August 2016 - Intel-nda - mailman3.cels.anl.gov

Re: [JLSE-Intel-NDA] questions about queues
by Allen, Benjamin S. 09 Aug '16

09 Aug '16

This should be fixed. Time was off on the logins. Ben > On Aug 9, 2016, at 10:56 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > Also I'll look into the odd time related issues in queue time and run time. > > Ben > >> On Aug 9, 2016, at 10:53 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >> >> Hi Steve, >> >> As of yesterday we're no longer testing Cobalt, so we've moved to the knl_7210 and knl_7250 queues. These queues are documented in https://wiki.jlse.anl.gov/display/knl/Intel+KNL. >> >> I'll remove the testingknl queue shortly to avoid confusion. >> >> Ben >> >>> On Aug 9, 2016, at 10:16 AM, Steven C. Pieper <spieper(a)anl.gov> wrote: >>> >>> 1) The testingknl queue is stopped. I found that I can use knl_7210. >>> Should I not do this? When will testingknl be started. >>> >>> >>> Name Users Groups MinTime MaxTime MaxRunning MaxQueued MaxUserNodes MaxNodeHours TotalNodes State >>> ===================================================================================================================== >>> knl_7210 None intel None None None None 9 None None running >>> knl_7250 None intel None None None None None None None running >>> testingknl None intel None None None None None None None stopped >>> >>> >>> 2) The times are messed up: >>> >>> JobID JobName User WallTime QueuedTime RunTime TimeRemaining Nodes State Location Mode Procs Preemptable Queue StartTime Index >>> ===================================================================================================================================================================================================== >>> 136 N/A lecompte 01:00:00 00:00:09 01:12:26 00:00:00 1 running knl08 script 1 False knl_7210 Tue Aug 09 09:59:24 2016 -0500 (CDT) None >>> 137 N/A lecompte 01:00:00 00:00:03 01:11:45 00:00:00 1 running knl04 script 1 False knl_7210 Tue Aug 09 10:00:05 2016 -0500 (CDT) None >>> 138 N/A lecompte 01:00:00 00:00:03 01:11:15 00:00:00 1 running knl05 script 1 False knl_7210 Tue Aug 09 10:00:36 2016 -0500 (CDT) None >>> 139 N/A lecompte 01:00:00 00:00:02 01:10:54 00:00:00 1 running knl06 script 1 False knl_7210 Tue Aug 09 10:00:56 2016 -0500 (CDT) None >>> 140 N/A lecompte 01:00:00 00:00:09 01:10:23 00:00:00 1 running knl07 script 1 False knl_7210 Tue Aug 09 10:01:27 2016 -0500 (CDT) None >>> 143 he4-test-pn-06b.vmc b21541 00:05:00 01:04:31 N/A N/A 1 queued None script 1 False testingknl N/A None >>> 144 he4-test-pn-06b.vmc b21541 00:05:00 00:00:08 01:00:44 00:00:00 1 starting knl00 script 1 False knl_7210 Tue Aug 09 10:11:06 2016 -0500 (CDT) None >>> >>> >>> Job 143 was submitted only 12 minutes ago but it shows up as being in the >>> queue 1h 4m. Lecompte's jobs appear to have exceeded their time but >>> are still running. >>> >>> Steve >>> >>> >>> >>> -- >>> Steven C. Pieper, Senior Physicist Emeritus: >>> spieper(a)anl.gov >>> >>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >>> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 >>> >>> http://www.phy.anl.gov/theory/staff/scp.html >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] questions about queues
by Allen, Benjamin S. 09 Aug '16

09 Aug '16

Also I'll look into the odd time related issues in queue time and run time. Ben > On Aug 9, 2016, at 10:53 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > Hi Steve, > > As of yesterday we're no longer testing Cobalt, so we've moved to the knl_7210 and knl_7250 queues. These queues are documented in https://wiki.jlse.anl.gov/display/knl/Intel+KNL. > > I'll remove the testingknl queue shortly to avoid confusion. > > Ben > >> On Aug 9, 2016, at 10:16 AM, Steven C. Pieper <spieper(a)anl.gov> wrote: >> >> 1) The testingknl queue is stopped. I found that I can use knl_7210. >> Should I not do this? When will testingknl be started. >> >> >> Name Users Groups MinTime MaxTime MaxRunning MaxQueued MaxUserNodes MaxNodeHours TotalNodes State >> ===================================================================================================================== >> knl_7210 None intel None None None None 9 None None running >> knl_7250 None intel None None None None None None None running >> testingknl None intel None None None None None None None stopped >> >> >> 2) The times are messed up: >> >> JobID JobName User WallTime QueuedTime RunTime TimeRemaining Nodes State Location Mode Procs Preemptable Queue StartTime Index >> ===================================================================================================================================================================================================== >> 136 N/A lecompte 01:00:00 00:00:09 01:12:26 00:00:00 1 running knl08 script 1 False knl_7210 Tue Aug 09 09:59:24 2016 -0500 (CDT) None >> 137 N/A lecompte 01:00:00 00:00:03 01:11:45 00:00:00 1 running knl04 script 1 False knl_7210 Tue Aug 09 10:00:05 2016 -0500 (CDT) None >> 138 N/A lecompte 01:00:00 00:00:03 01:11:15 00:00:00 1 running knl05 script 1 False knl_7210 Tue Aug 09 10:00:36 2016 -0500 (CDT) None >> 139 N/A lecompte 01:00:00 00:00:02 01:10:54 00:00:00 1 running knl06 script 1 False knl_7210 Tue Aug 09 10:00:56 2016 -0500 (CDT) None >> 140 N/A lecompte 01:00:00 00:00:09 01:10:23 00:00:00 1 running knl07 script 1 False knl_7210 Tue Aug 09 10:01:27 2016 -0500 (CDT) None >> 143 he4-test-pn-06b.vmc b21541 00:05:00 01:04:31 N/A N/A 1 queued None script 1 False testingknl N/A None >> 144 he4-test-pn-06b.vmc b21541 00:05:00 00:00:08 01:00:44 00:00:00 1 starting knl00 script 1 False knl_7210 Tue Aug 09 10:11:06 2016 -0500 (CDT) None >> >> >> Job 143 was submitted only 12 minutes ago but it shows up as being in the >> queue 1h 4m. Lecompte's jobs appear to have exceeded their time but >> are still running. >> >> Steve >> >> >> >> -- >> Steven C. Pieper, Senior Physicist Emeritus: >> spieper(a)anl.gov >> >> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 >> >> http://www.phy.anl.gov/theory/staff/scp.html >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] questions about queues
by Allen, Benjamin S. 09 Aug '16

09 Aug '16

Hi Steve, As of yesterday we're no longer testing Cobalt, so we've moved to the knl_7210 and knl_7250 queues. These queues are documented in https://wiki.jlse.anl.gov/display/knl/Intel+KNL. I'll remove the testingknl queue shortly to avoid confusion. Ben > On Aug 9, 2016, at 10:16 AM, Steven C. Pieper <spieper(a)anl.gov> wrote: > > 1) The testingknl queue is stopped. I found that I can use knl_7210. > Should I not do this? When will testingknl be started. > > > Name Users Groups MinTime MaxTime MaxRunning MaxQueued MaxUserNodes MaxNodeHours TotalNodes State > ===================================================================================================================== > knl_7210 None intel None None None None 9 None None running > knl_7250 None intel None None None None None None None running > testingknl None intel None None None None None None None stopped > > > 2) The times are messed up: > > JobID JobName User WallTime QueuedTime RunTime TimeRemaining Nodes State Location Mode Procs Preemptable Queue StartTime Index > ===================================================================================================================================================================================================== > 136 N/A lecompte 01:00:00 00:00:09 01:12:26 00:00:00 1 running knl08 script 1 False knl_7210 Tue Aug 09 09:59:24 2016 -0500 (CDT) None > 137 N/A lecompte 01:00:00 00:00:03 01:11:45 00:00:00 1 running knl04 script 1 False knl_7210 Tue Aug 09 10:00:05 2016 -0500 (CDT) None > 138 N/A lecompte 01:00:00 00:00:03 01:11:15 00:00:00 1 running knl05 script 1 False knl_7210 Tue Aug 09 10:00:36 2016 -0500 (CDT) None > 139 N/A lecompte 01:00:00 00:00:02 01:10:54 00:00:00 1 running knl06 script 1 False knl_7210 Tue Aug 09 10:00:56 2016 -0500 (CDT) None > 140 N/A lecompte 01:00:00 00:00:09 01:10:23 00:00:00 1 running knl07 script 1 False knl_7210 Tue Aug 09 10:01:27 2016 -0500 (CDT) None > 143 he4-test-pn-06b.vmc b21541 00:05:00 01:04:31 N/A N/A 1 queued None script 1 False testingknl N/A None > 144 he4-test-pn-06b.vmc b21541 00:05:00 00:00:08 01:00:44 00:00:00 1 starting knl00 script 1 False knl_7210 Tue Aug 09 10:11:06 2016 -0500 (CDT) None > > > Job 143 was submitted only 12 minutes ago but it shows up as being in the > queue 1h 4m. Lecompte's jobs appear to have exceeded their time but > are still running. > > Steve > > > > -- > Steven C. Pieper, Senior Physicist Emeritus: > spieper(a)anl.gov > > Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 > Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 > > http://www.phy.anl.gov/theory/staff/scp.html > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] questions about queues
by Steven C. Pieper 09 Aug '16

09 Aug '16

1) The testingknl queue is stopped. I found that I can use knl_7210. Should I not do this? When will testingknl be started. Name Users Groups MinTime MaxTime MaxRunning MaxQueued MaxUserNodes MaxNodeHours TotalNodes State ===================================================================================================================== knl_7210 None intel None None None None 9 None None running knl_7250 None intel None None None None None None None running testingknl None intel None None None None None None None stopped 2) The times are messed up: JobID JobName User WallTime QueuedTime RunTime TimeRemaining Nodes State Location Mode Procs Preemptable Queue StartTime Index ===================================================================================================================================================================================================== 136 N/A lecompte 01:00:00 00:00:09 01:12:26 00:00:00 1 running knl08 script 1 False knl_7210 Tue Aug 09 09:59:24 2016 -0500 (CDT) None 137 N/A lecompte 01:00:00 00:00:03 01:11:45 00:00:00 1 running knl04 script 1 False knl_7210 Tue Aug 09 10:00:05 2016 -0500 (CDT) None 138 N/A lecompte 01:00:00 00:00:03 01:11:15 00:00:00 1 running knl05 script 1 False knl_7210 Tue Aug 09 10:00:36 2016 -0500 (CDT) None 139 N/A lecompte 01:00:00 00:00:02 01:10:54 00:00:00 1 running knl06 script 1 False knl_7210 Tue Aug 09 10:00:56 2016 -0500 (CDT) None 140 N/A lecompte 01:00:00 00:00:09 01:10:23 00:00:00 1 running knl07 script 1 False knl_7210 Tue Aug 09 10:01:27 2016 -0500 (CDT) None 143 he4-test-pn-06b.vmc b21541 00:05:00 01:04:31 N/A N/A 1 queued None script 1 False testingknl N/A None 144 he4-test-pn-06b.vmc b21541 00:05:00 00:00:08 01:00:44 00:00:00 1 starting knl00 script 1 False knl_7210 Tue Aug 09 10:11:06 2016 -0500 (CDT) None Job 143 was submitted only 12 minutes ago but it shows up as being in the queue 1h 4m. Lecompte's jobs appear to have exceeded their time but are still running. Steve -- Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 http://www.phy.anl.gov/theory/staff/scp.html

1 0

Re: [JLSE-Intel-NDA] Cobalt Scheduling on KNL Nodes
by Allen, Benjamin S. 08 Aug '16

08 Aug '16

Forgot to mention, JLSE traditionally hasn't allowed outbound email from the environment. As such for now, job email notifications won't be delivered. Please don't include your email address in job submissions. I'll let you know when this issue is corrected. Ben > On Aug 8, 2016, at 5:23 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > This work is complete. Please see https://wiki.jlse.anl.gov/display/knl/Intel+KNL for examples on how to use the scheduler. > > Please let me know what breaks. > > Ben > >> On Aug 8, 2016, at 2:12 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >> >> Starting this work now. >> >> Ben >> >>> On Aug 8, 2016, at 10:58 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>> >>> All, >>> >>> I'd like to move the rest the nodes to be scheduled via Cobalt this after noon at 2pm. This will involve rebooting all KNL nodes, and should be limited to about 2 hours of downtime. >>> >>> Please let me know if this interferes with anyone's work. >>> >>> Thanks, >>> >>> Ben >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Cobalt Scheduling on KNL Nodes
by Allen, Benjamin S. 08 Aug '16

08 Aug '16

In the submission script you can add: #COBALT --attrs knl_memory_mode=Flat:knl_cluster_mode=Quadrant Full example: #!/bin/bash #COBALT --attrs knl_memory_mode=Flat:knl_cluster_mode=SNC-4 source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh mpirun -f $COBALT_NODEFILE -perhost 64 $HOME/bin/mpi_hello --- Note if you happen to specify both a value in the script and one on the CLI, the CLI wins. Ben > On Aug 8, 2016, at 5:34 PM, Luo, Ye <yeluo(a)anl.gov> wrote: > > I just read though the wiki page and didn't find much description about script mode of cobalt. > I'd like to know how to specify --attrs knl_memory_mode=Flat:knl_cluster_mode=Quadrant in a script. > Thanks, Ye. > =================== > Ye Luo, Ph.D. > Leadership Computing Facility > Argonne National Laboratory > > ________________________________________ > From: intel-nda-bounces(a)lists.jlse.anl.gov [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Allen, Benjamin S. [bsallen(a)alcf.anl.gov] > Sent: Monday, August 08, 2016 5:23 PM > To: intel-nda(a)lists.jlse.anl.gov > Subject: Re: [JLSE-Intel-NDA] Cobalt Scheduling on KNL Nodes > > This work is complete. Please see https://wiki.jlse.anl.gov/display/knl/Intel+KNL for examples on how to use the scheduler. > > Please let me know what breaks. > > Ben > >> On Aug 8, 2016, at 2:12 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >> >> Starting this work now. >> >> Ben >> >>> On Aug 8, 2016, at 10:58 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>> >>> All, >>> >>> I'd like to move the rest the nodes to be scheduled via Cobalt this after noon at 2pm. This will involve rebooting all KNL nodes, and should be limited to about 2 hours of downtime. >>> >>> Please let me know if this interferes with anyone's work. >>> >>> Thanks, >>> >>> Ben >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Cobalt Scheduling on KNL Nodes
by Luo, Ye 08 Aug '16

08 Aug '16

I just read though the wiki page and didn't find much description about script mode of cobalt. I'd like to know how to specify --attrs knl_memory_mode=Flat:knl_cluster_mode=Quadrant in a script. Thanks, Ye. =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory ________________________________________ From: intel-nda-bounces(a)lists.jlse.anl.gov [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Allen, Benjamin S. [bsallen(a)alcf.anl.gov] Sent: Monday, August 08, 2016 5:23 PM To: intel-nda(a)lists.jlse.anl.gov Subject: Re: [JLSE-Intel-NDA] Cobalt Scheduling on KNL Nodes This work is complete. Please see https://wiki.jlse.anl.gov/display/knl/Intel+KNL for examples on how to use the scheduler. Please let me know what breaks. Ben > On Aug 8, 2016, at 2:12 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > Starting this work now. > > Ben > >> On Aug 8, 2016, at 10:58 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >> >> All, >> >> I'd like to move the rest the nodes to be scheduled via Cobalt this after noon at 2pm. This will involve rebooting all KNL nodes, and should be limited to about 2 hours of downtime. >> >> Please let me know if this interferes with anyone's work. >> >> Thanks, >> >> Ben >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Cobalt Scheduling on KNL Nodes
by Allen, Benjamin S. 08 Aug '16

08 Aug '16

This work is complete. Please see https://wiki.jlse.anl.gov/display/knl/Intel+KNL for examples on how to use the scheduler. Please let me know what breaks. Ben > On Aug 8, 2016, at 2:12 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > Starting this work now. > > Ben > >> On Aug 8, 2016, at 10:58 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >> >> All, >> >> I'd like to move the rest the nodes to be scheduled via Cobalt this after noon at 2pm. This will involve rebooting all KNL nodes, and should be limited to about 2 hours of downtime. >> >> Please let me know if this interferes with anyone's work. >> >> Thanks, >> >> Ben >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] trying to use cobalt
by Steven C. Pieper 08 Aug '16

08 Aug '16

OK, that fixed it. Thanks Steve Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 http://www.phy.anl.gov/theory/staff/scp.html On 08/08/16 14:08, Allen, Benjamin S. wrote: > Should be fixed now. > > Ben > >> On Aug 8, 2016, at 2:03 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >> >> ulimit -l is set to the default 64k evidently. Will fix shortly. >> >> Ben >> >>> On Aug 8, 2016, at 1:49 PM, Steven C. Pieper <spieper(a)anl.gov> wrote: >>> >>> I can run a cobalt script on one node with -ppn 6 just fine. But >>> if I try two nodes it fails with the following in stderr: >>> >>> knl08.ftm.alcf.anl.gov.148827Driver initialization failure on /dev/hfi1 (err=23) >>> [6] DAPL startup: RLIMIT_MEMLOCK too small >>> knl08.ftm.alcf.anl.gov.148832Driver initialization failure on /dev/hfi1 (err=23) >>> [11] DAPL startup: RLIMIT_MEMLOCK too small >>> knl08.ftm.alcf.anl.gov.148831Driver initialization failure on /dev/hfi1 (err=23) >>> knl08.ftm.alcf.anl.gov.148828Driver initialization failure on /dev/hfi1 (err=23) >>> [10] DAPL startup: RLIMIT_MEMLOCK too small >>> knl08.ftm.alcf.anl.gov.148829Driver initialization failure on /dev/hfi1 (err=23) >>> knl08.ftm.alcf.anl.gov.148830Driver initialization failure on /dev/hfi1 (err=23) >>> [7] DAPL startup: RLIMIT_MEMLOCK too small >>> [8] DAPL startup: RLIMIT_MEMLOCK too small >>> [9] DAPL startup: RLIMIT_MEMLOCK too small >>> Fatal error in PMPI_Init_thread: Other MPI error, error stack: >>> MPIR_Init_thread(784)........: >>> MPID_Init(1332)..............: channel initialization failed >>> MPIDI_CH3_Init(136)..........: >>> MPIDI_CH3_VC_Init(256).......: >>> MPID_nem_vc_init(1927).......: >>> MPID_nem_tmi_vc_init(1107)...: >>> MPID_nem_tmi_vc_connect(942).: >>> MPID_nem_tmi_get_from_bc(889): no space for the host description >>> Fatal error in PMPI_Init_thread: Other MPI error, error stack: >>> MPIR_Init_thread(784)........: >>> MPID_Init(1332)..............: channel initialization failed >>> MPIDI_CH3_Init(136)..........: >>> MPIDI_CH3_VC_Init(256).......: >>> MPID_nem_vc_init(1927).......: >>> MPID_nem_tmi_vc_init(1107)...: >>> MPID_nem_tmi_vc_connect(942).: >>> MPID_nem_tmi_get_from_bc(889): no space for the host description >>> Fatal error in PMPI_Init_thread: Other MPI error, error stack: >>> MPIR_Init_thread(784)........: >>> MPID_Init(1332)..............: channel initialization failed >>> MPIDI_CH3_Init(136)..........: >>> MPIDI_CH3_VC_Init(256).......: >>> MPID_nem_vc_init(1927).......: >>> MPID_nem_tmi_vc_init(1107)...: >>> MPID_nem_tmi_vc_connect(942).: >>> MPID_nem_tmi_get_from_bc(889): no space for the host description >>> Fatal error in PMPI_Init_thread: Other MPI error, error stack: >>> MPIR_Init_thread(784)........: >>> MPID_Init(1332)..............: channel initialization failed >>> MPIDI_CH3_Init(136)..........: >>> MPIDI_CH3_VC_Init(256).......: >>> MPID_nem_vc_init(1927).......: >>> MPID_nem_tmi_vc_init(1107)...: >>> MPID_nem_tmi_vc_connect(942).: >>> MPID_nem_tmi_get_from_bc(889): no space for the host description >>> Fatal error in PMPI_Init_thread: Other MPI error, error stack: >>> MPIR_Init_thread(784)........: >>> MPID_Init(1332)..............: channel initialization failed >>> MPIDI_CH3_Init(136)..........: >>> MPIDI_CH3_VC_Init(256).......: >>> MPID_nem_vc_init(1927).......: >>> MPID_nem_tmi_vc_init(1107)...: >>> MPID_nem_tmi_vc_connect(942).: >>> MPID_nem_tmi_get_from_bc(889): no space for the host description >>> Fatal error in PMPI_Init_thread: Other MPI error, error stack: >>> MPIR_Init_thread(784)........: >>> MPID_Init(1332)..............: channel initialization failed >>> MPIDI_CH3_Init(136)..........: >>> MPIDI_CH3_VC_Init(256).......: >>> MPID_nem_vc_init(1927).......: >>> MPID_nem_tmi_vc_init(1107)...: >>> MPID_nem_tmi_vc_connect(942).: >>> MPID_nem_tmi_get_from_bc(889): no space for the host description >>> >>> >>> my script is: >>> >>> #!/bin/bash >>> >>> source /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh intel64 >>> source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh >>> >>> pwd >>> date >>> >>> mpirun -f $COBALT_NODEFILE -ppn 6 -env OMP_NUM_THREADS=10 $HOME/gfmc/mc/x.mc.4he-pn he4-test-pn-03.vmc.in >>> >>> >>> The pwd and date commands are executed. >>> >>> >>> The qsub command is >>> >>> qsub -n 2 -t 10 -q testingknl -M spieper(a)anl.gov ./doit >>> >>> >>> What is wrong? >>> >>> Steve >>> >>> >>> -- >>> Steven C. Pieper, Senior Physicist Emeritus: >>> spieper(a)anl.gov >>> >>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >>> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 >>> >>> http://www.phy.anl.gov/theory/staff/scp.html >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Cobalt Scheduling on KNL Nodes
by Allen, Benjamin S. 08 Aug '16

08 Aug '16

Starting this work now. Ben > On Aug 8, 2016, at 10:58 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > All, > > I'd like to move the rest the nodes to be scheduled via Cobalt this after noon at 2pm. This will involve rebooting all KNL nodes, and should be limited to about 2 hours of downtime. > > Please let me know if this interferes with anyone's work. > > Thanks, > > Ben > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0