August 2016 - Intel-nda - mailman3.cels.anl.gov

Re: [JLSE-Intel-NDA] trying to use cobalt
by Allen, Benjamin S. 08 Aug '16

08 Aug '16

Should be fixed now. Ben > On Aug 8, 2016, at 2:03 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > ulimit -l is set to the default 64k evidently. Will fix shortly. > > Ben > >> On Aug 8, 2016, at 1:49 PM, Steven C. Pieper <spieper(a)anl.gov> wrote: >> >> I can run a cobalt script on one node with -ppn 6 just fine. But >> if I try two nodes it fails with the following in stderr: >> >> knl08.ftm.alcf.anl.gov.148827Driver initialization failure on /dev/hfi1 (err=23) >> [6] DAPL startup: RLIMIT_MEMLOCK too small >> knl08.ftm.alcf.anl.gov.148832Driver initialization failure on /dev/hfi1 (err=23) >> [11] DAPL startup: RLIMIT_MEMLOCK too small >> knl08.ftm.alcf.anl.gov.148831Driver initialization failure on /dev/hfi1 (err=23) >> knl08.ftm.alcf.anl.gov.148828Driver initialization failure on /dev/hfi1 (err=23) >> [10] DAPL startup: RLIMIT_MEMLOCK too small >> knl08.ftm.alcf.anl.gov.148829Driver initialization failure on /dev/hfi1 (err=23) >> knl08.ftm.alcf.anl.gov.148830Driver initialization failure on /dev/hfi1 (err=23) >> [7] DAPL startup: RLIMIT_MEMLOCK too small >> [8] DAPL startup: RLIMIT_MEMLOCK too small >> [9] DAPL startup: RLIMIT_MEMLOCK too small >> Fatal error in PMPI_Init_thread: Other MPI error, error stack: >> MPIR_Init_thread(784)........: >> MPID_Init(1332)..............: channel initialization failed >> MPIDI_CH3_Init(136)..........: >> MPIDI_CH3_VC_Init(256).......: >> MPID_nem_vc_init(1927).......: >> MPID_nem_tmi_vc_init(1107)...: >> MPID_nem_tmi_vc_connect(942).: >> MPID_nem_tmi_get_from_bc(889): no space for the host description >> Fatal error in PMPI_Init_thread: Other MPI error, error stack: >> MPIR_Init_thread(784)........: >> MPID_Init(1332)..............: channel initialization failed >> MPIDI_CH3_Init(136)..........: >> MPIDI_CH3_VC_Init(256).......: >> MPID_nem_vc_init(1927).......: >> MPID_nem_tmi_vc_init(1107)...: >> MPID_nem_tmi_vc_connect(942).: >> MPID_nem_tmi_get_from_bc(889): no space for the host description >> Fatal error in PMPI_Init_thread: Other MPI error, error stack: >> MPIR_Init_thread(784)........: >> MPID_Init(1332)..............: channel initialization failed >> MPIDI_CH3_Init(136)..........: >> MPIDI_CH3_VC_Init(256).......: >> MPID_nem_vc_init(1927).......: >> MPID_nem_tmi_vc_init(1107)...: >> MPID_nem_tmi_vc_connect(942).: >> MPID_nem_tmi_get_from_bc(889): no space for the host description >> Fatal error in PMPI_Init_thread: Other MPI error, error stack: >> MPIR_Init_thread(784)........: >> MPID_Init(1332)..............: channel initialization failed >> MPIDI_CH3_Init(136)..........: >> MPIDI_CH3_VC_Init(256).......: >> MPID_nem_vc_init(1927).......: >> MPID_nem_tmi_vc_init(1107)...: >> MPID_nem_tmi_vc_connect(942).: >> MPID_nem_tmi_get_from_bc(889): no space for the host description >> Fatal error in PMPI_Init_thread: Other MPI error, error stack: >> MPIR_Init_thread(784)........: >> MPID_Init(1332)..............: channel initialization failed >> MPIDI_CH3_Init(136)..........: >> MPIDI_CH3_VC_Init(256).......: >> MPID_nem_vc_init(1927).......: >> MPID_nem_tmi_vc_init(1107)...: >> MPID_nem_tmi_vc_connect(942).: >> MPID_nem_tmi_get_from_bc(889): no space for the host description >> Fatal error in PMPI_Init_thread: Other MPI error, error stack: >> MPIR_Init_thread(784)........: >> MPID_Init(1332)..............: channel initialization failed >> MPIDI_CH3_Init(136)..........: >> MPIDI_CH3_VC_Init(256).......: >> MPID_nem_vc_init(1927).......: >> MPID_nem_tmi_vc_init(1107)...: >> MPID_nem_tmi_vc_connect(942).: >> MPID_nem_tmi_get_from_bc(889): no space for the host description >> >> >> my script is: >> >> #!/bin/bash >> >> source /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh intel64 >> source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh >> >> pwd >> date >> >> mpirun -f $COBALT_NODEFILE -ppn 6 -env OMP_NUM_THREADS=10 $HOME/gfmc/mc/x.mc.4he-pn he4-test-pn-03.vmc.in >> >> >> The pwd and date commands are executed. >> >> >> The qsub command is >> >> qsub -n 2 -t 10 -q testingknl -M spieper(a)anl.gov ./doit >> >> >> What is wrong? >> >> Steve >> >> >> -- >> Steven C. Pieper, Senior Physicist Emeritus: >> spieper(a)anl.gov >> >> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 >> >> http://www.phy.anl.gov/theory/staff/scp.html >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] trying to use cobalt
by Allen, Benjamin S. 08 Aug '16

08 Aug '16

ulimit -l is set to the default 64k evidently. Will fix shortly. Ben > On Aug 8, 2016, at 1:49 PM, Steven C. Pieper <spieper(a)anl.gov> wrote: > > I can run a cobalt script on one node with -ppn 6 just fine. But > if I try two nodes it fails with the following in stderr: > > knl08.ftm.alcf.anl.gov.148827Driver initialization failure on /dev/hfi1 (err=23) > [6] DAPL startup: RLIMIT_MEMLOCK too small > knl08.ftm.alcf.anl.gov.148832Driver initialization failure on /dev/hfi1 (err=23) > [11] DAPL startup: RLIMIT_MEMLOCK too small > knl08.ftm.alcf.anl.gov.148831Driver initialization failure on /dev/hfi1 (err=23) > knl08.ftm.alcf.anl.gov.148828Driver initialization failure on /dev/hfi1 (err=23) > [10] DAPL startup: RLIMIT_MEMLOCK too small > knl08.ftm.alcf.anl.gov.148829Driver initialization failure on /dev/hfi1 (err=23) > knl08.ftm.alcf.anl.gov.148830Driver initialization failure on /dev/hfi1 (err=23) > [7] DAPL startup: RLIMIT_MEMLOCK too small > [8] DAPL startup: RLIMIT_MEMLOCK too small > [9] DAPL startup: RLIMIT_MEMLOCK too small > Fatal error in PMPI_Init_thread: Other MPI error, error stack: > MPIR_Init_thread(784)........: > MPID_Init(1332)..............: channel initialization failed > MPIDI_CH3_Init(136)..........: > MPIDI_CH3_VC_Init(256).......: > MPID_nem_vc_init(1927).......: > MPID_nem_tmi_vc_init(1107)...: > MPID_nem_tmi_vc_connect(942).: > MPID_nem_tmi_get_from_bc(889): no space for the host description > Fatal error in PMPI_Init_thread: Other MPI error, error stack: > MPIR_Init_thread(784)........: > MPID_Init(1332)..............: channel initialization failed > MPIDI_CH3_Init(136)..........: > MPIDI_CH3_VC_Init(256).......: > MPID_nem_vc_init(1927).......: > MPID_nem_tmi_vc_init(1107)...: > MPID_nem_tmi_vc_connect(942).: > MPID_nem_tmi_get_from_bc(889): no space for the host description > Fatal error in PMPI_Init_thread: Other MPI error, error stack: > MPIR_Init_thread(784)........: > MPID_Init(1332)..............: channel initialization failed > MPIDI_CH3_Init(136)..........: > MPIDI_CH3_VC_Init(256).......: > MPID_nem_vc_init(1927).......: > MPID_nem_tmi_vc_init(1107)...: > MPID_nem_tmi_vc_connect(942).: > MPID_nem_tmi_get_from_bc(889): no space for the host description > Fatal error in PMPI_Init_thread: Other MPI error, error stack: > MPIR_Init_thread(784)........: > MPID_Init(1332)..............: channel initialization failed > MPIDI_CH3_Init(136)..........: > MPIDI_CH3_VC_Init(256).......: > MPID_nem_vc_init(1927).......: > MPID_nem_tmi_vc_init(1107)...: > MPID_nem_tmi_vc_connect(942).: > MPID_nem_tmi_get_from_bc(889): no space for the host description > Fatal error in PMPI_Init_thread: Other MPI error, error stack: > MPIR_Init_thread(784)........: > MPID_Init(1332)..............: channel initialization failed > MPIDI_CH3_Init(136)..........: > MPIDI_CH3_VC_Init(256).......: > MPID_nem_vc_init(1927).......: > MPID_nem_tmi_vc_init(1107)...: > MPID_nem_tmi_vc_connect(942).: > MPID_nem_tmi_get_from_bc(889): no space for the host description > Fatal error in PMPI_Init_thread: Other MPI error, error stack: > MPIR_Init_thread(784)........: > MPID_Init(1332)..............: channel initialization failed > MPIDI_CH3_Init(136)..........: > MPIDI_CH3_VC_Init(256).......: > MPID_nem_vc_init(1927).......: > MPID_nem_tmi_vc_init(1107)...: > MPID_nem_tmi_vc_connect(942).: > MPID_nem_tmi_get_from_bc(889): no space for the host description > > > my script is: > > #!/bin/bash > > source /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh intel64 > source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh > > pwd > date > > mpirun -f $COBALT_NODEFILE -ppn 6 -env OMP_NUM_THREADS=10 $HOME/gfmc/mc/x.mc.4he-pn he4-test-pn-03.vmc.in > > > The pwd and date commands are executed. > > > The qsub command is > > qsub -n 2 -t 10 -q testingknl -M spieper(a)anl.gov ./doit > > > What is wrong? > > Steve > > > -- > Steven C. Pieper, Senior Physicist Emeritus: > spieper(a)anl.gov > > Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 > Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 > > http://www.phy.anl.gov/theory/staff/scp.html > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] trying to use cobalt
by Steven C. Pieper 08 Aug '16

08 Aug '16

I can run a cobalt script on one node with -ppn 6 just fine. But if I try two nodes it fails with the following in stderr: knl08.ftm.alcf.anl.gov.148827Driver initialization failure on /dev/hfi1 (err=23) [6] DAPL startup: RLIMIT_MEMLOCK too small knl08.ftm.alcf.anl.gov.148832Driver initialization failure on /dev/hfi1 (err=23) [11] DAPL startup: RLIMIT_MEMLOCK too small knl08.ftm.alcf.anl.gov.148831Driver initialization failure on /dev/hfi1 (err=23) knl08.ftm.alcf.anl.gov.148828Driver initialization failure on /dev/hfi1 (err=23) [10] DAPL startup: RLIMIT_MEMLOCK too small knl08.ftm.alcf.anl.gov.148829Driver initialization failure on /dev/hfi1 (err=23) knl08.ftm.alcf.anl.gov.148830Driver initialization failure on /dev/hfi1 (err=23) [7] DAPL startup: RLIMIT_MEMLOCK too small [8] DAPL startup: RLIMIT_MEMLOCK too small [9] DAPL startup: RLIMIT_MEMLOCK too small Fatal error in PMPI_Init_thread: Other MPI error, error stack: MPIR_Init_thread(784)........: MPID_Init(1332)..............: channel initialization failed MPIDI_CH3_Init(136)..........: MPIDI_CH3_VC_Init(256).......: MPID_nem_vc_init(1927).......: MPID_nem_tmi_vc_init(1107)...: MPID_nem_tmi_vc_connect(942).: MPID_nem_tmi_get_from_bc(889): no space for the host description Fatal error in PMPI_Init_thread: Other MPI error, error stack: MPIR_Init_thread(784)........: MPID_Init(1332)..............: channel initialization failed MPIDI_CH3_Init(136)..........: MPIDI_CH3_VC_Init(256).......: MPID_nem_vc_init(1927).......: MPID_nem_tmi_vc_init(1107)...: MPID_nem_tmi_vc_connect(942).: MPID_nem_tmi_get_from_bc(889): no space for the host description Fatal error in PMPI_Init_thread: Other MPI error, error stack: MPIR_Init_thread(784)........: MPID_Init(1332)..............: channel initialization failed MPIDI_CH3_Init(136)..........: MPIDI_CH3_VC_Init(256).......: MPID_nem_vc_init(1927).......: MPID_nem_tmi_vc_init(1107)...: MPID_nem_tmi_vc_connect(942).: MPID_nem_tmi_get_from_bc(889): no space for the host description Fatal error in PMPI_Init_thread: Other MPI error, error stack: MPIR_Init_thread(784)........: MPID_Init(1332)..............: channel initialization failed MPIDI_CH3_Init(136)..........: MPIDI_CH3_VC_Init(256).......: MPID_nem_vc_init(1927).......: MPID_nem_tmi_vc_init(1107)...: MPID_nem_tmi_vc_connect(942).: MPID_nem_tmi_get_from_bc(889): no space for the host description Fatal error in PMPI_Init_thread: Other MPI error, error stack: MPIR_Init_thread(784)........: MPID_Init(1332)..............: channel initialization failed MPIDI_CH3_Init(136)..........: MPIDI_CH3_VC_Init(256).......: MPID_nem_vc_init(1927).......: MPID_nem_tmi_vc_init(1107)...: MPID_nem_tmi_vc_connect(942).: MPID_nem_tmi_get_from_bc(889): no space for the host description Fatal error in PMPI_Init_thread: Other MPI error, error stack: MPIR_Init_thread(784)........: MPID_Init(1332)..............: channel initialization failed MPIDI_CH3_Init(136)..........: MPIDI_CH3_VC_Init(256).......: MPID_nem_vc_init(1927).......: MPID_nem_tmi_vc_init(1107)...: MPID_nem_tmi_vc_connect(942).: MPID_nem_tmi_get_from_bc(889): no space for the host description my script is: #!/bin/bash source /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh intel64 source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh pwd date mpirun -f $COBALT_NODEFILE -ppn 6 -env OMP_NUM_THREADS=10 $HOME/gfmc/mc/x.mc.4he-pn he4-test-pn-03.vmc.in The pwd and date commands are executed. The qsub command is qsub -n 2 -t 10 -q testingknl -M spieper(a)anl.gov ./doit What is wrong? Steve -- Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 http://www.phy.anl.gov/theory/staff/scp.html

1 0

[JLSE-Intel-NDA] Cobalt Scheduling on KNL Nodes
by Allen, Benjamin S. 08 Aug '16

08 Aug '16

All, I'd like to move the rest the nodes to be scheduled via Cobalt this after noon at 2pm. This will involve rebooting all KNL nodes, and should be limited to about 2 hours of downtime. Please let me know if this interferes with anyone's work. Thanks, Ben

1 0

Re: [JLSE-Intel-NDA] Testing Cobalt Scheduling on knl07 and knl08
by Christopher Knight 05 Aug '16

05 Aug '16

I like the current behavior: if attributes not set, then run on first available node with whatever memory+cluster settings currently set (i.e. no reboot, just run). I’m able to submit jobs successfully now. Thanks, chris On Aug 5, 2016, at 11:31 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: If you don't specify the attributes for memory or cluster, Cobalt will pick any node and not modify the running settings. It doesn't have the ability today to schedule "resources", so we can't specify a requirement for the job to ideally pick a node that already has the needed settings. It simply picks the first available node(s). If we want a default configuration for memory and cluster modes to be set when a user doesn't specify a setting, we could do that too. Let me know what everyone thinks about this approach. Also what would the default be? Ben > On Aug 5, 2016, at 10:40 AM, Christopher Knight <knightc(a)anl.gov> wrote: > > Hi Ben, > > I’m getting “Host key verification failed.” errors. Am I goofing somewhere? > > Is it necessary to set attributes for memory+cluster or will not setting those let the job run on any available node? > > chris > > > =============== > > > [knightc@jlselogin2 daxpy]$ pwd > /home/knightc/blas/daxpy > > > qsub -n 1 -q testingknl -t 5 ./simple.cobalt > > > > [knightc@jlselogin2 daxpy]$ cat simple.cobalt > #!/bin/bash > > source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh > env OMP_NUM_THREADS=1 mpirun -f ${COBALT_NODEFILE} -ppn 1 ./test_daxpy > > > > > > On Aug 2, 2016, at 5:08 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > All, > > We have the Cobalt scheduler setup for the KNL nodes. For initial testing I've setup a queue called testingknl, that is limited to knl07 and knl08. Anyone that is in the "intel" unix group should be able to submit to it. No other restrictions have been put on the queue. > > We've added the ability to set MCDRAM Memory Mode and Cluster Mode via qsub attrs. This goes through the syscfg and power cycle process for you. Example: > > qsub -n 2 -q testingknl -t 10 --attrs knl_memory_mode=Flat:knl_cluster_mode=Quadrant /home/bsallen/test_mpi.sh > > The scheduler detects the current settings and won't power cycle the node if they're already correct. However it doesn't have the ability to pick available nodes that already have the correct settings. When the nodes need to reboot it takes about 10min 30secs before the job starts. This time isn't deducted from your walltime usage. > > Note the values for knl_memory_mode and knl_cluster_mode are case sensitive. The full list of accepted values: > > knl_memory_mode: > > Cache > Flat > Hybrid > Auto > > knl_cluster_mode: > > All2All > SNC-2 > SNC-4 > Hemisphere > Quadrant > Auto > > Example submission script: > > #!/bin/bash > > source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh > mpirun -f $COBALT_NODEFILE -ppn 64 $HOME/bin/mpi_hello > > Let me know what breaks. Once we're reasonably happy with it we'll schedule all the KNL nodes. > > Ben > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] Testing Cobalt Scheduling on knl07 and knl08
by Allen, Benjamin S. 05 Aug '16

05 Aug '16

If you don't specify the attributes for memory or cluster, Cobalt will pick any node and not modify the running settings. It doesn't have the ability today to schedule "resources", so we can't specify a requirement for the job to ideally pick a node that already has the needed settings. It simply picks the first available node(s). If we want a default configuration for memory and cluster modes to be set when a user doesn't specify a setting, we could do that too. Let me know what everyone thinks about this approach. Also what would the default be? Ben > On Aug 5, 2016, at 10:40 AM, Christopher Knight <knightc(a)anl.gov> wrote: > > Hi Ben, > > I’m getting “Host key verification failed.” errors. Am I goofing somewhere? > > Is it necessary to set attributes for memory+cluster or will not setting those let the job run on any available node? > > chris > > > =============== > > > [knightc@jlselogin2 daxpy]$ pwd > /home/knightc/blas/daxpy > > > qsub -n 1 -q testingknl -t 5 ./simple.cobalt > > > > [knightc@jlselogin2 daxpy]$ cat simple.cobalt > #!/bin/bash > > source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh > env OMP_NUM_THREADS=1 mpirun -f ${COBALT_NODEFILE} -ppn 1 ./test_daxpy > > > > > > On Aug 2, 2016, at 5:08 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > All, > > We have the Cobalt scheduler setup for the KNL nodes. For initial testing I've setup a queue called testingknl, that is limited to knl07 and knl08. Anyone that is in the "intel" unix group should be able to submit to it. No other restrictions have been put on the queue. > > We've added the ability to set MCDRAM Memory Mode and Cluster Mode via qsub attrs. This goes through the syscfg and power cycle process for you. Example: > > qsub -n 2 -q testingknl -t 10 --attrs knl_memory_mode=Flat:knl_cluster_mode=Quadrant /home/bsallen/test_mpi.sh > > The scheduler detects the current settings and won't power cycle the node if they're already correct. However it doesn't have the ability to pick available nodes that already have the correct settings. When the nodes need to reboot it takes about 10min 30secs before the job starts. This time isn't deducted from your walltime usage. > > Note the values for knl_memory_mode and knl_cluster_mode are case sensitive. The full list of accepted values: > > knl_memory_mode: > > Cache > Flat > Hybrid > Auto > > knl_cluster_mode: > > All2All > SNC-2 > SNC-4 > Hemisphere > Quadrant > Auto > > Example submission script: > > #!/bin/bash > > source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh > mpirun -f $COBALT_NODEFILE -ppn 64 $HOME/bin/mpi_hello > > Let me know what breaks. Once we're reasonably happy with it we'll schedule all the KNL nodes. > > Ben > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] gmon?
by Allen, Benjamin S. 05 Aug '16

05 Aug '16

Locate wastes quite a bit of metadata load to build its database. If we did enable it we wouldn't have it walk /soft. Will need a reference to which gmon you're referring to. Ben > On Aug 5, 2016, at 10:48 AM, Steven C. Pieper <spieper(a)anl.gov> wrote: > > Where is gmon on the knl nodes or gmon ? > > It would be nice to have the locate command! > > Thanks, > Steve > > -- > Steven C. Pieper, Senior Physicist Emeritus: > spieper(a)anl.gov > > Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 > Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 > > http://www.phy.anl.gov/theory/staff/scp.html > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Testing Cobalt Scheduling on knl07 and knl08
by Allen, Benjamin S. 05 Aug '16

05 Aug '16

Hi Chris, I missed populating /etc/ssh/ssh_known_hosts on the logins with the KNL machines. Try again, also ensure ~/.ssh/known_hosts is cleared of at least knl07 and knl08. Ben > On Aug 5, 2016, at 10:40 AM, Christopher Knight <knightc(a)anl.gov> wrote: > > Hi Ben, > > I’m getting “Host key verification failed.” errors. Am I goofing somewhere? > > Is it necessary to set attributes for memory+cluster or will not setting those let the job run on any available node? > > chris > > > =============== > > > [knightc@jlselogin2 daxpy]$ pwd > /home/knightc/blas/daxpy > > > qsub -n 1 -q testingknl -t 5 ./simple.cobalt > > > > [knightc@jlselogin2 daxpy]$ cat simple.cobalt > #!/bin/bash > > source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh > env OMP_NUM_THREADS=1 mpirun -f ${COBALT_NODEFILE} -ppn 1 ./test_daxpy > > > > > > On Aug 2, 2016, at 5:08 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > All, > > We have the Cobalt scheduler setup for the KNL nodes. For initial testing I've setup a queue called testingknl, that is limited to knl07 and knl08. Anyone that is in the "intel" unix group should be able to submit to it. No other restrictions have been put on the queue. > > We've added the ability to set MCDRAM Memory Mode and Cluster Mode via qsub attrs. This goes through the syscfg and power cycle process for you. Example: > > qsub -n 2 -q testingknl -t 10 --attrs knl_memory_mode=Flat:knl_cluster_mode=Quadrant /home/bsallen/test_mpi.sh > > The scheduler detects the current settings and won't power cycle the node if they're already correct. However it doesn't have the ability to pick available nodes that already have the correct settings. When the nodes need to reboot it takes about 10min 30secs before the job starts. This time isn't deducted from your walltime usage. > > Note the values for knl_memory_mode and knl_cluster_mode are case sensitive. The full list of accepted values: > > knl_memory_mode: > > Cache > Flat > Hybrid > Auto > > knl_cluster_mode: > > All2All > SNC-2 > SNC-4 > Hemisphere > Quadrant > Auto > > Example submission script: > > #!/bin/bash > > source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh > mpirun -f $COBALT_NODEFILE -ppn 64 $HOME/bin/mpi_hello > > Let me know what breaks. Once we're reasonably happy with it we'll schedule all the KNL nodes. > > Ben > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] gmon?
by Williams, Timothy J. 05 Aug '16

05 Aug '16

You mean gprof? If so, it’s /usr/bin/gprof ------- Timothy J. Williams, Ph.D. --- Argonne Leadership Computing Facility --- tjwilliams(a)anl.gov<mailto:[email protected]> --- 630-252-1154 ------- On Aug 5, 2016, at 10:48 AM, Steven C. Pieper <spieper(a)anl.gov<mailto:[email protected]>> wrote: Where is gmon on the knl nodes or gmon ? It would be nice to have the locate command! Thanks, Steve -- Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov<mailto:[email protected]> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 http://www.phy.anl.gov/theory/staff/scp.html _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] gmon?
by Steven C. Pieper 05 Aug '16

05 Aug '16

Where is gmon on the knl nodes or gmon ? It would be nice to have the locate command! Thanks, Steve -- Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 http://www.phy.anl.gov/theory/staff/scp.html

1 0