- Intel-nda - mailman3.cels.anl.gov

Re: [JLSE-Intel-NDA] jlse nodes
by Allen, Benjamin S. 23 Aug '16

23 Aug '16

Fixed the permissions on the knl_7250 queue, brought knl04 and knl09 back. knl03 and knl05 were panic'ed. Rebooting them now. I'll leave the policy question on Sarom Leang's 6 48-hour jobs to Tim and Kumar. Ben > On Aug 23, 2016, at 12:17 PM, Kenneth E Jansen <kenneth.jansen(a)colorado.edu> wrote: > > looks like 3 7210 nodes are down, a single user has submitted 6, 48 hour INTERACTIVE jobs to the only up 7210 nodes. > > one of the 7250 nodes is down, the other idle but trying to submit to it yields > > [kjansen@jlselogin1 64]$ qsub -n 1 -q knl_7250 -t 60 -I > <Fault 1001: "You are not allowed to submit to the 'knl_7250' queue\n"> > > > ok while writing this email it seems knl04 is back but are we really ok with 6 48 hour interactive one node jobs? > > And is it correct that I (other users too) can’t use the knl_7250 queue…thought I did before but maybe miss-remembering). > > Best regards, > > Ken > > ---------------------------------------------------------------- > Kenneth E. Jansen, Professor of Aerospace Engineering Sciences > ECAE 161 OFFICE (303) 492-4359 > 429 UCB FAX (303) 492-4990 > University of Colorado at Boulder jansenke(a)colorado.edu > Boulder, CO, 80309-0429 http://www.colorado.edu/aerospace > ---------------------------------------------------------------- > > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] jlse nodes
by Kenneth E Jansen 23 Aug '16

23 Aug '16

looks like 3 7210 nodes are down, a single user has submitted 6, 48 hour INTERACTIVE jobs to the only up 7210 nodes. one of the 7250 nodes is down, the other idle but trying to submit to it yields [kjansen@jlselogin1 64]$ qsub -n 1 -q knl_7250 -t 60 -I <Fault 1001: "You are not allowed to submit to the 'knl_7250' queue\n"> ok while writing this email it seems knl04 is back but are we really ok with 6 48 hour interactive one node jobs? And is it correct that I (other users too) can’t use the knl_7250 queue…thought I did before but maybe miss-remembering). Best regards, Ken ---------------------------------------------------------------- Kenneth E. Jansen, Professor of Aerospace Engineering Sciences ECAE 161 OFFICE (303) 492-4359 429 UCB FAX (303) 492-4990 University of Colorado at Boulder jansenke(a)colorado.edu Boulder, CO, 80309-0429 http://www.colorado.edu/aerospace ----------------------------------------------------------------

1 0

Re: [JLSE-Intel-NDA] JSLE KNL Scheduling Paused
by Allen, Benjamin S. 17 Aug '16

17 Aug '16

This is done, scheduling un-paused. Now at worst, if a node goes down when idle (between jobs), it will only cause one job to fail. Ben > On Aug 17, 2016, at 3:19 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > I've stopped scheduling for a bit while I test some changes to fix problems Steve and Ye Luo reported. Guessing less than 30 minutes. Running jobs are unaffected. > > Ben > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] JSLE KNL Scheduling Paused
by Allen, Benjamin S. 17 Aug '16

17 Aug '16

I've stopped scheduling for a bit while I test some changes to fix problems Steve and Ye Luo reported. Guessing less than 30 minutes. Running jobs are unaffected. Ben

1 0

Re: [JLSE-Intel-NDA] threading issue on KNL
by Kumaran, Kalyan 17 Aug '16

17 Aug '16

Maybe Jeff Hammond can help... > On Aug 17, 2016, at 1:41 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > We have very little experience with OmniPath so far. To use the OmniPath interfaces via Intel MPI add "-iface ib0 -PSM" to your mpirun. > > Unsure about the rank/thread placement questions. > > Ben > >> On Aug 17, 2016, at 12:35 PM, Luo, Ye <yeluo(a)anl.gov> wrote: >> >> Hi all, >> >> I just noticed a very strange behavior of the mpirun on KNL nodes. >> If I run 1 node with 1 MPI and 128 threads, everything goes well. >> If I run 2 nodes with 1 MPI and 128 threads on each nodes, all the 128 threads are actually running on a single core. >> This behavior is confirmed with KMP_AFFINITY=verbose. >> >> An extra question, is the currect mpirun aware of the omni path? I'm using the intel mpi comes with the compiler. >> >> Ye >> >> =================== >> Ye Luo, Ph.D. >> Leadership Computing Facility >> Argonne National Laboratory >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] threading issue on KNL
by Allen, Benjamin S. 17 Aug '16

17 Aug '16

We have very little experience with OmniPath so far. To use the OmniPath interfaces via Intel MPI add "-iface ib0 -PSM" to your mpirun. Unsure about the rank/thread placement questions. Ben > On Aug 17, 2016, at 12:35 PM, Luo, Ye <yeluo(a)anl.gov> wrote: > > Hi all, > > I just noticed a very strange behavior of the mpirun on KNL nodes. > If I run 1 node with 1 MPI and 128 threads, everything goes well. > If I run 2 nodes with 1 MPI and 128 threads on each nodes, all the 128 threads are actually running on a single core. > This behavior is confirmed with KMP_AFFINITY=verbose. > > An extra question, is the currect mpirun aware of the omni path? I'm using the intel mpi comes with the compiler. > > Ye > > =================== > Ye Luo, Ph.D. > Leadership Computing Facility > Argonne National Laboratory > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] threading issue on KNL
by Luo, Ye 17 Aug '16

17 Aug '16

Hi all, I just noticed a very strange behavior of the mpirun on KNL nodes. If I run 1 node with 1 MPI and 128 threads, everything goes well. If I run 2 nodes with 1 MPI and 128 threads on each nodes, all the 128 threads are actually running on a single core. This behavior is confirmed with KMP_AFFINITY=verbose. An extra question, is the currect mpirun aware of the omni path? I'm using the intel mpi comes with the compiler. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory

1 0

Re: [JLSE-Intel-NDA] immediate job exit on KNL nodes.
by Allen, Benjamin S. 17 Aug '16

17 Aug '16

knl05 went down (panic), and Cobalt kept scheduling jobs on it. Same issue as Steve reported. Don't quite have a fix for this yet. You have access to nodes were scheduled as part of job 548, the next job that comes along will correctly set permissions for the new job. I'll add to my todos to cleanup access after a failed job launch. Thanks, Ben > On Aug 17, 2016, at 10:05 AM, Luo, Ye <yeluo(a)anl.gov> wrote: > > Hi JLSE support, > > I'm not sure if this a known issue or not. > I noticed some strange behaviour last night on the cobalt for KNL. My jobs submitted to knl_7210 exited without starting. > I also noticed that I can ssh directly to several nodes without being allocated by cobalt for running jobs. > > The cobalt log and my job script is attached. > > Ye > > log: > Jobid: 548 > qsub --mode script -q knl_7210 -n 4 -t 120 -O FeSe --attrs knl_memory_mode=Cache:knl_cluster_mode=Quadrant qmcpack.sub > Wed Aug 17 02:48:30 2016 +0000 (UTC) submitted with cwd set to: /home/yeluo/SeTi-AB-collinear/dis_8.5/optimization-S1 > jobid 548 submitted from terminal /dev/pts/9 > Wed Aug 17 02:49:02 2016 +0000 (UTC) Info: resources released; starting job epilogue scripts > Jobid: 549 > qsub --mode script -q knl_7210 -n 1 -t 120 -O FeSe --attrs knl_memory_mode=Cache:knl_cluster_mode=Quadrant qmcpack.sub > Wed Aug 17 02:49:45 2016 +0000 (UTC) submitted with cwd set to: /home/yeluo/SeTi-AB-collinear/dis_8.5/optimization-S1 > jobid 549 submitted from terminal /dev/pts/9 > Wed Aug 17 02:50:03 2016 +0000 (UTC) Info: resources released; starting job epilogue scripts > Jobid: 550 > qsub --mode script -q knl_7210 -n 1 -t 120 -O FeSe --attrs knl_memory_mode=Cache:knl_cluster_mode=Quadrant qmcpack.sub > Wed Aug 17 02:53:31 2016 +0000 (UTC) submitted with cwd set to: /home/yeluo/SeTi-AB-collinear/dis_8.5/optimization-S1 > jobid 550 submitted from terminal /dev/pts/9 > Wed Aug 17 02:53:44 2016 +0000 (UTC) Info: resources released; starting job epilogue scripts > Jobid: 551 > qsub -q knl_7210 -n 1 -t 120 -O FeSe --attrs knl_memory_mode=Cache:knl_cluster_mode=Quadrant -I > Wed Aug 17 02:54:08 2016 +0000 (UTC) submitted with cwd set to: /home/yeluo/SeTi-AB-collinear/dis_8.5/optimization-S1 > jobid 551 submitted from terminal /dev/pts/9 > Wed Aug 17 02:54:35 2016 +0000 (UTC) Info: resources released; starting job epilogue scripts > > script: > #COBALT -q knl_7210 > #COBALT -n 2 > #COBALT -t 120 > #COBALT -O vmc > #COBALT --attrs knl_memory_mode=Cache:knl_cluster_mode=Quadrant > > =================== > Ye Luo, Ph.D. > Leadership Computing Facility > Argonne National Laboratory > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] immediate job exit on KNL nodes.
by Luo, Ye 17 Aug '16

17 Aug '16

Hi JLSE support, I'm not sure if this a known issue or not. I noticed some strange behaviour last night on the cobalt for KNL. My jobs submitted to knl_7210 exited without starting. I also noticed that I can ssh directly to several nodes without being allocated by cobalt for running jobs. The cobalt log and my job script is attached. Ye log: Jobid: 548 qsub --mode script -q knl_7210 -n 4 -t 120 -O FeSe --attrs knl_memory_mode=Cache:knl_cluster_mode=Quadrant qmcpack.sub Wed Aug 17 02:48:30 2016 +0000 (UTC) submitted with cwd set to: /home/yeluo/SeTi-AB-collinear/dis_8.5/optimization-S1 jobid 548 submitted from terminal /dev/pts/9 Wed Aug 17 02:49:02 2016 +0000 (UTC) Info: resources released; starting job epilogue scripts Jobid: 549 qsub --mode script -q knl_7210 -n 1 -t 120 -O FeSe --attrs knl_memory_mode=Cache:knl_cluster_mode=Quadrant qmcpack.sub Wed Aug 17 02:49:45 2016 +0000 (UTC) submitted with cwd set to: /home/yeluo/SeTi-AB-collinear/dis_8.5/optimization-S1 jobid 549 submitted from terminal /dev/pts/9 Wed Aug 17 02:50:03 2016 +0000 (UTC) Info: resources released; starting job epilogue scripts Jobid: 550 qsub --mode script -q knl_7210 -n 1 -t 120 -O FeSe --attrs knl_memory_mode=Cache:knl_cluster_mode=Quadrant qmcpack.sub Wed Aug 17 02:53:31 2016 +0000 (UTC) submitted with cwd set to: /home/yeluo/SeTi-AB-collinear/dis_8.5/optimization-S1 jobid 550 submitted from terminal /dev/pts/9 Wed Aug 17 02:53:44 2016 +0000 (UTC) Info: resources released; starting job epilogue scripts Jobid: 551 qsub -q knl_7210 -n 1 -t 120 -O FeSe --attrs knl_memory_mode=Cache:knl_cluster_mode=Quadrant -I Wed Aug 17 02:54:08 2016 +0000 (UTC) submitted with cwd set to: /home/yeluo/SeTi-AB-collinear/dis_8.5/optimization-S1 jobid 551 submitted from terminal /dev/pts/9 Wed Aug 17 02:54:35 2016 +0000 (UTC) Info: resources released; starting job epilogue scripts script: #COBALT -q knl_7210 #COBALT -n 2 #COBALT -t 120 #COBALT -O vmc #COBALT --attrs knl_memory_mode=Cache:knl_cluster_mode=Quadrant =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory

1 0

Re: [JLSE-Intel-NDA] knl03 in trouble?
by Allen, Benjamin S. 16 Aug '16

16 Aug '16

Looks like knl03 failed (likely kernel panic) sometime between its last job around 4:00 UTC and your jobs. Your first job should have trigger the node to be marked down however. We don't (today) have a way to reschedule a job when the node fails prologue, so in this case the first unlucky job will be sacrificed. Working on why the node wasn't marked down when the prologue failed. Ben > On Aug 16, 2016, at 10:26 AM, Tom LeCompte <lecompte(a)anl.gov> wrote: > > Hi everyone- > > My jobs on knl03 immediately exit. The same jobs run OK on 0-2 and 6-8. Is this a known problem? > > Cobalt doesn't say much: > > Tue Aug 16 15:20:40 2016 +0000 (UTC) submitted with cwd set to: /home/lecompte/g4/Geant4HepExpMTBenchmark/run > jobid 520 submitted from terminal /dev/pts/5 > Tue Aug 16 15:23:55 2016 +0000 (UTC) Info: resources released; starting job epilogue scripts > > Cheers, > > Tom > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0