August 2016 - Intel-nda - mailman3.cels.anl.gov

Re: [JLSE-Intel-NDA] threading issue on KNL
by Allen, Benjamin S. 17 Aug '16

17 Aug '16

We have very little experience with OmniPath so far. To use the OmniPath interfaces via Intel MPI add "-iface ib0 -PSM" to your mpirun. Unsure about the rank/thread placement questions. Ben > On Aug 17, 2016, at 12:35 PM, Luo, Ye <yeluo(a)anl.gov> wrote: > > Hi all, > > I just noticed a very strange behavior of the mpirun on KNL nodes. > If I run 1 node with 1 MPI and 128 threads, everything goes well. > If I run 2 nodes with 1 MPI and 128 threads on each nodes, all the 128 threads are actually running on a single core. > This behavior is confirmed with KMP_AFFINITY=verbose. > > An extra question, is the currect mpirun aware of the omni path? I'm using the intel mpi comes with the compiler. > > Ye > > =================== > Ye Luo, Ph.D. > Leadership Computing Facility > Argonne National Laboratory > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] threading issue on KNL
by Luo, Ye 17 Aug '16

17 Aug '16

Hi all, I just noticed a very strange behavior of the mpirun on KNL nodes. If I run 1 node with 1 MPI and 128 threads, everything goes well. If I run 2 nodes with 1 MPI and 128 threads on each nodes, all the 128 threads are actually running on a single core. This behavior is confirmed with KMP_AFFINITY=verbose. An extra question, is the currect mpirun aware of the omni path? I'm using the intel mpi comes with the compiler. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory

1 0

Re: [JLSE-Intel-NDA] immediate job exit on KNL nodes.
by Allen, Benjamin S. 17 Aug '16

17 Aug '16

knl05 went down (panic), and Cobalt kept scheduling jobs on it. Same issue as Steve reported. Don't quite have a fix for this yet. You have access to nodes were scheduled as part of job 548, the next job that comes along will correctly set permissions for the new job. I'll add to my todos to cleanup access after a failed job launch. Thanks, Ben > On Aug 17, 2016, at 10:05 AM, Luo, Ye <yeluo(a)anl.gov> wrote: > > Hi JLSE support, > > I'm not sure if this a known issue or not. > I noticed some strange behaviour last night on the cobalt for KNL. My jobs submitted to knl_7210 exited without starting. > I also noticed that I can ssh directly to several nodes without being allocated by cobalt for running jobs. > > The cobalt log and my job script is attached. > > Ye > > log: > Jobid: 548 > qsub --mode script -q knl_7210 -n 4 -t 120 -O FeSe --attrs knl_memory_mode=Cache:knl_cluster_mode=Quadrant qmcpack.sub > Wed Aug 17 02:48:30 2016 +0000 (UTC) submitted with cwd set to: /home/yeluo/SeTi-AB-collinear/dis_8.5/optimization-S1 > jobid 548 submitted from terminal /dev/pts/9 > Wed Aug 17 02:49:02 2016 +0000 (UTC) Info: resources released; starting job epilogue scripts > Jobid: 549 > qsub --mode script -q knl_7210 -n 1 -t 120 -O FeSe --attrs knl_memory_mode=Cache:knl_cluster_mode=Quadrant qmcpack.sub > Wed Aug 17 02:49:45 2016 +0000 (UTC) submitted with cwd set to: /home/yeluo/SeTi-AB-collinear/dis_8.5/optimization-S1 > jobid 549 submitted from terminal /dev/pts/9 > Wed Aug 17 02:50:03 2016 +0000 (UTC) Info: resources released; starting job epilogue scripts > Jobid: 550 > qsub --mode script -q knl_7210 -n 1 -t 120 -O FeSe --attrs knl_memory_mode=Cache:knl_cluster_mode=Quadrant qmcpack.sub > Wed Aug 17 02:53:31 2016 +0000 (UTC) submitted with cwd set to: /home/yeluo/SeTi-AB-collinear/dis_8.5/optimization-S1 > jobid 550 submitted from terminal /dev/pts/9 > Wed Aug 17 02:53:44 2016 +0000 (UTC) Info: resources released; starting job epilogue scripts > Jobid: 551 > qsub -q knl_7210 -n 1 -t 120 -O FeSe --attrs knl_memory_mode=Cache:knl_cluster_mode=Quadrant -I > Wed Aug 17 02:54:08 2016 +0000 (UTC) submitted with cwd set to: /home/yeluo/SeTi-AB-collinear/dis_8.5/optimization-S1 > jobid 551 submitted from terminal /dev/pts/9 > Wed Aug 17 02:54:35 2016 +0000 (UTC) Info: resources released; starting job epilogue scripts > > script: > #COBALT -q knl_7210 > #COBALT -n 2 > #COBALT -t 120 > #COBALT -O vmc > #COBALT --attrs knl_memory_mode=Cache:knl_cluster_mode=Quadrant > > =================== > Ye Luo, Ph.D. > Leadership Computing Facility > Argonne National Laboratory > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] immediate job exit on KNL nodes.
by Luo, Ye 17 Aug '16

17 Aug '16

Hi JLSE support, I'm not sure if this a known issue or not. I noticed some strange behaviour last night on the cobalt for KNL. My jobs submitted to knl_7210 exited without starting. I also noticed that I can ssh directly to several nodes without being allocated by cobalt for running jobs. The cobalt log and my job script is attached. Ye log: Jobid: 548 qsub --mode script -q knl_7210 -n 4 -t 120 -O FeSe --attrs knl_memory_mode=Cache:knl_cluster_mode=Quadrant qmcpack.sub Wed Aug 17 02:48:30 2016 +0000 (UTC) submitted with cwd set to: /home/yeluo/SeTi-AB-collinear/dis_8.5/optimization-S1 jobid 548 submitted from terminal /dev/pts/9 Wed Aug 17 02:49:02 2016 +0000 (UTC) Info: resources released; starting job epilogue scripts Jobid: 549 qsub --mode script -q knl_7210 -n 1 -t 120 -O FeSe --attrs knl_memory_mode=Cache:knl_cluster_mode=Quadrant qmcpack.sub Wed Aug 17 02:49:45 2016 +0000 (UTC) submitted with cwd set to: /home/yeluo/SeTi-AB-collinear/dis_8.5/optimization-S1 jobid 549 submitted from terminal /dev/pts/9 Wed Aug 17 02:50:03 2016 +0000 (UTC) Info: resources released; starting job epilogue scripts Jobid: 550 qsub --mode script -q knl_7210 -n 1 -t 120 -O FeSe --attrs knl_memory_mode=Cache:knl_cluster_mode=Quadrant qmcpack.sub Wed Aug 17 02:53:31 2016 +0000 (UTC) submitted with cwd set to: /home/yeluo/SeTi-AB-collinear/dis_8.5/optimization-S1 jobid 550 submitted from terminal /dev/pts/9 Wed Aug 17 02:53:44 2016 +0000 (UTC) Info: resources released; starting job epilogue scripts Jobid: 551 qsub -q knl_7210 -n 1 -t 120 -O FeSe --attrs knl_memory_mode=Cache:knl_cluster_mode=Quadrant -I Wed Aug 17 02:54:08 2016 +0000 (UTC) submitted with cwd set to: /home/yeluo/SeTi-AB-collinear/dis_8.5/optimization-S1 jobid 551 submitted from terminal /dev/pts/9 Wed Aug 17 02:54:35 2016 +0000 (UTC) Info: resources released; starting job epilogue scripts script: #COBALT -q knl_7210 #COBALT -n 2 #COBALT -t 120 #COBALT -O vmc #COBALT --attrs knl_memory_mode=Cache:knl_cluster_mode=Quadrant =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory

1 0

Re: [JLSE-Intel-NDA] knl03 in trouble?
by Allen, Benjamin S. 16 Aug '16

16 Aug '16

Looks like knl03 failed (likely kernel panic) sometime between its last job around 4:00 UTC and your jobs. Your first job should have trigger the node to be marked down however. We don't (today) have a way to reschedule a job when the node fails prologue, so in this case the first unlucky job will be sacrificed. Working on why the node wasn't marked down when the prologue failed. Ben > On Aug 16, 2016, at 10:26 AM, Tom LeCompte <lecompte(a)anl.gov> wrote: > > Hi everyone- > > My jobs on knl03 immediately exit. The same jobs run OK on 0-2 and 6-8. Is this a known problem? > > Cobalt doesn't say much: > > Tue Aug 16 15:20:40 2016 +0000 (UTC) submitted with cwd set to: /home/lecompte/g4/Geant4HepExpMTBenchmark/run > jobid 520 submitted from terminal /dev/pts/5 > Tue Aug 16 15:23:55 2016 +0000 (UTC) Info: resources released; starting job epilogue scripts > > Cheers, > > Tom > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] Fwd: [Early-users-discuss] KNL book is available as PDF online
by Kumaran, Kalyan 16 Aug '16

16 Aug '16

Begin forwarded message: From: "Hammond, Jeff R" <jeff.r.hammond(a)intel.com<mailto:[email protected]>> Subject: Re: [Early-users-discuss] KNL book is available as PDF online Date: August 16, 2016 at 10:49:37 AM CDT To: "early-users-discuss(a)lists.alcf.anl.gov<mailto:[email protected]>" <early-users-discuss(a)lists.alcf.anl.gov<mailto:[email protected]>> --------------------------------------------- No NDA material may be included in messages to early-users-discuss! _______________________________________________ I attached the two chapters that everyone should read this week. Jeff From: Jeff R Hammond <jeff.r.hammond(a)intel.com<mailto:[email protected]>> Date: Tuesday, August 16, 2016 at 10:46 AM To: "early-users-discuss(a)lists.alcf.anl.gov<mailto:[email protected]>" <early-users-discuss(a)lists.alcf.anl.gov<mailto:[email protected]>> Subject: KNL book is available as PDF online http://www.sciencedirect.com/science/book/9780128091944 This book has a number of important architectural details that I can’t find anywhere else online right now. I had no trouble downloading, but I don’t know if this is because I have an Intel IP address. Please let me know if you cannot download. Jeff _______________________________________________ No NDA material may be included in messages to early-users-discuss! early-users-discuss mailing list early-users-discuss(a)lists.alcf.anl.gov<mailto:[email protected]> https://lists.alcf.anl.gov/mailman/listinfo/early-users-discuss

1 0

[JLSE-Intel-NDA] knl03 in trouble?
by Tom LeCompte 16 Aug '16

16 Aug '16

Hi everyone- My jobs on knl03 immediately exit. The same jobs run OK on 0-2 and 6-8. Is this a known problem? Cobalt doesn't say much: Tue Aug 16 15:20:40 2016 +0000 (UTC) submitted with cwd set to: /home/lecompte/g4/Geant4HepExpMTBenchmark/run jobid 520 submitted from terminal /dev/pts/5 Tue Aug 16 15:23:55 2016 +0000 (UTC) Info: resources released; starting job epilogue scripts Cheers, Tom

1 0

Re: [JLSE-Intel-NDA] GCC Compilers on KNL
by Allen, Benjamin S. 15 Aug '16

15 Aug '16

Tom, sorry. Typo'ed your name. Ben > On Aug 15, 2016, at 10:48 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > Hi Tim, > > No we only have RHEL7's GCC 4.8 installed. Feel free to use one of the higher end nodes to speed up the compile instead of the logins. I also suggest using local storage (i.e /tmp) for the compile if possible. > > Ben > >> On Aug 15, 2016, at 10:40 AM, Tom LeCompte <lecompte(a)anl.gov> wrote: >> >> Hi everyone- >> >> I'm doing a bit of yak shaving here. >> >> I want to compile a piece of code (Geant4) under ICC/ICPC, and get an error in /usr/include/c++/4.8.5/bits/hashtable.h This turns out to be a known problem, and the authors tell me the fix is to use the headers for GCC 4.9+ (ICC uses the GCC headers). Before running around and installing my own GCC 4.9, is there one installed on the cluster? If not, are there any plans to install it? >> >> Cheers, >> Tom >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] GCC Compilers on KNL
by Allen, Benjamin S. 15 Aug '16

15 Aug '16

Hi Tim, No we only have RHEL7's GCC 4.8 installed. Feel free to use one of the higher end nodes to speed up the compile instead of the logins. I also suggest using local storage (i.e /tmp) for the compile if possible. Ben > On Aug 15, 2016, at 10:40 AM, Tom LeCompte <lecompte(a)anl.gov> wrote: > > Hi everyone- > > I'm doing a bit of yak shaving here. > > I want to compile a piece of code (Geant4) under ICC/ICPC, and get an error in /usr/include/c++/4.8.5/bits/hashtable.h This turns out to be a known problem, and the authors tell me the fix is to use the headers for GCC 4.9+ (ICC uses the GCC headers). Before running around and installing my own GCC 4.9, is there one installed on the cluster? If not, are there any plans to install it? > > Cheers, > Tom > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] GCC Compilers on KNL
by Tom LeCompte 15 Aug '16

15 Aug '16

Hi everyone- I'm doing a bit of yak shaving here. I want to compile a piece of code (Geant4) under ICC/ICPC, and get an error in /usr/include/c++/4.8.5/bits/hashtable.h This turns out to be a known problem, and the authors tell me the fix is to use the headers for GCC 4.9+ (ICC uses the GCC headers). Before running around and installing my own GCC 4.9, is there one installed on the cluster? If not, are there any plans to install it? Cheers, Tom

1 0