August 2016 - Intel-nda - mailman3.cels.anl.gov

[JLSE-Intel-NDA] Fwd: [catalysts] KNL in Architecture Optimization Manual
by Kumaran, Kalyan 14 Aug '16

14 Aug '16

> Begin forwarded message: > > > Hi everyone, > > FYI: The KNL is now covered in Intel's Architecture Optimization Manual (http://www.intel.com/content/www/us/en/architecture-and-technology/64-ia-32…) specifically chapter 16. > > -Hal > > -- > Hal Finkel > Assistant Computational Scientist > Leadership Computing Facility > Argonne National Laboratory > _______________________________________________ > catalysts mailing list > catalysts(a)lists.alcf.anl.gov > https://lists.alcf.anl.gov/cgi-bin/mailman/listinfo/catalysts

1 0

[JLSE-Intel-NDA] bus error?
by Steven C. Pieper 12 Aug '16

12 Aug '16

On the KNL, what is a bus error (signal 7) as distinguished from a seg fault (11)? I am getting these with (very) low probability in OpenMP regions of my Fortran code. The same code has worked fine for years on mira. Also on KNL the same subroutines have been successfully invoked up to 100,000 times. But attempts to scale up to bigger runs are being hit now. Removing the !$OMP statements for the given loop seems to get rid of the bus error, but of course this slows the program down -- we have to be able to use multithreading on this machine. Steve -- Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 http://www.phy.anl.gov/theory/staff/scp.html

1 0

[JLSE-Intel-NDA] Cobalt Fixes
by Allen, Benjamin S. 11 Aug '16

11 Aug '16

There were some value normalization fixes that needed to happen between /var/run/hwloc/knl_memoryside_cache, and what the BIOS values use that was causing Cobalt to not set knl_cluster_mode or knl_memory_mode in certain cases. Largely when SNC-2, SNC-4, or Hybrid were specified. This is now fixed. Please let me know if you run into any further instances where the values in /var/run/hwloc/knl_memoryside_cache don't match the values you selected. I also removed Auto from the possible values for both knl_memory_mode and knl_cluster_mode as Auto sets to Cached and Quadrant respectively. Thanks, Ben

1 0

Re: [JLSE-Intel-NDA] the mystery of COBALT_JOBSIZE
by Paul Hargrove 11 Aug '16

11 Aug '16

And why is cat even needed? Is there a reason not to use $(wc -l $COBALT_NODEFILE) ? -Paul On Thu, Aug 11, 2016 at 11:35 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > This is fixed. > > [bsallen@knl04 ~]$ cat $COBALT_NODEFILE > knl04 > knl08 > [bsallen@knl04 ~]$ cat $COBALT_NODEFILE | wc -l > 2 > > > > On Aug 11, 2016, at 1:03 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> > wrote: > > > > It's because wc -l counts newlines, and the node file is missing the > trailing newline. I'll fix this this afternoon. > > > > Ben > > > > > > From: "Steven C. Pieper" <spieper(a)anl.gov> > > Sent: Aug 11, 2016 12:49 PM > > To: intel-nda(a)lists.jlse.anl.gov > > Subject: Re: [JLSE-Intel-NDA] the mystery of COBALT_JOBSIZE > > > > The second possibility doesn't work for some reason I don't > > understand. I had observed that cat does not list two lines > > for a 2-node node file: > > > > [knl02 ~] echo $COBALT_NODEFILE > > /var/tmp/cobalt.327 > > [knl02 ~] cat /var/tmp/cobalt.327 > > knl07 > > knl02[knl02 ~] NODECOUNT=$(cat $COBALT_NODEFILE | wc -l) > > [knl02 ~] echo $NODECOUNT > > 1 > > > > however vi does show the two lines: > > > > knl07 > > knl02 > > ~ > > > > Seems very strange. > > > > Steve > > Steven C. Pieper, Senior Physicist Emeritus: > spieper(a)anl.gov > > > > Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL > 60439 > > Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, > -4100 > > > > http://www.phy.anl.gov/theory/staff/scp.html > > On 08/11/16 11:43, Allen, Benjamin S. wrote: > >> Couple ways to deal with this: > >> > >> mpirun -f $COBALT_NODEFILE -ppn 64 > >> > >> or > >> > >> NODECOUNT=$(cat $COBALT_NODEFILE | wc -l) > >> > >> COBALT_NODEFILE and COBALT_JOBID are the only two environmental > variables passed from Cobalt, similar to Cooley. > >> > >> Ben > >> > >> > >>> On Aug 10, 2016, at 6:08 PM, Alvaro <vama(a)alcf.anl.gov> > >>> wrote: > >>> > >>> hi. > >>> > >>> I am not getting any value for COBALT_JOBSIZE. > >>> > >>> So I cannot do > >>> > >>> proc=64 > >>> > >>> mpirun -n $((COBALT_JOBSIZE*proc)) > >>> > >>> How I can get the number of nodes in cobalt? > >>> > >>> > >>> -- > >>> -Alvaro > >>> http://web.alcf.anl.gov/~vama/ > >>> > >>> _______________________________________________ > >>> Intel-nda mailing list > >>> > >>> Intel-nda(a)lists.jlse.anl.gov > >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > >> _______________________________________________ > >> Intel-nda mailing list > >> > >> Intel-nda(a)lists.jlse.anl.gov > >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > > > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > -- Paul H. Hargrove PHHargrove(a)lbl.gov Computer Languages & Systems Software (CLaSS) Group Computer Science Department Tel: +1-510-495-2352 Lawrence Berkeley National Laboratory Fax: +1-510-486-6900

1 0

Re: [JLSE-Intel-NDA] the mystery of COBALT_JOBSIZE
by Allen, Benjamin S. 11 Aug '16

11 Aug '16

This is fixed. [bsallen@knl04 ~]$ cat $COBALT_NODEFILE knl04 knl08 [bsallen@knl04 ~]$ cat $COBALT_NODEFILE | wc -l 2 > On Aug 11, 2016, at 1:03 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > It's because wc -l counts newlines, and the node file is missing the trailing newline. I'll fix this this afternoon. > > Ben > > > From: "Steven C. Pieper" <spieper(a)anl.gov> > Sent: Aug 11, 2016 12:49 PM > To: intel-nda(a)lists.jlse.anl.gov > Subject: Re: [JLSE-Intel-NDA] the mystery of COBALT_JOBSIZE > > The second possibility doesn't work for some reason I don't > understand. I had observed that cat does not list two lines > for a 2-node node file: > > [knl02 ~] echo $COBALT_NODEFILE > /var/tmp/cobalt.327 > [knl02 ~] cat /var/tmp/cobalt.327 > knl07 > knl02[knl02 ~] NODECOUNT=$(cat $COBALT_NODEFILE | wc -l) > [knl02 ~] echo $NODECOUNT > 1 > > however vi does show the two lines: > > knl07 > knl02 > ~ > > Seems very strange. > > Steve > Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov > > Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 > Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 > > http://www.phy.anl.gov/theory/staff/scp.html > On 08/11/16 11:43, Allen, Benjamin S. wrote: >> Couple ways to deal with this: >> >> mpirun -f $COBALT_NODEFILE -ppn 64 >> >> or >> >> NODECOUNT=$(cat $COBALT_NODEFILE | wc -l) >> >> COBALT_NODEFILE and COBALT_JOBID are the only two environmental variables passed from Cobalt, similar to Cooley. >> >> Ben >> >> >>> On Aug 10, 2016, at 6:08 PM, Alvaro <vama(a)alcf.anl.gov> >>> wrote: >>> >>> hi. >>> >>> I am not getting any value for COBALT_JOBSIZE. >>> >>> So I cannot do >>> >>> proc=64 >>> >>> mpirun -n $((COBALT_JOBSIZE*proc)) >>> >>> How I can get the number of nodes in cobalt? >>> >>> >>> -- >>> -Alvaro >>> http://web.alcf.anl.gov/~vama/ >>> >>> _______________________________________________ >>> Intel-nda mailing list >>> >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> _______________________________________________ >> Intel-nda mailing list >> >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > >

1 0

Re: [JLSE-Intel-NDA] the mystery of COBALT_JOBSIZE
by Steven C. Pieper 11 Aug '16

11 Aug '16

The second possibility doesn't work for some reason I don't understand. I had observed that cat does not list two lines for a 2-node node file: [knl02 ~] echo $COBALT_NODEFILE /var/tmp/cobalt.327 [knl02 ~] cat /var/tmp/cobalt.327 knl07 knl02[knl02 ~] NODECOUNT=$(cat $COBALT_NODEFILE | wc -l) [knl02 ~] echo $NODECOUNT 1 however vi does show the two lines: knl07 knl02 ~ Seems very strange. Steve Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 http://www.phy.anl.gov/theory/staff/scp.html On 08/11/16 11:43, Allen, Benjamin S. wrote: > Couple ways to deal with this: > > mpirun -f $COBALT_NODEFILE -ppn 64 > > or > > NODECOUNT=$(cat $COBALT_NODEFILE | wc -l) > > COBALT_NODEFILE and COBALT_JOBID are the only two environmental variables passed from Cobalt, similar to Cooley. > > Ben > >> On Aug 10, 2016, at 6:08 PM, Alvaro <vama(a)alcf.anl.gov> wrote: >> >> hi. >> >> I am not getting any value for COBALT_JOBSIZE. >> >> So I cannot do >> >> proc=64 >> >> mpirun -n $((COBALT_JOBSIZE*proc)) >> >> How I can get the number of nodes in cobalt? >> >> >> -- >> -Alvaro http://web.alcf.anl.gov/~vama/ >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] the mystery of COBALT_JOBSIZE
by Allen, Benjamin S. 11 Aug '16

11 Aug '16

Couple ways to deal with this: mpirun -f $COBALT_NODEFILE -ppn 64 or NODECOUNT=$(cat $COBALT_NODEFILE | wc -l) COBALT_NODEFILE and COBALT_JOBID are the only two environmental variables passed from Cobalt, similar to Cooley. Ben > On Aug 10, 2016, at 6:08 PM, Alvaro <vama(a)alcf.anl.gov> wrote: > > hi. > > I am not getting any value for COBALT_JOBSIZE. > > So I cannot do > > proc=64 > > mpirun -n $((COBALT_JOBSIZE*proc)) > > How I can get the number of nodes in cobalt? > > > -- > -Alvaro http://web.alcf.anl.gov/~vama/ > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] the mystery of COBALT_JOBSIZE
by Alvaro 10 Aug '16

10 Aug '16

hi. I am not getting any value for COBALT_JOBSIZE. So I cannot do proc=64 mpirun -n $((COBALT_JOBSIZE*proc)) How I can get the number of nodes in cobalt? -- -Alvaro http://web.alcf.anl.gov/~vama/

1 0

Re: [JLSE-Intel-NDA] bus error
by Steven C. Pieper 10 Aug '16

10 Aug '16

I ran this job a second time and it completed correctly. The same 4 nodes were used. Steve Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 http://www.phy.anl.gov/theory/staff/scp.html On 08/10/16 13:45, Steven C. Pieper wrote: > I have been running the same executable with different > numbers of nodes (2 or 4) and ranks/node (7 or 10). It > has been working fine but I just had a job that failed. > I find nothing in stderr, but it made a core file. > gdb on the core file shows me: > > Program terminated with signal 7, Bus error. > #0 L_c12_vijay__234__par_loop0_2_2 () at c12_vijay.f:321 > 321 terms(num_perm2*(iperm1-1)+iperm2) = term > Missing separate debuginfos, use: debuginfo-install > glibc-2.17-106.el7_2.4.x86_64 libgcc-4.8.5-4.el7.x86_64 > libuuid-2.23.2-26.el7_2.2.x86_64 > (gdb) where > #0 L_c12_vijay__234__par_loop0_2_2 () at c12_vijay.f:321 > #1 0x00002b3815d713f3 in __kmp_invoke_microtask () from > /soft/compilers/intel_beta/compilers_and_libraries_2017.0.042/linux/compiler/lib/intel64/libiomp5.so > #2 0x00002b3815d3f057 in __kmp_invoke_task_func (gtid=0) at > ../../src/kmp_runtime.c:7055 > #3 0x00002b3815d3e745 in __kmp_launch_thread (this_thr=0x0) at > ../../src/kmp_runtime.c:5677 > #4 0x00002b3815d71773 in __kmp_launch_worker (thr=0x0) at > ../../src/z_Linux_util.c:769 > #5 0x00002b3813091dc5 in start_thread () from /lib64/libpthread.so.0 > #6 0x00002b3816371ced in __libc_ifunc_impl_list () from /lib64/libc.so.6 > #7 0x0000000000000000 in ?? () > > I was running in cache mode. > Is a bus error something that I can make by a programming error > or a machine error? I'm trying again. > > Steve > -- > Steven C. Pieper, Senior Physicist Emeritus:[email protected] > Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 > Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 > http://www.phy.anl.gov/theory/staff/scp.html > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] bus error
by Steven C. Pieper 10 Aug '16

10 Aug '16

I have been running the same executable with different numbers of nodes (2 or 4) and ranks/node (7 or 10). It has been working fine but I just had a job that failed. I find nothing in stderr, but it made a core file. gdb on the core file shows me: Program terminated with signal 7, Bus error. #0 L_c12_vijay__234__par_loop0_2_2 () at c12_vijay.f:321 321 terms(num_perm2*(iperm1-1)+iperm2) = term Missing separate debuginfos, use: debuginfo-install glibc-2.17-106.el7_2.4.x86_64 libgcc-4.8.5-4.el7.x86_64 libuuid-2.23.2-26.el7_2.2.x86_64 (gdb) where #0 L_c12_vijay__234__par_loop0_2_2 () at c12_vijay.f:321 #1 0x00002b3815d713f3 in __kmp_invoke_microtask () from /soft/compilers/intel_beta/compilers_and_libraries_2017.0.042/linux/compiler/lib/intel64/libiomp5.so #2 0x00002b3815d3f057 in __kmp_invoke_task_func (gtid=0) at ../../src/kmp_runtime.c:7055 #3 0x00002b3815d3e745 in __kmp_launch_thread (this_thr=0x0) at ../../src/kmp_runtime.c:5677 #4 0x00002b3815d71773 in __kmp_launch_worker (thr=0x0) at ../../src/z_Linux_util.c:769 #5 0x00002b3813091dc5 in start_thread () from /lib64/libpthread.so.0 #6 0x00002b3816371ced in __libc_ifunc_impl_list () from /lib64/libc.so.6 #7 0x0000000000000000 in ?? () I was running in cache mode. Is a bus error something that I can make by a programming error or a machine error? I'm trying again. Steve -- Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 http://www.phy.anl.gov/theory/staff/scp.html

1 0