August 2016 - Intel-nda - mailman3.cels.anl.gov

Re: [JLSE-Intel-NDA] OpenMP bug in Intel Fortran compiler
by Kumaran, Kalyan 23 Sep '16

23 Sep '16

Thanks, Martyn! I will open a Quad once I create a generic account. I have copied Steve Pieper to this thread. He reported the issue. -Kumar On Jul 28, 2016, at 6:42 PM, Corden, Martyn J <martyn.j.corden(a)intel.com<mailto:[email protected]>> wrote: Hello Kumar! Nice to hear from you. I shall be looking into your WORKSHARE issue, but haven’t yet had chance to do more than just reproduce.. We’ll want to create a support issue out of it eventually, so that it can be tracked. Thanks for the small test case. It’s only recently that the Intel Fortran compiler has made a parallel implementation of WORKSHARE, so it still has some restrictions. It looks like the loop over the index variable is challenging. Regards, Martyn From: Kirkendall, Keith G Sent: Thursday, July 28, 2016 7:05 AM To: Kumaran, Kalyan Cc: Corden, Martyn J; Borges, Leonardo; Allison, CM; Fromkin, Russ Subject: RE: [JLSE-Intel-NDA] OpenMP bug in Intel Fortran compiler I don’t think we mind if ANL wants to use a combined, single account. From: Kumaran, Kalyan [mailto:[email protected]] Sent: Wednesday, July 27, 2016 8:49 PM To: Kirkendall, Keith G <keith.g.kirkendall(a)intel.com<mailto:[email protected]>> Cc: Corden, Martyn J <martyn.j.corden(a)intel.com<mailto:[email protected]>>; Borges, Leonardo <leonardo.borges(a)intel.com<mailto:[email protected]>>; Allison, CM <cm.allison(a)intel.com<mailto:[email protected]>>; Fromkin, Russ <russ.fromkin(a)intel.com<mailto:[email protected]>> Subject: Re: [JLSE-Intel-NDA] OpenMP bug in Intel Fortran compiler I know Martyn from my SGI days as he was my main contact for the top 20 Itanium compiler bugs. Hello Martyn! Keith, regarding premier, do you want each user to open an account. I was hoping to set up a generic Argonne account that all users can use. Please advise. Thanks, Kumar On Jul 27, 2016, at 5:29 PM, Kirkendall, Keith G <keith.g.kirkendall(a)intel.com<mailto:[email protected]>> wrote: Hi Kumar, Martyn Corden is the smart compiler guy that was recommended to me. As an aside, he’ll be supporting the Theta ESP workshop, so you’ll meet him FTF in August. The general process for compiler bugs is to file a ticket through premier.intel.com<http://premier.intel.com/> through your support account, but since we like you, you get Martyn directly this time. ☺ Martyn, I copied the original email & files here. Thanks Keith From: Kumaran, Kalyan [mailto:[email protected]] Sent: Thursday, July 21, 2016 4:11 PM To: Allison, CM <cm.allison(a)intel.com<mailto:[email protected]>>; Kirkendall, Keith G <keith.g.kirkendall(a)intel.com<mailto:[email protected]>> Cc: Fromkin, Russ <russ.fromkin(a)intel.com<mailto:[email protected]>> Subject: Fwd: [JLSE-Intel-NDA] OpenMP bug in Intel Fortran compiler How do we report compiler bugs? Happens in 2016 and 2017 versions. -Kumar Begin forwarded message: From: "Steven C. Pieper" <spieper(a)anl.gov<mailto:[email protected]>> Subject: [JLSE-Intel-NDA] OpenMP bug in Intel Fortran compiler Date: July 12, 2016 at 1:55:50 PM CDT To: <intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]>> Reply-To: <spieper(a)anl.gov<mailto:[email protected]>> I have attached a simplified source code and demonstration script to show an OpenMP bug in the Intel Fortran compiler. I am using /soft/compilers/intel_beta/compilers_and_libraries_2017.0.042/linux/bin/intel64/ifort The bug occurs on both the jlse login node and knl00. It occurs using two or more OpenMP threads, but not if only one thread is used. The demonstration uses no optimization (O0) but the bug also happens with optimization. It does not happen on the IBM BG/Q. The bug is a segfault at line 113 of the source file: !$OMP PARALLEL WORKSHARE cof(:,:,1:min(2,num_op_emew),:,:) = 0 cfcl(:,:,:,:) = 0.d0 !$OMP END PARALLEL WORKSHARE It is triggered by line 128 when that line is activated (in the source file it is commented): cof(:,:,2,iqcross(:),1) = cof(:,:,1,iqcross(:),1) + cof(:,:,2,iqcross(:),1) The problem is the index iqcross(:); iqcross is a 2-element array with valid values as are printed. If this line is replaced by the equivalent two lines that follow: cof(:,:,2,iqcross(1),1) = cof(:,:,1,iqcross(1),1) + cof(:,:,2,iqcross(1),1) cof(:,:,2,iqcross(2),1) = cof(:,:,1,iqcross(2),1) + cof(:,:,2,iqcross(2),1) no segfault occurs. Note also that in fact this section of the code will never be reached due to the stop at line 119 and the if at line 121 which is false. In the original large code, this second OMP section was more than 100 lines removed from the first with a large intermediate OMP section. The script first compiles and runs the source with the commented line (bug-ok.f). This shows that things work with the line removed. Then the script uses sed to uncomment the line into bug-bad.f and compiles and runs that. This causes the segfault. Steve -- Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov<mailto:[email protected]> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 http://www.phy.anl.gov/theory/staff/scp.html _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda <makebug.txt><bug-ok.f>

2 1

[JLSE-Intel-NDA] Fwd: Two new KNL papers
by Allen, Benjamin S. 07 Sep '16

07 Sep '16

Some howtos from Colfax. Ben Begin forwarded message: From: Gautam Shah <shah(a)colfax-intl.com<mailto:[email protected]>> Subject: Two new KNL papers Date: May 5, 2016 at 1:56:29 AM CDT To: "Allen, Benjamin S." <bsallen(a)alcf.anl.gov<mailto:[email protected]>> Gautam Shah Colfax International 408-730-2275 x 141 shah(a)colfax-intl.com<mailto:[email protected]> www.colfax-intl.com<http://www.colfax-intl.com> www.colfaxdirect.com<http://www.colfaxdirect.com> www.colfaxresearch.com<http://www.colfaxresearch.com>

3 2

[JLSE-Intel-NDA] Fwd: User level KNL memory mode checker
by Kumaran, Kalyan 31 Aug '16

31 Aug '16

Begin forwarded message: From: "Nam, Hai Ah" <hnam(a)lanl.gov<mailto:[email protected]>> Subject: FW: User level KNL memory mode checker Date: August 31, 2016 at 12:44:44 PM CDT To: Jack Deslippe <jrdeslippe(a)lbl.gov<mailto:[email protected]>>, Kalyan Kumaran <kumaran(a)anl.gov<mailto:[email protected]>> Something our admin wrote to help identify the flavor of node. Thought it would be helpful to share with your user communities. --------------------------------------- Hai Ah Nam, Ph.D. Scientist, CCS-2 Los Alamos National Laboratory Phone: (505) 667-3768 Email: hnam(a)lanl.gov<mailto:[email protected]> From: rtc <rtc(a)lanl.gov<mailto:[email protected]>> Date: Tuesday, August 30, 2016 at 2:57 PM To: "Peltz Jr., Paul" <peltz(a)lanl.gov<mailto:[email protected]>> Cc: "Vernon, Louis James" <vernon(a)lanl.gov<mailto:[email protected]>>, Hai Ah Nam <hnam(a)lanl.gov<mailto:[email protected]>>, "Berry, Mike" <mrberry(a)lanl.gov<mailto:[email protected]>> Subject: Re: User level KNL memory mode checker Paul, this is great, thank you! Will update the docs. rob _______________________ Robert Cunningham HPC Consulting Team, HPC-DEV Los Alamos National Security, LLC http://hpc.lanl.gov<http://hpc.lanl.gov/> 505-665-4444 x05704 On Aug 30, 2016, at 1:22 PM, Peltz Jr., Paul <peltz(a)lanl.gov<mailto:[email protected]>> wrote: I heard during part of the boot camp today that users wanted to verify memory modes a node is configured in. I’ve adapted some simple bash scripts to do this when I do my verification testing. There are two ways to do this easily. First I have a function in my .bashrc file for interactive sessions. function knl_mode() { if [ -f /.hwinfo.cray ] ; then CNAME=`/usr/bin/cat /proc/cray_xt/cname` NODE=`echo -n $CNAME | tail -c1` NID=`/usr/bin/cat /proc/cray_xt/nid` NUMA=`/usr/bin/grep "numa_cfg\[$NODE" /.hwinfo.cray | awk -F "=" '{print $2}'` MCDRAM=`/usr/bin/grep "mcdram_cfg\[$NODE" /.hwinfo.cray| awk -F "=" '{print $2}'` if [ $NUMA ] ; then echo $NID "CLE_"$NUMA"_"$MCDRAM else echo "This is not a KNL compute node." fi else echo "This can only be used on a compute node." fi } And the same script for non-interactive sessions in my home area knl_mode.sh: #!/bin/bash if [ -f /.hwinfo.cray ] ; then CNAME=`/usr/bin/cat /proc/cray_xt/cname` NODE=`echo -n $CNAME | tail -c1` NID=`/usr/bin/cat /proc/cray_xt/nid` NUMA=`/usr/bin/grep "numa_cfg\[$NODE" /.hwinfo.cray | awk -F "=" '{print $2}'` MCDRAM=`/usr/bin/grep "mcdram_cfg\[$NODE" /.hwinfo.cray| awk -F "=" '{print $2}'` if [ $NUMA ] ; then echo $NID "CLE_"$NUMA"_"$MCDRAM else echo "This is not a KNL compute node." fi else echo "This can only be used on a compute node." fi CSH versions would need to be made, but this is what I’m doing currently and you might want to add this option to your documentation. Other options such as apstat -M don’t give as useful info. The memory mode isn’t obvious with this. Thanks, Paul =========================== Paul Peltz Jr HPC-DES Systems Integration Los Alamos National Laboratory ===========================

1 0

[JLSE-Intel-NDA] Fwd: parallelism training + DAAL, and Optimized Python
by Kumaran, Kalyan 31 Aug '16

31 Aug '16

Please contact Chris if you would like to attend and run into any issues. Begin forwarded message: From: "Allison, CM" <cm.allison(a)intel.com<mailto:[email protected]>> Subject: parallelism training + DAAL, and Optimized Python Argonne team - You (and your Chicago-area colleagues) are invited to participate in this free training we’ve organized with the Univ of Chicago: http://colfaxresearch.com/events/uchicago-16-10/ If you have questions, please contact me. I have coordinated this opportunity with Birali. You and your colleagues can find details and register at the above URL. Best regards, - Chris Chris Allison Research Computing, Field Intel Corporation 816-835-1106

1 0

[JLSE-Intel-NDA] Update XPPSL (MPSP) to 1.4.1
by Allen, Benjamin S. 25 Aug '16

25 Aug '16

Intel's XPPSL (MPSP) has been updated to 1.4.1. The KNL nodes will get the updates on their next reboot. Intel included a new kernel module to help deal with the cached memory mode performance degradation over time issue named zonesort_module. See page 21 in the attached doc. Note, I plan on loading this module and running "echo 0 > /sys/kernel/zone_sort_free_pages/nodeid" on job start where the node is not rebooted, and the mode is cached. If someone is interested in testing this out before I get to implementing it, let me know. Interesting known issues: - The hbmalloc library does not guarantee to allocate memory from the closest MCDRAM node when SNC-2 or SNC-4 Memory Mode is used. Ben

2 1

[JLSE-Intel-NDA] KNL BIOS Updates
by Allen, Benjamin S. 25 Aug '16

25 Aug '16

Fresh set of BIOS, BMC, and FRUSDR updates for the Adam's Pass KNL machines. They're not yet installed. Ben

1 0

[JLSE-Intel-NDA] Location of Vtune?
by Tom LeCompte 24 Aug '16

24 Aug '16

Hi everyone- I have an application I would like to profile. The Wiki says "Option 1" is to use Vtune and gives instructions execute the following command via Cobalt submission script $ amplxe-cl -c memory-access ... This givea a command not found for amplxe-cl. Is there something I need to set up? Cheers, Tom

4 5

[JLSE-Intel-NDA] Intel Parallel Studio 2017
by Allen, Benjamin S. 24 Aug '16

24 Aug '16

Intel provided us an early copy of the final release of Parallel Studio 2017. It's under NDA until officially released. As such it's installed under /soft/restricted/intel_nda. Currently the /soft/restricted mount is only available on the KNL nodes. Let me know if it is needed elsewhere. Ben

1 0

[JLSE-Intel-NDA] jlse nodes
by Kenneth E Jansen 23 Aug '16

23 Aug '16

looks like 3 7210 nodes are down, a single user has submitted 6, 48 hour INTERACTIVE jobs to the only up 7210 nodes. one of the 7250 nodes is down, the other idle but trying to submit to it yields [kjansen@jlselogin1 64]$ qsub -n 1 -q knl_7250 -t 60 -I <Fault 1001: "You are not allowed to submit to the 'knl_7250' queue\n"> ok while writing this email it seems knl04 is back but are we really ok with 6 48 hour interactive one node jobs? And is it correct that I (other users too) can’t use the knl_7250 queue…thought I did before but maybe miss-remembering). Best regards, Ken ---------------------------------------------------------------- Kenneth E. Jansen, Professor of Aerospace Engineering Sciences ECAE 161 OFFICE (303) 492-4359 429 UCB FAX (303) 492-4990 University of Colorado at Boulder jansenke(a)colorado.edu Boulder, CO, 80309-0429 http://www.colorado.edu/aerospace ----------------------------------------------------------------

5 4

[JLSE-Intel-NDA] JSLE KNL Scheduling Paused
by Allen, Benjamin S. 17 Aug '16

17 Aug '16

I've stopped scheduling for a bit while I test some changes to fix problems Steve and Ye Luo reported. Guessing less than 30 minutes. Running jobs are unaffected. Ben

1 1