- Intel-nda - mailman3.cels.anl.gov

[JLSE-Intel-NDA] HACC and NEKbone slides
by Kumaran, Kalyan 27 Apr '16

27 Apr '16

These two set of slides were shared with Intel. They might use these results internally and also for project launch at ISC. They have promised to keep us updated. -Kumar

1 0

[JLSE-Intel-NDA] KNL working group call
by Kumaran, Kalyan 27 Apr '16

27 Apr '16

There will be a monthly collaborative call with the weapons lab and NERSC on KNL. I will forward the meeting time and agenda to this list. You are welcome to send a message to Hai Ah and subscribe to the KNL chatter mailing list if you would like. I have reserved 1172 for ANL participants. -Kumar We will kick off our first KNL Working Group meeting Tuesday 11AM MDT (1PM EDT, 10AM PDT). This meeting was intended to be monthly, but if we find information should be shared at a more rapid pace, we can make it bi-weekly. If you can’t attend, we will try to make slides available to your COE leads who can distribute appropriately. Agenda: KNL updates – LANL white boxes (results, software notes, gotchas), Ron Green (Intel) New KNL results on Avalon, John Levesque (Cray) Open Forum (observations & questions encouraged) Feel free to forward this invite to others who should attend and are covered under the NDA. Note: The KNLchatter(a)lanl.gov email list has vendors (Intel & Cray) and participants from LANL, SNL, LLNL, NERSC/LBL and ANL covered under the NDA. The mailing list was setup so that we can share information about our experiences on KNLs and ask for help/feedback (e.g. Are you seeing what I’m seeing?). Each site has at least 1 KNL white box on site. Hai Ah ________________________________________________________________________ Slides will be shown so please ensure your webex is working prior to the meeting. -- Do not delete or change any of the following text. -- Join WebEx meeting<https://lanl-us.webex.com/lanl-us/j.php?MTID=mc34d4c4a32f6b86f76dada1c90e5f…> Meeting number: 801 593 665 If you are the host, you can use the meeting host key to pass the host privilege to another participant or to start the meeting from a video conferencing system or application. To find the host key for this meeting, go here.<https://lanl-us.webex.com/lanl-us/j.php?MTID=m7494c6df89017ff15f35e17d155bb…> Join by phone +1-415-655-0002 US Toll Access code: 801 593 665 Global call-in numbers<https://lanl-us.webex.com/lanl-us/globalcallin.php?serviceType=MC&ED=459245…> Can't join the meeting? Contact support.<https://lanl-us.webex.com/lanl-us/mc> IMPORTANT NOTICE: Please note that this WebEx service allows audio and other information sent during the session to be recorded, which may be discoverable in a legal matter. By joining this session, you automatically consent to such recordings. If you do not consent to being recorded, discuss your concerns with the host or do not join the session..

1 0

Re: [JLSE-Intel-NDA] Intel Meeting Notes
by Scott Parker 26 Apr '16

26 Apr '16

Re-ran stream in Quadrant mode and I was able to get 435 GB/s with 1 thread per core, and a very slightly better 440 GB/s with 2 threads per core. In SNC-4 mode I saw 311 GB/s with 1 thread and 442 GB/s with 2 threads. All2All hit 425 GB/s with 1 thread per core. A peak of around 440 GB/s is basically what we should be expecting to see for a Bin 3 part, so the MCDRAM seems to be capable of delivering the expected bandwidth. However, SNC-4 is an outlier in requiring 128 threads, and gives unexpected low (73 GB/s) bandwidth from one of the clusters (vs 112 GB/s from each of the other three). -Scott On 4/26/16 3:56 PM, Harms, Kevin wrote: > Meeting with Avinash and Intel - 2016/4/26 > > Discussion: > - We indicated that we are able to get 420,000 MB/s from stream in Flat All2All and Quadrant modes since the recent BIOS update > - Intel would like us to investigate the Quadrant model in Flat mode because they think we should get 440,000 MB/s > - Intel examined our NUMA output from the system and believes that we are not running a newer firmware because the logical processes should be nicely ordered. > - Intel said they only expect at 5-6% variance between NUMA clusters. > - Vitali present his HACC results on KNL (which were all good and have improved since A0) > - Irv from Intel joined to help debug our BIOS setup. > - He indicated that our BIOS is provided by PCSD division which does there own packaging and he doesn't know what they have. > - Intel development runs BIOS created by him. > - After some discussion, we need someone from Intel to help us debug the issue > - Avinash will work to assign someone to help us > - It appears Intel development has their own development boards and do not use Adams Pass boards > > Action Items: > - Argonne: Run our stream test cases in Flat-Quadrant configuration and try to achieve 440,000 MB/s using 1 or 2 threads per core > - Intel: Assign someone to help investigate the BIOS version/configuration on our box > - Argonne/Intel - setup a meeting in two weeks to discuss progress. Scott will present his NEK results. > > > kevin > > > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] Intel Meeting Notes
by Harms, Kevin 26 Apr '16

26 Apr '16

Meeting with Avinash and Intel - 2016/4/26 Discussion: - We indicated that we are able to get 420,000 MB/s from stream in Flat All2All and Quadrant modes since the recent BIOS update - Intel would like us to investigate the Quadrant model in Flat mode because they think we should get 440,000 MB/s - Intel examined our NUMA output from the system and believes that we are not running a newer firmware because the logical processes should be nicely ordered. - Intel said they only expect at 5-6% variance between NUMA clusters. - Vitali present his HACC results on KNL (which were all good and have improved since A0) - Irv from Intel joined to help debug our BIOS setup. - He indicated that our BIOS is provided by PCSD division which does there own packaging and he doesn't know what they have. - Intel development runs BIOS created by him. - After some discussion, we need someone from Intel to help us debug the issue - Avinash will work to assign someone to help us - It appears Intel development has their own development boards and do not use Adams Pass boards Action Items: - Argonne: Run our stream test cases in Flat-Quadrant configuration and try to achieve 440,000 MB/s using 1 or 2 threads per core - Intel: Assign someone to help investigate the BIOS version/configuration on our box - Argonne/Intel - setup a meeting in two weeks to discuss progress. Scott will present his NEK results. kevin

1 0

[JLSE-Intel-NDA] KNL SNC-4 mode performance irregularity
by Scott Parker 26 Apr '16

26 Apr '16

On our KNL B0 I'm seeing some irregularities in the MCDRAM performance when running Stream in SNC-4 mode. I'm running Stream with 16 threads all bound to either domain 0, 1, 2, or 3 and memory bound to just one of the four MCDRAM domains (4,5,6,7). Below is a table of the results showing memory bandwidth in MB/sec, where the thread domain used is specified on the left the MCDRAM domain on the top of the table. Domain 4 never achieves more than 73 GB/s while the other domains (5,6,7) all reach 112 GB/s. MCDRAM NUMA Domain 4 5 6 7 0 73,306 84,053 107,483 55,978 Thread Domain 1 64,837 112,380 55,521 77,765 2 63,575 67,508 111,569 77,957 3 65,506 102,203 76,131 111,617 numactl -H shows that nodes 0,1,2,3 are associated with the DDR and 16 cores each while nodes 4,5,6,7 are MCDRAM nodes with no associated processors. However the following nodes are closest: Node 4 -> Node 0 Node 5 -> Node 1 Node 6 -> Node 2 Node 7 -> Node 3

1 0

Re: [JLSE-Intel-NDA] Incorrect documentation
by Vitali A. Morozov 25 Apr '16

25 Apr '16

Thanks! Vitali On 04/25/2016 05:22 PM, Allen, Benjamin S. wrote: > Done. > >> On Apr 25, 2016, at 2:55 PM, Vitali A. Morozov <morozov(a)anl.gov> wrote: >> >> Ben, >> >> KNL is hanging ... Need a reboot. >> >> Thanks, >> >> Vitali

1 0

Re: [JLSE-Intel-NDA] Incorrect documentation
by Allen, Benjamin S. 25 Apr '16

25 Apr '16

Done. > On Apr 25, 2016, at 2:55 PM, Vitali A. Morozov <morozov(a)anl.gov> wrote: > > Ben, > > KNL is hanging ... Need a reboot. > > Thanks, > > Vitali

1 0

Re: [JLSE-Intel-NDA] MPI and process binding
by Vitali A. Morozov 25 Apr '16

25 Apr '16

I do not know if this problem have been solved yet, but the variable I_MPI_PIN_PROCESSOR_LIST=0,16,32,48 should do the trick. Vitali p.s. The question remains whether Intel MPI uses the same processor ids as KMP_AFFINITY, but I_MPI_DEBUG should help to determine that. On 04/22/2016 02:02 PM, Harms, Kevin wrote: > I found this environment variable that prints out the process bound list. I_MPI_DEBUG=5 > > [harms@ftlogin1 tests]$ I_MPI_DEBUG=5 mpirun -n 2 ./mpi_test > [0] MPI startup(): Multi-threaded optimized library > [0] MPI startup(): shm data transfer mode > [1] MPI startup(): shm data transfer mode > [0] MPI startup(): Rank Pid Node name Pin cpu > [0] MPI startup(): 0 29236 ftlogin1 {0,1,2,3,8,9,10,11} > [0] MPI startup(): 1 29237 ftlogin1 {4,5,6,7,12,13,14,15} > [0] MPI startup(): I_MPI_DEBUG=5 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=mlx4_0:0 > [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 > [0] MPI startup(): I_MPI_PIN_MAPPING=2:0 0,1 4 > > > If Intel MPI pins the processes sanely on KNL, then coupling that with KMP_AFFINITY=balanced,granularity=thread should give you a good result. Here's an explanation of balanced: https://software.intel.com/en-us/node/522518 > > If Intel MPI doesn't know KNL and does something weird then you will have to try to understand KNL's tile/core numbering scheme which appears complex. > > kevin > > > >> KMP_AFFINITY only works for thread affinity though, no? If I have four MPI processes, each of the processes will assign threads to the same cores rather than cores in a different sub-NUMA cluster. I need to be able to specify process affinity >> and my problem is that 1) numactl can only control a single process and 2) the usual >> mpirun -bind-to and -map-by options don't really have enough levels of abstraction to match the KNL's hierarchy. For what it's worth, I tried >> mpirun -bind-to numa -map-by numa ... and that resulted in an error. It's not even clear, to me at least, what to expect if you say 'numa' -- one could interpret it to mean a tile (that shares L2 cache) or a cluster (that shares MCDRAM). >> >> On Fri, Apr 22, 2016 at 9:46 AM, Harms, Kevin >> <harms(a)alcf.anl.gov> wrote: >> >> >> KMP_AFFINITY=balanced is supposed to do that but if it's not working, it might be that the MPI library does not have proper tuning for KNL yet. >> >> kevin >> >> >> >> >>> I'm trying to test a code using hybrid MPI/OpenMP on the KNL node but haven't been able to successfully set process affinity. I'd like to run four processes in SNC-4 mode with each process being bound to a cluster. >>> Does anyone know if this is possible? >>> >>> Thanks, >>> Paul >>> >> >> >> >> > > > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Incorrect documentation
by Vitali A. Morozov 25 Apr '16

25 Apr '16

Ben, KNL is hanging ... Need a reboot. Thanks, Vitali

1 0

[JLSE-Intel-NDA] Incorrect documentation
by Vitali A. Morozov 25 Apr '16

25 Apr '16

Hello, The latest Intel compiler is printing incorrect documentation for "-qno-openmp" option. [morozov@thing00 ~]$ icc -help 2>&1 |grep qno-openmp Use -qno-openmp to disable -qopenmp. Use -qno-openmp-simd to disable. Use -qno-openmp-offload to disable. [morozov@thing00 ~]$ icc -qno-openmp icc: command line error: option '-qno-openmp' not supported Vitali

1 0