April 2016 - Intel-nda - mailman3.cels.anl.gov

Re: [JLSE-Intel-NDA] MPI and process binding
by Luo, Ye 29 Apr '16

29 Apr '16

Hi All, I noticed some strange behavior when more than 1 rank mpi is launched. I was testing with Intel compiler and mpi version 2016.2.181. export I_MPI_PIN_DOMAIN=auto export I_MPI_DEBUG=5 export KMP_AFFINITY="verbose,granularity=fine,scatter" When I place only 1 rank, threads spread on cores 0-73 correctly with some gaps due to the actually total number of cores, 64. OS proc 52 maps to package 0 core 62 thread 0 OS proc 53 maps to package 0 core 63 thread 0 OS proc 54 maps to package 0 core 64 thread 0 ... OS proc 63 maps to package 0 core 73 thread 0 When I place 2 ranks, threads only spread on cores 0-63, only 54 cores. I noticed the following debug message when 256 OS procs are mapped: OS proc 52 maps to package 0 core 62 thread 0 OS proc 53 maps to package 0 core 63 thread 0 OS proc 54 maps to package 0 core 0 thread 0 OS proc 55 maps to package 0 core 1 thread 0 ... OS proc 63 maps to package 0 core 9 thread 0 So the logic OS proc seems not mapped to the physical cores properly by the MPI at the beginning. I checked that the MPI processes and threads are bound to 256 OS procs correctly. However, I got the same performance using 1 rank x 128 threads and 2 ranks x 64 threads. Very puzzling to me. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory ________________________________ From: intel-nda-bounces(a)lists.jlse.anl.gov [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Vitali A. Morozov [morozov(a)anl.gov] Sent: Monday, April 25, 2016 5:22 PM To: Harms, Kevin; paul.k.romano(a)anl.gov Cc: intel-nda(a)jlse.anl.gov Subject: Re: [JLSE-Intel-NDA] MPI and process binding I do not know if this problem have been solved yet, but the variable I_MPI_PIN_PROCESSOR_LIST=0,16,32,48 should do the trick. Vitali p.s. The question remains whether Intel MPI uses the same processor ids as KMP_AFFINITY, but I_MPI_DEBUG should help to determine that. On 04/22/2016 02:02 PM, Harms, Kevin wrote: I found this environment variable that prints out the process bound list. I_MPI_DEBUG=5 [harms@ftlogin1 tests]$ I_MPI_DEBUG=5 mpirun -n 2 ./mpi_test [0] MPI startup(): Multi-threaded optimized library [0] MPI startup(): shm data transfer mode [1] MPI startup(): shm data transfer mode [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 29236 ftlogin1 {0,1,2,3,8,9,10,11} [0] MPI startup(): 1 29237 ftlogin1 {4,5,6,7,12,13,14,15} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=mlx4_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=2:0 0,1 4 If Intel MPI pins the processes sanely on KNL, then coupling that with KMP_AFFINITY=balanced,granularity=thread should give you a good result. Here's an explanation of balanced: https://software.intel.com/en-us/node/522518 If Intel MPI doesn't know KNL and does something weird then you will have to try to understand KNL's tile/core numbering scheme which appears complex. kevin KMP_AFFINITY only works for thread affinity though, no? If I have four MPI processes, each of the processes will assign threads to the same cores rather than cores in a different sub-NUMA cluster. I need to be able to specify process affinity and my problem is that 1) numactl can only control a single process and 2) the usual mpirun -bind-to and -map-by options don't really have enough levels of abstraction to match the KNL's hierarchy. For what it's worth, I tried mpirun -bind-to numa -map-by numa ... and that resulted in an error. It's not even clear, to me at least, what to expect if you say 'numa' -- one could interpret it to mean a tile (that shares L2 cache) or a cluster (that shares MCDRAM). On Fri, Apr 22, 2016 at 9:46 AM, Harms, Kevin <harms(a)alcf.anl.gov><mailto:[email protected]> wrote: KMP_AFFINITY=balanced is supposed to do that but if it's not working, it might be that the MPI library does not have proper tuning for KNL yet. kevin I'm trying to test a code using hybrid MPI/OpenMP on the KNL node but haven't been able to successfully set process affinity. I'd like to run four processes in SNC-4 mode with each process being bound to a cluster. Does anyone know if this is possible? Thanks, Paul > _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] knl00 is hanging
by Allen, Benjamin S. 29 Apr '16

29 Apr '16

Yea I power cycled it. Ben > On Apr 29, 2016, at 8:58 PM, Luo, Ye <yeluo(a)anl.gov> wrote: > > The node is back by itself. Ye > > =================== > Ye Luo, Ph.D. > Leadership Computing Facility > Argonne National Laboratory > From: intel-nda-bounces(a)lists.jlse.anl.gov [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Luo, Ye [yeluo(a)anl.gov] > Sent: Friday, April 29, 2016 6:42 PM > To: intel-nda(a)lists.jlse.anl.gov > Subject: [JLSE-Intel-NDA] knl00 is hanging > > Hi Ben, > Could you please reset the node? And label the machine schedule back to free. > Sorry for the inconvenience to everyone. > Ye > =================== > Ye Luo, Ph.D. > Leadership Computing Facility > Argonne National Laboratory > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] knl00 is hanging
by Luo, Ye 29 Apr '16

29 Apr '16

The node is back by itself. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory ________________________________ From: intel-nda-bounces(a)lists.jlse.anl.gov [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Luo, Ye [yeluo(a)anl.gov] Sent: Friday, April 29, 2016 6:42 PM To: intel-nda(a)lists.jlse.anl.gov Subject: [JLSE-Intel-NDA] knl00 is hanging Hi Ben, Could you please reset the node? And label the machine schedule back to free. Sorry for the inconvenience to everyone. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory

1 0

[JLSE-Intel-NDA] knl00 is hanging
by Luo, Ye 29 Apr '16

29 Apr '16

Hi Ben, Could you please reset the node? And label the machine schedule back to free. Sorry for the inconvenience to everyone. Ye =================== Ye Luo, Ph.D. Leadership Computing Facility Argonne National Laboratory

1 0

[JLSE-Intel-NDA] HSCD progress update
by Vitali A Morozov 28 Apr '16

28 Apr '16

A quick update on HSCD. In cache mode, Marta and I see a 2.6x core-to-core (over 10x node-to-node) performance improvement over BG/Q. Vitali On 4/28/16 10:21 AM, Harms, Kevin wrote: > I briefly ran two of the CORAL benchmarks yesterday. > > AMG2013 > - showed a 2x improvement in FOM when running from IPM > - I ran it as 64 MPI ranks with -P 2 2 2. > > QBOX > - showed no improvement when running in IPM > - requires FFTW 2.1.5 and MKL > - I built FFTW but have no idea how well. (I just used -O3 -xMIC-AVX512) > - MKL has a number of libraries and variants. I selected ones that successfully linked but I have no idea if they were leveraging the KNL properly. > > kevin > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] Stream bandwidth in cached mode
by Scott Parker 28 Apr '16

28 Apr '16

Testing Stream in cache mode shows Triad bandwidth of 276 GB/s vs 420-440 GB/s in flat mode. So in cache mode we are only seeing about 65% of the flat mode bandwidth. In the context of Aurora Intel had previously mentioned an expected 25% drop in Stream bandwidth in cached mode. This was due to the fact that an effective additional read is required in cached mode for each write. The read operation is needed to determine if the line being written is in the IPM cache or not. This explanation was given in the context non-temporal streaming stores, but I believe they also mentioned the bandwidth would be the same for regular stores. ------------ Stream Results ------------ $ KMP_AFFINITY=scatter OMP_NUM_THREADS=64 ./stream_mod ------------------------------------------------------------- STREAM version $Revision: 5.10 $ ------------------------------------------------------------- This system uses 8 bytes per array element. ------------------------------------------------------------- Array size = 67108864 (elements), Offset = 0 (elements) Memory per array = 512.0 MiB (= 0.5 GiB). Total memory required = 1536.0 MiB (= 1.5 GiB). Each kernel will be executed 10 times. The *best* time for each kernel (excluding the first iteration) will be used to compute the reported bandwidth. ------------------------------------------------------------- Number of Threads requested = 64 Number of Threads counted = 64 ------------------------------------------------------------- Your clock granularity/precision appears to be 1 microseconds. Each test below will take on the order of 3100 microseconds. (= 3100 clock ticks) Increase the size of the arrays if this shows that you are not getting at least 20 clock ticks per test. ------------------------------------------------------------- WARNING -- The above is only a rough guideline. For best results, please be sure you know the precision of your system timer. ------------------------------------------------------------- Function Best Rate MB/s Avg time Min time Max time Copy: 223981.7 0.004823 0.004794 0.004843 Scale: 223415.0 0.004861 0.004806 0.005109 Add: 275595.6 0.005876 0.005844 0.005897 Triad: 275786.9 0.005873 0.005840 0.005899 ------------------------------------------------------------- Solution Validates: avg error less than 1.000000e-13 on all three arrays -------------------------------------------------------------

1 0

Re: [JLSE-Intel-NDA] AMG2013 and QBOX benchmarks
by Alvaro 28 Apr '16

28 Apr '16

Kevin, I compiled the FFTW libraries with the interfaces provided by MKL for KNL ( See: /soft/compilers/intel/compilers_and_libraries_2016.2.181/linux/mkl/interfaces/ ) they are presumably faster than those from fftw.org. You can find the headers here /soft/compilers/intel/compilers_and_libraries_2016.2.181/linux/mkl/include/fftw/ and the libraries here /home/vama/install/local/fftw2knl best On 04/28/2016 10:21 AM, Harms, Kevin wrote: > I briefly ran two of the CORAL benchmarks yesterday. > > AMG2013 > - showed a 2x improvement in FOM when running from IPM > - I ran it as 64 MPI ranks with -P 2 2 2. > > QBOX > - showed no improvement when running in IPM > - requires FFTW 2.1.5 and MKL > - I built FFTW but have no idea how well. (I just used -O3 -xMIC-AVX512) > - MKL has a number of libraries and variants. I selected ones that successfully linked but I have no idea if they were leveraging the KNL properly. > > kevin > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda -- -Alvaro http://web.alcf.anl.gov/~vama/

1 0

Re: [JLSE-Intel-NDA] AMG2013 and QBOX benchmarks
by Kumaran, Kalyan 28 Apr '16

28 Apr '16

Great. Scott was complaining that AMG did not run on A0 when compiled with AVX flags. -Kumar > On Apr 28, 2016, at 10:21 AM, Harms, Kevin <harms(a)alcf.anl.gov> wrote: > > > I briefly ran two of the CORAL benchmarks yesterday. > > AMG2013 > - showed a 2x improvement in FOM when running from IPM > - I ran it as 64 MPI ranks with -P 2 2 2. > > QBOX > - showed no improvement when running in IPM > - requires FFTW 2.1.5 and MKL > - I built FFTW but have no idea how well. (I just used -O3 -xMIC-AVX512) > - MKL has a number of libraries and variants. I selected ones that successfully linked but I have no idea if they were leveraging the KNL properly. > > kevin_______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] AMG2013 and QBOX benchmarks
by Harms, Kevin 28 Apr '16

28 Apr '16

I briefly ran two of the CORAL benchmarks yesterday. AMG2013 - showed a 2x improvement in FOM when running from IPM - I ran it as 64 MPI ranks with -P 2 2 2. QBOX - showed no improvement when running in IPM - requires FFTW 2.1.5 and MKL - I built FFTW but have no idea how well. (I just used -O3 -xMIC-AVX512) - MKL has a number of libraries and variants. I selected ones that successfully linked but I have no idea if they were leveraging the KNL properly. kevin

1 0

[JLSE-Intel-NDA] Fwd: Stream
by Kumaran, Kalyan 27 Apr '16

27 Apr '16

Begin forwarded message: From: "Chinthamani, Sundaram" <sundaram.chinthamani(a)intel.com<mailto:[email protected]>> Subject: RE: Stream Date: April 27, 2016 at 10:58:11 AM CDT To: "Sodani, Avinash" <avinash.sodani(a)intel.com<mailto:[email protected]>>, "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov<mailto:[email protected]>> Cc: "Allison, CM" <cm.allison(a)intel.com<mailto:[email protected]>>, "Kirkendall, Keith G" <keith.g.kirkendall(a)intel.com<mailto:[email protected]>>, "Fromkin, Russ" <russ.fromkin(a)intel.com<mailto:[email protected]>>, "Chinthamani, Sundaram" <sundaram.chinthamani(a)intel.com<mailto:[email protected]>> Thanks. Then, All2All (420 GB/s) and Quad mode (440 GB/s) are delivering performance as expected. At this point, we are checking on whether we can re-produce this issue internally? When I get time, I will also log-in into your system and see what is going on? -Thanks, -Sundaram From: Sodani, Avinash Sent: Wednesday, April 27, 2016 8:32 AM To: Kumaran, Kalyan <kumaran(a)alcf.anl.gov<mailto:[email protected]>>; Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com<mailto:[email protected]>> Cc: Allison, CM <cm.allison(a)intel.com<mailto:[email protected]>>; Kirkendall, Keith G <keith.g.kirkendall(a)intel.com<mailto:[email protected]>>; Fromkin, Russ <russ.fromkin(a)intel.com<mailto:[email protected]>>; Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>> Subject: RE: Stream Thanks Kumar. That at least confirms that quadrant mode bandwidth is as expected. -Avinash From: Kumaran, Kalyan [mailto:[email protected]] Sent: Wednesday, April 27, 2016 7:24 AM To: Sodani, Avinash; Chinthamani, Sundaram Cc: Allison, CM; Kirkendall, Keith G; Fromkin, Russ Subject: Stream Begin forwarded message: From: Scott Parker <sparker(a)anl.gov<mailto:[email protected]>> Subject: Re: [JLSE-Intel-NDA] Intel Meeting Notes Date: April 26, 2016 at 5:50:32 PM CDT Re-ran stream in Quadrant mode and I was able to get 435 GB/s with 1 thread per core, and a very slightly better 440 GB/s with 2 threads per core. In SNC-4 mode I saw 311 GB/s with 1 thread and 442 GB/s with 2 threads. All2All hit 425 GB/s with 1 thread per core. A peak of around 440 GB/s is basically what we should be expecting to see for a Bin 3 part, so the MCDRAM seems to be capable of delivering the expected bandwidth. However, SNC-4 is an outlier in requiring 128 threads, and gives unexpected low (73 GB/s) bandwidth from one of the clusters (vs 112 GB/s from each of the other three). -Scott

1 0