- Intel-nda - mailman3.cels.anl.gov

Re: [JLSE-Intel-NDA] KNL Status
by Harms, Kevin 21 Apr '16

21 Apr '16

> >I am curious what was changed (or added) in version 17... > >Vitali They removed FORTRAN support... OH SNAP! kevin

1 0

Re: [JLSE-Intel-NDA] KNL Status
by Vitali A Morozov 21 Apr '16

21 Apr '16

Yeah - as Ronald Green has figured out, the beta version of a compiler (17) does make something differently compared to production version 16. For some reason, it only matters in SNC-4 mode and not in All2All mode. I am curious what was changed (or added) in version 17... Vitali On 4/21/16 9:45 AM, Kumaran, Kalyan wrote: > Looks like Scott Parker gets better results: > > Adding another data point, I ran in SNC-4 mode with the configuration > below and observed a consistent BW around 440 GB/s > with the best run producing 442 GB/s. In this case 128 threads gave > the best results. > > I've noticed that the STREAM results sensitivity to build and run > parameters seems to vary depending on the memory configuration > in flat-all2all I saw noticeably better results with memory aligned on > 2MB boundaries and 64 threads, in SNC-4 mode the alignment > doesn't seem to matter and 128 threads is producing the best results. > > > Mode: flat & SNC-4 > Compiler: Intel 17 Beta > Compile Options: icc -O3 -fopenmp -DSTREAM_ARRAY_SIZE=134217728 > -xMIC-AVX512 -mcmodel medium -shared-intel > Run Command: OMP_NUM_THREADS=128 numactl -m 4,5,6,7 ./stream_mod > > Results: > Function Best Rate MB/s Avg time Min time Max time > Copy: 390378.3 0.005524 0.005501 0.005546 > Scale: 412738.8 0.005237 0.005203 0.005278 > Add: 448759.4 0.007199 0.007178 0.007218 > Triad: 441919.3 0.007355 0.007289 0.007548 > > > >> On Apr 20, 2016, at 7:39 PM, Vitali A Morozov <morozov(a)anl.gov >> <mailto:[email protected]>> wrote: >> >> Hi, >> >> a quick update on STREAM with the latest beta compiler >> >> [morozov@knl00 STREAM]$ icc -V >> Intel(R) C Intel(R) 64 Compiler for applications running on Intel(R) >> 64, Version 17.0.0.042 Beta Build 20160315 >> >> compiler options >> mpiicc -O3 -xMIC-AVX512 -qopt-report=5 -Wall -g -fopenmp -mcmodel >> medium -shared-intel -qopt-streaming-stores always -restrict stream.c >> -o stream >> >> Running in flat mode all2all >> >> morozov@knl00 STREAM]$ export OMP_NUM_THREADS=256 >> [morozov@knl00 STREAM]$ numactl -m 1 ./stream >> ------------------------------------------------------------- >> STREAM version $Revision: 5.9 $ >> ------------------------------------------------------------- >> This system uses 8 bytes per DOUBLE PRECISION word. >> ------------------------------------------------------------- >> Array size = 134217728, Offset = 0 >> Total memory required = 3072.0 MB. >> Each test is run 500 times, but only >> the *best*time for each is used. >> ------------------------------------------------------------- >> Number of Threads requested = 256 >> ------------------------------------------------------------- >> <skipped> >> ------------------------------------------------------------- >> Your clock granularity/precision appears to be 1 microseconds. >> Each test below will take on the order of 6356 microseconds. >> (= 6356 clock ticks) >> Increase the size of the arrays if this shows that >> you are not getting at least 20 clock ticks per test. >> ------------------------------------------------------------- >> WARNING -- The above is only a rough guideline. >> For best results, please be sure you know the >> precision of your system timer. >> ------------------------------------------------------------- >> Function Rate (MB/s) Avg time Min time Max time >> Copy: 330891.5637 0.0066 0.0065 0.0078 >> Scale: 330127.5200 0.0066 0.0065 0.0069 >> Add: 383622.4448 0.0085 0.0084 0.0093 >> Triad: 383252.4574 0.0085 0.0084 0.0098 >> ------------------------------------------------------------- >> Solution Validates >> ------------------------------------------------------------- >> >> Same with 64 threads scatter mode: >> >> morozov@knl00 STREAM]$ export OMP_NUM_THREADS=64 >> [morozov@knl00 STREAM]$ KMP_AFFINITY=scatter >> [morozov@knl00 STREAM]$ numactl -m 1 ./stream >> ------------------------------------------------------------- >> STREAM version $Revision: 5.9 $ >> ------------------------------------------------------------- >> This system uses 8 bytes per DOUBLE PRECISION word. >> ------------------------------------------------------------- >> Array size = 134217728, Offset = 0 >> Total memory required = 3072.0 MB. >> Each test is run 500 times, but only >> the *best*time for each is used. >> ------------------------------------------------------------- >> Number of Threads requested = 64 >> ------------------------------------------------------------- >> <skipped> >> ------------------------------------------------------------- >> Your clock granularity/precision appears to be 1 microseconds. >> Each test below will take on the order of 6056 microseconds. >> (= 6056 clock ticks) >> Increase the size of the arrays if this shows that >> you are not getting at least 20 clock ticks per test. >> ------------------------------------------------------------- >> WARNING -- The above is only a rough guideline. >> For best results, please be sure you know the >> precision of your system timer. >> ------------------------------------------------------------- >> Function Rate (MB/s) Avg time Min time Max time >> Copy: 353486.8826 0.0061 0.0061 0.0067 >> Scale: 353265.0608 0.0061 0.0061 0.0066 >> Add: 423961.3055 0.0077 0.0076 0.0079 >> Triad: 424907.9750 0.0076 0.0076 0.0084 >> ------------------------------------------------------------- >> Solution Validates >> ------------------------------------------------------------- >> >> Vitali >> >> >> >> >> On 4/18/16 12:43 PM, Kumaran, Kalyan wrote: >>> FYI. Karthik will get on the machine to debug. >>> >>> Sent from my iPhone >>> >>> Begin forwarded message: >>> >>>> *From:* "Chinthamani, Sundaram" <sundaram.chinthamani(a)intel.com> >>>> *Date:* April 18, 2016 at 2:05:26 PM CDT >>>> *To:* "Harms, Kevin" <harms(a)alcf.anl.gov >>>> <mailto:[email protected]>>, "Allen, Benjamin S." >>>> <bsallen(a)alcf.anl.gov <mailto:[email protected]>> >>>> *Cc:* "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov >>>> <mailto:[email protected]>>, "Kaana Balakrishna, Abhishek" >>>> <abhishek.kaana.balakrishna(a)intel.com >>>> <mailto:[email protected]>>, "Morozov, Vitali >>>> A." <morozov(a)alcf.anl.gov <mailto:[email protected]>>, "Raman, >>>> Karthik" <karthik.raman(a)intel.com >>>> <mailto:[email protected]>>, "Inkley, Benson" >>>> <benson.inkley(a)intel.com <mailto:[email protected]>>, >>>> "Sodani, Avinash" <avinash.sodani(a)intel.com >>>> <mailto:[email protected]>>, "Fromkin, Russ" >>>> <russ.fromkin(a)intel.com <mailto:[email protected]>>, >>>> "Chinthamani, Sundaram" <sundaram.chinthamani(a)intel.com >>>> <mailto:[email protected]>> >>>> *Subject:* *RE: [JLSE-Intel-NDA] KNL Status* >>>> >>>> I would expect Quad mode / SNC-4 mode to provide higher bandwidth >>>> compared to All2All mode. We haven't seen this in our internal >>>> measurements. >>>> >>>> -Thanks, >>>> -Sundaram >>>> >>>> -----Original Message----- >>>> From: Harms, Kevin [mailto:[email protected]] >>>> Sent: Monday, April 18, 2016 12:00 PM >>>> To: Allen, Benjamin S. <bsallen(a)alcf.anl.gov >>>> <mailto:[email protected]>> >>>> Cc: Kumaran, Kalyan <kumaran(a)alcf.anl.gov >>>> <mailto:[email protected]>>; Kaana Balakrishna, Abhishek >>>> <abhishek.kaana.balakrishna(a)intel.com >>>> <mailto:[email protected]>>; Morozov, Vitali A. >>>> <morozov(a)alcf.anl.gov <mailto:[email protected]>>; Chinthamani, >>>> Sundaram <sundaram.chinthamani(a)intel.com >>>> <mailto:[email protected]>>; Raman, Karthik >>>> <karthik.raman(a)intel.com <mailto:[email protected]>>; Inkley, >>>> Benson <benson.inkley(a)intel.com <mailto:[email protected]>>; >>>> Sodani, Avinash <avinash.sodani(a)intel.com >>>> <mailto:[email protected]>>; Fromkin, Russ >>>> <russ.fromkin(a)intel.com <mailto:[email protected]>> >>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>> >>>> >>>> I tested cached mode with all2all, snc4 and quad models. Cached >>>> all2all gave 275,000 MB/s. Does that look reasonable? SNC4 and quad >>>> mode gave 225,000 MB/s or so. >>>> >>>> kevin >>>> >>>> >>>> >>>> >>>>> The following versions are now installed: BIOS 01.01.0104, ME >>>>> 03.01.03.011.0, BMC 00.17.9699, FRUSDR 1.14. >>>>> >>>>> Ben >>>>> >>>>>> On Apr 18, 2016, at 1:27 PM, Harms, Kevin <harms(a)alcf.anl.gov >>>>>> <mailto:[email protected]>> wrote: >>>>>> >>>>>> >>>>>> Ben completed the BIOS upgrade, but we do not see any difference >>>>>> in performance. I have attached results from runs done under >>>>>> three configurations: flat+all2all, flat+quad, flat+snc4. >>>>>> >>>>>> flat+all2all = 427046.0546 MB/s >>>>>> flat+quad = 327253.8882 MB/s >>>>>> flat+snc4 = 308555.9588 MB/s >>>>>> >>>>>> I compiled the CORAL stream.c with the following options: >>>>>> icc -mcmodel medium -shared-intel -O3 -xMIC-AVX512 -DN=50000000 >>>>>> -DOFFSET=0 -DNTIMES=100 -openmp -opt-streaming-stores always >>>>>> stream.c -o stream >>>>>> >>>>>> I added the alignment directives to the arrays. The attached >>>>>> files show the NUMA domains and numactl commands used and run >>>>>> parameters. >>>>>> >>>>>> Have I missed a parameter? >>>>>> >>>>>> kevin >>>>>> >>>>>> >>>>>> >>>>>>> Suggest a time Friday afternoon. We are in central time zone. We >>>>>>> can get you access to this B0 too. >>>>>>> >>>>>>> -Kumar >>>>>>> >>>>>>> >>>>>>>> On Apr 13, 2016, at 5:06 PM, Kaana Balakrishna, Abhishek >>>>>>>> <abhishek.kaana.balakrishna(a)intel.com> wrote: >>>>>>>> >>>>>>>> + Benson >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Abhishek K Balakrishna (KB) >>>>>>>> Xeon Phiï¿½ Hardware Platform Applications Engineer, >>>>>>>> Technical Computing Group (TCG) >>>>>>>> Intel Corporation >>>>>>>> Ph - 512 466 9886 >>>>>>>> www.intel.com/xeonphi <http://www.intel.com/xeonphi> >>>>>>>> >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>> Sent: Wednesday, April 13, 2016 4:58 PM >>>>>>>> To: Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com >>>>>>>> <mailto:[email protected]>>; Raman, Karthik >>>>>>>> <karthik.raman(a)intel.com <mailto:[email protected]>>; >>>>>>>> Harms, Kevin <harms(a)alcf.anl.gov <mailto:[email protected]>>; >>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov >>>>>>>> <mailto:[email protected]>>; Kumaran, Kalyan >>>>>>>> <kumaran(a)alcf.anl.gov <mailto:[email protected]>> >>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; Fromkin, Russ >>>>>>>> <russ.fromkin(a)intel.com <mailto:[email protected]>>; >>>>>>>> Allen, Benjamin S. <bsallen(a)alcf.anl.gov >>>>>>>> <mailto:[email protected]>>; Kaana Balakrishna, Abhishek >>>>>>>> <abhishek.kaana.balakrishna(a)intel.com> >>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>> >>>>>>>> Sure! >>>>>>>> >>>>>>>> Vitali >>>>>>>> >>>>>>>> >>>>>>>> On 04/13/2016 04:54 PM, Chinthamani, Sundaram wrote: >>>>>>>>> Would it be possible to have a meeting to go over your >>>>>>>>> questions rather than exchanging E-mails back and forth? >>>>>>>>> >>>>>>>>> + KB from Intel side. He can help set-up the meeting between >>>>>>>>> Labs and Intel. >>>>>>>>> >>>>>>>>> -Thanks, >>>>>>>>> -Sundaram >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>> Sent: Wednesday, April 13, 2016 2:50 PM >>>>>>>>> To: Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com >>>>>>>>> <mailto:[email protected]>>; Raman, >>>>>>>>> Karthik <karthik.raman(a)intel.com>; Harms, Kevin >>>>>>>>> <harms(a)alcf.anl.gov <mailto:[email protected]>>; >>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov>; Kumaran, Kalyan >>>>>>>>> <kumaran(a)alcf.anl.gov> >>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; Fromkin, Russ >>>>>>>>> <russ.fromkin(a)intel.com>; Allen, Benjamin S. >>>>>>>>> <bsallen(a)alcf.anl.gov <mailto:[email protected]>> >>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>> >>>>>>>>> On 04/13/2016 04:42 PM, Chinthamani, Sundaram wrote: >>>>>>>>>> node 4 free: 2907 MB >>>>>>>>>> node 5 free: 2926 MB >>>>>>>>>> node 6 free: 2922 MB >>>>>>>>>> node 7 free: 2914 MB >>>>>>>>>> >>>>>>>>>> There is 16 GB of MC-DRAM and 4 GB per NUMA node. Is there >>>>>>>>>> any other process running on the machine? Why does "free >>>>>>>>>> memory" show only 3 GB per node? >>>>>>>>> This is EXACTLY why I am asking this question. Nothing is >>>>>>>>> running on the node - and I do not see full 4 GiB of available >>>>>>>>> MCDRAM memory. This might be reason why I do not see what is >>>>>>>>> expected. >>>>>>>>> >>>>>>>>> I understand that I can reduce the array size, but I am trying >>>>>>>>> to reproduce your results. >>>>>>>>> >>>>>>>>> I am still unclear on if you run "no mpi" version, how come >>>>>>>>> that SNC-4 mode is giving you advantage? All arrays are >>>>>>>>> allocated in the same quadrant. Requests from the other >>>>>>>>> quadrants will go to this one and create congestion. >>>>>>>>> >>>>>>>>> Vitali >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> If you want to test Stream triad bandwidth to MC-DRAM in >>>>>>>>>> Quad/Flat mode, you can use even smaller arrays of few 100 MBs. >>>>>>>>>> >>>>>>>>>> -Thanks, >>>>>>>>>> -Sundaram >>>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>> Sent: Wednesday, April 13, 2016 2:32 PM >>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com >>>>>>>>>> <mailto:[email protected]>>; Chinthamani, Sundaram >>>>>>>>>> <sundaram.chinthamani(a)intel.com>; Harms, Kevin >>>>>>>>>> <harms(a)alcf.anl.gov <mailto:[email protected]>>; >>>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov >>>>>>>>>> <mailto:[email protected]>>; Kumaran, Kalyan >>>>>>>>>> <kumaran(a)alcf.anl.gov> >>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>> <mailto:[email protected]>>; Fromkin, Russ >>>>>>>>>> <russ.fromkin(a)intel.com>; Allen, Benjamin S. >>>>>>>>>> <bsallen(a)alcf.anl.gov <mailto:[email protected]>> >>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>> >>>>>>>>>> ok, that is fine. >>>>>>>>>> >>>>>>>>>> Now the next issue I have is that for static arrays, all >>>>>>>>>> three arrays must land up on one of the numa nodes - the node >>>>>>>>>> that is attached to a master thread. >>>>>>>>>> >>>>>>>>>> You advice to use N = 134217728, that gives exactly 3 GiB of >>>>>>>>>> memory footprint for 3 arrays of triad. >>>>>>>>>> >>>>>>>>>> According to "numactl -H", I do not have 3 GiB of available >>>>>>>>>> memory on any of the MCDRAM numa nodes: >>>>>>>>>> >>>>>>>>>> numactl -H |grep free >>>>>>>>>> node 0 free: 46336 MB >>>>>>>>>> node 1 free: 46923 MB >>>>>>>>>> node 2 free: 46970 MB >>>>>>>>>> node 3 free: 46961 MB >>>>>>>>>> node 4 free: 2907 MB >>>>>>>>>> node 5 free: 2926 MB >>>>>>>>>> node 6 free: 2922 MB >>>>>>>>>> node 7 free: 2914 MB >>>>>>>>>> >>>>>>>>>> Therefore, at least one array will not be placed to the same >>>>>>>>>> numa node. >>>>>>>>>> Is this ok? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 04/13/2016 04:17 PM, Raman, Karthik wrote: >>>>>>>>>>> Yes we run pure OMP code too. >>>>>>>>>>> >>>>>>>>>>> The problem with KMP_AFFINITY with SNC4 mode (currently) is >>>>>>>>>>> that it >>>>>>>>>>> may not equally balance the threads per NUMA domain even with >>>>>>>>>>> scatter or balanced, you can check that by mapping the >>>>>>>>>>> "verbose" output (e.g. >>>>>>>>>>> KMP_AFFINITY=scatter,verbose) with "numactl -H" output to >>>>>>>>>>> see if the >>>>>>>>>>> OS thread to logical CPU mapping is balanced across all NUMA >>>>>>>>>>> domains >>>>>>>>>>> >>>>>>>>>>> With All2All/Quad this should not be a problem >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>> Sent: Wednesday, April 13, 2016 1:53 PM >>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com >>>>>>>>>>> <mailto:[email protected]>>; Chinthamani, Sundaram >>>>>>>>>>> <sundaram.chinthamani(a)intel.com>; Harms, Kevin >>>>>>>>>>> <harms(a)alcf.anl.gov <mailto:[email protected]>>; >>>>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov >>>>>>>>>>> <mailto:[email protected]>>; Kumaran, Kalyan >>>>>>>>>>> <kumaran(a)alcf.anl.gov> >>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>> <mailto:[email protected]>>; Fromkin, Russ >>>>>>>>>>> <russ.fromkin(a)intel.com>; Allen, Benjamin S. >>>>>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>> >>>>>>>>>>> Before doing that - let us check if we are running similar >>>>>>>>>>> benchmark. >>>>>>>>>>> >>>>>>>>>>> I am running 1 MPI process (in fact no mpi at all) and 64 >>>>>>>>>>> OpenMP threads version. The threads can be placed with >>>>>>>>>>> "balanced" or "scattered" >>>>>>>>>>> policy. I am using "scattered". >>>>>>>>>>> >>>>>>>>>>> Is this ok so far? >>>>>>>>>>> >>>>>>>>>>> Vitali >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 04/13/2016 03:48 PM, Raman, Karthik wrote: >>>>>>>>>>>> This still does not look right, and the reason could be >>>>>>>>>>>> that the >>>>>>>>>>>> threads are not balanced across all the NUMA domains >>>>>>>>>>>> (ideally you >>>>>>>>>>>> want >>>>>>>>>>>> 16 threads per quadrant) >>>>>>>>>>>> >>>>>>>>>>>> Before going to SNC4, can you please boot in "Quad" mode. >>>>>>>>>>>> If you >>>>>>>>>>>> look in the BIOS Cluster mode options there is another mode >>>>>>>>>>>> called "Quad" >>>>>>>>>>>> which is different from SNC4, only 2 NUMA domains (DDR, >>>>>>>>>>>> MCDRAM). >>>>>>>>>>>> Please run similar to All2All mode >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>> Sent: Wednesday, April 13, 2016 12:56 PM >>>>>>>>>>>> To: Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com >>>>>>>>>>>> <mailto:[email protected]>>; Harms, >>>>>>>>>>>> Kevin <harms(a)alcf.anl.gov>; Raman, Karthik >>>>>>>>>>>> <karthik.raman(a)intel.com>; Morozov, Vitali A. >>>>>>>>>>>> <morozov(a)alcf.anl.gov>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov> >>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>>> <mailto:[email protected]>>; Fromkin, Russ >>>>>>>>>>>> <russ.fromkin(a)intel.com>; Allen, Benjamin S. >>>>>>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>> >>>>>>>>>>>> ok, I am now in SNC-4/Flat. Same binary. >>>>>>>>>>>> >>>>>>>>>>>> OMP_NUM_THREADS=64 KMP_AFFINITY=scatter numactl -m 4,5,6,7 >>>>>>>>>>>> ./stream >>>>>>>>>>>> |tee running.dat >>>>>>>>>>>> >>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>> STREAM version $Revision: 5.9 $ >>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>> This system uses 8 bytes per DOUBLE PRECISION word. >>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>> Array size = 134217728, Offset = 0 >>>>>>>>>>>> Total memory required = 3072.0 MB. >>>>>>>>>>>> Each test is run 100 times, but only the *best* time for >>>>>>>>>>>> each is >>>>>>>>>>>> used. >>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>> Number of Threads requested = 64 >>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>> >>>>>>>>>>>> <skipped> >>>>>>>>>>>> >>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>> Function Rate (MB/s) Avg time Min time Max time >>>>>>>>>>>> Copy: 320997.8352 0.0068 0.0067 0.0071 >>>>>>>>>>>> Scale: 319994.2893 0.0068 0.0067 0.0073 >>>>>>>>>>>> Add: 306055.0205 0.0106 0.0105 0.0107 >>>>>>>>>>>> Triad: 306055.0205 0.0106 0.0105 0.0109 >>>>>>>>>>>> Store: 247056.8669 0.0044 0.0043 0.0051 >>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>> Solution Validates >>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 306055 MB/s >>>>>>>>>>>> >>>>>>>>>>>> Vitali >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 04/13/2016 02:01 PM, Chinthamani, Sundaram wrote: >>>>>>>>>>>>> 440 GB/s (base 10). To be precise, 440 bytes / ns >>>>>>>>>>>>> (nano-second). We always report memory bandwidth in GB/s >>>>>>>>>>>>> (base 10) within Intel. >>>>>>>>>>>>> >>>>>>>>>>>>> -Thanks, >>>>>>>>>>>>> -Sundaram >>>>>>>>>>>>> >>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>> From: Harms, Kevin [mailto:[email protected]] >>>>>>>>>>>>> Sent: Wednesday, April 13, 2016 11:52 AM >>>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com >>>>>>>>>>>>> <mailto:[email protected]>>; Chinthamani, >>>>>>>>>>>>> Sundaram <sundaram.chinthamani(a)intel.com>; Morozov, Vitali A. >>>>>>>>>>>>> <morozov(a)alcf.anl.gov>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>>>> <mailto:[email protected]>>; Fromkin, Russ >>>>>>>>>>>>> <russ.fromkin(a)intel.com>; Allen, Benjamin S. >>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Just to be clear, memory is always assumed to be >>>>>>>>>>>>> measured using base 2. It's only the storage field that >>>>>>>>>>>>> uses base 10. So we assume if you say 440 GB/s you mean >>>>>>>>>>>>> GiB, or 450,560 MB/s. >>>>>>>>>>>>> >>>>>>>>>>>>> kevin >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> And just some fun with units >>>>>>>>>>>>>> http://sites.utexas.edu/jdm4372/2013/01/05/counting-binary-vs-dec >>>>>>>>>>>>>> i >>>>>>>>>>>>>> m >>>>>>>>>>>>>> a >>>>>>>>>>>>>> l >>>>>>>>>>>>>> -powers-in-the-stream-benchmark/ >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> From: Chinthamani, Sundaram >>>>>>>>>>>>>> >>>>>>>>>>>>>> Sent: Wednesday, April 13, 2016 11:20 AM >>>>>>>>>>>>>> To: Vitali A. Morozov <morozov(a)anl.gov >>>>>>>>>>>>>> <mailto:[email protected]>>; Raman, Karthik >>>>>>>>>>>>>> <karthik.raman(a)intel.com>; Kumaran, Kalyan >>>>>>>>>>>>>> <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>>>>> <mailto:[email protected]>>; Fromkin, Russ >>>>>>>>>>>>>> <russ.fromkin(a)intel.com>; Harms, Kevin <harms(a)alcf.anl.gov>; >>>>>>>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov >>>>>>>>>>>>>> <mailto:[email protected]>>; Allen, Benjamin S. >>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov>; Chinthamani, Sundaram >>>>>>>>>>>>>> <sundaram.chinthamani(a)intel.com> >>>>>>>>>>>>>> Subject: RE: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 440 GB/s stream triad for MC-DRAM and 80 GB/s is our >>>>>>>>>>>>>> internal >>>>>>>>>>>>>> bin3 measurement. You should be able to get closer these >>>>>>>>>>>>>> numbers, if you run the benchmark in Quad/Flat or >>>>>>>>>>>>>> SNC-4/Flat mode. >>>>>>>>>>>>>> >>>>>>>>>>>>>> -Thanks, >>>>>>>>>>>>>> -Sundaram >>>>>>>>>>>>>> >>>>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>>>> >>>>>>>>>>>>>> Sent: Wednesday, April 13, 2016 11:11 AM >>>>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com >>>>>>>>>>>>>> <mailto:[email protected]>>; Chinthamani, >>>>>>>>>>>>>> Sundaram <sundaram.chinthamani(a)intel.com>; Kumaran, Kalyan >>>>>>>>>>>>>> <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>>>>> <mailto:[email protected]>>; Fromkin, Russ >>>>>>>>>>>>>> <russ.fromkin(a)intel.com>; Harms, Kevin <harms(a)alcf.anl.gov>; >>>>>>>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov >>>>>>>>>>>>>> <mailto:[email protected]>>; Allen, Benjamin S. >>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I see - I am treating 418122 MB/s as 418122/1024 GB/s (as >>>>>>>>>>>>>> this is memory unit). >>>>>>>>>>>>>> >>>>>>>>>>>>>> Anyway - great we are on the same page here. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I am in All2All/Flat mode. You seem to suggest that I >>>>>>>>>>>>>> must be on SNC-4/Flat to see 440000 MB/s? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>> >>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 04/13/2016 01:06 PM, Raman, Karthik wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Ok I read this as 418 GB/s >>>>>>>>>>>>>> J >>>>>>>>>>>>>> >>>>>>>>>>>>>> And ok, just realized you are running in All2All mode, so >>>>>>>>>>>>>> yes this looks good. For Quad/Flat you should see ~440 >>>>>>>>>>>>>> GB/s with the same run. >>>>>>>>>>>>>> >>>>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>>>> >>>>>>>>>>>>>> Sent: Wednesday, April 13, 2016 11:02 AM >>>>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com >>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>> <mailto:[email protected]>; Chinthamani, Sundaram >>>>>>>>>>>>>> <sundaram.chinthamani(a)intel.com> >>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>> Kumaran, Kalyan <kumaran(a)alcf.anl.gov >>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>> Fromkin, Russ <russ.fromkin(a)intel.com >>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>> <mailto:[email protected]>; Harms, Kevin >>>>>>>>>>>>>> <harms(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>> Morozov, Vitali A. >>>>>>>>>>>>>> <morozov(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>> Allen, Benjamin S. >>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> <mailto:[email protected]> >>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Karthik, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Repeating the same options you have: >>>>>>>>>>>>>> >>>>>>>>>>>>>> mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp >>>>>>>>>>>>>> -mcmodel >>>>>>>>>>>>>> medium -shared-intel -opt-streaming-stores always stream.c -o >>>>>>>>>>>>>> stream >>>>>>>>>>>>>> >>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>> STREAM version $Revision: 5.9 $ >>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>> This system uses 8 bytes per DOUBLE PRECISION word. >>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>> Array size = 134217728, Offset = 0 Total memory required >>>>>>>>>>>>>> = 3072.0 >>>>>>>>>>>>>> MB. >>>>>>>>>>>>>> Each test is run 100 times, but only the *best* time for >>>>>>>>>>>>>> each is >>>>>>>>>>>>>> used. >>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>> Number of Threads requested = 64 >>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>> >>>>>>>>>>>>>> < skiped > >>>>>>>>>>>>>> >>>>>>>>>>>>>> Function Rate (MB/s) Avg time Min time Max >>>>>>>>>>>>>> time >>>>>>>>>>>>>> Copy: 362331.5200 0.0060 0.0059 0.0061 >>>>>>>>>>>>>> Scale: 359281.9806 0.0060 0.0060 0.0062 >>>>>>>>>>>>>> Add: 418122.7024 0.0078 0.0077 0.0081 >>>>>>>>>>>>>> Triad: 418122.7024 0.0078 0.0077 0.0081 >>>>>>>>>>>>>> Store: 224685.6729 0.0048 0.0048 0.0049 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 408GB/s >>>>>>>>>>>>>> >>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 04/13/2016 12:15 PM, Raman, Karthik wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2. Effective MCDRAM bandwidth is around 406-408 GB/s. >>>>>>>>>>>>>> Larger values are possible for more high core count bins. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> For bin3 , you should be able to get Triad ~440 GB/s. Please >>>>>>>>>>>>>> check if you use the following Intel compiler flags -mcmodel >>>>>>>>>>>>>> medium -shared-intel -O3 -xMIC-AVX512 >>>>>>>>>>>>>> -DSTREAM_ARRAY_SIZE=134217728 -DOFFSET=0 -DNTIMES=100 >>>>>>>>>>>>>> -openmp -opt-streaming-stores always The array size is >>>>>>>>>>>>>> just allocating 1GB per array. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>>>> >>>>>>>>>>>>>> Sent: Wednesday, April 13, 2016 10:01 AM >>>>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com >>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>> <mailto:[email protected]>; Chinthamani, Sundaram >>>>>>>>>>>>>> <sundaram.chinthamani(a)intel.com> >>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>> Kumaran, Kalyan <kumaran(a)alcf.anl.gov >>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>> Fromkin, Russ <russ.fromkin(a)intel.com >>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>> <mailto:[email protected]>; Harms, Kevin >>>>>>>>>>>>>> <harms(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>> Morozov, Vitali A. >>>>>>>>>>>>>> <morozov(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>> Allen, Benjamin S. >>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> <mailto:[email protected]> >>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>> >>>>>>>>>>>>>> so, this is my understanding that on bin3 hardware (64 >>>>>>>>>>>>>> cores, 6 >>>>>>>>>>>>>> channel DDR4-2133) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1. Effective DDR bandwidth is 80 GB/s (around 78% of peak). >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2. Effective MCDRAM bandwidth is around 406-408 GB/s. >>>>>>>>>>>>>> Larger values are possible for more high core count bins. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 3. Alignment is not important for STREAM triad benchmark. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 4. For both static and dynamic variables, numactl -m >>>>>>>>>>>>>> [0|1] should be used to direct memory placement. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 5. For explicit MCDRAM allocation in flat mode, >>>>>>>>>>>>>> hbw_malloc.h API should be use. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Please correct if I am wrong... >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks a lot for you help! >>>>>>>>>>>>>> >>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 04/12/2016 11:07 AM, Vitali A. Morozov wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Karthik, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I was also wandering how the variable could be >>>>>>>>>>>>>> influencing the allocation if, in principle, I do not >>>>>>>>>>>>>> need to link against memkind in my case. Anyway, sorry >>>>>>>>>>>>>> for confusion! >>>>>>>>>>>>>> >>>>>>>>>>>>>> I think I did that before - see my first message. I was >>>>>>>>>>>>>> using "numactl --membind=1 ./stream" , which I believe is >>>>>>>>>>>>>> equivalent to "numactl -m 1 ./stream" that you suggest. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Here is the result of "numactl -H": >>>>>>>>>>>>>> >>>>>>>>>>>>>> [morozov@knl00 512_3_C64_MPI16_TH08_flat]$ numactl -H >>>>>>>>>>>>>> available: 2 nodes (0-1) >>>>>>>>>>>>>> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 >>>>>>>>>>>>>> 18 19 20 >>>>>>>>>>>>>> 21 >>>>>>>>>>>>>> 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 >>>>>>>>>>>>>> 41 42 43 >>>>>>>>>>>>>> 44 >>>>>>>>>>>>>> 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 >>>>>>>>>>>>>> 64 65 66 >>>>>>>>>>>>>> 67 >>>>>>>>>>>>>> 68 69 70 >>>>>>>>>>>>>> 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 >>>>>>>>>>>>>> 90 91 92 >>>>>>>>>>>>>> 93 >>>>>>>>>>>>>> 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 >>>>>>>>>>>>>> 110 111 >>>>>>>>>>>>>> 112 >>>>>>>>>>>>>> 113 114 115 116 117 118 119 120 121 122 123 124 125 126 >>>>>>>>>>>>>> 127 128 >>>>>>>>>>>>>> 129 >>>>>>>>>>>>>> 130 131 132 133 134 135 136 137 138 139 140 141 142 >>>>>>>>>>>>>> 143 144 145 146 147 148 149 150 151 152 153 154 155 156 >>>>>>>>>>>>>> 157 158 >>>>>>>>>>>>>> 159 >>>>>>>>>>>>>> 160 161 162 163 164 165 166 167 168 169 170 171 172 173 >>>>>>>>>>>>>> 174 175 >>>>>>>>>>>>>> 176 >>>>>>>>>>>>>> 177 178 179 180 181 182 183 184 185 186 187 188 189 190 >>>>>>>>>>>>>> 191 192 >>>>>>>>>>>>>> 193 >>>>>>>>>>>>>> 194 195 196 197 198 199 200 201 202 203 204 205 206 >>>>>>>>>>>>>> 207 208 209 210 211 212 213 214 215 216 217 218 219 220 >>>>>>>>>>>>>> 221 222 >>>>>>>>>>>>>> 223 >>>>>>>>>>>>>> 224 225 226 227 228 229 230 231 232 233 234 235 236 237 >>>>>>>>>>>>>> 238 239 >>>>>>>>>>>>>> 240 >>>>>>>>>>>>>> 241 242 243 244 245 246 247 248 249 250 251 252 253 254 >>>>>>>>>>>>>> 255 node >>>>>>>>>>>>>> 0 >>>>>>>>>>>>>> size: 196511 MB node 0 free: 186888 MB node 1 cpus: >>>>>>>>>>>>>> node 1 size: 16384 MB >>>>>>>>>>>>>> node 1 free: 11731 MB >>>>>>>>>>>>>> node distances: >>>>>>>>>>>>>> node 0 1 >>>>>>>>>>>>>> 0: 10 31 >>>>>>>>>>>>>> 1: 31 10 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Sounds like all as expected. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [morozov@knl00 STREAM]$ OMP_NUM_THREADS=64 >>>>>>>>>>>>>> KMP_AFFINITY=scatter numactl -m 1 ./stream >>>>>>>>>>>>>> Triad: 418262.7498 0.0058 0.0057 0.0060 >>>>>>>>>>>>>> >>>>>>>>>>>>>> For a series of runs, I see this: >>>>>>>>>>>>>> >>>>>>>>>>>>>> # MB N GB/s >>>>>>>>>>>>>> 228.88 10000000 408867.98 >>>>>>>>>>>>>> 457.76 20000000 405818.57 >>>>>>>>>>>>>> 686.65 30000000 414309.08 >>>>>>>>>>>>>> 915.53 40000000 415063.59 >>>>>>>>>>>>>> 1144.41 50000000 415792.22 >>>>>>>>>>>>>> 1373.29 60000000 415449.01 >>>>>>>>>>>>>> 1602.17 70000000 414106.18 >>>>>>>>>>>>>> 1831.05 80000000 416846.82 >>>>>>>>>>>>>> 2059.94 90000000 414992.29 >>>>>>>>>>>>>> 2288.82 100000000 417898.11 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 04/12/2016 10:32 AM, Raman, Karthik wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Vitali, >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1. >>>>>>>>>>>>>> The MEMKIND_HBW_NODES is *only* when you are allocating using >>>>>>>>>>>>>> hbwalloc (+ MCDRAM flat) in your application and when >>>>>>>>>>>>>> memkind is >>>>>>>>>>>>>> unable to find the high bandwidth nodes (pmtt not populated) >>>>>>>>>>>>>> >>>>>>>>>>>>>> 2. >>>>>>>>>>>>>> With the below >>>>>>>>>>>>>> static allocations, in MCDRAM Flat mode, you can simply >>>>>>>>>>>>>> use ï¿½numactl ï¿½m 1 ./streamï¿½ with propery affinity >>>>>>>>>>>>>> settings to get peak out of MCDRAM. >>>>>>>>>>>>>> a. >>>>>>>>>>>>>> Check ï¿½numactl ï¿½Hï¿½ to find the NUMA domains in your node. If >>>>>>>>>>>>>> booted in All2All/FLAT or Quad/Flat mode it should show 2 >>>>>>>>>>>>>> NUMA >>>>>>>>>>>>>> domains , one for DDR (96 GB) and other for MCDRAM (16GB) >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>> Karthik >>>>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>>>> >>>>>>>>>>>>>> Sent: Tuesday, April 12, 2016 7:27 AM >>>>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com >>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>> <mailto:[email protected]>; Chinthamani, Sundaram >>>>>>>>>>>>>> <sundaram.chinthamani(a)intel.com> >>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>> Kumaran, Kalyan <kumaran(a)alcf.anl.gov >>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>> Fromkin, Russ <russ.fromkin(a)intel.com >>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>> <mailto:[email protected]>; Harms, Kevin >>>>>>>>>>>>>> <harms(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>> Morozov, Vitali A. >>>>>>>>>>>>>> <morozov(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>> Allen, Benjamin S. >>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> <mailto:[email protected]> >>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Karthik, >>>>>>>>>>>>>> >>>>>>>>>>>>>> The node is in All2All mode for now. I made the following >>>>>>>>>>>>>> changes. The original code is using static allocation >>>>>>>>>>>>>> therefore, I follow the advice on 2MB static array alignment: >>>>>>>>>>>>>> >>>>>>>>>>>>>> [morozov@thing00 STREAM]$ head -n 108 < stream.c | tail -n 4 >>>>>>>>>>>>>> static double a[N+OFFSET] >>>>>>>>>>>>>> __attribute__((aligned(2097152))), >>>>>>>>>>>>>> b[N+OFFSET] >>>>>>>>>>>>>> __attribute__((aligned(2097152))), >>>>>>>>>>>>>> c[N+OFFSET] >>>>>>>>>>>>>> __attribute__((aligned(2097152))), >>>>>>>>>>>>>> d[N+OFFSET] >>>>>>>>>>>>>> __attribute__((aligned(2097152))); >>>>>>>>>>>>>> >>>>>>>>>>>>>> [morozov@knl00 STREAM]$ make >>>>>>>>>>>>>> mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp >>>>>>>>>>>>>> -mcmodel >>>>>>>>>>>>>> medium -shared-intel -fno-alias stream.c -o stream >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [morozov@knl00 STREAM]$ MEMKIND_HBW_NODES=1 >>>>>>>>>>>>>> OMP_NUM_THREADS=64 >>>>>>>>>>>>>> KMP_AFFINITY=scatter ./stream >>>>>>>>>>>>>> >>>>>>>>>>>>>> ..... >>>>>>>>>>>>>> >>>>>>>>>>>>>> Triad: 80251.3621 0.0300 0.0299 0.0305 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Apparently, the "MEMKIND_HBW_NODES=1" is ignored. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>> >>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 04/12/2016 01:51 AM, Raman, Karthik wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Also could you please modify the memory allocations in >>>>>>>>>>>>>> STREAM to align at 2MB boundary. >>>>>>>>>>>>>> >>>>>>>>>>>>>> For example: >>>>>>>>>>>>>> static allocations: >>>>>>>>>>>>>> >>>>>>>>>>>>>> static STREAM_TYPE a[STREAM_ARRAY_SIZE+OFFSET] >>>>>>>>>>>>>> __attribute__((aligned(2097152))) >>>>>>>>>>>>>> >>>>>>>>>>>>>> dynamic allocations: >>>>>>>>>>>>>> a = (STREAM_TYPE >>>>>>>>>>>>>> *)_mm_malloc(sizeof(STREAM_TYPE)*(STREAM_ARRAY_SIZE+OFFSET),20971 >>>>>>>>>>>>>> 5 >>>>>>>>>>>>>> 2 >>>>>>>>>>>>>> ) >>>>>>>>>>>>>> ; >>>>>>>>>>>>>> >>>>>>>>>>>>>> memkind/hbwalloc: >>>>>>>>>>>>>> (STREAM_TYPE *)hbw_posix_memalign((void **)&a, 2097152, >>>>>>>>>>>>>> sizeof(STREAM_TYPE)*(STREAM_ARRAY_SIZE+OFFSET)) >>>>>>>>>>>>>> >>>>>>>>>>>>>> For memkind/hbwmalloc to work out of the box, you need to >>>>>>>>>>>>>> have >>>>>>>>>>>>>> the PMTT tables populated so that it can identify the >>>>>>>>>>>>>> high bandwidth nodes. Alternatively you can set the >>>>>>>>>>>>>> MEMKIND_HBW_NODES environment variable to the high >>>>>>>>>>>>>> bandwidth node (MCDRAM). >>>>>>>>>>>>>> >>>>>>>>>>>>>> e.g. All2All/Flat or Quad/Flat : export >>>>>>>>>>>>>> MEMKIND_HBW_NODES=1 ; ./app_with_hbwmalloc >>>>>>>>>>>>>> SNC4/Flat: export MEMKIND_HBW_NODES=4,5,6,7 ; >>>>>>>>>>>>>> ./app_with_hbw_malloc >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>> Karthik >>>>>>>>>>>>>> >>>>>>>>>>>>>> From: Chinthamani, Sundaram >>>>>>>>>>>>>> >>>>>>>>>>>>>> Sent: Monday, April 11, 2016 8:15 PM >>>>>>>>>>>>>> To: Vitali Morozov <morozov(a)anl.gov >>>>>>>>>>>>>> <mailto:[email protected]>> <mailto:[email protected]>; >>>>>>>>>>>>>> Kumaran, Kalyan <kumaran(a)alcf.anl.gov >>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>> Fromkin, Russ <russ.fromkin(a)intel.com >>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>> <mailto:[email protected]>; Raman, Karthik >>>>>>>>>>>>>> <karthik.raman(a)intel.com> >>>>>>>>>>>>>> <mailto:[email protected]>; Harms, Kevin >>>>>>>>>>>>>> <harms(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>> Morozov, Vitali A. >>>>>>>>>>>>>> <morozov(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>> Allen, Benjamin S. >>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>> Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com >>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>>>> Subject: RE: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> You should use numactl ï¿½m 4,5,6,7 to get the memory >>>>>>>>>>>>>> bandwidth out of MC-DRAM. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> What kind of part is it? Is it bin1 or bin3? Even in >>>>>>>>>>>>>> All2All mode, we have measured up to 470 GB/s of stream >>>>>>>>>>>>>> triad bandwidth. Quad mode should get 490 GB/s. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> DDR-stream triad bandwidth on Bin1-part should be close >>>>>>>>>>>>>> to 90 GB/s. >>>>>>>>>>>>>> >>>>>>>>>>>>>> -Sundaram >>>>>>>>>>>>>> >>>>>>>>>>>>>> From: Vitali Morozov [mailto:[email protected]] >>>>>>>>>>>>>> >>>>>>>>>>>>>> Sent: Monday, April 11, 2016 6:59 PM >>>>>>>>>>>>>> To: Kumaran, Kalyan <kumaran(a)alcf.anl.gov >>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>>>>> <mailto:[email protected]>>; Fromkin, Russ >>>>>>>>>>>>>> <russ.fromkin(a)intel.com>; Chinthamani, Sundaram >>>>>>>>>>>>>> <sundaram.chinthamani(a)intel.com>; Raman, Karthik >>>>>>>>>>>>>> <karthik.raman(a)intel.com>; Harms, Kevin <harms(a)alcf.anl.gov>; >>>>>>>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov >>>>>>>>>>>>>> <mailto:[email protected]>>; Allen, Benjamin S. >>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Avinash and others, >>>>>>>>>>>>>> >>>>>>>>>>>>>> here is what we have so far for Flat more, All2All. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The results from memory were obtained by using default >>>>>>>>>>>>>> STREAM with static allocation. The results from MCDRAM >>>>>>>>>>>>>> were obtained by using modified STREAM with hbwmalloc.h >>>>>>>>>>>>>> interface. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1 MPI process, 64 threads, scatter affinity, no numactl >>>>>>>>>>>>>> is used. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Most results are as expected: 80 GB/s from DDR memory, >>>>>>>>>>>>>> 420 GB/s from MCDRAM, the MCDRAM size is about 16 GB, the >>>>>>>>>>>>>> aggregate L2 cache size is 32 MB. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Noticed inconsistencies for hbwmalloc benchmarks: >>>>>>>>>>>>>> >>>>>>>>>>>>>> 1. The result is not validated - will spend more time on >>>>>>>>>>>>>> that tomorrow. >>>>>>>>>>>>>> 2. hbw_malloc continues allocating data beyond the >>>>>>>>>>>>>> expected 16 GB up to 48 GB. This means it fails back to >>>>>>>>>>>>>> DDR if it cannot allocate in MCDRAM - not expected >>>>>>>>>>>>>> default behavior. It must fail if it does not have enough >>>>>>>>>>>>>> space. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The problem shows up when I am running static allocation >>>>>>>>>>>>>> and use numactl --membind=1 to ask for MCDRAM-only >>>>>>>>>>>>>> allocation. The variation is very large and inconsistent >>>>>>>>>>>>>> with the result observed when using hbw_malloc API: >>>>>>>>>>>>>> >>>>>>>>>>>>>> # MB N GB/s numactl GB/s >>>>>>>>>>>>>> hbw_malloc >>>>>>>>>>>>>> 228.88 10000000 255555.46 421009.18 >>>>>>>>>>>>>> 457.76 20000000 363930.93 416997.91 >>>>>>>>>>>>>> 686.65 30000000 309415.87 427808.31 >>>>>>>>>>>>>> 915.53 40000000 371691.30 423310.75 >>>>>>>>>>>>>> 1144.41 50000000 187002.22 431032.35 >>>>>>>>>>>>>> 1373.29 60000000 385954.23 427172.91 >>>>>>>>>>>>>> 1602.17 70000000 388254.49 426823.59 >>>>>>>>>>>>>> 1831.05 80000000 374491.43 428103.96 >>>>>>>>>>>>>> 2059.94 90000000 379622.74 428152.02 >>>>>>>>>>>>>> 2288.82 100000000 80868.34 430185.03 >>>>>>>>>>>>>> >>>>>>>>>>>>>> We also tried SNC-4 instead of All2All, but could not get >>>>>>>>>>>>>> more than 40 GB/s from MCDRAM even with hbw_malloc API. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Raw data: >>>>>>>>>>>>>> >>>>>>>>>>>>>> # flat memory, running from memory (default), alltoall >>>>>>>>>>>>>> # MB N GB/s >>>>>>>>>>>>>> 2.29 100000 272062.96 >>>>>>>>>>>>>> 4.58 200000 491040.47 >>>>>>>>>>>>>> 6.87 300000 603979.78 >>>>>>>>>>>>>> 9.16 400000 745654.04 >>>>>>>>>>>>>> 11.44 500000 811800.77 >>>>>>>>>>>>>> 13.73 600000 901462.35 >>>>>>>>>>>>>> 16.02 700000 891953.26 >>>>>>>>>>>>>> 18.31 800000 1019375.15 >>>>>>>>>>>>>> 20.60 900000 984749.63 >>>>>>>>>>>>>> 22.89 1000000 923516.48 >>>>>>>>>>>>>> 25.18 1100000 758422.09 >>>>>>>>>>>>>> 27.47 1200000 779328.74 >>>>>>>>>>>>>> 29.75 1300000 513185.43 >>>>>>>>>>>>>> 32.04 1400000 155550.35 >>>>>>>>>>>>>> 34.33 1500000 103209.12 >>>>>>>>>>>>>> 36.62 1600000 97435.74 >>>>>>>>>>>>>> 38.91 1700000 83968.40 >>>>>>>>>>>>>> 41.20 1800000 83692.35 >>>>>>>>>>>>>> 43.49 1900000 80734.60 >>>>>>>>>>>>>> 45.78 2000000 76317.89 >>>>>>>>>>>>>> 68.66 3000000 81180.08 >>>>>>>>>>>>>> 91.55 4000000 78889.73 >>>>>>>>>>>>>> 114.44 5000000 79891.50 >>>>>>>>>>>>>> 137.33 6000000 79954.96 >>>>>>>>>>>>>> 160.22 7000000 79477.00 >>>>>>>>>>>>>> 183.11 8000000 79145.59 >>>>>>>>>>>>>> 205.99 9000000 79821.12 >>>>>>>>>>>>>> 228.88 10000000 79519.15 >>>>>>>>>>>>>> 228.88 10000000 79212.54 >>>>>>>>>>>>>> 457.76 20000000 79692.27 >>>>>>>>>>>>>> 686.65 30000000 79866.15 >>>>>>>>>>>>>> 915.53 40000000 79828.15 >>>>>>>>>>>>>> 1144.41 50000000 80214.91 >>>>>>>>>>>>>> 1373.29 60000000 80226.84 >>>>>>>>>>>>>> 1602.17 70000000 80178.77 >>>>>>>>>>>>>> 1831.05 80000000 80177.06 >>>>>>>>>>>>>> 2059.94 90000000 80109.79 >>>>>>>>>>>>>> 2288.82 100000000 80175.94 >>>>>>>>>>>>>> >>>>>>>>>>>>>> # flat memory, running from MCDRAM, alltoall, hbwmalloc API >>>>>>>>>>>>>> # MB N GB/s >>>>>>>>>>>>>> 228.88 10000000 421009.18 >>>>>>>>>>>>>> 457.76 20000000 416997.91 >>>>>>>>>>>>>> 686.65 30000000 427808.31 >>>>>>>>>>>>>> 915.53 40000000 423310.75 >>>>>>>>>>>>>> 1144.41 50000000 431032.35 >>>>>>>>>>>>>> 1373.29 60000000 427172.91 >>>>>>>>>>>>>> 1602.17 70000000 426823.59 >>>>>>>>>>>>>> 1831.05 80000000 428103.96 >>>>>>>>>>>>>> 2059.94 90000000 428152.02 >>>>>>>>>>>>>> 2288.82 100000000 430185.03 >>>>>>>>>>>>>> 2343.75 102400000 425999.98 >>>>>>>>>>>>>> 4687.50 204800000 427821.10 >>>>>>>>>>>>>> 9155.27 400000000 429318.13 >>>>>>>>>>>>>> 9375.00 409600000 428376.70 >>>>>>>>>>>>>> 11444.09 500000000 428052.08 >>>>>>>>>>>>>> 13732.91 600000000 266061.01 >>>>>>>>>>>>>> 16021.73 700000000 200257.79 >>>>>>>>>>>>>> 18310.55 800000000 168164.19 >>>>>>>>>>>>>> 20599.37 900000000 150192.91 >>>>>>>>>>>>>> 22888.18 1000000000 137862.07 >>>>>>>>>>>>>> 25177.00 1100000000 129418.12 >>>>>>>>>>>>>> 27465.82 1200000000 123019.53 >>>>>>>>>>>>>> 29754.64 1300000000 118268.79 >>>>>>>>>>>>>> 32043.46 1400000000 114343.33 >>>>>>>>>>>>>> 34332.28 1500000000 110991.25 >>>>>>>>>>>>>> 36621.09 1600000000 108538.36 >>>>>>>>>>>>>> 38909.91 1700000000 106249.72 >>>>>>>>>>>>>> 41198.73 1800000000 104455.07 >>>>>>>>>>>>>> 43487.55 1900000000 102828.93 >>>>>>>>>>>>>> 45776.37 2000000000 101348.56 >>>>>>>>>>>>>> 48065.19 2100000000 100013.73 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> We appreciate any help on that. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 04/11/2016 08:42 PM, Kumaran, Kalyan wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Avinash, thanks for the prompt reply. Kevin/Vitali can >>>>>>>>>>>>>> provide >>>>>>>>>>>>>> all the details. The short summary is we are unable to >>>>>>>>>>>>>> reproduce the STREAM results from IPM or DDR. It would be >>>>>>>>>>>>>> great if Karthik or Sundaram can advise us on how to run >>>>>>>>>>>>>> (which mode etc) or if they could log into the machine >>>>>>>>>>>>>> and make sure we have set up things correctly. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Kumar >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Apr 11, 2016, at 8:07 PM, Sodani, Avinash >>>>>>>>>>>>>> <avinash.sodani(a)intel.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Copying Sundaram/Karthik from our side. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Kumar, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Can you send us more information on this? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -Avinash >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> From: Kumaran, >>>>>>>>>>>>>> Kalyan [mailto:[email protected]] >>>>>>>>>>>>>> Sent: Monday, April 11, 2016 5:32 PM >>>>>>>>>>>>>> To: Fromkin, Russ; Sodani, Avinash >>>>>>>>>>>>>> Cc: Harms, Kevin; Morozov, Vitali A. >>>>>>>>>>>>>> Subject: Fwd: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Russ, Avinash, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> we need help in understanding how to obtain STREAM >>>>>>>>>>>>>> results. Can anyone help? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Kumar >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Begin forwarded message: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> From: "Harms, Kevin" <harms(a)alcf.anl.gov >>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Subject: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Date: April 11, 2016 at 5:42:15 PM CDT >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> To: "intel-nda(a)jlse.anl.gov" >>>>>>>>>>>>>> <intel-nda(a)jlse.anl.gov> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> The knl00 node is currently in Flat mode and All2All >>>>>>>>>>>>>> model. With this setup we can get 420 GB/s from IPM >>>>>>>>>>>>>> memory on STREAM Triad when using hbw_malloc and friends >>>>>>>>>>>>>> which was confirmed by both Vitali and I. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Changing the BIOS settings and warm rebooting seem to be >>>>>>>>>>>>>> problematic. Vitali and I were running earlier and could >>>>>>>>>>>>>> not get over 80 GB/s with any method even though memory >>>>>>>>>>>>>> capacity indicated flat mode. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Using numactl --membind is also problematic. The STREAM >>>>>>>>>>>>>> results >>>>>>>>>>>>>> when using this to launch the application were erratic, >>>>>>>>>>>>>> so it is also suggested not to use that. That means you >>>>>>>>>>>>>> will need to modify your code to use dynamic memory and >>>>>>>>>>>>>> hbw_malloc to exploit the IPM. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Will followup with more detailed results later. >>>>>>>>>>>>>> >>>>>>>>>>>>>> kevin >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> Intel-nda mailing list >>>>>>>>>>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>>>>>>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>> >>>>>>> >>>>>> <flat_a2a_ddr.txt><flat_a2a_ipm.txt><flat_quad_ddr.txt><flat_quad_ipm.txt><flat_snc4_ddr.txt><flat_snc4_ipm.txt> >>>>> >>> >>> >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> > > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] KNL Status
by Scott Parker 21 Apr '16

21 Apr '16

Thank Ben. It's up and running in SNC-4 mode now. I performed an number of reboots yesterday and they seemed to work well for the other modes I tried, so I guess there must be something special about SNC-4 mode. -Scott On 4/21/16 5:49 AM, Allen, Benjamin S. wrote: > Kevin and I ran into this as well. It appears warm reboot with snc4 mode still hangs. Power cycling the machine now. > > Ben > >> On Apr 20, 2016, at 8:19 PM, Scott Parker <sparker(a)anl.gov> wrote: >> >> >> I believe the node is hung while attempting a reboot into snc4 mode >> >> -Scott >> >> On 4/13/16 12:46 PM, Allen, Benjamin S. wrote: >>> Yea, it hangs on warm reboot. Power cycling it via ipmi now. >>> >>> Ben >>> >>>> On Apr 13, 2016, at 2:44 PM, Vitali A. Morozov <morozov(a)anl.gov> wrote: >>>> >>>> I was using >>>> >>>> sudo /usr/bin/systemctl reboot >>>> >>>> and it is booting for over 30 minutes now. Something is not right... >>>> >>>> Vitali >>>> >>>> >>>> >>>> >>>> On 04/13/2016 02:05 PM, Vitali A. Morozov wrote: >>>>> As no complains have been posted, rebooting to SNC-4/Flat. >>>>> >>>>> Vitali >>>>> >>>>> >>>>> On 04/13/2016 01:17 PM, Vitali A. Morozov wrote: >>>>>> Hello, >>>>>> >>>>>> With current setup of All2All/Flat, we seem to have a stable 410 GB/s from MCDRAM and 78 GB/s from DDR. >>>>>> >>>>>> For STREAMing from MCDRAM, Intel people suggested to try SNC-4/Flat promising 430 GB/s. Our previous trials with other combinations of NOC settings lead to unstable results. >>>>>> >>>>>> Do we want to try it now? It will probably require hard reboot. >>>>>> >>>>>> Vitali >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On 04/13/2016 01:06 PM, Raman, Karthik wrote: >>>>>>> Ok I read this as 418 GB/s J >>>>>>> And ok, just realized you are running in All2All mode, so yes this looks good. For Quad/Flat you should see ~440 GB/s with the same run. >>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>> Sent: Wednesday, April 13, 2016 11:02 AM >>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com>; Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov> >>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; Fromkin, Russ <russ.fromkin(a)intel.com>; Harms, Kevin <harms(a)alcf.anl.gov>; Morozov, Vitali A. <morozov(a)alcf.anl.gov>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>> Hi Karthik, >>>>>>> >>>>>>> Repeating the same options you have: >>>>>>> >>>>>>> mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp -mcmodel medium -shared-intel -opt-streaming-stores always stream.c -o stream >>>>>>> >>>>>>> ------------------------------------------------------------- >>>>>>> STREAM version $Revision: 5.9 $ >>>>>>> ------------------------------------------------------------- >>>>>>> This system uses 8 bytes per DOUBLE PRECISION word. >>>>>>> ------------------------------------------------------------- >>>>>>> Array size = 134217728, Offset = 0 >>>>>>> Total memory required = 3072.0 MB. >>>>>>> Each test is run 100 times, but only >>>>>>> the *best* time for each is used. >>>>>>> ------------------------------------------------------------- >>>>>>> Number of Threads requested = 64 >>>>>>> ------------------------------------------------------------- >>>>>>> >>>>>>> < skiped > >>>>>>> >>>>>>> Function Rate (MB/s) Avg time Min time Max time >>>>>>> Copy: 362331.5200 0.0060 0.0059 0.0061 >>>>>>> Scale: 359281.9806 0.0060 0.0060 0.0062 >>>>>>> Add: 418122.7024 0.0078 0.0077 0.0081 >>>>>>> Triad: 418122.7024 0.0078 0.0077 0.0081 >>>>>>> Store: 224685.6729 0.0048 0.0048 0.0049 >>>>>>> >>>>>>> >>>>>>> 408GB/s >>>>>>> >>>>>>> Vitali >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Intel-nda mailing list >>>>>> >>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>> _______________________________________________ >>>> Intel-nda mailing list >>>> Intel-nda(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] KNL Status
by Kumaran, Kalyan 21 Apr '16

21 Apr '16

Looks like Scott Parker gets better results: Adding another data point, I ran in SNC-4 mode with the configuration below and observed a consistent BW around 440 GB/s with the best run producing 442 GB/s. In this case 128 threads gave the best results. I've noticed that the STREAM results sensitivity to build and run parameters seems to vary depending on the memory configuration in flat-all2all I saw noticeably better results with memory aligned on 2MB boundaries and 64 threads, in SNC-4 mode the alignment doesn't seem to matter and 128 threads is producing the best results. Mode: flat & SNC-4 Compiler: Intel 17 Beta Compile Options: icc -O3 -fopenmp -DSTREAM_ARRAY_SIZE=134217728 -xMIC-AVX512 -mcmodel medium -shared-intel Run Command: OMP_NUM_THREADS=128 numactl -m 4,5,6,7 ./stream_mod Results: Function Best Rate MB/s Avg time Min time Max time Copy: 390378.3 0.005524 0.005501 0.005546 Scale: 412738.8 0.005237 0.005203 0.005278 Add: 448759.4 0.007199 0.007178 0.007218 Triad: 441919.3 0.007355 0.007289 0.007548 On Apr 20, 2016, at 7:39 PM, Vitali A Morozov <morozov(a)anl.gov<mailto:[email protected]>> wrote: Hi, a quick update on STREAM with the latest beta compiler [morozov@knl00 STREAM]$ icc -V Intel(R) C Intel(R) 64 Compiler for applications running on Intel(R) 64, Version 17.0.0.042 Beta Build 20160315 compiler options mpiicc -O3 -xMIC-AVX512 -qopt-report=5 -Wall -g -fopenmp -mcmodel medium -shared-intel -qopt-streaming-stores always -restrict stream.c -o stream Running in flat mode all2all morozov@knl00 STREAM]$ export OMP_NUM_THREADS=256 [morozov@knl00 STREAM]$ numactl -m 1 ./stream ------------------------------------------------------------- STREAM version $Revision: 5.9 $ ------------------------------------------------------------- This system uses 8 bytes per DOUBLE PRECISION word. ------------------------------------------------------------- Array size = 134217728, Offset = 0 Total memory required = 3072.0 MB. Each test is run 500 times, but only the *best* time for each is used. ------------------------------------------------------------- Number of Threads requested = 256 ------------------------------------------------------------- <skipped> ------------------------------------------------------------- Your clock granularity/precision appears to be 1 microseconds. Each test below will take on the order of 6356 microseconds. (= 6356 clock ticks) Increase the size of the arrays if this shows that you are not getting at least 20 clock ticks per test. ------------------------------------------------------------- WARNING -- The above is only a rough guideline. For best results, please be sure you know the precision of your system timer. ------------------------------------------------------------- Function Rate (MB/s) Avg time Min time Max time Copy: 330891.5637 0.0066 0.0065 0.0078 Scale: 330127.5200 0.0066 0.0065 0.0069 Add: 383622.4448 0.0085 0.0084 0.0093 Triad: 383252.4574 0.0085 0.0084 0.0098 ------------------------------------------------------------- Solution Validates ------------------------------------------------------------- Same with 64 threads scatter mode: morozov@knl00 STREAM]$ export OMP_NUM_THREADS=64 [morozov@knl00 STREAM]$ KMP_AFFINITY=scatter [morozov@knl00 STREAM]$ numactl -m 1 ./stream ------------------------------------------------------------- STREAM version $Revision: 5.9 $ ------------------------------------------------------------- This system uses 8 bytes per DOUBLE PRECISION word. ------------------------------------------------------------- Array size = 134217728, Offset = 0 Total memory required = 3072.0 MB. Each test is run 500 times, but only the *best* time for each is used. ------------------------------------------------------------- Number of Threads requested = 64 ------------------------------------------------------------- <skipped> ------------------------------------------------------------- Your clock granularity/precision appears to be 1 microseconds. Each test below will take on the order of 6056 microseconds. (= 6056 clock ticks) Increase the size of the arrays if this shows that you are not getting at least 20 clock ticks per test. ------------------------------------------------------------- WARNING -- The above is only a rough guideline. For best results, please be sure you know the precision of your system timer. ------------------------------------------------------------- Function Rate (MB/s) Avg time Min time Max time Copy: 353486.8826 0.0061 0.0061 0.0067 Scale: 353265.0608 0.0061 0.0061 0.0066 Add: 423961.3055 0.0077 0.0076 0.0079 Triad: 424907.9750 0.0076 0.0076 0.0084 ------------------------------------------------------------- Solution Validates ------------------------------------------------------------- Vitali On 4/18/16 12:43 PM, Kumaran, Kalyan wrote: FYI. Karthik will get on the machine to debug. Sent from my iPhone Begin forwarded message: From: "Chinthamani, Sundaram" <<mailto:[email protected]>sundaram.chinthamani(a)intel.com<mailto:[email protected]>> Date: April 18, 2016 at 2:05:26 PM CDT To: "Harms, Kevin" <harms(a)alcf.anl.gov<mailto:[email protected]>>, "Allen, Benjamin S." <bsallen(a)alcf.anl.gov<mailto:[email protected]>> Cc: "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov<mailto:[email protected]>>, "Kaana Balakrishna, Abhishek" <abhishek.kaana.balakrishna(a)intel.com<mailto:[email protected]>>, "Morozov, Vitali A." <morozov(a)alcf.anl.gov<mailto:[email protected]>>, "Raman, Karthik" <karthik.raman(a)intel.com<mailto:[email protected]>>, "Inkley, Benson" <benson.inkley(a)intel.com<mailto:[email protected]>>, "Sodani, Avinash" <avinash.sodani(a)intel.com<mailto:[email protected]>>, "Fromkin, Russ" <russ.fromkin(a)intel.com<mailto:[email protected]>>, "Chinthamani, Sundaram" <sundaram.chinthamani(a)intel.com<mailto:[email protected]>> Subject: RE: [JLSE-Intel-NDA] KNL Status I would expect Quad mode / SNC-4 mode to provide higher bandwidth compared to All2All mode. We haven't seen this in our internal measurements. -Thanks, -Sundaram -----Original Message----- From: Harms, Kevin [mailto:[email protected]] Sent: Monday, April 18, 2016 12:00 PM To: Allen, Benjamin S. <bsallen(a)alcf.anl.gov<mailto:[email protected]>> Cc: Kumaran, Kalyan <kumaran(a)alcf.anl.gov<mailto:[email protected]>>; Kaana Balakrishna, Abhishek <abhishek.kaana.balakrishna(a)intel.com<mailto:[email protected]>>; Morozov, Vitali A. <morozov(a)alcf.anl.gov<mailto:[email protected]>>; Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com<mailto:[email protected]>>; Raman, Karthik <karthik.raman(a)intel.com<mailto:[email protected]>>; Inkley, Benson <benson.inkley(a)intel.com<mailto:[email protected]>>; Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>>; Fromkin, Russ <russ.fromkin(a)intel.com<mailto:[email protected]>> Subject: Re: [JLSE-Intel-NDA] KNL Status I tested cached mode with all2all, snc4 and quad models. Cached all2all gave 275,000 MB/s. Does that look reasonable? SNC4 and quad mode gave 225,000 MB/s or so. kevin The following versions are now installed: BIOS 01.01.0104, ME 03.01.03.011.0, BMC 00.17.9699, FRUSDR 1.14. Ben On Apr 18, 2016, at 1:27 PM, Harms, Kevin <harms(a)alcf.anl.gov<mailto:[email protected]>> wrote: Ben completed the BIOS upgrade, but we do not see any difference in performance. I have attached results from runs done under three configurations: flat+all2all, flat+quad, flat+snc4. flat+all2all = 427046.0546 MB/s flat+quad = 327253.8882 MB/s flat+snc4 = 308555.9588 MB/s I compiled the CORAL stream.c with the following options: icc -mcmodel medium -shared-intel -O3 -xMIC-AVX512 -DN=50000000 -DOFFSET=0 -DNTIMES=100 -openmp -opt-streaming-stores always stream.c -o stream I added the alignment directives to the arrays. The attached files show the NUMA domains and numactl commands used and run parameters. Have I missed a parameter? kevin Suggest a time Friday afternoon. We are in central time zone. We can get you access to this B0 too. -Kumar On Apr 13, 2016, at 5:06 PM, Kaana Balakrishna, Abhishek <<mailto:[email protected]>abhishek.kaana.balakrishna(a)intel.com<mailto:[email protected]>> wrote: + Benson Cheers, Abhishek K Balakrishna (KB) Xeon Phi™ Hardware Platform Applications Engineer, Technical Computing Group (TCG) Intel Corporation Ph - 512 466 9886 www.intel.com/xeonphi<http://www.intel.com/xeonphi> -----Original Message----- From: Vitali A. Morozov [<mailto:[email protected]>mailto:[email protected]] Sent: Wednesday, April 13, 2016 4:58 PM To: Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com<mailto:[email protected]>>; Raman, Karthik <karthik.raman(a)intel.com<mailto:[email protected]>>; Harms, Kevin <harms(a)alcf.anl.gov<mailto:[email protected]>>; Morozov, Vitali A. <morozov(a)alcf.anl.gov<mailto:[email protected]>>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov<mailto:[email protected]>> Cc: Sodani, Avinash <<mailto:[email protected]>avinash.sodani(a)intel.com<mailto:[email protected]>>; Fromkin, Russ <russ.fromkin(a)intel.com<mailto:[email protected]>>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov<mailto:[email protected]>>; Kaana Balakrishna, Abhishek <<mailto:[email protected]>abhishek.kaana.balakrishna(a)intel.com<mailto:[email protected]>> Subject: Re: [JLSE-Intel-NDA] KNL Status Sure! Vitali On 04/13/2016 04:54 PM, Chinthamani, Sundaram wrote: Would it be possible to have a meeting to go over your questions rather than exchanging E-mails back and forth? + KB from Intel side. He can help set-up the meeting between Labs and Intel. -Thanks, -Sundaram -----Original Message----- From: Vitali A. Morozov [mailto:[email protected]] Sent: Wednesday, April 13, 2016 2:50 PM To: Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com<mailto:[email protected]>>; Raman, Karthik <<mailto:[email protected]>karthik.raman(a)intel.com<mailto:[email protected]>>; Harms, Kevin <harms(a)alcf.anl.gov<mailto:[email protected]>>; Morozov, Vitali A. <<mailto:[email protected]>morozov(a)alcf.anl.gov<mailto:[email protected]>>; Kumaran, Kalyan <<mailto:[email protected]>kumaran(a)alcf.anl.gov<mailto:[email protected]>> Cc: Sodani, Avinash <<mailto:[email protected]>avinash.sodani(a)intel.com<mailto:[email protected]>>; Fromkin, Russ <<mailto:[email protected]>russ.fromkin(a)intel.com<mailto:[email protected]>>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov<mailto:[email protected]>> Subject: Re: [JLSE-Intel-NDA] KNL Status On 04/13/2016 04:42 PM, Chinthamani, Sundaram wrote: node 4 free: 2907 MB node 5 free: 2926 MB node 6 free: 2922 MB node 7 free: 2914 MB There is 16 GB of MC-DRAM and 4 GB per NUMA node. Is there any other process running on the machine? Why does "free memory" show only 3 GB per node? This is EXACTLY why I am asking this question. Nothing is running on the node - and I do not see full 4 GiB of available MCDRAM memory. This might be reason why I do not see what is expected. I understand that I can reduce the array size, but I am trying to reproduce your results. I am still unclear on if you run "no mpi" version, how come that SNC-4 mode is giving you advantage? All arrays are allocated in the same quadrant. Requests from the other quadrants will go to this one and create congestion. Vitali If you want to test Stream triad bandwidth to MC-DRAM in Quad/Flat mode, you can use even smaller arrays of few 100 MBs. -Thanks, -Sundaram -----Original Message----- From: Vitali A. Morozov [mailto:[email protected]] Sent: Wednesday, April 13, 2016 2:32 PM To: Raman, Karthik <karthik.raman(a)intel.com<mailto:[email protected]>>; Chinthamani, Sundaram <<mailto:[email protected]>sundaram.chinthamani(a)intel.com<mailto:[email protected]>>; Harms, Kevin <harms(a)alcf.anl.gov<mailto:[email protected]>>; Morozov, Vitali A. <morozov(a)alcf.anl.gov<mailto:[email protected]>>; Kumaran, Kalyan <<mailto:[email protected]>kumaran(a)alcf.anl.gov<mailto:[email protected]>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>>; Fromkin, Russ <<mailto:[email protected]>russ.fromkin(a)intel.com<mailto:[email protected]>>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov<mailto:[email protected]>> Subject: Re: [JLSE-Intel-NDA] KNL Status ok, that is fine. Now the next issue I have is that for static arrays, all three arrays must land up on one of the numa nodes - the node that is attached to a master thread. You advice to use N = 134217728, that gives exactly 3 GiB of memory footprint for 3 arrays of triad. According to "numactl -H", I do not have 3 GiB of available memory on any of the MCDRAM numa nodes: numactl -H |grep free node 0 free: 46336 MB node 1 free: 46923 MB node 2 free: 46970 MB node 3 free: 46961 MB node 4 free: 2907 MB node 5 free: 2926 MB node 6 free: 2922 MB node 7 free: 2914 MB Therefore, at least one array will not be placed to the same numa node. Is this ok? On 04/13/2016 04:17 PM, Raman, Karthik wrote: Yes we run pure OMP code too. The problem with KMP_AFFINITY with SNC4 mode (currently) is that it may not equally balance the threads per NUMA domain even with scatter or balanced, you can check that by mapping the "verbose" output (e.g. KMP_AFFINITY=scatter,verbose) with "numactl -H" output to see if the OS thread to logical CPU mapping is balanced across all NUMA domains With All2All/Quad this should not be a problem -----Original Message----- From: Vitali A. Morozov [mailto:[email protected]] Sent: Wednesday, April 13, 2016 1:53 PM To: Raman, Karthik <karthik.raman(a)intel.com<mailto:[email protected]>>; Chinthamani, Sundaram <<mailto:[email protected]>sundaram.chinthamani(a)intel.com<mailto:[email protected]>>; Harms, Kevin <harms(a)alcf.anl.gov<mailto:[email protected]>>; Morozov, Vitali A. <morozov(a)alcf.anl.gov<mailto:[email protected]>>; Kumaran, Kalyan <<mailto:[email protected]>kumaran(a)alcf.anl.gov<mailto:[email protected]>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>>; Fromkin, Russ <<mailto:[email protected]>russ.fromkin(a)intel.com<mailto:[email protected]>>; Allen, Benjamin S. <<mailto:[email protected]>bsallen(a)alcf.anl.gov<mailto:[email protected]>> Subject: Re: [JLSE-Intel-NDA] KNL Status Before doing that - let us check if we are running similar benchmark. I am running 1 MPI process (in fact no mpi at all) and 64 OpenMP threads version. The threads can be placed with "balanced" or "scattered" policy. I am using "scattered". Is this ok so far? Vitali On 04/13/2016 03:48 PM, Raman, Karthik wrote: This still does not look right, and the reason could be that the threads are not balanced across all the NUMA domains (ideally you want 16 threads per quadrant) Before going to SNC4, can you please boot in "Quad" mode. If you look in the BIOS Cluster mode options there is another mode called "Quad" which is different from SNC4, only 2 NUMA domains (DDR, MCDRAM). Please run similar to All2All mode -----Original Message----- From: Vitali A. Morozov [mailto:[email protected]] Sent: Wednesday, April 13, 2016 12:56 PM To: Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com<mailto:[email protected]>>; Harms, Kevin <<mailto:[email protected]>harms(a)alcf.anl.gov<mailto:[email protected]>>; Raman, Karthik <<mailto:[email protected]>karthik.raman(a)intel.com<mailto:[email protected]>>; Morozov, Vitali A. <<mailto:[email protected]>morozov(a)alcf.anl.gov<mailto:[email protected]>>; Kumaran, Kalyan <<mailto:[email protected]>kumaran(a)alcf.anl.gov<mailto:[email protected]>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>>; Fromkin, Russ <<mailto:[email protected]>russ.fromkin(a)intel.com<mailto:[email protected]>>; Allen, Benjamin S. <<mailto:[email protected]>bsallen(a)alcf.anl.gov<mailto:[email protected]>> Subject: Re: [JLSE-Intel-NDA] KNL Status ok, I am now in SNC-4/Flat. Same binary. OMP_NUM_THREADS=64 KMP_AFFINITY=scatter numactl -m 4,5,6,7 ./stream |tee running.dat ------------------------------------------------------------- STREAM version $Revision: 5.9 $ ------------------------------------------------------------- This system uses 8 bytes per DOUBLE PRECISION word. ------------------------------------------------------------- Array size = 134217728, Offset = 0 Total memory required = 3072.0 MB. Each test is run 100 times, but only the *best* time for each is used. ------------------------------------------------------------- Number of Threads requested = 64 ------------------------------------------------------------- <skipped> ------------------------------------------------------------- Function Rate (MB/s) Avg time Min time Max time Copy: 320997.8352 0.0068 0.0067 0.0071 Scale: 319994.2893 0.0068 0.0067 0.0073 Add: 306055.0205 0.0106 0.0105 0.0107 Triad: 306055.0205 0.0106 0.0105 0.0109 Store: 247056.8669 0.0044 0.0043 0.0051 ------------------------------------------------------------- Solution Validates ------------------------------------------------------------- 306055 MB/s Vitali On 04/13/2016 02:01 PM, Chinthamani, Sundaram wrote: 440 GB/s (base 10). To be precise, 440 bytes / ns (nano-second). We always report memory bandwidth in GB/s (base 10) within Intel. -Thanks, -Sundaram -----Original Message----- From: Harms, Kevin [mailto:[email protected]] Sent: Wednesday, April 13, 2016 11:52 AM To: Raman, Karthik <karthik.raman(a)intel.com<mailto:[email protected]>>; Chinthamani, Sundaram <<mailto:[email protected]>sundaram.chinthamani(a)intel.com<mailto:[email protected]>>; Morozov, Vitali A. <<mailto:[email protected]>morozov(a)alcf.anl.gov<mailto:[email protected]>>; Kumaran, Kalyan <<mailto:[email protected]>kumaran(a)alcf.anl.gov<mailto:[email protected]>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>>; Fromkin, Russ <<mailto:[email protected]>russ.fromkin(a)intel.com<mailto:[email protected]>>; Allen, Benjamin S. <<mailto:[email protected]>bsallen(a)alcf.anl.gov<mailto:[email protected]>> Subject: Re: [JLSE-Intel-NDA] KNL Status Just to be clear, memory is always assumed to be measured using base 2. It's only the storage field that uses base 10. So we assume if you say 440 GB/s you mean GiB, or 450,560 MB/s. kevin And just some fun with units <http://sites.utexas.edu/jdm4372/2013/01/05/counting-binary-vs-dec>http://sites.utexas.edu/jdm4372/2013/01/05/counting-binary-vs-dec i m a l -powers-in-the-stream-benchmark/ From: Chinthamani, Sundaram Sent: Wednesday, April 13, 2016 11:20 AM To: Vitali A. Morozov <morozov(a)anl.gov<mailto:[email protected]>>; Raman, Karthik <<mailto:[email protected]>karthik.raman(a)intel.com<mailto:[email protected]>>; Kumaran, Kalyan <<mailto:[email protected]>kumaran(a)alcf.anl.gov<mailto:[email protected]>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>>; Fromkin, Russ <<mailto:[email protected]>russ.fromkin(a)intel.com<mailto:[email protected]>>; Harms, Kevin <<mailto:[email protected]>harms(a)alcf.anl.gov<mailto:[email protected]>>; Morozov, Vitali A. <morozov(a)alcf.anl.gov<mailto:[email protected]>>; Allen, Benjamin S. <<mailto:[email protected]>bsallen(a)alcf.anl.gov<mailto:[email protected]>>; Chinthamani, Sundaram <<mailto:[email protected]>sundaram.chinthamani(a)intel.com<mailto:[email protected]>> Subject: RE: [JLSE-Intel-NDA] KNL Status 440 GB/s stream triad for MC-DRAM and 80 GB/s is our internal bin3 measurement. You should be able to get closer these numbers, if you run the benchmark in Quad/Flat or SNC-4/Flat mode. -Thanks, -Sundaram From: Vitali A. Morozov [mailto:[email protected]] Sent: Wednesday, April 13, 2016 11:11 AM To: Raman, Karthik <karthik.raman(a)intel.com<mailto:[email protected]>>; Chinthamani, Sundaram <<mailto:[email protected]>sundaram.chinthamani(a)intel.com<mailto:[email protected]>>; Kumaran, Kalyan <<mailto:[email protected]>kumaran(a)alcf.anl.gov<mailto:[email protected]>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>>; Fromkin, Russ <<mailto:[email protected]>russ.fromkin(a)intel.com<mailto:[email protected]>>; Harms, Kevin <<mailto:[email protected]>harms(a)alcf.anl.gov<mailto:[email protected]>>; Morozov, Vitali A. <morozov(a)alcf.anl.gov<mailto:[email protected]>>; Allen, Benjamin S. <<mailto:[email protected]>bsallen(a)alcf.anl.gov<mailto:[email protected]>> Subject: Re: [JLSE-Intel-NDA] KNL Status I see - I am treating 418122 MB/s as 418122/1024 GB/s (as this is memory unit). Anyway - great we are on the same page here. I am in All2All/Flat mode. You seem to suggest that I must be on SNC-4/Flat to see 440000 MB/s? Thanks! Vitali On 04/13/2016 01:06 PM, Raman, Karthik wrote: Ok I read this as 418 GB/s J And ok, just realized you are running in All2All mode, so yes this looks good. For Quad/Flat you should see ~440 GB/s with the same run. From: Vitali A. Morozov [mailto:[email protected]] Sent: Wednesday, April 13, 2016 11:02 AM To: Raman, Karthik <karthik.raman(a)intel.com<mailto:[email protected]>> <<mailto:[email protected]>mailto:[email protected]>; Chinthamani, Sundaram <<mailto:[email protected]>sundaram.chinthamani(a)intel.com<mailto:[email protected]>> <<mailto:[email protected]>mailto:[email protected]>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov<mailto:[email protected]>> <<mailto:[email protected]>mailto:[email protected]> Cc: Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>> <<mailto:[email protected]>mailto:[email protected]>; Fromkin, Russ <russ.fromkin(a)intel.com<mailto:[email protected]>> <<mailto:[email protected]>mailto:[email protected]>; Harms, Kevin <<mailto:[email protected]>harms(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]>; Morozov, Vitali A. <<mailto:[email protected]>morozov(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]>; Allen, Benjamin S. <<mailto:[email protected]>bsallen(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] KNL Status Hi Karthik, Repeating the same options you have: mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp -mcmodel medium -shared-intel -opt-streaming-stores always stream.c -o stream ------------------------------------------------------------- STREAM version $Revision: 5.9 $ ------------------------------------------------------------- This system uses 8 bytes per DOUBLE PRECISION word. ------------------------------------------------------------- Array size = 134217728, Offset = 0 Total memory required = 3072.0 MB. Each test is run 100 times, but only the *best* time for each is used. ------------------------------------------------------------- Number of Threads requested = 64 ------------------------------------------------------------- < skiped > Function Rate (MB/s) Avg time Min time Max time Copy: 362331.5200 0.0060 0.0059 0.0061 Scale: 359281.9806 0.0060 0.0060 0.0062 Add: 418122.7024 0.0078 0.0077 0.0081 Triad: 418122.7024 0.0078 0.0077 0.0081 Store: 224685.6729 0.0048 0.0048 0.0049 408GB/s Vitali On 04/13/2016 12:15 PM, Raman, Karthik wrote: 2. Effective MCDRAM bandwidth is around 406-408 GB/s. Larger values are possible for more high core count bins. For bin3 , you should be able to get Triad ~440 GB/s. Please check if you use the following Intel compiler flags -mcmodel medium -shared-intel -O3 -xMIC-AVX512 -DSTREAM_ARRAY_SIZE=134217728 -DOFFSET=0 -DNTIMES=100 -openmp -opt-streaming-stores always The array size is just allocating 1GB per array. From: Vitali A. Morozov [mailto:[email protected]] Sent: Wednesday, April 13, 2016 10:01 AM To: Raman, Karthik <karthik.raman(a)intel.com<mailto:[email protected]>> <<mailto:[email protected]>mailto:[email protected]>; Chinthamani, Sundaram <<mailto:[email protected]>sundaram.chinthamani(a)intel.com<mailto:[email protected]>> <<mailto:[email protected]>mailto:[email protected]>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov<mailto:[email protected]>> <<mailto:[email protected]>mailto:[email protected]> Cc: Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>> <<mailto:[email protected]>mailto:[email protected]>; Fromkin, Russ <russ.fromkin(a)intel.com<mailto:[email protected]>> <<mailto:[email protected]>mailto:[email protected]>; Harms, Kevin <<mailto:[email protected]>harms(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]>; Morozov, Vitali A. <<mailto:[email protected]>morozov(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]>; Allen, Benjamin S. <<mailto:[email protected]>bsallen(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] KNL Status Hello, so, this is my understanding that on bin3 hardware (64 cores, 6 channel DDR4-2133) 1. Effective DDR bandwidth is 80 GB/s (around 78% of peak). 2. Effective MCDRAM bandwidth is around 406-408 GB/s. Larger values are possible for more high core count bins. 3. Alignment is not important for STREAM triad benchmark. 4. For both static and dynamic variables, numactl -m [0|1] should be used to direct memory placement. 5. For explicit MCDRAM allocation in flat mode, hbw_malloc.h API should be use. Please correct if I am wrong... Thanks a lot for you help! Vitali On 04/12/2016 11:07 AM, Vitali A. Morozov wrote: Hi Karthik, I was also wandering how the variable could be influencing the allocation if, in principle, I do not need to link against memkind in my case. Anyway, sorry for confusion! I think I did that before - see my first message. I was using "numactl --membind=1 ./stream" , which I believe is equivalent to "numactl -m 1 ./stream" that you suggest. Here is the result of "numactl -H": [morozov@knl00 512_3_C64_MPI16_TH08_flat]$ numactl -H available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 node 0 size: 196511 MB node 0 free: 186888 MB node 1 cpus: node 1 size: 16384 MB node 1 free: 11731 MB node distances: node 0 1 0: 10 31 1: 31 10 Sounds like all as expected. [morozov@knl00 STREAM]$ OMP_NUM_THREADS=64 KMP_AFFINITY=scatter numactl -m 1 ./stream Triad: 418262.7498 0.0058 0.0057 0.0060 For a series of runs, I see this: # MB N GB/s 228.88 10000000 408867.98 457.76 20000000 405818.57 686.65 30000000 414309.08 915.53 40000000 415063.59 1144.41 50000000 415792.22 1373.29 60000000 415449.01 1602.17 70000000 414106.18 1831.05 80000000 416846.82 2059.94 90000000 414992.29 2288.82 100000000 417898.11 Vitali On 04/12/2016 10:32 AM, Raman, Karthik wrote: Hi Vitali, 1. The MEMKIND_HBW_NODES is *only* when you are allocating using hbwalloc (+ MCDRAM flat) in your application and when memkind is unable to find the high bandwidth nodes (pmtt not populated) 2. With the below static allocations, in MCDRAM Flat mode, you can simply use “numactl –m 1 ./stream” with propery affinity settings to get peak out of MCDRAM. a. Check “numactl –H” to find the NUMA domains in your node. If booted in All2All/FLAT or Quad/Flat mode it should show 2 NUMA domains , one for DDR (96 GB) and other for MCDRAM (16GB) Thanks Karthik From: Vitali A. Morozov [mailto:[email protected]] Sent: Tuesday, April 12, 2016 7:27 AM To: Raman, Karthik <karthik.raman(a)intel.com<mailto:[email protected]>> <<mailto:[email protected]>mailto:[email protected]>; Chinthamani, Sundaram <<mailto:[email protected]>sundaram.chinthamani(a)intel.com<mailto:[email protected]>> <<mailto:[email protected]>mailto:[email protected]>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov<mailto:[email protected]>> <<mailto:[email protected]>mailto:[email protected]> Cc: Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>> <<mailto:[email protected]>mailto:[email protected]>; Fromkin, Russ <russ.fromkin(a)intel.com<mailto:[email protected]>> <<mailto:[email protected]>mailto:[email protected]>; Harms, Kevin <<mailto:[email protected]>harms(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]>; Morozov, Vitali A. <<mailto:[email protected]>morozov(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]>; Allen, Benjamin S. <<mailto:[email protected]>bsallen(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] KNL Status Hi Karthik, The node is in All2All mode for now. I made the following changes. The original code is using static allocation therefore, I follow the advice on 2MB static array alignment: [morozov@thing00 STREAM]$ head -n 108 < stream.c | tail -n 4 static double a[N+OFFSET] __attribute__((aligned(2097152))), b[N+OFFSET] __attribute__((aligned(2097152))), c[N+OFFSET] __attribute__((aligned(2097152))), d[N+OFFSET] __attribute__((aligned(2097152))); [morozov@knl00 STREAM]$ make mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp -mcmodel medium -shared-intel -fno-alias stream.c -o stream [morozov@knl00 STREAM]$ MEMKIND_HBW_NODES=1 OMP_NUM_THREADS=64 KMP_AFFINITY=scatter ./stream ..... Triad: 80251.3621 0.0300 0.0299 0.0305 Apparently, the "MEMKIND_HBW_NODES=1" is ignored. Thanks! Vitali On 04/12/2016 01:51 AM, Raman, Karthik wrote: Also could you please modify the memory allocations in STREAM to align at 2MB boundary. For example: static allocations: static STREAM_TYPE a[STREAM_ARRAY_SIZE+OFFSET] __attribute__((aligned(2097152))) dynamic allocations: a = (STREAM_TYPE *)_mm_malloc(sizeof(STREAM_TYPE)*(STREAM_ARRAY_SIZE+OFFSET),20971 5 2 ) ; memkind/hbwalloc: (STREAM_TYPE *)hbw_posix_memalign((void **)&a, 2097152, sizeof(STREAM_TYPE)*(STREAM_ARRAY_SIZE+OFFSET)) For memkind/hbwmalloc to work out of the box, you need to have the PMTT tables populated so that it can identify the high bandwidth nodes. Alternatively you can set the MEMKIND_HBW_NODES environment variable to the high bandwidth node (MCDRAM). e.g. All2All/Flat or Quad/Flat : export MEMKIND_HBW_NODES=1 ; ./app_with_hbwmalloc SNC4/Flat: export MEMKIND_HBW_NODES=4,5,6,7 ; ./app_with_hbw_malloc Thanks Karthik From: Chinthamani, Sundaram Sent: Monday, April 11, 2016 8:15 PM To: Vitali Morozov <morozov(a)anl.gov<mailto:[email protected]>> <mailto:[email protected]>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov<mailto:[email protected]>> <<mailto:[email protected]>mailto:[email protected]> Cc: Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>> <<mailto:[email protected]>mailto:[email protected]>; Fromkin, Russ <russ.fromkin(a)intel.com<mailto:[email protected]>> <<mailto:[email protected]>mailto:[email protected]>; Raman, Karthik <<mailto:[email protected]>karthik.raman(a)intel.com<mailto:[email protected]>> <mailto:[email protected]>; Harms, Kevin <<mailto:[email protected]>harms(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]>; Morozov, Vitali A. <<mailto:[email protected]>morozov(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]>; Allen, Benjamin S. <<mailto:[email protected]>bsallen(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]>; Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com<mailto:[email protected]>> <<mailto:[email protected]>mailto:[email protected]> Subject: RE: [JLSE-Intel-NDA] KNL Status You should use numactl –m 4,5,6,7 to get the memory bandwidth out of MC-DRAM. What kind of part is it? Is it bin1 or bin3? Even in All2All mode, we have measured up to 470 GB/s of stream triad bandwidth. Quad mode should get 490 GB/s. DDR-stream triad bandwidth on Bin1-part should be close to 90 GB/s. -Sundaram From: Vitali Morozov [mailto:[email protected]] Sent: Monday, April 11, 2016 6:59 PM To: Kumaran, Kalyan <kumaran(a)alcf.anl.gov<mailto:[email protected]>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>>; Fromkin, Russ <<mailto:[email protected]>russ.fromkin(a)intel.com<mailto:[email protected]>>; Chinthamani, Sundaram <<mailto:[email protected]>sundaram.chinthamani(a)intel.com<mailto:[email protected]>>; Raman, Karthik <<mailto:[email protected]>karthik.raman(a)intel.com<mailto:[email protected]>>; Harms, Kevin <<mailto:[email protected]>harms(a)alcf.anl.gov<mailto:[email protected]>>; Morozov, Vitali A. <morozov(a)alcf.anl.gov<mailto:[email protected]>>; Allen, Benjamin S. <<mailto:[email protected]>bsallen(a)alcf.anl.gov<mailto:[email protected]>> Subject: Re: [JLSE-Intel-NDA] KNL Status Avinash and others, here is what we have so far for Flat more, All2All. The results from memory were obtained by using default STREAM with static allocation. The results from MCDRAM were obtained by using modified STREAM with hbwmalloc.h interface. 1 MPI process, 64 threads, scatter affinity, no numactl is used. Most results are as expected: 80 GB/s from DDR memory, 420 GB/s from MCDRAM, the MCDRAM size is about 16 GB, the aggregate L2 cache size is 32 MB. Noticed inconsistencies for hbwmalloc benchmarks: 1. The result is not validated - will spend more time on that tomorrow. 2. hbw_malloc continues allocating data beyond the expected 16 GB up to 48 GB. This means it fails back to DDR if it cannot allocate in MCDRAM - not expected default behavior. It must fail if it does not have enough space. The problem shows up when I am running static allocation and use numactl --membind=1 to ask for MCDRAM-only allocation. The variation is very large and inconsistent with the result observed when using hbw_malloc API: # MB N GB/s numactl GB/s hbw_malloc 228.88 10000000 255555.46 421009.18 457.76 20000000 363930.93 416997.91 686.65 30000000 309415.87 427808.31 915.53 40000000 371691.30 423310.75 1144.41 50000000 187002.22 431032.35 1373.29 60000000 385954.23 427172.91 1602.17 70000000 388254.49 426823.59 1831.05 80000000 374491.43 428103.96 2059.94 90000000 379622.74 428152.02 2288.82 100000000 80868.34 430185.03 We also tried SNC-4 instead of All2All, but could not get more than 40 GB/s from MCDRAM even with hbw_malloc API. Raw data: # flat memory, running from memory (default), alltoall # MB N GB/s 2.29 100000 272062.96 4.58 200000 491040.47 6.87 300000 603979.78 9.16 400000 745654.04 11.44 500000 811800.77 13.73 600000 901462.35 16.02 700000 891953.26 18.31 800000 1019375.15 20.60 900000 984749.63 22.89 1000000 923516.48 25.18 1100000 758422.09 27.47 1200000 779328.74 29.75 1300000 513185.43 32.04 1400000 155550.35 34.33 1500000 103209.12 36.62 1600000 97435.74 38.91 1700000 83968.40 41.20 1800000 83692.35 43.49 1900000 80734.60 45.78 2000000 76317.89 68.66 3000000 81180.08 91.55 4000000 78889.73 114.44 5000000 79891.50 137.33 6000000 79954.96 160.22 7000000 79477.00 183.11 8000000 79145.59 205.99 9000000 79821.12 228.88 10000000 79519.15 228.88 10000000 79212.54 457.76 20000000 79692.27 686.65 30000000 79866.15 915.53 40000000 79828.15 1144.41 50000000 80214.91 1373.29 60000000 80226.84 1602.17 70000000 80178.77 1831.05 80000000 80177.06 2059.94 90000000 80109.79 2288.82 100000000 80175.94 # flat memory, running from MCDRAM, alltoall, hbwmalloc API # MB N GB/s 228.88 10000000 421009.18 457.76 20000000 416997.91 686.65 30000000 427808.31 915.53 40000000 423310.75 1144.41 50000000 431032.35 1373.29 60000000 427172.91 1602.17 70000000 426823.59 1831.05 80000000 428103.96 2059.94 90000000 428152.02 2288.82 100000000 430185.03 2343.75 102400000 425999.98 4687.50 204800000 427821.10 9155.27 400000000 429318.13 9375.00 409600000 428376.70 11444.09 500000000 428052.08 13732.91 600000000 266061.01 16021.73 700000000 200257.79 18310.55 800000000 168164.19 20599.37 900000000 150192.91 22888.18 1000000000 137862.07 25177.00 1100000000 129418.12 27465.82 1200000000 123019.53 29754.64 1300000000 118268.79 32043.46 1400000000 114343.33 34332.28 1500000000 110991.25 36621.09 1600000000 108538.36 38909.91 1700000000 106249.72 41198.73 1800000000 104455.07 43487.55 1900000000 102828.93 45776.37 2000000000 101348.56 48065.19 2100000000 100013.73 We appreciate any help on that. Best, Vitali On 04/11/2016 08:42 PM, Kumaran, Kalyan wrote: Avinash, thanks for the prompt reply. Kevin/Vitali can provide all the details. The short summary is we are unable to reproduce the STREAM results from IPM or DDR. It would be great if Karthik or Sundaram can advise us on how to run (which mode etc) or if they could log into the machine and make sure we have set up things correctly. Thanks, Kumar On Apr 11, 2016, at 8:07 PM, Sodani, Avinash <<mailto:[email protected]>avinash.sodani(a)intel.com<mailto:[email protected]>> wrote: Copying Sundaram/Karthik from our side. Kumar, Can you send us more information on this? -Avinash From: Kumaran, Kalyan [<mailto:[email protected]>mailto:[email protected]] Sent: Monday, April 11, 2016 5:32 PM To: Fromkin, Russ; Sodani, Avinash Cc: Harms, Kevin; Morozov, Vitali A. Subject: Fwd: [JLSE-Intel-NDA] KNL Status Russ, Avinash, we need help in understanding how to obtain STREAM results. Can anyone help? Thanks, Kumar Begin forwarded message: From: "Harms, Kevin" <harms(a)alcf.anl.gov<mailto:[email protected]>> Subject: [JLSE-Intel-NDA] KNL Status Date: April 11, 2016 at 5:42:15 PM CDT To: "<mailto:[email protected]>intel-nda(a)jlse.anl.gov<mailto:[email protected]>" <<mailto:[email protected]>intel-nda(a)jlse.anl.gov<mailto:[email protected]>> The knl00 node is currently in Flat mode and All2All model. With this setup we can get 420 GB/s from IPM memory on STREAM Triad when using hbw_malloc and friends which was confirmed by both Vitali and I. Changing the BIOS settings and warm rebooting seem to be problematic. Vitali and I were running earlier and could not get over 80 GB/s with any method even though memory capacity indicated flat mode. Using numactl --membind is also problematic. The STREAM results when using this to launch the application were erratic, so it is also suggested not to use that. That means you will need to modify your code to use dynamic memory and hbw_malloc to exploit the IPM. Will followup with more detailed results later. kevin _______________________________________________ Intel-nda mailing list <mailto:[email protected]>Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> <https://lists.jlse.anl.gov/mailman/listinfo/intel-nda>https://lists.jlse.anl.gov/mailman/listinfo/intel-nda <flat_a2a_ddr.txt><flat_a2a_ipm.txt><flat_quad_ddr.txt><flat_quad_ipm.txt><flat_snc4_ddr.txt><flat_snc4_ipm.txt> _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] KNL Status
by Allen, Benjamin S. 21 Apr '16

21 Apr '16

Kevin and I ran into this as well. It appears warm reboot with snc4 mode still hangs. Power cycling the machine now. Ben > On Apr 20, 2016, at 8:19 PM, Scott Parker <sparker(a)anl.gov> wrote: > > > I believe the node is hung while attempting a reboot into snc4 mode > > -Scott > > On 4/13/16 12:46 PM, Allen, Benjamin S. wrote: >> Yea, it hangs on warm reboot. Power cycling it via ipmi now. >> >> Ben >> >>> On Apr 13, 2016, at 2:44 PM, Vitali A. Morozov <morozov(a)anl.gov> wrote: >>> >>> I was using >>> >>> sudo /usr/bin/systemctl reboot >>> >>> and it is booting for over 30 minutes now. Something is not right... >>> >>> Vitali >>> >>> >>> >>> >>> On 04/13/2016 02:05 PM, Vitali A. Morozov wrote: >>>> As no complains have been posted, rebooting to SNC-4/Flat. >>>> >>>> Vitali >>>> >>>> >>>> On 04/13/2016 01:17 PM, Vitali A. Morozov wrote: >>>>> Hello, >>>>> >>>>> With current setup of All2All/Flat, we seem to have a stable 410 GB/s from MCDRAM and 78 GB/s from DDR. >>>>> >>>>> For STREAMing from MCDRAM, Intel people suggested to try SNC-4/Flat promising 430 GB/s. Our previous trials with other combinations of NOC settings lead to unstable results. >>>>> >>>>> Do we want to try it now? It will probably require hard reboot. >>>>> >>>>> Vitali >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On 04/13/2016 01:06 PM, Raman, Karthik wrote: >>>>>> Ok I read this as 418 GB/s J >>>>>> And ok, just realized you are running in All2All mode, so yes this looks good. For Quad/Flat you should see ~440 GB/s with the same run. >>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>> Sent: Wednesday, April 13, 2016 11:02 AM >>>>>> To: Raman, Karthik <karthik.raman(a)intel.com>; Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov> >>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; Fromkin, Russ <russ.fromkin(a)intel.com>; Harms, Kevin <harms(a)alcf.anl.gov>; Morozov, Vitali A. <morozov(a)alcf.anl.gov>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>> Hi Karthik, >>>>>> >>>>>> Repeating the same options you have: >>>>>> >>>>>> mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp -mcmodel medium -shared-intel -opt-streaming-stores always stream.c -o stream >>>>>> >>>>>> ------------------------------------------------------------- >>>>>> STREAM version $Revision: 5.9 $ >>>>>> ------------------------------------------------------------- >>>>>> This system uses 8 bytes per DOUBLE PRECISION word. >>>>>> ------------------------------------------------------------- >>>>>> Array size = 134217728, Offset = 0 >>>>>> Total memory required = 3072.0 MB. >>>>>> Each test is run 100 times, but only >>>>>> the *best* time for each is used. >>>>>> ------------------------------------------------------------- >>>>>> Number of Threads requested = 64 >>>>>> ------------------------------------------------------------- >>>>>> >>>>>> < skiped > >>>>>> >>>>>> Function Rate (MB/s) Avg time Min time Max time >>>>>> Copy: 362331.5200 0.0060 0.0059 0.0061 >>>>>> Scale: 359281.9806 0.0060 0.0060 0.0062 >>>>>> Add: 418122.7024 0.0078 0.0077 0.0081 >>>>>> Triad: 418122.7024 0.0078 0.0077 0.0081 >>>>>> Store: 224685.6729 0.0048 0.0048 0.0049 >>>>>> >>>>>> >>>>>> 408GB/s >>>>>> >>>>>> Vitali >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Intel-nda mailing list >>>>> >>>>> Intel-nda(a)lists.jlse.anl.gov >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] KNL Status
by Scott Parker 20 Apr '16

20 Apr '16

I believe the node is hung while attempting a reboot into snc4 mode -Scott On 4/13/16 12:46 PM, Allen, Benjamin S. wrote: > Yea, it hangs on warm reboot. Power cycling it via ipmi now. > > Ben > >> On Apr 13, 2016, at 2:44 PM, Vitali A. Morozov <morozov(a)anl.gov> wrote: >> >> I was using >> >> sudo /usr/bin/systemctl reboot >> >> and it is booting for over 30 minutes now. Something is not right... >> >> Vitali >> >> >> >> >> On 04/13/2016 02:05 PM, Vitali A. Morozov wrote: >>> As no complains have been posted, rebooting to SNC-4/Flat. >>> >>> Vitali >>> >>> >>> On 04/13/2016 01:17 PM, Vitali A. Morozov wrote: >>>> Hello, >>>> >>>> With current setup of All2All/Flat, we seem to have a stable 410 GB/s from MCDRAM and 78 GB/s from DDR. >>>> >>>> For STREAMing from MCDRAM, Intel people suggested to try SNC-4/Flat promising 430 GB/s. Our previous trials with other combinations of NOC settings lead to unstable results. >>>> >>>> Do we want to try it now? It will probably require hard reboot. >>>> >>>> Vitali >>>> >>>> >>>> >>>> >>>> >>>> On 04/13/2016 01:06 PM, Raman, Karthik wrote: >>>>> Ok I read this as 418 GB/s J >>>>> >>>>> And ok, just realized you are running in All2All mode, so yes this looks good. For Quad/Flat you should see ~440 GB/s with the same run. >>>>> >>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>> Sent: Wednesday, April 13, 2016 11:02 AM >>>>> To: Raman, Karthik <karthik.raman(a)intel.com>; Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov> >>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; Fromkin, Russ <russ.fromkin(a)intel.com>; Harms, Kevin <harms(a)alcf.anl.gov>; Morozov, Vitali A. <morozov(a)alcf.anl.gov>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>> >>>>> Hi Karthik, >>>>> >>>>> Repeating the same options you have: >>>>> >>>>> mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp -mcmodel medium -shared-intel -opt-streaming-stores always stream.c -o stream >>>>> >>>>> ------------------------------------------------------------- >>>>> STREAM version $Revision: 5.9 $ >>>>> ------------------------------------------------------------- >>>>> This system uses 8 bytes per DOUBLE PRECISION word. >>>>> ------------------------------------------------------------- >>>>> Array size = 134217728, Offset = 0 >>>>> Total memory required = 3072.0 MB. >>>>> Each test is run 100 times, but only >>>>> the *best* time for each is used. >>>>> ------------------------------------------------------------- >>>>> Number of Threads requested = 64 >>>>> ------------------------------------------------------------- >>>>> >>>>> < skiped > >>>>> >>>>> Function Rate (MB/s) Avg time Min time Max time >>>>> Copy: 362331.5200 0.0060 0.0059 0.0061 >>>>> Scale: 359281.9806 0.0060 0.0060 0.0062 >>>>> Add: 418122.7024 0.0078 0.0077 0.0081 >>>>> Triad: 418122.7024 0.0078 0.0077 0.0081 >>>>> Store: 224685.6729 0.0048 0.0048 0.0049 >>>>> >>>>> >>>>> 408GB/s >>>>> >>>>> Vitali >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Intel-nda mailing list >>>> >>>> Intel-nda(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Fwd: KNL Status
by Vitali A Morozov 20 Apr '16

20 Apr '16

Hi, a quick update on STREAM with the latest beta compiler [morozov@knl00 STREAM]$ icc -V Intel(R) C Intel(R) 64 Compiler for applications running on Intel(R) 64, Version 17.0.0.042 Beta Build 20160315 compiler options mpiicc -O3 -xMIC-AVX512 -qopt-report=5 -Wall -g -fopenmp -mcmodel medium -shared-intel -qopt-streaming-stores always -restrict stream.c -o stream Running in flat mode all2all morozov@knl00 STREAM]$ export OMP_NUM_THREADS=256 [morozov@knl00 STREAM]$ numactl -m 1 ./stream ------------------------------------------------------------- STREAM version $Revision: 5.9 $ ------------------------------------------------------------- This system uses 8 bytes per DOUBLE PRECISION word. ------------------------------------------------------------- Array size = 134217728, Offset = 0 Total memory required = 3072.0 MB. Each test is run 500 times, but only the *best*time for each is used. ------------------------------------------------------------- Number of Threads requested = 256 ------------------------------------------------------------- <skipped> ------------------------------------------------------------- Your clock granularity/precision appears to be 1 microseconds. Each test below will take on the order of 6356 microseconds. (= 6356 clock ticks) Increase the size of the arrays if this shows that you are not getting at least 20 clock ticks per test. ------------------------------------------------------------- WARNING -- The above is only a rough guideline. For best results, please be sure you know the precision of your system timer. ------------------------------------------------------------- Function Rate (MB/s) Avg time Min time Max time Copy: 330891.5637 0.0066 0.0065 0.0078 Scale: 330127.5200 0.0066 0.0065 0.0069 Add: 383622.4448 0.0085 0.0084 0.0093 Triad: 383252.4574 0.0085 0.0084 0.0098 ------------------------------------------------------------- Solution Validates ------------------------------------------------------------- Same with 64 threads scatter mode: morozov@knl00 STREAM]$ export OMP_NUM_THREADS=64 [morozov@knl00 STREAM]$ KMP_AFFINITY=scatter [morozov@knl00 STREAM]$ numactl -m 1 ./stream ------------------------------------------------------------- STREAM version $Revision: 5.9 $ ------------------------------------------------------------- This system uses 8 bytes per DOUBLE PRECISION word. ------------------------------------------------------------- Array size = 134217728, Offset = 0 Total memory required = 3072.0 MB. Each test is run 500 times, but only the *best*time for each is used. ------------------------------------------------------------- Number of Threads requested = 64 ------------------------------------------------------------- <skipped> ------------------------------------------------------------- Your clock granularity/precision appears to be 1 microseconds. Each test below will take on the order of 6056 microseconds. (= 6056 clock ticks) Increase the size of the arrays if this shows that you are not getting at least 20 clock ticks per test. ------------------------------------------------------------- WARNING -- The above is only a rough guideline. For best results, please be sure you know the precision of your system timer. ------------------------------------------------------------- Function Rate (MB/s) Avg time Min time Max time Copy: 353486.8826 0.0061 0.0061 0.0067 Scale: 353265.0608 0.0061 0.0061 0.0066 Add: 423961.3055 0.0077 0.0076 0.0079 Triad: 424907.9750 0.0076 0.0076 0.0084 ------------------------------------------------------------- Solution Validates ------------------------------------------------------------- Vitali On 4/18/16 12:43 PM, Kumaran, Kalyan wrote: > FYI. Karthik will get on the machine to debug. > > Sent from my iPhone > > Begin forwarded message: > >> *From:* "Chinthamani, Sundaram" <sundaram.chinthamani(a)intel.com >> <mailto:[email protected]>> >> *Date:* April 18, 2016 at 2:05:26 PM CDT >> *To:* "Harms, Kevin" <harms(a)alcf.anl.gov >> <mailto:[email protected]>>, "Allen, Benjamin S." >> <bsallen(a)alcf.anl.gov <mailto:[email protected]>> >> *Cc:* "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov >> <mailto:[email protected]>>, "Kaana Balakrishna, Abhishek" >> <abhishek.kaana.balakrishna(a)intel.com >> <mailto:[email protected]>>, "Morozov, Vitali A." >> <morozov(a)alcf.anl.gov <mailto:[email protected]>>, "Raman, >> Karthik" <karthik.raman(a)intel.com <mailto:[email protected]>>, >> "Inkley, Benson" <benson.inkley(a)intel.com >> <mailto:[email protected]>>, "Sodani, Avinash" >> <avinash.sodani(a)intel.com <mailto:[email protected]>>, >> "Fromkin, Russ" <russ.fromkin(a)intel.com >> <mailto:[email protected]>>, "Chinthamani, Sundaram" >> <sundaram.chinthamani(a)intel.com <mailto:[email protected]>> >> *Subject:* *RE: [JLSE-Intel-NDA] KNL Status* >> >> I would expect Quad mode / SNC-4 mode to provide higher bandwidth >> compared to All2All mode. We haven't seen this in our internal >> measurements. >> >> -Thanks, >> -Sundaram >> >> -----Original Message----- >> From: Harms, Kevin [mailto:[email protected]] >> Sent: Monday, April 18, 2016 12:00 PM >> To: Allen, Benjamin S. <bsallen(a)alcf.anl.gov >> <mailto:[email protected]>> >> Cc: Kumaran, Kalyan <kumaran(a)alcf.anl.gov >> <mailto:[email protected]>>; Kaana Balakrishna, Abhishek >> <abhishek.kaana.balakrishna(a)intel.com >> <mailto:[email protected]>>; Morozov, Vitali A. >> <morozov(a)alcf.anl.gov <mailto:[email protected]>>; Chinthamani, >> Sundaram <sundaram.chinthamani(a)intel.com >> <mailto:[email protected]>>; Raman, Karthik >> <karthik.raman(a)intel.com <mailto:[email protected]>>; Inkley, >> Benson <benson.inkley(a)intel.com <mailto:[email protected]>>; >> Sodani, Avinash <avinash.sodani(a)intel.com >> <mailto:[email protected]>>; Fromkin, Russ >> <russ.fromkin(a)intel.com <mailto:[email protected]>> >> Subject: Re: [JLSE-Intel-NDA] KNL Status >> >> >> I tested cached mode with all2all, snc4 and quad models. Cached >> all2all gave 275,000 MB/s. Does that look reasonable? SNC4 and quad >> mode gave 225,000 MB/s or so. >> >> kevin >> >> >> >> >>> The following versions are now installed: BIOS 01.01.0104, ME >>> 03.01.03.011.0, BMC 00.17.9699, FRUSDR 1.14. >>> >>> Ben >>> >>>> On Apr 18, 2016, at 1:27 PM, Harms, Kevin <harms(a)alcf.anl.gov >>>> <mailto:[email protected]>> wrote: >>>> >>>> >>>> Ben completed the BIOS upgrade, but we do not see any difference in >>>> performance. I have attached results from runs done under three >>>> configurations: flat+all2all, flat+quad, flat+snc4. >>>> >>>> flat+all2all = 427046.0546 MB/s >>>> flat+quad = 327253.8882 MB/s >>>> flat+snc4 = 308555.9588 MB/s >>>> >>>> I compiled the CORAL stream.c with the following options: >>>> icc -mcmodel medium -shared-intel -O3 -xMIC-AVX512 -DN=50000000 >>>> -DOFFSET=0 -DNTIMES=100 -openmp -opt-streaming-stores always >>>> stream.c -o stream >>>> >>>> I added the alignment directives to the arrays. The attached files >>>> show the NUMA domains and numactl commands used and run parameters. >>>> >>>> Have I missed a parameter? >>>> >>>> kevin >>>> >>>> >>>> >>>>> Suggest a time Friday afternoon. We are in central time zone. We >>>>> can get you access to this B0 too. >>>>> >>>>> -Kumar >>>>> >>>>> >>>>>> On Apr 13, 2016, at 5:06 PM, Kaana Balakrishna, Abhishek >>>>>> <abhishek.kaana.balakrishna(a)intel.com >>>>>> <mailto:[email protected]>> wrote: >>>>>> >>>>>> + Benson >>>>>> >>>>>> Cheers, >>>>>> Abhishek K Balakrishna (KB) >>>>>> Xeon Phiï¿½ Hardware Platform Applications Engineer, >>>>>> Technical Computing Group (TCG) >>>>>> Intel Corporation >>>>>> Ph - 512 466 9886 >>>>>> www.intel.com/xeonphi <http://www.intel.com/xeonphi> >>>>>> >>>>>> >>>>>> -----Original Message----- >>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>> Sent: Wednesday, April 13, 2016 4:58 PM >>>>>> To: Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com >>>>>> <mailto:[email protected]>>; Raman, Karthik >>>>>> <karthik.raman(a)intel.com <mailto:[email protected]>>; >>>>>> Harms, Kevin <harms(a)alcf.anl.gov <mailto:[email protected]>>; >>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov >>>>>> <mailto:[email protected]>>; Kumaran, Kalyan >>>>>> <kumaran(a)alcf.anl.gov <mailto:[email protected]>> >>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>> <mailto:[email protected]>>; Fromkin, Russ >>>>>> <russ.fromkin(a)intel.com <mailto:[email protected]>>; Allen, >>>>>> Benjamin S. <bsallen(a)alcf.anl.gov <mailto:[email protected]>>; >>>>>> Kaana Balakrishna, Abhishek <abhishek.kaana.balakrishna(a)intel.com >>>>>> <mailto:[email protected]>> >>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>> >>>>>> Sure! >>>>>> >>>>>> Vitali >>>>>> >>>>>> >>>>>> On 04/13/2016 04:54 PM, Chinthamani, Sundaram wrote: >>>>>>> Would it be possible to have a meeting to go over your questions >>>>>>> rather than exchanging E-mails back and forth? >>>>>>> >>>>>>> + KB from Intel side. He can help set-up the meeting between >>>>>>> Labs and Intel. >>>>>>> >>>>>>> -Thanks, >>>>>>> -Sundaram >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>> Sent: Wednesday, April 13, 2016 2:50 PM >>>>>>> To: Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com >>>>>>> <mailto:[email protected]>>; Raman, >>>>>>> Karthik <karthik.raman(a)intel.com >>>>>>> <mailto:[email protected]>>; Harms, Kevin >>>>>>> <harms(a)alcf.anl.gov <mailto:[email protected]>>; >>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov >>>>>>> <mailto:[email protected]>>; Kumaran, Kalyan >>>>>>> <kumaran(a)alcf.anl.gov <mailto:[email protected]>> >>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>> <mailto:[email protected]>>; Fromkin, Russ >>>>>>> <russ.fromkin(a)intel.com <mailto:[email protected]>>; Allen, >>>>>>> Benjamin S. <bsallen(a)alcf.anl.gov <mailto:[email protected]>> >>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>> >>>>>>> On 04/13/2016 04:42 PM, Chinthamani, Sundaram wrote: >>>>>>>> node 4 free: 2907 MB >>>>>>>> node 5 free: 2926 MB >>>>>>>> node 6 free: 2922 MB >>>>>>>> node 7 free: 2914 MB >>>>>>>> >>>>>>>> There is 16 GB of MC-DRAM and 4 GB per NUMA node. Is there >>>>>>>> any other process running on the machine? Why does "free >>>>>>>> memory" show only 3 GB per node? >>>>>>> This is EXACTLY why I am asking this question. Nothing is >>>>>>> running on the node - and I do not see full 4 GiB of available >>>>>>> MCDRAM memory. This might be reason why I do not see what is >>>>>>> expected. >>>>>>> >>>>>>> I understand that I can reduce the array size, but I am trying >>>>>>> to reproduce your results. >>>>>>> >>>>>>> I am still unclear on if you run "no mpi" version, how come that >>>>>>> SNC-4 mode is giving you advantage? All arrays are allocated in >>>>>>> the same quadrant. Requests from the other quadrants will go to >>>>>>> this one and create congestion. >>>>>>> >>>>>>> Vitali >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> If you want to test Stream triad bandwidth to MC-DRAM in >>>>>>>> Quad/Flat mode, you can use even smaller arrays of few 100 MBs. >>>>>>>> >>>>>>>> -Thanks, >>>>>>>> -Sundaram >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>> Sent: Wednesday, April 13, 2016 2:32 PM >>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com >>>>>>>> <mailto:[email protected]>>; Chinthamani, Sundaram >>>>>>>> <sundaram.chinthamani(a)intel.com >>>>>>>> <mailto:[email protected]>>; Harms, Kevin >>>>>>>> <harms(a)alcf.anl.gov <mailto:[email protected]>>; >>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov >>>>>>>> <mailto:[email protected]>>; Kumaran, Kalyan >>>>>>>> <kumaran(a)alcf.anl.gov <mailto:[email protected]>> >>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>> <mailto:[email protected]>>; Fromkin, Russ >>>>>>>> <russ.fromkin(a)intel.com <mailto:[email protected]>>; >>>>>>>> Allen, Benjamin S. <bsallen(a)alcf.anl.gov >>>>>>>> <mailto:[email protected]>> >>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>> >>>>>>>> ok, that is fine. >>>>>>>> >>>>>>>> Now the next issue I have is that for static arrays, all three >>>>>>>> arrays must land up on one of the numa nodes - the node that is >>>>>>>> attached to a master thread. >>>>>>>> >>>>>>>> You advice to use N = 134217728, that gives exactly 3 GiB of >>>>>>>> memory footprint for 3 arrays of triad. >>>>>>>> >>>>>>>> According to "numactl -H", I do not have 3 GiB of available >>>>>>>> memory on any of the MCDRAM numa nodes: >>>>>>>> >>>>>>>> numactl -H |grep free >>>>>>>> node 0 free: 46336 MB >>>>>>>> node 1 free: 46923 MB >>>>>>>> node 2 free: 46970 MB >>>>>>>> node 3 free: 46961 MB >>>>>>>> node 4 free: 2907 MB >>>>>>>> node 5 free: 2926 MB >>>>>>>> node 6 free: 2922 MB >>>>>>>> node 7 free: 2914 MB >>>>>>>> >>>>>>>> Therefore, at least one array will not be placed to the same >>>>>>>> numa node. >>>>>>>> Is this ok? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 04/13/2016 04:17 PM, Raman, Karthik wrote: >>>>>>>>> Yes we run pure OMP code too. >>>>>>>>> >>>>>>>>> The problem with KMP_AFFINITY with SNC4 mode (currently) is >>>>>>>>> that it >>>>>>>>> may not equally balance the threads per NUMA domain even with >>>>>>>>> scatter or balanced, you can check that by mapping the >>>>>>>>> "verbose" output (e.g. >>>>>>>>> KMP_AFFINITY=scatter,verbose) with "numactl -H" output to see >>>>>>>>> if the >>>>>>>>> OS thread to logical CPU mapping is balanced across all NUMA >>>>>>>>> domains >>>>>>>>> >>>>>>>>> With All2All/Quad this should not be a problem >>>>>>>>> >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>> Sent: Wednesday, April 13, 2016 1:53 PM >>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com >>>>>>>>> <mailto:[email protected]>>; Chinthamani, Sundaram >>>>>>>>> <sundaram.chinthamani(a)intel.com >>>>>>>>> <mailto:[email protected]>>; Harms, Kevin >>>>>>>>> <harms(a)alcf.anl.gov <mailto:[email protected]>>; >>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov >>>>>>>>> <mailto:[email protected]>>; Kumaran, Kalyan >>>>>>>>> <kumaran(a)alcf.anl.gov <mailto:[email protected]>> >>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>> <mailto:[email protected]>>; Fromkin, Russ >>>>>>>>> <russ.fromkin(a)intel.com <mailto:[email protected]>>; >>>>>>>>> Allen, Benjamin S. <bsallen(a)alcf.anl.gov >>>>>>>>> <mailto:[email protected]>> >>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>> >>>>>>>>> Before doing that - let us check if we are running similar >>>>>>>>> benchmark. >>>>>>>>> >>>>>>>>> I am running 1 MPI process (in fact no mpi at all) and 64 >>>>>>>>> OpenMP threads version. The threads can be placed with >>>>>>>>> "balanced" or "scattered" >>>>>>>>> policy. I am using "scattered". >>>>>>>>> >>>>>>>>> Is this ok so far? >>>>>>>>> >>>>>>>>> Vitali >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On 04/13/2016 03:48 PM, Raman, Karthik wrote: >>>>>>>>>> This still does not look right, and the reason could be that the >>>>>>>>>> threads are not balanced across all the NUMA domains (ideally you >>>>>>>>>> want >>>>>>>>>> 16 threads per quadrant) >>>>>>>>>> >>>>>>>>>> Before going to SNC4, can you please boot in "Quad" mode. If you >>>>>>>>>> look in the BIOS Cluster mode options there is another mode >>>>>>>>>> called "Quad" >>>>>>>>>> which is different from SNC4, only 2 NUMA domains (DDR, MCDRAM). >>>>>>>>>> Please run similar to All2All mode >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>> Sent: Wednesday, April 13, 2016 12:56 PM >>>>>>>>>> To: Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com >>>>>>>>>> <mailto:[email protected]>>; Harms, >>>>>>>>>> Kevin <harms(a)alcf.anl.gov <mailto:[email protected]>>; >>>>>>>>>> Raman, Karthik >>>>>>>>>> <karthik.raman(a)intel.com <mailto:[email protected]>>; >>>>>>>>>> Morozov, Vitali A. >>>>>>>>>> <morozov(a)alcf.anl.gov <mailto:[email protected]>>; >>>>>>>>>> Kumaran, Kalyan <kumaran(a)alcf.anl.gov >>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>> <mailto:[email protected]>>; Fromkin, Russ >>>>>>>>>> <russ.fromkin(a)intel.com <mailto:[email protected]>>; >>>>>>>>>> Allen, Benjamin S. <bsallen(a)alcf.anl.gov >>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>> >>>>>>>>>> ok, I am now in SNC-4/Flat. Same binary. >>>>>>>>>> >>>>>>>>>> OMP_NUM_THREADS=64 KMP_AFFINITY=scatter numactl -m 4,5,6,7 >>>>>>>>>> ./stream >>>>>>>>>> |tee running.dat >>>>>>>>>> >>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>> STREAM version $Revision: 5.9 $ >>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>> This system uses 8 bytes per DOUBLE PRECISION word. >>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>> Array size = 134217728, Offset = 0 >>>>>>>>>> Total memory required = 3072.0 MB. >>>>>>>>>> Each test is run 100 times, but only the *best* time for each is >>>>>>>>>> used. >>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>> Number of Threads requested = 64 >>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>> >>>>>>>>>> <skipped> >>>>>>>>>> >>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>> Function Rate (MB/s) Avg time Min time Max time >>>>>>>>>> Copy: 320997.8352 0.0068 0.0067 0.0071 >>>>>>>>>> Scale: 319994.2893 0.0068 0.0067 0.0073 >>>>>>>>>> Add: 306055.0205 0.0106 0.0105 0.0107 >>>>>>>>>> Triad: 306055.0205 0.0106 0.0105 0.0109 >>>>>>>>>> Store: 247056.8669 0.0044 0.0043 0.0051 >>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>> Solution Validates >>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 306055 MB/s >>>>>>>>>> >>>>>>>>>> Vitali >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 04/13/2016 02:01 PM, Chinthamani, Sundaram wrote: >>>>>>>>>>> 440 GB/s (base 10). To be precise, 440 bytes / ns >>>>>>>>>>> (nano-second). We always report memory bandwidth in GB/s >>>>>>>>>>> (base 10) within Intel. >>>>>>>>>>> >>>>>>>>>>> -Thanks, >>>>>>>>>>> -Sundaram >>>>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: Harms, Kevin [mailto:[email protected]] >>>>>>>>>>> Sent: Wednesday, April 13, 2016 11:52 AM >>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com >>>>>>>>>>> <mailto:[email protected]>>; Chinthamani, >>>>>>>>>>> Sundaram <sundaram.chinthamani(a)intel.com >>>>>>>>>>> <mailto:[email protected]>>; Morozov, Vitali A. >>>>>>>>>>> <morozov(a)alcf.anl.gov <mailto:[email protected]>>; >>>>>>>>>>> Kumaran, Kalyan <kumaran(a)alcf.anl.gov >>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>> <mailto:[email protected]>>; Fromkin, Russ >>>>>>>>>>> <russ.fromkin(a)intel.com <mailto:[email protected]>>; >>>>>>>>>>> Allen, Benjamin S. >>>>>>>>>>> <bsallen(a)alcf.anl.gov <mailto:[email protected]>> >>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Just to be clear, memory is always assumed to be >>>>>>>>>>> measured using base 2. It's only the storage field that uses >>>>>>>>>>> base 10. So we assume if you say 440 GB/s you mean GiB, or >>>>>>>>>>> 450,560 MB/s. >>>>>>>>>>> >>>>>>>>>>> kevin >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> And just some fun with units >>>>>>>>>>>> http://sites.utexas.edu/jdm4372/2013/01/05/counting-binary-vs-dec >>>>>>>>>>>> i >>>>>>>>>>>> m >>>>>>>>>>>> a >>>>>>>>>>>> l >>>>>>>>>>>> -powers-in-the-stream-benchmark/ >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> From: Chinthamani, Sundaram >>>>>>>>>>>> >>>>>>>>>>>> Sent: Wednesday, April 13, 2016 11:20 AM >>>>>>>>>>>> To: Vitali A. Morozov <morozov(a)anl.gov >>>>>>>>>>>> <mailto:[email protected]>>; Raman, Karthik >>>>>>>>>>>> <karthik.raman(a)intel.com <mailto:[email protected]>>; >>>>>>>>>>>> Kumaran, Kalyan <kumaran(a)alcf.anl.gov >>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>>> <mailto:[email protected]>>; Fromkin, Russ >>>>>>>>>>>> <russ.fromkin(a)intel.com <mailto:[email protected]>>; >>>>>>>>>>>> Harms, Kevin <harms(a)alcf.anl.gov <mailto:[email protected]>>; >>>>>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov >>>>>>>>>>>> <mailto:[email protected]>>; Allen, Benjamin S. >>>>>>>>>>>> <bsallen(a)alcf.anl.gov <mailto:[email protected]>>; >>>>>>>>>>>> Chinthamani, Sundaram >>>>>>>>>>>> <sundaram.chinthamani(a)intel.com >>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>> Subject: RE: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 440 GB/s stream triad for MC-DRAM and 80 GB/s is our internal >>>>>>>>>>>> bin3 measurement. You should be able to get closer these >>>>>>>>>>>> numbers, if you run the benchmark in Quad/Flat or >>>>>>>>>>>> SNC-4/Flat mode. >>>>>>>>>>>> >>>>>>>>>>>> -Thanks, >>>>>>>>>>>> -Sundaram >>>>>>>>>>>> >>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>> >>>>>>>>>>>> Sent: Wednesday, April 13, 2016 11:11 AM >>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com >>>>>>>>>>>> <mailto:[email protected]>>; Chinthamani, >>>>>>>>>>>> Sundaram <sundaram.chinthamani(a)intel.com >>>>>>>>>>>> <mailto:[email protected]>>; Kumaran, Kalyan >>>>>>>>>>>> <kumaran(a)alcf.anl.gov <mailto:[email protected]>> >>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>>> <mailto:[email protected]>>; Fromkin, Russ >>>>>>>>>>>> <russ.fromkin(a)intel.com <mailto:[email protected]>>; >>>>>>>>>>>> Harms, Kevin <harms(a)alcf.anl.gov <mailto:[email protected]>>; >>>>>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov >>>>>>>>>>>> <mailto:[email protected]>>; Allen, Benjamin S. >>>>>>>>>>>> <bsallen(a)alcf.anl.gov <mailto:[email protected]>> >>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I see - I am treating 418122 MB/s as 418122/1024 GB/s (as >>>>>>>>>>>> this is memory unit). >>>>>>>>>>>> >>>>>>>>>>>> Anyway - great we are on the same page here. >>>>>>>>>>>> >>>>>>>>>>>> I am in All2All/Flat mode. You seem to suggest that I must >>>>>>>>>>>> be on SNC-4/Flat to see 440000 MB/s? >>>>>>>>>>>> >>>>>>>>>>>> Thanks! >>>>>>>>>>>> >>>>>>>>>>>> Vitali >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 04/13/2016 01:06 PM, Raman, Karthik wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Ok I read this as 418 GB/s >>>>>>>>>>>> J >>>>>>>>>>>> >>>>>>>>>>>> And ok, just realized you are running in All2All mode, so >>>>>>>>>>>> yes this looks good. For Quad/Flat you should see ~440 GB/s >>>>>>>>>>>> with the same run. >>>>>>>>>>>> >>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>> >>>>>>>>>>>> Sent: Wednesday, April 13, 2016 11:02 AM >>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com >>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]>; Chinthamani, Sundaram >>>>>>>>>>>> <sundaram.chinthamani(a)intel.com >>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>> Kumaran, Kalyan <kumaran(a)alcf.anl.gov >>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>> Fromkin, Russ <russ.fromkin(a)intel.com >>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]>; Harms, Kevin >>>>>>>>>>>> <harms(a)alcf.anl.gov <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]>; Morozov, Vitali A. >>>>>>>>>>>> <morozov(a)alcf.anl.gov <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]>; Allen, Benjamin S. >>>>>>>>>>>> <bsallen(a)alcf.anl.gov <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hi Karthik, >>>>>>>>>>>> >>>>>>>>>>>> Repeating the same options you have: >>>>>>>>>>>> >>>>>>>>>>>> mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp >>>>>>>>>>>> -mcmodel >>>>>>>>>>>> medium -shared-intel -opt-streaming-stores always stream.c -o >>>>>>>>>>>> stream >>>>>>>>>>>> >>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>> STREAM version $Revision: 5.9 $ >>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>> This system uses 8 bytes per DOUBLE PRECISION word. >>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>> Array size = 134217728, Offset = 0 Total memory required = >>>>>>>>>>>> 3072.0 >>>>>>>>>>>> MB. >>>>>>>>>>>> Each test is run 100 times, but only the *best* time for >>>>>>>>>>>> each is >>>>>>>>>>>> used. >>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>> Number of Threads requested = 64 >>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>> >>>>>>>>>>>> < skiped > >>>>>>>>>>>> >>>>>>>>>>>> Function Rate (MB/s) Avg time Min time Max time >>>>>>>>>>>> Copy: 362331.5200 0.0060 0.0059 0.0061 >>>>>>>>>>>> Scale: 359281.9806 0.0060 0.0060 0.0062 >>>>>>>>>>>> Add: 418122.7024 0.0078 0.0077 0.0081 >>>>>>>>>>>> Triad: 418122.7024 0.0078 0.0077 0.0081 >>>>>>>>>>>> Store: 224685.6729 0.0048 0.0048 0.0049 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 408GB/s >>>>>>>>>>>> >>>>>>>>>>>> Vitali >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 04/13/2016 12:15 PM, Raman, Karthik wrote: >>>>>>>>>>>> >>>>>>>>>>>> 2. Effective MCDRAM bandwidth is around 406-408 GB/s. >>>>>>>>>>>> Larger values are possible for more high core count bins. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> For bin3 , you should be able to get Triad ~440 GB/s. Please >>>>>>>>>>>> check if you use the following Intel compiler flags -mcmodel >>>>>>>>>>>> medium -shared-intel -O3 -xMIC-AVX512 >>>>>>>>>>>> -DSTREAM_ARRAY_SIZE=134217728 -DOFFSET=0 -DNTIMES=100 >>>>>>>>>>>> -openmp -opt-streaming-stores always The array size is just >>>>>>>>>>>> allocating 1GB per array. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>> >>>>>>>>>>>> Sent: Wednesday, April 13, 2016 10:01 AM >>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com >>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]>; Chinthamani, Sundaram >>>>>>>>>>>> <sundaram.chinthamani(a)intel.com >>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>> Kumaran, Kalyan <kumaran(a)alcf.anl.gov >>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>> Fromkin, Russ <russ.fromkin(a)intel.com >>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]>; Harms, Kevin >>>>>>>>>>>> <harms(a)alcf.anl.gov <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]>; Morozov, Vitali A. >>>>>>>>>>>> <morozov(a)alcf.anl.gov <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]>; Allen, Benjamin S. >>>>>>>>>>>> <bsallen(a)alcf.anl.gov <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hello, >>>>>>>>>>>> >>>>>>>>>>>> so, this is my understanding that on bin3 hardware (64 cores, 6 >>>>>>>>>>>> channel DDR4-2133) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> 1. Effective DDR bandwidth is 80 GB/s (around 78% of peak). >>>>>>>>>>>> >>>>>>>>>>>> 2. Effective MCDRAM bandwidth is around 406-408 GB/s. >>>>>>>>>>>> Larger values are possible for more high core count bins. >>>>>>>>>>>> >>>>>>>>>>>> 3. Alignment is not important for STREAM triad benchmark. >>>>>>>>>>>> >>>>>>>>>>>> 4. For both static and dynamic variables, numactl -m [0|1] >>>>>>>>>>>> should be used to direct memory placement. >>>>>>>>>>>> >>>>>>>>>>>> 5. For explicit MCDRAM allocation in flat mode, >>>>>>>>>>>> hbw_malloc.h API should be use. >>>>>>>>>>>> >>>>>>>>>>>> Please correct if I am wrong... >>>>>>>>>>>> >>>>>>>>>>>> Thanks a lot for you help! >>>>>>>>>>>> >>>>>>>>>>>> Vitali >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 04/12/2016 11:07 AM, Vitali A. Morozov wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi Karthik, >>>>>>>>>>>> >>>>>>>>>>>> I was also wandering how the variable could be influencing >>>>>>>>>>>> the allocation if, in principle, I do not need to link >>>>>>>>>>>> against memkind in my case. Anyway, sorry for confusion! >>>>>>>>>>>> >>>>>>>>>>>> I think I did that before - see my first message. I was >>>>>>>>>>>> using "numactl --membind=1 ./stream" , which I believe is >>>>>>>>>>>> equivalent to "numactl -m 1 ./stream" that you suggest. >>>>>>>>>>>> >>>>>>>>>>>> Here is the result of "numactl -H": >>>>>>>>>>>> >>>>>>>>>>>> [morozov@knl00 512_3_C64_MPI16_TH08_flat]$ numactl -H >>>>>>>>>>>> available: 2 nodes (0-1) >>>>>>>>>>>> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 >>>>>>>>>>>> 19 20 >>>>>>>>>>>> 21 >>>>>>>>>>>> 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 >>>>>>>>>>>> 42 43 >>>>>>>>>>>> 44 >>>>>>>>>>>> 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 >>>>>>>>>>>> 65 66 >>>>>>>>>>>> 67 >>>>>>>>>>>> 68 69 70 >>>>>>>>>>>> 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 >>>>>>>>>>>> 91 92 >>>>>>>>>>>> 93 >>>>>>>>>>>> 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 >>>>>>>>>>>> 110 111 >>>>>>>>>>>> 112 >>>>>>>>>>>> 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 >>>>>>>>>>>> 129 >>>>>>>>>>>> 130 131 132 133 134 135 136 137 138 139 140 141 142 >>>>>>>>>>>> 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 >>>>>>>>>>>> 159 >>>>>>>>>>>> 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 >>>>>>>>>>>> 176 >>>>>>>>>>>> 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 >>>>>>>>>>>> 193 >>>>>>>>>>>> 194 195 196 197 198 199 200 201 202 203 204 205 206 >>>>>>>>>>>> 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 >>>>>>>>>>>> 223 >>>>>>>>>>>> 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 >>>>>>>>>>>> 240 >>>>>>>>>>>> 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 >>>>>>>>>>>> node >>>>>>>>>>>> 0 >>>>>>>>>>>> size: 196511 MB node 0 free: 186888 MB node 1 cpus: >>>>>>>>>>>> node 1 size: 16384 MB >>>>>>>>>>>> node 1 free: 11731 MB >>>>>>>>>>>> node distances: >>>>>>>>>>>> node 0 1 >>>>>>>>>>>> 0: 10 31 >>>>>>>>>>>> 1: 31 10 >>>>>>>>>>>> >>>>>>>>>>>> Sounds like all as expected. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [morozov@knl00 STREAM]$ OMP_NUM_THREADS=64 >>>>>>>>>>>> KMP_AFFINITY=scatter numactl -m 1 ./stream >>>>>>>>>>>> Triad: 418262.7498 0.0058 0.0057 0.0060 >>>>>>>>>>>> >>>>>>>>>>>> For a series of runs, I see this: >>>>>>>>>>>> >>>>>>>>>>>> # MB N GB/s >>>>>>>>>>>> 228.88 10000000 408867.98 >>>>>>>>>>>> 457.76 20000000 405818.57 >>>>>>>>>>>> 686.65 30000000 414309.08 >>>>>>>>>>>> 915.53 40000000 415063.59 >>>>>>>>>>>> 1144.41 50000000 415792.22 >>>>>>>>>>>> 1373.29 60000000 415449.01 >>>>>>>>>>>> 1602.17 70000000 414106.18 >>>>>>>>>>>> 1831.05 80000000 416846.82 >>>>>>>>>>>> 2059.94 90000000 414992.29 >>>>>>>>>>>> 2288.82 100000000 417898.11 >>>>>>>>>>>> >>>>>>>>>>>> Vitali >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 04/12/2016 10:32 AM, Raman, Karthik wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi Vitali, >>>>>>>>>>>> >>>>>>>>>>>> 1. >>>>>>>>>>>> The MEMKIND_HBW_NODES is *only* when you are allocating using >>>>>>>>>>>> hbwalloc (+ MCDRAM flat) in your application and when >>>>>>>>>>>> memkind is >>>>>>>>>>>> unable to find the high bandwidth nodes (pmtt not populated) >>>>>>>>>>>> >>>>>>>>>>>> 2. >>>>>>>>>>>> With the below >>>>>>>>>>>> static allocations, in MCDRAM Flat mode, you can simply use >>>>>>>>>>>> ï¿½numactl ï¿½m 1 ./streamï¿½ with propery affinity settings to >>>>>>>>>>>> get peak out of MCDRAM. >>>>>>>>>>>> a. >>>>>>>>>>>> Check ï¿½numactl ï¿½Hï¿½ to find the NUMA domains in your node. If >>>>>>>>>>>> booted in All2All/FLAT or Quad/Flat mode it should show 2 NUMA >>>>>>>>>>>> domains , one for DDR (96 GB) and other for MCDRAM (16GB) >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> Karthik >>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>> >>>>>>>>>>>> Sent: Tuesday, April 12, 2016 7:27 AM >>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com >>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]>; Chinthamani, Sundaram >>>>>>>>>>>> <sundaram.chinthamani(a)intel.com >>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>> Kumaran, Kalyan <kumaran(a)alcf.anl.gov >>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>> Fromkin, Russ <russ.fromkin(a)intel.com >>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]>; Harms, Kevin >>>>>>>>>>>> <harms(a)alcf.anl.gov <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]>; Morozov, Vitali A. >>>>>>>>>>>> <morozov(a)alcf.anl.gov <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]>; Allen, Benjamin S. >>>>>>>>>>>> <bsallen(a)alcf.anl.gov <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hi Karthik, >>>>>>>>>>>> >>>>>>>>>>>> The node is in All2All mode for now. I made the following >>>>>>>>>>>> changes. The original code is using static allocation >>>>>>>>>>>> therefore, I follow the advice on 2MB static array alignment: >>>>>>>>>>>> >>>>>>>>>>>> [morozov@thing00 STREAM]$ head -n 108 < stream.c | tail -n 4 >>>>>>>>>>>> static double a[N+OFFSET] __attribute__((aligned(2097152))), >>>>>>>>>>>> b[N+OFFSET] __attribute__((aligned(2097152))), >>>>>>>>>>>> c[N+OFFSET] __attribute__((aligned(2097152))), >>>>>>>>>>>> d[N+OFFSET] >>>>>>>>>>>> __attribute__((aligned(2097152))); >>>>>>>>>>>> >>>>>>>>>>>> [morozov@knl00 STREAM]$ make >>>>>>>>>>>> mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp >>>>>>>>>>>> -mcmodel >>>>>>>>>>>> medium -shared-intel -fno-alias stream.c -o stream >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> [morozov@knl00 STREAM]$ MEMKIND_HBW_NODES=1 OMP_NUM_THREADS=64 >>>>>>>>>>>> KMP_AFFINITY=scatter ./stream >>>>>>>>>>>> >>>>>>>>>>>> ..... >>>>>>>>>>>> >>>>>>>>>>>> Triad: 80251.3621 0.0300 0.0299 0.0305 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Apparently, the "MEMKIND_HBW_NODES=1" is ignored. >>>>>>>>>>>> >>>>>>>>>>>> Thanks! >>>>>>>>>>>> >>>>>>>>>>>> Vitali >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 04/12/2016 01:51 AM, Raman, Karthik wrote: >>>>>>>>>>>> >>>>>>>>>>>> Also could you please modify the memory allocations in >>>>>>>>>>>> STREAM to align at 2MB boundary. >>>>>>>>>>>> >>>>>>>>>>>> For example: >>>>>>>>>>>> static allocations: >>>>>>>>>>>> >>>>>>>>>>>> static STREAM_TYPE a[STREAM_ARRAY_SIZE+OFFSET] >>>>>>>>>>>> __attribute__((aligned(2097152))) >>>>>>>>>>>> >>>>>>>>>>>> dynamic allocations: >>>>>>>>>>>> a = (STREAM_TYPE >>>>>>>>>>>> *)_mm_malloc(sizeof(STREAM_TYPE)*(STREAM_ARRAY_SIZE+OFFSET),20971 >>>>>>>>>>>> 5 >>>>>>>>>>>> 2 >>>>>>>>>>>> ) >>>>>>>>>>>> ; >>>>>>>>>>>> >>>>>>>>>>>> memkind/hbwalloc: >>>>>>>>>>>> (STREAM_TYPE *)hbw_posix_memalign((void **)&a, 2097152, >>>>>>>>>>>> sizeof(STREAM_TYPE)*(STREAM_ARRAY_SIZE+OFFSET)) >>>>>>>>>>>> >>>>>>>>>>>> For memkind/hbwmalloc to work out of the box, you need to have >>>>>>>>>>>> the PMTT tables populated so that it can identify the high >>>>>>>>>>>> bandwidth nodes. Alternatively you can set the >>>>>>>>>>>> MEMKIND_HBW_NODES environment variable to the high >>>>>>>>>>>> bandwidth node (MCDRAM). >>>>>>>>>>>> >>>>>>>>>>>> e.g. All2All/Flat or Quad/Flat : export MEMKIND_HBW_NODES=1 >>>>>>>>>>>> ; ./app_with_hbwmalloc >>>>>>>>>>>> SNC4/Flat: export MEMKIND_HBW_NODES=4,5,6,7 ; >>>>>>>>>>>> ./app_with_hbw_malloc >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> Karthik >>>>>>>>>>>> >>>>>>>>>>>> From: Chinthamani, Sundaram >>>>>>>>>>>> >>>>>>>>>>>> Sent: Monday, April 11, 2016 8:15 PM >>>>>>>>>>>> To: Vitali Morozov <morozov(a)anl.gov >>>>>>>>>>>> <mailto:[email protected]>> <mailto:[email protected]>; >>>>>>>>>>>> Kumaran, Kalyan <kumaran(a)alcf.anl.gov >>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>> Fromkin, Russ <russ.fromkin(a)intel.com >>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]>; Raman, Karthik >>>>>>>>>>>> <karthik.raman(a)intel.com <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]>; Harms, Kevin >>>>>>>>>>>> <harms(a)alcf.anl.gov <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]>; Morozov, Vitali A. >>>>>>>>>>>> <morozov(a)alcf.anl.gov <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]>; Allen, Benjamin S. >>>>>>>>>>>> <bsallen(a)alcf.anl.gov <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>> Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com >>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>> Subject: RE: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> You should use numactl ï¿½m 4,5,6,7 to get the memory >>>>>>>>>>>> bandwidth out of MC-DRAM. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> What kind of part is it? Is it bin1 or bin3? Even in >>>>>>>>>>>> All2All mode, we have measured up to 470 GB/s of stream >>>>>>>>>>>> triad bandwidth. Quad mode should get 490 GB/s. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> DDR-stream triad bandwidth on Bin1-part should be close to >>>>>>>>>>>> 90 GB/s. >>>>>>>>>>>> >>>>>>>>>>>> -Sundaram >>>>>>>>>>>> >>>>>>>>>>>> From: Vitali Morozov [mailto:[email protected]] >>>>>>>>>>>> >>>>>>>>>>>> Sent: Monday, April 11, 2016 6:59 PM >>>>>>>>>>>> To: Kumaran, Kalyan <kumaran(a)alcf.anl.gov >>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>>> <mailto:[email protected]>>; Fromkin, Russ >>>>>>>>>>>> <russ.fromkin(a)intel.com <mailto:[email protected]>>; >>>>>>>>>>>> Chinthamani, Sundaram >>>>>>>>>>>> <sundaram.chinthamani(a)intel.com >>>>>>>>>>>> <mailto:[email protected]>>; Raman, Karthik >>>>>>>>>>>> <karthik.raman(a)intel.com <mailto:[email protected]>>; >>>>>>>>>>>> Harms, Kevin <harms(a)alcf.anl.gov <mailto:[email protected]>>; >>>>>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov >>>>>>>>>>>> <mailto:[email protected]>>; Allen, Benjamin S. >>>>>>>>>>>> <bsallen(a)alcf.anl.gov <mailto:[email protected]>> >>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Avinash and others, >>>>>>>>>>>> >>>>>>>>>>>> here is what we have so far for Flat more, All2All. >>>>>>>>>>>> >>>>>>>>>>>> The results from memory were obtained by using default >>>>>>>>>>>> STREAM with static allocation. The results from MCDRAM were >>>>>>>>>>>> obtained by using modified STREAM with hbwmalloc.h interface. >>>>>>>>>>>> >>>>>>>>>>>> 1 MPI process, 64 threads, scatter affinity, no numactl is >>>>>>>>>>>> used. >>>>>>>>>>>> >>>>>>>>>>>> Most results are as expected: 80 GB/s from DDR memory, 420 >>>>>>>>>>>> GB/s from MCDRAM, the MCDRAM size is about 16 GB, the >>>>>>>>>>>> aggregate L2 cache size is 32 MB. >>>>>>>>>>>> >>>>>>>>>>>> Noticed inconsistencies for hbwmalloc benchmarks: >>>>>>>>>>>> >>>>>>>>>>>> 1. The result is not validated - will spend more time on >>>>>>>>>>>> that tomorrow. >>>>>>>>>>>> 2. hbw_malloc continues allocating data beyond the expected >>>>>>>>>>>> 16 GB up to 48 GB. This means it fails back to DDR if it >>>>>>>>>>>> cannot allocate in MCDRAM - not expected default behavior. >>>>>>>>>>>> It must fail if it does not have enough space. >>>>>>>>>>>> >>>>>>>>>>>> The problem shows up when I am running static allocation >>>>>>>>>>>> and use numactl --membind=1 to ask for MCDRAM-only >>>>>>>>>>>> allocation. The variation is very large and inconsistent >>>>>>>>>>>> with the result observed when using hbw_malloc API: >>>>>>>>>>>> >>>>>>>>>>>> # MB N GB/s numactl GB/s hbw_malloc >>>>>>>>>>>> 228.88 10000000 255555.46 421009.18 >>>>>>>>>>>> 457.76 20000000 363930.93 416997.91 >>>>>>>>>>>> 686.65 30000000 309415.87 427808.31 >>>>>>>>>>>> 915.53 40000000 371691.30 423310.75 >>>>>>>>>>>> 1144.41 50000000 187002.22 431032.35 >>>>>>>>>>>> 1373.29 60000000 385954.23 427172.91 >>>>>>>>>>>> 1602.17 70000000 388254.49 426823.59 >>>>>>>>>>>> 1831.05 80000000 374491.43 428103.96 >>>>>>>>>>>> 2059.94 90000000 379622.74 428152.02 >>>>>>>>>>>> 2288.82 100000000 80868.34 430185.03 >>>>>>>>>>>> >>>>>>>>>>>> We also tried SNC-4 instead of All2All, but could not get >>>>>>>>>>>> more than 40 GB/s from MCDRAM even with hbw_malloc API. >>>>>>>>>>>> >>>>>>>>>>>> Raw data: >>>>>>>>>>>> >>>>>>>>>>>> # flat memory, running from memory (default), alltoall >>>>>>>>>>>> # MB N GB/s >>>>>>>>>>>> 2.29 100000 272062.96 >>>>>>>>>>>> 4.58 200000 491040.47 >>>>>>>>>>>> 6.87 300000 603979.78 >>>>>>>>>>>> 9.16 400000 745654.04 >>>>>>>>>>>> 11.44 500000 811800.77 >>>>>>>>>>>> 13.73 600000 901462.35 >>>>>>>>>>>> 16.02 700000 891953.26 >>>>>>>>>>>> 18.31 800000 1019375.15 >>>>>>>>>>>> 20.60 900000 984749.63 >>>>>>>>>>>> 22.89 1000000 923516.48 >>>>>>>>>>>> 25.18 1100000 758422.09 >>>>>>>>>>>> 27.47 1200000 779328.74 >>>>>>>>>>>> 29.75 1300000 513185.43 >>>>>>>>>>>> 32.04 1400000 155550.35 >>>>>>>>>>>> 34.33 1500000 103209.12 >>>>>>>>>>>> 36.62 1600000 97435.74 >>>>>>>>>>>> 38.91 1700000 83968.40 >>>>>>>>>>>> 41.20 1800000 83692.35 >>>>>>>>>>>> 43.49 1900000 80734.60 >>>>>>>>>>>> 45.78 2000000 76317.89 >>>>>>>>>>>> 68.66 3000000 81180.08 >>>>>>>>>>>> 91.55 4000000 78889.73 >>>>>>>>>>>> 114.44 5000000 79891.50 >>>>>>>>>>>> 137.33 6000000 79954.96 >>>>>>>>>>>> 160.22 7000000 79477.00 >>>>>>>>>>>> 183.11 8000000 79145.59 >>>>>>>>>>>> 205.99 9000000 79821.12 >>>>>>>>>>>> 228.88 10000000 79519.15 >>>>>>>>>>>> 228.88 10000000 79212.54 >>>>>>>>>>>> 457.76 20000000 79692.27 >>>>>>>>>>>> 686.65 30000000 79866.15 >>>>>>>>>>>> 915.53 40000000 79828.15 >>>>>>>>>>>> 1144.41 50000000 80214.91 >>>>>>>>>>>> 1373.29 60000000 80226.84 >>>>>>>>>>>> 1602.17 70000000 80178.77 >>>>>>>>>>>> 1831.05 80000000 80177.06 >>>>>>>>>>>> 2059.94 90000000 80109.79 >>>>>>>>>>>> 2288.82 100000000 80175.94 >>>>>>>>>>>> >>>>>>>>>>>> # flat memory, running from MCDRAM, alltoall, hbwmalloc API >>>>>>>>>>>> # MB N GB/s >>>>>>>>>>>> 228.88 10000000 421009.18 >>>>>>>>>>>> 457.76 20000000 416997.91 >>>>>>>>>>>> 686.65 30000000 427808.31 >>>>>>>>>>>> 915.53 40000000 423310.75 >>>>>>>>>>>> 1144.41 50000000 431032.35 >>>>>>>>>>>> 1373.29 60000000 427172.91 >>>>>>>>>>>> 1602.17 70000000 426823.59 >>>>>>>>>>>> 1831.05 80000000 428103.96 >>>>>>>>>>>> 2059.94 90000000 428152.02 >>>>>>>>>>>> 2288.82 100000000 430185.03 >>>>>>>>>>>> 2343.75 102400000 425999.98 >>>>>>>>>>>> 4687.50 204800000 427821.10 >>>>>>>>>>>> 9155.27 400000000 429318.13 >>>>>>>>>>>> 9375.00 409600000 428376.70 >>>>>>>>>>>> 11444.09 500000000 428052.08 >>>>>>>>>>>> 13732.91 600000000 266061.01 >>>>>>>>>>>> 16021.73 700000000 200257.79 >>>>>>>>>>>> 18310.55 800000000 168164.19 >>>>>>>>>>>> 20599.37 900000000 150192.91 >>>>>>>>>>>> 22888.18 1000000000 137862.07 >>>>>>>>>>>> 25177.00 1100000000 129418.12 >>>>>>>>>>>> 27465.82 1200000000 123019.53 >>>>>>>>>>>> 29754.64 1300000000 118268.79 >>>>>>>>>>>> 32043.46 1400000000 114343.33 >>>>>>>>>>>> 34332.28 1500000000 110991.25 >>>>>>>>>>>> 36621.09 1600000000 108538.36 >>>>>>>>>>>> 38909.91 1700000000 106249.72 >>>>>>>>>>>> 41198.73 1800000000 104455.07 >>>>>>>>>>>> 43487.55 1900000000 102828.93 >>>>>>>>>>>> 45776.37 2000000000 101348.56 >>>>>>>>>>>> 48065.19 2100000000 100013.73 >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> We appreciate any help on that. >>>>>>>>>>>> >>>>>>>>>>>> Best, >>>>>>>>>>>> >>>>>>>>>>>> Vitali >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 04/11/2016 08:42 PM, Kumaran, Kalyan wrote: >>>>>>>>>>>> >>>>>>>>>>>> Avinash, thanks for the prompt reply. Kevin/Vitali can provide >>>>>>>>>>>> all the details. The short summary is we are unable to >>>>>>>>>>>> reproduce the STREAM results from IPM or DDR. It would be >>>>>>>>>>>> great if Karthik or Sundaram can advise us on how to run >>>>>>>>>>>> (which mode etc) or if they could log into the machine and >>>>>>>>>>>> make sure we have set up things correctly. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>>> Kumar >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Apr 11, 2016, at 8:07 PM, Sodani, Avinash >>>>>>>>>>>> <avinash.sodani(a)intel.com >>>>>>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Copying Sundaram/Karthik from our side. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Kumar, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Can you send us more information on this? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -Avinash >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> From: Kumaran, >>>>>>>>>>>> Kalyan [mailto:[email protected]] >>>>>>>>>>>> Sent: Monday, April 11, 2016 5:32 PM >>>>>>>>>>>> To: Fromkin, Russ; Sodani, Avinash >>>>>>>>>>>> Cc: Harms, Kevin; Morozov, Vitali A. >>>>>>>>>>>> Subject: Fwd: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Russ, Avinash, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> we need help in understanding how to obtain STREAM results. >>>>>>>>>>>> Can anyone help? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Kumar >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Begin forwarded message: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> From: "Harms, Kevin" <harms(a)alcf.anl.gov >>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Subject: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Date: April 11, 2016 at 5:42:15 PM CDT >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> To: "intel-nda(a)jlse.anl.gov <mailto:[email protected]>" >>>>>>>>>>>> <intel-nda(a)jlse.anl.gov <mailto:[email protected]>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> The knl00 node is currently in Flat mode and All2All model. >>>>>>>>>>>> With this setup we can get 420 GB/s from IPM memory on >>>>>>>>>>>> STREAM Triad when using hbw_malloc and friends which was >>>>>>>>>>>> confirmed by both Vitali and I. >>>>>>>>>>>> >>>>>>>>>>>> Changing the BIOS settings and warm rebooting seem to be >>>>>>>>>>>> problematic. Vitali and I were running earlier and could >>>>>>>>>>>> not get over 80 GB/s with any method even though memory >>>>>>>>>>>> capacity indicated flat mode. >>>>>>>>>>>> >>>>>>>>>>>> Using numactl --membind is also problematic. The STREAM results >>>>>>>>>>>> when using this to launch the application were erratic, so >>>>>>>>>>>> it is also suggested not to use that. That means you will >>>>>>>>>>>> need to modify your code to use dynamic memory and >>>>>>>>>>>> hbw_malloc to exploit the IPM. >>>>>>>>>>>> >>>>>>>>>>>> Will followup with more detailed results later. >>>>>>>>>>>> >>>>>>>>>>>> kevin >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Intel-nda mailing list >>>>>>>>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>> >>>>> >>>> <flat_a2a_ddr.txt><flat_a2a_ipm.txt><flat_quad_ddr.txt><flat_quad_ipm.txt><flat_snc4_ddr.txt><flat_snc4_ipm.txt> >>> > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] Call w Avinash Sodhani
by Kumaran, Kalyan 19 Apr '16

19 Apr '16

We have a call scheduled with Avinash Sodhani (Chief Architect of KNL) for next week. It would be good to raise any issues, share success stories we have so far. Date: April 26th (Tuesday) Time: 2:00pm Room: 4313 -Kumar

1 0

[JLSE-Intel-NDA] Fwd: KNL Status
by Kumaran, Kalyan 18 Apr '16

18 Apr '16

FYI. Karthik will get on the machine to debug. Sent from my iPhone Begin forwarded message: From: "Chinthamani, Sundaram" <sundaram.chinthamani(a)intel.com<mailto:[email protected]>> Date: April 18, 2016 at 2:05:26 PM CDT To: "Harms, Kevin" <harms(a)alcf.anl.gov<mailto:[email protected]>>, "Allen, Benjamin S." <bsallen(a)alcf.anl.gov<mailto:[email protected]>> Cc: "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov<mailto:[email protected]>>, "Kaana Balakrishna, Abhishek" <abhishek.kaana.balakrishna(a)intel.com<mailto:[email protected]>>, "Morozov, Vitali A." <morozov(a)alcf.anl.gov<mailto:[email protected]>>, "Raman, Karthik" <karthik.raman(a)intel.com<mailto:[email protected]>>, "Inkley, Benson" <benson.inkley(a)intel.com<mailto:[email protected]>>, "Sodani, Avinash" <avinash.sodani(a)intel.com<mailto:[email protected]>>, "Fromkin, Russ" <russ.fromkin(a)intel.com<mailto:[email protected]>>, "Chinthamani, Sundaram" <sundaram.chinthamani(a)intel.com<mailto:[email protected]>> Subject: RE: [JLSE-Intel-NDA] KNL Status I would expect Quad mode / SNC-4 mode to provide higher bandwidth compared to All2All mode. We haven't seen this in our internal measurements. -Thanks, -Sundaram -----Original Message----- From: Harms, Kevin [mailto:[email protected]] Sent: Monday, April 18, 2016 12:00 PM To: Allen, Benjamin S. <bsallen(a)alcf.anl.gov<mailto:[email protected]>> Cc: Kumaran, Kalyan <kumaran(a)alcf.anl.gov<mailto:[email protected]>>; Kaana Balakrishna, Abhishek <abhishek.kaana.balakrishna(a)intel.com<mailto:[email protected]>>; Morozov, Vitali A. <morozov(a)alcf.anl.gov<mailto:[email protected]>>; Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com<mailto:[email protected]>>; Raman, Karthik <karthik.raman(a)intel.com<mailto:[email protected]>>; Inkley, Benson <benson.inkley(a)intel.com<mailto:[email protected]>>; Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>>; Fromkin, Russ <russ.fromkin(a)intel.com<mailto:[email protected]>> Subject: Re: [JLSE-Intel-NDA] KNL Status I tested cached mode with all2all, snc4 and quad models. Cached all2all gave 275,000 MB/s. Does that look reasonable? SNC4 and quad mode gave 225,000 MB/s or so. kevin The following versions are now installed: BIOS 01.01.0104, ME 03.01.03.011.0, BMC 00.17.9699, FRUSDR 1.14. Ben On Apr 18, 2016, at 1:27 PM, Harms, Kevin <harms(a)alcf.anl.gov<mailto:[email protected]>> wrote: Ben completed the BIOS upgrade, but we do not see any difference in performance. I have attached results from runs done under three configurations: flat+all2all, flat+quad, flat+snc4. flat+all2all = 427046.0546 MB/s flat+quad = 327253.8882 MB/s flat+snc4 = 308555.9588 MB/s I compiled the CORAL stream.c with the following options: icc -mcmodel medium -shared-intel -O3 -xMIC-AVX512 -DN=50000000 -DOFFSET=0 -DNTIMES=100 -openmp -opt-streaming-stores always stream.c -o stream I added the alignment directives to the arrays. The attached files show the NUMA domains and numactl commands used and run parameters. Have I missed a parameter? kevin Suggest a time Friday afternoon. We are in central time zone. We can get you access to this B0 too. -Kumar On Apr 13, 2016, at 5:06 PM, Kaana Balakrishna, Abhishek <abhishek.kaana.balakrishna(a)intel.com<mailto:[email protected]>> wrote: + Benson Cheers, Abhishek K Balakrishna (KB) Xeon Phi™ Hardware Platform Applications Engineer, Technical Computing Group (TCG) Intel Corporation Ph - 512 466 9886 www.intel.com/xeonphi<http://www.intel.com/xeonphi> -----Original Message----- From: Vitali A. Morozov [mailto:[email protected]] Sent: Wednesday, April 13, 2016 4:58 PM To: Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com<mailto:[email protected]>>; Raman, Karthik <karthik.raman(a)intel.com<mailto:[email protected]>>; Harms, Kevin <harms(a)alcf.anl.gov<mailto:[email protected]>>; Morozov, Vitali A. <morozov(a)alcf.anl.gov<mailto:[email protected]>>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov<mailto:[email protected]>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>>; Fromkin, Russ <russ.fromkin(a)intel.com<mailto:[email protected]>>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov<mailto:[email protected]>>; Kaana Balakrishna, Abhishek <abhishek.kaana.balakrishna(a)intel.com<mailto:[email protected]>> Subject: Re: [JLSE-Intel-NDA] KNL Status Sure! Vitali On 04/13/2016 04:54 PM, Chinthamani, Sundaram wrote: Would it be possible to have a meeting to go over your questions rather than exchanging E-mails back and forth? + KB from Intel side. He can help set-up the meeting between Labs and Intel. -Thanks, -Sundaram -----Original Message----- From: Vitali A. Morozov [mailto:[email protected]] Sent: Wednesday, April 13, 2016 2:50 PM To: Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com<mailto:[email protected]>>; Raman, Karthik <karthik.raman(a)intel.com<mailto:[email protected]>>; Harms, Kevin <harms(a)alcf.anl.gov<mailto:[email protected]>>; Morozov, Vitali A. <morozov(a)alcf.anl.gov<mailto:[email protected]>>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov<mailto:[email protected]>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>>; Fromkin, Russ <russ.fromkin(a)intel.com<mailto:[email protected]>>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov<mailto:[email protected]>> Subject: Re: [JLSE-Intel-NDA] KNL Status On 04/13/2016 04:42 PM, Chinthamani, Sundaram wrote: node 4 free: 2907 MB node 5 free: 2926 MB node 6 free: 2922 MB node 7 free: 2914 MB There is 16 GB of MC-DRAM and 4 GB per NUMA node. Is there any other process running on the machine? Why does "free memory" show only 3 GB per node? This is EXACTLY why I am asking this question. Nothing is running on the node - and I do not see full 4 GiB of available MCDRAM memory. This might be reason why I do not see what is expected. I understand that I can reduce the array size, but I am trying to reproduce your results. I am still unclear on if you run "no mpi" version, how come that SNC-4 mode is giving you advantage? All arrays are allocated in the same quadrant. Requests from the other quadrants will go to this one and create congestion. Vitali If you want to test Stream triad bandwidth to MC-DRAM in Quad/Flat mode, you can use even smaller arrays of few 100 MBs. -Thanks, -Sundaram -----Original Message----- From: Vitali A. Morozov [mailto:[email protected]] Sent: Wednesday, April 13, 2016 2:32 PM To: Raman, Karthik <karthik.raman(a)intel.com<mailto:[email protected]>>; Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com<mailto:[email protected]>>; Harms, Kevin <harms(a)alcf.anl.gov<mailto:[email protected]>>; Morozov, Vitali A. <morozov(a)alcf.anl.gov<mailto:[email protected]>>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov<mailto:[email protected]>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>>; Fromkin, Russ <russ.fromkin(a)intel.com<mailto:[email protected]>>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov<mailto:[email protected]>> Subject: Re: [JLSE-Intel-NDA] KNL Status ok, that is fine. Now the next issue I have is that for static arrays, all three arrays must land up on one of the numa nodes - the node that is attached to a master thread. You advice to use N = 134217728, that gives exactly 3 GiB of memory footprint for 3 arrays of triad. According to "numactl -H", I do not have 3 GiB of available memory on any of the MCDRAM numa nodes: numactl -H |grep free node 0 free: 46336 MB node 1 free: 46923 MB node 2 free: 46970 MB node 3 free: 46961 MB node 4 free: 2907 MB node 5 free: 2926 MB node 6 free: 2922 MB node 7 free: 2914 MB Therefore, at least one array will not be placed to the same numa node. Is this ok? On 04/13/2016 04:17 PM, Raman, Karthik wrote: Yes we run pure OMP code too. The problem with KMP_AFFINITY with SNC4 mode (currently) is that it may not equally balance the threads per NUMA domain even with scatter or balanced, you can check that by mapping the "verbose" output (e.g. KMP_AFFINITY=scatter,verbose) with "numactl -H" output to see if the OS thread to logical CPU mapping is balanced across all NUMA domains With All2All/Quad this should not be a problem -----Original Message----- From: Vitali A. Morozov [mailto:[email protected]] Sent: Wednesday, April 13, 2016 1:53 PM To: Raman, Karthik <karthik.raman(a)intel.com<mailto:[email protected]>>; Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com<mailto:[email protected]>>; Harms, Kevin <harms(a)alcf.anl.gov<mailto:[email protected]>>; Morozov, Vitali A. <morozov(a)alcf.anl.gov<mailto:[email protected]>>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov<mailto:[email protected]>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>>; Fromkin, Russ <russ.fromkin(a)intel.com<mailto:[email protected]>>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov<mailto:[email protected]>> Subject: Re: [JLSE-Intel-NDA] KNL Status Before doing that - let us check if we are running similar benchmark. I am running 1 MPI process (in fact no mpi at all) and 64 OpenMP threads version. The threads can be placed with "balanced" or "scattered" policy. I am using "scattered". Is this ok so far? Vitali On 04/13/2016 03:48 PM, Raman, Karthik wrote: This still does not look right, and the reason could be that the threads are not balanced across all the NUMA domains (ideally you want 16 threads per quadrant) Before going to SNC4, can you please boot in "Quad" mode. If you look in the BIOS Cluster mode options there is another mode called "Quad" which is different from SNC4, only 2 NUMA domains (DDR, MCDRAM). Please run similar to All2All mode -----Original Message----- From: Vitali A. Morozov [mailto:[email protected]] Sent: Wednesday, April 13, 2016 12:56 PM To: Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com<mailto:[email protected]>>; Harms, Kevin <harms(a)alcf.anl.gov<mailto:[email protected]>>; Raman, Karthik <karthik.raman(a)intel.com<mailto:[email protected]>>; Morozov, Vitali A. <morozov(a)alcf.anl.gov<mailto:[email protected]>>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov<mailto:[email protected]>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>>; Fromkin, Russ <russ.fromkin(a)intel.com<mailto:[email protected]>>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov<mailto:[email protected]>> Subject: Re: [JLSE-Intel-NDA] KNL Status ok, I am now in SNC-4/Flat. Same binary. OMP_NUM_THREADS=64 KMP_AFFINITY=scatter numactl -m 4,5,6,7 ./stream |tee running.dat ------------------------------------------------------------- STREAM version $Revision: 5.9 $ ------------------------------------------------------------- This system uses 8 bytes per DOUBLE PRECISION word. ------------------------------------------------------------- Array size = 134217728, Offset = 0 Total memory required = 3072.0 MB. Each test is run 100 times, but only the *best* time for each is used. ------------------------------------------------------------- Number of Threads requested = 64 ------------------------------------------------------------- <skipped> ------------------------------------------------------------- Function Rate (MB/s) Avg time Min time Max time Copy: 320997.8352 0.0068 0.0067 0.0071 Scale: 319994.2893 0.0068 0.0067 0.0073 Add: 306055.0205 0.0106 0.0105 0.0107 Triad: 306055.0205 0.0106 0.0105 0.0109 Store: 247056.8669 0.0044 0.0043 0.0051 ------------------------------------------------------------- Solution Validates ------------------------------------------------------------- 306055 MB/s Vitali On 04/13/2016 02:01 PM, Chinthamani, Sundaram wrote: 440 GB/s (base 10). To be precise, 440 bytes / ns (nano-second). We always report memory bandwidth in GB/s (base 10) within Intel. -Thanks, -Sundaram -----Original Message----- From: Harms, Kevin [mailto:[email protected]] Sent: Wednesday, April 13, 2016 11:52 AM To: Raman, Karthik <karthik.raman(a)intel.com<mailto:[email protected]>>; Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com<mailto:[email protected]>>; Morozov, Vitali A. <morozov(a)alcf.anl.gov<mailto:[email protected]>>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov<mailto:[email protected]>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>>; Fromkin, Russ <russ.fromkin(a)intel.com<mailto:[email protected]>>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov<mailto:[email protected]>> Subject: Re: [JLSE-Intel-NDA] KNL Status Just to be clear, memory is always assumed to be measured using base 2. It's only the storage field that uses base 10. So we assume if you say 440 GB/s you mean GiB, or 450,560 MB/s. kevin And just some fun with units http://sites.utexas.edu/jdm4372/2013/01/05/counting-binary-vs-dec i m a l -powers-in-the-stream-benchmark/ From: Chinthamani, Sundaram Sent: Wednesday, April 13, 2016 11:20 AM To: Vitali A. Morozov <morozov(a)anl.gov<mailto:[email protected]>>; Raman, Karthik <karthik.raman(a)intel.com<mailto:[email protected]>>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov<mailto:[email protected]>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>>; Fromkin, Russ <russ.fromkin(a)intel.com<mailto:[email protected]>>; Harms, Kevin <harms(a)alcf.anl.gov<mailto:[email protected]>>; Morozov, Vitali A. <morozov(a)alcf.anl.gov<mailto:[email protected]>>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov<mailto:[email protected]>>; Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com<mailto:[email protected]>> Subject: RE: [JLSE-Intel-NDA] KNL Status 440 GB/s stream triad for MC-DRAM and 80 GB/s is our internal bin3 measurement. You should be able to get closer these numbers, if you run the benchmark in Quad/Flat or SNC-4/Flat mode. -Thanks, -Sundaram From: Vitali A. Morozov [mailto:[email protected]] Sent: Wednesday, April 13, 2016 11:11 AM To: Raman, Karthik <karthik.raman(a)intel.com<mailto:[email protected]>>; Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com<mailto:[email protected]>>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov<mailto:[email protected]>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>>; Fromkin, Russ <russ.fromkin(a)intel.com<mailto:[email protected]>>; Harms, Kevin <harms(a)alcf.anl.gov<mailto:[email protected]>>; Morozov, Vitali A. <morozov(a)alcf.anl.gov<mailto:[email protected]>>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov<mailto:[email protected]>> Subject: Re: [JLSE-Intel-NDA] KNL Status I see - I am treating 418122 MB/s as 418122/1024 GB/s (as this is memory unit). Anyway - great we are on the same page here. I am in All2All/Flat mode. You seem to suggest that I must be on SNC-4/Flat to see 440000 MB/s? Thanks! Vitali On 04/13/2016 01:06 PM, Raman, Karthik wrote: Ok I read this as 418 GB/s J And ok, just realized you are running in All2All mode, so yes this looks good. For Quad/Flat you should see ~440 GB/s with the same run. From: Vitali A. Morozov [mailto:[email protected]] Sent: Wednesday, April 13, 2016 11:02 AM To: Raman, Karthik <karthik.raman(a)intel.com<mailto:[email protected]>> <mailto:[email protected]>; Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com<mailto:[email protected]>> <mailto:[email protected]>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]> Cc: Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>> <mailto:[email protected]>; Fromkin, Russ <russ.fromkin(a)intel.com<mailto:[email protected]>> <mailto:[email protected]>; Harms, Kevin <harms(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]>; Morozov, Vitali A. <morozov(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] KNL Status Hi Karthik, Repeating the same options you have: mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp -mcmodel medium -shared-intel -opt-streaming-stores always stream.c -o stream ------------------------------------------------------------- STREAM version $Revision: 5.9 $ ------------------------------------------------------------- This system uses 8 bytes per DOUBLE PRECISION word. ------------------------------------------------------------- Array size = 134217728, Offset = 0 Total memory required = 3072.0 MB. Each test is run 100 times, but only the *best* time for each is used. ------------------------------------------------------------- Number of Threads requested = 64 ------------------------------------------------------------- < skiped > Function Rate (MB/s) Avg time Min time Max time Copy: 362331.5200 0.0060 0.0059 0.0061 Scale: 359281.9806 0.0060 0.0060 0.0062 Add: 418122.7024 0.0078 0.0077 0.0081 Triad: 418122.7024 0.0078 0.0077 0.0081 Store: 224685.6729 0.0048 0.0048 0.0049 408GB/s Vitali On 04/13/2016 12:15 PM, Raman, Karthik wrote: 2. Effective MCDRAM bandwidth is around 406-408 GB/s. Larger values are possible for more high core count bins. For bin3 , you should be able to get Triad ~440 GB/s. Please check if you use the following Intel compiler flags -mcmodel medium -shared-intel -O3 -xMIC-AVX512 -DSTREAM_ARRAY_SIZE=134217728 -DOFFSET=0 -DNTIMES=100 -openmp -opt-streaming-stores always The array size is just allocating 1GB per array. From: Vitali A. Morozov [mailto:[email protected]] Sent: Wednesday, April 13, 2016 10:01 AM To: Raman, Karthik <karthik.raman(a)intel.com<mailto:[email protected]>> <mailto:[email protected]>; Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com<mailto:[email protected]>> <mailto:[email protected]>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]> Cc: Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>> <mailto:[email protected]>; Fromkin, Russ <russ.fromkin(a)intel.com<mailto:[email protected]>> <mailto:[email protected]>; Harms, Kevin <harms(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]>; Morozov, Vitali A. <morozov(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] KNL Status Hello, so, this is my understanding that on bin3 hardware (64 cores, 6 channel DDR4-2133) 1. Effective DDR bandwidth is 80 GB/s (around 78% of peak). 2. Effective MCDRAM bandwidth is around 406-408 GB/s. Larger values are possible for more high core count bins. 3. Alignment is not important for STREAM triad benchmark. 4. For both static and dynamic variables, numactl -m [0|1] should be used to direct memory placement. 5. For explicit MCDRAM allocation in flat mode, hbw_malloc.h API should be use. Please correct if I am wrong... Thanks a lot for you help! Vitali On 04/12/2016 11:07 AM, Vitali A. Morozov wrote: Hi Karthik, I was also wandering how the variable could be influencing the allocation if, in principle, I do not need to link against memkind in my case. Anyway, sorry for confusion! I think I did that before - see my first message. I was using "numactl --membind=1 ./stream" , which I believe is equivalent to "numactl -m 1 ./stream" that you suggest. Here is the result of "numactl -H": [morozov@knl00 512_3_C64_MPI16_TH08_flat]$ numactl -H available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 node 0 size: 196511 MB node 0 free: 186888 MB node 1 cpus: node 1 size: 16384 MB node 1 free: 11731 MB node distances: node 0 1 0: 10 31 1: 31 10 Sounds like all as expected. [morozov@knl00 STREAM]$ OMP_NUM_THREADS=64 KMP_AFFINITY=scatter numactl -m 1 ./stream Triad: 418262.7498 0.0058 0.0057 0.0060 For a series of runs, I see this: # MB N GB/s 228.88 10000000 408867.98 457.76 20000000 405818.57 686.65 30000000 414309.08 915.53 40000000 415063.59 1144.41 50000000 415792.22 1373.29 60000000 415449.01 1602.17 70000000 414106.18 1831.05 80000000 416846.82 2059.94 90000000 414992.29 2288.82 100000000 417898.11 Vitali On 04/12/2016 10:32 AM, Raman, Karthik wrote: Hi Vitali, 1. The MEMKIND_HBW_NODES is *only* when you are allocating using hbwalloc (+ MCDRAM flat) in your application and when memkind is unable to find the high bandwidth nodes (pmtt not populated) 2. With the below static allocations, in MCDRAM Flat mode, you can simply use “numactl –m 1 ./stream” with propery affinity settings to get peak out of MCDRAM. a. Check “numactl –H” to find the NUMA domains in your node. If booted in All2All/FLAT or Quad/Flat mode it should show 2 NUMA domains , one for DDR (96 GB) and other for MCDRAM (16GB) Thanks Karthik From: Vitali A. Morozov [mailto:[email protected]] Sent: Tuesday, April 12, 2016 7:27 AM To: Raman, Karthik <karthik.raman(a)intel.com<mailto:[email protected]>> <mailto:[email protected]>; Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com<mailto:[email protected]>> <mailto:[email protected]>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]> Cc: Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>> <mailto:[email protected]>; Fromkin, Russ <russ.fromkin(a)intel.com<mailto:[email protected]>> <mailto:[email protected]>; Harms, Kevin <harms(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]>; Morozov, Vitali A. <morozov(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] KNL Status Hi Karthik, The node is in All2All mode for now. I made the following changes. The original code is using static allocation therefore, I follow the advice on 2MB static array alignment: [morozov@thing00 STREAM]$ head -n 108 < stream.c | tail -n 4 static double a[N+OFFSET] __attribute__((aligned(2097152))), b[N+OFFSET] __attribute__((aligned(2097152))), c[N+OFFSET] __attribute__((aligned(2097152))), d[N+OFFSET] __attribute__((aligned(2097152))); [morozov@knl00 STREAM]$ make mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp -mcmodel medium -shared-intel -fno-alias stream.c -o stream [morozov@knl00 STREAM]$ MEMKIND_HBW_NODES=1 OMP_NUM_THREADS=64 KMP_AFFINITY=scatter ./stream ..... Triad: 80251.3621 0.0300 0.0299 0.0305 Apparently, the "MEMKIND_HBW_NODES=1" is ignored. Thanks! Vitali On 04/12/2016 01:51 AM, Raman, Karthik wrote: Also could you please modify the memory allocations in STREAM to align at 2MB boundary. For example: static allocations: static STREAM_TYPE a[STREAM_ARRAY_SIZE+OFFSET] __attribute__((aligned(2097152))) dynamic allocations: a = (STREAM_TYPE *)_mm_malloc(sizeof(STREAM_TYPE)*(STREAM_ARRAY_SIZE+OFFSET),20971 5 2 ) ; memkind/hbwalloc: (STREAM_TYPE *)hbw_posix_memalign((void **)&a, 2097152, sizeof(STREAM_TYPE)*(STREAM_ARRAY_SIZE+OFFSET)) For memkind/hbwmalloc to work out of the box, you need to have the PMTT tables populated so that it can identify the high bandwidth nodes. Alternatively you can set the MEMKIND_HBW_NODES environment variable to the high bandwidth node (MCDRAM). e.g. All2All/Flat or Quad/Flat : export MEMKIND_HBW_NODES=1 ; ./app_with_hbwmalloc SNC4/Flat: export MEMKIND_HBW_NODES=4,5,6,7 ; ./app_with_hbw_malloc Thanks Karthik From: Chinthamani, Sundaram Sent: Monday, April 11, 2016 8:15 PM To: Vitali Morozov <morozov(a)anl.gov<mailto:[email protected]>> <mailto:[email protected]>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]> Cc: Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>> <mailto:[email protected]>; Fromkin, Russ <russ.fromkin(a)intel.com<mailto:[email protected]>> <mailto:[email protected]>; Raman, Karthik <karthik.raman(a)intel.com<mailto:[email protected]>> <mailto:[email protected]>; Harms, Kevin <harms(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]>; Morozov, Vitali A. <morozov(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov<mailto:[email protected]>> <mailto:[email protected]>; Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com<mailto:[email protected]>> <mailto:[email protected]> Subject: RE: [JLSE-Intel-NDA] KNL Status You should use numactl –m 4,5,6,7 to get the memory bandwidth out of MC-DRAM. What kind of part is it? Is it bin1 or bin3? Even in All2All mode, we have measured up to 470 GB/s of stream triad bandwidth. Quad mode should get 490 GB/s. DDR-stream triad bandwidth on Bin1-part should be close to 90 GB/s. -Sundaram From: Vitali Morozov [mailto:[email protected]] Sent: Monday, April 11, 2016 6:59 PM To: Kumaran, Kalyan <kumaran(a)alcf.anl.gov<mailto:[email protected]>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>>; Fromkin, Russ <russ.fromkin(a)intel.com<mailto:[email protected]>>; Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com<mailto:[email protected]>>; Raman, Karthik <karthik.raman(a)intel.com<mailto:[email protected]>>; Harms, Kevin <harms(a)alcf.anl.gov<mailto:[email protected]>>; Morozov, Vitali A. <morozov(a)alcf.anl.gov<mailto:[email protected]>>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov<mailto:[email protected]>> Subject: Re: [JLSE-Intel-NDA] KNL Status Avinash and others, here is what we have so far for Flat more, All2All. The results from memory were obtained by using default STREAM with static allocation. The results from MCDRAM were obtained by using modified STREAM with hbwmalloc.h interface. 1 MPI process, 64 threads, scatter affinity, no numactl is used. Most results are as expected: 80 GB/s from DDR memory, 420 GB/s from MCDRAM, the MCDRAM size is about 16 GB, the aggregate L2 cache size is 32 MB. Noticed inconsistencies for hbwmalloc benchmarks: 1. The result is not validated - will spend more time on that tomorrow. 2. hbw_malloc continues allocating data beyond the expected 16 GB up to 48 GB. This means it fails back to DDR if it cannot allocate in MCDRAM - not expected default behavior. It must fail if it does not have enough space. The problem shows up when I am running static allocation and use numactl --membind=1 to ask for MCDRAM-only allocation. The variation is very large and inconsistent with the result observed when using hbw_malloc API: # MB N GB/s numactl GB/s hbw_malloc 228.88 10000000 255555.46 421009.18 457.76 20000000 363930.93 416997.91 686.65 30000000 309415.87 427808.31 915.53 40000000 371691.30 423310.75 1144.41 50000000 187002.22 431032.35 1373.29 60000000 385954.23 427172.91 1602.17 70000000 388254.49 426823.59 1831.05 80000000 374491.43 428103.96 2059.94 90000000 379622.74 428152.02 2288.82 100000000 80868.34 430185.03 We also tried SNC-4 instead of All2All, but could not get more than 40 GB/s from MCDRAM even with hbw_malloc API. Raw data: # flat memory, running from memory (default), alltoall # MB N GB/s 2.29 100000 272062.96 4.58 200000 491040.47 6.87 300000 603979.78 9.16 400000 745654.04 11.44 500000 811800.77 13.73 600000 901462.35 16.02 700000 891953.26 18.31 800000 1019375.15 20.60 900000 984749.63 22.89 1000000 923516.48 25.18 1100000 758422.09 27.47 1200000 779328.74 29.75 1300000 513185.43 32.04 1400000 155550.35 34.33 1500000 103209.12 36.62 1600000 97435.74 38.91 1700000 83968.40 41.20 1800000 83692.35 43.49 1900000 80734.60 45.78 2000000 76317.89 68.66 3000000 81180.08 91.55 4000000 78889.73 114.44 5000000 79891.50 137.33 6000000 79954.96 160.22 7000000 79477.00 183.11 8000000 79145.59 205.99 9000000 79821.12 228.88 10000000 79519.15 228.88 10000000 79212.54 457.76 20000000 79692.27 686.65 30000000 79866.15 915.53 40000000 79828.15 1144.41 50000000 80214.91 1373.29 60000000 80226.84 1602.17 70000000 80178.77 1831.05 80000000 80177.06 2059.94 90000000 80109.79 2288.82 100000000 80175.94 # flat memory, running from MCDRAM, alltoall, hbwmalloc API # MB N GB/s 228.88 10000000 421009.18 457.76 20000000 416997.91 686.65 30000000 427808.31 915.53 40000000 423310.75 1144.41 50000000 431032.35 1373.29 60000000 427172.91 1602.17 70000000 426823.59 1831.05 80000000 428103.96 2059.94 90000000 428152.02 2288.82 100000000 430185.03 2343.75 102400000 425999.98 4687.50 204800000 427821.10 9155.27 400000000 429318.13 9375.00 409600000 428376.70 11444.09 500000000 428052.08 13732.91 600000000 266061.01 16021.73 700000000 200257.79 18310.55 800000000 168164.19 20599.37 900000000 150192.91 22888.18 1000000000 137862.07 25177.00 1100000000 129418.12 27465.82 1200000000 123019.53 29754.64 1300000000 118268.79 32043.46 1400000000 114343.33 34332.28 1500000000 110991.25 36621.09 1600000000 108538.36 38909.91 1700000000 106249.72 41198.73 1800000000 104455.07 43487.55 1900000000 102828.93 45776.37 2000000000 101348.56 48065.19 2100000000 100013.73 We appreciate any help on that. Best, Vitali On 04/11/2016 08:42 PM, Kumaran, Kalyan wrote: Avinash, thanks for the prompt reply. Kevin/Vitali can provide all the details. The short summary is we are unable to reproduce the STREAM results from IPM or DDR. It would be great if Karthik or Sundaram can advise us on how to run (which mode etc) or if they could log into the machine and make sure we have set up things correctly. Thanks, Kumar On Apr 11, 2016, at 8:07 PM, Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>> wrote: Copying Sundaram/Karthik from our side. Kumar, Can you send us more information on this? -Avinash From: Kumaran, Kalyan [mailto:[email protected]] Sent: Monday, April 11, 2016 5:32 PM To: Fromkin, Russ; Sodani, Avinash Cc: Harms, Kevin; Morozov, Vitali A. Subject: Fwd: [JLSE-Intel-NDA] KNL Status Russ, Avinash, we need help in understanding how to obtain STREAM results. Can anyone help? Thanks, Kumar Begin forwarded message: From: "Harms, Kevin" <harms(a)alcf.anl.gov<mailto:[email protected]>> Subject: [JLSE-Intel-NDA] KNL Status Date: April 11, 2016 at 5:42:15 PM CDT To: "intel-nda(a)jlse.anl.gov<mailto:[email protected]>" <intel-nda(a)jlse.anl.gov<mailto:[email protected]>> The knl00 node is currently in Flat mode and All2All model. With this setup we can get 420 GB/s from IPM memory on STREAM Triad when using hbw_malloc and friends which was confirmed by both Vitali and I. Changing the BIOS settings and warm rebooting seem to be problematic. Vitali and I were running earlier and could not get over 80 GB/s with any method even though memory capacity indicated flat mode. Using numactl --membind is also problematic. The STREAM results when using this to launch the application were erratic, so it is also suggested not to use that. That means you will need to modify your code to use dynamic memory and hbw_malloc to exploit the IPM. Will followup with more detailed results later. kevin _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda <flat_a2a_ddr.txt><flat_a2a_ipm.txt><flat_quad_ddr.txt><flat_quad_ipm.txt><flat_snc4_ddr.txt><flat_snc4_ipm.txt>

1 0

[JLSE-Intel-NDA] BIOS Updated
by Allen, Benjamin S. 18 Apr '16

18 Apr '16

BIOS and BMC has been updated to: BIOS 01.01.0104, ME 03.01.03.011.0, BMC 00.17.9699, FRUSDR 1.14. Release notes attached.

1 0