- Intel-nda - mailman3.cels.anl.gov

Re: [JLSE-Intel-NDA] MPI and process binding
by Harms, Kevin 22 Apr '16

22 Apr '16

I found this environment variable that prints out the process bound list. I_MPI_DEBUG=5 [harms@ftlogin1 tests]$ I_MPI_DEBUG=5 mpirun -n 2 ./mpi_test [0] MPI startup(): Multi-threaded optimized library [0] MPI startup(): shm data transfer mode [1] MPI startup(): shm data transfer mode [0] MPI startup(): Rank Pid Node name Pin cpu [0] MPI startup(): 0 29236 ftlogin1 {0,1,2,3,8,9,10,11} [0] MPI startup(): 1 29237 ftlogin1 {4,5,6,7,12,13,14,15} [0] MPI startup(): I_MPI_DEBUG=5 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_DIST=10 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_MAP=mlx4_0:0 [0] MPI startup(): I_MPI_INFO_NUMA_NODE_NUM=1 [0] MPI startup(): I_MPI_PIN_MAPPING=2:0 0,1 4 If Intel MPI pins the processes sanely on KNL, then coupling that with KMP_AFFINITY=balanced,granularity=thread should give you a good result. Here's an explanation of balanced: https://software.intel.com/en-us/node/522518 If Intel MPI doesn't know KNL and does something weird then you will have to try to understand KNL's tile/core numbering scheme which appears complex. kevin >KMP_AFFINITY only works for thread affinity though, no? If I have four MPI processes, each of the processes will assign threads to the same cores rather than cores in a different sub-NUMA cluster. I need to be able to specify process affinity > and my problem is that 1) numactl can only control a single process and 2) the usual >mpirun -bind-to and -map-by options don't really have enough levels of abstraction to match the KNL's hierarchy. For what it's worth, I tried >mpirun -bind-to numa -map-by numa ... and that resulted in an error. It's not even clear, to me at least, what to expect if you say 'numa' -- one could interpret it to mean a tile (that shares L2 cache) or a cluster (that shares MCDRAM). > >On Fri, Apr 22, 2016 at 9:46 AM, Harms, Kevin ><harms(a)alcf.anl.gov> wrote: > > > KMP_AFFINITY=balanced is supposed to do that but if it's not working, it might be that the MPI library does not have proper tuning for KNL yet. > >kevin > > > > >>I'm trying to test a code using hybrid MPI/OpenMP on the KNL node but haven't been able to successfully set process affinity. I'd like to run four processes in SNC-4 mode with each process being bound to a cluster. >> Does anyone know if this is possible? >> >>Thanks, >>Paul >> > > > > > >

1 0

Re: [JLSE-Intel-NDA] MCDRAM poster children
by Kumaran, Kalyan 22 Apr '16

22 Apr '16

Examples would be NEKbone (Scott Parker), HACCmk (Vitali Morozov) and MILC (James Osborn) - though I do not know if James has results on KNL. You can talk directly to them. I have ccd other users of our KNL incase they have results they would like to share. -Kumar On Apr 22, 2016, at 11:06 AM, David E. Martin <dem(a)alcf.anl.gov<mailto:[email protected]>> wrote: Kumar, Do we have anything to report? Intel is willing to waive NDA restrictions. -David -------- Forwarded Message -------- Subject: MCDRAM poster children Date: Fri, 22 Apr 2016 15:43:02 +0000 From: Newburn, Chris <chris.newburn(a)intel.com><mailto:[email protected]> To: Doug Doerfler <dwdoerf(a)lbl.gov><mailto:[email protected]>, Martin, David E. <dem(a)alcf.anl.gov><mailto:[email protected]> CC: Richard Gerber <ragerber(a)lbl.gov><mailto:[email protected]> Gentlemen, Do you have some poster-child apps for using MCDRAM, particularly images, video clips and perf improvement numbers that we can share at ISC? CJ

1 0

Re: [JLSE-Intel-NDA] MPI and process binding
by Paul Romano 22 Apr '16

22 Apr '16

KMP_AFFINITY only works for thread affinity though, no? If I have four MPI processes, each of the processes will assign threads to the same cores rather than cores in a different sub-NUMA cluster. I need to be able to specify process affinity and my problem is that 1) numactl can only control a single process and 2) the usual *mpirun -bind-to* and *-map-by* options don't really have enough levels of abstraction to match the KNL's hierarchy. For what it's worth, I tried *mpirun -bind-to numa -map-by numa ...* and that resulted in an error. It's not even clear, to me at least, what to expect if you say 'numa' -- one could interpret it to mean a tile (that shares L2 cache) or a cluster (that shares MCDRAM). On Fri, Apr 22, 2016 at 9:46 AM, Harms, Kevin <harms(a)alcf.anl.gov> wrote: > > KMP_AFFINITY=balanced is supposed to do that but if it's not working, it > might be that the MPI library does not have proper tuning for KNL yet. > > kevin > > > > > >I'm trying to test a code using hybrid MPI/OpenMP on the KNL node but > haven't been able to successfully set process affinity. I'd like to run > four processes in SNC-4 mode with each process being bound to a cluster. > > Does anyone know if this is possible? > > > >Thanks, > >Paul > > >

1 0

Re: [JLSE-Intel-NDA] MPI and process binding
by Harms, Kevin 22 Apr '16

22 Apr '16

KMP_AFFINITY=balanced is supposed to do that but if it's not working, it might be that the MPI library does not have proper tuning for KNL yet. kevin >I'm trying to test a code using hybrid MPI/OpenMP on the KNL node but haven't been able to successfully set process affinity. I'd like to run four processes in SNC-4 mode with each process being bound to a cluster. > Does anyone know if this is possible? > >Thanks, >Paul >

1 0

[JLSE-Intel-NDA] MPI and process binding
by Paul Romano 22 Apr '16

22 Apr '16

I'm trying to test a code using hybrid MPI/OpenMP on the KNL node but haven't been able to successfully set process affinity. I'd like to run four processes in SNC-4 mode with each process being bound to a cluster. Does anyone know if this is possible? Thanks, Paul

1 0

Re: [JLSE-Intel-NDA] KNL Status
by Scott Parker 21 Apr '16

21 Apr '16

Likely, with 64 threads I see Function Best Rate MB/s Avg time Min time Max time Copy: 324490.2 0.006682 0.006618 0.006773 Scale: 324350.0 0.006676 0.006621 0.006796 Add: 311022.1 0.010390 0.010357 0.010418 Triad: 310300.6 0.010407 0.010381 0.010460 On 4/21/16 10:32 AM, Vitali A Morozov wrote: > Then, the only difference on what you are doing and Kevin and I were > doing before is 128 threads versus 64 and 256 threads. I do not think > we have previously run 128 threads. > > Vitali > > On 4/21/16 10:29 AM, Scott Parker wrote: >> >> It's not the compiler, I see no significant difference in SNC-4 mode >> between Intel 16 and Intel 17 Beta. >> >> >icc -V >> Intel(R) C Intel(R) 64 Compiler for applications running on Intel(R) >> 64, Version 16.0.0.109 Build 20150815 >> Copyright (C) 1985-2015 Intel Corporation. All rights reserved. >> >> >icc -O3 -fopenmp -DSTREAM_ARRAY_SIZE=134217728 -xMIC-AVX512 -mcmodel >> medium -shared-intel stream.c -o stream_c.exe >> >> >OMP_NUM_THREADS=128 numactl -m 4,5,6,7 ./stream_c.exe >> Function Best Rate MB/s Avg time Min time Max time >> Copy: 390734.0 0.005547 0.005496 0.005631 >> Scale: 411250.1 0.005309 0.005222 0.005569 >> Add: 449326.5 0.007190 0.007169 0.007220 >> Triad: 440248.9 0.007343 0.007317 0.007391 >> >> On 4/21/16 10:12 AM, Vitali A Morozov wrote: >>> Yeah - as Ronald Green has figured out, the beta version of a >>> compiler (17) does make something differently compared to production >>> version 16. For some reason, it only matters in SNC-4 mode and not >>> in All2All mode. >>> >>> I am curious what was changed (or added) in version 17... >>> >>> Vitali >>> >>> >>> >>> On 4/21/16 9:45 AM, Kumaran, Kalyan wrote: >>>> Looks like Scott Parker gets better results: >>>> >>>> Adding another data point, I ran in SNC-4 mode with the >>>> configuration below and observed a consistent BW around 440 GB/s >>>> with the best run producing 442 GB/s. In this case 128 threads gave >>>> the best results. >>>> >>>> I've noticed that the STREAM results sensitivity to build and run >>>> parameters seems to vary depending on the memory configuration >>>> in flat-all2all I saw noticeably better results with memory aligned >>>> on 2MB boundaries and 64 threads, in SNC-4 mode the alignment >>>> doesn't seem to matter and 128 threads is producing the best results. >>>> >>>> >>>> Mode: flat & SNC-4 >>>> Compiler: Intel 17 Beta >>>> Compile Options: icc -O3 -fopenmp -DSTREAM_ARRAY_SIZE=134217728 >>>> -xMIC-AVX512 -mcmodel medium -shared-intel >>>> Run Command: OMP_NUM_THREADS=128 numactl -m 4,5,6,7 ./stream_mod >>>> >>>> Results: >>>> Function Best Rate MB/s Avg time Min time Max time >>>> Copy: 390378.3 0.005524 0.005501 0.005546 >>>> Scale: 412738.8 0.005237 0.005203 0.005278 >>>> Add: 448759.4 0.007199 0.007178 0.007218 >>>> Triad: 441919.3 0.007355 0.007289 0.007548 >>>> >>>> >>>> >>>>> On Apr 20, 2016, at 7:39 PM, Vitali A Morozov <morozov(a)anl.gov >>>>> <mailto:[email protected]>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> a quick update on STREAM with the latest beta compiler >>>>> >>>>> [morozov@knl00 STREAM]$ icc -V >>>>> Intel(R) C Intel(R) 64 Compiler for applications running on >>>>> Intel(R) 64, Version 17.0.0.042 Beta Build 20160315 >>>>> >>>>> compiler options >>>>> mpiicc -O3 -xMIC-AVX512 -qopt-report=5 -Wall -g -fopenmp -mcmodel >>>>> medium -shared-intel -qopt-streaming-stores always -restrict >>>>> stream.c -o stream >>>>> >>>>> Running in flat mode all2all >>>>> >>>>> morozov@knl00 STREAM]$ export OMP_NUM_THREADS=256 >>>>> [morozov@knl00 STREAM]$ numactl -m 1 ./stream >>>>> ------------------------------------------------------------- >>>>> STREAM version $Revision: 5.9 $ >>>>> ------------------------------------------------------------- >>>>> This system uses 8 bytes per DOUBLE PRECISION word. >>>>> ------------------------------------------------------------- >>>>> Array size = 134217728, Offset = 0 >>>>> Total memory required = 3072.0 MB. >>>>> Each test is run 500 times, but only >>>>> the *best*time for each is used. >>>>> ------------------------------------------------------------- >>>>> Number of Threads requested = 256 >>>>> ------------------------------------------------------------- >>>>> <skipped> >>>>> ------------------------------------------------------------- >>>>> Your clock granularity/precision appears to be 1 microseconds. >>>>> Each test below will take on the order of 6356 microseconds. >>>>> (= 6356 clock ticks) >>>>> Increase the size of the arrays if this shows that >>>>> you are not getting at least 20 clock ticks per test. >>>>> ------------------------------------------------------------- >>>>> WARNING -- The above is only a rough guideline. >>>>> For best results, please be sure you know the >>>>> precision of your system timer. >>>>> ------------------------------------------------------------- >>>>> Function Rate (MB/s) Avg time Min time Max time >>>>> Copy: 330891.5637 0.0066 0.0065 0.0078 >>>>> Scale: 330127.5200 0.0066 0.0065 0.0069 >>>>> Add: 383622.4448 0.0085 0.0084 0.0093 >>>>> Triad: 383252.4574 0.0085 0.0084 0.0098 >>>>> ------------------------------------------------------------- >>>>> Solution Validates >>>>> ------------------------------------------------------------- >>>>> >>>>> Same with 64 threads scatter mode: >>>>> >>>>> morozov@knl00 STREAM]$ export OMP_NUM_THREADS=64 >>>>> [morozov@knl00 STREAM]$ KMP_AFFINITY=scatter >>>>> [morozov@knl00 STREAM]$ numactl -m 1 ./stream >>>>> ------------------------------------------------------------- >>>>> STREAM version $Revision: 5.9 $ >>>>> ------------------------------------------------------------- >>>>> This system uses 8 bytes per DOUBLE PRECISION word. >>>>> ------------------------------------------------------------- >>>>> Array size = 134217728, Offset = 0 >>>>> Total memory required = 3072.0 MB. >>>>> Each test is run 500 times, but only >>>>> the *best*time for each is used. >>>>> ------------------------------------------------------------- >>>>> Number of Threads requested = 64 >>>>> ------------------------------------------------------------- >>>>> <skipped> >>>>> ------------------------------------------------------------- >>>>> Your clock granularity/precision appears to be 1 microseconds. >>>>> Each test below will take on the order of 6056 microseconds. >>>>> (= 6056 clock ticks) >>>>> Increase the size of the arrays if this shows that >>>>> you are not getting at least 20 clock ticks per test. >>>>> ------------------------------------------------------------- >>>>> WARNING -- The above is only a rough guideline. >>>>> For best results, please be sure you know the >>>>> precision of your system timer. >>>>> ------------------------------------------------------------- >>>>> Function Rate (MB/s) Avg time Min time Max time >>>>> Copy: 353486.8826 0.0061 0.0061 0.0067 >>>>> Scale: 353265.0608 0.0061 0.0061 0.0066 >>>>> Add: 423961.3055 0.0077 0.0076 0.0079 >>>>> Triad: 424907.9750 0.0076 0.0076 0.0084 >>>>> ------------------------------------------------------------- >>>>> Solution Validates >>>>> ------------------------------------------------------------- >>>>> >>>>> Vitali >>>>> >>>>> >>>>> >>>>> >>>>> On 4/18/16 12:43 PM, Kumaran, Kalyan wrote: >>>>>> FYI. Karthik will get on the machine to debug. >>>>>> >>>>>> Sent from my iPhone >>>>>> >>>>>> Begin forwarded message: >>>>>> >>>>>>> *From:* "Chinthamani, Sundaram" <sundaram.chinthamani(a)intel.com> >>>>>>> *Date:* April 18, 2016 at 2:05:26 PM CDT >>>>>>> *To:* "Harms, Kevin" <harms(a)alcf.anl.gov>, "Allen, Benjamin S." >>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>> *Cc:* "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov>, "Kaana >>>>>>> Balakrishna, Abhishek" <abhishek.kaana.balakrishna(a)intel.com>, >>>>>>> "Morozov, Vitali A." <morozov(a)alcf.anl.gov>, "Raman, Karthik" >>>>>>> <karthik.raman(a)intel.com <mailto:[email protected]>>, >>>>>>> "Inkley, Benson" <benson.inkley(a)intel.com >>>>>>> <mailto:[email protected]>>, "Sodani, Avinash" >>>>>>> <avinash.sodani(a)intel.com <mailto:[email protected]>>, >>>>>>> "Fromkin, Russ" <russ.fromkin(a)intel.com >>>>>>> <mailto:[email protected]>>, "Chinthamani, Sundaram" >>>>>>> <sundaram.chinthamani(a)intel.com> >>>>>>> *Subject:* *RE: [JLSE-Intel-NDA] KNL Status* >>>>>>> >>>>>>> I would expect Quad mode / SNC-4 mode to provide higher >>>>>>> bandwidth compared to All2All mode. We haven't seen this in >>>>>>> our internal measurements. >>>>>>> >>>>>>> -Thanks, >>>>>>> -Sundaram >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Harms, Kevin [mailto:[email protected]] >>>>>>> Sent: Monday, April 18, 2016 12:00 PM >>>>>>> To: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >>>>>>> Cc: Kumaran, Kalyan <kumaran(a)alcf.anl.gov>; Kaana Balakrishna, >>>>>>> Abhishek <abhishek.kaana.balakrishna(a)intel.com>; Morozov, Vitali >>>>>>> A. <morozov(a)alcf.anl.gov>; Chinthamani, Sundaram >>>>>>> <sundaram.chinthamani(a)intel.com>; Raman, Karthik >>>>>>> <karthik.raman(a)intel.com <mailto:[email protected]>>; >>>>>>> Inkley, Benson <benson.inkley(a)intel.com >>>>>>> <mailto:[email protected]>>; Sodani, Avinash >>>>>>> <avinash.sodani(a)intel.com <mailto:[email protected]>>; >>>>>>> Fromkin, Russ <russ.fromkin(a)intel.com >>>>>>> <mailto:[email protected]>> >>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>> >>>>>>> >>>>>>> I tested cached mode with all2all, snc4 and quad models. Cached >>>>>>> all2all gave 275,000 MB/s. Does that look reasonable? SNC4 and >>>>>>> quad mode gave 225,000 MB/s or so. >>>>>>> >>>>>>> kevin >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> The following versions are now installed: BIOS 01.01.0104, ME >>>>>>>> 03.01.03.011.0, BMC 00.17.9699, FRUSDR 1.14. >>>>>>>> >>>>>>>> Ben >>>>>>>> >>>>>>>>> On Apr 18, 2016, at 1:27 PM, Harms, Kevin <harms(a)alcf.anl.gov> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> Ben completed the BIOS upgrade, but we do not see any >>>>>>>>> difference in performance. I have attached results from runs >>>>>>>>> done under three configurations: flat+all2all, flat+quad, >>>>>>>>> flat+snc4. >>>>>>>>> >>>>>>>>> flat+all2all = 427046.0546 MB/s >>>>>>>>> flat+quad = 327253.8882 MB/s >>>>>>>>> flat+snc4 = 308555.9588 MB/s >>>>>>>>> >>>>>>>>> I compiled the CORAL stream.c with the following options: >>>>>>>>> icc -mcmodel medium -shared-intel -O3 -xMIC-AVX512 >>>>>>>>> -DN=50000000 -DOFFSET=0 -DNTIMES=100 -openmp >>>>>>>>> -opt-streaming-stores always stream.c -o stream >>>>>>>>> >>>>>>>>> I added the alignment directives to the arrays. The attached >>>>>>>>> files show the NUMA domains and numactl commands used and run >>>>>>>>> parameters. >>>>>>>>> >>>>>>>>> Have I missed a parameter? >>>>>>>>> >>>>>>>>> kevin >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Suggest a time Friday afternoon. We are in central time zone. >>>>>>>>>> We can get you access to this B0 too. >>>>>>>>>> >>>>>>>>>> -Kumar >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On Apr 13, 2016, at 5:06 PM, Kaana Balakrishna, Abhishek >>>>>>>>>>> <abhishek.kaana.balakrishna(a)intel.com> wrote: >>>>>>>>>>> >>>>>>>>>>> + Benson >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> Abhishek K Balakrishna (KB) >>>>>>>>>>> Xeon Phiï¿½ Hardware Platform Applications Engineer, >>>>>>>>>>> Technical Computing Group (TCG) >>>>>>>>>>> Intel Corporation >>>>>>>>>>> Ph - 512 466 9886 >>>>>>>>>>> www.intel.com/xeonphi <http://www.intel.com/xeonphi> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>> Sent: Wednesday, April 13, 2016 4:58 PM >>>>>>>>>>> To: Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com>; >>>>>>>>>>> Raman, Karthik <karthik.raman(a)intel.com>; Harms, Kevin >>>>>>>>>>> <harms(a)alcf.anl.gov>; Morozov, Vitali A. >>>>>>>>>>> <morozov(a)alcf.anl.gov>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov> >>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; Fromkin, >>>>>>>>>>> Russ <russ.fromkin(a)intel.com>; Allen, Benjamin S. >>>>>>>>>>> <bsallen(a)alcf.anl.gov>; Kaana Balakrishna, Abhishek >>>>>>>>>>> <abhishek.kaana.balakrishna(a)intel.com> >>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>> >>>>>>>>>>> Sure! >>>>>>>>>>> >>>>>>>>>>> Vitali >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 04/13/2016 04:54 PM, Chinthamani, Sundaram wrote: >>>>>>>>>>>> Would it be possible to have a meeting to go over your >>>>>>>>>>>> questions rather than exchanging E-mails back and forth? >>>>>>>>>>>> >>>>>>>>>>>> + KB from Intel side. He can help set-up the meeting >>>>>>>>>>>> between Labs and Intel. >>>>>>>>>>>> >>>>>>>>>>>> -Thanks, >>>>>>>>>>>> -Sundaram >>>>>>>>>>>> >>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>> Sent: Wednesday, April 13, 2016 2:50 PM >>>>>>>>>>>> To: Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com>; >>>>>>>>>>>> Raman, >>>>>>>>>>>> Karthik <karthik.raman(a)intel.com>; Harms, Kevin >>>>>>>>>>>> <harms(a)alcf.anl.gov>; >>>>>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov>; Kumaran, Kalyan >>>>>>>>>>>> <kumaran(a)alcf.anl.gov> >>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; Fromkin, Russ >>>>>>>>>>>> <russ.fromkin(a)intel.com>; Allen, Benjamin S. >>>>>>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>> >>>>>>>>>>>> On 04/13/2016 04:42 PM, Chinthamani, Sundaram wrote: >>>>>>>>>>>>> node 4 free: 2907 MB >>>>>>>>>>>>> node 5 free: 2926 MB >>>>>>>>>>>>> node 6 free: 2922 MB >>>>>>>>>>>>> node 7 free: 2914 MB >>>>>>>>>>>>> >>>>>>>>>>>>> There is 16 GB of MC-DRAM and 4 GB per NUMA node. Is >>>>>>>>>>>>> there any other process running on the machine? Why >>>>>>>>>>>>> does "free memory" show only 3 GB per node? >>>>>>>>>>>> This is EXACTLY why I am asking this question. Nothing is >>>>>>>>>>>> running on the node - and I do not see full 4 GiB of >>>>>>>>>>>> available MCDRAM memory. This might be reason why I do not >>>>>>>>>>>> see what is expected. >>>>>>>>>>>> >>>>>>>>>>>> I understand that I can reduce the array size, but I am >>>>>>>>>>>> trying to reproduce your results. >>>>>>>>>>>> >>>>>>>>>>>> I am still unclear on if you run "no mpi" version, how come >>>>>>>>>>>> that SNC-4 mode is giving you advantage? All arrays are >>>>>>>>>>>> allocated in the same quadrant. Requests from the other >>>>>>>>>>>> quadrants will go to this one and create congestion. >>>>>>>>>>>> >>>>>>>>>>>> Vitali >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> If you want to test Stream triad bandwidth to MC-DRAM in >>>>>>>>>>>>> Quad/Flat mode, you can use even smaller arrays of few 100 >>>>>>>>>>>>> MBs. >>>>>>>>>>>>> >>>>>>>>>>>>> -Thanks, >>>>>>>>>>>>> -Sundaram >>>>>>>>>>>>> >>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>>> Sent: Wednesday, April 13, 2016 2:32 PM >>>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com>; Chinthamani, >>>>>>>>>>>>> Sundaram >>>>>>>>>>>>> <sundaram.chinthamani(a)intel.com>; Harms, Kevin >>>>>>>>>>>>> <harms(a)alcf.anl.gov>; >>>>>>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov>; Kumaran, Kalyan >>>>>>>>>>>>> <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; Fromkin, Russ >>>>>>>>>>>>> <russ.fromkin(a)intel.com>; Allen, Benjamin S. >>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>> >>>>>>>>>>>>> ok, that is fine. >>>>>>>>>>>>> >>>>>>>>>>>>> Now the next issue I have is that for static arrays, all >>>>>>>>>>>>> three arrays must land up on one of the numa nodes - the >>>>>>>>>>>>> node that is attached to a master thread. >>>>>>>>>>>>> >>>>>>>>>>>>> You advice to use N = 134217728, that gives exactly 3 GiB >>>>>>>>>>>>> of memory footprint for 3 arrays of triad. >>>>>>>>>>>>> >>>>>>>>>>>>> According to "numactl -H", I do not have 3 GiB of >>>>>>>>>>>>> available memory on any of the MCDRAM numa nodes: >>>>>>>>>>>>> >>>>>>>>>>>>> numactl -H |grep free >>>>>>>>>>>>> node 0 free: 46336 MB >>>>>>>>>>>>> node 1 free: 46923 MB >>>>>>>>>>>>> node 2 free: 46970 MB >>>>>>>>>>>>> node 3 free: 46961 MB >>>>>>>>>>>>> node 4 free: 2907 MB >>>>>>>>>>>>> node 5 free: 2926 MB >>>>>>>>>>>>> node 6 free: 2922 MB >>>>>>>>>>>>> node 7 free: 2914 MB >>>>>>>>>>>>> >>>>>>>>>>>>> Therefore, at least one array will not be placed to the >>>>>>>>>>>>> same numa node. >>>>>>>>>>>>> Is this ok? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 04/13/2016 04:17 PM, Raman, Karthik wrote: >>>>>>>>>>>>>> Yes we run pure OMP code too. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The problem with KMP_AFFINITY with SNC4 mode (currently) >>>>>>>>>>>>>> is that it >>>>>>>>>>>>>> may not equally balance the threads per NUMA domain even with >>>>>>>>>>>>>> scatter or balanced, you can check that by mapping the >>>>>>>>>>>>>> "verbose" output (e.g. >>>>>>>>>>>>>> KMP_AFFINITY=scatter,verbose) with "numactl -H" output to >>>>>>>>>>>>>> see if the >>>>>>>>>>>>>> OS thread to logical CPU mapping is balanced across all >>>>>>>>>>>>>> NUMA domains >>>>>>>>>>>>>> >>>>>>>>>>>>>> With All2All/Quad this should not be a problem >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>>>> Sent: Wednesday, April 13, 2016 1:53 PM >>>>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com>; >>>>>>>>>>>>>> Chinthamani, Sundaram >>>>>>>>>>>>>> <sundaram.chinthamani(a)intel.com>; Harms, Kevin >>>>>>>>>>>>>> <harms(a)alcf.anl.gov>; >>>>>>>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov>; Kumaran, Kalyan >>>>>>>>>>>>>> <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; Fromkin, Russ >>>>>>>>>>>>>> <russ.fromkin(a)intel.com>; Allen, Benjamin S. >>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>> >>>>>>>>>>>>>> Before doing that - let us check if we are running >>>>>>>>>>>>>> similar benchmark. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I am running 1 MPI process (in fact no mpi at all) and 64 >>>>>>>>>>>>>> OpenMP threads version. The threads can be placed with >>>>>>>>>>>>>> "balanced" or "scattered" >>>>>>>>>>>>>> policy. I am using "scattered". >>>>>>>>>>>>>> >>>>>>>>>>>>>> Is this ok so far? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 04/13/2016 03:48 PM, Raman, Karthik wrote: >>>>>>>>>>>>>>> This still does not look right, and the reason could be >>>>>>>>>>>>>>> that the >>>>>>>>>>>>>>> threads are not balanced across all the NUMA domains >>>>>>>>>>>>>>> (ideally you >>>>>>>>>>>>>>> want >>>>>>>>>>>>>>> 16 threads per quadrant) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Before going to SNC4, can you please boot in "Quad" >>>>>>>>>>>>>>> mode. If you >>>>>>>>>>>>>>> look in the BIOS Cluster mode options there is another >>>>>>>>>>>>>>> mode called "Quad" >>>>>>>>>>>>>>> which is different from SNC4, only 2 NUMA domains (DDR, >>>>>>>>>>>>>>> MCDRAM). >>>>>>>>>>>>>>> Please run similar to All2All mode >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>>>>> Sent: Wednesday, April 13, 2016 12:56 PM >>>>>>>>>>>>>>> To: Chinthamani, Sundaram >>>>>>>>>>>>>>> <sundaram.chinthamani(a)intel.com>; Harms, >>>>>>>>>>>>>>> Kevin <harms(a)alcf.anl.gov>; Raman, Karthik >>>>>>>>>>>>>>> <karthik.raman(a)intel.com>; Morozov, Vitali A. >>>>>>>>>>>>>>> <morozov(a)alcf.anl.gov>; Kumaran, Kalyan >>>>>>>>>>>>>>> <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; Fromkin, >>>>>>>>>>>>>>> Russ >>>>>>>>>>>>>>> <russ.fromkin(a)intel.com>; Allen, Benjamin S. >>>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ok, I am now in SNC-4/Flat. Same binary. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> OMP_NUM_THREADS=64 KMP_AFFINITY=scatter numactl -m >>>>>>>>>>>>>>> 4,5,6,7 ./stream >>>>>>>>>>>>>>> |tee running.dat >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>>> STREAM version $Revision: 5.9 $ >>>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>>> This system uses 8 bytes per DOUBLE PRECISION word. >>>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>>> Array size = 134217728, Offset = 0 >>>>>>>>>>>>>>> Total memory required = 3072.0 MB. >>>>>>>>>>>>>>> Each test is run 100 times, but only the *best* time for >>>>>>>>>>>>>>> each is >>>>>>>>>>>>>>> used. >>>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>>> Number of Threads requested = 64 >>>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> <skipped> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>>> Function Rate (MB/s) Avg time Min time >>>>>>>>>>>>>>> Max time >>>>>>>>>>>>>>> Copy: 320997.8352 0.0068 0.0067 0.0071 >>>>>>>>>>>>>>> Scale: 319994.2893 0.0068 0.0067 0.0073 >>>>>>>>>>>>>>> Add: 306055.0205 0.0106 0.0105 0.0107 >>>>>>>>>>>>>>> Triad: 306055.0205 0.0106 0.0105 0.0109 >>>>>>>>>>>>>>> Store: 247056.8669 0.0044 0.0043 0.0051 >>>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>>> Solution Validates >>>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 306055 MB/s >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 04/13/2016 02:01 PM, Chinthamani, Sundaram wrote: >>>>>>>>>>>>>>>> 440 GB/s (base 10). To be precise, 440 bytes / ns >>>>>>>>>>>>>>>> (nano-second). We always report memory bandwidth in >>>>>>>>>>>>>>>> GB/s (base 10) within Intel. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -Thanks, >>>>>>>>>>>>>>>> -Sundaram >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>>>>> From: Harms, Kevin [mailto:[email protected]] >>>>>>>>>>>>>>>> Sent: Wednesday, April 13, 2016 11:52 AM >>>>>>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com>; Chinthamani, >>>>>>>>>>>>>>>> Sundaram <sundaram.chinthamani(a)intel.com>; Morozov, >>>>>>>>>>>>>>>> Vitali A. >>>>>>>>>>>>>>>> <morozov(a)alcf.anl.gov>; Kumaran, Kalyan >>>>>>>>>>>>>>>> <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; >>>>>>>>>>>>>>>> Fromkin, Russ >>>>>>>>>>>>>>>> <russ.fromkin(a)intel.com>; Allen, Benjamin S. >>>>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Just to be clear, memory is always assumed to be >>>>>>>>>>>>>>>> measured using base 2. It's only the storage field that >>>>>>>>>>>>>>>> uses base 10. So we assume if you say 440 GB/s you mean >>>>>>>>>>>>>>>> GiB, or 450,560 MB/s. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> kevin >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> And just some fun with units >>>>>>>>>>>>>>>>> http://sites.utexas.edu/jdm4372/2013/01/05/counting-binary-vs-dec >>>>>>>>>>>>>>>>> i >>>>>>>>>>>>>>>>> m >>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>> l >>>>>>>>>>>>>>>>> -powers-in-the-stream-benchmark/ >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> From: Chinthamani, Sundaram >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Sent: Wednesday, April 13, 2016 11:20 AM >>>>>>>>>>>>>>>>> To: Vitali A. Morozov <morozov(a)anl.gov>; Raman, Karthik >>>>>>>>>>>>>>>>> <karthik.raman(a)intel.com>; Kumaran, Kalyan >>>>>>>>>>>>>>>>> <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; >>>>>>>>>>>>>>>>> Fromkin, Russ >>>>>>>>>>>>>>>>> <russ.fromkin(a)intel.com>; Harms, Kevin >>>>>>>>>>>>>>>>> <harms(a)alcf.anl.gov>; >>>>>>>>>>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov>; Allen, >>>>>>>>>>>>>>>>> Benjamin S. >>>>>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov>; Chinthamani, Sundaram >>>>>>>>>>>>>>>>> <sundaram.chinthamani(a)intel.com> >>>>>>>>>>>>>>>>> Subject: RE: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 440 GB/s stream triad for MC-DRAM and 80 GB/s is our >>>>>>>>>>>>>>>>> internal >>>>>>>>>>>>>>>>> bin3 measurement. You should be able to get closer >>>>>>>>>>>>>>>>> these numbers, if you run the benchmark in Quad/Flat >>>>>>>>>>>>>>>>> or SNC-4/Flat mode. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -Thanks, >>>>>>>>>>>>>>>>> -Sundaram >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Sent: Wednesday, April 13, 2016 11:11 AM >>>>>>>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com>; Chinthamani, >>>>>>>>>>>>>>>>> Sundaram <sundaram.chinthamani(a)intel.com>; Kumaran, Kalyan >>>>>>>>>>>>>>>>> <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; >>>>>>>>>>>>>>>>> Fromkin, Russ >>>>>>>>>>>>>>>>> <russ.fromkin(a)intel.com>; Harms, Kevin >>>>>>>>>>>>>>>>> <harms(a)alcf.anl.gov>; >>>>>>>>>>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov>; Allen, >>>>>>>>>>>>>>>>> Benjamin S. >>>>>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I see - I am treating 418122 MB/s as 418122/1024 GB/s >>>>>>>>>>>>>>>>> (as this is memory unit). >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Anyway - great we are on the same page here. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I am in All2All/Flat mode. You seem to suggest that I >>>>>>>>>>>>>>>>> must be on SNC-4/Flat to see 440000 MB/s? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 04/13/2016 01:06 PM, Raman, Karthik wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Ok I read this as 418 GB/s >>>>>>>>>>>>>>>>> J >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> And ok, just realized you are running in All2All mode, >>>>>>>>>>>>>>>>> so yes this looks good. For Quad/Flat you should see >>>>>>>>>>>>>>>>> ~440 GB/s with the same run. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Sent: Wednesday, April 13, 2016 11:02 AM >>>>>>>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com> >>>>>>>>>>>>>>>>> <mailto:[email protected]>; Chinthamani, Sundaram >>>>>>>>>>>>>>>>> <sundaram.chinthamani(a)intel.com> >>>>>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>>>>> Kumaran, Kalyan <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com> >>>>>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>>>>> Fromkin, Russ <russ.fromkin(a)intel.com> >>>>>>>>>>>>>>>>> <mailto:[email protected]>; Harms, Kevin >>>>>>>>>>>>>>>>> <harms(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>>>> Morozov, Vitali A. >>>>>>>>>>>>>>>>> <morozov(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>>>> Allen, Benjamin S. >>>>>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> <mailto:[email protected]> >>>>>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Karthik, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Repeating the same options you have: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g >>>>>>>>>>>>>>>>> -fopenmp -mcmodel >>>>>>>>>>>>>>>>> medium -shared-intel -opt-streaming-stores always >>>>>>>>>>>>>>>>> stream.c -o >>>>>>>>>>>>>>>>> stream >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>>>>> STREAM version $Revision: 5.9 $ >>>>>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>>>>> This system uses 8 bytes per DOUBLE PRECISION word. >>>>>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>>>>> Array size = 134217728, Offset = 0 Total memory >>>>>>>>>>>>>>>>> required = 3072.0 >>>>>>>>>>>>>>>>> MB. >>>>>>>>>>>>>>>>> Each test is run 100 times, but only the *best* time >>>>>>>>>>>>>>>>> for each is >>>>>>>>>>>>>>>>> used. >>>>>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>>>>> Number of Threads requested = 64 >>>>>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> < skiped > >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Function Rate (MB/s) Avg time Min time >>>>>>>>>>>>>>>>> Max time >>>>>>>>>>>>>>>>> Copy: 362331.5200 0.0060 0.0059 >>>>>>>>>>>>>>>>> 0.0061 >>>>>>>>>>>>>>>>> Scale: 359281.9806 0.0060 0.0060 >>>>>>>>>>>>>>>>> 0.0062 >>>>>>>>>>>>>>>>> Add: 418122.7024 0.0078 0.0077 >>>>>>>>>>>>>>>>> 0.0081 >>>>>>>>>>>>>>>>> Triad: 418122.7024 0.0078 0.0077 >>>>>>>>>>>>>>>>> 0.0081 >>>>>>>>>>>>>>>>> Store: 224685.6729 0.0048 0.0048 >>>>>>>>>>>>>>>>> 0.0049 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 408GB/s >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 04/13/2016 12:15 PM, Raman, Karthik wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2. Effective MCDRAM bandwidth is around 406-408 GB/s. >>>>>>>>>>>>>>>>> Larger values are possible for more high core count bins. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> For bin3 , you should be able to get Triad ~440 GB/s. >>>>>>>>>>>>>>>>> Please >>>>>>>>>>>>>>>>> check if you use the following Intel compiler flags >>>>>>>>>>>>>>>>> -mcmodel >>>>>>>>>>>>>>>>> medium -shared-intel -O3 -xMIC-AVX512 >>>>>>>>>>>>>>>>> -DSTREAM_ARRAY_SIZE=134217728 -DOFFSET=0 -DNTIMES=100 >>>>>>>>>>>>>>>>> -openmp -opt-streaming-stores always The array size is >>>>>>>>>>>>>>>>> just allocating 1GB per array. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Sent: Wednesday, April 13, 2016 10:01 AM >>>>>>>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com> >>>>>>>>>>>>>>>>> <mailto:[email protected]>; Chinthamani, Sundaram >>>>>>>>>>>>>>>>> <sundaram.chinthamani(a)intel.com> >>>>>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>>>>> Kumaran, Kalyan <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com> >>>>>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>>>>> Fromkin, Russ <russ.fromkin(a)intel.com> >>>>>>>>>>>>>>>>> <mailto:[email protected]>; Harms, Kevin >>>>>>>>>>>>>>>>> <harms(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>>>> Morozov, Vitali A. >>>>>>>>>>>>>>>>> <morozov(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>>>> Allen, Benjamin S. >>>>>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> <mailto:[email protected]> >>>>>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> so, this is my understanding that on bin3 hardware (64 >>>>>>>>>>>>>>>>> cores, 6 >>>>>>>>>>>>>>>>> channel DDR4-2133) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 1. Effective DDR bandwidth is 80 GB/s (around 78% of >>>>>>>>>>>>>>>>> peak). >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2. Effective MCDRAM bandwidth is around 406-408 GB/s. >>>>>>>>>>>>>>>>> Larger values are possible for more high core count bins. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 3. Alignment is not important for STREAM triad benchmark. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 4. For both static and dynamic variables, numactl -m >>>>>>>>>>>>>>>>> [0|1] should be used to direct memory placement. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 5. For explicit MCDRAM allocation in flat mode, >>>>>>>>>>>>>>>>> hbw_malloc.h API should be use. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Please correct if I am wrong... >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks a lot for you help! >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 04/12/2016 11:07 AM, Vitali A. Morozov wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Karthik, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I was also wandering how the variable could be >>>>>>>>>>>>>>>>> influencing the allocation if, in principle, I do not >>>>>>>>>>>>>>>>> need to link against memkind in my case. Anyway, sorry >>>>>>>>>>>>>>>>> for confusion! >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I think I did that before - see my first message. I >>>>>>>>>>>>>>>>> was using "numactl --membind=1 ./stream" , which I >>>>>>>>>>>>>>>>> believe is equivalent to "numactl -m 1 ./stream" that >>>>>>>>>>>>>>>>> you suggest. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Here is the result of "numactl -H": >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [morozov@knl00 512_3_C64_MPI16_TH08_flat]$ numactl -H >>>>>>>>>>>>>>>>> available: 2 nodes (0-1) >>>>>>>>>>>>>>>>> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 >>>>>>>>>>>>>>>>> 17 18 19 20 >>>>>>>>>>>>>>>>> 21 >>>>>>>>>>>>>>>>> 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 >>>>>>>>>>>>>>>>> 40 41 42 43 >>>>>>>>>>>>>>>>> 44 >>>>>>>>>>>>>>>>> 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 >>>>>>>>>>>>>>>>> 63 64 65 66 >>>>>>>>>>>>>>>>> 67 >>>>>>>>>>>>>>>>> 68 69 70 >>>>>>>>>>>>>>>>> 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 >>>>>>>>>>>>>>>>> 89 90 91 92 >>>>>>>>>>>>>>>>> 93 >>>>>>>>>>>>>>>>> 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 >>>>>>>>>>>>>>>>> 109 110 111 >>>>>>>>>>>>>>>>> 112 >>>>>>>>>>>>>>>>> 113 114 115 116 117 118 119 120 121 122 123 124 125 >>>>>>>>>>>>>>>>> 126 127 128 >>>>>>>>>>>>>>>>> 129 >>>>>>>>>>>>>>>>> 130 131 132 133 134 135 136 137 138 139 140 141 142 >>>>>>>>>>>>>>>>> 143 144 145 146 147 148 149 150 151 152 153 154 155 >>>>>>>>>>>>>>>>> 156 157 158 >>>>>>>>>>>>>>>>> 159 >>>>>>>>>>>>>>>>> 160 161 162 163 164 165 166 167 168 169 170 171 172 >>>>>>>>>>>>>>>>> 173 174 175 >>>>>>>>>>>>>>>>> 176 >>>>>>>>>>>>>>>>> 177 178 179 180 181 182 183 184 185 186 187 188 189 >>>>>>>>>>>>>>>>> 190 191 192 >>>>>>>>>>>>>>>>> 193 >>>>>>>>>>>>>>>>> 194 195 196 197 198 199 200 201 202 203 204 205 206 >>>>>>>>>>>>>>>>> 207 208 209 210 211 212 213 214 215 216 217 218 219 >>>>>>>>>>>>>>>>> 220 221 222 >>>>>>>>>>>>>>>>> 223 >>>>>>>>>>>>>>>>> 224 225 226 227 228 229 230 231 232 233 234 235 236 >>>>>>>>>>>>>>>>> 237 238 239 >>>>>>>>>>>>>>>>> 240 >>>>>>>>>>>>>>>>> 241 242 243 244 245 246 247 248 249 250 251 252 253 >>>>>>>>>>>>>>>>> 254 255 node >>>>>>>>>>>>>>>>> 0 >>>>>>>>>>>>>>>>> size: 196511 MB node 0 free: 186888 MB node 1 cpus: >>>>>>>>>>>>>>>>> node 1 size: 16384 MB >>>>>>>>>>>>>>>>> node 1 free: 11731 MB >>>>>>>>>>>>>>>>> node distances: >>>>>>>>>>>>>>>>> node 0 1 >>>>>>>>>>>>>>>>> 0: 10 31 >>>>>>>>>>>>>>>>> 1: 31 10 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Sounds like all as expected. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [morozov@knl00 STREAM]$ OMP_NUM_THREADS=64 >>>>>>>>>>>>>>>>> KMP_AFFINITY=scatter numactl -m 1 ./stream >>>>>>>>>>>>>>>>> Triad: 418262.7498 0.0058 0.0057 >>>>>>>>>>>>>>>>> 0.0060 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> For a series of runs, I see this: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> # MB N GB/s >>>>>>>>>>>>>>>>> 228.88 10000000 408867.98 >>>>>>>>>>>>>>>>> 457.76 20000000 405818.57 >>>>>>>>>>>>>>>>> 686.65 30000000 414309.08 >>>>>>>>>>>>>>>>> 915.53 40000000 415063.59 >>>>>>>>>>>>>>>>> 1144.41 50000000 415792.22 >>>>>>>>>>>>>>>>> 1373.29 60000000 415449.01 >>>>>>>>>>>>>>>>> 1602.17 70000000 414106.18 >>>>>>>>>>>>>>>>> 1831.05 80000000 416846.82 >>>>>>>>>>>>>>>>> 2059.94 90000000 414992.29 >>>>>>>>>>>>>>>>> 2288.82 100000000 417898.11 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 04/12/2016 10:32 AM, Raman, Karthik wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Vitali, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 1. >>>>>>>>>>>>>>>>> The MEMKIND_HBW_NODES is *only* when you are >>>>>>>>>>>>>>>>> allocating using >>>>>>>>>>>>>>>>> hbwalloc (+ MCDRAM flat) in your application and when >>>>>>>>>>>>>>>>> memkind is >>>>>>>>>>>>>>>>> unable to find the high bandwidth nodes (pmtt not >>>>>>>>>>>>>>>>> populated) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2. >>>>>>>>>>>>>>>>> With the below >>>>>>>>>>>>>>>>> static allocations, in MCDRAM Flat mode, you can >>>>>>>>>>>>>>>>> simply use ï¿½numactl ï¿½m 1 ./streamï¿½ with propery >>>>>>>>>>>>>>>>> affinity settings to get peak out of MCDRAM. >>>>>>>>>>>>>>>>> a. >>>>>>>>>>>>>>>>> Check ï¿½numactl ï¿½Hï¿½ to find the NUMA domains in your >>>>>>>>>>>>>>>>> node. If >>>>>>>>>>>>>>>>> booted in All2All/FLAT or Quad/Flat mode it should >>>>>>>>>>>>>>>>> show 2 NUMA >>>>>>>>>>>>>>>>> domains , one for DDR (96 GB) and other for MCDRAM (16GB) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>> Karthik >>>>>>>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Sent: Tuesday, April 12, 2016 7:27 AM >>>>>>>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com> >>>>>>>>>>>>>>>>> <mailto:[email protected]>; Chinthamani, Sundaram >>>>>>>>>>>>>>>>> <sundaram.chinthamani(a)intel.com> >>>>>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>>>>> Kumaran, Kalyan <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com> >>>>>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>>>>> Fromkin, Russ <russ.fromkin(a)intel.com> >>>>>>>>>>>>>>>>> <mailto:[email protected]>; Harms, Kevin >>>>>>>>>>>>>>>>> <harms(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>>>> Morozov, Vitali A. >>>>>>>>>>>>>>>>> <morozov(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>>>> Allen, Benjamin S. >>>>>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> <mailto:[email protected]> >>>>>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Karthik, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The node is in All2All mode for now. I made the >>>>>>>>>>>>>>>>> following changes. The original code is using static >>>>>>>>>>>>>>>>> allocation therefore, I follow the advice on 2MB >>>>>>>>>>>>>>>>> static array alignment: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [morozov@thing00 STREAM]$ head -n 108 < stream.c | >>>>>>>>>>>>>>>>> tail -n 4 >>>>>>>>>>>>>>>>> static double a[N+OFFSET] >>>>>>>>>>>>>>>>> __attribute__((aligned(2097152))), >>>>>>>>>>>>>>>>> b[N+OFFSET] >>>>>>>>>>>>>>>>> __attribute__((aligned(2097152))), >>>>>>>>>>>>>>>>> c[N+OFFSET] >>>>>>>>>>>>>>>>> __attribute__((aligned(2097152))), >>>>>>>>>>>>>>>>> d[N+OFFSET] >>>>>>>>>>>>>>>>> __attribute__((aligned(2097152))); >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [morozov@knl00 STREAM]$ make >>>>>>>>>>>>>>>>> mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g >>>>>>>>>>>>>>>>> -fopenmp -mcmodel >>>>>>>>>>>>>>>>> medium -shared-intel -fno-alias stream.c -o stream >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [morozov@knl00 STREAM]$ MEMKIND_HBW_NODES=1 >>>>>>>>>>>>>>>>> OMP_NUM_THREADS=64 >>>>>>>>>>>>>>>>> KMP_AFFINITY=scatter ./stream >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> ..... >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Triad: 80251.3621 0.0300 0.0299 >>>>>>>>>>>>>>>>> 0.0305 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Apparently, the "MEMKIND_HBW_NODES=1" is ignored. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 04/12/2016 01:51 AM, Raman, Karthik wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Also could you please modify the memory allocations in >>>>>>>>>>>>>>>>> STREAM to align at 2MB boundary. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> For example: >>>>>>>>>>>>>>>>> static allocations: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> static STREAM_TYPE a[STREAM_ARRAY_SIZE+OFFSET] >>>>>>>>>>>>>>>>> __attribute__((aligned(2097152))) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> dynamic allocations: >>>>>>>>>>>>>>>>> a = (STREAM_TYPE >>>>>>>>>>>>>>>>> *)_mm_malloc(sizeof(STREAM_TYPE)*(STREAM_ARRAY_SIZE+OFFSET),20971 >>>>>>>>>>>>>>>>> 5 >>>>>>>>>>>>>>>>> 2 >>>>>>>>>>>>>>>>> ) >>>>>>>>>>>>>>>>> ; >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> memkind/hbwalloc: >>>>>>>>>>>>>>>>> (STREAM_TYPE *)hbw_posix_memalign((void **)&a, >>>>>>>>>>>>>>>>> 2097152, >>>>>>>>>>>>>>>>> sizeof(STREAM_TYPE)*(STREAM_ARRAY_SIZE+OFFSET)) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> For memkind/hbwmalloc to work out of the box, you need >>>>>>>>>>>>>>>>> to have >>>>>>>>>>>>>>>>> the PMTT tables populated so that it can identify the >>>>>>>>>>>>>>>>> high bandwidth nodes. Alternatively you can set the >>>>>>>>>>>>>>>>> MEMKIND_HBW_NODES environment variable to the high >>>>>>>>>>>>>>>>> bandwidth node (MCDRAM). >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> e.g. All2All/Flat or Quad/Flat : export >>>>>>>>>>>>>>>>> MEMKIND_HBW_NODES=1 ; ./app_with_hbwmalloc >>>>>>>>>>>>>>>>> SNC4/Flat: export MEMKIND_HBW_NODES=4,5,6,7 ; >>>>>>>>>>>>>>>>> ./app_with_hbw_malloc >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>> Karthik >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> From: Chinthamani, Sundaram >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Sent: Monday, April 11, 2016 8:15 PM >>>>>>>>>>>>>>>>> To: Vitali Morozov <morozov(a)anl.gov> >>>>>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>>>>> Kumaran, Kalyan <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com> >>>>>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>>>>> Fromkin, Russ <russ.fromkin(a)intel.com> >>>>>>>>>>>>>>>>> <mailto:[email protected]>; Raman, Karthik >>>>>>>>>>>>>>>>> <karthik.raman(a)intel.com> >>>>>>>>>>>>>>>>> <mailto:[email protected]>; Harms, Kevin >>>>>>>>>>>>>>>>> <harms(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>>>> Morozov, Vitali A. >>>>>>>>>>>>>>>>> <morozov(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>>>> Allen, Benjamin S. >>>>>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>>>> Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com> >>>>>>>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>>>>>>> Subject: RE: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> You should use numactl ï¿½m 4,5,6,7 to get the memory >>>>>>>>>>>>>>>>> bandwidth out of MC-DRAM. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> What kind of part is it? Is it bin1 or bin3? Even in >>>>>>>>>>>>>>>>> All2All mode, we have measured up to 470 GB/s of >>>>>>>>>>>>>>>>> stream triad bandwidth. Quad mode should get 490 GB/s. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> DDR-stream triad bandwidth on Bin1-part should be >>>>>>>>>>>>>>>>> close to 90 GB/s. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -Sundaram >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> From: Vitali Morozov [mailto:[email protected]] >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Sent: Monday, April 11, 2016 6:59 PM >>>>>>>>>>>>>>>>> To: Kumaran, Kalyan <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; >>>>>>>>>>>>>>>>> Fromkin, Russ >>>>>>>>>>>>>>>>> <russ.fromkin(a)intel.com>; Chinthamani, Sundaram >>>>>>>>>>>>>>>>> <sundaram.chinthamani(a)intel.com>; Raman, Karthik >>>>>>>>>>>>>>>>> <karthik.raman(a)intel.com>; Harms, Kevin >>>>>>>>>>>>>>>>> <harms(a)alcf.anl.gov>; >>>>>>>>>>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov>; Allen, >>>>>>>>>>>>>>>>> Benjamin S. >>>>>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Avinash and others, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> here is what we have so far for Flat more, All2All. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The results from memory were obtained by using default >>>>>>>>>>>>>>>>> STREAM with static allocation. The results from MCDRAM >>>>>>>>>>>>>>>>> were obtained by using modified STREAM with >>>>>>>>>>>>>>>>> hbwmalloc.h interface. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 1 MPI process, 64 threads, scatter affinity, no >>>>>>>>>>>>>>>>> numactl is used. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Most results are as expected: 80 GB/s from DDR memory, >>>>>>>>>>>>>>>>> 420 GB/s from MCDRAM, the MCDRAM size is about 16 GB, >>>>>>>>>>>>>>>>> the aggregate L2 cache size is 32 MB. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Noticed inconsistencies for hbwmalloc benchmarks: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 1. The result is not validated - will spend more time >>>>>>>>>>>>>>>>> on that tomorrow. >>>>>>>>>>>>>>>>> 2. hbw_malloc continues allocating data beyond the >>>>>>>>>>>>>>>>> expected 16 GB up to 48 GB. This means it fails back >>>>>>>>>>>>>>>>> to DDR if it cannot allocate in MCDRAM - not expected >>>>>>>>>>>>>>>>> default behavior. It must fail if it does not have >>>>>>>>>>>>>>>>> enough space. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The problem shows up when I am running static >>>>>>>>>>>>>>>>> allocation and use numactl --membind=1 to ask for >>>>>>>>>>>>>>>>> MCDRAM-only allocation. The variation is very large >>>>>>>>>>>>>>>>> and inconsistent with the result observed when using >>>>>>>>>>>>>>>>> hbw_malloc API: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> # MB N GB/s numactl GB/s >>>>>>>>>>>>>>>>> hbw_malloc >>>>>>>>>>>>>>>>> 228.88 10000000 255555.46 >>>>>>>>>>>>>>>>> 421009.18 >>>>>>>>>>>>>>>>> 457.76 20000000 363930.93 >>>>>>>>>>>>>>>>> 416997.91 >>>>>>>>>>>>>>>>> 686.65 30000000 309415.87 >>>>>>>>>>>>>>>>> 427808.31 >>>>>>>>>>>>>>>>> 915.53 40000000 371691.30 >>>>>>>>>>>>>>>>> 423310.75 >>>>>>>>>>>>>>>>> 1144.41 50000000 187002.22 431032.35 >>>>>>>>>>>>>>>>> 1373.29 60000000 385954.23 427172.91 >>>>>>>>>>>>>>>>> 1602.17 70000000 388254.49 426823.59 >>>>>>>>>>>>>>>>> 1831.05 80000000 374491.43 428103.96 >>>>>>>>>>>>>>>>> 2059.94 90000000 379622.74 428152.02 >>>>>>>>>>>>>>>>> 2288.82 100000000 80868.34 430185.03 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> We also tried SNC-4 instead of All2All, but could not >>>>>>>>>>>>>>>>> get more than 40 GB/s from MCDRAM even with hbw_malloc >>>>>>>>>>>>>>>>> API. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Raw data: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> # flat memory, running from memory (default), alltoall >>>>>>>>>>>>>>>>> # MB N GB/s >>>>>>>>>>>>>>>>> 2.29 100000 272062.96 >>>>>>>>>>>>>>>>> 4.58 200000 491040.47 >>>>>>>>>>>>>>>>> 6.87 300000 603979.78 >>>>>>>>>>>>>>>>> 9.16 400000 745654.04 >>>>>>>>>>>>>>>>> 11.44 500000 811800.77 >>>>>>>>>>>>>>>>> 13.73 600000 901462.35 >>>>>>>>>>>>>>>>> 16.02 700000 891953.26 >>>>>>>>>>>>>>>>> 18.31 800000 1019375.15 >>>>>>>>>>>>>>>>> 20.60 900000 984749.63 >>>>>>>>>>>>>>>>> 22.89 1000000 923516.48 >>>>>>>>>>>>>>>>> 25.18 1100000 758422.09 >>>>>>>>>>>>>>>>> 27.47 1200000 779328.74 >>>>>>>>>>>>>>>>> 29.75 1300000 513185.43 >>>>>>>>>>>>>>>>> 32.04 1400000 155550.35 >>>>>>>>>>>>>>>>> 34.33 1500000 103209.12 >>>>>>>>>>>>>>>>> 36.62 1600000 97435.74 >>>>>>>>>>>>>>>>> 38.91 1700000 83968.40 >>>>>>>>>>>>>>>>> 41.20 1800000 83692.35 >>>>>>>>>>>>>>>>> 43.49 1900000 80734.60 >>>>>>>>>>>>>>>>> 45.78 2000000 76317.89 >>>>>>>>>>>>>>>>> 68.66 3000000 81180.08 >>>>>>>>>>>>>>>>> 91.55 4000000 78889.73 >>>>>>>>>>>>>>>>> 114.44 5000000 79891.50 >>>>>>>>>>>>>>>>> 137.33 6000000 79954.96 >>>>>>>>>>>>>>>>> 160.22 7000000 79477.00 >>>>>>>>>>>>>>>>> 183.11 8000000 79145.59 >>>>>>>>>>>>>>>>> 205.99 9000000 79821.12 >>>>>>>>>>>>>>>>> 228.88 10000000 79519.15 >>>>>>>>>>>>>>>>> 228.88 10000000 79212.54 >>>>>>>>>>>>>>>>> 457.76 20000000 79692.27 >>>>>>>>>>>>>>>>> 686.65 30000000 79866.15 >>>>>>>>>>>>>>>>> 915.53 40000000 79828.15 >>>>>>>>>>>>>>>>> 1144.41 50000000 80214.91 >>>>>>>>>>>>>>>>> 1373.29 60000000 80226.84 >>>>>>>>>>>>>>>>> 1602.17 70000000 80178.77 >>>>>>>>>>>>>>>>> 1831.05 80000000 80177.06 >>>>>>>>>>>>>>>>> 2059.94 90000000 80109.79 >>>>>>>>>>>>>>>>> 2288.82 100000000 80175.94 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> # flat memory, running from MCDRAM, alltoall, >>>>>>>>>>>>>>>>> hbwmalloc API >>>>>>>>>>>>>>>>> # MB N GB/s >>>>>>>>>>>>>>>>> 228.88 10000000 421009.18 >>>>>>>>>>>>>>>>> 457.76 20000000 416997.91 >>>>>>>>>>>>>>>>> 686.65 30000000 427808.31 >>>>>>>>>>>>>>>>> 915.53 40000000 423310.75 >>>>>>>>>>>>>>>>> 1144.41 50000000 431032.35 >>>>>>>>>>>>>>>>> 1373.29 60000000 427172.91 >>>>>>>>>>>>>>>>> 1602.17 70000000 426823.59 >>>>>>>>>>>>>>>>> 1831.05 80000000 428103.96 >>>>>>>>>>>>>>>>> 2059.94 90000000 428152.02 >>>>>>>>>>>>>>>>> 2288.82 100000000 430185.03 >>>>>>>>>>>>>>>>> 2343.75 102400000 425999.98 >>>>>>>>>>>>>>>>> 4687.50 204800000 427821.10 >>>>>>>>>>>>>>>>> 9155.27 400000000 429318.13 >>>>>>>>>>>>>>>>> 9375.00 409600000 428376.70 >>>>>>>>>>>>>>>>> 11444.09 500000000 428052.08 >>>>>>>>>>>>>>>>> 13732.91 600000000 266061.01 >>>>>>>>>>>>>>>>> 16021.73 700000000 200257.79 >>>>>>>>>>>>>>>>> 18310.55 800000000 168164.19 >>>>>>>>>>>>>>>>> 20599.37 900000000 150192.91 >>>>>>>>>>>>>>>>> 22888.18 1000000000 137862.07 >>>>>>>>>>>>>>>>> 25177.00 1100000000 129418.12 >>>>>>>>>>>>>>>>> 27465.82 1200000000 123019.53 >>>>>>>>>>>>>>>>> 29754.64 1300000000 118268.79 >>>>>>>>>>>>>>>>> 32043.46 1400000000 114343.33 >>>>>>>>>>>>>>>>> 34332.28 1500000000 110991.25 >>>>>>>>>>>>>>>>> 36621.09 1600000000 108538.36 >>>>>>>>>>>>>>>>> 38909.91 1700000000 106249.72 >>>>>>>>>>>>>>>>> 41198.73 1800000000 104455.07 >>>>>>>>>>>>>>>>> 43487.55 1900000000 102828.93 >>>>>>>>>>>>>>>>> 45776.37 2000000000 101348.56 >>>>>>>>>>>>>>>>> 48065.19 2100000000 100013.73 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> We appreciate any help on that. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On 04/11/2016 08:42 PM, Kumaran, Kalyan wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Avinash, thanks for the prompt reply. Kevin/Vitali can >>>>>>>>>>>>>>>>> provide >>>>>>>>>>>>>>>>> all the details. The short summary is we are unable to >>>>>>>>>>>>>>>>> reproduce the STREAM results from IPM or DDR. It would >>>>>>>>>>>>>>>>> be great if Karthik or Sundaram can advise us on how >>>>>>>>>>>>>>>>> to run (which mode etc) or if they could log into the >>>>>>>>>>>>>>>>> machine and make sure we have set up things correctly. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Kumar >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Apr 11, 2016, at 8:07 PM, Sodani, Avinash >>>>>>>>>>>>>>>>> <avinash.sodani(a)intel.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Copying Sundaram/Karthik from our side. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Kumar, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Can you send us more information on this? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -Avinash >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> From: Kumaran, >>>>>>>>>>>>>>>>> Kalyan [mailto:[email protected]] >>>>>>>>>>>>>>>>> Sent: Monday, April 11, 2016 5:32 PM >>>>>>>>>>>>>>>>> To: Fromkin, Russ; Sodani, Avinash >>>>>>>>>>>>>>>>> Cc: Harms, Kevin; Morozov, Vitali A. >>>>>>>>>>>>>>>>> Subject: Fwd: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Russ, Avinash, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> we need help in understanding how to obtain STREAM >>>>>>>>>>>>>>>>> results. Can anyone help? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Kumar >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Begin forwarded message: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> From: "Harms, Kevin" <harms(a)alcf.anl.gov> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Subject: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Date: April 11, 2016 at 5:42:15 PM CDT >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> To: "intel-nda(a)jlse.anl.gov" >>>>>>>>>>>>>>>>> <intel-nda(a)jlse.anl.gov> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The knl00 node is currently in Flat mode and All2All >>>>>>>>>>>>>>>>> model. With this setup we can get 420 GB/s from IPM >>>>>>>>>>>>>>>>> memory on STREAM Triad when using hbw_malloc and >>>>>>>>>>>>>>>>> friends which was confirmed by both Vitali and I. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Changing the BIOS settings and warm rebooting seem to >>>>>>>>>>>>>>>>> be problematic. Vitali and I were running earlier and >>>>>>>>>>>>>>>>> could not get over 80 GB/s with any method even though >>>>>>>>>>>>>>>>> memory capacity indicated flat mode. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Using numactl --membind is also problematic. The >>>>>>>>>>>>>>>>> STREAM results >>>>>>>>>>>>>>>>> when using this to launch the application were >>>>>>>>>>>>>>>>> erratic, so it is also suggested not to use that. That >>>>>>>>>>>>>>>>> means you will need to modify your code to use dynamic >>>>>>>>>>>>>>>>> memory and hbw_malloc to exploit the IPM. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Will followup with more detailed results later. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> kevin >>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>> Intel-nda mailing list >>>>>>>>>>>>>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>>>>>>>>>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> <flat_a2a_ddr.txt><flat_a2a_ipm.txt><flat_quad_ddr.txt><flat_quad_ipm.txt><flat_snc4_ddr.txt><flat_snc4_ipm.txt> >>>>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Intel-nda mailing list >>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Intel-nda mailing list >>>> Intel-nda(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>> >>> >>> >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> >

1 0

Re: [JLSE-Intel-NDA] KNL Status
by Vitali A Morozov 21 Apr '16

21 Apr '16

Then, the only difference on what you are doing and Kevin and I were doing before is 128 threads versus 64 and 256 threads. I do not think we have previously run 128 threads. Vitali On 4/21/16 10:29 AM, Scott Parker wrote: > > It's not the compiler, I see no significant difference in SNC-4 mode > between Intel 16 and Intel 17 Beta. > > >icc -V > Intel(R) C Intel(R) 64 Compiler for applications running on Intel(R) > 64, Version 16.0.0.109 Build 20150815 > Copyright (C) 1985-2015 Intel Corporation. All rights reserved. > > >icc -O3 -fopenmp -DSTREAM_ARRAY_SIZE=134217728 -xMIC-AVX512 -mcmodel > medium -shared-intel stream.c -o stream_c.exe > > >OMP_NUM_THREADS=128 numactl -m 4,5,6,7 ./stream_c.exe > Function Best Rate MB/s Avg time Min time Max time > Copy: 390734.0 0.005547 0.005496 0.005631 > Scale: 411250.1 0.005309 0.005222 0.005569 > Add: 449326.5 0.007190 0.007169 0.007220 > Triad: 440248.9 0.007343 0.007317 0.007391 > > On 4/21/16 10:12 AM, Vitali A Morozov wrote: >> Yeah - as Ronald Green has figured out, the beta version of a >> compiler (17) does make something differently compared to production >> version 16. For some reason, it only matters in SNC-4 mode and not in >> All2All mode. >> >> I am curious what was changed (or added) in version 17... >> >> Vitali >> >> >> >> On 4/21/16 9:45 AM, Kumaran, Kalyan wrote: >>> Looks like Scott Parker gets better results: >>> >>> Adding another data point, I ran in SNC-4 mode with the >>> configuration below and observed a consistent BW around 440 GB/s >>> with the best run producing 442 GB/s. In this case 128 threads gave >>> the best results. >>> >>> I've noticed that the STREAM results sensitivity to build and run >>> parameters seems to vary depending on the memory configuration >>> in flat-all2all I saw noticeably better results with memory aligned >>> on 2MB boundaries and 64 threads, in SNC-4 mode the alignment >>> doesn't seem to matter and 128 threads is producing the best results. >>> >>> >>> Mode: flat & SNC-4 >>> Compiler: Intel 17 Beta >>> Compile Options: icc -O3 -fopenmp -DSTREAM_ARRAY_SIZE=134217728 >>> -xMIC-AVX512 -mcmodel medium -shared-intel >>> Run Command: OMP_NUM_THREADS=128 numactl -m 4,5,6,7 ./stream_mod >>> >>> Results: >>> Function Best Rate MB/s Avg time Min time Max time >>> Copy: 390378.3 0.005524 0.005501 0.005546 >>> Scale: 412738.8 0.005237 0.005203 0.005278 >>> Add: 448759.4 0.007199 0.007178 0.007218 >>> Triad: 441919.3 0.007355 0.007289 0.007548 >>> >>> >>> >>>> On Apr 20, 2016, at 7:39 PM, Vitali A Morozov <morozov(a)anl.gov >>>> <mailto:[email protected]>> wrote: >>>> >>>> Hi, >>>> >>>> a quick update on STREAM with the latest beta compiler >>>> >>>> [morozov@knl00 STREAM]$ icc -V >>>> Intel(R) C Intel(R) 64 Compiler for applications running on >>>> Intel(R) 64, Version 17.0.0.042 Beta Build 20160315 >>>> >>>> compiler options >>>> mpiicc -O3 -xMIC-AVX512 -qopt-report=5 -Wall -g -fopenmp -mcmodel >>>> medium -shared-intel -qopt-streaming-stores always -restrict >>>> stream.c -o stream >>>> >>>> Running in flat mode all2all >>>> >>>> morozov@knl00 STREAM]$ export OMP_NUM_THREADS=256 >>>> [morozov@knl00 STREAM]$ numactl -m 1 ./stream >>>> ------------------------------------------------------------- >>>> STREAM version $Revision: 5.9 $ >>>> ------------------------------------------------------------- >>>> This system uses 8 bytes per DOUBLE PRECISION word. >>>> ------------------------------------------------------------- >>>> Array size = 134217728, Offset = 0 >>>> Total memory required = 3072.0 MB. >>>> Each test is run 500 times, but only >>>> the *best*time for each is used. >>>> ------------------------------------------------------------- >>>> Number of Threads requested = 256 >>>> ------------------------------------------------------------- >>>> <skipped> >>>> ------------------------------------------------------------- >>>> Your clock granularity/precision appears to be 1 microseconds. >>>> Each test below will take on the order of 6356 microseconds. >>>> (= 6356 clock ticks) >>>> Increase the size of the arrays if this shows that >>>> you are not getting at least 20 clock ticks per test. >>>> ------------------------------------------------------------- >>>> WARNING -- The above is only a rough guideline. >>>> For best results, please be sure you know the >>>> precision of your system timer. >>>> ------------------------------------------------------------- >>>> Function Rate (MB/s) Avg time Min time Max time >>>> Copy: 330891.5637 0.0066 0.0065 0.0078 >>>> Scale: 330127.5200 0.0066 0.0065 0.0069 >>>> Add: 383622.4448 0.0085 0.0084 0.0093 >>>> Triad: 383252.4574 0.0085 0.0084 0.0098 >>>> ------------------------------------------------------------- >>>> Solution Validates >>>> ------------------------------------------------------------- >>>> >>>> Same with 64 threads scatter mode: >>>> >>>> morozov@knl00 STREAM]$ export OMP_NUM_THREADS=64 >>>> [morozov@knl00 STREAM]$ KMP_AFFINITY=scatter >>>> [morozov@knl00 STREAM]$ numactl -m 1 ./stream >>>> ------------------------------------------------------------- >>>> STREAM version $Revision: 5.9 $ >>>> ------------------------------------------------------------- >>>> This system uses 8 bytes per DOUBLE PRECISION word. >>>> ------------------------------------------------------------- >>>> Array size = 134217728, Offset = 0 >>>> Total memory required = 3072.0 MB. >>>> Each test is run 500 times, but only >>>> the *best*time for each is used. >>>> ------------------------------------------------------------- >>>> Number of Threads requested = 64 >>>> ------------------------------------------------------------- >>>> <skipped> >>>> ------------------------------------------------------------- >>>> Your clock granularity/precision appears to be 1 microseconds. >>>> Each test below will take on the order of 6056 microseconds. >>>> (= 6056 clock ticks) >>>> Increase the size of the arrays if this shows that >>>> you are not getting at least 20 clock ticks per test. >>>> ------------------------------------------------------------- >>>> WARNING -- The above is only a rough guideline. >>>> For best results, please be sure you know the >>>> precision of your system timer. >>>> ------------------------------------------------------------- >>>> Function Rate (MB/s) Avg time Min time Max time >>>> Copy: 353486.8826 0.0061 0.0061 0.0067 >>>> Scale: 353265.0608 0.0061 0.0061 0.0066 >>>> Add: 423961.3055 0.0077 0.0076 0.0079 >>>> Triad: 424907.9750 0.0076 0.0076 0.0084 >>>> ------------------------------------------------------------- >>>> Solution Validates >>>> ------------------------------------------------------------- >>>> >>>> Vitali >>>> >>>> >>>> >>>> >>>> On 4/18/16 12:43 PM, Kumaran, Kalyan wrote: >>>>> FYI. Karthik will get on the machine to debug. >>>>> >>>>> Sent from my iPhone >>>>> >>>>> Begin forwarded message: >>>>> >>>>>> *From:* "Chinthamani, Sundaram" <sundaram.chinthamani(a)intel.com> >>>>>> *Date:* April 18, 2016 at 2:05:26 PM CDT >>>>>> *To:* "Harms, Kevin" <harms(a)alcf.anl.gov>, "Allen, Benjamin S." >>>>>> <bsallen(a)alcf.anl.gov <mailto:[email protected]>> >>>>>> *Cc:* "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov>, "Kaana >>>>>> Balakrishna, Abhishek" <abhishek.kaana.balakrishna(a)intel.com>, >>>>>> "Morozov, Vitali A." <morozov(a)alcf.anl.gov >>>>>> <mailto:[email protected]>>, "Raman, Karthik" >>>>>> <karthik.raman(a)intel.com <mailto:[email protected]>>, >>>>>> "Inkley, Benson" <benson.inkley(a)intel.com >>>>>> <mailto:[email protected]>>, "Sodani, Avinash" >>>>>> <avinash.sodani(a)intel.com <mailto:[email protected]>>, >>>>>> "Fromkin, Russ" <russ.fromkin(a)intel.com >>>>>> <mailto:[email protected]>>, "Chinthamani, Sundaram" >>>>>> <sundaram.chinthamani(a)intel.com> >>>>>> *Subject:* *RE: [JLSE-Intel-NDA] KNL Status* >>>>>> >>>>>> I would expect Quad mode / SNC-4 mode to provide higher bandwidth >>>>>> compared to All2All mode. We haven't seen this in our internal >>>>>> measurements. >>>>>> >>>>>> -Thanks, >>>>>> -Sundaram >>>>>> >>>>>> -----Original Message----- >>>>>> From: Harms, Kevin [mailto:[email protected]] >>>>>> Sent: Monday, April 18, 2016 12:00 PM >>>>>> To: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >>>>>> Cc: Kumaran, Kalyan <kumaran(a)alcf.anl.gov>; Kaana Balakrishna, >>>>>> Abhishek <abhishek.kaana.balakrishna(a)intel.com>; Morozov, Vitali >>>>>> A. <morozov(a)alcf.anl.gov <mailto:[email protected]>>; >>>>>> Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com>; Raman, >>>>>> Karthik <karthik.raman(a)intel.com >>>>>> <mailto:[email protected]>>; Inkley, Benson >>>>>> <benson.inkley(a)intel.com <mailto:[email protected]>>; >>>>>> Sodani, Avinash <avinash.sodani(a)intel.com >>>>>> <mailto:[email protected]>>; Fromkin, Russ >>>>>> <russ.fromkin(a)intel.com <mailto:[email protected]>> >>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>> >>>>>> >>>>>> I tested cached mode with all2all, snc4 and quad models. Cached >>>>>> all2all gave 275,000 MB/s. Does that look reasonable? SNC4 and >>>>>> quad mode gave 225,000 MB/s or so. >>>>>> >>>>>> kevin >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> The following versions are now installed: BIOS 01.01.0104, ME >>>>>>> 03.01.03.011.0, BMC 00.17.9699, FRUSDR 1.14. >>>>>>> >>>>>>> Ben >>>>>>> >>>>>>>> On Apr 18, 2016, at 1:27 PM, Harms, Kevin <harms(a)alcf.anl.gov> >>>>>>>> wrote: >>>>>>>> >>>>>>>> >>>>>>>> Ben completed the BIOS upgrade, but we do not see any >>>>>>>> difference in performance. I have attached results from runs >>>>>>>> done under three configurations: flat+all2all, flat+quad, >>>>>>>> flat+snc4. >>>>>>>> >>>>>>>> flat+all2all = 427046.0546 MB/s >>>>>>>> flat+quad = 327253.8882 MB/s >>>>>>>> flat+snc4 = 308555.9588 MB/s >>>>>>>> >>>>>>>> I compiled the CORAL stream.c with the following options: >>>>>>>> icc -mcmodel medium -shared-intel -O3 -xMIC-AVX512 >>>>>>>> -DN=50000000 -DOFFSET=0 -DNTIMES=100 -openmp >>>>>>>> -opt-streaming-stores always stream.c -o stream >>>>>>>> >>>>>>>> I added the alignment directives to the arrays. The attached >>>>>>>> files show the NUMA domains and numactl commands used and run >>>>>>>> parameters. >>>>>>>> >>>>>>>> Have I missed a parameter? >>>>>>>> >>>>>>>> kevin >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> Suggest a time Friday afternoon. We are in central time zone. >>>>>>>>> We can get you access to this B0 too. >>>>>>>>> >>>>>>>>> -Kumar >>>>>>>>> >>>>>>>>> >>>>>>>>>> On Apr 13, 2016, at 5:06 PM, Kaana Balakrishna, Abhishek >>>>>>>>>> <abhishek.kaana.balakrishna(a)intel.com> wrote: >>>>>>>>>> >>>>>>>>>> + Benson >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Abhishek K Balakrishna (KB) >>>>>>>>>> Xeon Phiï¿½ Hardware Platform Applications Engineer, >>>>>>>>>> Technical Computing Group (TCG) >>>>>>>>>> Intel Corporation >>>>>>>>>> Ph - 512 466 9886 >>>>>>>>>> www.intel.com/xeonphi <http://www.intel.com/xeonphi> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>> Sent: Wednesday, April 13, 2016 4:58 PM >>>>>>>>>> To: Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com>; >>>>>>>>>> Raman, Karthik <karthik.raman(a)intel.com>; Harms, Kevin >>>>>>>>>> <harms(a)alcf.anl.gov>; Morozov, Vitali A. >>>>>>>>>> <morozov(a)alcf.anl.gov>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov> >>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; Fromkin, Russ >>>>>>>>>> <russ.fromkin(a)intel.com>; Allen, Benjamin S. >>>>>>>>>> <bsallen(a)alcf.anl.gov>; Kaana Balakrishna, Abhishek >>>>>>>>>> <abhishek.kaana.balakrishna(a)intel.com> >>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>> >>>>>>>>>> Sure! >>>>>>>>>> >>>>>>>>>> Vitali >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 04/13/2016 04:54 PM, Chinthamani, Sundaram wrote: >>>>>>>>>>> Would it be possible to have a meeting to go over your >>>>>>>>>>> questions rather than exchanging E-mails back and forth? >>>>>>>>>>> >>>>>>>>>>> + KB from Intel side. He can help set-up the meeting >>>>>>>>>>> between Labs and Intel. >>>>>>>>>>> >>>>>>>>>>> -Thanks, >>>>>>>>>>> -Sundaram >>>>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>> Sent: Wednesday, April 13, 2016 2:50 PM >>>>>>>>>>> To: Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com>; >>>>>>>>>>> Raman, >>>>>>>>>>> Karthik <karthik.raman(a)intel.com>; Harms, Kevin >>>>>>>>>>> <harms(a)alcf.anl.gov>; >>>>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov>; Kumaran, Kalyan >>>>>>>>>>> <kumaran(a)alcf.anl.gov> >>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; Fromkin, Russ >>>>>>>>>>> <russ.fromkin(a)intel.com>; Allen, Benjamin S. >>>>>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>> >>>>>>>>>>> On 04/13/2016 04:42 PM, Chinthamani, Sundaram wrote: >>>>>>>>>>>> node 4 free: 2907 MB >>>>>>>>>>>> node 5 free: 2926 MB >>>>>>>>>>>> node 6 free: 2922 MB >>>>>>>>>>>> node 7 free: 2914 MB >>>>>>>>>>>> >>>>>>>>>>>> There is 16 GB of MC-DRAM and 4 GB per NUMA node. Is >>>>>>>>>>>> there any other process running on the machine? Why does >>>>>>>>>>>> "free memory" show only 3 GB per node? >>>>>>>>>>> This is EXACTLY why I am asking this question. Nothing is >>>>>>>>>>> running on the node - and I do not see full 4 GiB of >>>>>>>>>>> available MCDRAM memory. This might be reason why I do not >>>>>>>>>>> see what is expected. >>>>>>>>>>> >>>>>>>>>>> I understand that I can reduce the array size, but I am >>>>>>>>>>> trying to reproduce your results. >>>>>>>>>>> >>>>>>>>>>> I am still unclear on if you run "no mpi" version, how come >>>>>>>>>>> that SNC-4 mode is giving you advantage? All arrays are >>>>>>>>>>> allocated in the same quadrant. Requests from the other >>>>>>>>>>> quadrants will go to this one and create congestion. >>>>>>>>>>> >>>>>>>>>>> Vitali >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> If you want to test Stream triad bandwidth to MC-DRAM in >>>>>>>>>>>> Quad/Flat mode, you can use even smaller arrays of few 100 MBs. >>>>>>>>>>>> >>>>>>>>>>>> -Thanks, >>>>>>>>>>>> -Sundaram >>>>>>>>>>>> >>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>> Sent: Wednesday, April 13, 2016 2:32 PM >>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com>; Chinthamani, >>>>>>>>>>>> Sundaram >>>>>>>>>>>> <sundaram.chinthamani(a)intel.com>; Harms, Kevin >>>>>>>>>>>> <harms(a)alcf.anl.gov>; >>>>>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov>; Kumaran, Kalyan >>>>>>>>>>>> <kumaran(a)alcf.anl.gov> >>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; Fromkin, Russ >>>>>>>>>>>> <russ.fromkin(a)intel.com>; Allen, Benjamin S. >>>>>>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>> >>>>>>>>>>>> ok, that is fine. >>>>>>>>>>>> >>>>>>>>>>>> Now the next issue I have is that for static arrays, all >>>>>>>>>>>> three arrays must land up on one of the numa nodes - the >>>>>>>>>>>> node that is attached to a master thread. >>>>>>>>>>>> >>>>>>>>>>>> You advice to use N = 134217728, that gives exactly 3 GiB >>>>>>>>>>>> of memory footprint for 3 arrays of triad. >>>>>>>>>>>> >>>>>>>>>>>> According to "numactl -H", I do not have 3 GiB of available >>>>>>>>>>>> memory on any of the MCDRAM numa nodes: >>>>>>>>>>>> >>>>>>>>>>>> numactl -H |grep free >>>>>>>>>>>> node 0 free: 46336 MB >>>>>>>>>>>> node 1 free: 46923 MB >>>>>>>>>>>> node 2 free: 46970 MB >>>>>>>>>>>> node 3 free: 46961 MB >>>>>>>>>>>> node 4 free: 2907 MB >>>>>>>>>>>> node 5 free: 2926 MB >>>>>>>>>>>> node 6 free: 2922 MB >>>>>>>>>>>> node 7 free: 2914 MB >>>>>>>>>>>> >>>>>>>>>>>> Therefore, at least one array will not be placed to the >>>>>>>>>>>> same numa node. >>>>>>>>>>>> Is this ok? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 04/13/2016 04:17 PM, Raman, Karthik wrote: >>>>>>>>>>>>> Yes we run pure OMP code too. >>>>>>>>>>>>> >>>>>>>>>>>>> The problem with KMP_AFFINITY with SNC4 mode (currently) >>>>>>>>>>>>> is that it >>>>>>>>>>>>> may not equally balance the threads per NUMA domain even with >>>>>>>>>>>>> scatter or balanced, you can check that by mapping the >>>>>>>>>>>>> "verbose" output (e.g. >>>>>>>>>>>>> KMP_AFFINITY=scatter,verbose) with "numactl -H" output to >>>>>>>>>>>>> see if the >>>>>>>>>>>>> OS thread to logical CPU mapping is balanced across all >>>>>>>>>>>>> NUMA domains >>>>>>>>>>>>> >>>>>>>>>>>>> With All2All/Quad this should not be a problem >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>>> Sent: Wednesday, April 13, 2016 1:53 PM >>>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com>; Chinthamani, >>>>>>>>>>>>> Sundaram >>>>>>>>>>>>> <sundaram.chinthamani(a)intel.com>; Harms, Kevin >>>>>>>>>>>>> <harms(a)alcf.anl.gov>; >>>>>>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov>; Kumaran, Kalyan >>>>>>>>>>>>> <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; Fromkin, Russ >>>>>>>>>>>>> <russ.fromkin(a)intel.com>; Allen, Benjamin S. >>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>> >>>>>>>>>>>>> Before doing that - let us check if we are running similar >>>>>>>>>>>>> benchmark. >>>>>>>>>>>>> >>>>>>>>>>>>> I am running 1 MPI process (in fact no mpi at all) and 64 >>>>>>>>>>>>> OpenMP threads version. The threads can be placed with >>>>>>>>>>>>> "balanced" or "scattered" >>>>>>>>>>>>> policy. I am using "scattered". >>>>>>>>>>>>> >>>>>>>>>>>>> Is this ok so far? >>>>>>>>>>>>> >>>>>>>>>>>>> Vitali >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 04/13/2016 03:48 PM, Raman, Karthik wrote: >>>>>>>>>>>>>> This still does not look right, and the reason could be >>>>>>>>>>>>>> that the >>>>>>>>>>>>>> threads are not balanced across all the NUMA domains >>>>>>>>>>>>>> (ideally you >>>>>>>>>>>>>> want >>>>>>>>>>>>>> 16 threads per quadrant) >>>>>>>>>>>>>> >>>>>>>>>>>>>> Before going to SNC4, can you please boot in "Quad" mode. >>>>>>>>>>>>>> If you >>>>>>>>>>>>>> look in the BIOS Cluster mode options there is another >>>>>>>>>>>>>> mode called "Quad" >>>>>>>>>>>>>> which is different from SNC4, only 2 NUMA domains (DDR, >>>>>>>>>>>>>> MCDRAM). >>>>>>>>>>>>>> Please run similar to All2All mode >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>>>> Sent: Wednesday, April 13, 2016 12:56 PM >>>>>>>>>>>>>> To: Chinthamani, Sundaram >>>>>>>>>>>>>> <sundaram.chinthamani(a)intel.com>; Harms, >>>>>>>>>>>>>> Kevin <harms(a)alcf.anl.gov>; Raman, Karthik >>>>>>>>>>>>>> <karthik.raman(a)intel.com>; Morozov, Vitali A. >>>>>>>>>>>>>> <morozov(a)alcf.anl.gov>; Kumaran, Kalyan >>>>>>>>>>>>>> <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; Fromkin, Russ >>>>>>>>>>>>>> <russ.fromkin(a)intel.com>; Allen, Benjamin S. >>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>> >>>>>>>>>>>>>> ok, I am now in SNC-4/Flat. Same binary. >>>>>>>>>>>>>> >>>>>>>>>>>>>> OMP_NUM_THREADS=64 KMP_AFFINITY=scatter numactl -m >>>>>>>>>>>>>> 4,5,6,7 ./stream >>>>>>>>>>>>>> |tee running.dat >>>>>>>>>>>>>> >>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>> STREAM version $Revision: 5.9 $ >>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>> This system uses 8 bytes per DOUBLE PRECISION word. >>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>> Array size = 134217728, Offset = 0 >>>>>>>>>>>>>> Total memory required = 3072.0 MB. >>>>>>>>>>>>>> Each test is run 100 times, but only the *best* time for >>>>>>>>>>>>>> each is >>>>>>>>>>>>>> used. >>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>> Number of Threads requested = 64 >>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>> >>>>>>>>>>>>>> <skipped> >>>>>>>>>>>>>> >>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>> Function Rate (MB/s) Avg time Min time Max >>>>>>>>>>>>>> time >>>>>>>>>>>>>> Copy: 320997.8352 0.0068 0.0067 0.0071 >>>>>>>>>>>>>> Scale: 319994.2893 0.0068 0.0067 0.0073 >>>>>>>>>>>>>> Add: 306055.0205 0.0106 0.0105 0.0107 >>>>>>>>>>>>>> Triad: 306055.0205 0.0106 0.0105 0.0109 >>>>>>>>>>>>>> Store: 247056.8669 0.0044 0.0043 0.0051 >>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>> Solution Validates >>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> 306055 MB/s >>>>>>>>>>>>>> >>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 04/13/2016 02:01 PM, Chinthamani, Sundaram wrote: >>>>>>>>>>>>>>> 440 GB/s (base 10). To be precise, 440 bytes / ns >>>>>>>>>>>>>>> (nano-second). We always report memory bandwidth in >>>>>>>>>>>>>>> GB/s (base 10) within Intel. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -Thanks, >>>>>>>>>>>>>>> -Sundaram >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>>>> From: Harms, Kevin [mailto:[email protected]] >>>>>>>>>>>>>>> Sent: Wednesday, April 13, 2016 11:52 AM >>>>>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com>; Chinthamani, >>>>>>>>>>>>>>> Sundaram <sundaram.chinthamani(a)intel.com>; Morozov, >>>>>>>>>>>>>>> Vitali A. >>>>>>>>>>>>>>> <morozov(a)alcf.anl.gov>; Kumaran, Kalyan >>>>>>>>>>>>>>> <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>>>>>> <mailto:[email protected]>>; Fromkin, Russ >>>>>>>>>>>>>>> <russ.fromkin(a)intel.com>; Allen, Benjamin S. >>>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Just to be clear, memory is always assumed to be >>>>>>>>>>>>>>> measured using base 2. It's only the storage field that >>>>>>>>>>>>>>> uses base 10. So we assume if you say 440 GB/s you mean >>>>>>>>>>>>>>> GiB, or 450,560 MB/s. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> kevin >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> And just some fun with units >>>>>>>>>>>>>>>> http://sites.utexas.edu/jdm4372/2013/01/05/counting-binary-vs-dec >>>>>>>>>>>>>>>> i >>>>>>>>>>>>>>>> m >>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>> l >>>>>>>>>>>>>>>> -powers-in-the-stream-benchmark/ >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> From: Chinthamani, Sundaram >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Sent: Wednesday, April 13, 2016 11:20 AM >>>>>>>>>>>>>>>> To: Vitali A. Morozov <morozov(a)anl.gov>; Raman, Karthik >>>>>>>>>>>>>>>> <karthik.raman(a)intel.com>; Kumaran, Kalyan >>>>>>>>>>>>>>>> <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; >>>>>>>>>>>>>>>> Fromkin, Russ >>>>>>>>>>>>>>>> <russ.fromkin(a)intel.com>; Harms, Kevin >>>>>>>>>>>>>>>> <harms(a)alcf.anl.gov>; >>>>>>>>>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov>; Allen, >>>>>>>>>>>>>>>> Benjamin S. >>>>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov>; Chinthamani, Sundaram >>>>>>>>>>>>>>>> <sundaram.chinthamani(a)intel.com> >>>>>>>>>>>>>>>> Subject: RE: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 440 GB/s stream triad for MC-DRAM and 80 GB/s is our >>>>>>>>>>>>>>>> internal >>>>>>>>>>>>>>>> bin3 measurement. You should be able to get closer >>>>>>>>>>>>>>>> these numbers, if you run the benchmark in Quad/Flat or >>>>>>>>>>>>>>>> SNC-4/Flat mode. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -Thanks, >>>>>>>>>>>>>>>> -Sundaram >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Sent: Wednesday, April 13, 2016 11:11 AM >>>>>>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com>; Chinthamani, >>>>>>>>>>>>>>>> Sundaram <sundaram.chinthamani(a)intel.com>; Kumaran, Kalyan >>>>>>>>>>>>>>>> <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; >>>>>>>>>>>>>>>> Fromkin, Russ >>>>>>>>>>>>>>>> <russ.fromkin(a)intel.com>; Harms, Kevin >>>>>>>>>>>>>>>> <harms(a)alcf.anl.gov>; >>>>>>>>>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov>; Allen, >>>>>>>>>>>>>>>> Benjamin S. >>>>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I see - I am treating 418122 MB/s as 418122/1024 GB/s >>>>>>>>>>>>>>>> (as this is memory unit). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Anyway - great we are on the same page here. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I am in All2All/Flat mode. You seem to suggest that I >>>>>>>>>>>>>>>> must be on SNC-4/Flat to see 440000 MB/s? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 04/13/2016 01:06 PM, Raman, Karthik wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Ok I read this as 418 GB/s >>>>>>>>>>>>>>>> J >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> And ok, just realized you are running in All2All mode, >>>>>>>>>>>>>>>> so yes this looks good. For Quad/Flat you should see >>>>>>>>>>>>>>>> ~440 GB/s with the same run. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Sent: Wednesday, April 13, 2016 11:02 AM >>>>>>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com> >>>>>>>>>>>>>>>> <mailto:[email protected]>; Chinthamani, Sundaram >>>>>>>>>>>>>>>> <sundaram.chinthamani(a)intel.com> >>>>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>>>> Kumaran, Kalyan <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com> >>>>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>>>> Fromkin, Russ <russ.fromkin(a)intel.com> >>>>>>>>>>>>>>>> <mailto:[email protected]>; Harms, Kevin >>>>>>>>>>>>>>>> <harms(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>>> Morozov, Vitali A. >>>>>>>>>>>>>>>> <morozov(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>>> Allen, Benjamin S. >>>>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> <mailto:[email protected]> >>>>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Karthik, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Repeating the same options you have: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp >>>>>>>>>>>>>>>> -mcmodel >>>>>>>>>>>>>>>> medium -shared-intel -opt-streaming-stores always >>>>>>>>>>>>>>>> stream.c -o >>>>>>>>>>>>>>>> stream >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>>>> STREAM version $Revision: 5.9 $ >>>>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>>>> This system uses 8 bytes per DOUBLE PRECISION word. >>>>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>>>> Array size = 134217728, Offset = 0 Total memory >>>>>>>>>>>>>>>> required = 3072.0 >>>>>>>>>>>>>>>> MB. >>>>>>>>>>>>>>>> Each test is run 100 times, but only the *best* time >>>>>>>>>>>>>>>> for each is >>>>>>>>>>>>>>>> used. >>>>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>>>> Number of Threads requested = 64 >>>>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> < skiped > >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Function Rate (MB/s) Avg time Min time >>>>>>>>>>>>>>>> Max time >>>>>>>>>>>>>>>> Copy: 362331.5200 0.0060 0.0059 >>>>>>>>>>>>>>>> 0.0061 >>>>>>>>>>>>>>>> Scale: 359281.9806 0.0060 0.0060 >>>>>>>>>>>>>>>> 0.0062 >>>>>>>>>>>>>>>> Add: 418122.7024 0.0078 0.0077 >>>>>>>>>>>>>>>> 0.0081 >>>>>>>>>>>>>>>> Triad: 418122.7024 0.0078 0.0077 >>>>>>>>>>>>>>>> 0.0081 >>>>>>>>>>>>>>>> Store: 224685.6729 0.0048 0.0048 >>>>>>>>>>>>>>>> 0.0049 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 408GB/s >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 04/13/2016 12:15 PM, Raman, Karthik wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2. Effective MCDRAM bandwidth is around 406-408 GB/s. >>>>>>>>>>>>>>>> Larger values are possible for more high core count bins. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> For bin3 , you should be able to get Triad ~440 GB/s. >>>>>>>>>>>>>>>> Please >>>>>>>>>>>>>>>> check if you use the following Intel compiler flags >>>>>>>>>>>>>>>> -mcmodel >>>>>>>>>>>>>>>> medium -shared-intel -O3 -xMIC-AVX512 >>>>>>>>>>>>>>>> -DSTREAM_ARRAY_SIZE=134217728 -DOFFSET=0 -DNTIMES=100 >>>>>>>>>>>>>>>> -openmp -opt-streaming-stores always The array size is >>>>>>>>>>>>>>>> just allocating 1GB per array. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Sent: Wednesday, April 13, 2016 10:01 AM >>>>>>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com> >>>>>>>>>>>>>>>> <mailto:[email protected]>; Chinthamani, Sundaram >>>>>>>>>>>>>>>> <sundaram.chinthamani(a)intel.com> >>>>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>>>> Kumaran, Kalyan <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com> >>>>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>>>> Fromkin, Russ <russ.fromkin(a)intel.com> >>>>>>>>>>>>>>>> <mailto:[email protected]>; Harms, Kevin >>>>>>>>>>>>>>>> <harms(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>>> Morozov, Vitali A. >>>>>>>>>>>>>>>> <morozov(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>>> Allen, Benjamin S. >>>>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> <mailto:[email protected]> >>>>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> so, this is my understanding that on bin3 hardware (64 >>>>>>>>>>>>>>>> cores, 6 >>>>>>>>>>>>>>>> channel DDR4-2133) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1. Effective DDR bandwidth is 80 GB/s (around 78% of peak). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2. Effective MCDRAM bandwidth is around 406-408 GB/s. >>>>>>>>>>>>>>>> Larger values are possible for more high core count bins. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 3. Alignment is not important for STREAM triad benchmark. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 4. For both static and dynamic variables, numactl -m >>>>>>>>>>>>>>>> [0|1] should be used to direct memory placement. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 5. For explicit MCDRAM allocation in flat mode, >>>>>>>>>>>>>>>> hbw_malloc.h API should be use. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Please correct if I am wrong... >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks a lot for you help! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 04/12/2016 11:07 AM, Vitali A. Morozov wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Karthik, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I was also wandering how the variable could be >>>>>>>>>>>>>>>> influencing the allocation if, in principle, I do not >>>>>>>>>>>>>>>> need to link against memkind in my case. Anyway, sorry >>>>>>>>>>>>>>>> for confusion! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I think I did that before - see my first message. I was >>>>>>>>>>>>>>>> using "numactl --membind=1 ./stream" , which I believe >>>>>>>>>>>>>>>> is equivalent to "numactl -m 1 ./stream" that you suggest. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Here is the result of "numactl -H": >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [morozov@knl00 512_3_C64_MPI16_TH08_flat]$ numactl -H >>>>>>>>>>>>>>>> available: 2 nodes (0-1) >>>>>>>>>>>>>>>> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 >>>>>>>>>>>>>>>> 17 18 19 20 >>>>>>>>>>>>>>>> 21 >>>>>>>>>>>>>>>> 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 >>>>>>>>>>>>>>>> 40 41 42 43 >>>>>>>>>>>>>>>> 44 >>>>>>>>>>>>>>>> 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 >>>>>>>>>>>>>>>> 63 64 65 66 >>>>>>>>>>>>>>>> 67 >>>>>>>>>>>>>>>> 68 69 70 >>>>>>>>>>>>>>>> 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 >>>>>>>>>>>>>>>> 89 90 91 92 >>>>>>>>>>>>>>>> 93 >>>>>>>>>>>>>>>> 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 >>>>>>>>>>>>>>>> 109 110 111 >>>>>>>>>>>>>>>> 112 >>>>>>>>>>>>>>>> 113 114 115 116 117 118 119 120 121 122 123 124 125 126 >>>>>>>>>>>>>>>> 127 128 >>>>>>>>>>>>>>>> 129 >>>>>>>>>>>>>>>> 130 131 132 133 134 135 136 137 138 139 140 141 142 >>>>>>>>>>>>>>>> 143 144 145 146 147 148 149 150 151 152 153 154 155 156 >>>>>>>>>>>>>>>> 157 158 >>>>>>>>>>>>>>>> 159 >>>>>>>>>>>>>>>> 160 161 162 163 164 165 166 167 168 169 170 171 172 173 >>>>>>>>>>>>>>>> 174 175 >>>>>>>>>>>>>>>> 176 >>>>>>>>>>>>>>>> 177 178 179 180 181 182 183 184 185 186 187 188 189 190 >>>>>>>>>>>>>>>> 191 192 >>>>>>>>>>>>>>>> 193 >>>>>>>>>>>>>>>> 194 195 196 197 198 199 200 201 202 203 204 205 206 >>>>>>>>>>>>>>>> 207 208 209 210 211 212 213 214 215 216 217 218 219 220 >>>>>>>>>>>>>>>> 221 222 >>>>>>>>>>>>>>>> 223 >>>>>>>>>>>>>>>> 224 225 226 227 228 229 230 231 232 233 234 235 236 237 >>>>>>>>>>>>>>>> 238 239 >>>>>>>>>>>>>>>> 240 >>>>>>>>>>>>>>>> 241 242 243 244 245 246 247 248 249 250 251 252 253 254 >>>>>>>>>>>>>>>> 255 node >>>>>>>>>>>>>>>> 0 >>>>>>>>>>>>>>>> size: 196511 MB node 0 free: 186888 MB node 1 cpus: >>>>>>>>>>>>>>>> node 1 size: 16384 MB >>>>>>>>>>>>>>>> node 1 free: 11731 MB >>>>>>>>>>>>>>>> node distances: >>>>>>>>>>>>>>>> node 0 1 >>>>>>>>>>>>>>>> 0: 10 31 >>>>>>>>>>>>>>>> 1: 31 10 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Sounds like all as expected. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [morozov@knl00 STREAM]$ OMP_NUM_THREADS=64 >>>>>>>>>>>>>>>> KMP_AFFINITY=scatter numactl -m 1 ./stream >>>>>>>>>>>>>>>> Triad: 418262.7498 0.0058 0.0057 >>>>>>>>>>>>>>>> 0.0060 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> For a series of runs, I see this: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> # MB N GB/s >>>>>>>>>>>>>>>> 228.88 10000000 408867.98 >>>>>>>>>>>>>>>> 457.76 20000000 405818.57 >>>>>>>>>>>>>>>> 686.65 30000000 414309.08 >>>>>>>>>>>>>>>> 915.53 40000000 415063.59 >>>>>>>>>>>>>>>> 1144.41 50000000 415792.22 >>>>>>>>>>>>>>>> 1373.29 60000000 415449.01 >>>>>>>>>>>>>>>> 1602.17 70000000 414106.18 >>>>>>>>>>>>>>>> 1831.05 80000000 416846.82 >>>>>>>>>>>>>>>> 2059.94 90000000 414992.29 >>>>>>>>>>>>>>>> 2288.82 100000000 417898.11 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 04/12/2016 10:32 AM, Raman, Karthik wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Vitali, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1. >>>>>>>>>>>>>>>> The MEMKIND_HBW_NODES is *only* when you are allocating >>>>>>>>>>>>>>>> using >>>>>>>>>>>>>>>> hbwalloc (+ MCDRAM flat) in your application and when >>>>>>>>>>>>>>>> memkind is >>>>>>>>>>>>>>>> unable to find the high bandwidth nodes (pmtt not >>>>>>>>>>>>>>>> populated) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2. >>>>>>>>>>>>>>>> With the below >>>>>>>>>>>>>>>> static allocations, in MCDRAM Flat mode, you can simply >>>>>>>>>>>>>>>> use ï¿½numactl ï¿½m 1 ./streamï¿½ with propery affinity >>>>>>>>>>>>>>>> settings to get peak out of MCDRAM. >>>>>>>>>>>>>>>> a. >>>>>>>>>>>>>>>> Check ï¿½numactl ï¿½Hï¿½ to find the NUMA domains in your >>>>>>>>>>>>>>>> node. If >>>>>>>>>>>>>>>> booted in All2All/FLAT or Quad/Flat mode it should show >>>>>>>>>>>>>>>> 2 NUMA >>>>>>>>>>>>>>>> domains , one for DDR (96 GB) and other for MCDRAM (16GB) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>> Karthik >>>>>>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Sent: Tuesday, April 12, 2016 7:27 AM >>>>>>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com> >>>>>>>>>>>>>>>> <mailto:[email protected]>; Chinthamani, Sundaram >>>>>>>>>>>>>>>> <sundaram.chinthamani(a)intel.com> >>>>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>>>> Kumaran, Kalyan <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com> >>>>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>>>> Fromkin, Russ <russ.fromkin(a)intel.com> >>>>>>>>>>>>>>>> <mailto:[email protected]>; Harms, Kevin >>>>>>>>>>>>>>>> <harms(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>>> Morozov, Vitali A. >>>>>>>>>>>>>>>> <morozov(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>>> Allen, Benjamin S. >>>>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> <mailto:[email protected]> >>>>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Karthik, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The node is in All2All mode for now. I made the >>>>>>>>>>>>>>>> following changes. The original code is using static >>>>>>>>>>>>>>>> allocation therefore, I follow the advice on 2MB static >>>>>>>>>>>>>>>> array alignment: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [morozov@thing00 STREAM]$ head -n 108 < stream.c | tail >>>>>>>>>>>>>>>> -n 4 >>>>>>>>>>>>>>>> static double a[N+OFFSET] >>>>>>>>>>>>>>>> __attribute__((aligned(2097152))), >>>>>>>>>>>>>>>> b[N+OFFSET] >>>>>>>>>>>>>>>> __attribute__((aligned(2097152))), >>>>>>>>>>>>>>>> c[N+OFFSET] >>>>>>>>>>>>>>>> __attribute__((aligned(2097152))), >>>>>>>>>>>>>>>> d[N+OFFSET] >>>>>>>>>>>>>>>> __attribute__((aligned(2097152))); >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [morozov@knl00 STREAM]$ make >>>>>>>>>>>>>>>> mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp >>>>>>>>>>>>>>>> -mcmodel >>>>>>>>>>>>>>>> medium -shared-intel -fno-alias stream.c -o stream >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [morozov@knl00 STREAM]$ MEMKIND_HBW_NODES=1 >>>>>>>>>>>>>>>> OMP_NUM_THREADS=64 >>>>>>>>>>>>>>>> KMP_AFFINITY=scatter ./stream >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ..... >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Triad: 80251.3621 0.0300 0.0299 >>>>>>>>>>>>>>>> 0.0305 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Apparently, the "MEMKIND_HBW_NODES=1" is ignored. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 04/12/2016 01:51 AM, Raman, Karthik wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Also could you please modify the memory allocations in >>>>>>>>>>>>>>>> STREAM to align at 2MB boundary. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> For example: >>>>>>>>>>>>>>>> static allocations: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> static STREAM_TYPE a[STREAM_ARRAY_SIZE+OFFSET] >>>>>>>>>>>>>>>> __attribute__((aligned(2097152))) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> dynamic allocations: >>>>>>>>>>>>>>>> a = (STREAM_TYPE >>>>>>>>>>>>>>>> *)_mm_malloc(sizeof(STREAM_TYPE)*(STREAM_ARRAY_SIZE+OFFSET),20971 >>>>>>>>>>>>>>>> 5 >>>>>>>>>>>>>>>> 2 >>>>>>>>>>>>>>>> ) >>>>>>>>>>>>>>>> ; >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> memkind/hbwalloc: >>>>>>>>>>>>>>>> (STREAM_TYPE *)hbw_posix_memalign((void **)&a, >>>>>>>>>>>>>>>> 2097152, >>>>>>>>>>>>>>>> sizeof(STREAM_TYPE)*(STREAM_ARRAY_SIZE+OFFSET)) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> For memkind/hbwmalloc to work out of the box, you need >>>>>>>>>>>>>>>> to have >>>>>>>>>>>>>>>> the PMTT tables populated so that it can identify the >>>>>>>>>>>>>>>> high bandwidth nodes. Alternatively you can set the >>>>>>>>>>>>>>>> MEMKIND_HBW_NODES environment variable to the high >>>>>>>>>>>>>>>> bandwidth node (MCDRAM). >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> e.g. All2All/Flat or Quad/Flat : export >>>>>>>>>>>>>>>> MEMKIND_HBW_NODES=1 ; ./app_with_hbwmalloc >>>>>>>>>>>>>>>> SNC4/Flat: export MEMKIND_HBW_NODES=4,5,6,7 ; >>>>>>>>>>>>>>>> ./app_with_hbw_malloc >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>> Karthik >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> From: Chinthamani, Sundaram >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Sent: Monday, April 11, 2016 8:15 PM >>>>>>>>>>>>>>>> To: Vitali Morozov <morozov(a)anl.gov> >>>>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>>>> Kumaran, Kalyan <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com> >>>>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>>>> Fromkin, Russ <russ.fromkin(a)intel.com> >>>>>>>>>>>>>>>> <mailto:[email protected]>; Raman, Karthik >>>>>>>>>>>>>>>> <karthik.raman(a)intel.com> >>>>>>>>>>>>>>>> <mailto:[email protected]>; Harms, Kevin >>>>>>>>>>>>>>>> <harms(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>>> Morozov, Vitali A. >>>>>>>>>>>>>>>> <morozov(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>>> Allen, Benjamin S. >>>>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>>> Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com> >>>>>>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>>>>>> Subject: RE: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> You should use numactl ï¿½m 4,5,6,7 to get the memory >>>>>>>>>>>>>>>> bandwidth out of MC-DRAM. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> What kind of part is it? Is it bin1 or bin3? Even in >>>>>>>>>>>>>>>> All2All mode, we have measured up to 470 GB/s of >>>>>>>>>>>>>>>> stream triad bandwidth. Quad mode should get 490 GB/s. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> DDR-stream triad bandwidth on Bin1-part should be close >>>>>>>>>>>>>>>> to 90 GB/s. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -Sundaram >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> From: Vitali Morozov [mailto:[email protected]] >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Sent: Monday, April 11, 2016 6:59 PM >>>>>>>>>>>>>>>> To: Kumaran, Kalyan <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; >>>>>>>>>>>>>>>> Fromkin, Russ >>>>>>>>>>>>>>>> <russ.fromkin(a)intel.com>; Chinthamani, Sundaram >>>>>>>>>>>>>>>> <sundaram.chinthamani(a)intel.com>; Raman, Karthik >>>>>>>>>>>>>>>> <karthik.raman(a)intel.com>; Harms, Kevin >>>>>>>>>>>>>>>> <harms(a)alcf.anl.gov>; >>>>>>>>>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov>; Allen, >>>>>>>>>>>>>>>> Benjamin S. >>>>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Avinash and others, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> here is what we have so far for Flat more, All2All. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The results from memory were obtained by using default >>>>>>>>>>>>>>>> STREAM with static allocation. The results from MCDRAM >>>>>>>>>>>>>>>> were obtained by using modified STREAM with hbwmalloc.h >>>>>>>>>>>>>>>> interface. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1 MPI process, 64 threads, scatter affinity, no numactl >>>>>>>>>>>>>>>> is used. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Most results are as expected: 80 GB/s from DDR memory, >>>>>>>>>>>>>>>> 420 GB/s from MCDRAM, the MCDRAM size is about 16 GB, >>>>>>>>>>>>>>>> the aggregate L2 cache size is 32 MB. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Noticed inconsistencies for hbwmalloc benchmarks: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 1. The result is not validated - will spend more time >>>>>>>>>>>>>>>> on that tomorrow. >>>>>>>>>>>>>>>> 2. hbw_malloc continues allocating data beyond the >>>>>>>>>>>>>>>> expected 16 GB up to 48 GB. This means it fails back to >>>>>>>>>>>>>>>> DDR if it cannot allocate in MCDRAM - not expected >>>>>>>>>>>>>>>> default behavior. It must fail if it does not have >>>>>>>>>>>>>>>> enough space. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The problem shows up when I am running static >>>>>>>>>>>>>>>> allocation and use numactl --membind=1 to ask for >>>>>>>>>>>>>>>> MCDRAM-only allocation. The variation is very large and >>>>>>>>>>>>>>>> inconsistent with the result observed when using >>>>>>>>>>>>>>>> hbw_malloc API: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> # MB N GB/s numactl GB/s >>>>>>>>>>>>>>>> hbw_malloc >>>>>>>>>>>>>>>> 228.88 10000000 255555.46 421009.18 >>>>>>>>>>>>>>>> 457.76 20000000 363930.93 416997.91 >>>>>>>>>>>>>>>> 686.65 30000000 309415.87 427808.31 >>>>>>>>>>>>>>>> 915.53 40000000 371691.30 423310.75 >>>>>>>>>>>>>>>> 1144.41 50000000 187002.22 431032.35 >>>>>>>>>>>>>>>> 1373.29 60000000 385954.23 427172.91 >>>>>>>>>>>>>>>> 1602.17 70000000 388254.49 426823.59 >>>>>>>>>>>>>>>> 1831.05 80000000 374491.43 428103.96 >>>>>>>>>>>>>>>> 2059.94 90000000 379622.74 428152.02 >>>>>>>>>>>>>>>> 2288.82 100000000 80868.34 430185.03 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> We also tried SNC-4 instead of All2All, but could not >>>>>>>>>>>>>>>> get more than 40 GB/s from MCDRAM even with hbw_malloc API. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Raw data: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> # flat memory, running from memory (default), alltoall >>>>>>>>>>>>>>>> # MB N GB/s >>>>>>>>>>>>>>>> 2.29 100000 272062.96 >>>>>>>>>>>>>>>> 4.58 200000 491040.47 >>>>>>>>>>>>>>>> 6.87 300000 603979.78 >>>>>>>>>>>>>>>> 9.16 400000 745654.04 >>>>>>>>>>>>>>>> 11.44 500000 811800.77 >>>>>>>>>>>>>>>> 13.73 600000 901462.35 >>>>>>>>>>>>>>>> 16.02 700000 891953.26 >>>>>>>>>>>>>>>> 18.31 800000 1019375.15 >>>>>>>>>>>>>>>> 20.60 900000 984749.63 >>>>>>>>>>>>>>>> 22.89 1000000 923516.48 >>>>>>>>>>>>>>>> 25.18 1100000 758422.09 >>>>>>>>>>>>>>>> 27.47 1200000 779328.74 >>>>>>>>>>>>>>>> 29.75 1300000 513185.43 >>>>>>>>>>>>>>>> 32.04 1400000 155550.35 >>>>>>>>>>>>>>>> 34.33 1500000 103209.12 >>>>>>>>>>>>>>>> 36.62 1600000 97435.74 >>>>>>>>>>>>>>>> 38.91 1700000 83968.40 >>>>>>>>>>>>>>>> 41.20 1800000 83692.35 >>>>>>>>>>>>>>>> 43.49 1900000 80734.60 >>>>>>>>>>>>>>>> 45.78 2000000 76317.89 >>>>>>>>>>>>>>>> 68.66 3000000 81180.08 >>>>>>>>>>>>>>>> 91.55 4000000 78889.73 >>>>>>>>>>>>>>>> 114.44 5000000 79891.50 >>>>>>>>>>>>>>>> 137.33 6000000 79954.96 >>>>>>>>>>>>>>>> 160.22 7000000 79477.00 >>>>>>>>>>>>>>>> 183.11 8000000 79145.59 >>>>>>>>>>>>>>>> 205.99 9000000 79821.12 >>>>>>>>>>>>>>>> 228.88 10000000 79519.15 >>>>>>>>>>>>>>>> 228.88 10000000 79212.54 >>>>>>>>>>>>>>>> 457.76 20000000 79692.27 >>>>>>>>>>>>>>>> 686.65 30000000 79866.15 >>>>>>>>>>>>>>>> 915.53 40000000 79828.15 >>>>>>>>>>>>>>>> 1144.41 50000000 80214.91 >>>>>>>>>>>>>>>> 1373.29 60000000 80226.84 >>>>>>>>>>>>>>>> 1602.17 70000000 80178.77 >>>>>>>>>>>>>>>> 1831.05 80000000 80177.06 >>>>>>>>>>>>>>>> 2059.94 90000000 80109.79 >>>>>>>>>>>>>>>> 2288.82 100000000 80175.94 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> # flat memory, running from MCDRAM, alltoall, >>>>>>>>>>>>>>>> hbwmalloc API >>>>>>>>>>>>>>>> # MB N GB/s >>>>>>>>>>>>>>>> 228.88 10000000 421009.18 >>>>>>>>>>>>>>>> 457.76 20000000 416997.91 >>>>>>>>>>>>>>>> 686.65 30000000 427808.31 >>>>>>>>>>>>>>>> 915.53 40000000 423310.75 >>>>>>>>>>>>>>>> 1144.41 50000000 431032.35 >>>>>>>>>>>>>>>> 1373.29 60000000 427172.91 >>>>>>>>>>>>>>>> 1602.17 70000000 426823.59 >>>>>>>>>>>>>>>> 1831.05 80000000 428103.96 >>>>>>>>>>>>>>>> 2059.94 90000000 428152.02 >>>>>>>>>>>>>>>> 2288.82 100000000 430185.03 >>>>>>>>>>>>>>>> 2343.75 102400000 425999.98 >>>>>>>>>>>>>>>> 4687.50 204800000 427821.10 >>>>>>>>>>>>>>>> 9155.27 400000000 429318.13 >>>>>>>>>>>>>>>> 9375.00 409600000 428376.70 >>>>>>>>>>>>>>>> 11444.09 500000000 428052.08 >>>>>>>>>>>>>>>> 13732.91 600000000 266061.01 >>>>>>>>>>>>>>>> 16021.73 700000000 200257.79 >>>>>>>>>>>>>>>> 18310.55 800000000 168164.19 >>>>>>>>>>>>>>>> 20599.37 900000000 150192.91 >>>>>>>>>>>>>>>> 22888.18 1000000000 137862.07 >>>>>>>>>>>>>>>> 25177.00 1100000000 129418.12 >>>>>>>>>>>>>>>> 27465.82 1200000000 123019.53 >>>>>>>>>>>>>>>> 29754.64 1300000000 118268.79 >>>>>>>>>>>>>>>> 32043.46 1400000000 114343.33 >>>>>>>>>>>>>>>> 34332.28 1500000000 110991.25 >>>>>>>>>>>>>>>> 36621.09 1600000000 108538.36 >>>>>>>>>>>>>>>> 38909.91 1700000000 106249.72 >>>>>>>>>>>>>>>> 41198.73 1800000000 104455.07 >>>>>>>>>>>>>>>> 43487.55 1900000000 102828.93 >>>>>>>>>>>>>>>> 45776.37 2000000000 101348.56 >>>>>>>>>>>>>>>> 48065.19 2100000000 100013.73 >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> We appreciate any help on that. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 04/11/2016 08:42 PM, Kumaran, Kalyan wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Avinash, thanks for the prompt reply. Kevin/Vitali can >>>>>>>>>>>>>>>> provide >>>>>>>>>>>>>>>> all the details. The short summary is we are unable to >>>>>>>>>>>>>>>> reproduce the STREAM results from IPM or DDR. It would >>>>>>>>>>>>>>>> be great if Karthik or Sundaram can advise us on how to >>>>>>>>>>>>>>>> run (which mode etc) or if they could log into the >>>>>>>>>>>>>>>> machine and make sure we have set up things correctly. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Kumar >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Apr 11, 2016, at 8:07 PM, Sodani, Avinash >>>>>>>>>>>>>>>> <avinash.sodani(a)intel.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Copying Sundaram/Karthik from our side. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Kumar, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Can you send us more information on this? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -Avinash >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> From: Kumaran, >>>>>>>>>>>>>>>> Kalyan [mailto:[email protected]] >>>>>>>>>>>>>>>> Sent: Monday, April 11, 2016 5:32 PM >>>>>>>>>>>>>>>> To: Fromkin, Russ; Sodani, Avinash >>>>>>>>>>>>>>>> Cc: Harms, Kevin; Morozov, Vitali A. >>>>>>>>>>>>>>>> Subject: Fwd: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Russ, Avinash, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> we need help in understanding how to obtain STREAM >>>>>>>>>>>>>>>> results. Can anyone help? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Kumar >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Begin forwarded message: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> From: "Harms, Kevin" <harms(a)alcf.anl.gov> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Subject: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Date: April 11, 2016 at 5:42:15 PM CDT >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> To: "intel-nda(a)jlse.anl.gov" >>>>>>>>>>>>>>>> <intel-nda(a)jlse.anl.gov> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> The knl00 node is currently in Flat mode and All2All >>>>>>>>>>>>>>>> model. With this setup we can get 420 GB/s from IPM >>>>>>>>>>>>>>>> memory on STREAM Triad when using hbw_malloc and >>>>>>>>>>>>>>>> friends which was confirmed by both Vitali and I. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Changing the BIOS settings and warm rebooting seem to >>>>>>>>>>>>>>>> be problematic. Vitali and I were running earlier and >>>>>>>>>>>>>>>> could not get over 80 GB/s with any method even though >>>>>>>>>>>>>>>> memory capacity indicated flat mode. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Using numactl --membind is also problematic. The STREAM >>>>>>>>>>>>>>>> results >>>>>>>>>>>>>>>> when using this to launch the application were erratic, >>>>>>>>>>>>>>>> so it is also suggested not to use that. That means you >>>>>>>>>>>>>>>> will need to modify your code to use dynamic memory and >>>>>>>>>>>>>>>> hbw_malloc to exploit the IPM. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Will followup with more detailed results later. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> kevin >>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>> Intel-nda mailing list >>>>>>>>>>>>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>>>>>>>>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> <flat_a2a_ddr.txt><flat_a2a_ipm.txt><flat_quad_ddr.txt><flat_quad_ipm.txt><flat_snc4_ddr.txt><flat_snc4_ipm.txt> >>>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Intel-nda mailing list >>>>> Intel-nda(a)lists.jlse.anl.gov >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>> >>> >>> >>> >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> >> >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] KNL Status
by Scott Parker 21 Apr '16

21 Apr '16

It's not the compiler, I see no significant difference in SNC-4 mode between Intel 16 and Intel 17 Beta. >icc -V Intel(R) C Intel(R) 64 Compiler for applications running on Intel(R) 64, Version 16.0.0.109 Build 20150815 Copyright (C) 1985-2015 Intel Corporation. All rights reserved. >icc -O3 -fopenmp -DSTREAM_ARRAY_SIZE=134217728 -xMIC-AVX512 -mcmodel medium -shared-intel stream.c -o stream_c.exe >OMP_NUM_THREADS=128 numactl -m 4,5,6,7 ./stream_c.exe Function Best Rate MB/s Avg time Min time Max time Copy: 390734.0 0.005547 0.005496 0.005631 Scale: 411250.1 0.005309 0.005222 0.005569 Add: 449326.5 0.007190 0.007169 0.007220 Triad: 440248.9 0.007343 0.007317 0.007391 On 4/21/16 10:12 AM, Vitali A Morozov wrote: > Yeah - as Ronald Green has figured out, the beta version of a > compiler (17) does make something differently compared to production > version 16. For some reason, it only matters in SNC-4 mode and not in > All2All mode. > > I am curious what was changed (or added) in version 17... > > Vitali > > > > On 4/21/16 9:45 AM, Kumaran, Kalyan wrote: >> Looks like Scott Parker gets better results: >> >> Adding another data point, I ran in SNC-4 mode with the configuration >> below and observed a consistent BW around 440 GB/s >> with the best run producing 442 GB/s. In this case 128 threads gave >> the best results. >> >> I've noticed that the STREAM results sensitivity to build and run >> parameters seems to vary depending on the memory configuration >> in flat-all2all I saw noticeably better results with memory aligned >> on 2MB boundaries and 64 threads, in SNC-4 mode the alignment >> doesn't seem to matter and 128 threads is producing the best results. >> >> >> Mode: flat & SNC-4 >> Compiler: Intel 17 Beta >> Compile Options: icc -O3 -fopenmp -DSTREAM_ARRAY_SIZE=134217728 >> -xMIC-AVX512 -mcmodel medium -shared-intel >> Run Command: OMP_NUM_THREADS=128 numactl -m 4,5,6,7 ./stream_mod >> >> Results: >> Function Best Rate MB/s Avg time Min time Max time >> Copy: 390378.3 0.005524 0.005501 0.005546 >> Scale: 412738.8 0.005237 0.005203 0.005278 >> Add: 448759.4 0.007199 0.007178 0.007218 >> Triad: 441919.3 0.007355 0.007289 0.007548 >> >> >> >>> On Apr 20, 2016, at 7:39 PM, Vitali A Morozov <morozov(a)anl.gov >>> <mailto:[email protected]>> wrote: >>> >>> Hi, >>> >>> a quick update on STREAM with the latest beta compiler >>> >>> [morozov@knl00 STREAM]$ icc -V >>> Intel(R) C Intel(R) 64 Compiler for applications running on Intel(R) >>> 64, Version 17.0.0.042 Beta Build 20160315 >>> >>> compiler options >>> mpiicc -O3 -xMIC-AVX512 -qopt-report=5 -Wall -g -fopenmp -mcmodel >>> medium -shared-intel -qopt-streaming-stores always -restrict >>> stream.c -o stream >>> >>> Running in flat mode all2all >>> >>> morozov@knl00 STREAM]$ export OMP_NUM_THREADS=256 >>> [morozov@knl00 STREAM]$ numactl -m 1 ./stream >>> ------------------------------------------------------------- >>> STREAM version $Revision: 5.9 $ >>> ------------------------------------------------------------- >>> This system uses 8 bytes per DOUBLE PRECISION word. >>> ------------------------------------------------------------- >>> Array size = 134217728, Offset = 0 >>> Total memory required = 3072.0 MB. >>> Each test is run 500 times, but only >>> the *best*time for each is used. >>> ------------------------------------------------------------- >>> Number of Threads requested = 256 >>> ------------------------------------------------------------- >>> <skipped> >>> ------------------------------------------------------------- >>> Your clock granularity/precision appears to be 1 microseconds. >>> Each test below will take on the order of 6356 microseconds. >>> (= 6356 clock ticks) >>> Increase the size of the arrays if this shows that >>> you are not getting at least 20 clock ticks per test. >>> ------------------------------------------------------------- >>> WARNING -- The above is only a rough guideline. >>> For best results, please be sure you know the >>> precision of your system timer. >>> ------------------------------------------------------------- >>> Function Rate (MB/s) Avg time Min time Max time >>> Copy: 330891.5637 0.0066 0.0065 0.0078 >>> Scale: 330127.5200 0.0066 0.0065 0.0069 >>> Add: 383622.4448 0.0085 0.0084 0.0093 >>> Triad: 383252.4574 0.0085 0.0084 0.0098 >>> ------------------------------------------------------------- >>> Solution Validates >>> ------------------------------------------------------------- >>> >>> Same with 64 threads scatter mode: >>> >>> morozov@knl00 STREAM]$ export OMP_NUM_THREADS=64 >>> [morozov@knl00 STREAM]$ KMP_AFFINITY=scatter >>> [morozov@knl00 STREAM]$ numactl -m 1 ./stream >>> ------------------------------------------------------------- >>> STREAM version $Revision: 5.9 $ >>> ------------------------------------------------------------- >>> This system uses 8 bytes per DOUBLE PRECISION word. >>> ------------------------------------------------------------- >>> Array size = 134217728, Offset = 0 >>> Total memory required = 3072.0 MB. >>> Each test is run 500 times, but only >>> the *best*time for each is used. >>> ------------------------------------------------------------- >>> Number of Threads requested = 64 >>> ------------------------------------------------------------- >>> <skipped> >>> ------------------------------------------------------------- >>> Your clock granularity/precision appears to be 1 microseconds. >>> Each test below will take on the order of 6056 microseconds. >>> (= 6056 clock ticks) >>> Increase the size of the arrays if this shows that >>> you are not getting at least 20 clock ticks per test. >>> ------------------------------------------------------------- >>> WARNING -- The above is only a rough guideline. >>> For best results, please be sure you know the >>> precision of your system timer. >>> ------------------------------------------------------------- >>> Function Rate (MB/s) Avg time Min time Max time >>> Copy: 353486.8826 0.0061 0.0061 0.0067 >>> Scale: 353265.0608 0.0061 0.0061 0.0066 >>> Add: 423961.3055 0.0077 0.0076 0.0079 >>> Triad: 424907.9750 0.0076 0.0076 0.0084 >>> ------------------------------------------------------------- >>> Solution Validates >>> ------------------------------------------------------------- >>> >>> Vitali >>> >>> >>> >>> >>> On 4/18/16 12:43 PM, Kumaran, Kalyan wrote: >>>> FYI. Karthik will get on the machine to debug. >>>> >>>> Sent from my iPhone >>>> >>>> Begin forwarded message: >>>> >>>>> *From:* "Chinthamani, Sundaram" <sundaram.chinthamani(a)intel.com> >>>>> *Date:* April 18, 2016 at 2:05:26 PM CDT >>>>> *To:* "Harms, Kevin" <harms(a)alcf.anl.gov>, "Allen, Benjamin S." >>>>> <bsallen(a)alcf.anl.gov <mailto:[email protected]>> >>>>> *Cc:* "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov>, "Kaana >>>>> Balakrishna, Abhishek" <abhishek.kaana.balakrishna(a)intel.com>, >>>>> "Morozov, Vitali A." <morozov(a)alcf.anl.gov >>>>> <mailto:[email protected]>>, "Raman, Karthik" >>>>> <karthik.raman(a)intel.com <mailto:[email protected]>>, >>>>> "Inkley, Benson" <benson.inkley(a)intel.com >>>>> <mailto:[email protected]>>, "Sodani, Avinash" >>>>> <avinash.sodani(a)intel.com <mailto:[email protected]>>, >>>>> "Fromkin, Russ" <russ.fromkin(a)intel.com >>>>> <mailto:[email protected]>>, "Chinthamani, Sundaram" >>>>> <sundaram.chinthamani(a)intel.com> >>>>> *Subject:* *RE: [JLSE-Intel-NDA] KNL Status* >>>>> >>>>> I would expect Quad mode / SNC-4 mode to provide higher bandwidth >>>>> compared to All2All mode. We haven't seen this in our internal >>>>> measurements. >>>>> >>>>> -Thanks, >>>>> -Sundaram >>>>> >>>>> -----Original Message----- >>>>> From: Harms, Kevin [mailto:[email protected]] >>>>> Sent: Monday, April 18, 2016 12:00 PM >>>>> To: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >>>>> Cc: Kumaran, Kalyan <kumaran(a)alcf.anl.gov>; Kaana Balakrishna, >>>>> Abhishek <abhishek.kaana.balakrishna(a)intel.com>; Morozov, Vitali >>>>> A. <morozov(a)alcf.anl.gov <mailto:[email protected]>>; >>>>> Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com>; Raman, >>>>> Karthik <karthik.raman(a)intel.com >>>>> <mailto:[email protected]>>; Inkley, Benson >>>>> <benson.inkley(a)intel.com <mailto:[email protected]>>; >>>>> Sodani, Avinash <avinash.sodani(a)intel.com >>>>> <mailto:[email protected]>>; Fromkin, Russ >>>>> <russ.fromkin(a)intel.com <mailto:[email protected]>> >>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>> >>>>> >>>>> I tested cached mode with all2all, snc4 and quad models. Cached >>>>> all2all gave 275,000 MB/s. Does that look reasonable? SNC4 and >>>>> quad mode gave 225,000 MB/s or so. >>>>> >>>>> kevin >>>>> >>>>> >>>>> >>>>> >>>>>> The following versions are now installed: BIOS 01.01.0104, ME >>>>>> 03.01.03.011.0, BMC 00.17.9699, FRUSDR 1.14. >>>>>> >>>>>> Ben >>>>>> >>>>>>> On Apr 18, 2016, at 1:27 PM, Harms, Kevin <harms(a)alcf.anl.gov> >>>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> Ben completed the BIOS upgrade, but we do not see any difference >>>>>>> in performance. I have attached results from runs done under >>>>>>> three configurations: flat+all2all, flat+quad, flat+snc4. >>>>>>> >>>>>>> flat+all2all = 427046.0546 MB/s >>>>>>> flat+quad = 327253.8882 MB/s >>>>>>> flat+snc4 = 308555.9588 MB/s >>>>>>> >>>>>>> I compiled the CORAL stream.c with the following options: >>>>>>> icc -mcmodel medium -shared-intel -O3 -xMIC-AVX512 -DN=50000000 >>>>>>> -DOFFSET=0 -DNTIMES=100 -openmp -opt-streaming-stores always >>>>>>> stream.c -o stream >>>>>>> >>>>>>> I added the alignment directives to the arrays. The attached >>>>>>> files show the NUMA domains and numactl commands used and run >>>>>>> parameters. >>>>>>> >>>>>>> Have I missed a parameter? >>>>>>> >>>>>>> kevin >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Suggest a time Friday afternoon. We are in central time zone. >>>>>>>> We can get you access to this B0 too. >>>>>>>> >>>>>>>> -Kumar >>>>>>>> >>>>>>>> >>>>>>>>> On Apr 13, 2016, at 5:06 PM, Kaana Balakrishna, Abhishek >>>>>>>>> <abhishek.kaana.balakrishna(a)intel.com> wrote: >>>>>>>>> >>>>>>>>> + Benson >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Abhishek K Balakrishna (KB) >>>>>>>>> Xeon Phiï¿½ Hardware Platform Applications Engineer, >>>>>>>>> Technical Computing Group (TCG) >>>>>>>>> Intel Corporation >>>>>>>>> Ph - 512 466 9886 >>>>>>>>> www.intel.com/xeonphi <http://www.intel.com/xeonphi> >>>>>>>>> >>>>>>>>> >>>>>>>>> -----Original Message----- >>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>> Sent: Wednesday, April 13, 2016 4:58 PM >>>>>>>>> To: Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com>; >>>>>>>>> Raman, Karthik <karthik.raman(a)intel.com>; Harms, Kevin >>>>>>>>> <harms(a)alcf.anl.gov <mailto:[email protected]>>; Morozov, >>>>>>>>> Vitali A. <morozov(a)alcf.anl.gov>; Kumaran, Kalyan >>>>>>>>> <kumaran(a)alcf.anl.gov> >>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; Fromkin, Russ >>>>>>>>> <russ.fromkin(a)intel.com>; Allen, Benjamin S. >>>>>>>>> <bsallen(a)alcf.anl.gov>; Kaana Balakrishna, Abhishek >>>>>>>>> <abhishek.kaana.balakrishna(a)intel.com> >>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>> >>>>>>>>> Sure! >>>>>>>>> >>>>>>>>> Vitali >>>>>>>>> >>>>>>>>> >>>>>>>>> On 04/13/2016 04:54 PM, Chinthamani, Sundaram wrote: >>>>>>>>>> Would it be possible to have a meeting to go over your >>>>>>>>>> questions rather than exchanging E-mails back and forth? >>>>>>>>>> >>>>>>>>>> + KB from Intel side. He can help set-up the meeting between >>>>>>>>>> Labs and Intel. >>>>>>>>>> >>>>>>>>>> -Thanks, >>>>>>>>>> -Sundaram >>>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>> Sent: Wednesday, April 13, 2016 2:50 PM >>>>>>>>>> To: Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com>; >>>>>>>>>> Raman, >>>>>>>>>> Karthik <karthik.raman(a)intel.com>; Harms, Kevin >>>>>>>>>> <harms(a)alcf.anl.gov>; >>>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov>; Kumaran, Kalyan >>>>>>>>>> <kumaran(a)alcf.anl.gov> >>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; Fromkin, Russ >>>>>>>>>> <russ.fromkin(a)intel.com>; Allen, Benjamin S. >>>>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>> >>>>>>>>>> On 04/13/2016 04:42 PM, Chinthamani, Sundaram wrote: >>>>>>>>>>> node 4 free: 2907 MB >>>>>>>>>>> node 5 free: 2926 MB >>>>>>>>>>> node 6 free: 2922 MB >>>>>>>>>>> node 7 free: 2914 MB >>>>>>>>>>> >>>>>>>>>>> There is 16 GB of MC-DRAM and 4 GB per NUMA node. Is >>>>>>>>>>> there any other process running on the machine? Why does >>>>>>>>>>> "free memory" show only 3 GB per node? >>>>>>>>>> This is EXACTLY why I am asking this question. Nothing is >>>>>>>>>> running on the node - and I do not see full 4 GiB of >>>>>>>>>> available MCDRAM memory. This might be reason why I do not >>>>>>>>>> see what is expected. >>>>>>>>>> >>>>>>>>>> I understand that I can reduce the array size, but I am >>>>>>>>>> trying to reproduce your results. >>>>>>>>>> >>>>>>>>>> I am still unclear on if you run "no mpi" version, how come >>>>>>>>>> that SNC-4 mode is giving you advantage? All arrays are >>>>>>>>>> allocated in the same quadrant. Requests from the other >>>>>>>>>> quadrants will go to this one and create congestion. >>>>>>>>>> >>>>>>>>>> Vitali >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> If you want to test Stream triad bandwidth to MC-DRAM in >>>>>>>>>>> Quad/Flat mode, you can use even smaller arrays of few 100 MBs. >>>>>>>>>>> >>>>>>>>>>> -Thanks, >>>>>>>>>>> -Sundaram >>>>>>>>>>> >>>>>>>>>>> -----Original Message----- >>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>> Sent: Wednesday, April 13, 2016 2:32 PM >>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com>; Chinthamani, >>>>>>>>>>> Sundaram >>>>>>>>>>> <sundaram.chinthamani(a)intel.com>; Harms, Kevin >>>>>>>>>>> <harms(a)alcf.anl.gov>; >>>>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov>; Kumaran, Kalyan >>>>>>>>>>> <kumaran(a)alcf.anl.gov> >>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; Fromkin, Russ >>>>>>>>>>> <russ.fromkin(a)intel.com>; Allen, Benjamin S. >>>>>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>> >>>>>>>>>>> ok, that is fine. >>>>>>>>>>> >>>>>>>>>>> Now the next issue I have is that for static arrays, all >>>>>>>>>>> three arrays must land up on one of the numa nodes - the >>>>>>>>>>> node that is attached to a master thread. >>>>>>>>>>> >>>>>>>>>>> You advice to use N = 134217728, that gives exactly 3 GiB >>>>>>>>>>> of memory footprint for 3 arrays of triad. >>>>>>>>>>> >>>>>>>>>>> According to "numactl -H", I do not have 3 GiB of available >>>>>>>>>>> memory on any of the MCDRAM numa nodes: >>>>>>>>>>> >>>>>>>>>>> numactl -H |grep free >>>>>>>>>>> node 0 free: 46336 MB >>>>>>>>>>> node 1 free: 46923 MB >>>>>>>>>>> node 2 free: 46970 MB >>>>>>>>>>> node 3 free: 46961 MB >>>>>>>>>>> node 4 free: 2907 MB >>>>>>>>>>> node 5 free: 2926 MB >>>>>>>>>>> node 6 free: 2922 MB >>>>>>>>>>> node 7 free: 2914 MB >>>>>>>>>>> >>>>>>>>>>> Therefore, at least one array will not be placed to the same >>>>>>>>>>> numa node. >>>>>>>>>>> Is this ok? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 04/13/2016 04:17 PM, Raman, Karthik wrote: >>>>>>>>>>>> Yes we run pure OMP code too. >>>>>>>>>>>> >>>>>>>>>>>> The problem with KMP_AFFINITY with SNC4 mode (currently) is >>>>>>>>>>>> that it >>>>>>>>>>>> may not equally balance the threads per NUMA domain even with >>>>>>>>>>>> scatter or balanced, you can check that by mapping the >>>>>>>>>>>> "verbose" output (e.g. >>>>>>>>>>>> KMP_AFFINITY=scatter,verbose) with "numactl -H" output to >>>>>>>>>>>> see if the >>>>>>>>>>>> OS thread to logical CPU mapping is balanced across all >>>>>>>>>>>> NUMA domains >>>>>>>>>>>> >>>>>>>>>>>> With All2All/Quad this should not be a problem >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>> Sent: Wednesday, April 13, 2016 1:53 PM >>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com>; Chinthamani, >>>>>>>>>>>> Sundaram >>>>>>>>>>>> <sundaram.chinthamani(a)intel.com>; Harms, Kevin >>>>>>>>>>>> <harms(a)alcf.anl.gov>; >>>>>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov>; Kumaran, Kalyan >>>>>>>>>>>> <kumaran(a)alcf.anl.gov> >>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; Fromkin, Russ >>>>>>>>>>>> <russ.fromkin(a)intel.com>; Allen, Benjamin S. >>>>>>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>> >>>>>>>>>>>> Before doing that - let us check if we are running similar >>>>>>>>>>>> benchmark. >>>>>>>>>>>> >>>>>>>>>>>> I am running 1 MPI process (in fact no mpi at all) and 64 >>>>>>>>>>>> OpenMP threads version. The threads can be placed with >>>>>>>>>>>> "balanced" or "scattered" >>>>>>>>>>>> policy. I am using "scattered". >>>>>>>>>>>> >>>>>>>>>>>> Is this ok so far? >>>>>>>>>>>> >>>>>>>>>>>> Vitali >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 04/13/2016 03:48 PM, Raman, Karthik wrote: >>>>>>>>>>>>> This still does not look right, and the reason could be >>>>>>>>>>>>> that the >>>>>>>>>>>>> threads are not balanced across all the NUMA domains >>>>>>>>>>>>> (ideally you >>>>>>>>>>>>> want >>>>>>>>>>>>> 16 threads per quadrant) >>>>>>>>>>>>> >>>>>>>>>>>>> Before going to SNC4, can you please boot in "Quad" mode. >>>>>>>>>>>>> If you >>>>>>>>>>>>> look in the BIOS Cluster mode options there is another >>>>>>>>>>>>> mode called "Quad" >>>>>>>>>>>>> which is different from SNC4, only 2 NUMA domains (DDR, >>>>>>>>>>>>> MCDRAM). >>>>>>>>>>>>> Please run similar to All2All mode >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>>> Sent: Wednesday, April 13, 2016 12:56 PM >>>>>>>>>>>>> To: Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com >>>>>>>>>>>>> <mailto:[email protected]>>; Harms, >>>>>>>>>>>>> Kevin <harms(a)alcf.anl.gov>; Raman, Karthik >>>>>>>>>>>>> <karthik.raman(a)intel.com>; Morozov, Vitali A. >>>>>>>>>>>>> <morozov(a)alcf.anl.gov>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; Fromkin, Russ >>>>>>>>>>>>> <russ.fromkin(a)intel.com>; Allen, Benjamin S. >>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>> >>>>>>>>>>>>> ok, I am now in SNC-4/Flat. Same binary. >>>>>>>>>>>>> >>>>>>>>>>>>> OMP_NUM_THREADS=64 KMP_AFFINITY=scatter numactl -m 4,5,6,7 >>>>>>>>>>>>> ./stream >>>>>>>>>>>>> |tee running.dat >>>>>>>>>>>>> >>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>> STREAM version $Revision: 5.9 $ >>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>> This system uses 8 bytes per DOUBLE PRECISION word. >>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>> Array size = 134217728, Offset = 0 >>>>>>>>>>>>> Total memory required = 3072.0 MB. >>>>>>>>>>>>> Each test is run 100 times, but only the *best* time for >>>>>>>>>>>>> each is >>>>>>>>>>>>> used. >>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>> Number of Threads requested = 64 >>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>> >>>>>>>>>>>>> <skipped> >>>>>>>>>>>>> >>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>> Function Rate (MB/s) Avg time Min time Max time >>>>>>>>>>>>> Copy: 320997.8352 0.0068 0.0067 0.0071 >>>>>>>>>>>>> Scale: 319994.2893 0.0068 0.0067 0.0073 >>>>>>>>>>>>> Add: 306055.0205 0.0106 0.0105 0.0107 >>>>>>>>>>>>> Triad: 306055.0205 0.0106 0.0105 0.0109 >>>>>>>>>>>>> Store: 247056.8669 0.0044 0.0043 0.0051 >>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>> Solution Validates >>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 306055 MB/s >>>>>>>>>>>>> >>>>>>>>>>>>> Vitali >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 04/13/2016 02:01 PM, Chinthamani, Sundaram wrote: >>>>>>>>>>>>>> 440 GB/s (base 10). To be precise, 440 bytes / ns >>>>>>>>>>>>>> (nano-second). We always report memory bandwidth in >>>>>>>>>>>>>> GB/s (base 10) within Intel. >>>>>>>>>>>>>> >>>>>>>>>>>>>> -Thanks, >>>>>>>>>>>>>> -Sundaram >>>>>>>>>>>>>> >>>>>>>>>>>>>> -----Original Message----- >>>>>>>>>>>>>> From: Harms, Kevin [mailto:[email protected]] >>>>>>>>>>>>>> Sent: Wednesday, April 13, 2016 11:52 AM >>>>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com>; Chinthamani, >>>>>>>>>>>>>> Sundaram <sundaram.chinthamani(a)intel.com>; Morozov, Vitali A. >>>>>>>>>>>>>> <morozov(a)alcf.anl.gov>; Kumaran, Kalyan >>>>>>>>>>>>>> <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>>>>> <mailto:[email protected]>>; Fromkin, Russ >>>>>>>>>>>>>> <russ.fromkin(a)intel.com>; Allen, Benjamin S. >>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Just to be clear, memory is always assumed to be >>>>>>>>>>>>>> measured using base 2. It's only the storage field that >>>>>>>>>>>>>> uses base 10. So we assume if you say 440 GB/s you mean >>>>>>>>>>>>>> GiB, or 450,560 MB/s. >>>>>>>>>>>>>> >>>>>>>>>>>>>> kevin >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> And just some fun with units >>>>>>>>>>>>>>> http://sites.utexas.edu/jdm4372/2013/01/05/counting-binary-vs-dec >>>>>>>>>>>>>>> i >>>>>>>>>>>>>>> m >>>>>>>>>>>>>>> a >>>>>>>>>>>>>>> l >>>>>>>>>>>>>>> -powers-in-the-stream-benchmark/ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> From: Chinthamani, Sundaram >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Sent: Wednesday, April 13, 2016 11:20 AM >>>>>>>>>>>>>>> To: Vitali A. Morozov <morozov(a)anl.gov >>>>>>>>>>>>>>> <mailto:[email protected]>>; Raman, Karthik >>>>>>>>>>>>>>> <karthik.raman(a)intel.com>; Kumaran, Kalyan >>>>>>>>>>>>>>> <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>>>>>> <mailto:[email protected]>>; Fromkin, Russ >>>>>>>>>>>>>>> <russ.fromkin(a)intel.com>; Harms, Kevin <harms(a)alcf.anl.gov>; >>>>>>>>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov >>>>>>>>>>>>>>> <mailto:[email protected]>>; Allen, Benjamin S. >>>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov>; Chinthamani, Sundaram >>>>>>>>>>>>>>> <sundaram.chinthamani(a)intel.com> >>>>>>>>>>>>>>> Subject: RE: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 440 GB/s stream triad for MC-DRAM and 80 GB/s is our >>>>>>>>>>>>>>> internal >>>>>>>>>>>>>>> bin3 measurement. You should be able to get closer >>>>>>>>>>>>>>> these numbers, if you run the benchmark in Quad/Flat or >>>>>>>>>>>>>>> SNC-4/Flat mode. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -Thanks, >>>>>>>>>>>>>>> -Sundaram >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Sent: Wednesday, April 13, 2016 11:11 AM >>>>>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com >>>>>>>>>>>>>>> <mailto:[email protected]>>; Chinthamani, >>>>>>>>>>>>>>> Sundaram <sundaram.chinthamani(a)intel.com>; Kumaran, Kalyan >>>>>>>>>>>>>>> <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>>>>>> <mailto:[email protected]>>; Fromkin, Russ >>>>>>>>>>>>>>> <russ.fromkin(a)intel.com>; Harms, Kevin <harms(a)alcf.anl.gov>; >>>>>>>>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov >>>>>>>>>>>>>>> <mailto:[email protected]>>; Allen, Benjamin S. >>>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I see - I am treating 418122 MB/s as 418122/1024 GB/s >>>>>>>>>>>>>>> (as this is memory unit). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Anyway - great we are on the same page here. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I am in All2All/Flat mode. You seem to suggest that I >>>>>>>>>>>>>>> must be on SNC-4/Flat to see 440000 MB/s? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 04/13/2016 01:06 PM, Raman, Karthik wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Ok I read this as 418 GB/s >>>>>>>>>>>>>>> J >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> And ok, just realized you are running in All2All mode, >>>>>>>>>>>>>>> so yes this looks good. For Quad/Flat you should see >>>>>>>>>>>>>>> ~440 GB/s with the same run. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Sent: Wednesday, April 13, 2016 11:02 AM >>>>>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com >>>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>>> <mailto:[email protected]>; Chinthamani, Sundaram >>>>>>>>>>>>>>> <sundaram.chinthamani(a)intel.com> >>>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>>> Kumaran, Kalyan <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>>> Fromkin, Russ <russ.fromkin(a)intel.com> >>>>>>>>>>>>>>> <mailto:[email protected]>; Harms, Kevin >>>>>>>>>>>>>>> <harms(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>> Morozov, Vitali A. >>>>>>>>>>>>>>> <morozov(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>> Allen, Benjamin S. >>>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> <mailto:[email protected]> >>>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Karthik, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Repeating the same options you have: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp >>>>>>>>>>>>>>> -mcmodel >>>>>>>>>>>>>>> medium -shared-intel -opt-streaming-stores always >>>>>>>>>>>>>>> stream.c -o >>>>>>>>>>>>>>> stream >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>>> STREAM version $Revision: 5.9 $ >>>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>>> This system uses 8 bytes per DOUBLE PRECISION word. >>>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>>> Array size = 134217728, Offset = 0 Total memory required >>>>>>>>>>>>>>> = 3072.0 >>>>>>>>>>>>>>> MB. >>>>>>>>>>>>>>> Each test is run 100 times, but only the *best* time for >>>>>>>>>>>>>>> each is >>>>>>>>>>>>>>> used. >>>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>>> Number of Threads requested = 64 >>>>>>>>>>>>>>> ------------------------------------------------------------- >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> < skiped > >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Function Rate (MB/s) Avg time Min time >>>>>>>>>>>>>>> Max time >>>>>>>>>>>>>>> Copy: 362331.5200 0.0060 0.0059 >>>>>>>>>>>>>>> 0.0061 >>>>>>>>>>>>>>> Scale: 359281.9806 0.0060 0.0060 >>>>>>>>>>>>>>> 0.0062 >>>>>>>>>>>>>>> Add: 418122.7024 0.0078 0.0077 >>>>>>>>>>>>>>> 0.0081 >>>>>>>>>>>>>>> Triad: 418122.7024 0.0078 0.0077 >>>>>>>>>>>>>>> 0.0081 >>>>>>>>>>>>>>> Store: 224685.6729 0.0048 0.0048 >>>>>>>>>>>>>>> 0.0049 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 408GB/s >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 04/13/2016 12:15 PM, Raman, Karthik wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2. Effective MCDRAM bandwidth is around 406-408 GB/s. >>>>>>>>>>>>>>> Larger values are possible for more high core count bins. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> For bin3 , you should be able to get Triad ~440 GB/s. Please >>>>>>>>>>>>>>> check if you use the following Intel compiler flags -mcmodel >>>>>>>>>>>>>>> medium -shared-intel -O3 -xMIC-AVX512 >>>>>>>>>>>>>>> -DSTREAM_ARRAY_SIZE=134217728 -DOFFSET=0 -DNTIMES=100 >>>>>>>>>>>>>>> -openmp -opt-streaming-stores always The array size is >>>>>>>>>>>>>>> just allocating 1GB per array. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Sent: Wednesday, April 13, 2016 10:01 AM >>>>>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com >>>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>>> <mailto:[email protected]>; Chinthamani, Sundaram >>>>>>>>>>>>>>> <sundaram.chinthamani(a)intel.com> >>>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>>> Kumaran, Kalyan <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>>> Fromkin, Russ <russ.fromkin(a)intel.com> >>>>>>>>>>>>>>> <mailto:[email protected]>; Harms, Kevin >>>>>>>>>>>>>>> <harms(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>> Morozov, Vitali A. >>>>>>>>>>>>>>> <morozov(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>> Allen, Benjamin S. >>>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> <mailto:[email protected]> >>>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> so, this is my understanding that on bin3 hardware (64 >>>>>>>>>>>>>>> cores, 6 >>>>>>>>>>>>>>> channel DDR4-2133) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1. Effective DDR bandwidth is 80 GB/s (around 78% of peak). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2. Effective MCDRAM bandwidth is around 406-408 GB/s. >>>>>>>>>>>>>>> Larger values are possible for more high core count bins. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 3. Alignment is not important for STREAM triad benchmark. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 4. For both static and dynamic variables, numactl -m >>>>>>>>>>>>>>> [0|1] should be used to direct memory placement. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 5. For explicit MCDRAM allocation in flat mode, >>>>>>>>>>>>>>> hbw_malloc.h API should be use. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Please correct if I am wrong... >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks a lot for you help! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 04/12/2016 11:07 AM, Vitali A. Morozov wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Karthik, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I was also wandering how the variable could be >>>>>>>>>>>>>>> influencing the allocation if, in principle, I do not >>>>>>>>>>>>>>> need to link against memkind in my case. Anyway, sorry >>>>>>>>>>>>>>> for confusion! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I think I did that before - see my first message. I was >>>>>>>>>>>>>>> using "numactl --membind=1 ./stream" , which I believe >>>>>>>>>>>>>>> is equivalent to "numactl -m 1 ./stream" that you suggest. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Here is the result of "numactl -H": >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [morozov@knl00 512_3_C64_MPI16_TH08_flat]$ numactl -H >>>>>>>>>>>>>>> available: 2 nodes (0-1) >>>>>>>>>>>>>>> node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 >>>>>>>>>>>>>>> 18 19 20 >>>>>>>>>>>>>>> 21 >>>>>>>>>>>>>>> 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 >>>>>>>>>>>>>>> 41 42 43 >>>>>>>>>>>>>>> 44 >>>>>>>>>>>>>>> 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 >>>>>>>>>>>>>>> 64 65 66 >>>>>>>>>>>>>>> 67 >>>>>>>>>>>>>>> 68 69 70 >>>>>>>>>>>>>>> 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 >>>>>>>>>>>>>>> 90 91 92 >>>>>>>>>>>>>>> 93 >>>>>>>>>>>>>>> 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 >>>>>>>>>>>>>>> 109 110 111 >>>>>>>>>>>>>>> 112 >>>>>>>>>>>>>>> 113 114 115 116 117 118 119 120 121 122 123 124 125 126 >>>>>>>>>>>>>>> 127 128 >>>>>>>>>>>>>>> 129 >>>>>>>>>>>>>>> 130 131 132 133 134 135 136 137 138 139 140 141 142 >>>>>>>>>>>>>>> 143 144 145 146 147 148 149 150 151 152 153 154 155 156 >>>>>>>>>>>>>>> 157 158 >>>>>>>>>>>>>>> 159 >>>>>>>>>>>>>>> 160 161 162 163 164 165 166 167 168 169 170 171 172 173 >>>>>>>>>>>>>>> 174 175 >>>>>>>>>>>>>>> 176 >>>>>>>>>>>>>>> 177 178 179 180 181 182 183 184 185 186 187 188 189 190 >>>>>>>>>>>>>>> 191 192 >>>>>>>>>>>>>>> 193 >>>>>>>>>>>>>>> 194 195 196 197 198 199 200 201 202 203 204 205 206 >>>>>>>>>>>>>>> 207 208 209 210 211 212 213 214 215 216 217 218 219 220 >>>>>>>>>>>>>>> 221 222 >>>>>>>>>>>>>>> 223 >>>>>>>>>>>>>>> 224 225 226 227 228 229 230 231 232 233 234 235 236 237 >>>>>>>>>>>>>>> 238 239 >>>>>>>>>>>>>>> 240 >>>>>>>>>>>>>>> 241 242 243 244 245 246 247 248 249 250 251 252 253 254 >>>>>>>>>>>>>>> 255 node >>>>>>>>>>>>>>> 0 >>>>>>>>>>>>>>> size: 196511 MB node 0 free: 186888 MB node 1 cpus: >>>>>>>>>>>>>>> node 1 size: 16384 MB >>>>>>>>>>>>>>> node 1 free: 11731 MB >>>>>>>>>>>>>>> node distances: >>>>>>>>>>>>>>> node 0 1 >>>>>>>>>>>>>>> 0: 10 31 >>>>>>>>>>>>>>> 1: 31 10 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Sounds like all as expected. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [morozov@knl00 STREAM]$ OMP_NUM_THREADS=64 >>>>>>>>>>>>>>> KMP_AFFINITY=scatter numactl -m 1 ./stream >>>>>>>>>>>>>>> Triad: 418262.7498 0.0058 0.0057 >>>>>>>>>>>>>>> 0.0060 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> For a series of runs, I see this: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> # MB N GB/s >>>>>>>>>>>>>>> 228.88 10000000 408867.98 >>>>>>>>>>>>>>> 457.76 20000000 405818.57 >>>>>>>>>>>>>>> 686.65 30000000 414309.08 >>>>>>>>>>>>>>> 915.53 40000000 415063.59 >>>>>>>>>>>>>>> 1144.41 50000000 415792.22 >>>>>>>>>>>>>>> 1373.29 60000000 415449.01 >>>>>>>>>>>>>>> 1602.17 70000000 414106.18 >>>>>>>>>>>>>>> 1831.05 80000000 416846.82 >>>>>>>>>>>>>>> 2059.94 90000000 414992.29 >>>>>>>>>>>>>>> 2288.82 100000000 417898.11 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 04/12/2016 10:32 AM, Raman, Karthik wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Vitali, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1. >>>>>>>>>>>>>>> The MEMKIND_HBW_NODES is *only* when you are allocating >>>>>>>>>>>>>>> using >>>>>>>>>>>>>>> hbwalloc (+ MCDRAM flat) in your application and when >>>>>>>>>>>>>>> memkind is >>>>>>>>>>>>>>> unable to find the high bandwidth nodes (pmtt not populated) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2. >>>>>>>>>>>>>>> With the below >>>>>>>>>>>>>>> static allocations, in MCDRAM Flat mode, you can simply >>>>>>>>>>>>>>> use ï¿½numactl ï¿½m 1 ./streamï¿½ with propery affinity >>>>>>>>>>>>>>> settings to get peak out of MCDRAM. >>>>>>>>>>>>>>> a. >>>>>>>>>>>>>>> Check ï¿½numactl ï¿½Hï¿½ to find the NUMA domains in your node. If >>>>>>>>>>>>>>> booted in All2All/FLAT or Quad/Flat mode it should show >>>>>>>>>>>>>>> 2 NUMA >>>>>>>>>>>>>>> domains , one for DDR (96 GB) and other for MCDRAM (16GB) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>> Karthik >>>>>>>>>>>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Sent: Tuesday, April 12, 2016 7:27 AM >>>>>>>>>>>>>>> To: Raman, Karthik <karthik.raman(a)intel.com >>>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>>> <mailto:[email protected]>; Chinthamani, Sundaram >>>>>>>>>>>>>>> <sundaram.chinthamani(a)intel.com> >>>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>>> Kumaran, Kalyan <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>>> Fromkin, Russ <russ.fromkin(a)intel.com> >>>>>>>>>>>>>>> <mailto:[email protected]>; Harms, Kevin >>>>>>>>>>>>>>> <harms(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>> Morozov, Vitali A. >>>>>>>>>>>>>>> <morozov(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>> Allen, Benjamin S. >>>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> <mailto:[email protected]> >>>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Karthik, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The node is in All2All mode for now. I made the >>>>>>>>>>>>>>> following changes. The original code is using static >>>>>>>>>>>>>>> allocation therefore, I follow the advice on 2MB static >>>>>>>>>>>>>>> array alignment: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [morozov@thing00 STREAM]$ head -n 108 < stream.c | tail -n 4 >>>>>>>>>>>>>>> static double a[N+OFFSET] >>>>>>>>>>>>>>> __attribute__((aligned(2097152))), >>>>>>>>>>>>>>> b[N+OFFSET] >>>>>>>>>>>>>>> __attribute__((aligned(2097152))), >>>>>>>>>>>>>>> c[N+OFFSET] >>>>>>>>>>>>>>> __attribute__((aligned(2097152))), >>>>>>>>>>>>>>> d[N+OFFSET] >>>>>>>>>>>>>>> __attribute__((aligned(2097152))); >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [morozov@knl00 STREAM]$ make >>>>>>>>>>>>>>> mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp >>>>>>>>>>>>>>> -mcmodel >>>>>>>>>>>>>>> medium -shared-intel -fno-alias stream.c -o stream >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [morozov@knl00 STREAM]$ MEMKIND_HBW_NODES=1 >>>>>>>>>>>>>>> OMP_NUM_THREADS=64 >>>>>>>>>>>>>>> KMP_AFFINITY=scatter ./stream >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ..... >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Triad: 80251.3621 0.0300 0.0299 >>>>>>>>>>>>>>> 0.0305 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Apparently, the "MEMKIND_HBW_NODES=1" is ignored. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 04/12/2016 01:51 AM, Raman, Karthik wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Also could you please modify the memory allocations in >>>>>>>>>>>>>>> STREAM to align at 2MB boundary. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> For example: >>>>>>>>>>>>>>> static allocations: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> static STREAM_TYPE a[STREAM_ARRAY_SIZE+OFFSET] >>>>>>>>>>>>>>> __attribute__((aligned(2097152))) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> dynamic allocations: >>>>>>>>>>>>>>> a = (STREAM_TYPE >>>>>>>>>>>>>>> *)_mm_malloc(sizeof(STREAM_TYPE)*(STREAM_ARRAY_SIZE+OFFSET),20971 >>>>>>>>>>>>>>> 5 >>>>>>>>>>>>>>> 2 >>>>>>>>>>>>>>> ) >>>>>>>>>>>>>>> ; >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> memkind/hbwalloc: >>>>>>>>>>>>>>> (STREAM_TYPE *)hbw_posix_memalign((void **)&a, 2097152, >>>>>>>>>>>>>>> sizeof(STREAM_TYPE)*(STREAM_ARRAY_SIZE+OFFSET)) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> For memkind/hbwmalloc to work out of the box, you need >>>>>>>>>>>>>>> to have >>>>>>>>>>>>>>> the PMTT tables populated so that it can identify the >>>>>>>>>>>>>>> high bandwidth nodes. Alternatively you can set the >>>>>>>>>>>>>>> MEMKIND_HBW_NODES environment variable to the high >>>>>>>>>>>>>>> bandwidth node (MCDRAM). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> e.g. All2All/Flat or Quad/Flat : export >>>>>>>>>>>>>>> MEMKIND_HBW_NODES=1 ; ./app_with_hbwmalloc >>>>>>>>>>>>>>> SNC4/Flat: export MEMKIND_HBW_NODES=4,5,6,7 ; >>>>>>>>>>>>>>> ./app_with_hbw_malloc >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>> Karthik >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> From: Chinthamani, Sundaram >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Sent: Monday, April 11, 2016 8:15 PM >>>>>>>>>>>>>>> To: Vitali Morozov <morozov(a)anl.gov >>>>>>>>>>>>>>> <mailto:[email protected]>> <mailto:[email protected]>; >>>>>>>>>>>>>>> Kumaran, Kalyan <kumaran(a)alcf.anl.gov> >>>>>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>>> <mailto:[email protected]>; >>>>>>>>>>>>>>> Fromkin, Russ <russ.fromkin(a)intel.com> >>>>>>>>>>>>>>> <mailto:[email protected]>; Raman, Karthik >>>>>>>>>>>>>>> <karthik.raman(a)intel.com> >>>>>>>>>>>>>>> <mailto:[email protected]>; Harms, Kevin >>>>>>>>>>>>>>> <harms(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>> Morozov, Vitali A. >>>>>>>>>>>>>>> <morozov(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>> Allen, Benjamin S. >>>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> <mailto:[email protected]>; >>>>>>>>>>>>>>> Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com >>>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>>>>> Subject: RE: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> You should use numactl ï¿½m 4,5,6,7 to get the memory >>>>>>>>>>>>>>> bandwidth out of MC-DRAM. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> What kind of part is it? Is it bin1 or bin3? Even in >>>>>>>>>>>>>>> All2All mode, we have measured up to 470 GB/s of stream >>>>>>>>>>>>>>> triad bandwidth. Quad mode should get 490 GB/s. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> DDR-stream triad bandwidth on Bin1-part should be close >>>>>>>>>>>>>>> to 90 GB/s. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -Sundaram >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> From: Vitali Morozov [mailto:[email protected]] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Sent: Monday, April 11, 2016 6:59 PM >>>>>>>>>>>>>>> To: Kumaran, Kalyan <kumaran(a)alcf.anl.gov >>>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com >>>>>>>>>>>>>>> <mailto:[email protected]>>; Fromkin, Russ >>>>>>>>>>>>>>> <russ.fromkin(a)intel.com>; Chinthamani, Sundaram >>>>>>>>>>>>>>> <sundaram.chinthamani(a)intel.com>; Raman, Karthik >>>>>>>>>>>>>>> <karthik.raman(a)intel.com>; Harms, Kevin >>>>>>>>>>>>>>> <harms(a)alcf.anl.gov>; >>>>>>>>>>>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov >>>>>>>>>>>>>>> <mailto:[email protected]>>; Allen, Benjamin S. >>>>>>>>>>>>>>> <bsallen(a)alcf.anl.gov> >>>>>>>>>>>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Avinash and others, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> here is what we have so far for Flat more, All2All. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The results from memory were obtained by using default >>>>>>>>>>>>>>> STREAM with static allocation. The results from MCDRAM >>>>>>>>>>>>>>> were obtained by using modified STREAM with hbwmalloc.h >>>>>>>>>>>>>>> interface. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1 MPI process, 64 threads, scatter affinity, no numactl >>>>>>>>>>>>>>> is used. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Most results are as expected: 80 GB/s from DDR memory, >>>>>>>>>>>>>>> 420 GB/s from MCDRAM, the MCDRAM size is about 16 GB, >>>>>>>>>>>>>>> the aggregate L2 cache size is 32 MB. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Noticed inconsistencies for hbwmalloc benchmarks: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1. The result is not validated - will spend more time on >>>>>>>>>>>>>>> that tomorrow. >>>>>>>>>>>>>>> 2. hbw_malloc continues allocating data beyond the >>>>>>>>>>>>>>> expected 16 GB up to 48 GB. This means it fails back to >>>>>>>>>>>>>>> DDR if it cannot allocate in MCDRAM - not expected >>>>>>>>>>>>>>> default behavior. It must fail if it does not have >>>>>>>>>>>>>>> enough space. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The problem shows up when I am running static allocation >>>>>>>>>>>>>>> and use numactl --membind=1 to ask for MCDRAM-only >>>>>>>>>>>>>>> allocation. The variation is very large and inconsistent >>>>>>>>>>>>>>> with the result observed when using hbw_malloc API: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> # MB N GB/s numactl GB/s >>>>>>>>>>>>>>> hbw_malloc >>>>>>>>>>>>>>> 228.88 10000000 255555.46 421009.18 >>>>>>>>>>>>>>> 457.76 20000000 363930.93 416997.91 >>>>>>>>>>>>>>> 686.65 30000000 309415.87 427808.31 >>>>>>>>>>>>>>> 915.53 40000000 371691.30 423310.75 >>>>>>>>>>>>>>> 1144.41 50000000 187002.22 431032.35 >>>>>>>>>>>>>>> 1373.29 60000000 385954.23 427172.91 >>>>>>>>>>>>>>> 1602.17 70000000 388254.49 426823.59 >>>>>>>>>>>>>>> 1831.05 80000000 374491.43 428103.96 >>>>>>>>>>>>>>> 2059.94 90000000 379622.74 428152.02 >>>>>>>>>>>>>>> 2288.82 100000000 80868.34 430185.03 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We also tried SNC-4 instead of All2All, but could not >>>>>>>>>>>>>>> get more than 40 GB/s from MCDRAM even with hbw_malloc API. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Raw data: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> # flat memory, running from memory (default), alltoall >>>>>>>>>>>>>>> # MB N GB/s >>>>>>>>>>>>>>> 2.29 100000 272062.96 >>>>>>>>>>>>>>> 4.58 200000 491040.47 >>>>>>>>>>>>>>> 6.87 300000 603979.78 >>>>>>>>>>>>>>> 9.16 400000 745654.04 >>>>>>>>>>>>>>> 11.44 500000 811800.77 >>>>>>>>>>>>>>> 13.73 600000 901462.35 >>>>>>>>>>>>>>> 16.02 700000 891953.26 >>>>>>>>>>>>>>> 18.31 800000 1019375.15 >>>>>>>>>>>>>>> 20.60 900000 984749.63 >>>>>>>>>>>>>>> 22.89 1000000 923516.48 >>>>>>>>>>>>>>> 25.18 1100000 758422.09 >>>>>>>>>>>>>>> 27.47 1200000 779328.74 >>>>>>>>>>>>>>> 29.75 1300000 513185.43 >>>>>>>>>>>>>>> 32.04 1400000 155550.35 >>>>>>>>>>>>>>> 34.33 1500000 103209.12 >>>>>>>>>>>>>>> 36.62 1600000 97435.74 >>>>>>>>>>>>>>> 38.91 1700000 83968.40 >>>>>>>>>>>>>>> 41.20 1800000 83692.35 >>>>>>>>>>>>>>> 43.49 1900000 80734.60 >>>>>>>>>>>>>>> 45.78 2000000 76317.89 >>>>>>>>>>>>>>> 68.66 3000000 81180.08 >>>>>>>>>>>>>>> 91.55 4000000 78889.73 >>>>>>>>>>>>>>> 114.44 5000000 79891.50 >>>>>>>>>>>>>>> 137.33 6000000 79954.96 >>>>>>>>>>>>>>> 160.22 7000000 79477.00 >>>>>>>>>>>>>>> 183.11 8000000 79145.59 >>>>>>>>>>>>>>> 205.99 9000000 79821.12 >>>>>>>>>>>>>>> 228.88 10000000 79519.15 >>>>>>>>>>>>>>> 228.88 10000000 79212.54 >>>>>>>>>>>>>>> 457.76 20000000 79692.27 >>>>>>>>>>>>>>> 686.65 30000000 79866.15 >>>>>>>>>>>>>>> 915.53 40000000 79828.15 >>>>>>>>>>>>>>> 1144.41 50000000 80214.91 >>>>>>>>>>>>>>> 1373.29 60000000 80226.84 >>>>>>>>>>>>>>> 1602.17 70000000 80178.77 >>>>>>>>>>>>>>> 1831.05 80000000 80177.06 >>>>>>>>>>>>>>> 2059.94 90000000 80109.79 >>>>>>>>>>>>>>> 2288.82 100000000 80175.94 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> # flat memory, running from MCDRAM, alltoall, hbwmalloc API >>>>>>>>>>>>>>> # MB N GB/s >>>>>>>>>>>>>>> 228.88 10000000 421009.18 >>>>>>>>>>>>>>> 457.76 20000000 416997.91 >>>>>>>>>>>>>>> 686.65 30000000 427808.31 >>>>>>>>>>>>>>> 915.53 40000000 423310.75 >>>>>>>>>>>>>>> 1144.41 50000000 431032.35 >>>>>>>>>>>>>>> 1373.29 60000000 427172.91 >>>>>>>>>>>>>>> 1602.17 70000000 426823.59 >>>>>>>>>>>>>>> 1831.05 80000000 428103.96 >>>>>>>>>>>>>>> 2059.94 90000000 428152.02 >>>>>>>>>>>>>>> 2288.82 100000000 430185.03 >>>>>>>>>>>>>>> 2343.75 102400000 425999.98 >>>>>>>>>>>>>>> 4687.50 204800000 427821.10 >>>>>>>>>>>>>>> 9155.27 400000000 429318.13 >>>>>>>>>>>>>>> 9375.00 409600000 428376.70 >>>>>>>>>>>>>>> 11444.09 500000000 428052.08 >>>>>>>>>>>>>>> 13732.91 600000000 266061.01 >>>>>>>>>>>>>>> 16021.73 700000000 200257.79 >>>>>>>>>>>>>>> 18310.55 800000000 168164.19 >>>>>>>>>>>>>>> 20599.37 900000000 150192.91 >>>>>>>>>>>>>>> 22888.18 1000000000 137862.07 >>>>>>>>>>>>>>> 25177.00 1100000000 129418.12 >>>>>>>>>>>>>>> 27465.82 1200000000 123019.53 >>>>>>>>>>>>>>> 29754.64 1300000000 118268.79 >>>>>>>>>>>>>>> 32043.46 1400000000 114343.33 >>>>>>>>>>>>>>> 34332.28 1500000000 110991.25 >>>>>>>>>>>>>>> 36621.09 1600000000 108538.36 >>>>>>>>>>>>>>> 38909.91 1700000000 106249.72 >>>>>>>>>>>>>>> 41198.73 1800000000 104455.07 >>>>>>>>>>>>>>> 43487.55 1900000000 102828.93 >>>>>>>>>>>>>>> 45776.37 2000000000 101348.56 >>>>>>>>>>>>>>> 48065.19 2100000000 100013.73 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> We appreciate any help on that. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Vitali >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On 04/11/2016 08:42 PM, Kumaran, Kalyan wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Avinash, thanks for the prompt reply. Kevin/Vitali can >>>>>>>>>>>>>>> provide >>>>>>>>>>>>>>> all the details. The short summary is we are unable to >>>>>>>>>>>>>>> reproduce the STREAM results from IPM or DDR. It would >>>>>>>>>>>>>>> be great if Karthik or Sundaram can advise us on how to >>>>>>>>>>>>>>> run (which mode etc) or if they could log into the >>>>>>>>>>>>>>> machine and make sure we have set up things correctly. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Kumar >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Apr 11, 2016, at 8:07 PM, Sodani, Avinash >>>>>>>>>>>>>>> <avinash.sodani(a)intel.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Copying Sundaram/Karthik from our side. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Kumar, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Can you send us more information on this? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -Avinash >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> From: Kumaran, >>>>>>>>>>>>>>> Kalyan [mailto:[email protected]] >>>>>>>>>>>>>>> Sent: Monday, April 11, 2016 5:32 PM >>>>>>>>>>>>>>> To: Fromkin, Russ; Sodani, Avinash >>>>>>>>>>>>>>> Cc: Harms, Kevin; Morozov, Vitali A. >>>>>>>>>>>>>>> Subject: Fwd: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Russ, Avinash, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> we need help in understanding how to obtain STREAM >>>>>>>>>>>>>>> results. Can anyone help? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Kumar >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Begin forwarded message: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> From: "Harms, Kevin" <harms(a)alcf.anl.gov >>>>>>>>>>>>>>> <mailto:[email protected]>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Subject: [JLSE-Intel-NDA] KNL Status >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Date: April 11, 2016 at 5:42:15 PM CDT >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> To: "intel-nda(a)jlse.anl.gov" >>>>>>>>>>>>>>> <intel-nda(a)jlse.anl.gov> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The knl00 node is currently in Flat mode and All2All >>>>>>>>>>>>>>> model. With this setup we can get 420 GB/s from IPM >>>>>>>>>>>>>>> memory on STREAM Triad when using hbw_malloc and friends >>>>>>>>>>>>>>> which was confirmed by both Vitali and I. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Changing the BIOS settings and warm rebooting seem to be >>>>>>>>>>>>>>> problematic. Vitali and I were running earlier and could >>>>>>>>>>>>>>> not get over 80 GB/s with any method even though memory >>>>>>>>>>>>>>> capacity indicated flat mode. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Using numactl --membind is also problematic. The STREAM >>>>>>>>>>>>>>> results >>>>>>>>>>>>>>> when using this to launch the application were erratic, >>>>>>>>>>>>>>> so it is also suggested not to use that. That means you >>>>>>>>>>>>>>> will need to modify your code to use dynamic memory and >>>>>>>>>>>>>>> hbw_malloc to exploit the IPM. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Will followup with more detailed results later. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> kevin >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> Intel-nda mailing list >>>>>>>>>>>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>>>>>>>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> <flat_a2a_ddr.txt><flat_a2a_ipm.txt><flat_quad_ddr.txt><flat_quad_ipm.txt><flat_snc4_ddr.txt><flat_snc4_ipm.txt> >>>>>> >>>> >>>> >>>> _______________________________________________ >>>> Intel-nda mailing list >>>> Intel-nda(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>> >> >> >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] KNL Status
by Vitali A Morozov 21 Apr '16

21 Apr '16

It was "a kind of" joke, Ben. Kevin and Kumar like to make jokes about Fortran on me. They think that we do not need Fortran support anymore, and we should abandon pushing the latest features into the Fortran language ... On 4/21/16 10:21 AM, Allen, Benjamin S. wrote: > /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/intel64/ifort > > ? > > Ben > >> On Apr 21, 2016, at 12:19 PM, Harms, Kevin <harms(a)alcf.anl.gov> wrote: >> >>> I am curious what was changed (or added) in version 17... >>> >>> Vitali >> They removed FORTRAN support... OH SNAP! >> >> kevin >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] KNL Status
by Allen, Benjamin S. 21 Apr '16

21 Apr '16

/soft/compilers/intel_beta/compilers_and_libraries/linux/bin/intel64/ifort ? Ben > On Apr 21, 2016, at 12:19 PM, Harms, Kevin <harms(a)alcf.anl.gov> wrote: > >> >> I am curious what was changed (or added) in version 17... >> >> Vitali > > They removed FORTRAN support... OH SNAP! > > kevin > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0