It's not the compiler, I see no significant difference in SNC-4 mode between Intel 16 and Intel 17 Beta.

>icc -V
Intel(R) C Intel(R) 64 Compiler for applications running on Intel(R) 64, Version 16.0.0.109 Build 20150815
Copyright (C) 1985-2015 Intel Corporation.� All rights reserved.

>icc -O3 -fopenmp -DSTREAM_ARRAY_SIZE=134217728 -xMIC-AVX512 -mcmodel medium -shared-intel� stream.c -o stream_c.exe

>OMP_NUM_THREADS=128� numactl -m 4,5,6,7 ./stream_c.exe
Function��� Best Rate MB/s� Avg time���� Min time���� Max time
Copy:��������� 390734.0���� 0.005547���� 0.005496���� 0.005631
Scale:�������� 411250.1���� 0.005309���� 0.005222���� 0.005569
Add:���������� 449326.5���� 0.007190���� 0.007169���� 0.007220
Triad:�������� 440248.9���� 0.007343���� 0.007317���� 0.007391

On 4/21/16 10:12 AM, Vitali A Morozov wrote:
Yeah -� as Ronald Green has figured out, the beta version of a compiler (17) does make something differently compared to production version 16. For some reason, it only matters in SNC-4 mode and not in All2All mode.

I am curious what was changed (or added) in version 17...

Vitali



On 4/21/16 9:45 AM, Kumaran, Kalyan wrote:
Looks like Scott Parker gets better results:

Adding another data point, I ran in SNC-4 mode with the configuration below and observed a consistent BW around 440 GB/s
with the best run producing 442 GB/s. In this case 128 threads gave the best results.

I've noticed that the STREAM results �sensitivity to build and run parameters seems to vary depending on the memory configuration
in flat-all2all I saw noticeably better results with memory aligned on 2MB boundaries and 64 threads, in SNC-4 mode the alignment
doesn't seem to matter and 128 threads is producing the best results.


��Mode: flat & SNC-4
��Compiler: Intel 17 Beta
��Compile Options: icc -O3 -fopenmp -DSTREAM_ARRAY_SIZE=134217728 -xMIC-AVX512 -mcmodel medium -shared-intel
��Run Command: �OMP_NUM_THREADS=128 numactl -m 4,5,6,7 ./stream_mod

��Results:
������Function ���Best Rate MB/s �Avg time ����Min time ����Max time
������Copy: ���������390378.3 ����0.005524 ����0.005501 0.005546
������Scale: ��������412738.8 ����0.005237 ����0.005203 0.005278
������Add: ����������448759.4 ����0.007199 ����0.007178 0.007218
������Triad: ��������441919.3 ����0.007355 ����0.007289 0.007548



On Apr 20, 2016, at 7:39 PM, Vitali A Morozov <morozov@anl.gov> wrote:

Hi,

a quick update on STREAM with the latest beta compiler

[morozov@knl00 STREAM]$ icc -V
Intel(R) C Intel(R) 64 Compiler for applications running on Intel(R) 64, Version 17.0.0.042 Beta Build 20160315

compiler options
mpiicc -O3 -xMIC-AVX512 -qopt-report=5 -Wall -g -fopenmp -mcmodel medium -shared-intel -qopt-streaming-stores always -restrict stream.c -o stream

Running in flat mode all2all

morozov@knl00 STREAM]$ export OMP_NUM_THREADS=256
[morozov@knl00 STREAM]$ numactl -m 1 ./stream
-------------------------------------------------------------
STREAM version $Revision: 5.9 $
-------------------------------------------------------------
This system uses 8 bytes per DOUBLE PRECISION word.
-------------------------------------------------------------
Array size = 134217728, Offset = 0
Total memory required = 3072.0 MB.
Each test is run 500 times, but only
the *best* time for each is used.
-------------------------------------------------------------
Number of Threads requested = 256
-------------------------------------------------------------
<skipped>
-------------------------------------------------------------
Your clock granularity/precision appears to be 1 microseconds.
Each test below will take on the order of 6356 microseconds.
�� (= 6356 clock ticks)
Increase the size of the arrays if this shows that
you are not getting at least 20 clock ticks per test.
-------------------------------------------------------------
WARNING -- The above is only a rough guideline.
For best results, please be sure you know the
precision of your system timer.
-------------------------------------------------------------
Function����� Rate (MB/s)�� Avg time���� Min time���� Max time
Copy:����� 330891.5637������ 0.0066������ 0.0065������ 0.0078
Scale:���� 330127.5200������ 0.0066������ 0.0065������ 0.0069
Add:������ 383622.4448������ 0.0085������ 0.0084������ 0.0093
Triad:���� 383252.4574������ 0.0085������ 0.0084������ 0.0098
-------------------------------------------------------------
Solution Validates
-------------------------------------------------------------

Same with 64 threads scatter mode:

morozov@knl00 STREAM]$ export OMP_NUM_THREADS=64
[morozov@knl00 STREAM]$ KMP_AFFINITY=scatter
[morozov@knl00 STREAM]$ numactl -m 1 ./stream
-------------------------------------------------------------
STREAM version $Revision: 5.9 $
-------------------------------------------------------------
This system uses 8 bytes per DOUBLE PRECISION word.
-------------------------------------------------------------
Array size = 134217728, Offset = 0
Total memory required = 3072.0 MB.
Each test is run 500 times, but only
the *best* time for each is used.
-------------------------------------------------------------
Number of Threads requested = 64
-------------------------------------------------------------
<skipped>
-------------------------------------------------------------
Your clock granularity/precision appears to be 1 microseconds.
Each test below will take on the order of 6056 microseconds.
�� (= 6056 clock ticks)
Increase the size of the arrays if this shows that
you are not getting at least 20 clock ticks per test.
-------------------------------------------------------------
WARNING -- The above is only a rough guideline.
For best results, please be sure you know the
precision of your system timer.
-------------------------------------------------------------
Function����� Rate (MB/s)�� Avg time���� Min time���� Max time
Copy:����� 353486.8826������ 0.0061������ 0.0061������ 0.0067
Scale:���� 353265.0608������ 0.0061������ 0.0061������ 0.0066
Add:������ 423961.3055������ 0.0077������ 0.0076������ 0.0079
Triad:���� 424907.9750������ 0.0076������ 0.0076������ 0.0084
-------------------------------------------------------------
Solution Validates
-------------------------------------------------------------

Vitali




On 4/18/16 12:43 PM, Kumaran, Kalyan wrote:
FYI. Karthik will get on the machine to debug.

Sent from my iPhone

Begin forwarded message:

From: "Chinthamani, Sundaram" <sundaram.chinthamani@intel.com>
Date: April 18, 2016 at 2:05:26 PM CDT
To: "Harms, Kevin" <harms@alcf.anl.gov>, "Allen, Benjamin S." <bsallen@alcf.anl.gov>
Cc: "Kumaran, Kalyan" <kumaran@alcf.anl.gov>, "Kaana Balakrishna, Abhishek" <abhishek.kaana.balakrishna@intel.com>, "Morozov, Vitali A." <morozov@alcf.anl.gov>, "Raman, Karthik" <karthik.raman@intel.com>, "Inkley, Benson" <benson.inkley@intel.com>, "Sodani, Avinash" <avinash.sodani@intel.com>, "Fromkin, Russ" <russ.fromkin@intel.com>, "Chinthamani, Sundaram" <sundaram.chinthamani@intel.com>
Subject: RE: [JLSE-Intel-NDA] KNL Status

I would expect Quad mode / SNC-4 mode to provide higher bandwidth compared to All2All mode. ��We haven't seen this in our internal measurements. ���

-Thanks,
-Sundaram

-----Original Message-----
From: Harms, Kevin [mailto:harms@alcf.anl.gov]
Sent: Monday, April 18, 2016 12:00 PM
To: Allen, Benjamin S. <bsallen@alcf.anl.gov>
Cc: Kumaran, Kalyan <kumaran@alcf.anl.gov>; Kaana Balakrishna, Abhishek <abhishek.kaana.balakrishna@intel.com>; Morozov, Vitali A. <morozov@alcf.anl.gov>; Chinthamani, Sundaram <sundaram.chinthamani@intel.com>; Raman, Karthik <karthik.raman@intel.com>; Inkley, Benson <benson.inkley@intel.com>; Sodani, Avinash <avinash.sodani@intel.com>; Fromkin, Russ <russ.fromkin@intel.com>
Subject: Re: [JLSE-Intel-NDA] KNL Status


�I tested cached mode with all2all, snc4 and quad models. Cached all2all gave 275,000 MB/s. Does that look reasonable? SNC4 and quad mode gave 225,000 MB/s or so.

kevin




The following versions are now installed: BIOS 01.01.0104, ME 03.01.03.011.0, BMC 00.17.9699, FRUSDR 1.14.

Ben

On Apr 18, 2016, at 1:27 PM, Harms, Kevin <harms@alcf.anl.gov> wrote:


Ben completed the BIOS upgrade, but we do not see any difference in performance. I have attached results from runs done under three configurations: flat+all2all, flat+quad, flat+snc4.

flat+all2all = 427046.0546 MB/s
flat+quad ���= 327253.8882 MB/s
flat+snc4 ���= 308555.9588 MB/s

I compiled the CORAL stream.c with the following options:
icc -mcmodel medium -shared-intel -O3 �-xMIC-AVX512 -DN=50000000 -DOFFSET=0 -DNTIMES=100 -openmp -opt-streaming-stores always stream.c -o stream

I added the alignment directives to the arrays. The attached files show the NUMA domains and numactl commands used and run parameters.

Have I missed a parameter?

kevin



Suggest a time Friday afternoon. We are in central time zone. We can get you access to this B0 too.

-Kumar


On Apr 13, 2016, at 5:06 PM, Kaana Balakrishna, Abhishek <abhishek.kaana.balakrishna@intel.com> wrote:

+ Benson

Cheers,
Abhishek K Balakrishna (KB)
Xeon Phi� Hardware Platform Applications Engineer,
Technical Computing Group (TCG)
Intel Corporation
Ph - 512 466 9886
www.intel.com/xeonphi


-----Original Message-----
From: Vitali A. Morozov [mailto:morozov@anl.gov]
Sent: Wednesday, April 13, 2016 4:58 PM
To: Chinthamani, Sundaram <sundaram.chinthamani@intel.com>; Raman, Karthik <karthik.raman@intel.com>; Harms, Kevin <harms@alcf.anl.gov>; Morozov, Vitali A. <morozov@alcf.anl.gov>; Kumaran, Kalyan <kumaran@alcf.anl.gov>
Cc: Sodani, Avinash <avinash.sodani@intel.com>; Fromkin, Russ <russ.fromkin@intel.com>; Allen, Benjamin S. <bsallen@alcf.anl.gov>; Kaana Balakrishna, Abhishek <abhishek.kaana.balakrishna@intel.com>
Subject: Re: [JLSE-Intel-NDA] KNL Status

Sure!

Vitali


On 04/13/2016 04:54 PM, Chinthamani, Sundaram wrote:
Would it be possible to have a meeting to go over your questions rather than exchanging E-mails back and forth?

+ KB from Intel side. �He can help set-up the meeting between Labs and Intel.

-Thanks,
-Sundaram

-----Original Message-----
From: Vitali A. Morozov [mailto:morozov@anl.gov]
Sent: Wednesday, April 13, 2016 2:50 PM
To: Chinthamani, Sundaram <sundaram.chinthamani@intel.com>; Raman,
Karthik <karthik.raman@intel.com>; Harms, Kevin <harms@alcf.anl.gov>;
Morozov, Vitali A. <morozov@alcf.anl.gov>; Kumaran, Kalyan
<kumaran@alcf.anl.gov>
Cc: Sodani, Avinash <avinash.sodani@intel.com>; Fromkin, Russ
<russ.fromkin@intel.com>; Allen, Benjamin S. <bsallen@alcf.anl.gov>
Subject: Re: [JLSE-Intel-NDA] KNL Status

On 04/13/2016 04:42 PM, Chinthamani, Sundaram wrote:
node 4 free: 2907 MB
node 5 free: 2926 MB
node 6 free: 2922 MB
node 7 free: 2914 MB

There is 16 GB of MC-DRAM and 4 GB per NUMA node. ���Is there any other process running on the machine? ���Why does "free memory" show only 3 GB per node?
This is EXACTLY why I am asking this question. Nothing is running on the node - and I do not see full 4 GiB of available MCDRAM memory. This might be reason why I do not see what is expected.

I understand that I can reduce the array size, but I am trying to reproduce your results.

I am still unclear on if you run "no mpi" version, how come that SNC-4 mode is giving you advantage? All arrays are allocated in the same quadrant. Requests from the other quadrants will go to this one and create congestion.

Vitali






If you want to test Stream triad bandwidth to MC-DRAM in Quad/Flat mode, you can use even smaller arrays of few 100 MBs.

-Thanks,
-Sundaram

-----Original Message-----
From: Vitali A. Morozov [mailto:morozov@anl.gov]
Sent: Wednesday, April 13, 2016 2:32 PM
To: Raman, Karthik <karthik.raman@intel.com>; Chinthamani, Sundaram
<sundaram.chinthamani@intel.com>; Harms, Kevin <harms@alcf.anl.gov>;
Morozov, Vitali A. <morozov@alcf.anl.gov>; Kumaran, Kalyan
<kumaran@alcf.anl.gov>
Cc: Sodani, Avinash <avinash.sodani@intel.com>; Fromkin, Russ
<russ.fromkin@intel.com>; Allen, Benjamin S. <bsallen@alcf.anl.gov>
Subject: Re: [JLSE-Intel-NDA] KNL Status

ok, that is fine.

Now the next issue I have is that for static arrays, all three arrays must land up on one of the numa nodes - the node that is attached to a master thread.

You advice to use �N = 134217728, that gives exactly 3 GiB of memory footprint for 3 arrays of triad.

According to "numactl -H", I do not have 3 GiB of available memory on any of the MCDRAM numa nodes:

numactl -H |grep free
node 0 free: 46336 MB
node 1 free: 46923 MB
node 2 free: 46970 MB
node 3 free: 46961 MB
node 4 free: 2907 MB
node 5 free: 2926 MB
node 6 free: 2922 MB
node 7 free: 2914 MB

Therefore, at least one array will not be placed to the same numa node.
Is this ok?






On 04/13/2016 04:17 PM, Raman, Karthik wrote:
Yes we run pure OMP code too.

The problem with KMP_AFFINITY with SNC4 mode (currently) is that it
may not equally balance the threads per NUMA domain even with
scatter or balanced, you can check that by mapping �the "verbose" output (e.g.
KMP_AFFINITY=scatter,verbose) with "numactl -H" output to see if the
OS thread to logical CPU mapping is balanced across all NUMA domains

With All2All/Quad this should not be a problem


-----Original Message-----
From: Vitali A. Morozov [mailto:morozov@anl.gov]
Sent: Wednesday, April 13, 2016 1:53 PM
To: Raman, Karthik <karthik.raman@intel.com>; Chinthamani, Sundaram
<sundaram.chinthamani@intel.com>; Harms, Kevin <harms@alcf.anl.gov>;
Morozov, Vitali A. <morozov@alcf.anl.gov>; Kumaran, Kalyan
<kumaran@alcf.anl.gov>
Cc: Sodani, Avinash <avinash.sodani@intel.com>; Fromkin, Russ
<russ.fromkin@intel.com>; Allen, Benjamin S. <bsallen@alcf.anl.gov>
Subject: Re: [JLSE-Intel-NDA] KNL Status

Before doing that - let us check if we are running similar benchmark.

I am running 1 MPI process (in fact no mpi at all) and 64 OpenMP threads version. The threads can be placed with "balanced" or "scattered"
policy. I am using "scattered".

Is this ok so far?

Vitali






On 04/13/2016 03:48 PM, Raman, Karthik wrote:
This still does not look right, and the reason could be that the
threads are not balanced across all the NUMA domains (ideally you
want
16 threads per quadrant)

Before going to SNC4, can you please boot in "Quad" mode. If you
look in the BIOS Cluster mode options there is another mode called "Quad"
which is different from SNC4, only 2 NUMA domains (DDR, MCDRAM).
Please run similar to All2All mode



-----Original Message-----
From: Vitali A. Morozov [mailto:morozov@anl.gov]
Sent: Wednesday, April 13, 2016 12:56 PM
To: Chinthamani, Sundaram <sundaram.chinthamani@intel.com>; Harms,
Kevin <harms@alcf.anl.gov>; Raman, Karthik
<karthik.raman@intel.com>; Morozov, Vitali A.
<morozov@alcf.anl.gov>; Kumaran, Kalyan <kumaran@alcf.anl.gov>
Cc: Sodani, Avinash <avinash.sodani@intel.com>; Fromkin, Russ
<russ.fromkin@intel.com>; Allen, Benjamin S. <bsallen@alcf.anl.gov>
Subject: Re: [JLSE-Intel-NDA] KNL Status

ok, I am now in SNC-4/Flat. Same binary.

OMP_NUM_THREADS=64 KMP_AFFINITY=scatter numactl -m 4,5,6,7 ./stream
|tee running.dat

-------------------------------------------------------------
STREAM version $Revision: 5.9 $
-------------------------------------------------------------
This system uses 8 bytes per DOUBLE PRECISION word.
-------------------------------------------------------------
Array size = 134217728, Offset = 0
Total memory required = 3072.0 MB.
Each test is run 100 times, but only the *best* time for each is
used.
-------------------------------------------------------------
Number of Threads requested = 64
-------------------------------------------------------------

<skipped>

-------------------------------------------------------------
Function �����Rate (MB/s) ��Avg time ����Min time ����Max time
Copy: �����320997.8352 ������0.0068 ������0.0067 0.0071
Scale: ����319994.2893 ������0.0068 ������0.0067 0.0073
Add: ������306055.0205 ������0.0106 ������0.0105 0.0107
Triad: ����306055.0205 ������0.0106 ������0.0105 0.0109
Store: ����247056.8669 ������0.0044 ������0.0043 0.0051
-------------------------------------------------------------
Solution Validates
-------------------------------------------------------------


306055 MB/s

Vitali



On 04/13/2016 02:01 PM, Chinthamani, Sundaram wrote:
440 GB/s (base 10). ��To be precise, 440 bytes / ns (nano-second). ��We always report memory bandwidth in GB/s (base 10) within Intel.

-Thanks,
-Sundaram

-----Original Message-----
From: Harms, Kevin [mailto:harms@alcf.anl.gov]
Sent: Wednesday, April 13, 2016 11:52 AM
To: Raman, Karthik <karthik.raman@intel.com>; Chinthamani,
Sundaram <sundaram.chinthamani@intel.com>; Morozov, Vitali A.
<morozov@alcf.anl.gov>; Kumaran, Kalyan <kumaran@alcf.anl.gov>
Cc: Sodani, Avinash <avinash.sodani@intel.com>; Fromkin, Russ
<russ.fromkin@intel.com>; Allen, Benjamin S.
<bsallen@alcf.anl.gov>
Subject: Re: [JLSE-Intel-NDA] KNL Status


����Just to be clear, memory is always assumed to be measured using base 2. It's only the storage field that uses base 10. So we assume if you say 440 GB/s you mean GiB, or 450,560 MB/s.

kevin



And just some fun with units
http://sites.utexas.edu/jdm4372/2013/01/05/counting-binary-vs-dec
i
m
a
l
-powers-in-the-stream-benchmark/


From: Chinthamani, Sundaram

Sent: Wednesday, April 13, 2016 11:20 AM
To: Vitali A. Morozov <morozov@anl.gov>; Raman, Karthik
<karthik.raman@intel.com>; Kumaran, Kalyan <kumaran@alcf.anl.gov>
Cc: Sodani, Avinash <avinash.sodani@intel.com>; Fromkin, Russ
<russ.fromkin@intel.com>; Harms, Kevin <harms@alcf.anl.gov>;
Morozov, Vitali A. <morozov@alcf.anl.gov>; Allen, Benjamin S.
<bsallen@alcf.anl.gov>; Chinthamani, Sundaram
<sundaram.chinthamani@intel.com>
Subject: RE: [JLSE-Intel-NDA] KNL Status



440 GB/s �stream triad for MC-DRAM and 80 GB/s is our internal
bin3 measurement. �You should be able to get closer these numbers, if you run the benchmark in Quad/Flat or SNC-4/Flat mode.

-Thanks,
-Sundaram

From: Vitali A. Morozov [mailto:morozov@anl.gov]

Sent: Wednesday, April 13, 2016 11:11 AM
To: Raman, Karthik <karthik.raman@intel.com>; Chinthamani,
Sundaram <sundaram.chinthamani@intel.com>; Kumaran, Kalyan
<kumaran@alcf.anl.gov>
Cc: Sodani, Avinash <avinash.sodani@intel.com>; Fromkin, Russ
<russ.fromkin@intel.com>; Harms, Kevin <harms@alcf.anl.gov>;
Morozov, Vitali A. <morozov@alcf.anl.gov>; Allen, Benjamin S.
<bsallen@alcf.anl.gov>
Subject: Re: [JLSE-Intel-NDA] KNL Status



I see - I am treating 418122 MB/s as 418122/1024 GB/s (as this is memory unit).

Anyway - great we are on the same page here.

I am in All2All/Flat mode. You seem to suggest that I must be on SNC-4/Flat to see 440000 MB/s?

Thanks!

Vitali


On 04/13/2016 01:06 PM, Raman, Karthik wrote:


Ok I read this as 418 GB/s
J

And ok, just realized you are running in All2All mode, so yes this looks good. For Quad/Flat you should see ~440 GB/s with the same run.

From: Vitali A. Morozov [mailto:morozov@anl.gov]

Sent: Wednesday, April 13, 2016 11:02 AM
To: Raman, Karthik <karthik.raman@intel.com>
<mailto:karthik.raman@intel.com>; Chinthamani, Sundaram
<sundaram.chinthamani@intel.com>
<mailto:sundaram.chinthamani@intel.com>;
Kumaran, Kalyan <kumaran@alcf.anl.gov>
<mailto:kumaran@alcf.anl.gov>
Cc: Sodani, Avinash <avinash.sodani@intel.com>
<mailto:avinash.sodani@intel.com>;
Fromkin, Russ <russ.fromkin@intel.com>
<mailto:russ.fromkin@intel.com>; Harms, Kevin <harms@alcf.anl.gov> <mailto:harms@alcf.anl.gov>; Morozov, Vitali A.
<morozov@alcf.anl.gov> <mailto:morozov@alcf.anl.gov>; Allen, Benjamin S.
<bsallen@alcf.anl.gov> <mailto:bsallen@alcf.anl.gov>
Subject: Re: [JLSE-Intel-NDA] KNL Status



Hi Karthik,

Repeating the same options you have:

mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp -mcmodel
medium -shared-intel -opt-streaming-stores always stream.c -o
stream

-------------------------------------------------------------
STREAM version $Revision: 5.9 $
-------------------------------------------------------------
This system uses 8 bytes per DOUBLE PRECISION word.
-------------------------------------------------------------
Array size = 134217728, Offset = 0 Total memory required = 3072.0
MB.
Each test is run 100 times, but only the *best* time for each is
used.
-------------------------------------------------------------
Number of Threads requested = 64
-------------------------------------------------------------

< skiped >

Function �����Rate (MB/s) ��Avg time ����Min time ����Max time
Copy: �����362331.5200 ������0.0060 ������0.0059 ������0.0061
Scale: ����359281.9806 ������0.0060 ������0.0060 ������0.0062
Add: ������418122.7024 ������0.0078 ������0.0077 ������0.0081
Triad: ����418122.7024 ������0.0078 ������0.0077 ������0.0081
Store: ����224685.6729 ������0.0048 ������0.0048 ������0.0049


408GB/s

Vitali


On 04/13/2016 12:15 PM, Raman, Karthik wrote:

2. Effective MCDRAM bandwidth is around 406-408 GB/s. Larger values are possible for more high core count bins.



For bin3 , you should be able to get Triad ~440 GB/s. Please
check if you use the following Intel compiler flags -mcmodel
medium -shared-intel -O3 �-xMIC-AVX512 -DSTREAM_ARRAY_SIZE=134217728 -DOFFSET=0 -DNTIMES=100 -openmp -opt-streaming-stores always The array size is just allocating 1GB per array.


From: Vitali A. Morozov [mailto:morozov@anl.gov]

Sent: Wednesday, April 13, 2016 10:01 AM
To: Raman, Karthik <karthik.raman@intel.com>
<mailto:karthik.raman@intel.com>; Chinthamani, Sundaram
<sundaram.chinthamani@intel.com>
<mailto:sundaram.chinthamani@intel.com>;
Kumaran, Kalyan <kumaran@alcf.anl.gov>
<mailto:kumaran@alcf.anl.gov>
Cc: Sodani, Avinash <avinash.sodani@intel.com>
<mailto:avinash.sodani@intel.com>;
Fromkin, Russ <russ.fromkin@intel.com>
<mailto:russ.fromkin@intel.com>; Harms, Kevin <harms@alcf.anl.gov> <mailto:harms@alcf.anl.gov>; Morozov, Vitali A.
<morozov@alcf.anl.gov> <mailto:morozov@alcf.anl.gov>; Allen, Benjamin S.
<bsallen@alcf.anl.gov> <mailto:bsallen@alcf.anl.gov>
Subject: Re: [JLSE-Intel-NDA] KNL Status



Hello,

so, this is my understanding that on bin3 hardware (64 cores, 6
channel DDR4-2133)


1. Effective DDR bandwidth is 80 GB/s (around 78% of peak).

2. Effective MCDRAM bandwidth is around 406-408 GB/s. Larger values are possible for more high core count bins.

3. Alignment is not important for STREAM triad benchmark.

4. For both static and dynamic variables, numactl -m [0|1] should be used to direct memory placement.

5. For explicit MCDRAM allocation in flat mode, �hbw_malloc.h API should be use.

Please correct if I am wrong...

Thanks a lot for you help!

Vitali




On 04/12/2016 11:07 AM, Vitali A. Morozov wrote:

Hi Karthik,

I was also wandering how the variable could be influencing the allocation if, in principle, I do not need to link against memkind in my case. Anyway, sorry for confusion!

I think I did that before - see my first message. I was using "numactl --membind=1 ./stream" , which I believe is equivalent to "numactl -m 1 ./stream" that you suggest.

Here is the result of "numactl -H":

[morozov@knl00 512_3_C64_MPI16_TH08_flat]$ numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
21
22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43
44
45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66
67
68 69 70
71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92
93
94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111
112
113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128
129
130 131 132 133 134 135 136 137 138 139 140 141 142
143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158
159
160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175
176
177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192
193
194 195 196 197 198 199 200 201 202 203 204 205 206
207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222
223
224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239
240
241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 node
0
size: 196511 MB node 0 free: 186888 MB node 1 cpus:
node 1 size: 16384 MB
node 1 free: 11731 MB
node distances:
node ��0 ��1
���0: �10 �31
���1: �31 �10

Sounds like all as expected.


[morozov@knl00 STREAM]$ OMP_NUM_THREADS=64 KMP_AFFINITY=scatter numactl -m 1 ./stream
Triad: ����418262.7498 ������0.0058 ������0.0057 ������0.0060

For a series of runs, I see this:

# ��MB ��������������N ��������������GB/s
���228.88 ���������10000000 �������408867.98
���457.76 ���������20000000 �������405818.57
���686.65 ���������30000000 �������414309.08
���915.53 ���������40000000 �������415063.59
1144.41 ���������50000000 �������415792.22
1373.29 ���������60000000 �������415449.01
1602.17 ���������70000000 �������414106.18
1831.05 ���������80000000 �������416846.82
2059.94 ���������90000000 �������414992.29
2288.82 ��������100000000 �������417898.11

Vitali



On 04/12/2016 10:32 AM, Raman, Karthik wrote:

Hi Vitali,

1.
The MEMKIND_HBW_NODES is *only* when you are allocating using
hbwalloc (+ MCDRAM flat) in your application and when memkind is
unable to find the high bandwidth nodes (pmtt not populated)

2.
With the below
static allocations, in MCDRAM Flat mode, you can simply use �numactl �m �1 ./stream� with propery affinity settings to get peak out of MCDRAM.
a.
Check �numactl �H� to find the NUMA domains in your node. If
booted in All2All/FLAT or Quad/Flat mode it should show 2 NUMA
domains , one for DDR (96 GB) and other for MCDRAM (16GB)

Thanks
Karthik
From: Vitali A. Morozov [mailto:morozov@anl.gov]

Sent: Tuesday, April 12, 2016 7:27 AM
To: Raman, Karthik <karthik.raman@intel.com>
<mailto:karthik.raman@intel.com>; Chinthamani, Sundaram
<sundaram.chinthamani@intel.com>
<mailto:sundaram.chinthamani@intel.com>;
Kumaran, Kalyan <kumaran@alcf.anl.gov>
<mailto:kumaran@alcf.anl.gov>
Cc: Sodani, Avinash <avinash.sodani@intel.com>
<mailto:avinash.sodani@intel.com>;
Fromkin, Russ <russ.fromkin@intel.com>
<mailto:russ.fromkin@intel.com>; Harms, Kevin <harms@alcf.anl.gov> <mailto:harms@alcf.anl.gov>; Morozov, Vitali A.
<morozov@alcf.anl.gov> <mailto:morozov@alcf.anl.gov>; Allen, Benjamin S.
<bsallen@alcf.anl.gov> <mailto:bsallen@alcf.anl.gov>
Subject: Re: [JLSE-Intel-NDA] KNL Status



Hi Karthik,

The node is in All2All mode for now. I made the following changes. The original code is using static allocation therefore, I follow the advice on 2MB static array alignment:

[morozov@thing00 STREAM]$ head -n 108 < stream.c | tail -n 4
static double ��a[N+OFFSET] __attribute__((aligned(2097152))),
�����������������b[N+OFFSET] __attribute__((aligned(2097152))),
�����������������c[N+OFFSET] __attribute__((aligned(2097152))),
�����������������d[N+OFFSET]
__attribute__((aligned(2097152)));

[morozov@knl00 STREAM]$ make
mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp -mcmodel
medium -shared-intel -fno-alias stream.c -o stream


[morozov@knl00 STREAM]$ MEMKIND_HBW_NODES=1 OMP_NUM_THREADS=64
KMP_AFFINITY=scatter ./stream

.....

Triad: �����80251.3621 ������0.0300 ������0.0299 ������0.0305


Apparently, the "MEMKIND_HBW_NODES=1" is ignored.

Thanks!

Vitali



On 04/12/2016 01:51 AM, Raman, Karthik wrote:

Also could you please modify the memory allocations in STREAM to align at 2MB boundary.

For example:
static allocations:

static STREAM_TYPE ��a[STREAM_ARRAY_SIZE+OFFSET] __attribute__((aligned(2097152)))

dynamic allocations:
�����������������a = (STREAM_TYPE
*)_mm_malloc(sizeof(STREAM_TYPE)*(STREAM_ARRAY_SIZE+OFFSET),20971
5
2
)
;

memkind/hbwalloc:
�����(STREAM_TYPE *)hbw_posix_memalign((void **)&a, 2097152,
sizeof(STREAM_TYPE)*(STREAM_ARRAY_SIZE+OFFSET))

For memkind/hbwmalloc to work out of the box, you need to have
the PMTT tables populated so that it can identify the high bandwidth nodes. Alternatively you can set the MEMKIND_HBW_NODES environment variable to the high bandwidth node (MCDRAM).

e.g. All2All/Flat or Quad/Flat : export MEMKIND_HBW_NODES=1 ; ./app_with_hbwmalloc
����������SNC4/Flat: �export MEMKIND_HBW_NODES=4,5,6,7 ;
./app_with_hbw_malloc

Thanks
Karthik

From: Chinthamani, Sundaram

Sent: Monday, April 11, 2016 8:15 PM
To: Vitali Morozov <morozov@anl.gov> <mailto:morozov@anl.gov>;
Kumaran, Kalyan <kumaran@alcf.anl.gov>
<mailto:kumaran@alcf.anl.gov>
Cc: Sodani, Avinash <avinash.sodani@intel.com>
<mailto:avinash.sodani@intel.com>;
Fromkin, Russ <russ.fromkin@intel.com>
<mailto:russ.fromkin@intel.com>; Raman, Karthik
<karthik.raman@intel.com> <mailto:karthik.raman@intel.com>; Harms, Kevin <harms@alcf.anl.gov> <mailto:harms@alcf.anl.gov>; Morozov, Vitali A.
<morozov@alcf.anl.gov> <mailto:morozov@alcf.anl.gov>; Allen, Benjamin S.
<bsallen@alcf.anl.gov> <mailto:bsallen@alcf.anl.gov>;
Chinthamani, Sundaram <sundaram.chinthamani@intel.com>
<mailto:sundaram.chinthamani@intel.com>
Subject: RE: [JLSE-Intel-NDA] KNL Status



You should use numactl �m 4,5,6,7 �to get the memory bandwidth out of MC-DRAM.


What kind of part is it? Is it bin1 or bin3? ��Even in All2All mode, �we have measured up to 470 GB/s of stream triad bandwidth. ��Quad mode should get 490 GB/s.


DDR-stream triad bandwidth on Bin1-part should be close to 90 GB/s.

-Sundaram

From: Vitali Morozov [mailto:morozov@anl.gov]

Sent: Monday, April 11, 2016 6:59 PM
To: Kumaran, Kalyan <kumaran@alcf.anl.gov>
Cc: Sodani, Avinash <avinash.sodani@intel.com>; Fromkin, Russ
<russ.fromkin@intel.com>; Chinthamani, Sundaram
<sundaram.chinthamani@intel.com>; Raman, Karthik
<karthik.raman@intel.com>; Harms, Kevin <harms@alcf.anl.gov>;
Morozov, Vitali A. <morozov@alcf.anl.gov>; Allen, Benjamin S.
<bsallen@alcf.anl.gov>
Subject: Re: [JLSE-Intel-NDA] KNL Status



Avinash and others,

here is what we have so far for Flat more, All2All.

The results from memory were obtained by using default STREAM with static allocation. The results from MCDRAM were obtained by using modified STREAM with hbwmalloc.h interface.

1 MPI process, 64 threads, scatter affinity, no numactl is used.

Most results are as expected: 80 GB/s from DDR memory, 420 GB/s from MCDRAM, the MCDRAM size is about 16 GB, the aggregate L2 cache size is 32 MB.

Noticed inconsistencies for hbwmalloc benchmarks:

1. The result is not validated - will spend more time on that tomorrow.
2. hbw_malloc continues allocating data beyond the expected 16 GB up to 48 GB. This means it fails back to DDR if it cannot allocate in MCDRAM - not expected default behavior. It must fail if it does not have enough space.

The problem shows up when I am running static allocation and use numactl --membind=1 to ask for MCDRAM-only allocation. The variation is very large and inconsistent with the result observed when using hbw_malloc API:

# �MB ����������������N ���������GB/s numactl �GB/s hbw_malloc
���228.88 ���������10000000 �������255555.46 �����421009.18
���457.76 ���������20000000 �������363930.93 �����416997.91
���686.65 ���������30000000 �������309415.87 �����427808.31
���915.53 ���������40000000 �������371691.30 �����423310.75
1144.41 ���������50000000 �������187002.22 �����431032.35
1373.29 ���������60000000 �������385954.23 �����427172.91
1602.17 ���������70000000 �������388254.49 �����426823.59
1831.05 ���������80000000 �������374491.43 �����428103.96
2059.94 ���������90000000 �������379622.74 �����428152.02
2288.82 ��������100000000 ��������80868.34 �����430185.03

We also tried SNC-4 instead of All2All, but could not get more than 40 GB/s from MCDRAM even with hbw_malloc API.

Raw data:

# �flat memory, running from memory (default), alltoall
# �MB ����������������N �������������GB/s
�����2.29 �����������100000 �������272062.96
�����4.58 �����������200000 �������491040.47
�����6.87 �����������300000 �������603979.78
�����9.16 �����������400000 �������745654.04
����11.44 �����������500000 �������811800.77
����13.73 �����������600000 �������901462.35
����16.02 �����������700000 �������891953.26
����18.31 �����������800000 ������1019375.15
����20.60 �����������900000 �������984749.63
����22.89 ����������1000000 �������923516.48
����25.18 ����������1100000 �������758422.09
����27.47 ����������1200000 �������779328.74
����29.75 ����������1300000 �������513185.43
����32.04 ����������1400000 �������155550.35
����34.33 ����������1500000 �������103209.12
����36.62 ����������1600000 ��������97435.74
����38.91 ����������1700000 ��������83968.40
����41.20 ����������1800000 ��������83692.35
����43.49 ����������1900000 ��������80734.60
����45.78 ����������2000000 ��������76317.89
����68.66 ����������3000000 ��������81180.08
����91.55 ����������4000000 ��������78889.73
���114.44 ����������5000000 ��������79891.50
���137.33 ����������6000000 ��������79954.96
���160.22 ����������7000000 ��������79477.00
���183.11 ����������8000000 ��������79145.59
���205.99 ����������9000000 ��������79821.12
���228.88 ���������10000000 ��������79519.15
���228.88 ���������10000000 ��������79212.54
���457.76 ���������20000000 ��������79692.27
���686.65 ���������30000000 ��������79866.15
���915.53 ���������40000000 ��������79828.15
1144.41 ���������50000000 ��������80214.91
1373.29 ���������60000000 ��������80226.84
1602.17 ���������70000000 ��������80178.77
1831.05 ���������80000000 ��������80177.06
2059.94 ���������90000000 ��������80109.79
2288.82 ��������100000000 ��������80175.94

# �flat memory, running from MCDRAM, alltoall, hbwmalloc API
# �MB ����������������N �������������GB/s
���228.88 ���������10000000 �������421009.18
���457.76 ���������20000000 �������416997.91
���686.65 ���������30000000 �������427808.31
���915.53 ���������40000000 �������423310.75
1144.41 ���������50000000 �������431032.35
1373.29 ���������60000000 �������427172.91
1602.17 ���������70000000 �������426823.59
1831.05 ���������80000000 �������428103.96
2059.94 ���������90000000 �������428152.02
2288.82 ��������100000000 �������430185.03
2343.75 ��������102400000 �������425999.98
4687.50 ��������204800000 �������427821.10
9155.27 ��������400000000 �������429318.13
9375.00 ��������409600000 �������428376.70
11444.09 ��������500000000 �������428052.08
13732.91 ��������600000000 �������266061.01
16021.73 ��������700000000 �������200257.79
18310.55 ��������800000000 �������168164.19
20599.37 ��������900000000 �������150192.91
22888.18 �������1000000000 �������137862.07
25177.00 �������1100000000 �������129418.12
27465.82 �������1200000000 �������123019.53
29754.64 �������1300000000 �������118268.79
32043.46 �������1400000000 �������114343.33
34332.28 �������1500000000 �������110991.25
36621.09 �������1600000000 �������108538.36
38909.91 �������1700000000 �������106249.72
41198.73 �������1800000000 �������104455.07
43487.55 �������1900000000 �������102828.93
45776.37 �������2000000000 �������101348.56
48065.19 �������2100000000 �������100013.73


We appreciate any help on that.

Best,

Vitali



On 04/11/2016 08:42 PM, Kumaran, Kalyan wrote:

Avinash, thanks for the prompt reply. Kevin/Vitali can provide
all the details. The short summary is we are unable to reproduce the STREAM results from IPM or DDR. It would be great if Karthik or Sundaram can advise us on how to run (which mode etc) or if they could log into the machine and make sure we have set up things correctly.



Thanks,

Kumar





On Apr 11, 2016, at 8:07 PM, Sodani, Avinash <avinash.sodani@intel.com> wrote:


Copying Sundaram/Karthik from our side.



Kumar,



Can you send us more information on this?



-Avinash



From: Kumaran,
Kalyan [mailto:kumaran@alcf.anl.gov]
Sent: Monday, April 11, 2016 5:32 PM
To: Fromkin, Russ; Sodani, Avinash
Cc: Harms, Kevin; Morozov, Vitali A.
Subject: Fwd: [JLSE-Intel-NDA] KNL Status





Russ, Avinash,





we need help in understanding how to obtain STREAM results. Can anyone help?





Thanks,


Kumar














Begin forwarded message:




From: "Harms, Kevin" <harms@alcf.anl.gov>


Subject: [JLSE-Intel-NDA] KNL Status


Date: April 11, 2016 at 5:42:15 PM CDT


To: "intel-nda@jlse.anl.gov"
<intel-nda@jlse.anl.gov>





The knl00 node is currently in Flat mode and All2All model. With this setup we can get 420 GB/s from IPM memory on STREAM Triad when using hbw_malloc and friends which was confirmed by both Vitali and I.

Changing the BIOS settings and warm rebooting seem to be problematic. Vitali and I were running earlier and could not get over 80 GB/s with any method even though memory capacity indicated flat mode.

Using numactl --membind is also problematic. The STREAM results
when using this to launch the application were erratic, so it is also suggested not to use that. That means you will need to modify your code to use dynamic memory and hbw_malloc to exploit the IPM.

Will followup with more detailed results later.

kevin
_______________________________________________
Intel-nda mailing list
Intel-nda@lists.jlse.anl.gov
https://lists.jlse.anl.gov/mailman/listinfo/intel-nda






























<flat_a2a_ddr.txt><flat_a2a_ipm.txt><flat_quad_ddr.txt><flat_quad_ipm.txt><flat_snc4_ddr.txt><flat_snc4_ipm.txt>



_______________________________________________
Intel-nda mailing list
Intel-nda@lists.jlse.anl.gov
https://lists.jlse.anl.gov/mailman/listinfo/intel-nda





_______________________________________________
Intel-nda mailing list
Intel-nda@lists.jlse.anl.gov
https://lists.jlse.anl.gov/mailman/listinfo/intel-nda




_______________________________________________
Intel-nda mailing list
Intel-nda@lists.jlse.anl.gov
https://lists.jlse.anl.gov/mailman/listinfo/intel-nda