I would expect Quad mode / SNC-4 mode to provide higher bandwidth compared to All2All mode. We haven't seen this in our internal measurements.

-Thanks,
-Sundaram

-----Original Message-----
From: Harms, Kevin [mailto:harms@alcf.anl.gov]
Sent: Monday, April 18, 2016 12:00 PM
To: Allen, Benjamin S. <bsallen@alcf.anl.gov>
Cc: Kumaran, Kalyan <kumaran@alcf.anl.gov>; Kaana Balakrishna, Abhishek <abhishek.kaana.balakrishna@intel.com>; Morozov, Vitali A. <morozov@alcf.anl.gov>; Chinthamani, Sundaram <sundaram.chinthamani@intel.com>; Raman, Karthik <karthik.raman@intel.com>; Inkley, Benson <benson.inkley@intel.com>; Sodani, Avinash <avinash.sodani@intel.com>; Fromkin, Russ <russ.fromkin@intel.com>
Subject: Re: [JLSE-Intel-NDA] KNL Status

I tested cached mode with all2all, snc4 and quad models. Cached all2all gave 275,000 MB/s. Does that look reasonable? SNC4 and quad mode gave 225,000 MB/s or so.

kevin

The following versions are now installed: BIOS 01.01.0104, ME 03.01.03.011.0, BMC 00.17.9699, FRUSDR 1.14.

Ben

On Apr 18, 2016, at 1:27 PM, Harms, Kevin <harms@alcf.anl.gov> wrote:

Ben completed the BIOS upgrade, but we do not see any difference in performance. I have attached results from runs done under three configurations: flat+all2all, flat+quad, flat+snc4.

flat+all2all = 427046.0546 MB/s

flat+quad = 327253.8882 MB/s

flat+snc4 = 308555.9588 MB/s

I compiled the CORAL stream.c with the following options:

icc -mcmodel medium -shared-intel -O3 -xMIC-AVX512 -DN=50000000 -DOFFSET=0 -DNTIMES=100 -openmp -opt-streaming-stores always stream.c -o stream

I added the alignment directives to the arrays. The attached files show the NUMA domains and numactl commands used and run parameters.

Have I missed a parameter?

kevin

Suggest a time Friday afternoon. We are in central time zone. We can get you access to this B0 too.

-Kumar

On Apr 13, 2016, at 5:06 PM, Kaana Balakrishna, Abhishek <abhishek.kaana.balakrishna@intel.com> wrote:

+ Benson

Cheers,

Abhishek K Balakrishna (KB)

Xeon Phi™ Hardware Platform Applications Engineer,

Technical Computing Group (TCG)

Intel Corporation

Ph - 512 466 9886

www.intel.com/xeonphi

-----Original Message-----

From: Vitali A. Morozov [mailto:morozov@anl.gov]

Sent: Wednesday, April 13, 2016 4:58 PM

To: Chinthamani, Sundaram <sundaram.chinthamani@intel.com>; Raman, Karthik <karthik.raman@intel.com>; Harms, Kevin <harms@alcf.anl.gov>; Morozov, Vitali A. <morozov@alcf.anl.gov>; Kumaran, Kalyan <kumaran@alcf.anl.gov>

Cc: Sodani, Avinash <avinash.sodani@intel.com>; Fromkin, Russ <russ.fromkin@intel.com>; Allen, Benjamin S. <bsallen@alcf.anl.gov>; Kaana Balakrishna, Abhishek <abhishek.kaana.balakrishna@intel.com>

Subject: Re: [JLSE-Intel-NDA] KNL Status

Sure!

Vitali

On 04/13/2016 04:54 PM, Chinthamani, Sundaram wrote:

Would it be possible to have a meeting to go over your questions rather than exchanging E-mails back and forth?

+ KB from Intel side. He can help set-up the meeting between Labs and Intel.

-Thanks,

-Sundaram

-----Original Message-----

From: Vitali A. Morozov [mailto:morozov@anl.gov]

Sent: Wednesday, April 13, 2016 2:50 PM

To: Chinthamani, Sundaram <sundaram.chinthamani@intel.com>; Raman,

Karthik <karthik.raman@intel.com>; Harms, Kevin <harms@alcf.anl.gov>;

Morozov, Vitali A. <morozov@alcf.anl.gov>; Kumaran, Kalyan

<kumaran@alcf.anl.gov>

Cc: Sodani, Avinash <avinash.sodani@intel.com>; Fromkin, Russ

<russ.fromkin@intel.com>; Allen, Benjamin S. <bsallen@alcf.anl.gov>

Subject: Re: [JLSE-Intel-NDA] KNL Status

On 04/13/2016 04:42 PM, Chinthamani, Sundaram wrote:

node 4 free: 2907 MB

node 5 free: 2926 MB

node 6 free: 2922 MB

node 7 free: 2914 MB

There is 16 GB of MC-DRAM and 4 GB per NUMA node. Is there any other process running on the machine? Why does "free memory" show only 3 GB per node?

This is EXACTLY why I am asking this question. Nothing is running on the node - and I do not see full 4 GiB of available MCDRAM memory. This might be reason why I do not see what is expected.

I understand that I can reduce the array size, but I am trying to reproduce your results.

I am still unclear on if you run "no mpi" version, how come that SNC-4 mode is giving you advantage? All arrays are allocated in the same quadrant. Requests from the other quadrants will go to this one and create congestion.

Vitali

If you want to test Stream triad bandwidth to MC-DRAM in Quad/Flat mode, you can use even smaller arrays of few 100 MBs.

-Thanks,

-Sundaram

-----Original Message-----

From: Vitali A. Morozov [mailto:morozov@anl.gov]

Sent: Wednesday, April 13, 2016 2:32 PM

To: Raman, Karthik <karthik.raman@intel.com>; Chinthamani, Sundaram

<sundaram.chinthamani@intel.com>; Harms, Kevin <harms@alcf.anl.gov>;

Morozov, Vitali A. <morozov@alcf.anl.gov>; Kumaran, Kalyan

<kumaran@alcf.anl.gov>

Cc: Sodani, Avinash <avinash.sodani@intel.com>; Fromkin, Russ

<russ.fromkin@intel.com>; Allen, Benjamin S. <bsallen@alcf.anl.gov>

Subject: Re: [JLSE-Intel-NDA] KNL Status

ok, that is fine.

Now the next issue I have is that for static arrays, all three arrays must land up on one of the numa nodes - the node that is attached to a master thread.

You advice to use N = 134217728, that gives exactly 3 GiB of memory footprint for 3 arrays of triad.

According to "numactl -H", I do not have 3 GiB of available memory on any of the MCDRAM numa nodes:

numactl -H |grep free

node 0 free: 46336 MB

node 1 free: 46923 MB

node 2 free: 46970 MB

node 3 free: 46961 MB

node 4 free: 2907 MB

node 5 free: 2926 MB

node 6 free: 2922 MB

node 7 free: 2914 MB

Therefore, at least one array will not be placed to the same numa node.

Is this ok?

On 04/13/2016 04:17 PM, Raman, Karthik wrote:

Yes we run pure OMP code too.

The problem with KMP_AFFINITY with SNC4 mode (currently) is that it

may not equally balance the threads per NUMA domain even with

scatter or balanced, you can check that by mapping the "verbose" output (e.g.

KMP_AFFINITY=scatter,verbose) with "numactl -H" output to see if the

OS thread to logical CPU mapping is balanced across all NUMA domains

With All2All/Quad this should not be a problem

-----Original Message-----

From: Vitali A. Morozov [mailto:morozov@anl.gov]

Sent: Wednesday, April 13, 2016 1:53 PM

To: Raman, Karthik <karthik.raman@intel.com>; Chinthamani, Sundaram

<sundaram.chinthamani@intel.com>; Harms, Kevin <harms@alcf.anl.gov>;

Morozov, Vitali A. <morozov@alcf.anl.gov>; Kumaran, Kalyan

<kumaran@alcf.anl.gov>

Cc: Sodani, Avinash <avinash.sodani@intel.com>; Fromkin, Russ

<russ.fromkin@intel.com>; Allen, Benjamin S. <bsallen@alcf.anl.gov>

Subject: Re: [JLSE-Intel-NDA] KNL Status

Before doing that - let us check if we are running similar benchmark.

I am running 1 MPI process (in fact no mpi at all) and 64 OpenMP threads version. The threads can be placed with "balanced" or "scattered"

policy. I am using "scattered".

Is this ok so far?

Vitali

On 04/13/2016 03:48 PM, Raman, Karthik wrote:

This still does not look right, and the reason could be that the

threads are not balanced across all the NUMA domains (ideally you

want

16 threads per quadrant)

Before going to SNC4, can you please boot in "Quad" mode. If you

look in the BIOS Cluster mode options there is another mode called "Quad"

which is different from SNC4, only 2 NUMA domains (DDR, MCDRAM).

Please run similar to All2All mode

-----Original Message-----

From: Vitali A. Morozov [mailto:morozov@anl.gov]

Sent: Wednesday, April 13, 2016 12:56 PM

To: Chinthamani, Sundaram <sundaram.chinthamani@intel.com>; Harms,

Kevin <harms@alcf.anl.gov>; Raman, Karthik

<karthik.raman@intel.com>; Morozov, Vitali A.

<morozov@alcf.anl.gov>; Kumaran, Kalyan <kumaran@alcf.anl.gov>

Cc: Sodani, Avinash <avinash.sodani@intel.com>; Fromkin, Russ

<russ.fromkin@intel.com>; Allen, Benjamin S. <bsallen@alcf.anl.gov>

Subject: Re: [JLSE-Intel-NDA] KNL Status

ok, I am now in SNC-4/Flat. Same binary.

OMP_NUM_THREADS=64 KMP_AFFINITY=scatter numactl -m 4,5,6,7 ./stream

|tee running.dat

-------------------------------------------------------------

STREAM version $Revision: 5.9 $

-------------------------------------------------------------

This system uses 8 bytes per DOUBLE PRECISION word.

-------------------------------------------------------------

Array size = 134217728, Offset = 0

Total memory required = 3072.0 MB.

Each test is run 100 times, but only the *best* time for each is

used.

-------------------------------------------------------------

Number of Threads requested = 64

-------------------------------------------------------------

<skipped>

-------------------------------------------------------------

Function Rate (MB/s) Avg time Min time Max time

Copy: 320997.8352 0.0068 0.0067 0.0071

Scale: 319994.2893 0.0068 0.0067 0.0073

Add: 306055.0205 0.0106 0.0105 0.0107

Triad: 306055.0205 0.0106 0.0105 0.0109

Store: 247056.8669 0.0044 0.0043 0.0051

-------------------------------------------------------------

Solution Validates

-------------------------------------------------------------

306055 MB/s

Vitali

On 04/13/2016 02:01 PM, Chinthamani, Sundaram wrote:

440 GB/s (base 10). To be precise, 440 bytes / ns (nano-second). We always report memory bandwidth in GB/s (base 10) within Intel.

-Thanks,

-Sundaram

-----Original Message-----

From: Harms, Kevin [mailto:harms@alcf.anl.gov]

Sent: Wednesday, April 13, 2016 11:52 AM

To: Raman, Karthik <karthik.raman@intel.com>; Chinthamani,

Sundaram <sundaram.chinthamani@intel.com>; Morozov, Vitali A.

<morozov@alcf.anl.gov>; Kumaran, Kalyan <kumaran@alcf.anl.gov>

Cc: Sodani, Avinash <avinash.sodani@intel.com>; Fromkin, Russ

<russ.fromkin@intel.com>; Allen, Benjamin S.

<bsallen@alcf.anl.gov>

Subject: Re: [JLSE-Intel-NDA] KNL Status

Just to be clear, memory is always assumed to be measured using base 2. It's only the storage field that uses base 10. So we assume if you say 440 GB/s you mean GiB, or 450,560 MB/s.

kevin

And just some fun with units

http://sites.utexas.edu/jdm4372/2013/01/05/counting-binary-vs-dec

i

m

a

l

-powers-in-the-stream-benchmark/

From: Chinthamani, Sundaram

Sent: Wednesday, April 13, 2016 11:20 AM

To: Vitali A. Morozov <morozov@anl.gov>; Raman, Karthik

<karthik.raman@intel.com>; Kumaran, Kalyan <kumaran@alcf.anl.gov>

Cc: Sodani, Avinash <avinash.sodani@intel.com>; Fromkin, Russ

<russ.fromkin@intel.com>; Harms, Kevin <harms@alcf.anl.gov>;

Morozov, Vitali A. <morozov@alcf.anl.gov>; Allen, Benjamin S.

<bsallen@alcf.anl.gov>; Chinthamani, Sundaram

<sundaram.chinthamani@intel.com>

Subject: RE: [JLSE-Intel-NDA] KNL Status

440 GB/s stream triad for MC-DRAM and 80 GB/s is our internal

bin3 measurement. You should be able to get closer these numbers, if you run the benchmark in Quad/Flat or SNC-4/Flat mode.

-Thanks,

-Sundaram

From: Vitali A. Morozov [mailto:morozov@anl.gov]

Sent: Wednesday, April 13, 2016 11:11 AM

To: Raman, Karthik <karthik.raman@intel.com>; Chinthamani,

Sundaram <sundaram.chinthamani@intel.com>; Kumaran, Kalyan

<kumaran@alcf.anl.gov>

Cc: Sodani, Avinash <avinash.sodani@intel.com>; Fromkin, Russ

<russ.fromkin@intel.com>; Harms, Kevin <harms@alcf.anl.gov>;

Morozov, Vitali A. <morozov@alcf.anl.gov>; Allen, Benjamin S.

<bsallen@alcf.anl.gov>

Subject: Re: [JLSE-Intel-NDA] KNL Status

I see - I am treating 418122 MB/s as 418122/1024 GB/s (as this is memory unit).

Anyway - great we are on the same page here.

I am in All2All/Flat mode. You seem to suggest that I must be on SNC-4/Flat to see 440000 MB/s?

Thanks!

Vitali

On 04/13/2016 01:06 PM, Raman, Karthik wrote:

Ok I read this as 418 GB/s

J

And ok, just realized you are running in All2All mode, so yes this looks good. For Quad/Flat you should see ~440 GB/s with the same run.

From: Vitali A. Morozov [mailto:morozov@anl.gov]

Sent: Wednesday, April 13, 2016 11:02 AM

To: Raman, Karthik <karthik.raman@intel.com>

<mailto:karthik.raman@intel.com>; Chinthamani, Sundaram

<sundaram.chinthamani@intel.com>

<mailto:sundaram.chinthamani@intel.com>;

Kumaran, Kalyan <kumaran@alcf.anl.gov>

<mailto:kumaran@alcf.anl.gov>

Cc: Sodani, Avinash <avinash.sodani@intel.com>

<mailto:avinash.sodani@intel.com>;

Fromkin, Russ <russ.fromkin@intel.com>

<mailto:russ.fromkin@intel.com>; Harms, Kevin <harms@alcf.anl.gov> <mailto:harms@alcf.anl.gov>; Morozov, Vitali A.

<morozov@alcf.anl.gov> <mailto:morozov@alcf.anl.gov>; Allen, Benjamin S.

<bsallen@alcf.anl.gov> <mailto:bsallen@alcf.anl.gov>

Subject: Re: [JLSE-Intel-NDA] KNL Status

Hi Karthik,

Repeating the same options you have:

mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp -mcmodel

medium -shared-intel -opt-streaming-stores always stream.c -o

stream

-------------------------------------------------------------

STREAM version $Revision: 5.9 $

-------------------------------------------------------------

This system uses 8 bytes per DOUBLE PRECISION word.

-------------------------------------------------------------

Array size = 134217728, Offset = 0 Total memory required = 3072.0

MB.

Each test is run 100 times, but only the *best* time for each is

used.

-------------------------------------------------------------

Number of Threads requested = 64

-------------------------------------------------------------

< skiped >

Function Rate (MB/s) Avg time Min time Max time

Copy: 362331.5200 0.0060 0.0059 0.0061

Scale: 359281.9806 0.0060 0.0060 0.0062

Add: 418122.7024 0.0078 0.0077 0.0081

Triad: 418122.7024 0.0078 0.0077 0.0081

Store: 224685.6729 0.0048 0.0048 0.0049

408GB/s

Vitali

On 04/13/2016 12:15 PM, Raman, Karthik wrote:

2. Effective MCDRAM bandwidth is around 406-408 GB/s. Larger values are possible for more high core count bins.

For bin3 , you should be able to get Triad ~440 GB/s. Please

check if you use the following Intel compiler flags -mcmodel

medium -shared-intel -O3 -xMIC-AVX512 -DSTREAM_ARRAY_SIZE=134217728 -DOFFSET=0 -DNTIMES=100 -openmp -opt-streaming-stores always The array size is just allocating 1GB per array.

From: Vitali A. Morozov [mailto:morozov@anl.gov]

Sent: Wednesday, April 13, 2016 10:01 AM

To: Raman, Karthik <karthik.raman@intel.com>

<mailto:karthik.raman@intel.com>; Chinthamani, Sundaram

<sundaram.chinthamani@intel.com>

<mailto:sundaram.chinthamani@intel.com>;

Kumaran, Kalyan <kumaran@alcf.anl.gov>

<mailto:kumaran@alcf.anl.gov>

Cc: Sodani, Avinash <avinash.sodani@intel.com>

<mailto:avinash.sodani@intel.com>;

Fromkin, Russ <russ.fromkin@intel.com>

<mailto:russ.fromkin@intel.com>; Harms, Kevin <harms@alcf.anl.gov> <mailto:harms@alcf.anl.gov>; Morozov, Vitali A.

<morozov@alcf.anl.gov> <mailto:morozov@alcf.anl.gov>; Allen, Benjamin S.

<bsallen@alcf.anl.gov> <mailto:bsallen@alcf.anl.gov>

Subject: Re: [JLSE-Intel-NDA] KNL Status

Hello,

so, this is my understanding that on bin3 hardware (64 cores, 6

channel DDR4-2133)

1. Effective DDR bandwidth is 80 GB/s (around 78% of peak).

2. Effective MCDRAM bandwidth is around 406-408 GB/s. Larger values are possible for more high core count bins.

3. Alignment is not important for STREAM triad benchmark.

4. For both static and dynamic variables, numactl -m [0|1] should be used to direct memory placement.

5. For explicit MCDRAM allocation in flat mode, hbw_malloc.h API should be use.

Please correct if I am wrong...

Thanks a lot for you help!

Vitali

On 04/12/2016 11:07 AM, Vitali A. Morozov wrote:

Hi Karthik,

I was also wandering how the variable could be influencing the allocation if, in principle, I do not need to link against memkind in my case. Anyway, sorry for confusion!

I think I did that before - see my first message. I was using "numactl --membind=1 ./stream" , which I believe is equivalent to "numactl -m 1 ./stream" that you suggest.

Here is the result of "numactl -H":

[morozov@knl00 512_3_C64_MPI16_TH08_flat]$ numactl -H

available: 2 nodes (0-1)

node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

21

22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43

44

45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66

67

68 69 70

71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92

93

94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111

112

113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128

129

130 131 132 133 134 135 136 137 138 139 140 141 142

143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158

159

160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175

176

177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192

193

194 195 196 197 198 199 200 201 202 203 204 205 206

207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222

223

224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239

240

241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 node

0

size: 196511 MB node 0 free: 186888 MB node 1 cpus:

node 1 size: 16384 MB

node 1 free: 11731 MB

node distances:

node 0 1

0: 10 31

1: 31 10

Sounds like all as expected.

[morozov@knl00 STREAM]$ OMP_NUM_THREADS=64 KMP_AFFINITY=scatter numactl -m 1 ./stream

Triad: 418262.7498 0.0058 0.0057 0.0060

For a series of runs, I see this:

# MB N GB/s

228.88 10000000 408867.98

457.76 20000000 405818.57

686.65 30000000 414309.08

915.53 40000000 415063.59

1144.41 50000000 415792.22

1373.29 60000000 415449.01

1602.17 70000000 414106.18

1831.05 80000000 416846.82

2059.94 90000000 414992.29

2288.82 100000000 417898.11

Vitali

On 04/12/2016 10:32 AM, Raman, Karthik wrote:

Hi Vitali,

1.

The MEMKIND_HBW_NODES is *only* when you are allocating using

hbwalloc (+ MCDRAM flat) in your application and when memkind is

unable to find the high bandwidth nodes (pmtt not populated)

2.

With the below

static allocations, in MCDRAM Flat mode, you can simply use “numactl –m 1 ./stream” with propery affinity settings to get peak out of MCDRAM.

a.

Check “numactl –H” to find the NUMA domains in your node. If

booted in All2All/FLAT or Quad/Flat mode it should show 2 NUMA

domains , one for DDR (96 GB) and other for MCDRAM (16GB)

Thanks

Karthik

From: Vitali A. Morozov [mailto:morozov@anl.gov]

Sent: Tuesday, April 12, 2016 7:27 AM

To: Raman, Karthik <karthik.raman@intel.com>

<mailto:karthik.raman@intel.com>; Chinthamani, Sundaram

<sundaram.chinthamani@intel.com>

<mailto:sundaram.chinthamani@intel.com>;

Kumaran, Kalyan <kumaran@alcf.anl.gov>

<mailto:kumaran@alcf.anl.gov>

Cc: Sodani, Avinash <avinash.sodani@intel.com>

<mailto:avinash.sodani@intel.com>;

Fromkin, Russ <russ.fromkin@intel.com>

<mailto:russ.fromkin@intel.com>; Harms, Kevin <harms@alcf.anl.gov> <mailto:harms@alcf.anl.gov>; Morozov, Vitali A.

<morozov@alcf.anl.gov> <mailto:morozov@alcf.anl.gov>; Allen, Benjamin S.

<bsallen@alcf.anl.gov> <mailto:bsallen@alcf.anl.gov>

Subject: Re: [JLSE-Intel-NDA] KNL Status

Hi Karthik,

The node is in All2All mode for now. I made the following changes. The original code is using static allocation therefore, I follow the advice on 2MB static array alignment:

[morozov@thing00 STREAM]$ head -n 108 < stream.c | tail -n 4

static double a[N+OFFSET] __attribute__((aligned(2097152))),

b[N+OFFSET] __attribute__((aligned(2097152))),

c[N+OFFSET] __attribute__((aligned(2097152))),

d[N+OFFSET]

__attribute__((aligned(2097152)));

[morozov@knl00 STREAM]$ make

mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp -mcmodel

medium -shared-intel -fno-alias stream.c -o stream

[morozov@knl00 STREAM]$ MEMKIND_HBW_NODES=1 OMP_NUM_THREADS=64

KMP_AFFINITY=scatter ./stream

.....

Triad: 80251.3621 0.0300 0.0299 0.0305

Apparently, the "MEMKIND_HBW_NODES=1" is ignored.

Thanks!

Vitali

On 04/12/2016 01:51 AM, Raman, Karthik wrote:

Also could you please modify the memory allocations in STREAM to align at 2MB boundary.

For example:

static allocations:

static STREAM_TYPE a[STREAM_ARRAY_SIZE+OFFSET] __attribute__((aligned(2097152)))

dynamic allocations:

a = (STREAM_TYPE

*)_mm_malloc(sizeof(STREAM_TYPE)*(STREAM_ARRAY_SIZE+OFFSET),20971

5

2

)

;

memkind/hbwalloc:

(STREAM_TYPE *)hbw_posix_memalign((void **)&a, 2097152,

sizeof(STREAM_TYPE)*(STREAM_ARRAY_SIZE+OFFSET))

For memkind/hbwmalloc to work out of the box, you need to have

the PMTT tables populated so that it can identify the high bandwidth nodes. Alternatively you can set the MEMKIND_HBW_NODES environment variable to the high bandwidth node (MCDRAM).

e.g. All2All/Flat or Quad/Flat : export MEMKIND_HBW_NODES=1 ; ./app_with_hbwmalloc

SNC4/Flat: export MEMKIND_HBW_NODES=4,5,6,7 ;

./app_with_hbw_malloc

Thanks

Karthik

From: Chinthamani, Sundaram

Sent: Monday, April 11, 2016 8:15 PM

To: Vitali Morozov <morozov@anl.gov> <mailto:morozov@anl.gov>;

Kumaran, Kalyan <kumaran@alcf.anl.gov>

<mailto:kumaran@alcf.anl.gov>

Cc: Sodani, Avinash <avinash.sodani@intel.com>

<mailto:avinash.sodani@intel.com>;

Fromkin, Russ <russ.fromkin@intel.com>

<mailto:russ.fromkin@intel.com>; Raman, Karthik

<karthik.raman@intel.com> <mailto:karthik.raman@intel.com>; Harms, Kevin <harms@alcf.anl.gov> <mailto:harms@alcf.anl.gov>; Morozov, Vitali A.

<morozov@alcf.anl.gov> <mailto:morozov@alcf.anl.gov>; Allen, Benjamin S.

<bsallen@alcf.anl.gov> <mailto:bsallen@alcf.anl.gov>;

Chinthamani, Sundaram <sundaram.chinthamani@intel.com>

<mailto:sundaram.chinthamani@intel.com>

Subject: RE: [JLSE-Intel-NDA] KNL Status

You should use numactl –m 4,5,6,7 to get the memory bandwidth out of MC-DRAM.

What kind of part is it? Is it bin1 or bin3? Even in All2All mode, we have measured up to 470 GB/s of stream triad bandwidth. Quad mode should get 490 GB/s.

DDR-stream triad bandwidth on Bin1-part should be close to 90 GB/s.

-Sundaram

From: Vitali Morozov [mailto:morozov@anl.gov]

Sent: Monday, April 11, 2016 6:59 PM

To: Kumaran, Kalyan <kumaran@alcf.anl.gov>

Cc: Sodani, Avinash <avinash.sodani@intel.com>; Fromkin, Russ

<russ.fromkin@intel.com>; Chinthamani, Sundaram

<sundaram.chinthamani@intel.com>; Raman, Karthik

<karthik.raman@intel.com>; Harms, Kevin <harms@alcf.anl.gov>;

Morozov, Vitali A. <morozov@alcf.anl.gov>; Allen, Benjamin S.

<bsallen@alcf.anl.gov>

Subject: Re: [JLSE-Intel-NDA] KNL Status

Avinash and others,

here is what we have so far for Flat more, All2All.

The results from memory were obtained by using default STREAM with static allocation. The results from MCDRAM were obtained by using modified STREAM with hbwmalloc.h interface.

1 MPI process, 64 threads, scatter affinity, no numactl is used.

Most results are as expected: 80 GB/s from DDR memory, 420 GB/s from MCDRAM, the MCDRAM size is about 16 GB, the aggregate L2 cache size is 32 MB.

Noticed inconsistencies for hbwmalloc benchmarks:

1. The result is not validated - will spend more time on that tomorrow.

2. hbw_malloc continues allocating data beyond the expected 16 GB up to 48 GB. This means it fails back to DDR if it cannot allocate in MCDRAM - not expected default behavior. It must fail if it does not have enough space.

The problem shows up when I am running static allocation and use numactl --membind=1 to ask for MCDRAM-only allocation. The variation is very large and inconsistent with the result observed when using hbw_malloc API:

# MB N GB/s numactl GB/s hbw_malloc

228.88 10000000 255555.46 421009.18

457.76 20000000 363930.93 416997.91

686.65 30000000 309415.87 427808.31

915.53 40000000 371691.30 423310.75

1144.41 50000000 187002.22 431032.35

1373.29 60000000 385954.23 427172.91

1602.17 70000000 388254.49 426823.59

1831.05 80000000 374491.43 428103.96

2059.94 90000000 379622.74 428152.02

2288.82 100000000 80868.34 430185.03

We also tried SNC-4 instead of All2All, but could not get more than 40 GB/s from MCDRAM even with hbw_malloc API.

Raw data:

# flat memory, running from memory (default), alltoall

# MB N GB/s

2.29 100000 272062.96

4.58 200000 491040.47

6.87 300000 603979.78

9.16 400000 745654.04

11.44 500000 811800.77

13.73 600000 901462.35

16.02 700000 891953.26

18.31 800000 1019375.15

20.60 900000 984749.63

22.89 1000000 923516.48

25.18 1100000 758422.09

27.47 1200000 779328.74

29.75 1300000 513185.43

32.04 1400000 155550.35

34.33 1500000 103209.12

36.62 1600000 97435.74

38.91 1700000 83968.40

41.20 1800000 83692.35

43.49 1900000 80734.60

45.78 2000000 76317.89

68.66 3000000 81180.08

91.55 4000000 78889.73

114.44 5000000 79891.50

137.33 6000000 79954.96

160.22 7000000 79477.00

183.11 8000000 79145.59

205.99 9000000 79821.12

228.88 10000000 79519.15

228.88 10000000 79212.54

457.76 20000000 79692.27

686.65 30000000 79866.15

915.53 40000000 79828.15

1144.41 50000000 80214.91

1373.29 60000000 80226.84

1602.17 70000000 80178.77

1831.05 80000000 80177.06

2059.94 90000000 80109.79

2288.82 100000000 80175.94

# flat memory, running from MCDRAM, alltoall, hbwmalloc API

# MB N GB/s

228.88 10000000 421009.18

457.76 20000000 416997.91

686.65 30000000 427808.31

915.53 40000000 423310.75

1144.41 50000000 431032.35

1373.29 60000000 427172.91

1602.17 70000000 426823.59

1831.05 80000000 428103.96

2059.94 90000000 428152.02

2288.82 100000000 430185.03

2343.75 102400000 425999.98

4687.50 204800000 427821.10

9155.27 400000000 429318.13

9375.00 409600000 428376.70

11444.09 500000000 428052.08

13732.91 600000000 266061.01

16021.73 700000000 200257.79

18310.55 800000000 168164.19

20599.37 900000000 150192.91

22888.18 1000000000 137862.07

25177.00 1100000000 129418.12

27465.82 1200000000 123019.53

29754.64 1300000000 118268.79

32043.46 1400000000 114343.33

34332.28 1500000000 110991.25

36621.09 1600000000 108538.36

38909.91 1700000000 106249.72

41198.73 1800000000 104455.07

43487.55 1900000000 102828.93

45776.37 2000000000 101348.56

48065.19 2100000000 100013.73

We appreciate any help on that.

Best,

Vitali

On 04/11/2016 08:42 PM, Kumaran, Kalyan wrote:

Avinash, thanks for the prompt reply. Kevin/Vitali can provide

all the details. The short summary is we are unable to reproduce the STREAM results from IPM or DDR. It would be great if Karthik or Sundaram can advise us on how to run (which mode etc) or if they could log into the machine and make sure we have set up things correctly.

Thanks,

Kumar

On Apr 11, 2016, at 8:07 PM, Sodani, Avinash <avinash.sodani@intel.com> wrote:

Copying Sundaram/Karthik from our side.

Kumar,

Can you send us more information on this?

-Avinash

From: Kumaran,

Kalyan [mailto:kumaran@alcf.anl.gov]

Sent: Monday, April 11, 2016 5:32 PM

To: Fromkin, Russ; Sodani, Avinash

Cc: Harms, Kevin; Morozov, Vitali A.

Subject: Fwd: [JLSE-Intel-NDA] KNL Status

Russ, Avinash,

we need help in understanding how to obtain STREAM results. Can anyone help?

Thanks,

Kumar

Begin forwarded message:

From: "Harms, Kevin" <harms@alcf.anl.gov>

Subject: [JLSE-Intel-NDA] KNL Status

Date: April 11, 2016 at 5:42:15 PM CDT

To: "intel-nda@jlse.anl.gov"

<intel-nda@jlse.anl.gov>

The knl00 node is currently in Flat mode and All2All model. With this setup we can get 420 GB/s from IPM memory on STREAM Triad when using hbw_malloc and friends which was confirmed by both Vitali and I.

Changing the BIOS settings and warm rebooting seem to be problematic. Vitali and I were running earlier and could not get over 80 GB/s with any method even though memory capacity indicated flat mode.

Using numactl --membind is also problematic. The STREAM results

when using this to launch the application were erratic, so it is also suggested not to use that. That means you will need to modify your code to use dynamic memory and hbw_malloc to exploit the IPM.

Will followup with more detailed results later.

kevin

_______________________________________________

Intel-nda mailing list

Intel-nda@lists.jlse.anl.gov

https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

<flat_a2a_ddr.txt><flat_a2a_ipm.txt><flat_quad_ddr.txt><flat_quad_ipm.txt><flat_snc4_ddr.txt><flat_snc4_ipm.txt>