Hello,

With current setup of All2All/Flat, we seem to have a stable 410 GB/s from MCDRAM and 78 GB/s from DDR.

For STREAMing from MCDRAM, Intel people suggested to try SNC-4/Flat promising 430 GB/s. Our previous trials with other combinations of NOC settings lead to unstable results.

Do we want to try it now? It will probably require hard reboot.

Vitali





On 04/13/2016 01:06 PM, Raman, Karthik wrote:

Ok I read this as 418 GB/s J

And ok, just realized you are running in All2All mode, so yes this looks good. For Quad/Flat you should see ~440 GB/s with the same run.

From: Vitali A. Morozov [mailto:morozov@anl.gov]
Sent: Wednesday, April 13, 2016 11:02 AM
To: Raman, Karthik <karthik.raman@intel.com>; Chinthamani, Sundaram <sundaram.chinthamani@intel.com>; Kumaran, Kalyan <kumaran@alcf.anl.gov>
Cc: Sodani, Avinash <avinash.sodani@intel.com>; Fromkin, Russ <russ.fromkin@intel.com>; Harms, Kevin <harms@alcf.anl.gov>; Morozov, Vitali A. <morozov@alcf.anl.gov>; Allen, Benjamin S. <bsallen@alcf.anl.gov>
Subject: Re: [JLSE-Intel-NDA] KNL Status

Hi Karthik,

Repeating the same options you have:

mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp -mcmodel medium -shared-intel -opt-streaming-stores always stream.c -o stream

-------------------------------------------------------------
STREAM version $Revision: 5.9 $
-------------------------------------------------------------
This system uses 8 bytes per DOUBLE PRECISION word.
-------------------------------------------------------------
Array size = 134217728, Offset = 0
Total memory required = 3072.0 MB.
Each test is run 100 times, but only
the *best* time for each is used.
-------------------------------------------------------------
Number of Threads requested = 64
-------------------------------------------------------------

< skiped >

Function����� Rate (MB/s)�� Avg time���� Min time���� Max time
Copy:����� 362331.5200������ 0.0060������ 0.0059������ 0.0061
Scale:���� 359281.9806������ 0.0060������ 0.0060������ 0.0062
Add:������ 418122.7024������ 0.0078������ 0.0077������ 0.0081
Triad:���� 418122.7024������ 0.0078������ 0.0077������ 0.0081
Store:���� 224685.6729������ 0.0048������ 0.0048������ 0.0049



408GB/s

Vitali