On our KNL B0 I'm seeing some irregularities in the MCDRAM
performance when running Stream in SNC-4 mode.
I'm running Stream with 16 threads all bound to either domain 0, 1,
2, or 3 and memory bound to just one of the
four MCDRAM domains (4,5,6,7). Below is a table of the results
showing memory bandwidth in MB/sec, where
the thread domain used is specified on the left the MCDRAM domain on
the top of the table.
Domain 4 never achieves more than 73 GB/s while the other domains
(5,6,7) all reach 112 GB/s.
MCDRAM NUMA Domain
4
5
6
7
0
73,306
84,053
107,483
55,978
Thread Domain
1
64,837
112,380
55,521
77,765
2
63,575
67,508
111,569
77,957
3
65,506
102,203
76,131
111,617
numactl -H shows that nodes 0,1,2,3 are associated with the DDR and
16 cores each
while nodes 4,5,6,7 are MCDRAM nodes with no associated processors.
However the
following nodes are closest:
Node 4 -> Node 0
Node 5 -> Node 1
Node 6 -> Node 2
Node 7 -> Node 3