A little more results for STREAM. Flat more, alltoall. The results from memory were obtained by using default STREAM with static allocation. The results from MCDRAM were obtained by using modified STREAM with hbwmalloc.h interface. 1 MPI process, 64 threads, scatter affinity, no numactl is used. Most results are as expected: 80 GB/s from DDR memory, 420 GB/s from MCDRAM, the MCDRAM size is about 16 GB, the aggregate L2 cache size is 32 MB. Noticed inconsistencies for hbwmalloc benchmarks: 1. The result is not validated - will spend more time on that tomorrow. 2. hbw_malloc continues allocating data beyond the expected 16 GB up to 48 GB. This means it fails back to DDR if it cannot allocate in MCDRAM - not expected default behavior. It must fail if it does not have enough space. Vitali Raw data: # flat memory, running from memory (default), alltoall # MB N GB/s 2.29 100000 272062.96 4.58 200000 491040.47 6.87 300000 603979.78 9.16 400000 745654.04 11.44 500000 811800.77 13.73 600000 901462.35 16.02 700000 891953.26 18.31 800000 1019375.15 20.60 900000 984749.63 22.89 1000000 923516.48 25.18 1100000 758422.09 27.47 1200000 779328.74 29.75 1300000 513185.43 32.04 1400000 155550.35 34.33 1500000 103209.12 36.62 1600000 97435.74 38.91 1700000 83968.40 41.20 1800000 83692.35 43.49 1900000 80734.60 45.78 2000000 76317.89 68.66 3000000 81180.08 91.55 4000000 78889.73 114.44 5000000 79891.50 137.33 6000000 79954.96 160.22 7000000 79477.00 183.11 8000000 79145.59 205.99 9000000 79821.12 228.88 10000000 79519.15 228.88 10000000 79212.54 457.76 20000000 79692.27 686.65 30000000 79866.15 915.53 40000000 79828.15 1144.41 50000000 80214.91 1373.29 60000000 80226.84 1602.17 70000000 80178.77 1831.05 80000000 80177.06 2059.94 90000000 80109.79 2288.82 100000000 80175.94 # flat memory, running from MCDRAM, alltoall, hbwmalloc API # MB N GB/s 228.88 10000000 421009.18 457.76 20000000 416997.91 686.65 30000000 427808.31 915.53 40000000 423310.75 1144.41 50000000 431032.35 1373.29 60000000 427172.91 1602.17 70000000 426823.59 1831.05 80000000 428103.96 2059.94 90000000 428152.02 2288.82 100000000 430185.03 2343.75 102400000 425999.98 4687.50 204800000 427821.10 9155.27 400000000 429318.13 9375.00 409600000 428376.70 11444.09 500000000 428052.08 13732.91 600000000 266061.01 16021.73 700000000 200257.79 18310.55 800000000 168164.19 20599.37 900000000 150192.91 22888.18 1000000000 137862.07 25177.00 1100000000 129418.12 27465.82 1200000000 123019.53 29754.64 1300000000 118268.79 32043.46 1400000000 114343.33 34332.28 1500000000 110991.25 36621.09 1600000000 108538.36 38909.91 1700000000 106249.72 41198.73 1800000000 104455.07 43487.55 1900000000 102828.93 45776.37 2000000000 101348.56 48065.19 2100000000 100013.73 On 04/11/2016 05:42 PM, Harms, Kevin wrote:
The knl00 node is currently in Flat mode and All2All model. With this setup we can get 420 GB/s from IPM memory on STREAM Triad when using hbw_malloc and friends which was confirmed by both Vitali and I.
Changing the BIOS settings and warm rebooting seem to be problematic. Vitali and I were running earlier and could not get over 80 GB/s with any method even though memory capacity indicated flat mode.
Using numactl --membind is also problematic. The STREAM results when using this to launch the application were erratic, so it is also suggested not to use that. That means you will need to modify your code to use dynamic memory and hbw_malloc to exploit the IPM.
Will followup with more detailed results later.
kevin
_______________________________________________ Intel-nda mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/intel-nda