March 2016 - Intel-nda - mailman3.cels.anl.gov

[JLSE-Intel-NDA] B0 Stream triad
by Vitali A. Morozov 30 Mar '16

30 Mar '16

A quick update on STREAM triad on KNL B0 after BIOS upgrade. The first figure is for 1 core. About 35 GB/s from L2 cache, drops down to 11.5 GB/s up to 64 MB, but then increases to a little over 13 GB/s. All exactly as expected. The second figure is for 8 tiles (a quadrant). Near linear increase in bandwidth up to 284 GB/s when data fit into all L2 caches - 8 MB. This is near 80% of peak L2 bandwidth, which is expected at 357.6 GB/s then drops to about 78-80 GB/s, a little less then one forth of expected peak of 120 GB/s (67% of peak). Two points were obtained for the footprint outside MCDRAM size - both show 79 GB/s. The last figure is for entire node. The accumulative L2 cache bandwidth is a little over 1 TB/s (1.4 TB/s is a peak) with 32 MB capacity. After this footprint, I observe the drop down of the bandwidth to 80 GB/s. This is strange as MCDRAM bandwidth is rated 480 GB/s. Issues: 1. Full node MCDRAM bandwidth does not reach expected 420-430 GB/s bandwidth. 2. 75 GB memory footprint case failed all the the runs. Vitali On 03/25/2016 03:29 PM, Vitali A. Morozov wrote: > Hello, > > here are the figures that I have obtained for STREAM triad. The peak > numbers are: > > L2 peak bandwidth 44.7 GB/s, based on the size of the request table, > L2 hit latency, and core capabilities to issue memory requests clocked > at 1.3 GHz. > MCDRAM peak bandwidth 480 GB/s scaled down to 1 port of L2 tile => 15 > GB/s. Could be slightly less if the port was provisioned to 36 tiles > (13.3 GB/s in this case). > DDR4 peak bandwidth 112 GB/s is based on 6 channel DDR4-2400. Being > scaled down to 2134 GT/s, it will be 100 GB/s. > > The first graph provides STREAM triad bandwidth for 1 core and 1 tile > (2 cores, 1 thread is running on each core). > > I do not look at L2 cache bandwidth, as 44.7 GB/s number is very hard > to reach and there might be many factors that can reduce effective > bandwidth (moreover, I was not carefully coding stream to get the full > L2 bandwidth anyway). The measured 32 GB/s from a core and 37 GB/s for > 2 cores on a tile are within 80% of expected tolerance. Running more > then 2 threads on a tile severely degrades performance (20 GB/s at > best), which is expected as the L2 request table gets quickly full. > > When data fits into MCDRAM, I have measured 10-12 GB/s for a core and > 10-13 GB/s for a tile with strange repeatable artifacts at around 64 > MB and 640 MB. The tile performance is more uniform, but starts > quickly degrade at around 900 MB, which is strange as we should have > plenty of room in MCDRAM. Also, I am measuring triad, so the cache > must be warmed out by the time I started it as triad is one of the > last benchmarks in STREAM suite. In fact I do not expect any > performance drop between about 8 MB up to 192 GB - the memory > bandwidth should be capable to provide single port MCDRAM-available > bandwidth at ease. > > The second graph provides STREAM triad numbers for 8 tiles. Again L2 > cache bandwidth is a kind of OK except that it quickly gets down not > reaching the expected max of 8 MB combined L2 cache. The MCDRAM > bandwidth is within the expected range (with a very strange dip in > between 12 - 40 MB. Then, the bandwidth decreases with a very unusual > change of curvature at around 2 GB. Even if MCDRAM does not work as > cache, I would expect to see around 90 GB/s - I am measuring 4 GB/s at > 38 GB. > > Sounds like Ben have installed CR instead of DDR :) > > The second graph numbers can slightly be improved with careful mapping > the processes to the cores, but not very much. > > Vitali > > > > > > > > > > > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Kernel Panic on KNL Box
by Vitali A. Morozov 30 Mar '16

30 Mar '16

What happened is I was submitting a script with OMP_NUM_THREAD variable set to 64444 as opposed to 64. Expected crash for an immature software stack. Vitali On 03/30/2016 01:58 PM, Allen, Benjamin S. wrote: > Your panic dumped to /var/crash/10.17.6.104-2016-03-30-18:36:37 if interested. Appears to be a "divide error: 0000" in intel_pstate. This panic was immediately preceded by: > > [78206.294474] hrtimer: interrupt took 11325201 ns > [78208.724730] [sched_delayed] sched: RT throttling activated > > This appears to match this RHEL Solution article: https://access.redhat.com/solutions/1471663. However we're already running a newer kernel than is listed as the resolution. > > Ben > >> On Mar 30, 2016, at 1:38 PM, Vitali A. Morozov <morozov(a)anl.gov> wrote: >> >> Finally, I could make it. The KNL box has died... >> >> Ben, can you reboot it, please? >> >> Thanks, >> >> Vitali

1 0

Re: [JLSE-Intel-NDA] Kernel Panic on KNL Box
by Allen, Benjamin S. 30 Mar '16

30 Mar '16

Your panic dumped to /var/crash/10.17.6.104-2016-03-30-18:36:37 if interested. Appears to be a "divide error: 0000" in intel_pstate. This panic was immediately preceded by: [78206.294474] hrtimer: interrupt took 11325201 ns [78208.724730] [sched_delayed] sched: RT throttling activated This appears to match this RHEL Solution article: https://access.redhat.com/solutions/1471663. However we're already running a newer kernel than is listed as the resolution. Ben > On Mar 30, 2016, at 1:38 PM, Vitali A. Morozov <morozov(a)anl.gov> wrote: > > Finally, I could make it. The KNL box has died... > > Ben, can you reboot it, please? > > Thanks, > > Vitali

1 0

[JLSE-Intel-NDA] KNL Box is free
by Vitali A. Morozov 30 Mar '16

30 Mar '16

I have released the KNL box. Feel free to use. Vitali

1 0

Re: [JLSE-Intel-NDA] Kernel Panic on KNL Box
by Vitali A. Morozov 30 Mar '16

30 Mar '16

Finally, I could make it. The KNL box has died... Ben, can you reboot it, please? Thanks, Vitali

1 0

Re: [JLSE-Intel-NDA] Fwd: B0 KNL System
by Vitali A. Morozov 30 Mar '16

30 Mar '16

With this kernel, real run for 320^3 has finished for 2278 seconds on BG/Q node (2x2x2 + 8 threads each) and for 440 seconds on KNL node (same 2x2x2 + 8 threads). It translates to 5.18x core-to-core performance improvement. Vitali On 03/30/2016 11:17 AM, Kumaran, Kalyan wrote: > > >> Begin forwarded message: >> >> *From: *"Vitali A. Morozov" <morozov(a)anl.gov <mailto:[email protected]>> >> *Subject: **Re: B0 KNL System* >> *Date: *March 30, 2016 at 11:14:19 AM CDT >> *To: *"Fromkin, Russ" <russ.fromkin(a)intel.com >> <mailto:[email protected]>>, Scott Parker <sparker(a)anl.gov >> <mailto:[email protected]>> >> *Cc: *"Allen, Benjamin S." <bsallen(a)alcf.anl.gov >> <mailto:[email protected]>>, "Kumaran, Kalyan" >> <kumaran(a)alcf.anl.gov <mailto:[email protected]>>, "Kirkendall, >> Keith G" <keith.g.kirkendall(a)intel.com >> <mailto:[email protected]>>, "Allison, CM" >> <cm.allison(a)intel.com <mailto:[email protected]>>, "Arney, >> Rebecca" <rebecca.arney(a)intel.com <mailto:[email protected]>>, >> Kalyanasundaram Kumaran <kumaran(a)alcf.anl.gov >> <mailto:[email protected]>> >> >> Hi Russ, >> >> I am intensively benchmarking it for the last couple of days - it is >> too early to share much of the data that I am getting, I am afraid to >> make a mistake. >> >> However, of what I can share so far. I am very happy to see what is >> coming out after the BIOS update. The STREAM benchmark was fixed and >> I am measuring what I am expecting (not finished yet). HACCmk is >> running at 1.6 IPC (60% of peak)! >> >> I am very much surprised to see such a terrific match of the B0 >> hardware versus Sniper model. For HACCmk, Sniper was projecting >> >> n = 1488: 486725 cycles, IPC 1.5933, 49.9847% of peak >> >> For the same problem size and slightly modified coding, I have measured >> >> n = 1488: 501949 cycles, IPC = 1.5601, 57.5565% of peak. >> >> Amazing! >> >> Vitali >> >> >> >> On 03/30/2016 11:00 AM, Fromkin, Russ wrote: >>> Vitali, Scott, >>> >>> Just wanted to check in to see how things are coming along with the >>> updated system. Ben has provided some initial data that you have >>> seen much better behaviors/performance, at least with the memory. >>> >>> Please let us know how things are coming along and if you run into >>> any issues or have any questions. It would be good to hear how some >>> of your other tests are comparing vs. the December work in OR and >>> the more recent A0 remote access results. >>> >>> Thanks, >>> Russ >>> >>> >> > > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] Fwd: B0 KNL System
by Kumaran, Kalyan 30 Mar '16

30 Mar '16

Begin forwarded message: From: "Vitali A. Morozov" <morozov(a)anl.gov<mailto:[email protected]>> Subject: Re: B0 KNL System Date: March 30, 2016 at 11:14:19 AM CDT To: "Fromkin, Russ" <russ.fromkin(a)intel.com<mailto:[email protected]>>, Scott Parker <sparker(a)anl.gov<mailto:[email protected]>> Cc: "Allen, Benjamin S." <bsallen(a)alcf.anl.gov<mailto:[email protected]>>, "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov<mailto:[email protected]>>, "Kirkendall, Keith G" <keith.g.kirkendall(a)intel.com<mailto:[email protected]>>, "Allison, CM" <cm.allison(a)intel.com<mailto:[email protected]>>, "Arney, Rebecca" <rebecca.arney(a)intel.com<mailto:[email protected]>>, Kalyanasundaram Kumaran <kumaran(a)alcf.anl.gov<mailto:[email protected]>> Hi Russ, I am intensively benchmarking it for the last couple of days - it is too early to share much of the data that I am getting, I am afraid to make a mistake. However, of what I can share so far. I am very happy to see what is coming out after the BIOS update. The STREAM benchmark was fixed and I am measuring what I am expecting (not finished yet). HACCmk is running at 1.6 IPC (60% of peak)! I am very much surprised to see such a terrific match of the B0 hardware versus Sniper model. For HACCmk, Sniper was projecting n = 1488: 486725 cycles, IPC 1.5933, 49.9847% of peak For the same problem size and slightly modified coding, I have measured n = 1488: 501949 cycles, IPC = 1.5601, 57.5565% of peak. Amazing! Vitali On 03/30/2016 11:00 AM, Fromkin, Russ wrote: Vitali, Scott, Just wanted to check in to see how things are coming along with the updated system. Ben has provided some initial data that you have seen much better behaviors/performance, at least with the memory. Please let us know how things are coming along and if you run into any issues or have any questions. It would be good to hear how some of your other tests are comparing vs. the December work in OR and the more recent A0 remote access results. Thanks, Russ

1 0

Re: [JLSE-Intel-NDA] Kernel Panic on KNL Box
by Vitali A. Morozov 29 Mar '16

29 Mar '16

The memory at 2134 MT/s is the BO bin 3 limitation (I thought it must be 2133, strange 4). Also, the product will be at 1.4 GHz base frequency. I have a good "preliminary" news. I ran a single tile STREAM triad for two data points, right in MCDRAM footprint and well beyond that. In both cases I am getting exactly what was projected: a little over 13 GB/s. I will need to make more intensive benchmarking but I am happy so far. Memory before_update after_update 1373.3 MB 8598.3969 GB/s 13594.6362GB/s 22888.2 MB1849.2670 GB/s 13169.3377 GB/s I will try something else to see if this converts into performance. Vitali On 03/29/2016 04:46 PM, Allen, Benjamin S. wrote: > I did change two BIOS settings: the acoustic profile in the BIOS to favor performance over noise. As well as changed a the Performance Bias of the CPU to be "Performance" instead of Balanced. This is supposed to be a 1.3Ghz CPU. In addition, the new BIOS includes a number of microcode updates for the B0. I suppose it could be any of the above. > > Note the memory still came up at 2134MT/s. > > Ben > >> On Mar 29, 2016, at 4:39 PM, Vitali A. Morozov <morozov(a)anl.gov> wrote: >> >> Hi Ben, >> >> thanks a lot - will start a few tests right away. /proc/cpuinfo gives higher frequency now: >> >> cpu MHz : 1299.644 >> >> it was lower before: >> >> cpu MHz : 1200.519 >> >> I am curious if this is reporting fix or sensor fix or indeed changes in frequency. >> >> Vitali >> >> >> On 03/29/2016 04:17 PM, Allen, Benjamin S. wrote: >>> Vitali, >>> >>> Upgraded the BIOS with A0 and swapped the B0 back in. Appears to be a lot more happy now. I can't reproduce the panic as I had before. Various release notes and updated manuals attached. Feel free to start testing. >>> >>> Thanks, >>> >>> Ben

1 0

Re: [JLSE-Intel-NDA] Kernel Panic on KNL Box
by Allen, Benjamin S. 29 Mar '16

29 Mar '16

I did change two BIOS settings: the acoustic profile in the BIOS to favor performance over noise. As well as changed a the Performance Bias of the CPU to be "Performance" instead of Balanced. This is supposed to be a 1.3Ghz CPU. In addition, the new BIOS includes a number of microcode updates for the B0. I suppose it could be any of the above. Note the memory still came up at 2134MT/s. Ben > On Mar 29, 2016, at 4:39 PM, Vitali A. Morozov <morozov(a)anl.gov> wrote: > > Hi Ben, > > thanks a lot - will start a few tests right away. /proc/cpuinfo gives higher frequency now: > > cpu MHz : 1299.644 > > it was lower before: > > cpu MHz : 1200.519 > > I am curious if this is reporting fix or sensor fix or indeed changes in frequency. > > Vitali > > > On 03/29/2016 04:17 PM, Allen, Benjamin S. wrote: >> Vitali, >> >> Upgraded the BIOS with A0 and swapped the B0 back in. Appears to be a lot more happy now. I can't reproduce the panic as I had before. Various release notes and updated manuals attached. Feel free to start testing. >> >> Thanks, >> >> Ben >

1 0

Re: [JLSE-Intel-NDA] Kernel Panic on KNL Box
by Vitali A. Morozov 29 Mar '16

29 Mar '16

Hi Ben, thanks a lot - will start a few tests right away. /proc/cpuinfo gives higher frequency now: cpu MHz : 1299.644 it was lower before: cpu MHz : 1200.519 I am curious if this is reporting fix or sensor fix or indeed changes in frequency. Vitali On 03/29/2016 04:17 PM, Allen, Benjamin S. wrote: > Vitali, > > Upgraded the BIOS with A0 and swapped the B0 back in. Appears to be a lot more happy now. I can't reproduce the panic as I had before. Various release notes and updated manuals attached. Feel free to start testing. > > Thanks, > > Ben

1 0