- Intel-nda - mailman3.cels.anl.gov

Re: [JLSE-Intel-NDA] Fwd: syscfg for Adams Pass
by Allen, Benjamin S. 11 Apr '16

11 Apr '16

Added to allowed command list: /usr/bin/systemctl reboot Since one needs to reboot to actually make said changes effective. Ben > On Apr 11, 2016, at 2:00 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > The following is now possible: > > # Display current settings > sudo /usr/bin/syscfg/syscfg /d biossettings "Memory Mode for 2LM" > sudo /usr/bin/syscfg/syscfg /d biossettings "Memory Model" > > # Changing settings > sudo /usr/bin/syscfg/syscfg /bcs "" "Memory Mode for 2LM" 1 > sudo /usr/bin/syscfg/syscfg /bcs "" "Memory Model" 4 > > I currently allow sparker,harms,kumaran,morozov to run these without a password. Please let me know if your testing includes testing against the variables memory modes, and I'll add your user. > > Please note, that displaying settings is showing the current BIOS value. If the node hasn't been rebooted since the setting was changed, this will not reflect the system's current state. > > Ben > >> On Apr 11, 2016, at 12:48 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >> >> FYI, working on making a limited set of syscfg options available via sudo to allow users to swap around the MCDRAM Memory Mode and Memory Model, eg: >> >> syscfg /bcs "" "Memory Mode for 2LM" 1 >> syscfg /bcs "" "Memory Model" 4 >> >> syscfg /d biossettings "Memory Mode for 2LM" >> syscfg /d biossettings "Memory Model" >> >> [root@knl00 ~]# syscfg /d biossettings "Memory Mode for 2LM" >> >> Memory Mode for 2LM >> =================== >> Current Value : Flat >> -------------------- >> Possible Values >> --------------- >> Cache : 00 >> Flat : 01 >> Hybrid : 02 >> Auto : 03 >> [root@knl00 ~]# syscfg /d biossettings "Memory Model" >> >> Memory Model >> ============ >> Current Value : Quadrant >> ------------------------ >> Possible Values >> --------------- >> All2All : 00 >> SNC-2 : 01 >> SNC-4 : 02 >> Hemisphere : 03 >> Quadrant : 04 >> Auto : 05 >> >> So far, booting in Flat + Quadrant mostly hangs or at least takes a very very long time for the kernel to boot. Flat + All2All boots fine. >> >> Ben >> >>> Some input based on Ben’s earlier questions: >>> >>> Flat mode (separate DDR and MCDRAM): >>> syscfg /bcs "" "Memory Mode" 1 >>> >>> Cache mode: >>> syscfg /bcs "" "Memory Mode" 0 >>> >>> The other setting is "Cluster Mode" where you choose between All2all, Quadrant and SNC4. Quadrant seems to be good starting point. All2all shouldn't be used as it is slower. >>> >>> You can see available values as usual: >>> syscfg /d biossettings "Memory Mode" >>> syscfg /d biossettings "Cluster Mode" >>> >>> Make sure it comes up as expected. numactl -H is the best way to tell. >>> In flat mode you would see 2 NUMA nodes, #0 with DDR size and #1 with MCDRAM size. >>> In cache mode you see single NUMA node with DDR size. >>> For sub-numa clustering, SNC4+cache you see 4 NUMAs, and SNC4+flat you see 8 NUMAs. >>> >>> Let me know if you have any questions. >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Fwd: syscfg for Adams Pass
by Allen, Benjamin S. 11 Apr '16

11 Apr '16

The following is now possible: # Display current settings sudo /usr/bin/syscfg/syscfg /d biossettings "Memory Mode for 2LM" sudo /usr/bin/syscfg/syscfg /d biossettings "Memory Model" # Changing settings sudo /usr/bin/syscfg/syscfg /bcs "" "Memory Mode for 2LM" 1 sudo /usr/bin/syscfg/syscfg /bcs "" "Memory Model" 4 I currently allow sparker,harms,kumaran,morozov to run these without a password. Please let me know if your testing includes testing against the variables memory modes, and I'll add your user. Please note, that displaying settings is showing the current BIOS value. If the node hasn't been rebooted since the setting was changed, this will not reflect the system's current state. Ben > On Apr 11, 2016, at 12:48 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > FYI, working on making a limited set of syscfg options available via sudo to allow users to swap around the MCDRAM Memory Mode and Memory Model, eg: > > syscfg /bcs "" "Memory Mode for 2LM" 1 > syscfg /bcs "" "Memory Model" 4 > > syscfg /d biossettings "Memory Mode for 2LM" > syscfg /d biossettings "Memory Model" > > [root@knl00 ~]# syscfg /d biossettings "Memory Mode for 2LM" > > Memory Mode for 2LM > =================== > Current Value : Flat > -------------------- > Possible Values > --------------- > Cache : 00 > Flat : 01 > Hybrid : 02 > Auto : 03 > [root@knl00 ~]# syscfg /d biossettings "Memory Model" > > Memory Model > ============ > Current Value : Quadrant > ------------------------ > Possible Values > --------------- > All2All : 00 > SNC-2 : 01 > SNC-4 : 02 > Hemisphere : 03 > Quadrant : 04 > Auto : 05 > > So far, booting in Flat + Quadrant mostly hangs or at least takes a very very long time for the kernel to boot. Flat + All2All boots fine. > > Ben > >> Some input based on Ben’s earlier questions: >> >> Flat mode (separate DDR and MCDRAM): >> syscfg /bcs "" "Memory Mode" 1 >> >> Cache mode: >> syscfg /bcs "" "Memory Mode" 0 >> >> The other setting is "Cluster Mode" where you choose between All2all, Quadrant and SNC4. Quadrant seems to be good starting point. All2all shouldn't be used as it is slower. >> >> You can see available values as usual: >> syscfg /d biossettings "Memory Mode" >> syscfg /d biossettings "Cluster Mode" >> >> Make sure it comes up as expected. numactl -H is the best way to tell. >> In flat mode you would see 2 NUMA nodes, #0 with DDR size and #1 with MCDRAM size. >> In cache mode you see single NUMA node with DDR size. >> For sub-numa clustering, SNC4+cache you see 4 NUMAs, and SNC4+flat you see 8 NUMAs. >> >> Let me know if you have any questions. > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Intel MPSP
by Allen, Benjamin S. 11 Apr '16

11 Apr '16

It includes the kernel SRPM if you want to go spelunking for patches. Ben > On Apr 11, 2016, at 12:45 PM, Harms, Kevin <harms(a)alcf.anl.gov> wrote: > > > Getting the updated jemalloc and memkind libraries would be worth it if we are going to test the flat mode. It might be interesting to get and run the STREAM test in the micperf package to demonstrate the memory bandwidth seems to be incorrect. > > The document doesn't really go into what the kernel changes were other than vague statements of fixes or support of additional features. > > kevin > > > > >> Intel has a patched RHEL7.2 kernel and a number of other tools that appear to better support KNL. Attached the user guide. Let me know if I should install this on knl00. >> >> Ben

1 0

[JLSE-Intel-NDA] Fwd: syscfg for Adams Pass
by Allen, Benjamin S. 11 Apr '16

11 Apr '16

FYI, working on making a limited set of syscfg options available via sudo to allow users to swap around the MCDRAM Memory Mode and Memory Model, eg: syscfg /bcs "" "Memory Mode for 2LM" 1 syscfg /bcs "" "Memory Model" 4 syscfg /d biossettings "Memory Mode for 2LM" syscfg /d biossettings "Memory Model" [root@knl00 ~]# syscfg /d biossettings "Memory Mode for 2LM" Memory Mode for 2LM =================== Current Value : Flat -------------------- Possible Values --------------- Cache : 00 Flat : 01 Hybrid : 02 Auto : 03 [root@knl00 ~]# syscfg /d biossettings "Memory Model" Memory Model ============ Current Value : Quadrant ------------------------ Possible Values --------------- All2All : 00 SNC-2 : 01 SNC-4 : 02 Hemisphere : 03 Quadrant : 04 Auto : 05 So far, booting in Flat + Quadrant mostly hangs or at least takes a very very long time for the kernel to boot. Flat + All2All boots fine. Ben Some input based on Ben’s earlier questions: Flat mode (separate DDR and MCDRAM): syscfg /bcs "" "Memory Mode" 1 Cache mode: syscfg /bcs "" "Memory Mode" 0 The other setting is "Cluster Mode" where you choose between All2all, Quadrant and SNC4. Quadrant seems to be good starting point. All2all shouldn't be used as it is slower. You can see available values as usual: syscfg /d biossettings "Memory Mode" syscfg /d biossettings "Cluster Mode" Make sure it comes up as expected. numactl -H is the best way to tell. In flat mode you would see 2 NUMA nodes, #0 with DDR size and #1 with MCDRAM size. In cache mode you see single NUMA node with DDR size. For sub-numa clustering, SNC4+cache you see 4 NUMAs, and SNC4+flat you see 8 NUMAs. Let me know if you have any questions.

1 0

Re: [JLSE-Intel-NDA] Intel MPSP
by Harms, Kevin 11 Apr '16

11 Apr '16

Getting the updated jemalloc and memkind libraries would be worth it if we are going to test the flat mode. It might be interesting to get and run the STREAM test in the micperf package to demonstrate the memory bandwidth seems to be incorrect. The document doesn't really go into what the kernel changes were other than vague statements of fixes or support of additional features. kevin >Intel has a patched RHEL7.2 kernel and a number of other tools that appear to better support KNL. Attached the user guide. Let me know if I should install this on knl00. > >Ben >

1 0

[JLSE-Intel-NDA] Intel MPSP
by Allen, Benjamin S. 11 Apr '16

11 Apr '16

Intel has a patched RHEL7.2 kernel and a number of other tools that appear to better support KNL. Attached the user guide. Let me know if I should install this on knl00. Ben

1 0

[JLSE-Intel-NDA] Call w Avinash Sodhani
by Kumaran, Kalyan 08 Apr '16

08 Apr '16

He is the Chief Architect of KNL. We have a 1 hour call scheduled for April 26th at 2:00pm central. I will follow up with a room # closer to the date. We can share our results with him and also ask any questions. -Kumar

1 0

[JLSE-Intel-NDA] KNK usage
by Kumaran, Kalyan 07 Apr '16

07 Apr '16

How is the usage currently on the KNL node? Is it OK to open up to more users? Thanks, Kumar

1 0

[JLSE-Intel-NDA] KNL MPI_Test performance
by Harms, Kevin 06 Apr '16

06 Apr '16

I was seeing odd results from the kmi_hash benchmark which heavily uses the MPI_Test interface. KNL was much slower than a BGQ node. Looking at a standalone test case of MPI_Test, KNL with Intel MPI is much slower to execute MPI_Test than bgq. results for 64 ranks: bgq: loops: 5000000 time: 9.306008 test/s: 537287.314173 knl: loops: 5000000 time: 78.315594 test/s: 63844.245215 I'm guessing most codes don't use MPI_Test but we should probably figure out what's going on. kevin

1 0

Re: [JLSE-Intel-NDA] B0 Stream triad
by Allen, Benjamin S. 04 Apr '16

04 Apr '16

Hi Scott, We're checking with Intel if there's a way to change this option programatically. As far as detecting current mode: Cache mode total memory shows up as 192GB total memory. Flat mode total memory shows up as 192GB + 16GB, and numactl show should an additional memory node. Hybrid mode total memory would show up as 192GB + 8GB (I think), and numactl show should an additional memory node. It would be nice to have something to query to give this mode however. Ben > On Apr 4, 2016, at 1:41 PM, Scott Parker <sparker(a)anl.gov> wrote: > > > We are going to need a way of changing from flat, to hybrid, to cache modes that can be done by the user > (and a way of checking the mode if we don't already have it). Is this something that could be baked into a > script? I assume a reboot is required, but we really do need to be able to test out the different modes against > various application. > > -Scott > > On 4/4/16 12:05 PM, Allen, Benjamin S. wrote: >> Hi Scott, >> >> Swapping into flat mode is a BIOS option. I can change that if you'd like, and no one else is using the machine. >> >> Ben >> >>> On Apr 1, 2016, at 6:27 PM, Scott Parker <sparker(a)anl.gov> wrote: >>> >>> >>> Are we able to boot the KNL into flat mode? I'm seeing the same Stream bandwidth as Vitali and >>> the resulting Nekbone performance, while still 4x better than BG/Q, is 2x slower than what I was >>> seeing on A0 hardware. Best nekbone performance requires full IPM bandwidth. >>> >>> -Scott >>> >>> On 3/30/16 3:21 PM, Allen, Benjamin S. wrote: >>>> Yes correct it's in cache mode. >>>> >>>> Ben >>>> >>>>> On Mar 30, 2016, at 3:16 PM, Vitali A. Morozov <morozov(a)anl.gov> wrote: >>>>> >>>>> >>>>> I believe I am running in cache mode, at least this was previously a default setting. Ben can confirm. >>>>> >>>>> Vitali >>>>> >>>>> >>>>> >>>>> On 03/30/2016 03:14 PM, Kumaran, Kalyan wrote: >>>>>> Are you running in flat mode? Can you choose a size that fits in IPM in this mode? You are seeing only DDR bandwidth and not IPM. >>>>>> >>>>>> This is the notes for KNH from SOW: >>>>>> >>>>>> KNH node performance was projected for the STREAM benchmark for three cases: >>>>>> >>>>>> A) IPM is configured in flat mode with all accesses targeting IPM memory addresses. In this mode, STREAM performance is within 90% of the peak IPM performance. >>>>>> >>>>>> B) IPM is configured in cache mode. In this mode, non-temporal stores are used to minimize ‘Read-For-Ownership’ cache coherency overhead. The use of non-temporal stores, however, requires an additional read for each store. This overhead causes a drop in BW from 603 GB/s to 404 GB/s (33%) for STREAM copy and 603 GB/s to 452 GB/s (25%) for STREAM Triad. Note that the additional read is required only for non-temporal stores. It is not needed for write-backs from the processor caches. >>>>>> >>>>>> C) IPM is configured in flat mode with all accesses targeting NVM memory addresses: The controller reads 256B from the NVM and places it in the write data buffer. The subsequent writes to the same 256B does not take NVM BW. Eventually, the 256B data is written back to NVM, which consumes its BW. This results in 25% BW loss for STREAM Triad and 33% BW loss for STREAM copy. >>>>>> >>>>>> >>>>>> >>>>>>> On Mar 30, 2016, at 2:51 PM, Vitali A. Morozov <morozov(a)anl.gov> wrote: >>>>>>> >>>>>>> A quick update on STREAM triad on KNL B0 after BIOS upgrade. >>>>>>> >>>>>>> The first figure is for 1 core. About 35 GB/s from L2 cache, drops down to 11.5 GB/s up to 64 MB, but then increases to a little over 13 GB/s. All exactly as expected. >>>>>>> >>>>>>> The second figure is for 8 tiles (a quadrant). Near linear increase in bandwidth up to 284 GB/s when data fit into all L2 caches - 8 MB. This is near 80% of peak L2 bandwidth, which is expected at 357.6 GB/s then drops to about 78-80 GB/s, a little less then one forth of expected peak of 120 GB/s (67% of peak). Two points were obtained for the footprint outside MCDRAM size - both show 79 GB/s. >>>>>>> >>>>>>> The last figure is for entire node. The accumulative L2 cache bandwidth is a little over 1 TB/s (1.4 TB/s is a peak) with 32 MB capacity. After this footprint, I observe the drop down of the bandwidth to 80 GB/s. This is strange as MCDRAM bandwidth is rated 480 GB/s. >>>>>>> >>>>>>> Issues: >>>>>>> >>>>>>> 1. Full node MCDRAM bandwidth does not reach expected 420-430 GB/s bandwidth. >>>>>>> 2. 75 GB memory footprint case failed all the the runs. >>>>>>> >>>>>>> Vitali >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 03/25/2016 03:29 PM, Vitali A. Morozov wrote: >>>>>>>> Hello, >>>>>>>> >>>>>>>> here are the figures that I have obtained for STREAM triad. The peak numbers are: >>>>>>>> >>>>>>>> L2 peak bandwidth 44.7 GB/s, based on the size of the request table, L2 hit latency, and core capabilities to issue memory requests clocked at 1.3 GHz. >>>>>>>> MCDRAM peak bandwidth 480 GB/s scaled down to 1 port of L2 tile => 15 GB/s. Could be slightly less if the port was provisioned to 36 tiles (13.3 GB/s in this case). >>>>>>>> DDR4 peak bandwidth 112 GB/s is based on 6 channel DDR4-2400. Being scaled down to 2134 GT/s, it will be 100 GB/s. >>>>>>>> >>>>>>>> The first graph provides STREAM triad bandwidth for 1 core and 1 tile (2 cores, 1 thread is running on each core). >>>>>>>> >>>>>>>> I do not look at L2 cache bandwidth, as 44.7 GB/s number is very hard to reach and there might be many factors that can reduce effective bandwidth (moreover, I was not carefully coding stream to get the full L2 bandwidth anyway). The measured 32 GB/s from a core and 37 GB/s for 2 cores on a tile are within 80% of expected tolerance. Running more then 2 threads on a tile severely degrades performance (20 GB/s at best), which is expected as the L2 request table gets quickly full. >>>>>>>> >>>>>>>> When data fits into MCDRAM, I have measured 10-12 GB/s for a core and 10-13 GB/s for a tile with strange repeatable artifacts at around 64 MB and 640 MB. The tile performance is more uniform, but starts quickly degrade at around 900 MB, which is strange as we should have plenty of room in MCDRAM. Also, I am measuring triad, so the cache must be warmed out by the time I started it as triad is one of the last benchmarks in STREAM suite. In fact I do not expect any performance drop between about 8 MB up to 192 GB - the memory bandwidth should be capable to provide single port MCDRAM-available bandwidth at ease. >>>>>>>> >>>>>>>> The second graph provides STREAM triad numbers for 8 tiles. Again L2 cache bandwidth is a kind of OK except that it quickly gets down not reaching the expected max of 8 MB combined L2 cache. The MCDRAM bandwidth is within the expected range (with a very strange dip in between 12 - 40 MB. Then, the bandwidth decreases with a very unusual change of curvature at around 2 GB. Even if MCDRAM does not work as cache, I would expect to see around 90 GB/s - I am measuring 4 GB/s at 38 GB. >>>>>>>> >>>>>>>> Sounds like Ben have installed CR instead of DDR :) >>>>>>>> >>>>>>>> The second graph numbers can slightly be improved with careful mapping the processes to the cores, but not very much. >>>>>>>> >>>>>>>> Vitali >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Intel-nda mailing list >>>>>>>> >>>>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>>>> <stream-knlb0-1core.png><stream-knlb0-8tiles.png><stream-knlb0-32tiles.png>_______________________________________________ >>>>>>> Intel-nda mailing list >>>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>> _______________________________________________ >>>>> Intel-nda mailing list >>>>> Intel-nda(a)lists.jlse.anl.gov >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>> _______________________________________________ >>>> Intel-nda mailing list >>>> Intel-nda(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0