- Intel-nda - mailman3.cels.anl.gov

[JLSE-Intel-NDA] Fwd: Memory Mode for MCDRAM
by Kumaran, Kalyan 04 Apr '16

04 Apr '16

forwarding to this list as Scott asked this question. -Kumar Begin forwarded message: From: "Allen, Benjamin S." <bsallen(a)alcf.anl.gov<mailto:[email protected]>> Subject: Re: Memory Mode for MCDRAM Date: April 4, 2016 at 12:22:49 PM CDT To: "Kirkendall, Keith G" <keith.g.kirkendall(a)intel.com<mailto:[email protected]>> Cc: "Fromkin, Russ" <russ.fromkin(a)intel.com<mailto:[email protected]>>, "Allison, CM" <cm.allison(a)intel.com<mailto:[email protected]>>, "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov<mailto:[email protected]>> I'm hoping to not have to use expect scripts or similar via a serial console over IPMI to edit the BIOS. Ideally something exposed via IPMI that could be set via "ipmitool raw" or similar. Thanks, Ben On Apr 4, 2016, at 12:13 PM, Kirkendall, Keith G <keith.g.kirkendall(a)intel.com<mailto:[email protected]>> wrote: I will check. It seems reasonable that you could change a BIOS setting via a script somehow & have it take effect on reboot. I know we do that with core count settings on some of our clusters. -----Original Message----- From: Allen, Benjamin S. [mailto:[email protected]] Sent: Monday, April 4, 2016 12:10 PM To: Fromkin, Russ <russ.fromkin(a)intel.com<mailto:[email protected]>>; Kirkendall, Keith G <keith.g.kirkendall(a)intel.com<mailto:[email protected]>> Cc: Allison, CM <cm.allison(a)intel.com<mailto:[email protected]>>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov<mailto:[email protected]>> Subject: Memory Mode for MCDRAM Russ and Keith, Do you know or can you find out if there's a way to change the Memory model for the 2LM (MCDRAM) via some mechanism from Linux or IPMI, (eg. from cached to flat mode). We're hoping to provide the users some self-service ability to change the mode and have it in effect on reboot. Thanks, Ben

1 0

Re: [JLSE-Intel-NDA] B0 Stream triad
by Vitali A. Morozov 04 Apr '16

04 Apr '16

I totally agree with that. Vitali On 04/04/2016 01:41 PM, Scott Parker wrote: > > We are going to need a way of changing from flat, to hybrid, to cache > modes that can be done by the user > (and a way of checking the mode if we don't already have it). Is this > something that could be baked into a > script? I assume a reboot is required, but we really do need to be > able to test out the different modes against > various application. > > -Scott > > On 4/4/16 12:05 PM, Allen, Benjamin S. wrote: >> Hi Scott, >> >> Swapping into flat mode is a BIOS option. I can change that if you'd >> like, and no one else is using the machine. >> >> Ben >> >>> On Apr 1, 2016, at 6:27 PM, Scott Parker <sparker(a)anl.gov> wrote: >>> >>> >>> Are we able to boot the KNL into flat mode? I'm seeing the same >>> Stream bandwidth as Vitali and >>> the resulting Nekbone performance, while still 4x better than BG/Q, >>> is 2x slower than what I was >>> seeing on A0 hardware. Best nekbone performance requires full IPM >>> bandwidth. >>> >>> -Scott >>> >>> On 3/30/16 3:21 PM, Allen, Benjamin S. wrote: >>>> Yes correct it's in cache mode. >>>> >>>> Ben >>>> >>>>> On Mar 30, 2016, at 3:16 PM, Vitali A. Morozov <morozov(a)anl.gov> >>>>> wrote: >>>>> >>>>> >>>>> I believe I am running in cache mode, at least this was previously >>>>> a default setting. Ben can confirm. >>>>> >>>>> Vitali >>>>> >>>>> >>>>> >>>>> On 03/30/2016 03:14 PM, Kumaran, Kalyan wrote: >>>>>> Are you running in flat mode? Can you choose a size that fits in >>>>>> IPM in this mode? You are seeing only DDR bandwidth and not IPM. >>>>>> >>>>>> This is the notes for KNH from SOW: >>>>>> >>>>>> KNH node performance was projected for the STREAM benchmark for >>>>>> three cases: >>>>>> >>>>>> A) IPM is configured in flat mode with all accesses targeting >>>>>> IPM memory addresses. In this mode, STREAM performance is within >>>>>> 90% of the peak IPM performance. >>>>>> >>>>>> B) IPM is configured in cache mode. In this mode, >>>>>> non-temporal stores are used to minimize ï¿½Read-For-Ownershipï¿½ >>>>>> cache coherency overhead. The use of non-temporal stores, >>>>>> however, requires an additional read for each store. This >>>>>> overhead causes a drop in BW from 603 GB/s to 404 GB/s (33%) for >>>>>> STREAM copy and 603 GB/s to 452 GB/s (25%) for STREAM Triad. Note >>>>>> that the additional read is required only for non-temporal >>>>>> stores. It is not needed for write-backs from the processor caches. >>>>>> >>>>>> C) IPM is configured in flat mode with all accesses targeting >>>>>> NVM memory addresses: The controller reads 256B from the NVM and >>>>>> places it in the write data buffer. The subsequent writes to the >>>>>> same 256B does not take NVM BW. Eventually, the 256B data is >>>>>> written back to NVM, which consumes its BW. This results in 25% >>>>>> BW loss for STREAM Triad and 33% BW loss for STREAM copy. >>>>>> >>>>>> >>>>>> >>>>>>> On Mar 30, 2016, at 2:51 PM, Vitali A. Morozov <morozov(a)anl.gov> >>>>>>> wrote: >>>>>>> >>>>>>> A quick update on STREAM triad on KNL B0 after BIOS upgrade. >>>>>>> >>>>>>> The first figure is for 1 core. About 35 GB/s from L2 cache, >>>>>>> drops down to 11.5 GB/s up to 64 MB, but then increases to a >>>>>>> little over 13 GB/s. All exactly as expected. >>>>>>> >>>>>>> The second figure is for 8 tiles (a quadrant). Near linear >>>>>>> increase in bandwidth up to 284 GB/s when data fit into all L2 >>>>>>> caches - 8 MB. This is near 80% of peak L2 bandwidth, which is >>>>>>> expected at 357.6 GB/s then drops to about 78-80 GB/s, a little >>>>>>> less then one forth of expected peak of 120 GB/s (67% of peak). >>>>>>> Two points were obtained for the footprint outside MCDRAM size - >>>>>>> both show 79 GB/s. >>>>>>> >>>>>>> The last figure is for entire node. The accumulative L2 cache >>>>>>> bandwidth is a little over 1 TB/s (1.4 TB/s is a peak) with 32 >>>>>>> MB capacity. After this footprint, I observe the drop down of >>>>>>> the bandwidth to 80 GB/s. This is strange as MCDRAM bandwidth is >>>>>>> rated 480 GB/s. >>>>>>> >>>>>>> Issues: >>>>>>> >>>>>>> 1. Full node MCDRAM bandwidth does not reach expected 420-430 >>>>>>> GB/s bandwidth. >>>>>>> 2. 75 GB memory footprint case failed all the the runs. >>>>>>> >>>>>>> Vitali >>>>>>> >>>>>>> >>>>>>> >>>>>>> On 03/25/2016 03:29 PM, Vitali A. Morozov wrote: >>>>>>>> Hello, >>>>>>>> >>>>>>>> here are the figures that I have obtained for STREAM triad. The >>>>>>>> peak numbers are: >>>>>>>> >>>>>>>> L2 peak bandwidth 44.7 GB/s, based on the size of the request >>>>>>>> table, L2 hit latency, and core capabilities to issue memory >>>>>>>> requests clocked at 1.3 GHz. >>>>>>>> MCDRAM peak bandwidth 480 GB/s scaled down to 1 port of L2 tile >>>>>>>> => 15 GB/s. Could be slightly less if the port was provisioned >>>>>>>> to 36 tiles (13.3 GB/s in this case). >>>>>>>> DDR4 peak bandwidth 112 GB/s is based on 6 channel DDR4-2400. >>>>>>>> Being scaled down to 2134 GT/s, it will be 100 GB/s. >>>>>>>> >>>>>>>> The first graph provides STREAM triad bandwidth for 1 core and >>>>>>>> 1 tile (2 cores, 1 thread is running on each core). >>>>>>>> >>>>>>>> I do not look at L2 cache bandwidth, as 44.7 GB/s number is >>>>>>>> very hard to reach and there might be many factors that can >>>>>>>> reduce effective bandwidth (moreover, I was not carefully >>>>>>>> coding stream to get the full L2 bandwidth anyway). The >>>>>>>> measured 32 GB/s from a core and 37 GB/s for 2 cores on a tile >>>>>>>> are within 80% of expected tolerance. Running more then 2 >>>>>>>> threads on a tile severely degrades performance (20 GB/s at >>>>>>>> best), which is expected as the L2 request table gets quickly >>>>>>>> full. >>>>>>>> >>>>>>>> When data fits into MCDRAM, I have measured 10-12 GB/s for a >>>>>>>> core and 10-13 GB/s for a tile with strange repeatable >>>>>>>> artifacts at around 64 MB and 640 MB. The tile performance is >>>>>>>> more uniform, but starts quickly degrade at around 900 MB, >>>>>>>> which is strange as we should have plenty of room in MCDRAM. >>>>>>>> Also, I am measuring triad, so the cache must be warmed out by >>>>>>>> the time I started it as triad is one of the last benchmarks in >>>>>>>> STREAM suite. In fact I do not expect any performance drop >>>>>>>> between about 8 MB up to 192 GB - the memory bandwidth should >>>>>>>> be capable to provide single port MCDRAM-available bandwidth at >>>>>>>> ease. >>>>>>>> >>>>>>>> The second graph provides STREAM triad numbers for 8 tiles. >>>>>>>> Again L2 cache bandwidth is a kind of OK except that it quickly >>>>>>>> gets down not reaching the expected max of 8 MB combined L2 >>>>>>>> cache. The MCDRAM bandwidth is within the expected range (with >>>>>>>> a very strange dip in between 12 - 40 MB. Then, the bandwidth >>>>>>>> decreases with a very unusual change of curvature at around 2 >>>>>>>> GB. Even if MCDRAM does not work as cache, I would expect to >>>>>>>> see around 90 GB/s - I am measuring 4 GB/s at 38 GB. >>>>>>>> >>>>>>>> Sounds like Ben have installed CR instead of DDR :) >>>>>>>> >>>>>>>> The second graph numbers can slightly be improved with careful >>>>>>>> mapping the processes to the cores, but not very much. >>>>>>>> >>>>>>>> Vitali >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Intel-nda mailing list >>>>>>>> >>>>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>>>> <stream-knlb0-1core.png><stream-knlb0-8tiles.png><stream-knlb0-32tiles.png>_______________________________________________ >>>>>>> >>>>>>> Intel-nda mailing list >>>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>> _______________________________________________ >>>>> Intel-nda mailing list >>>>> Intel-nda(a)lists.jlse.anl.gov >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>> _______________________________________________ >>>> Intel-nda mailing list >>>> Intel-nda(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] B0 Stream triad
by Scott Parker 04 Apr '16

04 Apr '16

We are going to need a way of changing from flat, to hybrid, to cache modes that can be done by the user (and a way of checking the mode if we don't already have it). Is this something that could be baked into a script? I assume a reboot is required, but we really do need to be able to test out the different modes against various application. -Scott On 4/4/16 12:05 PM, Allen, Benjamin S. wrote: > Hi Scott, > > Swapping into flat mode is a BIOS option. I can change that if you'd like, and no one else is using the machine. > > Ben > >> On Apr 1, 2016, at 6:27 PM, Scott Parker <sparker(a)anl.gov> wrote: >> >> >> Are we able to boot the KNL into flat mode? I'm seeing the same Stream bandwidth as Vitali and >> the resulting Nekbone performance, while still 4x better than BG/Q, is 2x slower than what I was >> seeing on A0 hardware. Best nekbone performance requires full IPM bandwidth. >> >> -Scott >> >> On 3/30/16 3:21 PM, Allen, Benjamin S. wrote: >>> Yes correct it's in cache mode. >>> >>> Ben >>> >>>> On Mar 30, 2016, at 3:16 PM, Vitali A. Morozov <morozov(a)anl.gov> wrote: >>>> >>>> >>>> I believe I am running in cache mode, at least this was previously a default setting. Ben can confirm. >>>> >>>> Vitali >>>> >>>> >>>> >>>> On 03/30/2016 03:14 PM, Kumaran, Kalyan wrote: >>>>> Are you running in flat mode? Can you choose a size that fits in IPM in this mode? You are seeing only DDR bandwidth and not IPM. >>>>> >>>>> This is the notes for KNH from SOW: >>>>> >>>>> KNH node performance was projected for the STREAM benchmark for three cases: >>>>> >>>>> A) IPM is configured in flat mode with all accesses targeting IPM memory addresses. In this mode, STREAM performance is within 90% of the peak IPM performance. >>>>> >>>>> B) IPM is configured in cache mode. In this mode, non-temporal stores are used to minimize ï¿½Read-For-Ownershipï¿½ cache coherency overhead. The use of non-temporal stores, however, requires an additional read for each store. This overhead causes a drop in BW from 603 GB/s to 404 GB/s (33%) for STREAM copy and 603 GB/s to 452 GB/s (25%) for STREAM Triad. Note that the additional read is required only for non-temporal stores. It is not needed for write-backs from the processor caches. >>>>> >>>>> C) IPM is configured in flat mode with all accesses targeting NVM memory addresses: The controller reads 256B from the NVM and places it in the write data buffer. The subsequent writes to the same 256B does not take NVM BW. Eventually, the 256B data is written back to NVM, which consumes its BW. This results in 25% BW loss for STREAM Triad and 33% BW loss for STREAM copy. >>>>> >>>>> >>>>> >>>>>> On Mar 30, 2016, at 2:51 PM, Vitali A. Morozov <morozov(a)anl.gov> wrote: >>>>>> >>>>>> A quick update on STREAM triad on KNL B0 after BIOS upgrade. >>>>>> >>>>>> The first figure is for 1 core. About 35 GB/s from L2 cache, drops down to 11.5 GB/s up to 64 MB, but then increases to a little over 13 GB/s. All exactly as expected. >>>>>> >>>>>> The second figure is for 8 tiles (a quadrant). Near linear increase in bandwidth up to 284 GB/s when data fit into all L2 caches - 8 MB. This is near 80% of peak L2 bandwidth, which is expected at 357.6 GB/s then drops to about 78-80 GB/s, a little less then one forth of expected peak of 120 GB/s (67% of peak). Two points were obtained for the footprint outside MCDRAM size - both show 79 GB/s. >>>>>> >>>>>> The last figure is for entire node. The accumulative L2 cache bandwidth is a little over 1 TB/s (1.4 TB/s is a peak) with 32 MB capacity. After this footprint, I observe the drop down of the bandwidth to 80 GB/s. This is strange as MCDRAM bandwidth is rated 480 GB/s. >>>>>> >>>>>> Issues: >>>>>> >>>>>> 1. Full node MCDRAM bandwidth does not reach expected 420-430 GB/s bandwidth. >>>>>> 2. 75 GB memory footprint case failed all the the runs. >>>>>> >>>>>> Vitali >>>>>> >>>>>> >>>>>> >>>>>> On 03/25/2016 03:29 PM, Vitali A. Morozov wrote: >>>>>>> Hello, >>>>>>> >>>>>>> here are the figures that I have obtained for STREAM triad. The peak numbers are: >>>>>>> >>>>>>> L2 peak bandwidth 44.7 GB/s, based on the size of the request table, L2 hit latency, and core capabilities to issue memory requests clocked at 1.3 GHz. >>>>>>> MCDRAM peak bandwidth 480 GB/s scaled down to 1 port of L2 tile => 15 GB/s. Could be slightly less if the port was provisioned to 36 tiles (13.3 GB/s in this case). >>>>>>> DDR4 peak bandwidth 112 GB/s is based on 6 channel DDR4-2400. Being scaled down to 2134 GT/s, it will be 100 GB/s. >>>>>>> >>>>>>> The first graph provides STREAM triad bandwidth for 1 core and 1 tile (2 cores, 1 thread is running on each core). >>>>>>> >>>>>>> I do not look at L2 cache bandwidth, as 44.7 GB/s number is very hard to reach and there might be many factors that can reduce effective bandwidth (moreover, I was not carefully coding stream to get the full L2 bandwidth anyway). The measured 32 GB/s from a core and 37 GB/s for 2 cores on a tile are within 80% of expected tolerance. Running more then 2 threads on a tile severely degrades performance (20 GB/s at best), which is expected as the L2 request table gets quickly full. >>>>>>> >>>>>>> When data fits into MCDRAM, I have measured 10-12 GB/s for a core and 10-13 GB/s for a tile with strange repeatable artifacts at around 64 MB and 640 MB. The tile performance is more uniform, but starts quickly degrade at around 900 MB, which is strange as we should have plenty of room in MCDRAM. Also, I am measuring triad, so the cache must be warmed out by the time I started it as triad is one of the last benchmarks in STREAM suite. In fact I do not expect any performance drop between about 8 MB up to 192 GB - the memory bandwidth should be capable to provide single port MCDRAM-available bandwidth at ease. >>>>>>> >>>>>>> The second graph provides STREAM triad numbers for 8 tiles. Again L2 cache bandwidth is a kind of OK except that it quickly gets down not reaching the expected max of 8 MB combined L2 cache. The MCDRAM bandwidth is within the expected range (with a very strange dip in between 12 - 40 MB. Then, the bandwidth decreases with a very unusual change of curvature at around 2 GB. Even if MCDRAM does not work as cache, I would expect to see around 90 GB/s - I am measuring 4 GB/s at 38 GB. >>>>>>> >>>>>>> Sounds like Ben have installed CR instead of DDR :) >>>>>>> >>>>>>> The second graph numbers can slightly be improved with careful mapping the processes to the cores, but not very much. >>>>>>> >>>>>>> Vitali >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Intel-nda mailing list >>>>>>> >>>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>>> <stream-knlb0-1core.png><stream-knlb0-8tiles.png><stream-knlb0-32tiles.png>_______________________________________________ >>>>>> Intel-nda mailing list >>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>> _______________________________________________ >>>> Intel-nda mailing list >>>> Intel-nda(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] KNL node
by Kumaran, Kalyan 04 Apr '16

04 Apr '16

Should we open it up to more apps folks (beyond folks running acceptance)? Is the node kept busy? Thanks, Kumar

1 0

Re: [JLSE-Intel-NDA] B0 Stream triad
by Allen, Benjamin S. 04 Apr '16

04 Apr '16

Hi Scott, Swapping into flat mode is a BIOS option. I can change that if you'd like, and no one else is using the machine. Ben > On Apr 1, 2016, at 6:27 PM, Scott Parker <sparker(a)anl.gov> wrote: > > > Are we able to boot the KNL into flat mode? I'm seeing the same Stream bandwidth as Vitali and > the resulting Nekbone performance, while still 4x better than BG/Q, is 2x slower than what I was > seeing on A0 hardware. Best nekbone performance requires full IPM bandwidth. > > -Scott > > On 3/30/16 3:21 PM, Allen, Benjamin S. wrote: >> Yes correct it's in cache mode. >> >> Ben >> >>> On Mar 30, 2016, at 3:16 PM, Vitali A. Morozov <morozov(a)anl.gov> wrote: >>> >>> >>> I believe I am running in cache mode, at least this was previously a default setting. Ben can confirm. >>> >>> Vitali >>> >>> >>> >>> On 03/30/2016 03:14 PM, Kumaran, Kalyan wrote: >>>> Are you running in flat mode? Can you choose a size that fits in IPM in this mode? You are seeing only DDR bandwidth and not IPM. >>>> >>>> This is the notes for KNH from SOW: >>>> >>>> KNH node performance was projected for the STREAM benchmark for three cases: >>>> >>>> A) IPM is configured in flat mode with all accesses targeting IPM memory addresses. In this mode, STREAM performance is within 90% of the peak IPM performance. >>>> >>>> B) IPM is configured in cache mode. In this mode, non-temporal stores are used to minimize ‘Read-For-Ownership’ cache coherency overhead. The use of non-temporal stores, however, requires an additional read for each store. This overhead causes a drop in BW from 603 GB/s to 404 GB/s (33%) for STREAM copy and 603 GB/s to 452 GB/s (25%) for STREAM Triad. Note that the additional read is required only for non-temporal stores. It is not needed for write-backs from the processor caches. >>>> >>>> C) IPM is configured in flat mode with all accesses targeting NVM memory addresses: The controller reads 256B from the NVM and places it in the write data buffer. The subsequent writes to the same 256B does not take NVM BW. Eventually, the 256B data is written back to NVM, which consumes its BW. This results in 25% BW loss for STREAM Triad and 33% BW loss for STREAM copy. >>>> >>>> >>>> >>>>> On Mar 30, 2016, at 2:51 PM, Vitali A. Morozov <morozov(a)anl.gov> wrote: >>>>> >>>>> A quick update on STREAM triad on KNL B0 after BIOS upgrade. >>>>> >>>>> The first figure is for 1 core. About 35 GB/s from L2 cache, drops down to 11.5 GB/s up to 64 MB, but then increases to a little over 13 GB/s. All exactly as expected. >>>>> >>>>> The second figure is for 8 tiles (a quadrant). Near linear increase in bandwidth up to 284 GB/s when data fit into all L2 caches - 8 MB. This is near 80% of peak L2 bandwidth, which is expected at 357.6 GB/s then drops to about 78-80 GB/s, a little less then one forth of expected peak of 120 GB/s (67% of peak). Two points were obtained for the footprint outside MCDRAM size - both show 79 GB/s. >>>>> >>>>> The last figure is for entire node. The accumulative L2 cache bandwidth is a little over 1 TB/s (1.4 TB/s is a peak) with 32 MB capacity. After this footprint, I observe the drop down of the bandwidth to 80 GB/s. This is strange as MCDRAM bandwidth is rated 480 GB/s. >>>>> >>>>> Issues: >>>>> >>>>> 1. Full node MCDRAM bandwidth does not reach expected 420-430 GB/s bandwidth. >>>>> 2. 75 GB memory footprint case failed all the the runs. >>>>> >>>>> Vitali >>>>> >>>>> >>>>> >>>>> On 03/25/2016 03:29 PM, Vitali A. Morozov wrote: >>>>>> Hello, >>>>>> >>>>>> here are the figures that I have obtained for STREAM triad. The peak numbers are: >>>>>> >>>>>> L2 peak bandwidth 44.7 GB/s, based on the size of the request table, L2 hit latency, and core capabilities to issue memory requests clocked at 1.3 GHz. >>>>>> MCDRAM peak bandwidth 480 GB/s scaled down to 1 port of L2 tile => 15 GB/s. Could be slightly less if the port was provisioned to 36 tiles (13.3 GB/s in this case). >>>>>> DDR4 peak bandwidth 112 GB/s is based on 6 channel DDR4-2400. Being scaled down to 2134 GT/s, it will be 100 GB/s. >>>>>> >>>>>> The first graph provides STREAM triad bandwidth for 1 core and 1 tile (2 cores, 1 thread is running on each core). >>>>>> >>>>>> I do not look at L2 cache bandwidth, as 44.7 GB/s number is very hard to reach and there might be many factors that can reduce effective bandwidth (moreover, I was not carefully coding stream to get the full L2 bandwidth anyway). The measured 32 GB/s from a core and 37 GB/s for 2 cores on a tile are within 80% of expected tolerance. Running more then 2 threads on a tile severely degrades performance (20 GB/s at best), which is expected as the L2 request table gets quickly full. >>>>>> >>>>>> When data fits into MCDRAM, I have measured 10-12 GB/s for a core and 10-13 GB/s for a tile with strange repeatable artifacts at around 64 MB and 640 MB. The tile performance is more uniform, but starts quickly degrade at around 900 MB, which is strange as we should have plenty of room in MCDRAM. Also, I am measuring triad, so the cache must be warmed out by the time I started it as triad is one of the last benchmarks in STREAM suite. In fact I do not expect any performance drop between about 8 MB up to 192 GB - the memory bandwidth should be capable to provide single port MCDRAM-available bandwidth at ease. >>>>>> >>>>>> The second graph provides STREAM triad numbers for 8 tiles. Again L2 cache bandwidth is a kind of OK except that it quickly gets down not reaching the expected max of 8 MB combined L2 cache. The MCDRAM bandwidth is within the expected range (with a very strange dip in between 12 - 40 MB. Then, the bandwidth decreases with a very unusual change of curvature at around 2 GB. Even if MCDRAM does not work as cache, I would expect to see around 90 GB/s - I am measuring 4 GB/s at 38 GB. >>>>>> >>>>>> Sounds like Ben have installed CR instead of DDR :) >>>>>> >>>>>> The second graph numbers can slightly be improved with careful mapping the processes to the cores, but not very much. >>>>>> >>>>>> Vitali >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Intel-nda mailing list >>>>>> >>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>> <stream-knlb0-1core.png><stream-knlb0-8tiles.png><stream-knlb0-32tiles.png>_______________________________________________ >>>>> Intel-nda mailing list >>>>> Intel-nda(a)lists.jlse.anl.gov >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] B0 Stream triad
by Scott Parker 01 Apr '16

01 Apr '16

Are we able to boot the KNL into flat mode? I'm seeing the same Stream bandwidth as Vitali and the resulting Nekbone performance, while still 4x better than BG/Q, is 2x slower than what I was seeing on A0 hardware. Best nekbone performance requires full IPM bandwidth. -Scott On 3/30/16 3:21 PM, Allen, Benjamin S. wrote: > Yes correct it's in cache mode. > > Ben > >> On Mar 30, 2016, at 3:16 PM, Vitali A. Morozov <morozov(a)anl.gov> wrote: >> >> >> I believe I am running in cache mode, at least this was previously a default setting. Ben can confirm. >> >> Vitali >> >> >> >> On 03/30/2016 03:14 PM, Kumaran, Kalyan wrote: >>> Are you running in flat mode? Can you choose a size that fits in IPM in this mode? You are seeing only DDR bandwidth and not IPM. >>> >>> This is the notes for KNH from SOW: >>> >>> KNH node performance was projected for the STREAM benchmark for three cases: >>> >>> A) IPM is configured in flat mode with all accesses targeting IPM memory addresses. In this mode, STREAM performance is within 90% of the peak IPM performance. >>> >>> B) IPM is configured in cache mode. In this mode, non-temporal stores are used to minimize ï¿½Read-For-Ownershipï¿½ cache coherency overhead. The use of non-temporal stores, however, requires an additional read for each store. This overhead causes a drop in BW from 603 GB/s to 404 GB/s (33%) for STREAM copy and 603 GB/s to 452 GB/s (25%) for STREAM Triad. Note that the additional read is required only for non-temporal stores. It is not needed for write-backs from the processor caches. >>> >>> C) IPM is configured in flat mode with all accesses targeting NVM memory addresses: The controller reads 256B from the NVM and places it in the write data buffer. The subsequent writes to the same 256B does not take NVM BW. Eventually, the 256B data is written back to NVM, which consumes its BW. This results in 25% BW loss for STREAM Triad and 33% BW loss for STREAM copy. >>> >>> >>> >>>> On Mar 30, 2016, at 2:51 PM, Vitali A. Morozov <morozov(a)anl.gov> wrote: >>>> >>>> A quick update on STREAM triad on KNL B0 after BIOS upgrade. >>>> >>>> The first figure is for 1 core. About 35 GB/s from L2 cache, drops down to 11.5 GB/s up to 64 MB, but then increases to a little over 13 GB/s. All exactly as expected. >>>> >>>> The second figure is for 8 tiles (a quadrant). Near linear increase in bandwidth up to 284 GB/s when data fit into all L2 caches - 8 MB. This is near 80% of peak L2 bandwidth, which is expected at 357.6 GB/s then drops to about 78-80 GB/s, a little less then one forth of expected peak of 120 GB/s (67% of peak). Two points were obtained for the footprint outside MCDRAM size - both show 79 GB/s. >>>> >>>> The last figure is for entire node. The accumulative L2 cache bandwidth is a little over 1 TB/s (1.4 TB/s is a peak) with 32 MB capacity. After this footprint, I observe the drop down of the bandwidth to 80 GB/s. This is strange as MCDRAM bandwidth is rated 480 GB/s. >>>> >>>> Issues: >>>> >>>> 1. Full node MCDRAM bandwidth does not reach expected 420-430 GB/s bandwidth. >>>> 2. 75 GB memory footprint case failed all the the runs. >>>> >>>> Vitali >>>> >>>> >>>> >>>> On 03/25/2016 03:29 PM, Vitali A. Morozov wrote: >>>>> Hello, >>>>> >>>>> here are the figures that I have obtained for STREAM triad. The peak numbers are: >>>>> >>>>> L2 peak bandwidth 44.7 GB/s, based on the size of the request table, L2 hit latency, and core capabilities to issue memory requests clocked at 1.3 GHz. >>>>> MCDRAM peak bandwidth 480 GB/s scaled down to 1 port of L2 tile => 15 GB/s. Could be slightly less if the port was provisioned to 36 tiles (13.3 GB/s in this case). >>>>> DDR4 peak bandwidth 112 GB/s is based on 6 channel DDR4-2400. Being scaled down to 2134 GT/s, it will be 100 GB/s. >>>>> >>>>> The first graph provides STREAM triad bandwidth for 1 core and 1 tile (2 cores, 1 thread is running on each core). >>>>> >>>>> I do not look at L2 cache bandwidth, as 44.7 GB/s number is very hard to reach and there might be many factors that can reduce effective bandwidth (moreover, I was not carefully coding stream to get the full L2 bandwidth anyway). The measured 32 GB/s from a core and 37 GB/s for 2 cores on a tile are within 80% of expected tolerance. Running more then 2 threads on a tile severely degrades performance (20 GB/s at best), which is expected as the L2 request table gets quickly full. >>>>> >>>>> When data fits into MCDRAM, I have measured 10-12 GB/s for a core and 10-13 GB/s for a tile with strange repeatable artifacts at around 64 MB and 640 MB. The tile performance is more uniform, but starts quickly degrade at around 900 MB, which is strange as we should have plenty of room in MCDRAM. Also, I am measuring triad, so the cache must be warmed out by the time I started it as triad is one of the last benchmarks in STREAM suite. In fact I do not expect any performance drop between about 8 MB up to 192 GB - the memory bandwidth should be capable to provide single port MCDRAM-available bandwidth at ease. >>>>> >>>>> The second graph provides STREAM triad numbers for 8 tiles. Again L2 cache bandwidth is a kind of OK except that it quickly gets down not reaching the expected max of 8 MB combined L2 cache. The MCDRAM bandwidth is within the expected range (with a very strange dip in between 12 - 40 MB. Then, the bandwidth decreases with a very unusual change of curvature at around 2 GB. Even if MCDRAM does not work as cache, I would expect to see around 90 GB/s - I am measuring 4 GB/s at 38 GB. >>>>> >>>>> Sounds like Ben have installed CR instead of DDR :) >>>>> >>>>> The second graph numbers can slightly be improved with careful mapping the processes to the cores, but not very much. >>>>> >>>>> Vitali >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Intel-nda mailing list >>>>> >>>>> Intel-nda(a)lists.jlse.anl.gov >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>> <stream-knlb0-1core.png><stream-knlb0-8tiles.png><stream-knlb0-32tiles.png>_______________________________________________ >>>> Intel-nda mailing list >>>> Intel-nda(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] MILCmk results on KNL b0
by Harms, Kevin 31 Mar '16

31 Mar '16

James, here's an updated spreadsheet with results for all the sizes and with 1, 2 and 4 threads per core. The 2 threads/core is almost universally better for both double and single precision and 4 threads/core is mostly better for single precision and more variable for double. kevin >Please use intel-nda list. > >-Kumar > > >> On Mar 14, 2016, at 3:29 PM, Harms, Kevin <harms(a)alcf.anl.gov> wrote: >> >> >> ok. I will take a look at that one and see what I can find out. >> >> kevin >> >> >> >> >>> Hi Kevin, >>> >>> Probably 65536 is the most relevant. This code is old now so I don't >>> care about it too much now anyway. The performance is still slower than >>> I would have expected even without any vectorization. It may be nice to >>> understand what the bottleneck is for say the single precision, 65536, >>> QLA_V_vpeq_M_times_pV case (or for simplicity, even the >>> QLA_V_veq_Ma_times_V one since that one has a more regular memory access >>> pattern). >>> >>> Cheers, >>> James >>> >>> >>> On 03/11/2016 12:10 PM, Harms, Kevin wrote: >>>> James, >>>> >>>> I ran the MILCmk CORAL benchmarks on the KNL b0. I'm attaching the results. For larger sizes, there's a 2-3x speedup over mira. What sizes do you consider to be the important ones? >>>> >>>> kevin >>>> >>>> MILC-v1 benchmark from CORAL site >>>> A2 results from running defaults in Makefile >>>> KNL B0 results compiled with Intel using basic optimization >>>> >>>> icc -std=c99 -O3 -xMIC-AVX512 -openmp -Iqla-1.7.1 -DQLA_Precision=\'F\' -DQLA_Colors=3 -DQLA_Nc=3 -c qla-1.7.1/QLA_F3_c1_veq_V_dot_V.c -o qla-1.7.1/QLA_F3_c1_veq_V_dot_V.o >>>> >>>> Run with OMP_NUM_THREADS=64 (have not tried 128 yet) >>>> >>>> I could not run the larger sizes on KNL because lack of memory. >>>> >>>> >

1 0

[JLSE-Intel-NDA] Fwd: Running HACC on KNL B0
by Kumaran, Kalyan 31 Mar '16

31 Mar '16

It would be good to share any positive results with Avinash. We should also have a call with him to discuss the STREAM results or anything else we cannot figure out. Begin forwarded message: From: "Sodani, Avinash" <avinash.sodani(a)intel.com<mailto:[email protected]>> Subject: RE: [JLSE-Intel-NDA] Running HACC on KNL B0 Date: March 31, 2016 at 8:05:28 AM CDT Very happy and relieved to see these results! Thanks for forwarding, Kumar. -Avinash From: Kumaran, Kalyan [mailto:[email protected]] Sent: Wednesday, March 30, 2016 4:04 PM To: Sodani, Avinash; Gara, Alan; Fromkin, Russ; Hur, Ibrahim Cc: Morozov, Vitali A.; Coghlan, Susan M. Subject: Fwd: [JLSE-Intel-NDA] Running HACC on KNL B0 Our experience has been very positive on B0 so far. HACCmk back to running very well and we do not see the some of the issues we saw on A0. Also Sniper did a great job in predicting the result. They agree very well. -Kumar Begin forwarded message: From: "Vitali A. Morozov" <morozov(a)anl.gov<mailto:[email protected]>> Subject: [JLSE-Intel-NDA] Running HACC on KNL B0 Date: March 30, 2016 at 5:48:48 PM CDT To: "intel-nda(a)jlse.anl.gov<mailto:[email protected]>" <intel-nda(a)jlse.anl.gov<mailto:[email protected]>> This is summary of results running HACC on KNL B0 bin 3 hardware. Quick summary: HACC is running over 5 times faster on KNL core compared to BG/Q core due to double FPU units, advantage in single precision arithmetic, and code tuning. Long write-up. 1. The peak throughput of independent AVX512 FMA instruction stream was measured at 1.7784 instructions per clock cycle. This is 81.3% of rated peak. Previous result was measured at 1.66 (75.9% of peak). 2. HACCmk was written in assembler and tuned to run on 4 hardware threads of a core. It fully utilizes AVX512 instructions, specifics of Intel ISA, single precision arithmetic, all possible optimizations and tricks. Sniper simulator was used for development. For very small problem size of N = 1488, I have measured: 1 thread, 686276 cycles, 1.1411 IPC, 42.1% of peak. 2 threads, 563599 cycles, 1.3894 IPC, 51.3% of peak. 4 threads, 501949 cycles, 1.5601 IPC, 57.6% of peak. Absolute peak of this kernel for the given mixture of instructions is 74% of peak (423902 cycles, 1.8473 IPC). For comparison, the PowerA2-based kernel is written in C with intrinsic QPX instructions, uses full double precision (except for loads) and shows the following performance: 4 threads, 4195007 cycles, 1.4064 IPC, 66.7% of peak. It is therefore expected to have the best core-to-care performance ratio close to 8.34x on cycle by cycle basis. Taking the AVX frequency of 1.1 GHz and Power A2 frequency of 1.6 GHz, the core-to-core ratio becomes 5.73x. 3. Performance of HACCmk C code version compiled by Intel compiler Version 16.0.0.109 Build 20150815 measured for N = 1488 as following. Correct IPC and Flop rate count requires additional work, and therefore not reported. 1 thread, 965060 cycles, +40.6% 2 threads, 804636 cycles, +42.8% 4 threads, 637610 cycles, +27.0% 3. The tuned HACCmk was incorporated into the full CORAL HACC application. The code was running on 16 cores of BG/Q (full node) and 16 cores of KNL B0 (a quadrant). The problem size is 320^3. The following results were measured: Cores RPN OMP TH BG/Q Time, s KNL B0, Time, s Ratio 16 4 4 16 4297 616.3308 6.98 16 4 8 32 2677 543.7294 4.92 16 4 16 64 2504 530.2267 4.72 16 8 2 16 4362 544.7519 8.00 16 8 4 32 2571 459.5265 5.59 16 8 8 64 2278 437.2058 5.18 16 16 4 64 2581 468.5037 5.50 The largest performance improvement is noticed when baseline code is running only 1 thread per core - we know that this limitation of the BG/Q architecture does not exist on KNL. Other then that, the obtained result is exactly as expected. 4. The full HACC runs on 4 nodes of BG/Q and compares with full node of KNL B0 for large 512^3 problem size. The following result was measured. Cores RPN OMP TH BG/Q Time, s KNL B0, Time, s Ratio 64 32 4 128 2824 536.1650 5.27 64 32 8 256 2503 503.0927 4.98 64 64 4 256 2539 510.3745 4.98 A very stable performance improvement with about 5x speedup is observed. Vitali _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] Running HACC on KNL B0
by Vitali A. Morozov 30 Mar '16

30 Mar '16

This is summary of results running HACC on KNL B0 bin 3 hardware. Quick summary: HACC is running over 5 times faster on KNL core compared to BG/Q core due to double FPU units, advantage in single precision arithmetic, and code tuning. Long write-up. 1. The peak throughput of independent AVX512 FMA instruction stream was measured at 1.7784 instructions per clock cycle. This is 81.3% of rated peak. Previous result was measured at 1.66 (75.9% of peak). 2. HACCmk was written in assembler and tuned to run on 4 hardware threads of a core. It fully utilizes AVX512 instructions, specifics of Intel ISA, single precision arithmetic, all possible optimizations and tricks. Sniper simulator was used for development. For very small problem size of N = 1488, I have measured: 1 thread, 686276 cycles, 1.1411 IPC, 42.1% of peak. 2 threads, 563599 cycles, 1.3894 IPC, 51.3% of peak. 4 threads, 501949 cycles, 1.5601 IPC, 57.6% of peak. Absolute peak of this kernel for the given mixture of instructions is 74% of peak (423902 cycles, 1.8473 IPC). For comparison, the PowerA2-based kernel is written in C with intrinsic QPX instructions, uses full double precision (except for loads) and shows the following performance: 4 threads, 4195007 cycles, 1.4064 IPC, 66.7% of peak. It is therefore expected to have the best core-to-care performance ratio close to 8.34x on cycle by cycle basis. Taking the AVX frequency of 1.1 GHz and Power A2 frequency of 1.6 GHz, the core-to-core ratio becomes 5.73x. 3. Performance of HACCmk C code version compiled by Intel compiler Version 16.0.0.109 Build 20150815 measured for N = 1488 as following. Correct IPC and Flop rate count requires additional work, and therefore not reported. 1 thread, 965060 cycles, +40.6% 2 threads, 804636 cycles, +42.8% 4 threads, 637610 cycles, +27.0% 3. The tuned HACCmk was incorporated into the full CORAL HACC application. The code was running on 16 cores of BG/Q (full node) and 16 cores of KNL B0 (a quadrant). The problem size is 320^3. The following results were measured: Cores RPN OMP TH BG/Q Time, s KNL B0, Time, s Ratio 16 4 4 16 4297 616.3308 6.98 16 4 8 32 2677 543.7294 4.92 16 4 16 64 2504 530.2267 4.72 16 8 2 16 4362 544.7519 8.00 16 8 4 32 2571 459.5265 5.59 16 8 8 64 2278 437.2058 5.18 16 16 4 64 2581 468.5037 5.50 The largest performance improvement is noticed when baseline code is running only 1 thread per core - we know that this limitation of the BG/Q architecture does not exist on KNL. Other then that, the obtained result is exactly as expected. 4. The full HACC runs on 4 nodes of BG/Q and compares with full node of KNL B0 for large 512^3 problem size. The following result was measured. Cores RPN OMP TH BG/Q Time, s KNL B0, Time, s Ratio 64 32 4 128 2824 536.1650 5.27 64 32 8 256 2503 503.0927 4.98 64 64 4 256 2539 510.3745 4.98 A very stable performance improvement with about 5x speedup is observed. Vitali

1 0

Re: [JLSE-Intel-NDA] B0 Stream triad
by Vitali A. Morozov 30 Mar '16

30 Mar '16

No, I cannot allocate in MCDRAM as I think I am in cache mode. On 03/30/2016 03:27 PM, Scott Parker wrote: > > Are you running with numctl set to allocate into the IPM instead of DDR? > > On 3/30/16 2:51 PM, Vitali A. Morozov wrote: >> A quick update on STREAM triad on KNL B0 after BIOS upgrade. >> >> The first figure is for 1 core. About 35 GB/s from L2 cache, drops >> down to 11.5 GB/s up to 64 MB, but then increases to a little over 13 >> GB/s. All exactly as expected. >> >> The second figure is for 8 tiles (a quadrant). Near linear increase >> in bandwidth up to 284 GB/s when data fit into all L2 caches - 8 MB. >> This is near 80% of peak L2 bandwidth, which is expected at 357.6 >> GB/s then drops to about 78-80 GB/s, a little less then one forth of >> expected peak of 120 GB/s (67% of peak). Two points were obtained for >> the footprint outside MCDRAM size - both show 79 GB/s. >> >> The last figure is for entire node. The accumulative L2 cache >> bandwidth is a little over 1 TB/s (1.4 TB/s is a peak) with 32 MB >> capacity. After this footprint, I observe the drop down of the >> bandwidth to 80 GB/s. This is strange as MCDRAM bandwidth is rated >> 480 GB/s. >> >> Issues: >> >> 1. Full node MCDRAM bandwidth does not reach expected 420-430 GB/s >> bandwidth. >> 2. 75 GB memory footprint case failed all the the runs. >> >> Vitali >> >> >> >> On 03/25/2016 03:29 PM, Vitali A. Morozov wrote: >>> Hello, >>> >>> here are the figures that I have obtained for STREAM triad. The peak >>> numbers are: >>> >>> L2 peak bandwidth 44.7 GB/s, based on the size of the request table, >>> L2 hit latency, and core capabilities to issue memory requests >>> clocked at 1.3 GHz. >>> MCDRAM peak bandwidth 480 GB/s scaled down to 1 port of L2 tile => >>> 15 GB/s. Could be slightly less if the port was provisioned to 36 >>> tiles (13.3 GB/s in this case). >>> DDR4 peak bandwidth 112 GB/s is based on 6 channel DDR4-2400. Being >>> scaled down to 2134 GT/s, it will be 100 GB/s. >>> >>> The first graph provides STREAM triad bandwidth for 1 core and 1 >>> tile (2 cores, 1 thread is running on each core). >>> >>> I do not look at L2 cache bandwidth, as 44.7 GB/s number is very >>> hard to reach and there might be many factors that can reduce >>> effective bandwidth (moreover, I was not carefully coding stream to >>> get the full L2 bandwidth anyway). The measured 32 GB/s from a core >>> and 37 GB/s for 2 cores on a tile are within 80% of expected >>> tolerance. Running more then 2 threads on a tile severely degrades >>> performance (20 GB/s at best), which is expected as the L2 request >>> table gets quickly full. >>> >>> When data fits into MCDRAM, I have measured 10-12 GB/s for a core >>> and 10-13 GB/s for a tile with strange repeatable artifacts at >>> around 64 MB and 640 MB. The tile performance is more uniform, but >>> starts quickly degrade at around 900 MB, which is strange as we >>> should have plenty of room in MCDRAM. Also, I am measuring triad, so >>> the cache must be warmed out by the time I started it as triad is >>> one of the last benchmarks in STREAM suite. In fact I do not expect >>> any performance drop between about 8 MB up to 192 GB - the memory >>> bandwidth should be capable to provide single port MCDRAM-available >>> bandwidth at ease. >>> >>> The second graph provides STREAM triad numbers for 8 tiles. Again L2 >>> cache bandwidth is a kind of OK except that it quickly gets down not >>> reaching the expected max of 8 MB combined L2 cache. The MCDRAM >>> bandwidth is within the expected range (with a very strange dip in >>> between 12 - 40 MB. Then, the bandwidth decreases with a very >>> unusual change of curvature at around 2 GB. Even if MCDRAM does not >>> work as cache, I would expect to see around 90 GB/s - I am measuring >>> 4 GB/s at 38 GB. >>> >>> Sounds like Ben have installed CR instead of DDR :) >>> >>> The second graph numbers can slightly be improved with careful >>> mapping the processes to the cores, but not very much. >>> >>> Vitali >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> >> >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0