April 2016 - Intel-nda - mailman3.cels.anl.gov

Re: [JLSE-Intel-NDA] KNL Status
by Kumaran, Kalyan 15 Apr '16

15 Apr '16

Hi Avinash, Thanks for checking on this. We did have a good call with Karthik and Sundaram. Our issues are: * We seem to see DDR bw in cache mode when data is in IPM for NEKbone * We are unable to get repeatable results for SNC-4 and quad modes. Only for all2all. * Our performance for IPM bw is in the 420GB/s range (lower than the 440GB/s achievable only by the above two modes). Our plan of action is to install the bios that was released today. Do not know what this would fix. We also found that we were using large pages and we can turn these off. We will all be in the COE meetings next week in Phoenix. We have a call scheduled with you on Tuesday (26th) and we will share our results then. Thanks and have a good weekend, Kumar > On Apr 15, 2016, at 6:06 PM, Sodani, Avinash <avinash.sodani(a)intel.com> wrote: > > Kumar, > > I've been travelling for last two weeks, so have not been able to follow this closely. I know we had a meeting today 1-2pm pst; I missed that meetng because I was still on road. I'm back now and should be able to pay attention to his starting next week if needed. > > -Avinash > >

1 0

[JLSE-Intel-NDA] STREAM discussion w Intel @3:00pm today
by Kumaran, Kalyan 15 Apr '16

15 Apr '16

We have a conf call with Intel scheduled for 3:00pm this afternoon to go over the STREAM results. We have tried various settings and options and still do not get the result they advertise with bin3. You are welcome to join if you would like. I hope Vitali, Scott and Kevin can attend for sure! Time: 3:00pm central Date: April 15th (Friday) Room: 1172 -Kumar https://meet.intel.com/cm.allison/5L88GT60 Join by phone +1(916)356-2663 Choose bridge 5. Conference ID: 176343430

1 0

Re: [JLSE-Intel-NDA] KNL Status
by Vitali A. Morozov 13 Apr '16

13 Apr '16

I freed the node. It is in SNC-4 mode now - the best MCDRAM bandwidth is 307000 MB/s. No success so far. Vitali On 04/13/2016 02:47 PM, Vitali A. Morozov wrote: > > I guess warm reboot should be avoided. Last time it also hanged. > > > > > > On 04/13/2016 02:46 PM, Allen, Benjamin S. wrote: >> Yea, it hangs on warm reboot. Power cycling it via ipmi now. >> >> Ben >> >>> On Apr 13, 2016, at 2:44 PM, Vitali A. Morozov <morozov(a)anl.gov> wrote: >>> >>> I was using >>> >>> sudo /usr/bin/systemctl reboot >>> >>> and it is booting for over 30 minutes now. Something is not right... >>> >>> Vitali >>> >>> >>> >>> >>> On 04/13/2016 02:05 PM, Vitali A. Morozov wrote: >>>> As no complains have been posted, rebooting to SNC-4/Flat. >>>> >>>> Vitali >>>> >>>> >>>> On 04/13/2016 01:17 PM, Vitali A. Morozov wrote: >>>>> Hello, >>>>> >>>>> With current setup of All2All/Flat, we seem to have a stable 410 >>>>> GB/s from MCDRAM and 78 GB/s from DDR. >>>>> >>>>> For STREAMing from MCDRAM, Intel people suggested to try >>>>> SNC-4/Flat promising 430 GB/s. Our previous trials with other >>>>> combinations of NOC settings lead to unstable results. >>>>> >>>>> Do we want to try it now? It will probably require hard reboot. >>>>> >>>>> Vitali >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On 04/13/2016 01:06 PM, Raman, Karthik wrote: >>>>>> Ok I read this as 418 GB/s J >>>>>> >>>>>> And ok, just realized you are running in All2All mode, so yes >>>>>> this looks good. For Quad/Flat you should see ~440 GB/s with the >>>>>> same run. >>>>>> >>>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>>> Sent: Wednesday, April 13, 2016 11:02 AM >>>>>> To: Raman, Karthik <karthik.raman(a)intel.com>; Chinthamani, >>>>>> Sundaram <sundaram.chinthamani(a)intel.com>; Kumaran, Kalyan >>>>>> <kumaran(a)alcf.anl.gov> >>>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; Fromkin, Russ >>>>>> <russ.fromkin(a)intel.com>; Harms, Kevin <harms(a)alcf.anl.gov>; >>>>>> Morozov, Vitali A. <morozov(a)alcf.anl.gov>; Allen, Benjamin S. >>>>>> <bsallen(a)alcf.anl.gov> >>>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>>> >>>>>> Hi Karthik, >>>>>> >>>>>> Repeating the same options you have: >>>>>> >>>>>> mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp -mcmodel >>>>>> medium -shared-intel -opt-streaming-stores always stream.c -o stream >>>>>> >>>>>> ------------------------------------------------------------- >>>>>> STREAM version $Revision: 5.9 $ >>>>>> ------------------------------------------------------------- >>>>>> This system uses 8 bytes per DOUBLE PRECISION word. >>>>>> ------------------------------------------------------------- >>>>>> Array size = 134217728, Offset = 0 >>>>>> Total memory required = 3072.0 MB. >>>>>> Each test is run 100 times, but only >>>>>> the *best* time for each is used. >>>>>> ------------------------------------------------------------- >>>>>> Number of Threads requested = 64 >>>>>> ------------------------------------------------------------- >>>>>> >>>>>> < skiped > >>>>>> >>>>>> Function Rate (MB/s) Avg time Min time Max time >>>>>> Copy: 362331.5200 0.0060 0.0059 0.0061 >>>>>> Scale: 359281.9806 0.0060 0.0060 0.0062 >>>>>> Add: 418122.7024 0.0078 0.0077 0.0081 >>>>>> Triad: 418122.7024 0.0078 0.0077 0.0081 >>>>>> Store: 224685.6729 0.0048 0.0048 0.0049 >>>>>> >>>>>> >>>>>> 408GB/s >>>>>> >>>>>> Vitali >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Intel-nda mailing list >>>>> >>>>> Intel-nda(a)lists.jlse.anl.gov >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] KNL Status
by Vitali A. Morozov 13 Apr '16

13 Apr '16

I guess warm reboot should be avoided. Last time it also hanged. On 04/13/2016 02:46 PM, Allen, Benjamin S. wrote: > Yea, it hangs on warm reboot. Power cycling it via ipmi now. > > Ben > >> On Apr 13, 2016, at 2:44 PM, Vitali A. Morozov <morozov(a)anl.gov> wrote: >> >> I was using >> >> sudo /usr/bin/systemctl reboot >> >> and it is booting for over 30 minutes now. Something is not right... >> >> Vitali >> >> >> >> >> On 04/13/2016 02:05 PM, Vitali A. Morozov wrote: >>> As no complains have been posted, rebooting to SNC-4/Flat. >>> >>> Vitali >>> >>> >>> On 04/13/2016 01:17 PM, Vitali A. Morozov wrote: >>>> Hello, >>>> >>>> With current setup of All2All/Flat, we seem to have a stable 410 GB/s from MCDRAM and 78 GB/s from DDR. >>>> >>>> For STREAMing from MCDRAM, Intel people suggested to try SNC-4/Flat promising 430 GB/s. Our previous trials with other combinations of NOC settings lead to unstable results. >>>> >>>> Do we want to try it now? It will probably require hard reboot. >>>> >>>> Vitali >>>> >>>> >>>> >>>> >>>> >>>> On 04/13/2016 01:06 PM, Raman, Karthik wrote: >>>>> Ok I read this as 418 GB/s J >>>>> >>>>> And ok, just realized you are running in All2All mode, so yes this looks good. For Quad/Flat you should see ~440 GB/s with the same run. >>>>> >>>>> From: Vitali A. Morozov [mailto:[email protected]] >>>>> Sent: Wednesday, April 13, 2016 11:02 AM >>>>> To: Raman, Karthik <karthik.raman(a)intel.com>; Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov> >>>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; Fromkin, Russ <russ.fromkin(a)intel.com>; Harms, Kevin <harms(a)alcf.anl.gov>; Morozov, Vitali A. <morozov(a)alcf.anl.gov>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >>>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>>> >>>>> Hi Karthik, >>>>> >>>>> Repeating the same options you have: >>>>> >>>>> mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp -mcmodel medium -shared-intel -opt-streaming-stores always stream.c -o stream >>>>> >>>>> ------------------------------------------------------------- >>>>> STREAM version $Revision: 5.9 $ >>>>> ------------------------------------------------------------- >>>>> This system uses 8 bytes per DOUBLE PRECISION word. >>>>> ------------------------------------------------------------- >>>>> Array size = 134217728, Offset = 0 >>>>> Total memory required = 3072.0 MB. >>>>> Each test is run 100 times, but only >>>>> the *best* time for each is used. >>>>> ------------------------------------------------------------- >>>>> Number of Threads requested = 64 >>>>> ------------------------------------------------------------- >>>>> >>>>> < skiped > >>>>> >>>>> Function Rate (MB/s) Avg time Min time Max time >>>>> Copy: 362331.5200 0.0060 0.0059 0.0061 >>>>> Scale: 359281.9806 0.0060 0.0060 0.0062 >>>>> Add: 418122.7024 0.0078 0.0077 0.0081 >>>>> Triad: 418122.7024 0.0078 0.0077 0.0081 >>>>> Store: 224685.6729 0.0048 0.0048 0.0049 >>>>> >>>>> >>>>> 408GB/s >>>>> >>>>> Vitali >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Intel-nda mailing list >>>> >>>> Intel-nda(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] KNL Status
by Allen, Benjamin S. 13 Apr '16

13 Apr '16

Yea, it hangs on warm reboot. Power cycling it via ipmi now. Ben > On Apr 13, 2016, at 2:44 PM, Vitali A. Morozov <morozov(a)anl.gov> wrote: > > I was using > > sudo /usr/bin/systemctl reboot > > and it is booting for over 30 minutes now. Something is not right... > > Vitali > > > > > On 04/13/2016 02:05 PM, Vitali A. Morozov wrote: >> >> As no complains have been posted, rebooting to SNC-4/Flat. >> >> Vitali >> >> >> On 04/13/2016 01:17 PM, Vitali A. Morozov wrote: >>> Hello, >>> >>> With current setup of All2All/Flat, we seem to have a stable 410 GB/s from MCDRAM and 78 GB/s from DDR. >>> >>> For STREAMing from MCDRAM, Intel people suggested to try SNC-4/Flat promising 430 GB/s. Our previous trials with other combinations of NOC settings lead to unstable results. >>> >>> Do we want to try it now? It will probably require hard reboot. >>> >>> Vitali >>> >>> >>> >>> >>> >>> On 04/13/2016 01:06 PM, Raman, Karthik wrote: >>>> Ok I read this as 418 GB/s J >>>> >>>> And ok, just realized you are running in All2All mode, so yes this looks good. For Quad/Flat you should see ~440 GB/s with the same run. >>>> >>>> From: Vitali A. Morozov [mailto:[email protected]] >>>> Sent: Wednesday, April 13, 2016 11:02 AM >>>> To: Raman, Karthik <karthik.raman(a)intel.com>; Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov> >>>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com>; Fromkin, Russ <russ.fromkin(a)intel.com>; Harms, Kevin <harms(a)alcf.anl.gov>; Morozov, Vitali A. <morozov(a)alcf.anl.gov>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >>>> Subject: Re: [JLSE-Intel-NDA] KNL Status >>>> >>>> Hi Karthik, >>>> >>>> Repeating the same options you have: >>>> >>>> mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp -mcmodel medium -shared-intel -opt-streaming-stores always stream.c -o stream >>>> >>>> ------------------------------------------------------------- >>>> STREAM version $Revision: 5.9 $ >>>> ------------------------------------------------------------- >>>> This system uses 8 bytes per DOUBLE PRECISION word. >>>> ------------------------------------------------------------- >>>> Array size = 134217728, Offset = 0 >>>> Total memory required = 3072.0 MB. >>>> Each test is run 100 times, but only >>>> the *best* time for each is used. >>>> ------------------------------------------------------------- >>>> Number of Threads requested = 64 >>>> ------------------------------------------------------------- >>>> >>>> < skiped > >>>> >>>> Function Rate (MB/s) Avg time Min time Max time >>>> Copy: 362331.5200 0.0060 0.0059 0.0061 >>>> Scale: 359281.9806 0.0060 0.0060 0.0062 >>>> Add: 418122.7024 0.0078 0.0077 0.0081 >>>> Triad: 418122.7024 0.0078 0.0077 0.0081 >>>> Store: 224685.6729 0.0048 0.0048 0.0049 >>>> >>>> >>>> 408GB/s >>>> >>>> Vitali >>>> >>> >>> >>> >>> _______________________________________________ >>> Intel-nda mailing list >>> >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] KNL Status
by Vitali A. Morozov 13 Apr '16

13 Apr '16

I was using sudo /usr/bin/systemctl reboot and it is booting for over 30 minutes now. Something is not right... Vitali On 04/13/2016 02:05 PM, Vitali A. Morozov wrote: > > As no complains have been posted, rebooting to SNC-4/Flat. > > Vitali > > > On 04/13/2016 01:17 PM, Vitali A. Morozov wrote: >> Hello, >> >> With current setup of All2All/Flat, we seem to have a stable 410 GB/s >> from MCDRAM and 78 GB/s from DDR. >> >> For STREAMing from MCDRAM, Intel people suggested to try SNC-4/Flat >> promising 430 GB/s. Our previous trials with other combinations of >> NOC settings lead to unstable results. >> >> Do we want to try it now? It will probably require hard reboot. >> >> Vitali >> >> >> >> >> >> On 04/13/2016 01:06 PM, Raman, Karthik wrote: >>> >>> Ok I read this as 418 GB/s J >>> >>> And ok, just realized you are running in All2All mode, so yes this >>> looks good. For Quad/Flat you should see ~440 GB/s with the same run. >>> >>> *From:*Vitali A. Morozov [mailto:[email protected]] >>> *Sent:* Wednesday, April 13, 2016 11:02 AM >>> *To:* Raman, Karthik <karthik.raman(a)intel.com>; Chinthamani, >>> Sundaram <sundaram.chinthamani(a)intel.com>; Kumaran, Kalyan >>> <kumaran(a)alcf.anl.gov> >>> *Cc:* Sodani, Avinash <avinash.sodani(a)intel.com>; Fromkin, Russ >>> <russ.fromkin(a)intel.com>; Harms, Kevin <harms(a)alcf.anl.gov>; >>> Morozov, Vitali A. <morozov(a)alcf.anl.gov>; Allen, Benjamin S. >>> <bsallen(a)alcf.anl.gov> >>> *Subject:* Re: [JLSE-Intel-NDA] KNL Status >>> >>> Hi Karthik, >>> >>> Repeating the same options you have: >>> >>> mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp -mcmodel >>> medium -shared-intel -opt-streaming-stores always stream.c -o stream >>> >>> ------------------------------------------------------------- >>> STREAM version $Revision: 5.9 $ >>> ------------------------------------------------------------- >>> This system uses 8 bytes per DOUBLE PRECISION word. >>> ------------------------------------------------------------- >>> Array size = 134217728, Offset = 0 >>> Total memory required = 3072.0 MB. >>> Each test is run 100 times, but only >>> the *best* time for each is used. >>> ------------------------------------------------------------- >>> Number of Threads requested = 64 >>> ------------------------------------------------------------- >>> >>> < skiped > >>> >>> Function Rate (MB/s) Avg time Min time Max time >>> Copy: 362331.5200 0.0060 0.0059 0.0061 >>> Scale: 359281.9806 0.0060 0.0060 0.0062 >>> Add: 418122.7024 0.0078 0.0077 0.0081 >>> Triad: 418122.7024 0.0078 0.0077 0.0081 >>> Store: 224685.6729 0.0048 0.0048 0.0049 >>> >>> >>> 408GB/s >>> >>> Vitali >>> >> >> >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] KNL Status
by Vitali A. Morozov 13 Apr '16

13 Apr '16

As no complains have been posted, rebooting to SNC-4/Flat. Vitali On 04/13/2016 01:17 PM, Vitali A. Morozov wrote: > Hello, > > With current setup of All2All/Flat, we seem to have a stable 410 GB/s > from MCDRAM and 78 GB/s from DDR. > > For STREAMing from MCDRAM, Intel people suggested to try SNC-4/Flat > promising 430 GB/s. Our previous trials with other combinations of NOC > settings lead to unstable results. > > Do we want to try it now? It will probably require hard reboot. > > Vitali > > > > > > On 04/13/2016 01:06 PM, Raman, Karthik wrote: >> >> Ok I read this as 418 GB/s J >> >> And ok, just realized you are running in All2All mode, so yes this >> looks good. For Quad/Flat you should see ~440 GB/s with the same run. >> >> *From:*Vitali A. Morozov [mailto:[email protected]] >> *Sent:* Wednesday, April 13, 2016 11:02 AM >> *To:* Raman, Karthik <karthik.raman(a)intel.com>; Chinthamani, Sundaram >> <sundaram.chinthamani(a)intel.com>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov> >> *Cc:* Sodani, Avinash <avinash.sodani(a)intel.com>; Fromkin, Russ >> <russ.fromkin(a)intel.com>; Harms, Kevin <harms(a)alcf.anl.gov>; Morozov, >> Vitali A. <morozov(a)alcf.anl.gov>; Allen, Benjamin S. >> <bsallen(a)alcf.anl.gov> >> *Subject:* Re: [JLSE-Intel-NDA] KNL Status >> >> Hi Karthik, >> >> Repeating the same options you have: >> >> mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp -mcmodel >> medium -shared-intel -opt-streaming-stores always stream.c -o stream >> >> ------------------------------------------------------------- >> STREAM version $Revision: 5.9 $ >> ------------------------------------------------------------- >> This system uses 8 bytes per DOUBLE PRECISION word. >> ------------------------------------------------------------- >> Array size = 134217728, Offset = 0 >> Total memory required = 3072.0 MB. >> Each test is run 100 times, but only >> the *best* time for each is used. >> ------------------------------------------------------------- >> Number of Threads requested = 64 >> ------------------------------------------------------------- >> >> < skiped > >> >> Function Rate (MB/s) Avg time Min time Max time >> Copy: 362331.5200 0.0060 0.0059 0.0061 >> Scale: 359281.9806 0.0060 0.0060 0.0062 >> Add: 418122.7024 0.0078 0.0077 0.0081 >> Triad: 418122.7024 0.0078 0.0077 0.0081 >> Store: 224685.6729 0.0048 0.0048 0.0049 >> >> >> 408GB/s >> >> Vitali >> > > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] KNL Status
by Vitali A. Morozov 13 Apr '16

13 Apr '16

Hello, With current setup of All2All/Flat, we seem to have a stable 410 GB/s from MCDRAM and 78 GB/s from DDR. For STREAMing from MCDRAM, Intel people suggested to try SNC-4/Flat promising 430 GB/s. Our previous trials with other combinations of NOC settings lead to unstable results. Do we want to try it now? It will probably require hard reboot. Vitali On 04/13/2016 01:06 PM, Raman, Karthik wrote: > > Ok I read this as 418 GB/s J > > And ok, just realized you are running in All2All mode, so yes this > looks good. For Quad/Flat you should see ~440 GB/s with the same run. > > *From:*Vitali A. Morozov [mailto:[email protected]] > *Sent:* Wednesday, April 13, 2016 11:02 AM > *To:* Raman, Karthik <karthik.raman(a)intel.com>; Chinthamani, Sundaram > <sundaram.chinthamani(a)intel.com>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov> > *Cc:* Sodani, Avinash <avinash.sodani(a)intel.com>; Fromkin, Russ > <russ.fromkin(a)intel.com>; Harms, Kevin <harms(a)alcf.anl.gov>; Morozov, > Vitali A. <morozov(a)alcf.anl.gov>; Allen, Benjamin S. > <bsallen(a)alcf.anl.gov> > *Subject:* Re: [JLSE-Intel-NDA] KNL Status > > Hi Karthik, > > Repeating the same options you have: > > mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp -mcmodel > medium -shared-intel -opt-streaming-stores always stream.c -o stream > > ------------------------------------------------------------- > STREAM version $Revision: 5.9 $ > ------------------------------------------------------------- > This system uses 8 bytes per DOUBLE PRECISION word. > ------------------------------------------------------------- > Array size = 134217728, Offset = 0 > Total memory required = 3072.0 MB. > Each test is run 100 times, but only > the *best* time for each is used. > ------------------------------------------------------------- > Number of Threads requested = 64 > ------------------------------------------------------------- > > < skiped > > > Function Rate (MB/s) Avg time Min time Max time > Copy: 362331.5200 0.0060 0.0059 0.0061 > Scale: 359281.9806 0.0060 0.0060 0.0062 > Add: 418122.7024 0.0078 0.0077 0.0081 > Triad: 418122.7024 0.0078 0.0077 0.0081 > Store: 224685.6729 0.0048 0.0048 0.0049 > > > 408GB/s > > Vitali >

1 0

[JLSE-Intel-NDA] Fwd: KNL Status
by Kumaran, Kalyan 13 Apr '16

13 Apr '16

Keeping the list abreast. Short summary - DDR bw of 80GB/sec is what you see for bin3 parts as memory runs at a lower clock compared to bin1. IPM bw is 420 GB/sec for 64 cores. The 470 or 480 numbers that Intel floats around is for 72 cores A0 part. Let us see what Intel has to say. -Kumar Begin forwarded message: From: "Vitali A. Morozov" <morozov(a)anl.gov<mailto:[email protected]>> Subject: Re: [JLSE-Intel-NDA] KNL Status Date: April 13, 2016 at 12:00:36 PM CDT To: "Raman, Karthik" <karthik.raman(a)intel.com<mailto:[email protected]>>, "Chinthamani, Sundaram" <sundaram.chinthamani(a)intel.com<mailto:[email protected]>>, "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov<mailto:[email protected]>> Cc: "Sodani, Avinash" <avinash.sodani(a)intel.com<mailto:[email protected]>>, "Fromkin, Russ" <russ.fromkin(a)intel.com<mailto:[email protected]>>, "Harms, Kevin" <harms(a)alcf.anl.gov<mailto:[email protected]>>, "Morozov, Vitali A." <morozov(a)alcf.anl.gov<mailto:[email protected]>>, "Allen, Benjamin S." <bsallen(a)alcf.anl.gov<mailto:[email protected]>> Hello, so, this is my understanding that on bin3 hardware (64 cores, 6 channel DDR4-2133) 1. Effective DDR bandwidth is 80 GB/s (around 78% of peak). 2. Effective MCDRAM bandwidth is around 406-408 GB/s. Larger values are possible for more high core count bins. 3. Alignment is not important for STREAM triad benchmark. 4. For both static and dynamic variables, numactl -m [0|1] should be used to direct memory placement. 5. For explicit MCDRAM allocation in flat mode, hbw_malloc.h API should be use. Please correct if I am wrong... Thanks a lot for you help! Vitali On 04/12/2016 11:07 AM, Vitali A. Morozov wrote: Hi Karthik, I was also wandering how the variable could be influencing the allocation if, in principle, I do not need to link against memkind in my case. Anyway, sorry for confusion! I think I did that before - see my first message. I was using "numactl --membind=1 ./stream" , which I believe is equivalent to "numactl -m 1 ./stream" that you suggest. Here is the result of "numactl -H": [morozov@knl00 512_3_C64_MPI16_TH08_flat]$ numactl -H available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 node 0 size: 196511 MB node 0 free: 186888 MB node 1 cpus: node 1 size: 16384 MB node 1 free: 11731 MB node distances: node 0 1 0: 10 31 1: 31 10 Sounds like all as expected. [morozov@knl00 STREAM]$ OMP_NUM_THREADS=64 KMP_AFFINITY=scatter numactl -m 1 ./stream Triad: 418262.7498 0.0058 0.0057 0.0060 For a series of runs, I see this: # MB N GB/s 228.88 10000000 408867.98 457.76 20000000 405818.57 686.65 30000000 414309.08 915.53 40000000 415063.59 1144.41 50000000 415792.22 1373.29 60000000 415449.01 1602.17 70000000 414106.18 1831.05 80000000 416846.82 2059.94 90000000 414992.29 2288.82 100000000 417898.11 Vitali On 04/12/2016 10:32 AM, Raman, Karthik wrote: Hi Vitali, 1. The MEMKIND_HBW_NODES is *only* when you are allocating using hbwalloc (+ MCDRAM flat) in your application and when memkind is unable to find the high bandwidth nodes (pmtt not populated) 2. With the below static allocations, in MCDRAM Flat mode, you can simply use “numactl –m 1 ./stream” with propery affinity settings to get peak out of MCDRAM. a. Check “numactl –H” to find the NUMA domains in your node. If booted in All2All/FLAT or Quad/Flat mode it should show 2 NUMA domains , one for DDR (96 GB) and other for MCDRAM (16GB) Thanks Karthik From: Vitali A. Morozov [mailto:[email protected]] Sent: Tuesday, April 12, 2016 7:27 AM To: Raman, Karthik <karthik.raman(a)intel.com><mailto:[email protected]>; Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com><mailto:[email protected]>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov><mailto:[email protected]> Cc: Sodani, Avinash <avinash.sodani(a)intel.com><mailto:[email protected]>; Fromkin, Russ <russ.fromkin(a)intel.com><mailto:[email protected]>; Harms, Kevin <harms(a)alcf.anl.gov><mailto:[email protected]>; Morozov, Vitali A. <morozov(a)alcf.anl.gov><mailto:[email protected]>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov><mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] KNL Status Hi Karthik, The node is in All2All mode for now. I made the following changes. The original code is using static allocation therefore, I follow the advice on 2MB static array alignment: [morozov@thing00 STREAM]$ head -n 108 < stream.c | tail -n 4 static double a[N+OFFSET] __attribute__((aligned(2097152))), b[N+OFFSET] __attribute__((aligned(2097152))), c[N+OFFSET] __attribute__((aligned(2097152))), d[N+OFFSET] __attribute__((aligned(2097152))); [morozov@knl00 STREAM]$ make mpiicc -O3 -xMIC-AVX512 -opt-report=5 -Wall -g -fopenmp -mcmodel medium -shared-intel -fno-alias stream.c -o stream [morozov@knl00 STREAM]$ MEMKIND_HBW_NODES=1 OMP_NUM_THREADS=64 KMP_AFFINITY=scatter ./stream ..... Triad: 80251.3621 0.0300 0.0299 0.0305 Apparently, the "MEMKIND_HBW_NODES=1" is ignored. Thanks! Vitali On 04/12/2016 01:51 AM, Raman, Karthik wrote: Also could you please modify the memory allocations in STREAM to align at 2MB boundary. For example: static allocations: static STREAM_TYPE a[STREAM_ARRAY_SIZE+OFFSET] __attribute__((aligned(2097152))) dynamic allocations: a = (STREAM_TYPE *)_mm_malloc(sizeof(STREAM_TYPE)*(STREAM_ARRAY_SIZE+OFFSET),2097152); memkind/hbwalloc: (STREAM_TYPE *)hbw_posix_memalign((void **)&a, 2097152, sizeof(STREAM_TYPE)*(STREAM_ARRAY_SIZE+OFFSET)) For memkind/hbwmalloc to work out of the box, you need to have the PMTT tables populated so that it can identify the high bandwidth nodes. Alternatively you can set the MEMKIND_HBW_NODES environment variable to the high bandwidth node (MCDRAM). e.g. All2All/Flat or Quad/Flat : export MEMKIND_HBW_NODES=1 ; ./app_with_hbwmalloc SNC4/Flat: export MEMKIND_HBW_NODES=4,5,6,7 ; ./app_with_hbw_malloc Thanks Karthik From: Chinthamani, Sundaram Sent: Monday, April 11, 2016 8:15 PM To: Vitali Morozov <morozov(a)anl.gov><mailto:[email protected]>; Kumaran, Kalyan <kumaran(a)alcf.anl.gov><mailto:[email protected]> Cc: Sodani, Avinash <avinash.sodani(a)intel.com><mailto:[email protected]>; Fromkin, Russ <russ.fromkin(a)intel.com><mailto:[email protected]>; Raman, Karthik <karthik.raman(a)intel.com><mailto:[email protected]>; Harms, Kevin <harms(a)alcf.anl.gov><mailto:[email protected]>; Morozov, Vitali A. <morozov(a)alcf.anl.gov><mailto:[email protected]>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov><mailto:[email protected]>; Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com><mailto:[email protected]> Subject: RE: [JLSE-Intel-NDA] KNL Status You should use numactl –m 4,5,6,7 to get the memory bandwidth out of MC-DRAM. What kind of part is it? Is it bin1 or bin3? Even in All2All mode, we have measured up to 470 GB/s of stream triad bandwidth. Quad mode should get 490 GB/s. DDR-stream triad bandwidth on Bin1-part should be close to 90 GB/s. -Sundaram From: Vitali Morozov [mailto:[email protected]] Sent: Monday, April 11, 2016 6:59 PM To: Kumaran, Kalyan <kumaran(a)alcf.anl.gov<mailto:[email protected]>> Cc: Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>>; Fromkin, Russ <russ.fromkin(a)intel.com<mailto:[email protected]>>; Chinthamani, Sundaram <sundaram.chinthamani(a)intel.com<mailto:[email protected]>>; Raman, Karthik <karthik.raman(a)intel.com<mailto:[email protected]>>; Harms, Kevin <harms(a)alcf.anl.gov<mailto:[email protected]>>; Morozov, Vitali A. <morozov(a)alcf.anl.gov<mailto:[email protected]>>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov<mailto:[email protected]>> Subject: Re: [JLSE-Intel-NDA] KNL Status Avinash and others, here is what we have so far for Flat more, All2All. The results from memory were obtained by using default STREAM with static allocation. The results from MCDRAM were obtained by using modified STREAM with hbwmalloc.h interface. 1 MPI process, 64 threads, scatter affinity, no numactl is used. Most results are as expected: 80 GB/s from DDR memory, 420 GB/s from MCDRAM, the MCDRAM size is about 16 GB, the aggregate L2 cache size is 32 MB. Noticed inconsistencies for hbwmalloc benchmarks: 1. The result is not validated - will spend more time on that tomorrow. 2. hbw_malloc continues allocating data beyond the expected 16 GB up to 48 GB. This means it fails back to DDR if it cannot allocate in MCDRAM - not expected default behavior. It must fail if it does not have enough space. The problem shows up when I am running static allocation and use numactl --membind=1 to ask for MCDRAM-only allocation. The variation is very large and inconsistent with the result observed when using hbw_malloc API: # MB N GB/s numactl GB/s hbw_malloc 228.88 10000000 255555.46 421009.18 457.76 20000000 363930.93 416997.91 686.65 30000000 309415.87 427808.31 915.53 40000000 371691.30 423310.75 1144.41 50000000 187002.22 431032.35 1373.29 60000000 385954.23 427172.91 1602.17 70000000 388254.49 426823.59 1831.05 80000000 374491.43 428103.96 2059.94 90000000 379622.74 428152.02 2288.82 100000000 80868.34 430185.03 We also tried SNC-4 instead of All2All, but could not get more than 40 GB/s from MCDRAM even with hbw_malloc API. Raw data: # flat memory, running from memory (default), alltoall # MB N GB/s 2.29 100000 272062.96 4.58 200000 491040.47 6.87 300000 603979.78 9.16 400000 745654.04 11.44 500000 811800.77 13.73 600000 901462.35 16.02 700000 891953.26 18.31 800000 1019375.15 20.60 900000 984749.63 22.89 1000000 923516.48 25.18 1100000 758422.09 27.47 1200000 779328.74 29.75 1300000 513185.43 32.04 1400000 155550.35 34.33 1500000 103209.12 36.62 1600000 97435.74 38.91 1700000 83968.40 41.20 1800000 83692.35 43.49 1900000 80734.60 45.78 2000000 76317.89 68.66 3000000 81180.08 91.55 4000000 78889.73 114.44 5000000 79891.50 137.33 6000000 79954.96 160.22 7000000 79477.00 183.11 8000000 79145.59 205.99 9000000 79821.12 228.88 10000000 79519.15 228.88 10000000 79212.54 457.76 20000000 79692.27 686.65 30000000 79866.15 915.53 40000000 79828.15 1144.41 50000000 80214.91 1373.29 60000000 80226.84 1602.17 70000000 80178.77 1831.05 80000000 80177.06 2059.94 90000000 80109.79 2288.82 100000000 80175.94 # flat memory, running from MCDRAM, alltoall, hbwmalloc API # MB N GB/s 228.88 10000000 421009.18 457.76 20000000 416997.91 686.65 30000000 427808.31 915.53 40000000 423310.75 1144.41 50000000 431032.35 1373.29 60000000 427172.91 1602.17 70000000 426823.59 1831.05 80000000 428103.96 2059.94 90000000 428152.02 2288.82 100000000 430185.03 2343.75 102400000 425999.98 4687.50 204800000 427821.10 9155.27 400000000 429318.13 9375.00 409600000 428376.70 11444.09 500000000 428052.08 13732.91 600000000 266061.01 16021.73 700000000 200257.79 18310.55 800000000 168164.19 20599.37 900000000 150192.91 22888.18 1000000000 137862.07 25177.00 1100000000 129418.12 27465.82 1200000000 123019.53 29754.64 1300000000 118268.79 32043.46 1400000000 114343.33 34332.28 1500000000 110991.25 36621.09 1600000000 108538.36 38909.91 1700000000 106249.72 41198.73 1800000000 104455.07 43487.55 1900000000 102828.93 45776.37 2000000000 101348.56 48065.19 2100000000 100013.73 We appreciate any help on that. Best, Vitali On 04/11/2016 08:42 PM, Kumaran, Kalyan wrote: Avinash, thanks for the prompt reply. Kevin/Vitali can provide all the details. The short summary is we are unable to reproduce the STREAM results from IPM or DDR. It would be great if Karthik or Sundaram can advise us on how to run (which mode etc) or if they could log into the machine and make sure we have set up things correctly. Thanks, Kumar On Apr 11, 2016, at 8:07 PM, Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>> wrote: Copying Sundaram/Karthik from our side. Kumar, Can you send us more information on this? -Avinash From: Kumaran, Kalyan [mailto:[email protected]] Sent: Monday, April 11, 2016 5:32 PM To: Fromkin, Russ; Sodani, Avinash Cc: Harms, Kevin; Morozov, Vitali A. Subject: Fwd: [JLSE-Intel-NDA] KNL Status Russ, Avinash, we need help in understanding how to obtain STREAM results. Can anyone help? Thanks, Kumar Begin forwarded message: From: "Harms, Kevin" <harms(a)alcf.anl.gov<mailto:[email protected]>> Subject: [JLSE-Intel-NDA] KNL Status Date: April 11, 2016 at 5:42:15 PM CDT To: "intel-nda(a)jlse.anl.gov<mailto:[email protected]>" <intel-nda(a)jlse.anl.gov<mailto:[email protected]>> The knl00 node is currently in Flat mode and All2All model. With this setup we can get 420 GB/s from IPM memory on STREAM Triad when using hbw_malloc and friends which was confirmed by both Vitali and I. Changing the BIOS settings and warm rebooting seem to be problematic. Vitali and I were running earlier and could not get over 80 GB/s with any method even though memory capacity indicated flat mode. Using numactl --membind is also problematic. The STREAM results when using this to launch the application were erratic, so it is also suggested not to use that. That means you will need to modify your code to use dynamic memory and hbw_malloc to exploit the IPM. Will followup with more detailed results later. kevin _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] Kernel BUG!
by Vitali A. Morozov 12 Apr '16

12 Apr '16

Message from syslogd@localhost at Apr 12 16:55:33 ... kernel:BUG: soft lockup - CPU#232 stuck for 22s! [crond:36140] Message from syslogd@localhost at Apr 12 16:55:33 ... kernel:BUG: soft lockup - CPU#233 stuck for 22s! [sendmail:36148] Message from syslogd@localhost at Apr 12 16:55:33 ... kernel:BUG: soft lockup - CPU#237 stuck for 22s! [sh:36145] Message from syslogd@localhost at Apr 12 16:55:33 ... kernel:BUG: soft lockup - CPU#4 stuck for 22s! [crond:36141] Message from syslogd@localhost at Apr 12 16:55:33 ... kernel:BUG: soft lockup - CPU#40 stuck for 23s! [sendmail:36144]

1 0