May 2016 - Intel-nda - mailman3.cels.anl.gov

Re: [JLSE-Intel-NDA] LAMMPS and QBox on KNL
by Harms, Kevin 16 May '16

16 May '16

Chris, for flat+quadrant mode, did you try running with 'mpirun numactl -m 1 ./app.exe' ? kevin > > > >Hello, > > >I placed some updated slides on the KNL confluence page. The pdf now includes additional comparisons and some timings for the WEST code used in one of the ESP projects. > > >I noticed that “cache+quadrant” mode can in some cases yield ~2x speedups in runtime for the planewave DFT codes compared to “flat+quadrant” with most of the runtime difference coming from the 3D FFT related kernels. Maybe explicitly allocating > memory to HBM will recover performance in “flat+quadrant” mode? > > >chris > > > > > > > >On May 9, 2016, at 10:16 AM, Christopher Knight <knightc(a)anl.gov> wrote: > >Hello, > >Here’s quick summary of what I’ve seen so far with LAMMPS and QBox on KNL. WEST code is work in progress. > > >For the LAMMPS rhodo benchmark, I see core-to-core speedup of 3.39x on KNL and a node-to-node speedup of 5.40x using 64 MPI ranks and 2 OpenMP threads on KNL. Using Intel optimized code on KNL (can’t run on BG/Q), I see core-to-core speedup of 10.49x and node-to-node > speedup of 12.47x. > >For the LAMMPS reaxc benchmark using multi-threaded package, I see core-to-core speedup of 4.05x on KNL and a node-to-node speedup of 5.19x using 16 MPI ranks and 8 OpenMP threads on KNL. > > >For QBox, I’m using inputs related to an ESP project. I see core-to-core speedups in the range of 2.5 – 3.5x on KNL and a node-to-node speedups in range of 3.8 – 4.5x depending on model, system, and compilation flags. Runtimes sensitive to FFT interface selected > for MKL. > > >chris > > > > > > > > > >_______________________________________________ >Intel-nda mailing list >Intel-nda(a)lists.jlse.anl.gov >https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > > ><knl_LAMMPS_QBOX_May2016.pdf> > > >

1 0

Re: [JLSE-Intel-NDA] LAMMPS and QBox on KNL
by Christopher Knight 16 May '16

16 May '16

Hello, I placed some updated slides on the KNL confluence page. The pdf now includes additional comparisons and some timings for the WEST code used in one of the ESP projects. I noticed that “cache+quadrant” mode can in some cases yield ~2x speedups in runtime for the planewave DFT codes compared to “flat+quadrant” with most of the runtime difference coming from the 3D FFT related kernels. Maybe explicitly allocating memory to HBM will recover performance in “flat+quadrant” mode? chris On May 9, 2016, at 10:16 AM, Christopher Knight <knightc(a)anl.gov> wrote: Hello, Here’s quick summary of what I’ve seen so far with LAMMPS and QBox on KNL. WEST code is work in progress. For the LAMMPS rhodo benchmark, I see core-to-core speedup of 3.39x on KNL and a node-to-node speedup of 5.40x using 64 MPI ranks and 2 OpenMP threads on KNL. Using Intel optimized code on KNL (can’t run on BG/Q), I see core-to-core speedup of 10.49x and node-to-node speedup of 12.47x. For the LAMMPS reaxc benchmark using multi-threaded package, I see core-to-core speedup of 4.05x on KNL and a node-to-node speedup of 5.19x using 16 MPI ranks and 8 OpenMP threads on KNL. For QBox, I’m using inputs related to an ESP project. I see core-to-core speedups in the range of 2.5 – 3.5x on KNL and a node-to-node speedups in range of 3.8 – 4.5x depending on model, system, and compilation flags. Runtimes sensitive to FFT interface selected for MKL. chris _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov https://lists.jlse.anl.gov/mailman/listinfo/intel-nda <https://lists.jlse.anl.gov/mailman/listinfo/intel-nda> <knl_LAMMPS_QBOX_May2016.pdf>

1 0

Re: [JLSE-Intel-NDA] KNL User Guide
by Ray Loy 12 May '16

12 May '16

thanks that solved it for me too. it anyone has trouble finding it, it's under JLSE Account and Identity Management Account Information Assigned Resources > On May 10, 2016, at 10:22 AM, Christopher Knight <knightc(a)anl.gov> wrote: > > Looks like list was dropped. I'm able to login after requesting access to confluence from my jlse accounts page. > > chris > > > On May 10, 2016, at 9:38 AM, Harms, Kevin <harms(a)alcf.anl.gov> wrote: > > > you need to login to the accounts.jlse page and then select the confluence resource. > > kevin > > > > >> Hi Kevin, >> >> Tried viewing the page with jlse user/passwd and it returned an “operation not permitted” message. >> >> Does anyone else see similar message or has succeeded in viewing page? >> >> chris >> >> >> >> On May 9, 2016, at 4:27 PM, Harms, Kevin <harms(a)alcf.anl.gov> wrote: >> >> >> Here's a user guide for the KNL node. >> >> https://wiki.jlse.anl.gov/display/knl/Intel+KNL >> >> I believe anyone with Confluence access and in the 'intel' group should be able to edit it. Please make any corrections as needed. >> >> kevin >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] KNL IEEE Journal Article
by Kumaran, Kalyan 12 May '16

12 May '16

Thanks, Hal! I downloaded and added to our KNL wikipage. -Kumar > On May 12, 2016, at 11:16 AM, Hal Finkel <hfinkel(a)anl.gov> wrote: > > Hi Everyone, > > In case you didn't know... > > The IEEE Micro article describing the KNL microarchiecture has been published: > > Knights Landing: Second-Generation Intel Xeon Phi Product > Avinash Sodani, et al. > > http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7453080 > > There is a lot of good information, diagrams, etc. in there. > > -Hal > > -- > Hal Finkel > Assistant Computational Scientist > Leadership Computing Facility > Argonne National Laboratory

1 0

Re: [JLSE-Intel-NDA] Action items from today's Intel call
by Scott Parker 11 May '16

11 May '16

Trying again, hopefully the attachment won't get filtered this time. On 5/11/16 11:32 AM, Scott Parker wrote: > > Attached are the latest stream results for SNC-4 mode along with the > source and make file used. The node was powered > down and powered back up into flat/snc-4 mode. The results are the > same as before - numa domain 4 shows significantly > less bandwidth than the other numa domains. > > -Scott > > On 5/10/16 4:33 PM, Kumaran, Kalyan wrote: >> * Intel is seeing difference in STREAM triad results in the 4 >> clusters. They are aware of this issue and debugging it. They would >> like Scott to rerun using 1 thread and 2 threads per core and submit >> the results to them. They will hopefully have a fix in two weeks. >> >> * We made it clear to Intel that we need a BIOS fix for the issue >> with changing modes. Every time the mode is changed a cold reboot is >> required to fix the mapping on the cores. >> >> * Intel would like us to rerun the STREAM triad results in cache >> mode. They are expecting to see 300GB/s for Bin 3 parts. Scott was >> seeng 270GB/s. >> >> * HACC and NeKbone slides will be used for ISC. >> >> * Intel to look into getting us the VTune update for KNL. >> >> -Kumar >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Action items from today's Intel call
by Scott Parker 11 May '16

11 May '16

Attached are the latest stream results for SNC-4 mode along with the source and make file used. The node was powered down and powered back up into flat/snc-4 mode. The results are the same as before - numa domain 4 shows significantly less bandwidth than the other numa domains. -Scott On 5/10/16 4:33 PM, Kumaran, Kalyan wrote: > * Intel is seeing difference in STREAM triad results in the 4 clusters. They are aware of this issue and debugging it. They would like Scott to rerun using 1 thread and 2 threads per core and submit the results to them. They will hopefully have a fix in two weeks. > > * We made it clear to Intel that we need a BIOS fix for the issue with changing modes. Every time the mode is changed a cold reboot is required to fix the mapping on the cores. > > * Intel would like us to rerun the STREAM triad results in cache mode. They are expecting to see 300GB/s for Bin 3 parts. Scott was seeng 270GB/s. > > * HACC and NeKbone slides will be used for ISC. > > * Intel to look into getting us the VTune update for KNL. > > -Kumar > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Action items from today's Intel call
by Kumaran, Kalyan 10 May '16

10 May '16

Could we add this to the confluence page? Thanks, Kumar > On May 10, 2016, at 5:22 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > Yea I can hand out a shared account on the BMC that'll let you power cycle the machine from the logins via ipmitool to those that need it. > > The procedure will be something along the lines of: > > 1. run on knl00: shutdown -h now > 2. Wait for the node to be powered off check via: ipmitool -I lanplus -H knl00.fto -U <user> power status > 3. From login node: ipmitool -I lanplus -H knl00.fto -U <user> power on > 4. Wait 5-10 minutes for the machine to come up. > > We need to validate that this has the same affect as pulling the power from the machine. > > Ben > >> On May 10, 2016, at 5:09 PM, Scott Parker <sparker(a)anl.gov> wrote: >> >> >> Do we have a way for non-admin users to initiate a cold reboot? It would be useful to have that in the interim until >> we get a BIOS fix from Intel. Otherwise we are going to be severely limited in how frequently we can change the mode, >> and it will be hard to be certain if the node is in a good state. >> >> -Scott >> >> On 5/10/16 4:33 PM, Kumaran, Kalyan wrote: >>> * Intel is seeing difference in STREAM triad results in the 4 clusters. They are aware of this issue and debugging it. They would like Scott to rerun using 1 thread and 2 threads per core and submit the results to them. They will hopefully have a fix in two weeks. >>> >>> * We made it clear to Intel that we need a BIOS fix for the issue with changing modes. Every time the mode is changed a cold reboot is required to fix the mapping on the cores. >>> >>> * Intel would like us to rerun the STREAM triad results in cache mode. They are expecting to see 300GB/s for Bin 3 parts. Scott was seeng 270GB/s. >>> >>> * HACC and NeKbone slides will be used for ISC. >>> >>> * Intel to look into getting us the VTune update for KNL. >>> >>> -Kumar >>> >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Action items from today's Intel call
by Ray Loy 10 May '16

10 May '16

mode. Isn't the goal to reboot the node in a different mode? > On May 10, 2016, at 5:44 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > New node? > > Ben > >> On May 10, 2016, at 5:31 PM, Ray Loy <rloy(a)alcf.anl.gov> wrote: >> >> >> Before the shutdown I assume you'd select the new mode somehow... >> >> >>> On May 10, 2016, at 5:22 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>> >>> Yea I can hand out a shared account on the BMC that'll let you power cycle the machine from the logins via ipmitool to those that need it. >>> >>> The procedure will be something along the lines of: >>> >>> 1. run on knl00: shutdown -h now >>> 2. Wait for the node to be powered off check via: ipmitool -I lanplus -H knl00.fto -U <user> power status >>> 3. From login node: ipmitool -I lanplus -H knl00.fto -U <user> power on >>> 4. Wait 5-10 minutes for the machine to come up. >>> >>> We need to validate that this has the same affect as pulling the power from the machine. >>> >>> Ben >>> >>>> On May 10, 2016, at 5:09 PM, Scott Parker <sparker(a)anl.gov> wrote: >>>> >>>> >>>> Do we have a way for non-admin users to initiate a cold reboot? It would be useful to have that in the interim until >>>> we get a BIOS fix from Intel. Otherwise we are going to be severely limited in how frequently we can change the mode, >>>> and it will be hard to be certain if the node is in a good state. >>>> >>>> -Scott >>>> >>>> On 5/10/16 4:33 PM, Kumaran, Kalyan wrote: >>>>> * Intel is seeing difference in STREAM triad results in the 4 clusters. They are aware of this issue and debugging it. They would like Scott to rerun using 1 thread and 2 threads per core and submit the results to them. They will hopefully have a fix in two weeks. >>>>> >>>>> * We made it clear to Intel that we need a BIOS fix for the issue with changing modes. Every time the mode is changed a cold reboot is required to fix the mapping on the cores. >>>>> >>>>> * Intel would like us to rerun the STREAM triad results in cache mode. They are expecting to see 300GB/s for Bin 3 parts. Scott was seeng 270GB/s. >>>>> >>>>> * HACC and NeKbone slides will be used for ISC. >>>>> >>>>> * Intel to look into getting us the VTune update for KNL. >>>>> >>>>> -Kumar >>>>> >>>>> _______________________________________________ >>>>> Intel-nda mailing list >>>>> Intel-nda(a)lists.jlse.anl.gov >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>> >>>> _______________________________________________ >>>> Intel-nda mailing list >>>> Intel-nda(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>> >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> >

1 0

Re: [JLSE-Intel-NDA] Action items from today's Intel call
by Allen, Benjamin S. 10 May '16

10 May '16

New node? Ben > On May 10, 2016, at 5:31 PM, Ray Loy <rloy(a)alcf.anl.gov> wrote: > > > Before the shutdown I assume you'd select the new mode somehow... > > >> On May 10, 2016, at 5:22 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >> >> Yea I can hand out a shared account on the BMC that'll let you power cycle the machine from the logins via ipmitool to those that need it. >> >> The procedure will be something along the lines of: >> >> 1. run on knl00: shutdown -h now >> 2. Wait for the node to be powered off check via: ipmitool -I lanplus -H knl00.fto -U <user> power status >> 3. From login node: ipmitool -I lanplus -H knl00.fto -U <user> power on >> 4. Wait 5-10 minutes for the machine to come up. >> >> We need to validate that this has the same affect as pulling the power from the machine. >> >> Ben >> >>> On May 10, 2016, at 5:09 PM, Scott Parker <sparker(a)anl.gov> wrote: >>> >>> >>> Do we have a way for non-admin users to initiate a cold reboot? It would be useful to have that in the interim until >>> we get a BIOS fix from Intel. Otherwise we are going to be severely limited in how frequently we can change the mode, >>> and it will be hard to be certain if the node is in a good state. >>> >>> -Scott >>> >>> On 5/10/16 4:33 PM, Kumaran, Kalyan wrote: >>>> * Intel is seeing difference in STREAM triad results in the 4 clusters. They are aware of this issue and debugging it. They would like Scott to rerun using 1 thread and 2 threads per core and submit the results to them. They will hopefully have a fix in two weeks. >>>> >>>> * We made it clear to Intel that we need a BIOS fix for the issue with changing modes. Every time the mode is changed a cold reboot is required to fix the mapping on the cores. >>>> >>>> * Intel would like us to rerun the STREAM triad results in cache mode. They are expecting to see 300GB/s for Bin 3 parts. Scott was seeng 270GB/s. >>>> >>>> * HACC and NeKbone slides will be used for ISC. >>>> >>>> * Intel to look into getting us the VTune update for KNL. >>>> >>>> -Kumar >>>> >>>> _______________________________________________ >>>> Intel-nda mailing list >>>> Intel-nda(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>> >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] Action items from today's Intel call
by Ray Loy 10 May '16

10 May '16

Before the shutdown I assume you'd select the new mode somehow... > On May 10, 2016, at 5:22 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > Yea I can hand out a shared account on the BMC that'll let you power cycle the machine from the logins via ipmitool to those that need it. > > The procedure will be something along the lines of: > > 1. run on knl00: shutdown -h now > 2. Wait for the node to be powered off check via: ipmitool -I lanplus -H knl00.fto -U <user> power status > 3. From login node: ipmitool -I lanplus -H knl00.fto -U <user> power on > 4. Wait 5-10 minutes for the machine to come up. > > We need to validate that this has the same affect as pulling the power from the machine. > > Ben > >> On May 10, 2016, at 5:09 PM, Scott Parker <sparker(a)anl.gov> wrote: >> >> >> Do we have a way for non-admin users to initiate a cold reboot? It would be useful to have that in the interim until >> we get a BIOS fix from Intel. Otherwise we are going to be severely limited in how frequently we can change the mode, >> and it will be hard to be certain if the node is in a good state. >> >> -Scott >> >> On 5/10/16 4:33 PM, Kumaran, Kalyan wrote: >>> * Intel is seeing difference in STREAM triad results in the 4 clusters. They are aware of this issue and debugging it. They would like Scott to rerun using 1 thread and 2 threads per core and submit the results to them. They will hopefully have a fix in two weeks. >>> >>> * We made it clear to Intel that we need a BIOS fix for the issue with changing modes. Every time the mode is changed a cold reboot is required to fix the mapping on the cores. >>> >>> * Intel would like us to rerun the STREAM triad results in cache mode. They are expecting to see 300GB/s for Bin 3 parts. Scott was seeng 270GB/s. >>> >>> * HACC and NeKbone slides will be used for ISC. >>> >>> * Intel to look into getting us the VTune update for KNL. >>> >>> -Kumar >>> >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0