- Intel-nda - mailman3.cels.anl.gov

[JLSE-Intel-NDA] Method for Determining Current Memory and Cluster Mode
by Allen, Benjamin S. 06 Jun '16

06 Jun '16

[bsallen@knl00 ~]$ /soft/restricted/intel-vtune-nda/vtune_amplifier_xe_2016.3.0.463186/bin64/sep -version Sampling Enabling Product version: 4.0 (private) built by patbbinn on Apr 20 2016 02:39:28 SEP User Mode Version: 4.0.0 SEP Driver Version: 4.0.0 PAX Driver Version: 1.0.1 Platform type: 102 CPU name: Intel(R) Unreleased Processor PMU: knl Cluster mode: Quadrant Memory mode: Flat

1 0

Re: [JLSE-Intel-NDA] !DIR$ ATTRIBUTES FASTMEM in cache mode
by Vitali A. Morozov 03 Jun '16

03 Jun '16

Hello, FYI: I have confirmed with the memkind developers that the policy is broken at this moment and documentation is not reflecting the boundary behaviour of the hbw_malloc for out-of-memory conditions. Vitali On 05/03/2016 09:07 AM, Vitali A. Morozov wrote: > Thanks for reporting it, Marta - I have been distracted yesterday and > failed sending a message on this issue. > > We are facing with tho issues. > > First of, we see a non-intuitive behaviour of the default policy > HBW_POLICY_PREFERRED. As documented > > "If insufficient memory is available from the high bandwidth NUMA node > closest at allocation time, fall back to standard memory (default) > with the smallest NUMA distance." > > This brings two questions. > > What happens if there is no HBW memory on the system? The expected > answer - hbw_malloc (and its Fortran equivalent ATTRIBUTES FASTMEM) > should behave equivalently to malloc. > > What happens on KNL in cache mode, where the is no memory other then > DDR memory? The expected answer - hbw_malloc (and its Fortran > equivalent) should behave equivalently to malloc. > > Marta and I have found that on both Xeon and KNL in cache mode both > Fortran and C codes failed to allocate a small chunk of memory with > hbw_malloc interface. We think this is a bug and it need to be > reported and fixed. > > We found a workaround. Additionally to the mentioned conditional > compilation (which is not an appropriate way to fix it), the user may > define > > MEMKIND_HBW_NODE=0 > > This seems to direct all memory requests to numa node 0, and this > exactly what hbw_malloc is expected to do if it runs out of memory on > other numa nodes. > > Secondly, we seems to fail setting a policy from Fortran (not that the > policy works as follows from the first report, but still). In C, the > function "hbw_set_policy" changes the policy and the following > "hbw_get_policy" returns a new policy. In Fortran the equivalent > > ierr = hbw_set_policy( %VAL(1) ) > > returns an error and the following > > ierr = hbw_get_policy() > > returns the default policy as if the previous function is ignored. > > The correspondent codes are in JLSE servers in > /home/morozov/KNL/HSCD-intel/test5 for those of you who want to verify > the reports. > > Best, > > Vitali > > > On 05/02/2016 10:19 PM, Garcia Martinez, Marta wrote: >> Hi, >> >> Vitali and I are using FASTMEM in a subroutine and it works well in >> flat mode. However, trying to run the same test in cache mode is >> causing problems (issues with not enough memory) at runtime (not >> compilation). >> >> !DIR$ ATTRIBUTES FASTMEM :: var >> >> A turnaround for the moment is to enclose that between an if defined >> declaration like this: >> >> !DEC$ IF DEFINED(name) >> !DIR$ ATTRIBUTES FASTMEM :: var >> !DEC$ ENDIF >> >> and then compile with the variable ‘name'. >> >> Marta >> >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

[JLSE-Intel-NDA] New KNL Machines
by Allen, Benjamin S. 02 Jun '16

02 Jun '16

An update for everyone here. We've received two additional KNL machines. They're Intel Software Development Platform (SDP) machines, which means they're not quite the same as knl00. They have "Groveport" main boards instead of Adam's Pass. Groveport refers to the Intel internal reference platform, where Adam's pass is the public (or soon to be) product. This of course means they have a different BIOS from our existing Adam's Pass board, so we're still working out how that relates to all the bugs we've fixed on the Adam's Pass via BIOS updates. It's also not clear yet how we change the MCDRAM Memory and Cluster modes. One good bit is these systems have "bin1" Intel Xeon Phi 7250 68c 1.4Ghz CPUs. We're working out if we can flash over the Groveport BIOS with an Adam's Pass BIOS or vice versa, and the other details above. I'll let you know when they're to a point where some early testing (STREAM, etc) is needed. Thanks, Ben

1 0

Re: [JLSE-Intel-NDA] VTune 2017 Beta Update 1
by Delalondre Fabien Jonathan 30 May '16

30 May '16

Hi Ben, I can confirm that I was able to get Vtune 2016+NDA to get some KNL profiling. Thanks for the new installs. Cheers, Fabien ________________________________________ From: Allen, Benjamin S. <bsallen(a)alcf.anl.gov> Sent: Friday, May 27, 2016 6:37 PM To: Delalondre Fabien Jonathan Cc: intel-nda(a)lists.jlse.anl.gov Subject: Re: VTune 2017 Beta Update 1 Installed at: /soft/compilers/intel_beta/vtune_amplifier_xe_2017.0.1.466719. This adds Broadwell-EP support. Note, VTune 2017 Beta 1, still doesn't have KNL support. For VTune KNL support use /soft/restricted/intel-vtune-nda. This has the NDA KNL add-on for VTune 2016 installed. I installed Advisor 2017 Beta1 yesterday, which added KNL support to Advisor. Thanks, Ben > On May 27, 2016, at 10:44 AM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: > > Last time I checked (a few ago before the update), this functionality was not supported: > "HPC Characterization analysis used to monitor utilization of the CPU, memory, and FPU > for a compute-intensive or throughput application and to identify floating point operation > and memory optimization opportunities" > > I have not have time to test it yesterday. I ll do it today and will back to you. > > Fabien > ________________________________________ > From: intel-nda-bounces(a)lists.jlse.anl.gov <intel-nda-bounces(a)lists.jlse.anl.gov> on behalf of Allen, Benjamin S. <bsallen(a)alcf.anl.gov> > Sent: Friday, May 27, 2016 5:40 PM > To: intel-nda(a)lists.jlse.anl.gov > Subject: [JLSE-Intel-NDA] VTune 2017 Beta Update 1 > > https://software.intel.com/sites/default/files/managed/02/27/release_notes_… > > No mention of KNL, anyone see anything we need in this update? > > Ben > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] VTune 2017 Beta Update 1
by Allen, Benjamin S. 27 May '16

27 May '16

Installed at: /soft/compilers/intel_beta/vtune_amplifier_xe_2017.0.1.466719. This adds Broadwell-EP support. Note, VTune 2017 Beta 1, still doesn't have KNL support. For VTune KNL support use /soft/restricted/intel-vtune-nda. This has the NDA KNL add-on for VTune 2016 installed. I installed Advisor 2017 Beta1 yesterday, which added KNL support to Advisor. Thanks, Ben > On May 27, 2016, at 10:44 AM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: > > Last time I checked (a few ago before the update), this functionality was not supported: > "HPC Characterization analysis used to monitor utilization of the CPU, memory, and FPU > for a compute-intensive or throughput application and to identify floating point operation > and memory optimization opportunities" > > I have not have time to test it yesterday. I ll do it today and will back to you. > > Fabien > ________________________________________ > From: intel-nda-bounces(a)lists.jlse.anl.gov <intel-nda-bounces(a)lists.jlse.anl.gov> on behalf of Allen, Benjamin S. <bsallen(a)alcf.anl.gov> > Sent: Friday, May 27, 2016 5:40 PM > To: intel-nda(a)lists.jlse.anl.gov > Subject: [JLSE-Intel-NDA] VTune 2017 Beta Update 1 > > https://software.intel.com/sites/default/files/managed/02/27/release_notes_… > > No mention of KNL, anyone see anything we need in this update? > > Ben > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] VTune 2017 Beta Update 1
by Delalondre Fabien Jonathan 27 May '16

27 May '16

Last time I checked (a few ago before the update), this functionality was not supported: "HPC Characterization analysis used to monitor utilization of the CPU, memory, and FPU for a compute-intensive or throughput application and to identify floating point operation and memory optimization opportunities" I have not have time to test it yesterday. I ll do it today and will back to you. Fabien ________________________________________ From: intel-nda-bounces(a)lists.jlse.anl.gov <intel-nda-bounces(a)lists.jlse.anl.gov> on behalf of Allen, Benjamin S. <bsallen(a)alcf.anl.gov> Sent: Friday, May 27, 2016 5:40 PM To: intel-nda(a)lists.jlse.anl.gov Subject: [JLSE-Intel-NDA] VTune 2017 Beta Update 1 https://software.intel.com/sites/default/files/managed/02/27/release_notes_… No mention of KNL, anyone see anything we need in this update? Ben _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] VTune 2017 Beta Update 1
by Allen, Benjamin S. 27 May '16

27 May '16

https://software.intel.com/sites/default/files/managed/02/27/release_notes_… No mention of KNL, anyone see anything we need in this update? Ben

1 0

Re: [JLSE-Intel-NDA] Resolution to the two ANL issues with SNC4
by Allen, Benjamin S. 26 May '16

26 May '16

The machines in SNC-4 / flat mode at the moment. The patch does nothing to deal with the BIOS issues. Note, the cold reboot procedure doesn't run into the hang on boot. See "Changing Node Configuration" on the wiki. Ben > On May 26, 2016, at 5:29 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: > > Do you guys know if this patch also solves the hanging issue at reboot ? I d like to do some testing in SNC-4 mode and would like to avoid getting it stuck. > > Best, > > Fabien > > ________________________________________ > From: intel-nda-bounces(a)lists.jlse.anl.gov <intel-nda-bounces(a)lists.jlse.anl.gov> on behalf of Allen, Benjamin S. <bsallen(a)alcf.anl.gov> > Sent: Thursday, May 26, 2016 11:58 PM > To: Kumaran, Kalyan > Cc: intel-nda(a)lists.jlse.anl.gov > Subject: Re: [JLSE-Intel-NDA] Resolution to the two ANL issues with SNC4 > > Kumar, > > Can you ask Andi and Avinash to integrate this into MPSP, as well as request RedHat to include the fix in probably what will end up being RHEL 7.3? Additionally SUSE 12 for Theta. > > Thanks, > > Ben > >> On May 26, 2016, at 4:52 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >> >> Excellent. We'll continue running the patched kernel on knl00, which is "3.10.0-327.el7.mpsp_1.3.0.66_725fc629ff.x86_64". >> >> Ben >> >>> On May 26, 2016, at 4:45 PM, Scott Parker <sparker(a)anl.gov> wrote: >>> >>> >>> The kernel patch appears to have resolved the SNC-4 mode memory bandwidth issue. Each numa domain is >>> now showing ~112 GB/s (previously numa domain 4 was around 70-80 GB/s). Total memory bandwidth is >>> ~440 GB/s with 64 threads (previously ~310 GB/s) and ~445 GB/s with 128 threads. These results now match >>> expectations. >>> >>> -Scott >>> >>> >>> On 5/25/16 8:37 PM, Kumaran, Kalyan wrote: >>>> And Kleen said that the patch looks good. >>>> >>>> >>>>> On May 25, 2016, at 6:54 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>>> >>>>> Cleaned up spacing to match the kernel coding standard. >>>>> >>>>> Ben >>>>> >>>>> <725fc629ff_3.10.0-327.el7_backport.patch> >>>>>> On May 25, 2016, at 6:41 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>>>> >>>>>> I'll test this on knl00 tomorrow. I created a backport to the Intel provided MPSP RHEL7.2 kernel as attached. I've booted a kernel with this patch on a Haswell box successfully so far. >>>>>> >>>>>> If anyone has some spare cycles please review it against the mainline patch https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/kern…. >>>>>> >>>>>> Kumar, feel free to pass this patch back to Avinash for review. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Ben >>>>>> >>>>>> >>>>>> <725fc629ff_3.10.0-327.el7_backport.patch> >>>>>>> On May 25, 2016, at 3:47 PM, Kumaran, Kalyan <kumaran(a)alcf.anl.gov> wrote: >>>>>>> >>>>>>> Ben, fyi. >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Begin forwarded message: >>>>>>>> >>>>>>>> From: "Sodani, Avinash" <avinash.sodani(a)intel.com> >>>>>>>> Subject: Resolution to the two ANL issues with SNC4 >>>>>>>> Date: May 25, 2016 at 3:42:09 PM CDT >>>>>>>> To: "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov>, "Harms, Kevin" <harms(a)alcf.anl.gov>, "Parker, Scott J." <sparker(a)alcf.anl.gov> >>>>>>>> Cc: "Kleen, Andi" <andi.kleen(a)intel.com>, "Chinthamani, Sundaram" <sundaram.chinthamani(a)intel.com>, "Sodani, Avinash" <avinash.sodani(a)intel.com>, "Allison, CM" <cm.allison(a)intel.com>, "Kirkendall, Keith G" <keith.g.kirkendall(a)intel.com>, "Chaffin, Benjamin C" <benjamin.c.chaffin(a)intel.com> >>>>>>>> >>>>>>>> Hi Kalyan, >>>>>>>> >>>>>>>> Here is the summary of the resolution to the two SNC4 issues that we were discussing in our meetings. The link to upstream OS patch is attached. Andi will provide any pointers to MPSP distributions as necessary. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> -Avinash >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Issue 1: “Cluster 0/4” in SNC4 does not get full bandwidth even for local accesses. >>>>>>>> >>>>>>>> Memory NUMA Domain >>>>>>>> 4 5 6 7 >>>>>>>> CPU NUMA Domain 0 73,475.60 84,302.20 107,423.70 55,924.00 >>>>>>>> 1 65,511.00 112,528.40 55,557.00 77,935.00 >>>>>>>> 2 60,390.10 67,572.70 111,543.80 77,941.20 >>>>>>>> 3 63,518.40 102,543.80 76,201.40 111622.4 >>>>>>>> >>>>>>>> >>>>>>>> Root Cause: OS puts idle threads in mwait. In SNC4, the addresses for these OS mwaits were being assigned such they all (288 of them, one per thread) were getting mapped into the tag directories in cluster 0. This was overwhelming the mechanism architected to handle mwait wakeup, causing significant slowdowns for the memory traffic getting routed through the tag directories in cluster 0. We don’t see this behavior for Quadrant and All-to-all modes because the mwait addresses are distributed to all tag directories, and therefore the mwait mechanisms do not get overwhelmed. >>>>>>>> Fix: OS patch has been released that distributes the OS mwait lines more uniformly over all clusters in SNC4, preventing this issue. >>>>>>>> Link: (https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/kern…) >>>>>>>> >>>>>>>> >>>>>>>> Issue 2: In SNC4, the traffic targeting only remote clusters show much lower BW compared to the traffic targeting only local clusters >>>>>>>> Node 4 5 6 7 >>>>>>>> 0 132,090 107,477 120,639 72,496 >>>>>>>> 1 111,967 132,738 84,004 123,338 >>>>>>>> 2 111,637 84,380 125,716 96,188 >>>>>>>> 3 86,456 120,387 100,434 125,238 >>>>>>>> >>>>>>>> Root Cause: KNL follows YX routing for sending messages on the mesh. In SNC4 mode, when targeting all traffic to a remote cluster, the traffic (both address and data) predominantly flows in one direction: for example same Y and same X direction to go to the remote cluster that is diagonally across. This largely unidirectional flow cuts down effective BW on the mesh and causes congestion that results on further slowdowns. Mesh resources are provisioned for uniform distribution of traffic, such as those seen in Quadrant, all-to-all modes and local traffic in SNC4 mode. It is not provisioned for predominantly remote traffic in SNC4 mode. >>>>>>>> Fix: No HW or SW fix for this. Usage model needs to avoid drawing most of the traffic from remote clusters in SNC4 mode. They should keep the traffic local. If the access pattern is such that large amounts of remote traffic cannot be avoided then consider using Quadrant mode. >>>>>> _______________________________________________ >>>>>> Intel-nda mailing list >>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>> _______________________________________________ >>>> Intel-nda mailing list >>>> Intel-nda(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>> >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Resolution to the two ANL issues with SNC4
by Delalondre Fabien Jonathan 26 May '16

26 May '16

Do you guys know if this patch also solves the hanging issue at reboot ? I d like to do some testing in SNC-4 mode and would like to avoid getting it stuck. Best, Fabien ________________________________________ From: intel-nda-bounces(a)lists.jlse.anl.gov <intel-nda-bounces(a)lists.jlse.anl.gov> on behalf of Allen, Benjamin S. <bsallen(a)alcf.anl.gov> Sent: Thursday, May 26, 2016 11:58 PM To: Kumaran, Kalyan Cc: intel-nda(a)lists.jlse.anl.gov Subject: Re: [JLSE-Intel-NDA] Resolution to the two ANL issues with SNC4 Kumar, Can you ask Andi and Avinash to integrate this into MPSP, as well as request RedHat to include the fix in probably what will end up being RHEL 7.3? Additionally SUSE 12 for Theta. Thanks, Ben > On May 26, 2016, at 4:52 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > Excellent. We'll continue running the patched kernel on knl00, which is "3.10.0-327.el7.mpsp_1.3.0.66_725fc629ff.x86_64". > > Ben > >> On May 26, 2016, at 4:45 PM, Scott Parker <sparker(a)anl.gov> wrote: >> >> >> The kernel patch appears to have resolved the SNC-4 mode memory bandwidth issue. Each numa domain is >> now showing ~112 GB/s (previously numa domain 4 was around 70-80 GB/s). Total memory bandwidth is >> ~440 GB/s with 64 threads (previously ~310 GB/s) and ~445 GB/s with 128 threads. These results now match >> expectations. >> >> -Scott >> >> >> On 5/25/16 8:37 PM, Kumaran, Kalyan wrote: >>> And Kleen said that the patch looks good. >>> >>> >>>> On May 25, 2016, at 6:54 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>> >>>> Cleaned up spacing to match the kernel coding standard. >>>> >>>> Ben >>>> >>>> <725fc629ff_3.10.0-327.el7_backport.patch> >>>>> On May 25, 2016, at 6:41 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>>> >>>>> I'll test this on knl00 tomorrow. I created a backport to the Intel provided MPSP RHEL7.2 kernel as attached. I've booted a kernel with this patch on a Haswell box successfully so far. >>>>> >>>>> If anyone has some spare cycles please review it against the mainline patch https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/kern…. >>>>> >>>>> Kumar, feel free to pass this patch back to Avinash for review. >>>>> >>>>> Thanks, >>>>> >>>>> Ben >>>>> >>>>> >>>>> <725fc629ff_3.10.0-327.el7_backport.patch> >>>>>> On May 25, 2016, at 3:47 PM, Kumaran, Kalyan <kumaran(a)alcf.anl.gov> wrote: >>>>>> >>>>>> Ben, fyi. >>>>>> >>>>>> >>>>>> >>>>>>> Begin forwarded message: >>>>>>> >>>>>>> From: "Sodani, Avinash" <avinash.sodani(a)intel.com> >>>>>>> Subject: Resolution to the two ANL issues with SNC4 >>>>>>> Date: May 25, 2016 at 3:42:09 PM CDT >>>>>>> To: "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov>, "Harms, Kevin" <harms(a)alcf.anl.gov>, "Parker, Scott J." <sparker(a)alcf.anl.gov> >>>>>>> Cc: "Kleen, Andi" <andi.kleen(a)intel.com>, "Chinthamani, Sundaram" <sundaram.chinthamani(a)intel.com>, "Sodani, Avinash" <avinash.sodani(a)intel.com>, "Allison, CM" <cm.allison(a)intel.com>, "Kirkendall, Keith G" <keith.g.kirkendall(a)intel.com>, "Chaffin, Benjamin C" <benjamin.c.chaffin(a)intel.com> >>>>>>> >>>>>>> Hi Kalyan, >>>>>>> >>>>>>> Here is the summary of the resolution to the two SNC4 issues that we were discussing in our meetings. The link to upstream OS patch is attached. Andi will provide any pointers to MPSP distributions as necessary. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> -Avinash >>>>>>> >>>>>>> >>>>>>> >>>>>>> Issue 1: “Cluster 0/4” in SNC4 does not get full bandwidth even for local accesses. >>>>>>> >>>>>>> Memory NUMA Domain >>>>>>> 4 5 6 7 >>>>>>> CPU NUMA Domain 0 73,475.60 84,302.20 107,423.70 55,924.00 >>>>>>> 1 65,511.00 112,528.40 55,557.00 77,935.00 >>>>>>> 2 60,390.10 67,572.70 111,543.80 77,941.20 >>>>>>> 3 63,518.40 102,543.80 76,201.40 111622.4 >>>>>>> >>>>>>> >>>>>>> Root Cause: OS puts idle threads in mwait. In SNC4, the addresses for these OS mwaits were being assigned such they all (288 of them, one per thread) were getting mapped into the tag directories in cluster 0. This was overwhelming the mechanism architected to handle mwait wakeup, causing significant slowdowns for the memory traffic getting routed through the tag directories in cluster 0. We don’t see this behavior for Quadrant and All-to-all modes because the mwait addresses are distributed to all tag directories, and therefore the mwait mechanisms do not get overwhelmed. >>>>>>> Fix: OS patch has been released that distributes the OS mwait lines more uniformly over all clusters in SNC4, preventing this issue. >>>>>>> Link: (https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/kern…) >>>>>>> >>>>>>> >>>>>>> Issue 2: In SNC4, the traffic targeting only remote clusters show much lower BW compared to the traffic targeting only local clusters >>>>>>> Node 4 5 6 7 >>>>>>> 0 132,090 107,477 120,639 72,496 >>>>>>> 1 111,967 132,738 84,004 123,338 >>>>>>> 2 111,637 84,380 125,716 96,188 >>>>>>> 3 86,456 120,387 100,434 125,238 >>>>>>> >>>>>>> Root Cause: KNL follows YX routing for sending messages on the mesh. In SNC4 mode, when targeting all traffic to a remote cluster, the traffic (both address and data) predominantly flows in one direction: for example same Y and same X direction to go to the remote cluster that is diagonally across. This largely unidirectional flow cuts down effective BW on the mesh and causes congestion that results on further slowdowns. Mesh resources are provisioned for uniform distribution of traffic, such as those seen in Quadrant, all-to-all modes and local traffic in SNC4 mode. It is not provisioned for predominantly remote traffic in SNC4 mode. >>>>>>> Fix: No HW or SW fix for this. Usage model needs to avoid drawing most of the traffic from remote clusters in SNC4 mode. They should keep the traffic local. If the access pattern is such that large amounts of remote traffic cannot be avoided then consider using Quadrant mode. >>>>> _______________________________________________ >>>>> Intel-nda mailing list >>>>> Intel-nda(a)lists.jlse.anl.gov >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Resolution to the two ANL issues with SNC4
by Allen, Benjamin S. 26 May '16

26 May '16

Kumar, Can you ask Andi and Avinash to integrate this into MPSP, as well as request RedHat to include the fix in probably what will end up being RHEL 7.3? Additionally SUSE 12 for Theta. Thanks, Ben > On May 26, 2016, at 4:52 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > Excellent. We'll continue running the patched kernel on knl00, which is "3.10.0-327.el7.mpsp_1.3.0.66_725fc629ff.x86_64". > > Ben > >> On May 26, 2016, at 4:45 PM, Scott Parker <sparker(a)anl.gov> wrote: >> >> >> The kernel patch appears to have resolved the SNC-4 mode memory bandwidth issue. Each numa domain is >> now showing ~112 GB/s (previously numa domain 4 was around 70-80 GB/s). Total memory bandwidth is >> ~440 GB/s with 64 threads (previously ~310 GB/s) and ~445 GB/s with 128 threads. These results now match >> expectations. >> >> -Scott >> >> >> On 5/25/16 8:37 PM, Kumaran, Kalyan wrote: >>> And Kleen said that the patch looks good. >>> >>> >>>> On May 25, 2016, at 6:54 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>> >>>> Cleaned up spacing to match the kernel coding standard. >>>> >>>> Ben >>>> >>>> <725fc629ff_3.10.0-327.el7_backport.patch> >>>>> On May 25, 2016, at 6:41 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>>> >>>>> I'll test this on knl00 tomorrow. I created a backport to the Intel provided MPSP RHEL7.2 kernel as attached. I've booted a kernel with this patch on a Haswell box successfully so far. >>>>> >>>>> If anyone has some spare cycles please review it against the mainline patch https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/kern…. >>>>> >>>>> Kumar, feel free to pass this patch back to Avinash for review. >>>>> >>>>> Thanks, >>>>> >>>>> Ben >>>>> >>>>> >>>>> <725fc629ff_3.10.0-327.el7_backport.patch> >>>>>> On May 25, 2016, at 3:47 PM, Kumaran, Kalyan <kumaran(a)alcf.anl.gov> wrote: >>>>>> >>>>>> Ben, fyi. >>>>>> >>>>>> >>>>>> >>>>>>> Begin forwarded message: >>>>>>> >>>>>>> From: "Sodani, Avinash" <avinash.sodani(a)intel.com> >>>>>>> Subject: Resolution to the two ANL issues with SNC4 >>>>>>> Date: May 25, 2016 at 3:42:09 PM CDT >>>>>>> To: "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov>, "Harms, Kevin" <harms(a)alcf.anl.gov>, "Parker, Scott J." <sparker(a)alcf.anl.gov> >>>>>>> Cc: "Kleen, Andi" <andi.kleen(a)intel.com>, "Chinthamani, Sundaram" <sundaram.chinthamani(a)intel.com>, "Sodani, Avinash" <avinash.sodani(a)intel.com>, "Allison, CM" <cm.allison(a)intel.com>, "Kirkendall, Keith G" <keith.g.kirkendall(a)intel.com>, "Chaffin, Benjamin C" <benjamin.c.chaffin(a)intel.com> >>>>>>> >>>>>>> Hi Kalyan, >>>>>>> >>>>>>> Here is the summary of the resolution to the two SNC4 issues that we were discussing in our meetings. The link to upstream OS patch is attached. Andi will provide any pointers to MPSP distributions as necessary. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> -Avinash >>>>>>> >>>>>>> >>>>>>> >>>>>>> Issue 1: “Cluster 0/4” in SNC4 does not get full bandwidth even for local accesses. >>>>>>> >>>>>>> Memory NUMA Domain >>>>>>> 4 5 6 7 >>>>>>> CPU NUMA Domain 0 73,475.60 84,302.20 107,423.70 55,924.00 >>>>>>> 1 65,511.00 112,528.40 55,557.00 77,935.00 >>>>>>> 2 60,390.10 67,572.70 111,543.80 77,941.20 >>>>>>> 3 63,518.40 102,543.80 76,201.40 111622.4 >>>>>>> >>>>>>> >>>>>>> Root Cause: OS puts idle threads in mwait. In SNC4, the addresses for these OS mwaits were being assigned such they all (288 of them, one per thread) were getting mapped into the tag directories in cluster 0. This was overwhelming the mechanism architected to handle mwait wakeup, causing significant slowdowns for the memory traffic getting routed through the tag directories in cluster 0. We don’t see this behavior for Quadrant and All-to-all modes because the mwait addresses are distributed to all tag directories, and therefore the mwait mechanisms do not get overwhelmed. >>>>>>> Fix: OS patch has been released that distributes the OS mwait lines more uniformly over all clusters in SNC4, preventing this issue. >>>>>>> Link: (https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/kern…) >>>>>>> >>>>>>> >>>>>>> Issue 2: In SNC4, the traffic targeting only remote clusters show much lower BW compared to the traffic targeting only local clusters >>>>>>> Node 4 5 6 7 >>>>>>> 0 132,090 107,477 120,639 72,496 >>>>>>> 1 111,967 132,738 84,004 123,338 >>>>>>> 2 111,637 84,380 125,716 96,188 >>>>>>> 3 86,456 120,387 100,434 125,238 >>>>>>> >>>>>>> Root Cause: KNL follows YX routing for sending messages on the mesh. In SNC4 mode, when targeting all traffic to a remote cluster, the traffic (both address and data) predominantly flows in one direction: for example same Y and same X direction to go to the remote cluster that is diagonally across. This largely unidirectional flow cuts down effective BW on the mesh and causes congestion that results on further slowdowns. Mesh resources are provisioned for uniform distribution of traffic, such as those seen in Quadrant, all-to-all modes and local traffic in SNC4 mode. It is not provisioned for predominantly remote traffic in SNC4 mode. >>>>>>> Fix: No HW or SW fix for this. Usage model needs to avoid drawing most of the traffic from remote clusters in SNC4 mode. They should keep the traffic local. If the access pattern is such that large amounts of remote traffic cannot be avoided then consider using Quadrant mode. >>>>> _______________________________________________ >>>>> Intel-nda mailing list >>>>> Intel-nda(a)lists.jlse.anl.gov >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0