- Intel-nda - mailman3.cels.anl.gov

Re: [JLSE-Intel-NDA] Resolution to the two ANL issues with SNC4
by Allen, Benjamin S. 26 May '16

26 May '16

Excellent. We'll continue running the patched kernel on knl00, which is "3.10.0-327.el7.mpsp_1.3.0.66_725fc629ff.x86_64". Ben > On May 26, 2016, at 4:45 PM, Scott Parker <sparker(a)anl.gov> wrote: > > > The kernel patch appears to have resolved the SNC-4 mode memory bandwidth issue. Each numa domain is > now showing ~112 GB/s (previously numa domain 4 was around 70-80 GB/s). Total memory bandwidth is > ~440 GB/s with 64 threads (previously ~310 GB/s) and ~445 GB/s with 128 threads. These results now match > expectations. > > -Scott > > > On 5/25/16 8:37 PM, Kumaran, Kalyan wrote: >> And Kleen said that the patch looks good. >> >> >>> On May 25, 2016, at 6:54 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>> >>> Cleaned up spacing to match the kernel coding standard. >>> >>> Ben >>> >>> <725fc629ff_3.10.0-327.el7_backport.patch> >>>> On May 25, 2016, at 6:41 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>> >>>> I'll test this on knl00 tomorrow. I created a backport to the Intel provided MPSP RHEL7.2 kernel as attached. I've booted a kernel with this patch on a Haswell box successfully so far. >>>> >>>> If anyone has some spare cycles please review it against the mainline patch https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/kern…. >>>> >>>> Kumar, feel free to pass this patch back to Avinash for review. >>>> >>>> Thanks, >>>> >>>> Ben >>>> >>>> >>>> <725fc629ff_3.10.0-327.el7_backport.patch> >>>>> On May 25, 2016, at 3:47 PM, Kumaran, Kalyan <kumaran(a)alcf.anl.gov> wrote: >>>>> >>>>> Ben, fyi. >>>>> >>>>> >>>>> >>>>>> Begin forwarded message: >>>>>> >>>>>> From: "Sodani, Avinash" <avinash.sodani(a)intel.com> >>>>>> Subject: Resolution to the two ANL issues with SNC4 >>>>>> Date: May 25, 2016 at 3:42:09 PM CDT >>>>>> To: "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov>, "Harms, Kevin" <harms(a)alcf.anl.gov>, "Parker, Scott J." <sparker(a)alcf.anl.gov> >>>>>> Cc: "Kleen, Andi" <andi.kleen(a)intel.com>, "Chinthamani, Sundaram" <sundaram.chinthamani(a)intel.com>, "Sodani, Avinash" <avinash.sodani(a)intel.com>, "Allison, CM" <cm.allison(a)intel.com>, "Kirkendall, Keith G" <keith.g.kirkendall(a)intel.com>, "Chaffin, Benjamin C" <benjamin.c.chaffin(a)intel.com> >>>>>> >>>>>> Hi Kalyan, >>>>>> >>>>>> Here is the summary of the resolution to the two SNC4 issues that we were discussing in our meetings. The link to upstream OS patch is attached. Andi will provide any pointers to MPSP distributions as necessary. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> -Avinash >>>>>> >>>>>> >>>>>> >>>>>> Issue 1: “Cluster 0/4” in SNC4 does not get full bandwidth even for local accesses. >>>>>> >>>>>> Memory NUMA Domain >>>>>> 4 5 6 7 >>>>>> CPU NUMA Domain 0 73,475.60 84,302.20 107,423.70 55,924.00 >>>>>> 1 65,511.00 112,528.40 55,557.00 77,935.00 >>>>>> 2 60,390.10 67,572.70 111,543.80 77,941.20 >>>>>> 3 63,518.40 102,543.80 76,201.40 111622.4 >>>>>> >>>>>> >>>>>> Root Cause: OS puts idle threads in mwait. In SNC4, the addresses for these OS mwaits were being assigned such they all (288 of them, one per thread) were getting mapped into the tag directories in cluster 0. This was overwhelming the mechanism architected to handle mwait wakeup, causing significant slowdowns for the memory traffic getting routed through the tag directories in cluster 0. We don’t see this behavior for Quadrant and All-to-all modes because the mwait addresses are distributed to all tag directories, and therefore the mwait mechanisms do not get overwhelmed. >>>>>> Fix: OS patch has been released that distributes the OS mwait lines more uniformly over all clusters in SNC4, preventing this issue. >>>>>> Link: (https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/kern…) >>>>>> >>>>>> >>>>>> Issue 2: In SNC4, the traffic targeting only remote clusters show much lower BW compared to the traffic targeting only local clusters >>>>>> Node 4 5 6 7 >>>>>> 0 132,090 107,477 120,639 72,496 >>>>>> 1 111,967 132,738 84,004 123,338 >>>>>> 2 111,637 84,380 125,716 96,188 >>>>>> 3 86,456 120,387 100,434 125,238 >>>>>> >>>>>> Root Cause: KNL follows YX routing for sending messages on the mesh. In SNC4 mode, when targeting all traffic to a remote cluster, the traffic (both address and data) predominantly flows in one direction: for example same Y and same X direction to go to the remote cluster that is diagonally across. This largely unidirectional flow cuts down effective BW on the mesh and causes congestion that results on further slowdowns. Mesh resources are provisioned for uniform distribution of traffic, such as those seen in Quadrant, all-to-all modes and local traffic in SNC4 mode. It is not provisioned for predominantly remote traffic in SNC4 mode. >>>>>> Fix: No HW or SW fix for this. Usage model needs to avoid drawing most of the traffic from remote clusters in SNC4 mode. They should keep the traffic local. If the access pattern is such that large amounts of remote traffic cannot be avoided then consider using Quadrant mode. >>>> _______________________________________________ >>>> Intel-nda mailing list >>>> Intel-nda(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] Resolution to the two ANL issues with SNC4
by Scott Parker 26 May '16

26 May '16

The kernel patch appears to have resolved the SNC-4 mode memory bandwidth issue. Each numa domain is now showing ~112 GB/s (previously numa domain 4 was around 70-80 GB/s). Total memory bandwidth is ~440 GB/s with 64 threads (previously ~310 GB/s) and ~445 GB/s with 128 threads. These results now match expectations. -Scott On 5/25/16 8:37 PM, Kumaran, Kalyan wrote: > And Kleen said that the patch looks good. > > >> On May 25, 2016, at 6:54 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >> >> Cleaned up spacing to match the kernel coding standard. >> >> Ben >> >> <725fc629ff_3.10.0-327.el7_backport.patch> >>> On May 25, 2016, at 6:41 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>> >>> I'll test this on knl00 tomorrow. I created a backport to the Intel provided MPSP RHEL7.2 kernel as attached. I've booted a kernel with this patch on a Haswell box successfully so far. >>> >>> If anyone has some spare cycles please review it against the mainline patch https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/kern…. >>> >>> Kumar, feel free to pass this patch back to Avinash for review. >>> >>> Thanks, >>> >>> Ben >>> >>> >>> <725fc629ff_3.10.0-327.el7_backport.patch> >>>> On May 25, 2016, at 3:47 PM, Kumaran, Kalyan <kumaran(a)alcf.anl.gov> wrote: >>>> >>>> Ben, fyi. >>>> >>>> >>>> >>>>> Begin forwarded message: >>>>> >>>>> From: "Sodani, Avinash" <avinash.sodani(a)intel.com> >>>>> Subject: Resolution to the two ANL issues with SNC4 >>>>> Date: May 25, 2016 at 3:42:09 PM CDT >>>>> To: "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov>, "Harms, Kevin" <harms(a)alcf.anl.gov>, "Parker, Scott J." <sparker(a)alcf.anl.gov> >>>>> Cc: "Kleen, Andi" <andi.kleen(a)intel.com>, "Chinthamani, Sundaram" <sundaram.chinthamani(a)intel.com>, "Sodani, Avinash" <avinash.sodani(a)intel.com>, "Allison, CM" <cm.allison(a)intel.com>, "Kirkendall, Keith G" <keith.g.kirkendall(a)intel.com>, "Chaffin, Benjamin C" <benjamin.c.chaffin(a)intel.com> >>>>> >>>>> Hi Kalyan, >>>>> >>>>> Here is the summary of the resolution to the two SNC4 issues that we were discussing in our meetings. The link to upstream OS patch is attached. Andi will provide any pointers to MPSP distributions as necessary. >>>>> >>>>> Thanks, >>>>> >>>>> -Avinash >>>>> >>>>> >>>>> >>>>> Issue 1: “Cluster 0/4” in SNC4 does not get full bandwidth even for local accesses. >>>>> >>>>> Memory NUMA Domain >>>>> 4 5 6 7 >>>>> CPU NUMA Domain 0 73,475.60 84,302.20 107,423.70 55,924.00 >>>>> 1 65,511.00 112,528.40 55,557.00 77,935.00 >>>>> 2 60,390.10 67,572.70 111,543.80 77,941.20 >>>>> 3 63,518.40 102,543.80 76,201.40 111622.4 >>>>> >>>>> >>>>> Root Cause: OS puts idle threads in mwait. In SNC4, the addresses for these OS mwaits were being assigned such they all (288 of them, one per thread) were getting mapped into the tag directories in cluster 0. This was overwhelming the mechanism architected to handle mwait wakeup, causing significant slowdowns for the memory traffic getting routed through the tag directories in cluster 0. We don’t see this behavior for Quadrant and All-to-all modes because the mwait addresses are distributed to all tag directories, and therefore the mwait mechanisms do not get overwhelmed. >>>>> Fix: OS patch has been released that distributes the OS mwait lines more uniformly over all clusters in SNC4, preventing this issue. >>>>> Link: (https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/kern…) >>>>> >>>>> >>>>> Issue 2: In SNC4, the traffic targeting only remote clusters show much lower BW compared to the traffic targeting only local clusters >>>>> Node 4 5 6 7 >>>>> 0 132,090 107,477 120,639 72,496 >>>>> 1 111,967 132,738 84,004 123,338 >>>>> 2 111,637 84,380 125,716 96,188 >>>>> 3 86,456 120,387 100,434 125,238 >>>>> >>>>> Root Cause: KNL follows YX routing for sending messages on the mesh. In SNC4 mode, when targeting all traffic to a remote cluster, the traffic (both address and data) predominantly flows in one direction: for example same Y and same X direction to go to the remote cluster that is diagonally across. This largely unidirectional flow cuts down effective BW on the mesh and causes congestion that results on further slowdowns. Mesh resources are provisioned for uniform distribution of traffic, such as those seen in Quadrant, all-to-all modes and local traffic in SNC4 mode. It is not provisioned for predominantly remote traffic in SNC4 mode. >>>>> Fix: No HW or SW fix for this. Usage model needs to avoid drawing most of the traffic from remote clusters in SNC4 mode. They should keep the traffic local. If the access pattern is such that large amounts of remote traffic cannot be avoided then consider using Quadrant mode. >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Intel Advisor XE Beta 2017 Update 1
by Delalondre Fabien Jonathan 26 May '16

26 May '16

Hi Ben, You can use the system now if you want to go ahead. Let me know when done and I can give it a try before lunch. Best, Fabien ________________________________________ From: intel-nda-bounces(a)lists.jlse.anl.gov <intel-nda-bounces(a)lists.jlse.anl.gov> on behalf of Allen, Benjamin S. <bsallen(a)alcf.anl.gov> Sent: Thursday, May 26, 2016 5:16 PM To: intel-nda(a)lists.jlse.anl.gov Subject: [JLSE-Intel-NDA] Intel Advisor XE Beta 2017 Update 1 Looks like KNL support has been added into Advisor: https://software.intel.com/sites/default/files/managed/94/a2/release_notes_…. Installing this shortly. Ben _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] Intel Advisor XE Beta 2017 Update 1
by Allen, Benjamin S. 26 May '16

26 May '16

Looks like KNL support has been added into Advisor: https://software.intel.com/sites/default/files/managed/94/a2/release_notes_…. Installing this shortly. Ben

1 0

[JLSE-Intel-NDA] Fwd: WW21-22 Xeon Phi Biweekly Product Update
by Kumaran, Kalyan 26 May '16

26 May '16

Begin forwarded message: _____________________________________________ From: Linthwaite, Kirsten A Sent: Wednesday, May 25, 2016 5:37 PM Subject: WW21-22 Xeon Phi Biweekly Product Update [X] [X] Xeon Phi™ Home Page<http://www.intel.com/content/www/us/en/processors/xeon/xeon-phi-detail.html> | Xeon Phi™ App Catalogs<https://software.intel.com/en-us/mic-developer/app-catalogs> | Developer Zone<http://software.intel.com/en-us/mic-developer> | IPCCs<http://software.intel.com/en-us/articles/intel-parallel-computing-centers> | Product Overview (IBL)<http://www.intel.com/cd/edesign/library/asmo-na/eng/538824.htm> Spotlight Knights Landing Launch: We are 4 weeks away from the launch of KNL at ISC ’16! The KNL launch will kick-off June 20th at 9am CEST. If you would like to participate in the launch celebration, social media amplification, or other launch materials, please work with your Intel representative as soon as possible to ensure we can accommodate your participation. Recent News: Colfax Research just published three new whitepapers focused enabling software for KNL. The paper titles are: “Automatic Vectorization with Intel AVX-512 Instructions in KNL”, “Clustering Modes in Knights Landing Processors”, and “MCDRAM as High-Bandwidth Memory (HBM) in KNL Processors”. These papers will be a great resource for anyone preparing to move to KNL soon. An overview of each paper and a link to download them is available on Colfax’s website<http://colfaxresearch.com/get-ready-for-intel-knights-landing-3-papers/>. New Video: Looking to get more information on the benefits of the KNL Developer Access Program? This short video<http://www.go-parallel.com/benefits-of-intel-xeon-phi-developers-program/> gives an excerpt of an interview of Intel expert James Reinders explaining the benefits of the KNL DAP. More Launch Activities: The Texas Advanced Computing Center (TACC) at the University of Texas at Austin, in partnership with Intel Corporation, is proud to announce a special tutorial event as part of the Intel Xeon Phi User group Meeting (IXPUG) at ISC16 in Frankfurt, Germany. This event will provide hands-on experience on the latest Intel product, the second generation Intel Xeon Phi, also known as Knights Landing (KNL). Attendees to this half-day event will be some of the first researchers in the world to have access to the KNL platform. The event will be June 19th from 8am-12pm at the Frankfurt Hotel Marriott. You can register for the event here<https://portal.tacc.utexas.edu/training/>. More News Training Highlight FREE Intel® Xeon Phi™ Code Modernization Workshops Upcoming Dates Location Link May 27 Almaty, Kazakhstan Register<http://www.inteldevconference.com/events/almaty-code-modernization/> May 31-June 1 Gothenburg, Sweden Register<http://www.inteldevconference.com/events/gothenburg/> June 2-3 Copenhagen, Denmark Register<http://www.inteldevconference.com/events/copenhagen/> June 6-7 Cambridge, England Register<http://www.inteldevconference.com/events/cambridge-united-kingdom/> June 8-9 Toulouse, France Register<http://www.inteldevconference.com/events/toulouse/> June 15-16 Zurich, Switzerland Register<http://www.inteldevconference.com/events/zurich-hpc-code-modernization/> Checkout the curriculums and course resource materials that were designed by University of Oregon<http://ipcc.cs.uoregon.edu/curriculum.html>, Carnegie Mellon University<http://15418.courses.cs.cmu.edu/spring2015/exercises>, University of Michigan<http://www.eecs.umich.edu/courses/eecs570/>, University of Washington<http://homes.cs.washington.edu/~nelson/csep524/>, Duke University<http://www.cs.duke.edu/courses/fall15/compsci590.3/>, and University of Illinois at Urbana–Champaign<https://wiki.cites.illinois.edu/wiki/display/cs484sp16/Assignments?src=spac…>to see how these universities are teaching parallel programming. FREE TRAINING: Colfax Research continues to offer their “Developer’s Guide to the Next-Generation Intel® Xeon Phi™ Processor” webinars. This is a one-hour FREE webinar that is held at various times to accommodate developers worldwide. The webinars are both an introduction to KNL and a programmer’s guide to KNL. You can find the times and register on the Colfax website<http://colfaxresearch.com/knl-webinar/>. Knights Landing Key Information Software Resources * Developer Access Program (DAP)<http://dap.xeonphi.com/> * Knights Landing – What’s Public<https://software.intel.com/en-us/articles/what-disclosures-has-intel-made-a…> * Intel® Parallel Studio XE Tools and Libraries<https://software.intel.com/en-us/free_tools_and_libraries> * Intel® MIC Developer Zone<http://software.intel.com/mic-developer> * Intel® Modern Code<https://software.intel.com/en-us/modern-code> * Intel® Xeon Phi™ Software Code Recipes<http://software.intel.com/en-us/articles/code-recipes-for-intelr-xeon-phitm…> * Intel® Xeon Phi™ SW Tools and Libraries<http://software.intel.com/en-us/articles/intel-and-third-party-tools-and-li…> * Intel® Software Tools Technical Webinar Series<http://software.intel.com/en-us/articles/intel-software-tools-technical-web…> * Intel® MIC Technical Update (CDI doc 473027) – CNDA<http://www.intel.com/cd/edesign/library/asmo-na/eng/473027.htm> * Intel® Xeon Phi™ Coprocessor – Applications and Solutions Catalog<https://software.intel.com/en-us/articles/intel-xeon-phi-coprocessor-applic…> * Intel® Xeon Phi™ Processor Product Family Early Release Software and Tools<https://software.intel.com/en-us/xeon-phi-nda/tools> (request access to the site here<https://software.intel.com/en-us/form/intel-xeon-phi-nda-site-access-request>) Knights Corner Collateral * Intel® Xeon Phi™ Public Performance Data<http://www.intel.com/content/www/us/en/benchmarks/server/xeon-phi/xeon-phi-…> * Accelerating Energy Exploration with Intel® Xeon Phi™ Coprocessors Case Study<https://software.intel.com/sites/default/files/managed/f1/e6/Intel-Cirrasca…> * Taking Oil and Gas Exploration to the Next Level Case Study<https://www-ssl.intel.com/content/www/us/en/high-performance-computing/dug-…> * SGI® Unveiling the Early Universe: Optimizing Cosmology Workloads for Intel® Xeon Phi™ Coprocessors Case Study<http://www.sgi.com/pdfs/4523.pdf> * OEM Platform Supporting Intel® Xeon Phi™ – CNDA<http://www.intel.com/cd/edesign/library/asmo-na/eng/540160.htm> Copyright © 2016 Intel Corporation. All rights reserved. Legal<http://targetmailer.intel.com/TMService/Redir.aspx?ID=1266509>. Privacy.<http://targetmailer.intel.com/TMService/Redir.aspx?ID=1266510> Intel Confidential – For Use under CNDA Only *Other names and brands may be claimed as the property of others.

1 0

Re: [JLSE-Intel-NDA] Resolution to the two ANL issues with SNC4
by Kumaran, Kalyan 25 May '16

25 May '16

And Kleen said that the patch looks good. > On May 25, 2016, at 6:54 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > Cleaned up spacing to match the kernel coding standard. > > Ben > > <725fc629ff_3.10.0-327.el7_backport.patch> >> On May 25, 2016, at 6:41 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >> >> I'll test this on knl00 tomorrow. I created a backport to the Intel provided MPSP RHEL7.2 kernel as attached. I've booted a kernel with this patch on a Haswell box successfully so far. >> >> If anyone has some spare cycles please review it against the mainline patch https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/kern…. >> >> Kumar, feel free to pass this patch back to Avinash for review. >> >> Thanks, >> >> Ben >> >> >> <725fc629ff_3.10.0-327.el7_backport.patch> >>> On May 25, 2016, at 3:47 PM, Kumaran, Kalyan <kumaran(a)alcf.anl.gov> wrote: >>> >>> Ben, fyi. >>> >>> >>> >>>> Begin forwarded message: >>>> >>>> From: "Sodani, Avinash" <avinash.sodani(a)intel.com> >>>> Subject: Resolution to the two ANL issues with SNC4 >>>> Date: May 25, 2016 at 3:42:09 PM CDT >>>> To: "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov>, "Harms, Kevin" <harms(a)alcf.anl.gov>, "Parker, Scott J." <sparker(a)alcf.anl.gov> >>>> Cc: "Kleen, Andi" <andi.kleen(a)intel.com>, "Chinthamani, Sundaram" <sundaram.chinthamani(a)intel.com>, "Sodani, Avinash" <avinash.sodani(a)intel.com>, "Allison, CM" <cm.allison(a)intel.com>, "Kirkendall, Keith G" <keith.g.kirkendall(a)intel.com>, "Chaffin, Benjamin C" <benjamin.c.chaffin(a)intel.com> >>>> >>>> Hi Kalyan, >>>> >>>> Here is the summary of the resolution to the two SNC4 issues that we were discussing in our meetings. The link to upstream OS patch is attached. Andi will provide any pointers to MPSP distributions as necessary. >>>> >>>> Thanks, >>>> >>>> -Avinash >>>> >>>> >>>> >>>> Issue 1: “Cluster 0/4” in SNC4 does not get full bandwidth even for local accesses. >>>> >>>> Memory NUMA Domain >>>> 4 5 6 7 >>>> CPU NUMA Domain 0 73,475.60 84,302.20 107,423.70 55,924.00 >>>> 1 65,511.00 112,528.40 55,557.00 77,935.00 >>>> 2 60,390.10 67,572.70 111,543.80 77,941.20 >>>> 3 63,518.40 102,543.80 76,201.40 111622.4 >>>> >>>> >>>> Root Cause: OS puts idle threads in mwait. In SNC4, the addresses for these OS mwaits were being assigned such they all (288 of them, one per thread) were getting mapped into the tag directories in cluster 0. This was overwhelming the mechanism architected to handle mwait wakeup, causing significant slowdowns for the memory traffic getting routed through the tag directories in cluster 0. We don’t see this behavior for Quadrant and All-to-all modes because the mwait addresses are distributed to all tag directories, and therefore the mwait mechanisms do not get overwhelmed. >>>> Fix: OS patch has been released that distributes the OS mwait lines more uniformly over all clusters in SNC4, preventing this issue. >>>> Link: (https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/kern…) >>>> >>>> >>>> Issue 2: In SNC4, the traffic targeting only remote clusters show much lower BW compared to the traffic targeting only local clusters >>>> Node 4 5 6 7 >>>> 0 132,090 107,477 120,639 72,496 >>>> 1 111,967 132,738 84,004 123,338 >>>> 2 111,637 84,380 125,716 96,188 >>>> 3 86,456 120,387 100,434 125,238 >>>> >>>> Root Cause: KNL follows YX routing for sending messages on the mesh. In SNC4 mode, when targeting all traffic to a remote cluster, the traffic (both address and data) predominantly flows in one direction: for example same Y and same X direction to go to the remote cluster that is diagonally across. This largely unidirectional flow cuts down effective BW on the mesh and causes congestion that results on further slowdowns. Mesh resources are provisioned for uniform distribution of traffic, such as those seen in Quadrant, all-to-all modes and local traffic in SNC4 mode. It is not provisioned for predominantly remote traffic in SNC4 mode. >>>> Fix: No HW or SW fix for this. Usage model needs to avoid drawing most of the traffic from remote clusters in SNC4 mode. They should keep the traffic local. If the access pattern is such that large amounts of remote traffic cannot be avoided then consider using Quadrant mode. >>> >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] Resolution to the two ANL issues with SNC4
by Allen, Benjamin S. 25 May '16

25 May '16

Cleaned up spacing to match the kernel coding standard. Ben

1 0

Re: [JLSE-Intel-NDA] Resolution to the two ANL issues with SNC4
by Allen, Benjamin S. 25 May '16

25 May '16

I'll test this on knl00 tomorrow. I created a backport to the Intel provided MPSP RHEL7.2 kernel as attached. I've booted a kernel with this patch on a Haswell box successfully so far. If anyone has some spare cycles please review it against the mainline patch https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/kern…. Kumar, feel free to pass this patch back to Avinash for review. Thanks, Ben

1 0

[JLSE-Intel-NDA] Fwd: Resolution to the two ANL issues with SNC4
by Kumaran, Kalyan 25 May '16

25 May '16

Ben, fyi. Begin forwarded message: From: "Sodani, Avinash" <avinash.sodani(a)intel.com<mailto:[email protected]>> Subject: Resolution to the two ANL issues with SNC4 Date: May 25, 2016 at 3:42:09 PM CDT To: "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov<mailto:[email protected]>>, "Harms, Kevin" <harms(a)alcf.anl.gov<mailto:[email protected]>>, "Parker, Scott J." <sparker(a)alcf.anl.gov<mailto:[email protected]>> Cc: "Kleen, Andi" <andi.kleen(a)intel.com<mailto:[email protected]>>, "Chinthamani, Sundaram" <sundaram.chinthamani(a)intel.com<mailto:[email protected]>>, "Sodani, Avinash" <avinash.sodani(a)intel.com<mailto:[email protected]>>, "Allison, CM" <cm.allison(a)intel.com<mailto:[email protected]>>, "Kirkendall, Keith G" <keith.g.kirkendall(a)intel.com<mailto:[email protected]>>, "Chaffin, Benjamin C" <benjamin.c.chaffin(a)intel.com<mailto:[email protected]>> Hi Kalyan, Here is the summary of the resolution to the two SNC4 issues that we were discussing in our meetings. The link to upstream OS patch is attached. Andi will provide any pointers to MPSP distributions as necessary. Thanks, -Avinash Issue 1: “Cluster 0/4” in SNC4 does not get full bandwidth even for local accesses. Memory NUMA Domain 4 5 6 7 CPU NUMA Domain 0 73,475.60 84,302.20 107,423.70 55,924.00 1 65,511.00 112,528.40 55,557.00 77,935.00 2 60,390.10 67,572.70 111,543.80 77,941.20 3 63,518.40 102,543.80 76,201.40 111622.4 Root Cause: OS puts idle threads in mwait. In SNC4, the addresses for these OS mwaits were being assigned such they all (288 of them, one per thread) were getting mapped into the tag directories in cluster 0. This was overwhelming the mechanism architected to handle mwait wakeup, causing significant slowdowns for the memory traffic getting routed through the tag directories in cluster 0. We don’t see this behavior for Quadrant and All-to-all modes because the mwait addresses are distributed to all tag directories, and therefore the mwait mechanisms do not get overwhelmed. Fix: OS patch has been released that distributes the OS mwait lines more uniformly over all clusters in SNC4, preventing this issue. Link: (https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/kern…) Issue 2: In SNC4, the traffic targeting only remote clusters show much lower BW compared to the traffic targeting only local clusters Node 4 5 6 7 0 132,090 107,477 120,639 72,496 1 111,967 132,738 84,004 123,338 2 111,637 84,380 125,716 96,188 3 86,456 120,387 100,434 125,238 Root Cause: KNL follows YX routing for sending messages on the mesh. In SNC4 mode, when targeting all traffic to a remote cluster, the traffic (both address and data) predominantly flows in one direction: for example same Y and same X direction to go to the remote cluster that is diagonally across. This largely unidirectional flow cuts down effective BW on the mesh and causes congestion that results on further slowdowns. Mesh resources are provisioned for uniform distribution of traffic, such as those seen in Quadrant, all-to-all modes and local traffic in SNC4 mode. It is not provisioned for predominantly remote traffic in SNC4 mode. Fix: No HW or SW fix for this. Usage model needs to avoid drawing most of the traffic from remote clusters in SNC4 mode. They should keep the traffic local. If the access pattern is such that large amounts of remote traffic cannot be avoided then consider using Quadrant mode.

1 0

Re: [JLSE-Intel-NDA] KNL00 BIOS Update
by Scott Parker 23 May '16

23 May '16

Yes, I was following the power-off / power-on procedure. -Scott On 5/23/16 5:20 PM, Allen, Benjamin S. wrote: > Fabien, > > I suspect that Scott was powering the machine off on each mode change, then using IPMI to power it back up. We know that the warm reboot issue remains in this BIOS, so this procedure is still needed. As such you'll never actually warm reboot into SNC-4 and experience the hang. Scott correct me if I'm wrong here in what you tried. > > Ben > >> On May 23, 2016, at 4:20 PM, Delalondre Fabien Jonathan <fabien.delalondre(a)epfl.ch> wrote: >> >> Ok thanks. >> Does this sentence on the wiki still applies ? "Please note that currently [..] reboot into SNC mode hangs" >> >> Best, >> >> Fabien >> ________________________________________ >> From: Scott Parker <sparker(a)anl.gov> >> Sent: Monday, May 23, 2016 11:16 PM >> To: Delalondre Fabien Jonathan; Allen, Benjamin S.; intel-nda(a)lists.jlse.anl.gov >> Subject: Re: [JLSE-Intel-NDA] KNL00 BIOS Update >> >> Nothing significant - there was a small bump for flat-quad mode from a >> previously measure peak of 435 to 449 GB/s >> but this is only 3% and more or less in the noise range of the >> measurements. SNC-4 mode continues to exhibit problems >> with one of the NUMA domains being slower and peak bandwidth of 440 GB/s >> only achievable with 128 threads (vs 64 threads >> for peak of very close to peak in other modes). Cache mode wasn't tested. >> >> -Scott >> >> >> On 5/23/16 4:08 PM, Delalondre Fabien Jonathan wrote: >>> Did you observe any performance improvements for STREAM ? >>> >>> Cheers, >>> >>> Fabien >>> ________________________________________ >>> From: intel-nda-bounces(a)lists.jlse.anl.gov <intel-nda-bounces(a)lists.jlse.anl.gov> on behalf of Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >>> Sent: Monday, May 23, 2016 11:05 PM >>> To: intel-nda(a)lists.jlse.anl.gov >>> Subject: Re: [JLSE-Intel-NDA] KNL00 BIOS Update >>> >>> We're done testing. Sorry for taking the machine away for a bit longer than expected. >>> >>> Ben >>> >>>> On May 23, 2016, at 3:03 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>> >>>> This work is done. Note, we're still doing a few benchmarks to see if the BIOS has fixed any of our standing issues. >>>> >>>> Also, this BIOS update renamed the BIOS parameters for the Memory Model and Mode. They are now "Memory Mode" and "Cluster Mode". The Wiki has been updated to show this. >>>> >>>> A separate Intel Vtune 2016 copy has been installed under /soft/restricted/intel-vtune-nda. This install contains the Intel NDA KNL add-on. I've not tested it, so please let me know if it works. I have compiled the sep4_0 kernel module from this VTune install, and set knl00 the load it on boot. >>>> >>>> Ben >>>> >>>> >>>>> On May 23, 2016, at 9:20 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>>>> >>>>> I'll be installing BIOS 01.01.0132, ME03.01.03.012.0, BMC FW 00.17.9699, and FRUSDR 1.14 later today. Release notes attached. >>>>> >>>>> Ben >>>>> >>>>> <BIOS Release Notes 01.01.0132.txt><BMC Release Notes 00.17.9699.txt><FRUSDR Release Notes 1.14.txt>_______________________________________________ >>>>> Intel-nda mailing list >>>>> Intel-nda(a)lists.jlse.anl.gov >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>> _______________________________________________ >>>> Intel-nda mailing list >>>> Intel-nda(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0