Do you guys know if this patch also solves the hanging issue at reboot ? I d like to do some testing in SNC-4 mode and would like to avoid getting it stuck. Best, Fabien ________________________________________ From: [email protected] <[email protected]> on behalf of Allen, Benjamin S. <[email protected]> Sent: Thursday, May 26, 2016 11:58 PM To: Kumaran, Kalyan Cc: [email protected] Subject: Re: [JLSE-Intel-NDA] Resolution to the two ANL issues with SNC4 Kumar, Can you ask Andi and Avinash to integrate this into MPSP, as well as request RedHat to include the fix in probably what will end up being RHEL 7.3? Additionally SUSE 12 for Theta. Thanks, Ben
On May 26, 2016, at 4:52 PM, Allen, Benjamin S. <[email protected]> wrote:
Excellent. We'll continue running the patched kernel on knl00, which is "3.10.0-327.el7.mpsp_1.3.0.66_725fc629ff.x86_64".
Ben
On May 26, 2016, at 4:45 PM, Scott Parker <[email protected]> wrote:
The kernel patch appears to have resolved the SNC-4 mode memory bandwidth issue. Each numa domain is now showing ~112 GB/s (previously numa domain 4 was around 70-80 GB/s). Total memory bandwidth is ~440 GB/s with 64 threads (previously ~310 GB/s) and ~445 GB/s with 128 threads. These results now match expectations.
-Scott
On 5/25/16 8:37 PM, Kumaran, Kalyan wrote:
And Kleen said that the patch looks good.
On May 25, 2016, at 6:54 PM, Allen, Benjamin S. <[email protected]> wrote:
Cleaned up spacing to match the kernel coding standard.
Ben
<725fc629ff_3.10.0-327.el7_backport.patch>
On May 25, 2016, at 6:41 PM, Allen, Benjamin S. <[email protected]> wrote:
I'll test this on knl00 tomorrow. I created a backport to the Intel provided MPSP RHEL7.2 kernel as attached. I've booted a kernel with this patch on a Haswell box successfully so far.
If anyone has some spare cycles please review it against the mainline patch https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/kerne....
Kumar, feel free to pass this patch back to Avinash for review.
Thanks,
Ben
<725fc629ff_3.10.0-327.el7_backport.patch>
On May 25, 2016, at 3:47 PM, Kumaran, Kalyan <[email protected]> wrote:
Ben, fyi.
Begin forwarded message:
From: "Sodani, Avinash" <[email protected]> Subject: Resolution to the two ANL issues with SNC4 Date: May 25, 2016 at 3:42:09 PM CDT To: "Kumaran, Kalyan" <[email protected]>, "Harms, Kevin" <[email protected]>, "Parker, Scott J." <[email protected]> Cc: "Kleen, Andi" <[email protected]>, "Chinthamani, Sundaram" <[email protected]>, "Sodani, Avinash" <[email protected]>, "Allison, CM" <[email protected]>, "Kirkendall, Keith G" <[email protected]>, "Chaffin, Benjamin C" <[email protected]>
Hi Kalyan,
Here is the summary of the resolution to the two SNC4 issues that we were discussing in our meetings. The link to upstream OS patch is attached. Andi will provide any pointers to MPSP distributions as necessary.
Thanks,
-Avinash
Issue 1: “Cluster 0/4” in SNC4 does not get full bandwidth even for local accesses.
Memory NUMA Domain 4 5 6 7 CPU NUMA Domain 0 73,475.60 84,302.20 107,423.70 55,924.00 1 65,511.00 112,528.40 55,557.00 77,935.00 2 60,390.10 67,572.70 111,543.80 77,941.20 3 63,518.40 102,543.80 76,201.40 111622.4
Root Cause: OS puts idle threads in mwait. In SNC4, the addresses for these OS mwaits were being assigned such they all (288 of them, one per thread) were getting mapped into the tag directories in cluster 0. This was overwhelming the mechanism architected to handle mwait wakeup, causing significant slowdowns for the memory traffic getting routed through the tag directories in cluster 0. We don’t see this behavior for Quadrant and All-to-all modes because the mwait addresses are distributed to all tag directories, and therefore the mwait mechanisms do not get overwhelmed. Fix: OS patch has been released that distributes the OS mwait lines more uniformly over all clusters in SNC4, preventing this issue. Link: (https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/kerne...)
Issue 2: In SNC4, the traffic targeting only remote clusters show much lower BW compared to the traffic targeting only local clusters Node 4 5 6 7 0 132,090 107,477 120,639 72,496 1 111,967 132,738 84,004 123,338 2 111,637 84,380 125,716 96,188 3 86,456 120,387 100,434 125,238
Root Cause: KNL follows YX routing for sending messages on the mesh. In SNC4 mode, when targeting all traffic to a remote cluster, the traffic (both address and data) predominantly flows in one direction: for example same Y and same X direction to go to the remote cluster that is diagonally across. This largely unidirectional flow cuts down effective BW on the mesh and causes congestion that results on further slowdowns. Mesh resources are provisioned for uniform distribution of traffic, such as those seen in Quadrant, all-to-all modes and local traffic in SNC4 mode. It is not provisioned for predominantly remote traffic in SNC4 mode. Fix: No HW or SW fix for this. Usage model needs to avoid drawing most of the traffic from remote clusters in SNC4 mode. They should keep the traffic local. If the access pattern is such that large amounts of remote traffic cannot be avoided then consider using Quadrant mode.
Intel-nda mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/intel-nda
Intel-nda mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/intel-nda
_______________________________________________ Intel-nda mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/intel-nda
_______________________________________________ Intel-nda mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/intel-nda