The issue is that over time, the Linux free page list becomes effectively randomized, so a program will receive pages with physical addresses in several different 16 GiB regions, resulting in collisions in the direct-mapped MCDRAM cache.

We have had success with a kernel patch that provides a sysctl to sort the free list. However, that patch is seen as a hack by the linux community and is proving difficult to upstream. We are investigating other approaches.

The sorting patch requires that a utility be run as root before every job and does not completely mitigate the problem but does help a lot (effectively zero collisions up to 12-14 GiB of application memory usage).

I am interested in ANL's views on using such a patch in your enviroment.

-- Larry

On 6/29/16, 9:21 AM, "Kumaran, Kalyan" <kumaran@alcf.anl.gov> wrote:

Is this the same issue reported by Trinity folks?

Sent from my iPhone

On Jun 29, 2016, at 8:44 PM, Sodani, Avinash <avinash.sodani@intel.com> wrote:

Ccing Larry and Antonio.

-Avinash

-----Original Message-----

From: Hal Finkel [mailto:hfinkel@anl.gov]

Sent: Tuesday, June 28, 2016 6:44 PM

To: Kalyan Kumaran

Cc: Kevin Harms; Vitali A. Morozov; Kirkendall, Keith G; Benjamin S. Allen; Allison, CM; Sodani, Avinash

Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up

----- Original Message -----

From: "Kalyan Kumaran" <kumaran@alcf.anl.gov>

To: "Hal J. Finkel" <hfinkel@anl.gov>, "avinash sodani" <avinash.sodani@intel.com>

Cc: "Kevin Harms" <harms@alcf.anl.gov>, "Vitali A. Morozov" <morozov@alcf.anl.gov>, "keith g kirkendall"

<keith.g.kirkendall@intel.com>, "Benjamin S. Allen" <bsallen@alcf.anl.gov>, "cm allison" <cm.allison@intel.com>

Sent: Tuesday, June 28, 2016 8:40:15 PM

Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up

Copying Avinash.

Sent from my iPhone

On Jun 29, 2016, at 6:12 AM, Hal Finkel <hfinkel@anl.gov> wrote:

Hi Kevin,

Thanks!

Also, from the release notes:

Issue number: 0000003

Description: Application Performance can degrade over time

when

MCDRAM is used in cache mode

Impact: The direct-mapped nature of MCDRAM-as-cache is

by

design. Real applications that need more than

16GB

of memory will not exhibit this time-based

degradation; it is really only a problem for

benchmarking when running smaller datasets.

Mitigation: Workaround is to reboot the node

That seems scary. I believe we were considering booting the nodes

in cache mode by default on Theta. Maybe that's a bad idea.

Intel Tracking ID: 0001418

Affected OS: CentOS 7.2, RHEL 7.2, SLES 12, SLES 12 SP1

Description: [Tools] rpm rebuild not working

Notes: Will not be Fixed; Not a defect

Why not?

-Hal

----- Original Message -----

From: "Kevin Harms" <harms@alcf.anl.gov>

To: "Hal J. Finkel" <hfinkel@anl.gov>

Cc: "Vitali A. Morozov" <morozov@alcf.anl.gov>,

intel-nda@lists.jlse.anl.gov

Sent: Tuesday, June 28, 2016 7:30:30 PM

Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up

I believe this kernel is just a set of selected backports from

the

main kernel. It’s possible this bug is related to a problem with

PTE handling. Intel reported some bug where dirty pages were

handled incorrectly under some conditions. That’s all the info we

got. I haven’t seen the fix yet. The mpsp 1.3.3 is out as well,

renamed xppsl_1.3.3.

I installed this version. Same problem:

Message from syslogd@kingly at Jun 28 20:14:29 ...

kernel:BUG: soft lockup - CPU#223 stuck for 22s! [sh:24755]

Message from syslogd@kingly at Jun 28 20:14:36 ...

kernel:BUG: soft lockup - CPU#247 stuck for 22s! [as:25704]

...

$ uname -a

Linux kingly 3.10.0-327.13.1.el7.xppsl_1.3.3.151.x86_64 #1 SMP Fri Jun 10 15:04:35 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

-Hal

https://software.intel.com/en-us/articles/intel-xeon-phi-processor-software

http://registrationcenter-download.intel.com/akdlm/irc_nas/9325/xppsl-1.3.3-releasenotes-linux.txt

kevin

Hi Kevin,

FWIW, no, the problem happened again with the new kernel:

Message from syslogd@kingly at Jun 28 19:12:49 ...

kernel:BUG: soft lockup - CPU#104 stuck for 23s! [cc1plus:77778]

Message from syslogd@kingly at Jun 28 19:12:51 ...

kernel:BUG: soft lockup - CPU#177 stuck for 23s! [cc1plus:77662]

...

What's special about this kernel? Could I build a mainline kernel

and expect it to work reasonably?

-Hal

----- Original Message -----

From: "Hal Finkel" <hfinkel@anl.gov>

To: "Kevin Harms" <harms@alcf.anl.gov>

Cc: "Vitali A. Morozov" <morozov@alcf.anl.gov>,

intel-nda@lists.jlse.anl.gov

Sent: Tuesday, June 28, 2016 12:47:02 PM

Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up

Thanks! I'll upgrade.

-Hal

----- Original Message -----

From: "Kevin Harms" <harms@alcf.anl.gov>

To: "Hal J. Finkel" <hfinkel@anl.gov>, "Vitali A. Morozov"

<morozov@alcf.anl.gov>

Cc: intel-nda@lists.jlse.anl.gov

Sent: Tuesday, June 28, 2016 11:59:20 AM

Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up

I updated this page to indicate how to upgrade to the mpsp

1.3.2

kernel. https://wiki.jlse.anl.gov/display/knl/Intel+DAP+Setup

No idea, if that will fix this problem though :)

kevin

FYI: James and I have been experimenting with our box. We had

to

reset it after we hit some kernel bug:

Message from syslogd@kingly at Jun 27 14:29:44 ...

kernel:BUG: soft lockup - CPU#77 stuck for 22s!

[cc1plus:15316]

Message from syslogd@kingly at Jun 27 14:30:43 ...

kernel:BUG: soft lockup - CPU#149 stuck for 23s!

[cc1plus:14840]

Message from syslogd@kingly at Jun 27 14:30:44 ...

kernel:BUG: soft lockup - CPU#8 stuck for 23s! [cc1plus:15337]

Message from syslogd@kingly at Jun 27 14:30:46 ...

kernel:BUG: soft lockup - CPU#49 stuck for 22s! [as:16806]

Message from syslogd@kingly at Jun 27 14:30:47 ...

kernel:BUG: soft lockup - CPU#27 stuck for 22s!

[cc1plus:16548]

Message from syslogd@kingly at Jun 27 14:30:50 ...

kernel:BUG: soft lockup - CPU#34 stuck for 22s! [cmake:16807]

Message from syslogd@kingly at Jun 27 14:30:52 ...

kernel:BUG: soft lockup - CPU#154 stuck for 22s! [cmake:16810]

Message from syslogd@kingly at Jun 27 14:30:54 ...

kernel:BUG: soft lockup - CPU#35 stuck for 23s!

[cc1plus:15807]

Message from syslogd@kingly at Jun 27 14:30:55 ...

kernel:BUG: soft lockup - CPU#77 stuck for 22s!

[cc1plus:15316]

Message from syslogd@kingly at Jun 27 14:30:55 ...

kernel:BUG: soft lockup - CPU#91 stuck for 22s! [cmake:16813]

Message from syslogd@kingly at Jun 27 14:30:58 ...

kernel:BUG: soft lockup - CPU#124 stuck for 23s! [make:16811]

Message from syslogd@kingly at Jun 27 14:31:00 ...

kernel:BUG: soft lockup - CPU#149 stuck for 23s!

[cc1plus:14840]

... (this went on for a while)

The kernel (3.10.0-327.el7.centos.mpsp_1.3.1.45.x86_64) seems

like

some special build?

-Hal

----- Original Message -----

From: "Vitali A. Morozov" <morozov@anl.gov>

To: "Kalyan Kumaran" <kumaran@alcf.anl.gov>,

intel-nda@lists.jlse.anl.gov

Sent: Thursday, June 23, 2016 10:47:37 AM

Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up

I have updated the Ethernet port issue that I have

encountered.

Vitali

On 06/23/2016 10:42 AM, Kumaran, Kalyan wrote:

For folks setting these up please see instructions at:

https://wiki.jlse.anl.gov/display/knl/Intel+DAP+Setup

Kevin, Scott and Vitali will be adding information as they

set

the

machines up. Others are welcome to add stuff.

-Kumar

_______________________________________________

Intel-nda mailing list Intel-nda@lists.jlse.anl.gov

https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

_______________________________________________

Intel-nda mailing list

Intel-nda@lists.jlse.anl.gov

https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

--

Hal Finkel

Assistant Computational Scientist

Leadership Computing Facility

Argonne National Laboratory

_______________________________________________

Intel-nda mailing list

Intel-nda@lists.jlse.anl.gov

https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

--

Hal Finkel

Assistant Computational Scientist

Leadership Computing Facility

Argonne National Laboratory

_______________________________________________

Intel-nda mailing list

Intel-nda@lists.jlse.anl.gov

https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

--

Hal Finkel

Assistant Computational Scientist

Leadership Computing Facility

Argonne National Laboratory

--

Hal Finkel

Assistant Computational Scientist

Leadership Computing Facility

Argonne National Laboratory

_______________________________________________

Intel-nda mailing list

Intel-nda@lists.jlse.anl.gov

https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

--

Hal Finkel

Assistant Computational Scientist

Leadership Computing Facility

Argonne National Laboratory