Sent from my iPhone Begin forwarded message: From: "Meadows, Lawrence F" <[email protected]<mailto:[email protected]>> Date: June 29, 2016 at 10:02:13 PM GMT+5:30 To: "Kumaran, Kalyan" <[email protected]<mailto:[email protected]>>, "Sodani, Avinash" <[email protected]<mailto:[email protected]>> Cc: "Finkel, Hal J." <[email protected]<mailto:[email protected]>>, "Harms, Kevin" <[email protected]<mailto:[email protected]>>, "Morozov, Vitali A." <[email protected]<mailto:[email protected]>>, "Kirkendall, Keith G" <[email protected]<mailto:[email protected]>>, "Allen, Benjamin S." <[email protected]<mailto:[email protected]>>, "Allison, CM" <[email protected]<mailto:[email protected]>>, "Valles, Antonio C" <[email protected]<mailto:[email protected]>> Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up The issue is that over time, the Linux free page list becomes effectively randomized, so a program will receive pages with physical addresses in several different 16 GiB regions, resulting in collisions in the direct-mapped MCDRAM cache. We have had success with a kernel patch that provides a sysctl to sort the free list. However, that patch is seen as a hack by the linux community and is proving difficult to upstream. We are investigating other approaches. The sorting patch requires that a utility be run as root before every job and does not completely mitigate the problem but does help a lot (effectively zero collisions up to 12-14 GiB of application memory usage). I am interested in ANL's views on using such a patch in your enviroment. -- Larry On 6/29/16, 9:21 AM, "Kumaran, Kalyan" <[email protected]<mailto:[email protected]>> wrote: Is this the same issue reported by Trinity folks? Sent from my iPhone On Jun 29, 2016, at 8:44 PM, Sodani, Avinash <[email protected]<mailto:[email protected]>> wrote: Ccing Larry and Antonio. -Avinash -----Original Message----- From: Hal Finkel [mailto:[email protected]] Sent: Tuesday, June 28, 2016 6:44 PM To: Kalyan Kumaran Cc: Kevin Harms; Vitali A. Morozov; Kirkendall, Keith G; Benjamin S. Allen; Allison, CM; Sodani, Avinash Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up ----- Original Message ----- From: "Kalyan Kumaran" <[email protected]<mailto:[email protected]>> To: "Hal J. Finkel" <[email protected]<mailto:[email protected]>>, "avinash sodani" <[email protected]<mailto:[email protected]>> Cc: "Kevin Harms" <[email protected]<mailto:[email protected]>>, "Vitali A. Morozov" <[email protected]<mailto:[email protected]>>, "keith g kirkendall" <[email protected]<mailto:[email protected]>>, "Benjamin S. Allen" <[email protected]<mailto:[email protected]>>, "cm allison" <[email protected]<mailto:[email protected]>> Sent: Tuesday, June 28, 2016 8:40:15 PM Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up Copying Avinash. Sent from my iPhone On Jun 29, 2016, at 6:12 AM, Hal Finkel <[email protected]<mailto:[email protected]>> wrote: Hi Kevin, Thanks! Also, from the release notes: Issue number: 0000003 Description: Application Performance can degrade over time when MCDRAM is used in cache mode Impact: The direct-mapped nature of MCDRAM-as-cache is by design. Real applications that need more than 16GB of memory will not exhibit this time-based degradation; it is really only a problem for benchmarking when running smaller datasets. Mitigation: Workaround is to reboot the node That seems scary. I believe we were considering booting the nodes in cache mode by default on Theta. Maybe that's a bad idea. Intel Tracking ID: 0001418 Affected OS: CentOS 7.2, RHEL 7.2, SLES 12, SLES 12 SP1 Description: [Tools] rpm rebuild not working Notes: Will not be Fixed; Not a defect Why not? -Hal ----- Original Message ----- From: "Kevin Harms" <[email protected]<mailto:[email protected]>> To: "Hal J. Finkel" <[email protected]<mailto:[email protected]>> Cc: "Vitali A. Morozov" <[email protected]<mailto:[email protected]>>, [email protected]<mailto:[email protected]> Sent: Tuesday, June 28, 2016 7:30:30 PM Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up I believe this kernel is just a set of selected backports from the main kernel. It’s possible this bug is related to a problem with PTE handling. Intel reported some bug where dirty pages were handled incorrectly under some conditions. That’s all the info we got. I haven’t seen the fix yet. The mpsp 1.3.3 is out as well, renamed xppsl_1.3.3. I installed this version. Same problem: Message from syslogd@kingly at Jun 28 20:14:29 ... kernel:BUG: soft lockup - CPU#223 stuck for 22s! [sh:24755] Message from syslogd@kingly at Jun 28 20:14:36 ... kernel:BUG: soft lockup - CPU#247 stuck for 22s! [as:25704] ... $ uname -a Linux kingly 3.10.0-327.13.1.el7.xppsl_1.3.3.151.x86_64 #1 SMP Fri Jun 10 15:04:35 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux -Hal https://software.intel.com/en-us/articles/intel-xeon-phi-processor-software http://registrationcenter-download.intel.com/akdlm/irc_nas/9325/xppsl-1.3.3-... kevin Hi Kevin, FWIW, no, the problem happened again with the new kernel: Message from syslogd@kingly at Jun 28 19:12:49 ... kernel:BUG: soft lockup - CPU#104 stuck for 23s! [cc1plus:77778] Message from syslogd@kingly at Jun 28 19:12:51 ... kernel:BUG: soft lockup - CPU#177 stuck for 23s! [cc1plus:77662] ... What's special about this kernel? Could I build a mainline kernel and expect it to work reasonably? -Hal ----- Original Message ----- From: "Hal Finkel" <[email protected]<mailto:[email protected]>> To: "Kevin Harms" <[email protected]<mailto:[email protected]>> Cc: "Vitali A. Morozov" <[email protected]<mailto:[email protected]>>, [email protected]<mailto:[email protected]> Sent: Tuesday, June 28, 2016 12:47:02 PM Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up Thanks! I'll upgrade. -Hal ----- Original Message ----- From: "Kevin Harms" <[email protected]<mailto:[email protected]>> To: "Hal J. Finkel" <[email protected]<mailto:[email protected]>>, "Vitali A. Morozov" <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]> Sent: Tuesday, June 28, 2016 11:59:20 AM Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up I updated this page to indicate how to upgrade to the mpsp 1.3.2 kernel. https://wiki.jlse.anl.gov/display/knl/Intel+DAP+Setup No idea, if that will fix this problem though :) kevin FYI: James and I have been experimenting with our box. We had to reset it after we hit some kernel bug: Message from syslogd@kingly at Jun 27 14:29:44 ... kernel:BUG: soft lockup - CPU#77 stuck for 22s! [cc1plus:15316] Message from syslogd@kingly at Jun 27 14:30:43 ... kernel:BUG: soft lockup - CPU#149 stuck for 23s! [cc1plus:14840] Message from syslogd@kingly at Jun 27 14:30:44 ... kernel:BUG: soft lockup - CPU#8 stuck for 23s! [cc1plus:15337] Message from syslogd@kingly at Jun 27 14:30:46 ... kernel:BUG: soft lockup - CPU#49 stuck for 22s! [as:16806] Message from syslogd@kingly at Jun 27 14:30:47 ... kernel:BUG: soft lockup - CPU#27 stuck for 22s! [cc1plus:16548] Message from syslogd@kingly at Jun 27 14:30:50 ... kernel:BUG: soft lockup - CPU#34 stuck for 22s! [cmake:16807] Message from syslogd@kingly at Jun 27 14:30:52 ... kernel:BUG: soft lockup - CPU#154 stuck for 22s! [cmake:16810] Message from syslogd@kingly at Jun 27 14:30:54 ... kernel:BUG: soft lockup - CPU#35 stuck for 23s! [cc1plus:15807] Message from syslogd@kingly at Jun 27 14:30:55 ... kernel:BUG: soft lockup - CPU#77 stuck for 22s! [cc1plus:15316] Message from syslogd@kingly at Jun 27 14:30:55 ... kernel:BUG: soft lockup - CPU#91 stuck for 22s! [cmake:16813] Message from syslogd@kingly at Jun 27 14:30:58 ... kernel:BUG: soft lockup - CPU#124 stuck for 23s! [make:16811] Message from syslogd@kingly at Jun 27 14:31:00 ... kernel:BUG: soft lockup - CPU#149 stuck for 23s! [cc1plus:14840] ... (this went on for a while) The kernel (3.10.0-327.el7.centos.mpsp_1.3.1.45.x86_64) seems like some special build? -Hal ----- Original Message ----- From: "Vitali A. Morozov" <[email protected]<mailto:[email protected]>> To: "Kalyan Kumaran" <[email protected]<mailto:[email protected]>>, [email protected]<mailto:[email protected]> Sent: Thursday, June 23, 2016 10:47:37 AM Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up I have updated the Ethernet port issue that I have encountered. Vitali On 06/23/2016 10:42 AM, Kumaran, Kalyan wrote: For folks setting these up please see instructions at: https://wiki.jlse.anl.gov/display/knl/Intel+DAP+Setup Kevin, Scott and Vitali will be adding information as they set the machines up. Others are welcome to add stuff. -Kumar _______________________________________________ Intel-nda mailing list [email protected]<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list [email protected]<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory _______________________________________________ Intel-nda mailing list [email protected]<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory _______________________________________________ Intel-nda mailing list [email protected]<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory _______________________________________________ Intel-nda mailing list [email protected]<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory