- Intel-nda - mailman3.cels.anl.gov

[JLSE-Intel-NDA] Fwd: Intel DAP boxes set up
by Kumaran, Kalyan 29 Jun '16

29 Jun '16

Sent from my iPhone Begin forwarded message: From: "Meadows, Lawrence F" <lawrence.f.meadows(a)intel.com<mailto:[email protected]>> Date: June 29, 2016 at 10:02:13 PM GMT+5:30 To: "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov<mailto:[email protected]>>, "Sodani, Avinash" <avinash.sodani(a)intel.com<mailto:[email protected]>> Cc: "Finkel, Hal J." <hfinkel(a)anl.gov<mailto:[email protected]>>, "Harms, Kevin" <harms(a)alcf.anl.gov<mailto:[email protected]>>, "Morozov, Vitali A." <morozov(a)alcf.anl.gov<mailto:[email protected]>>, "Kirkendall, Keith G" <keith.g.kirkendall(a)intel.com<mailto:[email protected]>>, "Allen, Benjamin S." <bsallen(a)alcf.anl.gov<mailto:[email protected]>>, "Allison, CM" <cm.allison(a)intel.com<mailto:[email protected]>>, "Valles, Antonio C" <antonio.c.valles(a)intel.com<mailto:[email protected]>> Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up The issue is that over time, the Linux free page list becomes effectively randomized, so a program will receive pages with physical addresses in several different 16 GiB regions, resulting in collisions in the direct-mapped MCDRAM cache. We have had success with a kernel patch that provides a sysctl to sort the free list. However, that patch is seen as a hack by the linux community and is proving difficult to upstream. We are investigating other approaches. The sorting patch requires that a utility be run as root before every job and does not completely mitigate the problem but does help a lot (effectively zero collisions up to 12-14 GiB of application memory usage). I am interested in ANL's views on using such a patch in your enviroment. -- Larry On 6/29/16, 9:21 AM, "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov<mailto:[email protected]>> wrote: Is this the same issue reported by Trinity folks? Sent from my iPhone On Jun 29, 2016, at 8:44 PM, Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>> wrote: Ccing Larry and Antonio. -Avinash -----Original Message----- From: Hal Finkel [mailto:[email protected]] Sent: Tuesday, June 28, 2016 6:44 PM To: Kalyan Kumaran Cc: Kevin Harms; Vitali A. Morozov; Kirkendall, Keith G; Benjamin S. Allen; Allison, CM; Sodani, Avinash Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up ----- Original Message ----- From: "Kalyan Kumaran" <kumaran(a)alcf.anl.gov<mailto:[email protected]>> To: "Hal J. Finkel" <hfinkel(a)anl.gov<mailto:[email protected]>>, "avinash sodani" <avinash.sodani(a)intel.com<mailto:[email protected]>> Cc: "Kevin Harms" <harms(a)alcf.anl.gov<mailto:[email protected]>>, "Vitali A. Morozov" <morozov(a)alcf.anl.gov<mailto:[email protected]>>, "keith g kirkendall" <keith.g.kirkendall(a)intel.com<mailto:[email protected]>>, "Benjamin S. Allen" <bsallen(a)alcf.anl.gov<mailto:[email protected]>>, "cm allison" <cm.allison(a)intel.com<mailto:[email protected]>> Sent: Tuesday, June 28, 2016 8:40:15 PM Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up Copying Avinash. Sent from my iPhone On Jun 29, 2016, at 6:12 AM, Hal Finkel <hfinkel(a)anl.gov<mailto:[email protected]>> wrote: Hi Kevin, Thanks! Also, from the release notes: Issue number: 0000003 Description: Application Performance can degrade over time when MCDRAM is used in cache mode Impact: The direct-mapped nature of MCDRAM-as-cache is by design. Real applications that need more than 16GB of memory will not exhibit this time-based degradation; it is really only a problem for benchmarking when running smaller datasets. Mitigation: Workaround is to reboot the node That seems scary. I believe we were considering booting the nodes in cache mode by default on Theta. Maybe that's a bad idea. Intel Tracking ID: 0001418 Affected OS: CentOS 7.2, RHEL 7.2, SLES 12, SLES 12 SP1 Description: [Tools] rpm rebuild not working Notes: Will not be Fixed; Not a defect Why not? -Hal ----- Original Message ----- From: "Kevin Harms" <harms(a)alcf.anl.gov<mailto:[email protected]>> To: "Hal J. Finkel" <hfinkel(a)anl.gov<mailto:[email protected]>> Cc: "Vitali A. Morozov" <morozov(a)alcf.anl.gov<mailto:[email protected]>>, intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> Sent: Tuesday, June 28, 2016 7:30:30 PM Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up I believe this kernel is just a set of selected backports from the main kernel. It’s possible this bug is related to a problem with PTE handling. Intel reported some bug where dirty pages were handled incorrectly under some conditions. That’s all the info we got. I haven’t seen the fix yet. The mpsp 1.3.3 is out as well, renamed xppsl_1.3.3. I installed this version. Same problem: Message from syslogd@kingly at Jun 28 20:14:29 ... kernel:BUG: soft lockup - CPU#223 stuck for 22s! [sh:24755] Message from syslogd@kingly at Jun 28 20:14:36 ... kernel:BUG: soft lockup - CPU#247 stuck for 22s! [as:25704] ... $ uname -a Linux kingly 3.10.0-327.13.1.el7.xppsl_1.3.3.151.x86_64 #1 SMP Fri Jun 10 15:04:35 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux -Hal https://software.intel.com/en-us/articles/intel-xeon-phi-processor-software http://registrationcenter-download.intel.com/akdlm/irc_nas/9325/xppsl-1.3.3… kevin Hi Kevin, FWIW, no, the problem happened again with the new kernel: Message from syslogd@kingly at Jun 28 19:12:49 ... kernel:BUG: soft lockup - CPU#104 stuck for 23s! [cc1plus:77778] Message from syslogd@kingly at Jun 28 19:12:51 ... kernel:BUG: soft lockup - CPU#177 stuck for 23s! [cc1plus:77662] ... What's special about this kernel? Could I build a mainline kernel and expect it to work reasonably? -Hal ----- Original Message ----- From: "Hal Finkel" <hfinkel(a)anl.gov<mailto:[email protected]>> To: "Kevin Harms" <harms(a)alcf.anl.gov<mailto:[email protected]>> Cc: "Vitali A. Morozov" <morozov(a)alcf.anl.gov<mailto:[email protected]>>, intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> Sent: Tuesday, June 28, 2016 12:47:02 PM Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up Thanks! I'll upgrade. -Hal ----- Original Message ----- From: "Kevin Harms" <harms(a)alcf.anl.gov<mailto:[email protected]>> To: "Hal J. Finkel" <hfinkel(a)anl.gov<mailto:[email protected]>>, "Vitali A. Morozov" <morozov(a)alcf.anl.gov<mailto:[email protected]>> Cc: intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> Sent: Tuesday, June 28, 2016 11:59:20 AM Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up I updated this page to indicate how to upgrade to the mpsp 1.3.2 kernel. https://wiki.jlse.anl.gov/display/knl/Intel+DAP+Setup No idea, if that will fix this problem though :) kevin FYI: James and I have been experimenting with our box. We had to reset it after we hit some kernel bug: Message from syslogd@kingly at Jun 27 14:29:44 ... kernel:BUG: soft lockup - CPU#77 stuck for 22s! [cc1plus:15316] Message from syslogd@kingly at Jun 27 14:30:43 ... kernel:BUG: soft lockup - CPU#149 stuck for 23s! [cc1plus:14840] Message from syslogd@kingly at Jun 27 14:30:44 ... kernel:BUG: soft lockup - CPU#8 stuck for 23s! [cc1plus:15337] Message from syslogd@kingly at Jun 27 14:30:46 ... kernel:BUG: soft lockup - CPU#49 stuck for 22s! [as:16806] Message from syslogd@kingly at Jun 27 14:30:47 ... kernel:BUG: soft lockup - CPU#27 stuck for 22s! [cc1plus:16548] Message from syslogd@kingly at Jun 27 14:30:50 ... kernel:BUG: soft lockup - CPU#34 stuck for 22s! [cmake:16807] Message from syslogd@kingly at Jun 27 14:30:52 ... kernel:BUG: soft lockup - CPU#154 stuck for 22s! [cmake:16810] Message from syslogd@kingly at Jun 27 14:30:54 ... kernel:BUG: soft lockup - CPU#35 stuck for 23s! [cc1plus:15807] Message from syslogd@kingly at Jun 27 14:30:55 ... kernel:BUG: soft lockup - CPU#77 stuck for 22s! [cc1plus:15316] Message from syslogd@kingly at Jun 27 14:30:55 ... kernel:BUG: soft lockup - CPU#91 stuck for 22s! [cmake:16813] Message from syslogd@kingly at Jun 27 14:30:58 ... kernel:BUG: soft lockup - CPU#124 stuck for 23s! [make:16811] Message from syslogd@kingly at Jun 27 14:31:00 ... kernel:BUG: soft lockup - CPU#149 stuck for 23s! [cc1plus:14840] ... (this went on for a while) The kernel (3.10.0-327.el7.centos.mpsp_1.3.1.45.x86_64) seems like some special build? -Hal ----- Original Message ----- From: "Vitali A. Morozov" <morozov(a)anl.gov<mailto:[email protected]>> To: "Kalyan Kumaran" <kumaran(a)alcf.anl.gov<mailto:[email protected]>>, intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> Sent: Thursday, June 23, 2016 10:47:37 AM Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up I have updated the Ethernet port issue that I have encountered. Vitali On 06/23/2016 10:42 AM, Kumaran, Kalyan wrote: For folks setting these up please see instructions at: https://wiki.jlse.anl.gov/display/knl/Intel+DAP+Setup Kevin, Scott and Vitali will be adding information as they set the machines up. Others are welcome to add stuff. -Kumar _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory

1 0

Re: [JLSE-Intel-NDA] Intel DAP boxes set up
by Vitali A. Morozov 29 Jun '16

29 Jun '16

Alternatively, we may partition the machine with a partition of all nodes in cache mode and a partition of all nodes in flat mode. The user job should specify where it wants to run and it gets scheduled there. The question on how many nodes we need in each partition is open. Vitali On 06/29/2016 11:32 AM, Scott Parker wrote: > > > That 15 minute boot time could be a real efficiency loss. The only way > we have to change the memory mode between cache > and flat is a reboot and I think we may see a mix of flat and cache > mode jobs on the system given the roughly 30% memory > bandwidth reduction seen in cache mode. Applications that can be made > to run in flat mode could see a real performance advantage > for doing so. So we may see a need for frequent reboots if even a > moderate fraction of our applications opt to use flat mode. > > -Scott > > On 6/29/16 11:15 AM, Susan Coghlan wrote: >> Unfortunately, a reboot is 15 mins. That's a lot of time so I think >> we don't want to reboot for each job. We've asked Cray and Intel to >> analyze and document where the time is spent during the boot to see >> if there is a way to reduce it, but for now we are stuck with 15 min >> boots. I do like the idea of allowing the user to request a reboot >> even if they are running in the default mode. That seems like it >> should be doable. >> >> -Susan >> >> >> Kamil Iskra wrote: >>> On Tue, Jun 28, 2016 at 19:42:29 -0500, Hal Finkel wrote: >>> >>>> Description: Application Performance can degrade over time when >>>> MCDRAM is used in cache mode >>>> Impact: The direct-mapped nature of MCDRAM-as-cache is by >>>> design. Real applications that need more than 16GB >>>> of memory will not exhibit this time-based >>>> degradation; it is really only a problem for >>>> benchmarking when running smaller datasets. >>>> Mitigation: Workaround is to reboot the node >>>> >>>> That seems scary. I believe we were considering booting the nodes in cache mode by default on Theta. Maybe that's a bad idea. >>> >>> If I understand it correctly then, this is due to physical memory >>> fragmentation of DDR, typical for page-based memory management as used in >>> Linux. This effect can be observed with any non-fully-associative cache >>> (including regular CPU caches), although it will be the worst for >>> one-way-associative (direct-mapped) one. >>> >>> For the most predictable, and generally best, performance, it may make >>> sense to consider rebooting nodes between each job on Theta, or at least >>> make node reboot a user-accessible option at job submission time... >>> >>> Kamil >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> >> >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Intel DAP boxes set up
by Scott Parker 29 Jun '16

29 Jun '16

That 15 minute boot time could be a real efficiency loss. The only way we have to change the memory mode between cache and flat is a reboot and I think we may see a mix of flat and cache mode jobs on the system given the roughly 30% memory bandwidth reduction seen in cache mode. Applications that can be made to run in flat mode could see a real performance advantage for doing so. So we may see a need for frequent reboots if even a moderate fraction of our applications opt to use flat mode. -Scott On 6/29/16 11:15 AM, Susan Coghlan wrote: > Unfortunately, a reboot is 15 mins. That's a lot of time so I think we > don't want to reboot for each job. We've asked Cray and Intel to > analyze and document where the time is spent during the boot to see if > there is a way to reduce it, but for now we are stuck with 15 min > boots. I do like the idea of allowing the user to request a reboot > even if they are running in the default mode. That seems like it > should be doable. > > -Susan > > > Kamil Iskra wrote: >> On Tue, Jun 28, 2016 at 19:42:29 -0500, Hal Finkel wrote: >> >>> Description: Application Performance can degrade over time when >>> MCDRAM is used in cache mode >>> Impact: The direct-mapped nature of MCDRAM-as-cache is by >>> design. Real applications that need more than 16GB >>> of memory will not exhibit this time-based >>> degradation; it is really only a problem for >>> benchmarking when running smaller datasets. >>> Mitigation: Workaround is to reboot the node >>> >>> That seems scary. I believe we were considering booting the nodes in cache mode by default on Theta. Maybe that's a bad idea. >> >> If I understand it correctly then, this is due to physical memory >> fragmentation of DDR, typical for page-based memory management as used in >> Linux. This effect can be observed with any non-fully-associative cache >> (including regular CPU caches), although it will be the worst for >> one-way-associative (direct-mapped) one. >> >> For the most predictable, and generally best, performance, it may make >> sense to consider rebooting nodes between each job on Theta, or at least >> make node reboot a user-accessible option at job submission time... >> >> Kamil >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Intel DAP boxes set up
by Susan Coghlan 29 Jun '16

29 Jun '16

Unfortunately, a reboot is 15 mins. That's a lot of time so I think we don't want to reboot for each job. We've asked Cray and Intel to analyze and document where the time is spent during the boot to see if there is a way to reduce it, but for now we are stuck with 15 min boots. I do like the idea of allowing the user to request a reboot even if they are running in the default mode. That seems like it should be doable. -Susan Kamil Iskra wrote: > On Tue, Jun 28, 2016 at 19:42:29 -0500, Hal Finkel wrote: > >> Description: Application Performance can degrade over time when >> MCDRAM is used in cache mode >> Impact: The direct-mapped nature of MCDRAM-as-cache is by >> design. Real applications that need more than 16GB >> of memory will not exhibit this time-based >> degradation; it is really only a problem for >> benchmarking when running smaller datasets. >> Mitigation: Workaround is to reboot the node >> >> That seems scary. I believe we were considering booting the nodes in cache mode by default on Theta. Maybe that's a bad idea. > > If I understand it correctly then, this is due to physical memory > fragmentation of DDR, typical for page-based memory management as used in > Linux. This effect can be observed with any non-fully-associative cache > (including regular CPU caches), although it will be the worst for > one-way-associative (direct-mapped) one. > > For the most predictable, and generally best, performance, it may make > sense to consider rebooting nodes between each job on Theta, or at least > make node reboot a user-accessible option at job submission time... > > Kamil > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Intel DAP boxes set up
by Scott Parker 29 Jun '16

29 Jun '16

This seem like something we should understand better. The descriptions is a little ambiguous, I wonder if this mean a single application running in cache mode for multiple hours will see performance degradation? Or it's the standard Linux memory fragmentation that occurs on systems over weeks or months of having numerous processes allocate and deallocate memory? Does it only occur when applications are in cache mode and use less that 16 GB? -Scott On 6/29/16 10:27 AM, Kamil Iskra wrote: > On Tue, Jun 28, 2016 at 19:42:29 -0500, Hal Finkel wrote: > >> Description: Application Performance can degrade over time when >> MCDRAM is used in cache mode >> Impact: The direct-mapped nature of MCDRAM-as-cache is by >> design. Real applications that need more than 16GB >> of memory will not exhibit this time-based >> degradation; it is really only a problem for >> benchmarking when running smaller datasets. >> Mitigation: Workaround is to reboot the node >> >> That seems scary. I believe we were considering booting the nodes in cache mode by default on Theta. Maybe that's a bad idea. > If I understand it correctly then, this is due to physical memory > fragmentation of DDR, typical for page-based memory management as used in > Linux. This effect can be observed with any non-fully-associative cache > (including regular CPU caches), although it will be the worst for > one-way-associative (direct-mapped) one. > > For the most predictable, and generally best, performance, it may make > sense to consider rebooting nodes between each job on Theta, or at least > make node reboot a user-accessible option at job submission time... > > Kamil > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Intel DAP boxes set up
by Swann Perarnau 29 Jun '16

29 Jun '16

On Wed, Jun 29, 2016 at 10:27:00AM -0500, Kamil Iskra wrote: > On Tue, Jun 28, 2016 at 19:42:29 -0500, Hal Finkel wrote: > > > Description: Application Performance can degrade over time when > > MCDRAM is used in cache mode > > Impact: The direct-mapped nature of MCDRAM-as-cache is by > > design. Real applications that need more than 16GB > > of memory will not exhibit this time-based > > degradation; it is really only a problem for > > benchmarking when running smaller datasets. > > Mitigation: Workaround is to reboot the node > > > > That seems scary. I believe we were considering booting the nodes in cache mode by default on Theta. Maybe that's a bad idea. > > If I understand it correctly then, this is due to physical memory > fragmentation of DDR, typical for page-based memory management as used in > Linux. This effect can be observed with any non-fully-associative cache > (including regular CPU caches), although it will be the worst for > one-way-associative (direct-mapped) one. > > For the most predictable, and generally best, performance, it may make > sense to consider rebooting nodes between each job on Theta, or at least > make node reboot a user-accessible option at job submission time... > If direct-mapped MCDRAM-as-cache is a long term feature of Xeon Phis, you could try to tell Intel to implement page coloring in Linux. Torvalds only killed 4 attempts at it in the last ten years :p -- Swann

1 0

Re: [JLSE-Intel-NDA] Intel DAP boxes set up
by Kamil Iskra 29 Jun '16

29 Jun '16

On Tue, Jun 28, 2016 at 19:42:29 -0500, Hal Finkel wrote: > Description: Application Performance can degrade over time when > MCDRAM is used in cache mode > Impact: The direct-mapped nature of MCDRAM-as-cache is by > design. Real applications that need more than 16GB > of memory will not exhibit this time-based > degradation; it is really only a problem for > benchmarking when running smaller datasets. > Mitigation: Workaround is to reboot the node > > That seems scary. I believe we were considering booting the nodes in cache mode by default on Theta. Maybe that's a bad idea. If I understand it correctly then, this is due to physical memory fragmentation of DDR, typical for page-based memory management as used in Linux. This effect can be observed with any non-fully-associative cache (including regular CPU caches), although it will be the worst for one-way-associative (direct-mapped) one. For the most predictable, and generally best, performance, it may make sense to consider rebooting nodes between each job on Theta, or at least make node reboot a user-accessible option at job submission time... Kamil

1 0

Re: [JLSE-Intel-NDA] Intel DAP boxes set up
by Hal Finkel 28 Jun '16

28 Jun '16

----- Original Message ----- > From: "Hal Finkel" <hfinkel(a)anl.gov> > To: "Vitali Morozov" <morozov(a)anl.gov> > Cc: "Vitali A. Morozov" <morozov(a)alcf.anl.gov>, intel-nda(a)lists.jlse.anl.gov > Sent: Tuesday, June 28, 2016 8:41:03 PM > Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up > > ----- Original Message ----- > > From: "Vitali Morozov" <morozov(a)anl.gov> > > To: "Hal Finkel" <hfinkel(a)anl.gov> > > Cc: "Kevin Harms" <harms(a)alcf.anl.gov>, "Vitali A. Morozov" > > <morozov(a)alcf.anl.gov>, intel-nda(a)lists.jlse.anl.gov > > Sent: Tuesday, June 28, 2016 8:29:17 PM > > Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up > > > > > > Interesting, what are you doing such that others do not? Why no one > > else > > reports it? > > I'm just building LLVM trunk with the system gcc. make -j256. Also, the build directory is in /dev/shm (source directory is in my home directory): $ df -h /dev/shm Filesystem Size Used Avail Use% Mounted on tmpfs 55G 3.1G 52G 6% /dev/shm That's how I build on the regular Xeons and on the POWER machines. It looks like there's still plenty of room. -Hal > > -Hal > > > > > Vitali > > > > > > On 06/28/2016 07:18 PM, Hal Finkel wrote: > > > Hi Kevin, > > > > > > FWIW, no, the problem happened again with the new kernel: > > > > > > Message from syslogd@kingly at Jun 28 19:12:49 ... > > > kernel:BUG: soft lockup - CPU#104 stuck for 23s! > > > [cc1plus:77778] > > > > > > Message from syslogd@kingly at Jun 28 19:12:51 ... > > > kernel:BUG: soft lockup - CPU#177 stuck for 23s! > > > [cc1plus:77662] > > > > > > ... > > > > > > What's special about this kernel? Could I build a mainline kernel > > > and expect it to work reasonably? > > > > > > -Hal > > > > > > ----- Original Message ----- > > >> From: "Hal Finkel" <hfinkel(a)anl.gov> > > >> To: "Kevin Harms" <harms(a)alcf.anl.gov> > > >> Cc: "Vitali A. Morozov" <morozov(a)alcf.anl.gov>, > > >> intel-nda(a)lists.jlse.anl.gov > > >> Sent: Tuesday, June 28, 2016 12:47:02 PM > > >> Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up > > >> > > >> Thanks! I'll upgrade. > > >> > > >> -Hal > > >> > > >> ----- Original Message ----- > > >>> From: "Kevin Harms" <harms(a)alcf.anl.gov> > > >>> To: "Hal J. Finkel" <hfinkel(a)anl.gov>, "Vitali A. Morozov" > > >>> <morozov(a)alcf.anl.gov> > > >>> Cc: intel-nda(a)lists.jlse.anl.gov > > >>> Sent: Tuesday, June 28, 2016 11:59:20 AM > > >>> Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up > > >>> > > >>> > > >>> I updated this page to indicate how to upgrade to the mpsp > > >>> 1.3.2 > > >>> kernel. > > >>> https://wiki.jlse.anl.gov/display/knl/Intel+DAP+Setup > > >>> > > >>> No idea, if that will fix this problem though :) > > >>> > > >>> kevin > > >>> > > >>>> FYI: James and I have been experimenting with our box. We had > > >>>> to > > >>>> reset it after we hit some kernel bug: > > >>>> > > >>>> Message from syslogd@kingly at Jun 27 14:29:44 ... > > >>>> kernel:BUG: soft lockup - CPU#77 stuck for 22s! > > >>>> [cc1plus:15316] > > >>>> > > >>>> Message from syslogd@kingly at Jun 27 14:30:43 ... > > >>>> kernel:BUG: soft lockup - CPU#149 stuck for 23s! > > >>>> [cc1plus:14840] > > >>>> > > >>>> Message from syslogd@kingly at Jun 27 14:30:44 ... > > >>>> kernel:BUG: soft lockup - CPU#8 stuck for 23s! [cc1plus:15337] > > >>>> > > >>>> Message from syslogd@kingly at Jun 27 14:30:46 ... > > >>>> kernel:BUG: soft lockup - CPU#49 stuck for 22s! [as:16806] > > >>>> > > >>>> Message from syslogd@kingly at Jun 27 14:30:47 ... > > >>>> kernel:BUG: soft lockup - CPU#27 stuck for 22s! > > >>>> [cc1plus:16548] > > >>>> > > >>>> Message from syslogd@kingly at Jun 27 14:30:50 ... > > >>>> kernel:BUG: soft lockup - CPU#34 stuck for 22s! [cmake:16807] > > >>>> > > >>>> Message from syslogd@kingly at Jun 27 14:30:52 ... > > >>>> kernel:BUG: soft lockup - CPU#154 stuck for 22s! [cmake:16810] > > >>>> > > >>>> Message from syslogd@kingly at Jun 27 14:30:54 ... > > >>>> kernel:BUG: soft lockup - CPU#35 stuck for 23s! > > >>>> [cc1plus:15807] > > >>>> > > >>>> Message from syslogd@kingly at Jun 27 14:30:55 ... > > >>>> kernel:BUG: soft lockup - CPU#77 stuck for 22s! > > >>>> [cc1plus:15316] > > >>>> > > >>>> Message from syslogd@kingly at Jun 27 14:30:55 ... > > >>>> kernel:BUG: soft lockup - CPU#91 stuck for 22s! [cmake:16813] > > >>>> > > >>>> Message from syslogd@kingly at Jun 27 14:30:58 ... > > >>>> kernel:BUG: soft lockup - CPU#124 stuck for 23s! [make:16811] > > >>>> > > >>>> Message from syslogd@kingly at Jun 27 14:31:00 ... > > >>>> kernel:BUG: soft lockup - CPU#149 stuck for 23s! > > >>>> [cc1plus:14840] > > >>>> > > >>>> ... (this went on for a while) > > >>>> > > >>>> The kernel (3.10.0-327.el7.centos.mpsp_1.3.1.45.x86_64) seems > > >>>> like > > >>>> some special build? > > >>>> > > >>>> -Hal > > >>>> > > >>>> ----- Original Message ----- > > >>>>> From: "Vitali A. Morozov" <morozov(a)anl.gov> > > >>>>> To: "Kalyan Kumaran" <kumaran(a)alcf.anl.gov>, > > >>>>> intel-nda(a)lists.jlse.anl.gov > > >>>>> Sent: Thursday, June 23, 2016 10:47:37 AM > > >>>>> Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up > > >>>>> > > >>>>> > > >>>>> > > >>>>> I have updated the Ethernet port issue that I have > > >>>>> encountered. > > >>>>> > > >>>>> Vitali > > >>>>> > > >>>>> On 06/23/2016 10:42 AM, Kumaran, Kalyan wrote: > > >>>>> > > >>>>> > > >>>>> For folks setting these up please see instructions at: > > >>>>> > > >>>>> > > >>>>> https://wiki.jlse.anl.gov/display/knl/Intel+DAP+Setup > > >>>>> > > >>>>> > > >>>>> > > >>>>> Kevin, Scott and Vitali will be adding information as they > > >>>>> set > > >>>>> the > > >>>>> machines up. Others are welcome to add stuff. > > >>>>> > > >>>>> > > >>>>> -Kumar > > >>>>> > > >>>>> _______________________________________________ > > >>>>> Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov > > >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > >>>>> > > >>>>> _______________________________________________ > > >>>>> Intel-nda mailing list > > >>>>> Intel-nda(a)lists.jlse.anl.gov > > >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > >>>>> > > >>>> -- > > >>>> Hal Finkel > > >>>> Assistant Computational Scientist > > >>>> Leadership Computing Facility > > >>>> Argonne National Laboratory > > >>>> _______________________________________________ > > >>>> Intel-nda mailing list > > >>>> Intel-nda(a)lists.jlse.anl.gov > > >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > >> -- > > >> Hal Finkel > > >> Assistant Computational Scientist > > >> Leadership Computing Facility > > >> Argonne National Laboratory > > >> _______________________________________________ > > >> Intel-nda mailing list > > >> Intel-nda(a)lists.jlse.anl.gov > > >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > >> > > > > > > -- > Hal Finkel > Assistant Computational Scientist > Leadership Computing Facility > Argonne National Laboratory > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory

1 0

Re: [JLSE-Intel-NDA] Intel DAP boxes set up
by Hal Finkel 28 Jun '16

28 Jun '16

----- Original Message ----- > From: "Vitali Morozov" <morozov(a)anl.gov> > To: "Hal Finkel" <hfinkel(a)anl.gov> > Cc: "Kevin Harms" <harms(a)alcf.anl.gov>, "Vitali A. Morozov" <morozov(a)alcf.anl.gov>, intel-nda(a)lists.jlse.anl.gov > Sent: Tuesday, June 28, 2016 8:29:17 PM > Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up > > > Interesting, what are you doing such that others do not? Why no one > else > reports it? I'm just building LLVM trunk with the system gcc. make -j256. -Hal > > Vitali > > > On 06/28/2016 07:18 PM, Hal Finkel wrote: > > Hi Kevin, > > > > FWIW, no, the problem happened again with the new kernel: > > > > Message from syslogd@kingly at Jun 28 19:12:49 ... > > kernel:BUG: soft lockup - CPU#104 stuck for 23s! [cc1plus:77778] > > > > Message from syslogd@kingly at Jun 28 19:12:51 ... > > kernel:BUG: soft lockup - CPU#177 stuck for 23s! [cc1plus:77662] > > > > ... > > > > What's special about this kernel? Could I build a mainline kernel > > and expect it to work reasonably? > > > > -Hal > > > > ----- Original Message ----- > >> From: "Hal Finkel" <hfinkel(a)anl.gov> > >> To: "Kevin Harms" <harms(a)alcf.anl.gov> > >> Cc: "Vitali A. Morozov" <morozov(a)alcf.anl.gov>, > >> intel-nda(a)lists.jlse.anl.gov > >> Sent: Tuesday, June 28, 2016 12:47:02 PM > >> Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up > >> > >> Thanks! I'll upgrade. > >> > >> -Hal > >> > >> ----- Original Message ----- > >>> From: "Kevin Harms" <harms(a)alcf.anl.gov> > >>> To: "Hal J. Finkel" <hfinkel(a)anl.gov>, "Vitali A. Morozov" > >>> <morozov(a)alcf.anl.gov> > >>> Cc: intel-nda(a)lists.jlse.anl.gov > >>> Sent: Tuesday, June 28, 2016 11:59:20 AM > >>> Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up > >>> > >>> > >>> I updated this page to indicate how to upgrade to the mpsp > >>> 1.3.2 > >>> kernel. https://wiki.jlse.anl.gov/display/knl/Intel+DAP+Setup > >>> > >>> No idea, if that will fix this problem though :) > >>> > >>> kevin > >>> > >>>> FYI: James and I have been experimenting with our box. We had to > >>>> reset it after we hit some kernel bug: > >>>> > >>>> Message from syslogd@kingly at Jun 27 14:29:44 ... > >>>> kernel:BUG: soft lockup - CPU#77 stuck for 22s! [cc1plus:15316] > >>>> > >>>> Message from syslogd@kingly at Jun 27 14:30:43 ... > >>>> kernel:BUG: soft lockup - CPU#149 stuck for 23s! [cc1plus:14840] > >>>> > >>>> Message from syslogd@kingly at Jun 27 14:30:44 ... > >>>> kernel:BUG: soft lockup - CPU#8 stuck for 23s! [cc1plus:15337] > >>>> > >>>> Message from syslogd@kingly at Jun 27 14:30:46 ... > >>>> kernel:BUG: soft lockup - CPU#49 stuck for 22s! [as:16806] > >>>> > >>>> Message from syslogd@kingly at Jun 27 14:30:47 ... > >>>> kernel:BUG: soft lockup - CPU#27 stuck for 22s! [cc1plus:16548] > >>>> > >>>> Message from syslogd@kingly at Jun 27 14:30:50 ... > >>>> kernel:BUG: soft lockup - CPU#34 stuck for 22s! [cmake:16807] > >>>> > >>>> Message from syslogd@kingly at Jun 27 14:30:52 ... > >>>> kernel:BUG: soft lockup - CPU#154 stuck for 22s! [cmake:16810] > >>>> > >>>> Message from syslogd@kingly at Jun 27 14:30:54 ... > >>>> kernel:BUG: soft lockup - CPU#35 stuck for 23s! [cc1plus:15807] > >>>> > >>>> Message from syslogd@kingly at Jun 27 14:30:55 ... > >>>> kernel:BUG: soft lockup - CPU#77 stuck for 22s! [cc1plus:15316] > >>>> > >>>> Message from syslogd@kingly at Jun 27 14:30:55 ... > >>>> kernel:BUG: soft lockup - CPU#91 stuck for 22s! [cmake:16813] > >>>> > >>>> Message from syslogd@kingly at Jun 27 14:30:58 ... > >>>> kernel:BUG: soft lockup - CPU#124 stuck for 23s! [make:16811] > >>>> > >>>> Message from syslogd@kingly at Jun 27 14:31:00 ... > >>>> kernel:BUG: soft lockup - CPU#149 stuck for 23s! [cc1plus:14840] > >>>> > >>>> ... (this went on for a while) > >>>> > >>>> The kernel (3.10.0-327.el7.centos.mpsp_1.3.1.45.x86_64) seems > >>>> like > >>>> some special build? > >>>> > >>>> -Hal > >>>> > >>>> ----- Original Message ----- > >>>>> From: "Vitali A. Morozov" <morozov(a)anl.gov> > >>>>> To: "Kalyan Kumaran" <kumaran(a)alcf.anl.gov>, > >>>>> intel-nda(a)lists.jlse.anl.gov > >>>>> Sent: Thursday, June 23, 2016 10:47:37 AM > >>>>> Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up > >>>>> > >>>>> > >>>>> > >>>>> I have updated the Ethernet port issue that I have encountered. > >>>>> > >>>>> Vitali > >>>>> > >>>>> On 06/23/2016 10:42 AM, Kumaran, Kalyan wrote: > >>>>> > >>>>> > >>>>> For folks setting these up please see instructions at: > >>>>> > >>>>> > >>>>> https://wiki.jlse.anl.gov/display/knl/Intel+DAP+Setup > >>>>> > >>>>> > >>>>> > >>>>> Kevin, Scott and Vitali will be adding information as they set > >>>>> the > >>>>> machines up. Others are welcome to add stuff. > >>>>> > >>>>> > >>>>> -Kumar > >>>>> > >>>>> _______________________________________________ > >>>>> Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov > >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > >>>>> > >>>>> _______________________________________________ > >>>>> Intel-nda mailing list > >>>>> Intel-nda(a)lists.jlse.anl.gov > >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > >>>>> > >>>> -- > >>>> Hal Finkel > >>>> Assistant Computational Scientist > >>>> Leadership Computing Facility > >>>> Argonne National Laboratory > >>>> _______________________________________________ > >>>> Intel-nda mailing list > >>>> Intel-nda(a)lists.jlse.anl.gov > >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > >> -- > >> Hal Finkel > >> Assistant Computational Scientist > >> Leadership Computing Facility > >> Argonne National Laboratory > >> _______________________________________________ > >> Intel-nda mailing list > >> Intel-nda(a)lists.jlse.anl.gov > >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > >> > > -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory

1 0

Re: [JLSE-Intel-NDA] Intel DAP boxes set up
by Vitali Morozov 28 Jun '16

28 Jun '16

Interesting, what are you doing such that others do not? Why no one else reports it? Vitali On 06/28/2016 07:18 PM, Hal Finkel wrote: > Hi Kevin, > > FWIW, no, the problem happened again with the new kernel: > > Message from syslogd@kingly at Jun 28 19:12:49 ... > kernel:BUG: soft lockup - CPU#104 stuck for 23s! [cc1plus:77778] > > Message from syslogd@kingly at Jun 28 19:12:51 ... > kernel:BUG: soft lockup - CPU#177 stuck for 23s! [cc1plus:77662] > > ... > > What's special about this kernel? Could I build a mainline kernel and expect it to work reasonably? > > -Hal > > ----- Original Message ----- >> From: "Hal Finkel" <hfinkel(a)anl.gov> >> To: "Kevin Harms" <harms(a)alcf.anl.gov> >> Cc: "Vitali A. Morozov" <morozov(a)alcf.anl.gov>, intel-nda(a)lists.jlse.anl.gov >> Sent: Tuesday, June 28, 2016 12:47:02 PM >> Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up >> >> Thanks! I'll upgrade. >> >> -Hal >> >> ----- Original Message ----- >>> From: "Kevin Harms" <harms(a)alcf.anl.gov> >>> To: "Hal J. Finkel" <hfinkel(a)anl.gov>, "Vitali A. Morozov" >>> <morozov(a)alcf.anl.gov> >>> Cc: intel-nda(a)lists.jlse.anl.gov >>> Sent: Tuesday, June 28, 2016 11:59:20 AM >>> Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up >>> >>> >>> I updated this page to indicate how to upgrade to the mpsp 1.3.2 >>> kernel. https://wiki.jlse.anl.gov/display/knl/Intel+DAP+Setup >>> >>> No idea, if that will fix this problem though :) >>> >>> kevin >>> >>>> FYI: James and I have been experimenting with our box. We had to >>>> reset it after we hit some kernel bug: >>>> >>>> Message from syslogd@kingly at Jun 27 14:29:44 ... >>>> kernel:BUG: soft lockup - CPU#77 stuck for 22s! [cc1plus:15316] >>>> >>>> Message from syslogd@kingly at Jun 27 14:30:43 ... >>>> kernel:BUG: soft lockup - CPU#149 stuck for 23s! [cc1plus:14840] >>>> >>>> Message from syslogd@kingly at Jun 27 14:30:44 ... >>>> kernel:BUG: soft lockup - CPU#8 stuck for 23s! [cc1plus:15337] >>>> >>>> Message from syslogd@kingly at Jun 27 14:30:46 ... >>>> kernel:BUG: soft lockup - CPU#49 stuck for 22s! [as:16806] >>>> >>>> Message from syslogd@kingly at Jun 27 14:30:47 ... >>>> kernel:BUG: soft lockup - CPU#27 stuck for 22s! [cc1plus:16548] >>>> >>>> Message from syslogd@kingly at Jun 27 14:30:50 ... >>>> kernel:BUG: soft lockup - CPU#34 stuck for 22s! [cmake:16807] >>>> >>>> Message from syslogd@kingly at Jun 27 14:30:52 ... >>>> kernel:BUG: soft lockup - CPU#154 stuck for 22s! [cmake:16810] >>>> >>>> Message from syslogd@kingly at Jun 27 14:30:54 ... >>>> kernel:BUG: soft lockup - CPU#35 stuck for 23s! [cc1plus:15807] >>>> >>>> Message from syslogd@kingly at Jun 27 14:30:55 ... >>>> kernel:BUG: soft lockup - CPU#77 stuck for 22s! [cc1plus:15316] >>>> >>>> Message from syslogd@kingly at Jun 27 14:30:55 ... >>>> kernel:BUG: soft lockup - CPU#91 stuck for 22s! [cmake:16813] >>>> >>>> Message from syslogd@kingly at Jun 27 14:30:58 ... >>>> kernel:BUG: soft lockup - CPU#124 stuck for 23s! [make:16811] >>>> >>>> Message from syslogd@kingly at Jun 27 14:31:00 ... >>>> kernel:BUG: soft lockup - CPU#149 stuck for 23s! [cc1plus:14840] >>>> >>>> ... (this went on for a while) >>>> >>>> The kernel (3.10.0-327.el7.centos.mpsp_1.3.1.45.x86_64) seems like >>>> some special build? >>>> >>>> -Hal >>>> >>>> ----- Original Message ----- >>>>> From: "Vitali A. Morozov" <morozov(a)anl.gov> >>>>> To: "Kalyan Kumaran" <kumaran(a)alcf.anl.gov>, >>>>> intel-nda(a)lists.jlse.anl.gov >>>>> Sent: Thursday, June 23, 2016 10:47:37 AM >>>>> Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up >>>>> >>>>> >>>>> >>>>> I have updated the Ethernet port issue that I have encountered. >>>>> >>>>> Vitali >>>>> >>>>> On 06/23/2016 10:42 AM, Kumaran, Kalyan wrote: >>>>> >>>>> >>>>> For folks setting these up please see instructions at: >>>>> >>>>> >>>>> https://wiki.jlse.anl.gov/display/knl/Intel+DAP+Setup >>>>> >>>>> >>>>> >>>>> Kevin, Scott and Vitali will be adding information as they set >>>>> the >>>>> machines up. Others are welcome to add stuff. >>>>> >>>>> >>>>> -Kumar >>>>> >>>>> _______________________________________________ >>>>> Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>> >>>>> _______________________________________________ >>>>> Intel-nda mailing list >>>>> Intel-nda(a)lists.jlse.anl.gov >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>> >>>> -- >>>> Hal Finkel >>>> Assistant Computational Scientist >>>> Leadership Computing Facility >>>> Argonne National Laboratory >>>> _______________________________________________ >>>> Intel-nda mailing list >>>> Intel-nda(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> -- >> Hal Finkel >> Assistant Computational Scientist >> Leadership Computing Facility >> Argonne National Laboratory >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>

1 0