June 2016 - Intel-nda - mailman3.cels.anl.gov

Re: [JLSE-Intel-NDA] Intel DAP boxes set up
by Paul Rich 30 Jun '16

30 Jun '16

Yes, a flag to force a reboot no mater should be pretty easy at this point. Paul Rich Software Development Specialist ALCF Operations -- AIG richp(a)alcf.anl.gov On 6/29/2016 11:15, Susan Coghlan wrote: > Unfortunately, a reboot is 15 mins. That's a lot of time so I think we > don't want to reboot for each job. We've asked Cray and Intel to > analyze and document where the time is spent during the boot to see if > there is a way to reduce it, but for now we are stuck with 15 min > boots. I do like the idea of allowing the user to request a reboot > even if they are running in the default mode. That seems like it > should be doable. > > -Susan > > > Kamil Iskra wrote: >> On Tue, Jun 28, 2016 at 19:42:29 -0500, Hal Finkel wrote: >> >>> Description: Application Performance can degrade over time when >>> MCDRAM is used in cache mode >>> Impact: The direct-mapped nature of MCDRAM-as-cache is by >>> design. Real applications that need more than 16GB >>> of memory will not exhibit this time-based >>> degradation; it is really only a problem for >>> benchmarking when running smaller datasets. >>> Mitigation: Workaround is to reboot the node >>> >>> That seems scary. I believe we were considering booting the nodes in cache mode by default on Theta. Maybe that's a bad idea. >> >> If I understand it correctly then, this is due to physical memory >> fragmentation of DDR, typical for page-based memory management as used in >> Linux. This effect can be observed with any non-fully-associative cache >> (including regular CPU caches), although it will be the worst for >> one-way-associative (direct-mapped) one. >> >> For the most predictable, and generally best, performance, it may make >> sense to consider rebooting nodes between each job on Theta, or at least >> make node reboot a user-accessible option at job submission time... >> >> Kamil >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Intel DAP boxes set up
by Hal Finkel 29 Jun '16

29 Jun '16

----- Original Message ----- > From: "Swann Perarnau" <swann(a)anl.gov> > To: "Hal Finkel" <hfinkel(a)anl.gov> > Cc: "Kamil Iskra" <iskra(a)mcs.anl.gov>, intel-nda(a)lists.jlse.anl.gov, "Scott Parker" <sparker(a)anl.gov> > Sent: Wednesday, June 29, 2016 2:52:04 PM > Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up > > On Wed, Jun 29, 2016 at 02:24:20PM -0500, Hal Finkel wrote: > > ----- Original Message ----- > > > From: "Kamil Iskra" <iskra(a)mcs.anl.gov> To: "Scott Parker" > > > <sparker(a)anl.gov> Cc: intel-nda(a)lists.jlse.anl.gov Sent: > > > Wednesday, June > > > 29, 2016 1:20:55 PM Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes > > > set up > > > > > > > I wonder if this mean a single application running in cache > > > > mode for > > > > multiple hours will see performance degradation? > > > > > > It's possible, but only if that application performed a lot of > > > memory > > > allocations and releases throughout its execution. In practice I > > > would say > > > it's unlikely, and nothing we can do about anyway, so I wouldn't > > > worry > > > about it. > > > > We have applications that do large allocations in phases, and that > > combined > > with other allocations from the MPI library, etc. could certainly > > lead to > > problems in this regard. > > > > Regarding whether we can do anything about it, I'm not sure the > > situation is > > so clear cut. We can't change the fact that the cache is direct > > mapped, true, > > but how the free list is managed, and how page migration is > > handled, are > > kernel concerns. The kernel could certainly move pages with cache > > conflicts > > being accessed in temporal proximity to non-conflicting addresses > > (unless > > they're pinned, etc.). If nothing else, PEBS provides a natural > > facility to > > handle collecting this data in the kernel. > > This is what is traditionally called "page coloring" or "cache > coloring" by OS > people. The kernel *colors* pages that can conflict in cache, and > tries to > keep a balance in the colors used by a process. > > There is extensive research work on it, as it allows all kinds of > cool stuff, > including: > - limiting the cache impact of thrashing access patterns > - avoiding cache conflicts between structures accessed concurrently > - and all kinds of QoS when multiple processes are accessing the same > caches. > > It can also benefit from real time performance monitoring of > applications by the kernel, using PEBS indeed or other methods (page > fault > injection). I specifically did not mention page-fault injection as an option because I suspect the overhead would be too high for any reasonable accuracy. I could certainly be wrong. PEBS seems more realistic, and I imagine there might be other options too. > > It is quite tricky to implement, and most attempts at mainlining page > coloring > in Linux ended up in failures, Torvalds not being a fan of it. > See http://yarchive.net/comp/linux/page_coloring.html for some of > that. > > The idea died for a while also because Intel and AMD consistently > refused to > disclose the hashing function used by their caches, which is required > to > implement coloring. Interesting. Funny thing is, however, that it can be reverse engineered regardless (https://eprint.iacr.org/2015/905.pdf, https://eprint.iacr.org/2015/690.pdf) In our case, anyway, my understanding is that MCDRAM cache mapping is trivial by design. Thanks again, Hal > > -- > Swann > -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory

1 0

Re: [JLSE-Intel-NDA] Intel DAP boxes set up
by Swann Perarnau 29 Jun '16

29 Jun '16

On Wed, Jun 29, 2016 at 02:24:20PM -0500, Hal Finkel wrote: > ----- Original Message ----- > > From: "Kamil Iskra" <iskra(a)mcs.anl.gov> To: "Scott Parker" > > <sparker(a)anl.gov> Cc: intel-nda(a)lists.jlse.anl.gov Sent: Wednesday, June > > 29, 2016 1:20:55 PM Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up > > > > > I wonder if this mean a single application running in cache mode for > > > multiple hours will see performance degradation? > > > > It's possible, but only if that application performed a lot of memory > > allocations and releases throughout its execution. In practice I would say > > it's unlikely, and nothing we can do about anyway, so I wouldn't worry > > about it. > > We have applications that do large allocations in phases, and that combined > with other allocations from the MPI library, etc. could certainly lead to > problems in this regard. > > Regarding whether we can do anything about it, I'm not sure the situation is > so clear cut. We can't change the fact that the cache is direct mapped, true, > but how the free list is managed, and how page migration is handled, are > kernel concerns. The kernel could certainly move pages with cache conflicts > being accessed in temporal proximity to non-conflicting addresses (unless > they're pinned, etc.). If nothing else, PEBS provides a natural facility to > handle collecting this data in the kernel. This is what is traditionally called "page coloring" or "cache coloring" by OS people. The kernel *colors* pages that can conflict in cache, and tries to keep a balance in the colors used by a process. There is extensive research work on it, as it allows all kinds of cool stuff, including: - limiting the cache impact of thrashing access patterns - avoiding cache conflicts between structures accessed concurrently - and all kinds of QoS when multiple processes are accessing the same caches. It can also benefit from real time performance monitoring of applications by the kernel, using PEBS indeed or other methods (page fault injection). It is quite tricky to implement, and most attempts at mainlining page coloring in Linux ended up in failures, Torvalds not being a fan of it. See http://yarchive.net/comp/linux/page_coloring.html for some of that. The idea died for a while also because Intel and AMD consistently refused to disclose the hashing function used by their caches, which is required to implement coloring. -- Swann

1 0

Re: [JLSE-Intel-NDA] Intel DAP boxes set up
by Hal Finkel 29 Jun '16

29 Jun '16

----- Original Message ----- > From: "Kamil Iskra" <iskra(a)mcs.anl.gov> > To: "Scott Parker" <sparker(a)anl.gov> > Cc: intel-nda(a)lists.jlse.anl.gov > Sent: Wednesday, June 29, 2016 1:20:55 PM > Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up > > On Wed, Jun 29, 2016 at 11:04:04 -0500, Scott Parker wrote: > > > This seem like something we should understand better. The > > descriptions > > is a little > > ambiguous, > > Yes, it's missing a lot of detail, presumably to keep it short. > > > I wonder if this mean a single application running in cache > > mode for multiple > > hours will see performance degradation? > > It's possible, but only if that application performed a lot of memory > allocations and releases throughout its execution. In practice I > would say > it's unlikely, and nothing we can do about anyway, so I wouldn't > worry > about it. We have applications that do large allocations in phases, and that combined with other allocations from the MPI library, etc. could certainly lead to problems in this regard. Regarding whether we can do anything about it, I'm not sure the situation is so clear cut. We can't change the fact that the cache is direct mapped, true, but how the free list is managed, and how page migration is handled, are kernel concerns. The kernel could certainly move pages with cache conflicts being accessed in temporal proximity to non-conflicting addresses (unless they're pinned, etc.). If nothing else, PEBS provides a natural facility to handle collecting this data in the kernel. > > > Or it's the standard Linux > > memory fragmentation > > that occurs on systems over weeks or months of having numerous > > processes > > allocate > > and deallocate memory? > > That's exactly it. > > > Does it only occur when applications are in cache > > mode and use > > less that 16 GB? > > Intel's statement about applications above 16 GB not exhibiting the > problem > is a little disingenuous. I'm guessing the problem is the easiest to > observe for smaller apps, but there is absolutely no guarantee that > an app > with a problem size above 16 GB will consistently be able to use > MCDRAM > cache for all of its critical data structures. It depends on which > physical pages the kernel allocates for it, and that's pretty much > random > right now. Now, now, those just aren't "real applications" ;) > > Now, regarding the email from Intel forwarded by Kumar. While a > mainline > Linux kernel solution would obviously be preferable long-term, the > free > memory list sorting hack that Intel has implemented strikes me as an > eminently reasonable workaround for the time being. Given the 15 > minute > reboot time, I would definitely advocate for it to be deployed for > every > job launch I agree that this is reasonable. There might be practical issues involving support from Cray, however, that we'd need to work out. >, with reboot being available to users on-demand for > critical > performance measurement experiments and charged to the user > allocation (as > it was on BG/P; I don't actually know what you guys do on BG/Q). We also reboot nodes on the BG/Q. -Hal > > Kamil > > > -Scott > > > > > > On 6/29/16 10:27 AM, Kamil Iskra wrote: > > > On Tue, Jun 28, 2016 at 19:42:29 -0500, Hal Finkel wrote: > > > > > >> Description: Application Performance can degrade over > > >> time when > > >> MCDRAM is used in cache mode > > >> Impact: The direct-mapped nature of MCDRAM-as-cache > > >> is by > > >> design. Real applications that need more > > >> than 16GB > > >> of memory will not exhibit this time-based > > >> degradation; it is really only a problem > > >> for > > >> benchmarking when running smaller datasets. > > >> Mitigation: Workaround is to reboot the node > > >> > > >> That seems scary. I believe we were considering booting the > > >> nodes in cache mode by default on Theta. Maybe that's a bad > > >> idea. > > > If I understand it correctly then, this is due to physical memory > > > fragmentation of DDR, typical for page-based memory management as > > > used in > > > Linux. This effect can be observed with any > > > non-fully-associative cache > > > (including regular CPU caches), although it will be the worst for > > > one-way-associative (direct-mapped) one. > > > > > > For the most predictable, and generally best, performance, it may > > > make > > > sense to consider rebooting nodes between each job on Theta, or > > > at least > > > make node reboot a user-accessible option at job submission > > > time... > > > > > > Kamil > > > _______________________________________________ > > > Intel-nda mailing list > > > Intel-nda(a)lists.jlse.anl.gov > > > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > > > -- > Kamil Iskra, PhD > Argonne National Laboratory, Mathematics and Computer Science > Division > 9700 South Cass Avenue, Building 240, Argonne, IL 60439, USA > phone: +1-630-252-7197 fax: +1-630-252-5986 > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory

1 0

[JLSE-Intel-NDA] Intel DAP boxes: performance degradation
by Vitali A. Morozov 29 Jun '16

29 Jun '16

The Intel DAP box is not delivering the 2 instructions per cycle throughput in my simple micro benchmark. The same problem was previously observed on knl00 rack mounted system before the BIOS update, which has fixed the problem. More details. A program with a known mixture and number of independent instructions should provide the peak 2 IPCs throughput. This was previously confirmed on the knl00 machine with 1, 2 and 4 threads running on a core: knl00: 1 thread per core: 4512: optimized time 405239 result 0.00000000 IPC = 1.9911 SF/cycle = 58.2539 Peak rate = 91.0217 4528: optimized time 408436 result 0.00000000 IPC = 1.9825 SF/cycle = 58.0029 Peak rate = 90.6296 4544: optimized time 400927 result 0.00000000 IPC = 2.0268 SF/cycle = 59.2980 Peak rate = 92.6532 2 threads per core: 4512: optimized time 405415 result 0.00000000 IPC = 1.9902 SF/cycle = 58.2286 Peak rate = 90.9822 4528: optimized time 408624 result 0.00000000 IPC = 1.9816 SF/cycle = 57.9762 Peak rate = 90.5878 4544: optimized time 401053 result 0.00000000 IPC = 2.0262 SF/cycle = 59.2794 Peak rate = 92.6241 4 threads per core: 4512: optimized time 407851 result 0.00000000 IPC = 1.9783 SF/cycle = 57.8808 Peak rate = 90.4388 4528: optimized time 408742 result 0.00000000 IPC = 1.9810 SF/cycle = 57.9594 Peak rate = 90.5616 4544: optimized time 401151 result 0.00000000 IPC = 2.0257 SF/cycle = 59.2649 Peak rate = 92.6015 Same executable on Intel DAP box: 1 thread per core: 4512: optimized time 461229 result 0.00000000 IPC = 1.7494 SF/cycle = 51.1823 Peak rate = 79.9724 4528: optimized time 464888 result 0.00000000 IPC = 1.7418 SF/cycle = 50.9596 Peak rate = 79.6243 4544: optimized time 456222 result 0.00000000 IPC = 1.7811 SF/cycle = 52.1110 Peak rate = 81.4234 2 threads per core: 4512: optimized time 461132 result 0.00000000 IPC = 1.7498 SF/cycle = 51.1930 Peak rate = 79.9891 4528: optimized time 464981 result 0.00000000 IPC = 1.7414 SF/cycle = 50.9494 Peak rate = 79.6084 4544: optimized time 456272 result 0.00000000 IPC = 1.7809 SF/cycle = 52.1053 Peak rate = 81.4145 4 threads per core: 4512: optimized time 461073 result 0.00000000 IPC = 1.7500 SF/cycle = 51.1996 Peak rate = 79.9993 4528: optimized time 464830 result 0.00000000 IPC = 1.7420 SF/cycle = 50.9659 Peak rate = 79.6343 4544: optimized time 456215 result 0.00000000 IPC = 1.7812 SF/cycle = 52.1117 Peak rate = 81.4246 Same issue is confirmed when running HACCmk. On knl00, it is delivering 67% of peak: 2944: optimized time 858274 result 1354864.52650320 IPC = 1.8052 SF/cycle = 42.6229 Peak rate = 66.5983 2960: optimized time 860064 result 1367795.59051550 IPC = 1.8112 SF/cycle = 42.7654 Peak rate = 66.8209 2976: optimized time 863812 result 1380780.63478190 IPC = 1.8131 SF/cycle = 42.8099 Peak rate = 66.8905 2992: optimized time 865702 result 1393825.84251841 IPC = 1.8189 SF/cycle = 42.9462 Peak rate = 67.1034 Intel DAP provides less then 60% of peak: 2944: optimized time 982627 result 1354864.52650320 IPC = 1.5767 SF/cycle = 37.2289 Peak rate = 58.1702 2960: optimized time 989038 result 1367795.59051550 IPC = 1.5750 SF/cycle = 37.1886 Peak rate = 58.1072 2976: optimized time 991330 result 1380780.63478190 IPC = 1.5799 SF/cycle = 37.3032 Peak rate = 58.2862 2992: optimized time 995241 result 1393825.84251841 IPC = 1.5821 SF/cycle = 37.3563 Peak rate = 58.3693 In these results, the time is measured in cycles, therefore the possible differences in frequency should not lead to performance difference. Vitali

1 0

Re: [JLSE-Intel-NDA] Intel DAP boxes set up
by Kamil Iskra 29 Jun '16

29 Jun '16

On Wed, Jun 29, 2016 at 11:04:04 -0500, Scott Parker wrote: > This seem like something we should understand better. The descriptions > is a little > ambiguous, Yes, it's missing a lot of detail, presumably to keep it short. > I wonder if this mean a single application running in cache > mode for multiple > hours will see performance degradation? It's possible, but only if that application performed a lot of memory allocations and releases throughout its execution. In practice I would say it's unlikely, and nothing we can do about anyway, so I wouldn't worry about it. > Or it's the standard Linux > memory fragmentation > that occurs on systems over weeks or months of having numerous processes > allocate > and deallocate memory? That's exactly it. > Does it only occur when applications are in cache > mode and use > less that 16 GB? Intel's statement about applications above 16 GB not exhibiting the problem is a little disingenuous. I'm guessing the problem is the easiest to observe for smaller apps, but there is absolutely no guarantee that an app with a problem size above 16 GB will consistently be able to use MCDRAM cache for all of its critical data structures. It depends on which physical pages the kernel allocates for it, and that's pretty much random right now. Now, regarding the email from Intel forwarded by Kumar. While a mainline Linux kernel solution would obviously be preferable long-term, the free memory list sorting hack that Intel has implemented strikes me as an eminently reasonable workaround for the time being. Given the 15 minute reboot time, I would definitely advocate for it to be deployed for every job launch, with reboot being available to users on-demand for critical performance measurement experiments and charged to the user allocation (as it was on BG/P; I don't actually know what you guys do on BG/Q). Kamil > -Scott > > > On 6/29/16 10:27 AM, Kamil Iskra wrote: > > On Tue, Jun 28, 2016 at 19:42:29 -0500, Hal Finkel wrote: > > > >> Description: Application Performance can degrade over time when > >> MCDRAM is used in cache mode > >> Impact: The direct-mapped nature of MCDRAM-as-cache is by > >> design. Real applications that need more than 16GB > >> of memory will not exhibit this time-based > >> degradation; it is really only a problem for > >> benchmarking when running smaller datasets. > >> Mitigation: Workaround is to reboot the node > >> > >> That seems scary. I believe we were considering booting the nodes in cache mode by default on Theta. Maybe that's a bad idea. > > If I understand it correctly then, this is due to physical memory > > fragmentation of DDR, typical for page-based memory management as used in > > Linux. This effect can be observed with any non-fully-associative cache > > (including regular CPU caches), although it will be the worst for > > one-way-associative (direct-mapped) one. > > > > For the most predictable, and generally best, performance, it may make > > sense to consider rebooting nodes between each job on Theta, or at least > > make node reboot a user-accessible option at job submission time... > > > > Kamil > > _______________________________________________ > > Intel-nda mailing list > > Intel-nda(a)lists.jlse.anl.gov > > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > -- Kamil Iskra, PhD Argonne National Laboratory, Mathematics and Computer Science Division 9700 South Cass Avenue, Building 240, Argonne, IL 60439, USA phone: +1-630-252-7197 fax: +1-630-252-5986

1 0

Re: [JLSE-Intel-NDA] Intel DAP boxes set up
by Christopher Knight 29 Jun '16

29 Jun '16

FYI, a recommendation for rebooting the Theta nodes into alternate memory configurations will be discussed and organized at the next scheduling meeting, July 5th at 2pm. I can summarize discussion from this thread at the meeting. chris On Jun 29, 2016, at 11:35 AM, Vitali A. Morozov <morozov(a)anl.gov> wrote: Alternatively, we may partition the machine with a partition of all nodes in cache mode and a partition of all nodes in flat mode. The user job should specify where it wants to run and it gets scheduled there. The question on how many nodes we need in each partition is open. Vitali On 06/29/2016 11:32 AM, Scott Parker wrote: > > That 15 minute boot time could be a real efficiency loss. The only way we have to change the memory mode between cache > and flat is a reboot and I think we may see a mix of flat and cache mode jobs on the system given the roughly 30% memory > bandwidth reduction seen in cache mode. Applications that can be made to run in flat mode could see a real performance advantage > for doing so. So we may see a need for frequent reboots if even a moderate fraction of our applications opt to use flat mode. > -Scott > On 6/29/16 11:15 AM, Susan Coghlan wrote: >> Unfortunately, a reboot is 15 mins. That's a lot of time so I think we don't want to reboot for each job. We've asked Cray and Intel to analyze and document where the time is spent during the boot to see if there is a way to reduce it, but for now we are stuck with 15 min boots. I do like the idea of allowing the user to request a reboot even if they are running in the default mode. That seems like it should be doable. >> >> -Susan >> >> >> Kamil Iskra wrote: >>> On Tue, Jun 28, 2016 at 19:42:29 -0500, Hal Finkel wrote: >>> >>>> Description: Application Performance can degrade over time when >>>> MCDRAM is used in cache mode >>>> Impact: The direct-mapped nature of MCDRAM-as-cache is by >>>> design. Real applications that need more than 16GB >>>> of memory will not exhibit this time-based >>>> degradation; it is really only a problem for >>>> benchmarking when running smaller datasets. >>>> Mitigation: Workaround is to reboot the node >>>> >>>> That seems scary. I believe we were considering booting the nodes in cache mode by default on Theta. Maybe that's a bad idea. >>> >>> If I understand it correctly then, this is due to physical memory >>> fragmentation of DDR, typical for page-based memory management as used in >>> Linux. This effect can be observed with any non-fully-associative cache >>> (including regular CPU caches), although it will be the worst for >>> one-way-associative (direct-mapped) one. >>> >>> For the most predictable, and generally best, performance, it may make >>> sense to consider rebooting nodes between each job on Theta, or at least >>> make node reboot a user-accessible option at job submission time... >>> >>> Kamil >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov <mailto:[email protected]> >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda <https://lists.jlse.anl.gov/mailman/listinfo/intel-nda> >> >> >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov <mailto:[email protected]> >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda <https://lists.jlse.anl.gov/mailman/listinfo/intel-nda> > > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov <mailto:[email protected]> > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda <https://lists.jlse.anl.gov/mailman/listinfo/intel-nda> _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] Fwd: Intel DAP boxes set up
by Kumaran, Kalyan 29 Jun '16

29 Jun '16

Sent from my iPhone Begin forwarded message: From: "Meadows, Lawrence F" <lawrence.f.meadows(a)intel.com<mailto:[email protected]>> Date: June 29, 2016 at 10:02:13 PM GMT+5:30 To: "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov<mailto:[email protected]>>, "Sodani, Avinash" <avinash.sodani(a)intel.com<mailto:[email protected]>> Cc: "Finkel, Hal J." <hfinkel(a)anl.gov<mailto:[email protected]>>, "Harms, Kevin" <harms(a)alcf.anl.gov<mailto:[email protected]>>, "Morozov, Vitali A." <morozov(a)alcf.anl.gov<mailto:[email protected]>>, "Kirkendall, Keith G" <keith.g.kirkendall(a)intel.com<mailto:[email protected]>>, "Allen, Benjamin S." <bsallen(a)alcf.anl.gov<mailto:[email protected]>>, "Allison, CM" <cm.allison(a)intel.com<mailto:[email protected]>>, "Valles, Antonio C" <antonio.c.valles(a)intel.com<mailto:[email protected]>> Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up The issue is that over time, the Linux free page list becomes effectively randomized, so a program will receive pages with physical addresses in several different 16 GiB regions, resulting in collisions in the direct-mapped MCDRAM cache. We have had success with a kernel patch that provides a sysctl to sort the free list. However, that patch is seen as a hack by the linux community and is proving difficult to upstream. We are investigating other approaches. The sorting patch requires that a utility be run as root before every job and does not completely mitigate the problem but does help a lot (effectively zero collisions up to 12-14 GiB of application memory usage). I am interested in ANL's views on using such a patch in your enviroment. -- Larry On 6/29/16, 9:21 AM, "Kumaran, Kalyan" <kumaran(a)alcf.anl.gov<mailto:[email protected]>> wrote: Is this the same issue reported by Trinity folks? Sent from my iPhone On Jun 29, 2016, at 8:44 PM, Sodani, Avinash <avinash.sodani(a)intel.com<mailto:[email protected]>> wrote: Ccing Larry and Antonio. -Avinash -----Original Message----- From: Hal Finkel [mailto:[email protected]] Sent: Tuesday, June 28, 2016 6:44 PM To: Kalyan Kumaran Cc: Kevin Harms; Vitali A. Morozov; Kirkendall, Keith G; Benjamin S. Allen; Allison, CM; Sodani, Avinash Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up ----- Original Message ----- From: "Kalyan Kumaran" <kumaran(a)alcf.anl.gov<mailto:[email protected]>> To: "Hal J. Finkel" <hfinkel(a)anl.gov<mailto:[email protected]>>, "avinash sodani" <avinash.sodani(a)intel.com<mailto:[email protected]>> Cc: "Kevin Harms" <harms(a)alcf.anl.gov<mailto:[email protected]>>, "Vitali A. Morozov" <morozov(a)alcf.anl.gov<mailto:[email protected]>>, "keith g kirkendall" <keith.g.kirkendall(a)intel.com<mailto:[email protected]>>, "Benjamin S. Allen" <bsallen(a)alcf.anl.gov<mailto:[email protected]>>, "cm allison" <cm.allison(a)intel.com<mailto:[email protected]>> Sent: Tuesday, June 28, 2016 8:40:15 PM Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up Copying Avinash. Sent from my iPhone On Jun 29, 2016, at 6:12 AM, Hal Finkel <hfinkel(a)anl.gov<mailto:[email protected]>> wrote: Hi Kevin, Thanks! Also, from the release notes: Issue number: 0000003 Description: Application Performance can degrade over time when MCDRAM is used in cache mode Impact: The direct-mapped nature of MCDRAM-as-cache is by design. Real applications that need more than 16GB of memory will not exhibit this time-based degradation; it is really only a problem for benchmarking when running smaller datasets. Mitigation: Workaround is to reboot the node That seems scary. I believe we were considering booting the nodes in cache mode by default on Theta. Maybe that's a bad idea. Intel Tracking ID: 0001418 Affected OS: CentOS 7.2, RHEL 7.2, SLES 12, SLES 12 SP1 Description: [Tools] rpm rebuild not working Notes: Will not be Fixed; Not a defect Why not? -Hal ----- Original Message ----- From: "Kevin Harms" <harms(a)alcf.anl.gov<mailto:[email protected]>> To: "Hal J. Finkel" <hfinkel(a)anl.gov<mailto:[email protected]>> Cc: "Vitali A. Morozov" <morozov(a)alcf.anl.gov<mailto:[email protected]>>, intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> Sent: Tuesday, June 28, 2016 7:30:30 PM Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up I believe this kernel is just a set of selected backports from the main kernel. It’s possible this bug is related to a problem with PTE handling. Intel reported some bug where dirty pages were handled incorrectly under some conditions. That’s all the info we got. I haven’t seen the fix yet. The mpsp 1.3.3 is out as well, renamed xppsl_1.3.3. I installed this version. Same problem: Message from syslogd@kingly at Jun 28 20:14:29 ... kernel:BUG: soft lockup - CPU#223 stuck for 22s! [sh:24755] Message from syslogd@kingly at Jun 28 20:14:36 ... kernel:BUG: soft lockup - CPU#247 stuck for 22s! [as:25704] ... $ uname -a Linux kingly 3.10.0-327.13.1.el7.xppsl_1.3.3.151.x86_64 #1 SMP Fri Jun 10 15:04:35 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux -Hal https://software.intel.com/en-us/articles/intel-xeon-phi-processor-software http://registrationcenter-download.intel.com/akdlm/irc_nas/9325/xppsl-1.3.3… kevin Hi Kevin, FWIW, no, the problem happened again with the new kernel: Message from syslogd@kingly at Jun 28 19:12:49 ... kernel:BUG: soft lockup - CPU#104 stuck for 23s! [cc1plus:77778] Message from syslogd@kingly at Jun 28 19:12:51 ... kernel:BUG: soft lockup - CPU#177 stuck for 23s! [cc1plus:77662] ... What's special about this kernel? Could I build a mainline kernel and expect it to work reasonably? -Hal ----- Original Message ----- From: "Hal Finkel" <hfinkel(a)anl.gov<mailto:[email protected]>> To: "Kevin Harms" <harms(a)alcf.anl.gov<mailto:[email protected]>> Cc: "Vitali A. Morozov" <morozov(a)alcf.anl.gov<mailto:[email protected]>>, intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> Sent: Tuesday, June 28, 2016 12:47:02 PM Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up Thanks! I'll upgrade. -Hal ----- Original Message ----- From: "Kevin Harms" <harms(a)alcf.anl.gov<mailto:[email protected]>> To: "Hal J. Finkel" <hfinkel(a)anl.gov<mailto:[email protected]>>, "Vitali A. Morozov" <morozov(a)alcf.anl.gov<mailto:[email protected]>> Cc: intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> Sent: Tuesday, June 28, 2016 11:59:20 AM Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up I updated this page to indicate how to upgrade to the mpsp 1.3.2 kernel. https://wiki.jlse.anl.gov/display/knl/Intel+DAP+Setup No idea, if that will fix this problem though :) kevin FYI: James and I have been experimenting with our box. We had to reset it after we hit some kernel bug: Message from syslogd@kingly at Jun 27 14:29:44 ... kernel:BUG: soft lockup - CPU#77 stuck for 22s! [cc1plus:15316] Message from syslogd@kingly at Jun 27 14:30:43 ... kernel:BUG: soft lockup - CPU#149 stuck for 23s! [cc1plus:14840] Message from syslogd@kingly at Jun 27 14:30:44 ... kernel:BUG: soft lockup - CPU#8 stuck for 23s! [cc1plus:15337] Message from syslogd@kingly at Jun 27 14:30:46 ... kernel:BUG: soft lockup - CPU#49 stuck for 22s! [as:16806] Message from syslogd@kingly at Jun 27 14:30:47 ... kernel:BUG: soft lockup - CPU#27 stuck for 22s! [cc1plus:16548] Message from syslogd@kingly at Jun 27 14:30:50 ... kernel:BUG: soft lockup - CPU#34 stuck for 22s! [cmake:16807] Message from syslogd@kingly at Jun 27 14:30:52 ... kernel:BUG: soft lockup - CPU#154 stuck for 22s! [cmake:16810] Message from syslogd@kingly at Jun 27 14:30:54 ... kernel:BUG: soft lockup - CPU#35 stuck for 23s! [cc1plus:15807] Message from syslogd@kingly at Jun 27 14:30:55 ... kernel:BUG: soft lockup - CPU#77 stuck for 22s! [cc1plus:15316] Message from syslogd@kingly at Jun 27 14:30:55 ... kernel:BUG: soft lockup - CPU#91 stuck for 22s! [cmake:16813] Message from syslogd@kingly at Jun 27 14:30:58 ... kernel:BUG: soft lockup - CPU#124 stuck for 23s! [make:16811] Message from syslogd@kingly at Jun 27 14:31:00 ... kernel:BUG: soft lockup - CPU#149 stuck for 23s! [cc1plus:14840] ... (this went on for a while) The kernel (3.10.0-327.el7.centos.mpsp_1.3.1.45.x86_64) seems like some special build? -Hal ----- Original Message ----- From: "Vitali A. Morozov" <morozov(a)anl.gov<mailto:[email protected]>> To: "Kalyan Kumaran" <kumaran(a)alcf.anl.gov<mailto:[email protected]>>, intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> Sent: Thursday, June 23, 2016 10:47:37 AM Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up I have updated the Ethernet port issue that I have encountered. Vitali On 06/23/2016 10:42 AM, Kumaran, Kalyan wrote: For folks setting these up please see instructions at: https://wiki.jlse.anl.gov/display/knl/Intel+DAP+Setup Kevin, Scott and Vitali will be adding information as they set the machines up. Others are welcome to add stuff. -Kumar _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda -- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory

1 0

Re: [JLSE-Intel-NDA] Intel DAP boxes set up
by Vitali A. Morozov 29 Jun '16

29 Jun '16

Alternatively, we may partition the machine with a partition of all nodes in cache mode and a partition of all nodes in flat mode. The user job should specify where it wants to run and it gets scheduled there. The question on how many nodes we need in each partition is open. Vitali On 06/29/2016 11:32 AM, Scott Parker wrote: > > > That 15 minute boot time could be a real efficiency loss. The only way > we have to change the memory mode between cache > and flat is a reboot and I think we may see a mix of flat and cache > mode jobs on the system given the roughly 30% memory > bandwidth reduction seen in cache mode. Applications that can be made > to run in flat mode could see a real performance advantage > for doing so. So we may see a need for frequent reboots if even a > moderate fraction of our applications opt to use flat mode. > > -Scott > > On 6/29/16 11:15 AM, Susan Coghlan wrote: >> Unfortunately, a reboot is 15 mins. That's a lot of time so I think >> we don't want to reboot for each job. We've asked Cray and Intel to >> analyze and document where the time is spent during the boot to see >> if there is a way to reduce it, but for now we are stuck with 15 min >> boots. I do like the idea of allowing the user to request a reboot >> even if they are running in the default mode. That seems like it >> should be doable. >> >> -Susan >> >> >> Kamil Iskra wrote: >>> On Tue, Jun 28, 2016 at 19:42:29 -0500, Hal Finkel wrote: >>> >>>> Description: Application Performance can degrade over time when >>>> MCDRAM is used in cache mode >>>> Impact: The direct-mapped nature of MCDRAM-as-cache is by >>>> design. Real applications that need more than 16GB >>>> of memory will not exhibit this time-based >>>> degradation; it is really only a problem for >>>> benchmarking when running smaller datasets. >>>> Mitigation: Workaround is to reboot the node >>>> >>>> That seems scary. I believe we were considering booting the nodes in cache mode by default on Theta. Maybe that's a bad idea. >>> >>> If I understand it correctly then, this is due to physical memory >>> fragmentation of DDR, typical for page-based memory management as used in >>> Linux. This effect can be observed with any non-fully-associative cache >>> (including regular CPU caches), although it will be the worst for >>> one-way-associative (direct-mapped) one. >>> >>> For the most predictable, and generally best, performance, it may make >>> sense to consider rebooting nodes between each job on Theta, or at least >>> make node reboot a user-accessible option at job submission time... >>> >>> Kamil >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> >> >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Intel DAP boxes set up
by Scott Parker 29 Jun '16

29 Jun '16

That 15 minute boot time could be a real efficiency loss. The only way we have to change the memory mode between cache and flat is a reboot and I think we may see a mix of flat and cache mode jobs on the system given the roughly 30% memory bandwidth reduction seen in cache mode. Applications that can be made to run in flat mode could see a real performance advantage for doing so. So we may see a need for frequent reboots if even a moderate fraction of our applications opt to use flat mode. -Scott On 6/29/16 11:15 AM, Susan Coghlan wrote: > Unfortunately, a reboot is 15 mins. That's a lot of time so I think we > don't want to reboot for each job. We've asked Cray and Intel to > analyze and document where the time is spent during the boot to see if > there is a way to reduce it, but for now we are stuck with 15 min > boots. I do like the idea of allowing the user to request a reboot > even if they are running in the default mode. That seems like it > should be doable. > > -Susan > > > Kamil Iskra wrote: >> On Tue, Jun 28, 2016 at 19:42:29 -0500, Hal Finkel wrote: >> >>> Description: Application Performance can degrade over time when >>> MCDRAM is used in cache mode >>> Impact: The direct-mapped nature of MCDRAM-as-cache is by >>> design. Real applications that need more than 16GB >>> of memory will not exhibit this time-based >>> degradation; it is really only a problem for >>> benchmarking when running smaller datasets. >>> Mitigation: Workaround is to reboot the node >>> >>> That seems scary. I believe we were considering booting the nodes in cache mode by default on Theta. Maybe that's a bad idea. >> >> If I understand it correctly then, this is due to physical memory >> fragmentation of DDR, typical for page-based memory management as used in >> Linux. This effect can be observed with any non-fully-associative cache >> (including regular CPU caches), although it will be the worst for >> one-way-associative (direct-mapped) one. >> >> For the most predictable, and generally best, performance, it may make >> sense to consider rebooting nodes between each job on Theta, or at least >> make node reboot a user-accessible option at job submission time... >> >> Kamil >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0