----- Original Message -----

From: "Kamil Iskra" <[email protected]> To: "Scott Parker" <[email protected]> Cc: [email protected] Sent: Wednesday, June 29, 2016 1:20:55 PM Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up

On Wed, Jun 29, 2016 at 11:04:04 -0500, Scott Parker wrote:

...

This seem like something we should understand better. The descriptions is a little ambiguous,

Yes, it's missing a lot of detail, presumably to keep it short.

...

I wonder if this mean a single application running in cache mode for multiple hours will see performance degradation?

It's possible, but only if that application performed a lot of memory allocations and releases throughout its execution. In practice I would say it's unlikely, and nothing we can do about anyway, so I wouldn't worry about it.

We have applications that do large allocations in phases, and that combined with other allocations from the MPI library, etc. could certainly lead to problems in this regard. Regarding whether we can do anything about it, I'm not sure the situation is so clear cut. We can't change the fact that the cache is direct mapped, true, but how the free list is managed, and how page migration is handled, are kernel concerns. The kernel could certainly move pages with cache conflicts being accessed in temporal proximity to non-conflicting addresses (unless they're pinned, etc.). If nothing else, PEBS provides a natural facility to handle collecting this data in the kernel.

...

Or it's the standard Linux memory fragmentation that occurs on systems over weeks or months of having numerous processes allocate and deallocate memory?

That's exactly it.

...

Does it only occur when applications are in cache mode and use less that 16 GB?

Intel's statement about applications above 16 GB not exhibiting the problem is a little disingenuous. I'm guessing the problem is the easiest to observe for smaller apps, but there is absolutely no guarantee that an app with a problem size above 16 GB will consistently be able to use MCDRAM cache for all of its critical data structures. It depends on which physical pages the kernel allocates for it, and that's pretty much random right now.

Now, now, those just aren't "real applications" ;)

Now, regarding the email from Intel forwarded by Kumar. While a mainline Linux kernel solution would obviously be preferable long-term, the free memory list sorting hack that Intel has implemented strikes me as an eminently reasonable workaround for the time being. Given the 15 minute reboot time, I would definitely advocate for it to be deployed for every job launch

I agree that this is reasonable. There might be practical issues involving support from Cray, however, that we'd need to work out.

, with reboot being available to users on-demand for critical performance measurement experiments and charged to the user allocation (as it was on BG/P; I don't actually know what you guys do on BG/Q).

We also reboot nodes on the BG/Q. -Hal

Kamil

...

-Scott

On 6/29/16 10:27 AM, Kamil Iskra wrote:

...

On Tue, Jun 28, 2016 at 19:42:29 -0500, Hal Finkel wrote:

...

Description: Application Performance can degrade over time when MCDRAM is used in cache mode Impact: The direct-mapped nature of MCDRAM-as-cache is by design. Real applications that need more than 16GB of memory will not exhibit this time-based degradation; it is really only a problem for benchmarking when running smaller datasets. Mitigation: Workaround is to reboot the node

That seems scary. I believe we were considering booting the nodes in cache mode by default on Theta. Maybe that's a bad idea. If I understand it correctly then, this is due to physical memory fragmentation of DDR, typical for page-based memory management as used in Linux. This effect can be observed with any non-fully-associative cache (including regular CPU caches), although it will be the worst for one-way-associative (direct-mapped) one.

For the most predictable, and generally best, performance, it may make sense to consider rebooting nodes between each job on Theta, or at least make node reboot a user-accessible option at job submission time...

Kamil _______________________________________________ Intel-nda mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

-- Kamil Iskra, PhD Argonne National Laboratory, Mathematics and Computer Science Division 9700 South Cass Avenue, Building 240, Argonne, IL 60439, USA phone: +1-630-252-7197 fax: +1-630-252-5986 _______________________________________________ Intel-nda mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

-- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory