----- Original Message -----
From: "Swann Perarnau" <[email protected]> To: "Hal Finkel" <[email protected]> Cc: "Kamil Iskra" <[email protected]>, [email protected], "Scott Parker" <[email protected]> Sent: Wednesday, June 29, 2016 2:52:04 PM Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up
On Wed, Jun 29, 2016 at 02:24:20PM -0500, Hal Finkel wrote:
----- Original Message -----
From: "Kamil Iskra" <[email protected]> To: "Scott Parker" <[email protected]> Cc: [email protected] Sent: Wednesday, June 29, 2016 1:20:55 PM Subject: Re: [JLSE-Intel-NDA] Intel DAP boxes set up
I wonder if this mean a single application running in cache mode for multiple hours will see performance degradation?
It's possible, but only if that application performed a lot of memory allocations and releases throughout its execution. In practice I would say it's unlikely, and nothing we can do about anyway, so I wouldn't worry about it.
We have applications that do large allocations in phases, and that combined with other allocations from the MPI library, etc. could certainly lead to problems in this regard.
Regarding whether we can do anything about it, I'm not sure the situation is so clear cut. We can't change the fact that the cache is direct mapped, true, but how the free list is managed, and how page migration is handled, are kernel concerns. The kernel could certainly move pages with cache conflicts being accessed in temporal proximity to non-conflicting addresses (unless they're pinned, etc.). If nothing else, PEBS provides a natural facility to handle collecting this data in the kernel.
This is what is traditionally called "page coloring" or "cache coloring" by OS people. The kernel *colors* pages that can conflict in cache, and tries to keep a balance in the colors used by a process.
There is extensive research work on it, as it allows all kinds of cool stuff, including: - limiting the cache impact of thrashing access patterns - avoiding cache conflicts between structures accessed concurrently - and all kinds of QoS when multiple processes are accessing the same caches.
It can also benefit from real time performance monitoring of applications by the kernel, using PEBS indeed or other methods (page fault injection).
I specifically did not mention page-fault injection as an option because I suspect the overhead would be too high for any reasonable accuracy. I could certainly be wrong. PEBS seems more realistic, and I imagine there might be other options too.
It is quite tricky to implement, and most attempts at mainlining page coloring in Linux ended up in failures, Torvalds not being a fan of it. See http://yarchive.net/comp/linux/page_coloring.html for some of that.
The idea died for a while also because Intel and AMD consistently refused to disclose the hashing function used by their caches, which is required to implement coloring.
Interesting. Funny thing is, however, that it can be reverse engineered regardless (https://eprint.iacr.org/2015/905.pdf, https://eprint.iacr.org/2015/690.pdf). In our case, anyway, my understanding is that MCDRAM cache mapping is trivial by design. Thanks again, Hal
-- Swann
-- Hal Finkel Assistant Computational Scientist Leadership Computing Facility Argonne National Laboratory