On Wed, Jun 29, 2016 at 10:27:00AM -0500, Kamil Iskra wrote:

On Tue, Jun 28, 2016 at 19:42:29 -0500, Hal Finkel wrote:

...

Description: Application Performance can degrade over time when MCDRAM is used in cache mode Impact: The direct-mapped nature of MCDRAM-as-cache is by design. Real applications that need more than 16GB of memory will not exhibit this time-based degradation; it is really only a problem for benchmarking when running smaller datasets. Mitigation: Workaround is to reboot the node

That seems scary. I believe we were considering booting the nodes in cache mode by default on Theta. Maybe that's a bad idea.

If I understand it correctly then, this is due to physical memory fragmentation of DDR, typical for page-based memory management as used in Linux. This effect can be observed with any non-fully-associative cache (including regular CPU caches), although it will be the worst for one-way-associative (direct-mapped) one.

For the most predictable, and generally best, performance, it may make sense to consider rebooting nodes between each job on Theta, or at least make node reboot a user-accessible option at job submission time...

If direct-mapped MCDRAM-as-cache is a long term feature of Xeon Phis, you could try to tell Intel to implement page coloring in Linux. Torvalds only killed 4 attempts at it in the last ten years :p -- Swann