This seem like something we should understand better. The descriptions is a little ambiguous, I wonder if this mean a single application running in cache mode for multiple hours will see performance degradation? Or it's the standard Linux memory fragmentation that occurs on systems over weeks or months of having numerous processes allocate and deallocate memory? Does it only occur when applications are in cache mode and use less that 16 GB? -Scott On 6/29/16 10:27 AM, Kamil Iskra wrote:

On Tue, Jun 28, 2016 at 19:42:29 -0500, Hal Finkel wrote:

...

Description: Application Performance can degrade over time when MCDRAM is used in cache mode Impact: The direct-mapped nature of MCDRAM-as-cache is by design. Real applications that need more than 16GB of memory will not exhibit this time-based degradation; it is really only a problem for benchmarking when running smaller datasets. Mitigation: Workaround is to reboot the node

That seems scary. I believe we were considering booting the nodes in cache mode by default on Theta. Maybe that's a bad idea. If I understand it correctly then, this is due to physical memory fragmentation of DDR, typical for page-based memory management as used in Linux. This effect can be observed with any non-fully-associative cache (including regular CPU caches), although it will be the worst for one-way-associative (direct-mapped) one.

For the most predictable, and generally best, performance, it may make sense to consider rebooting nodes between each job on Theta, or at least make node reboot a user-accessible option at job submission time...

Kamil _______________________________________________ Intel-nda mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/intel-nda