Unfortunately, a reboot is 15 mins. That's a lot of time so I think we don't want to reboot for each job. We've asked Cray and Intel to analyze and document where the time is spent during the boot to see if there is a way to reduce it, but for now we are stuck with 15 min boots. I do like the idea of allowing the user to request a reboot even if they are running in the default mode. That seems like it should be doable. -Susan Kamil Iskra wrote:

On Tue, Jun 28, 2016 at 19:42:29 -0500, Hal Finkel wrote:

...

Description: Application Performance can degrade over time when MCDRAM is used in cache mode Impact: The direct-mapped nature of MCDRAM-as-cache is by design. Real applications that need more than 16GB of memory will not exhibit this time-based degradation; it is really only a problem for benchmarking when running smaller datasets. Mitigation: Workaround is to reboot the node

That seems scary. I believe we were considering booting the nodes in cache mode by default on Theta. Maybe that's a bad idea.

If I understand it correctly then, this is due to physical memory fragmentation of DDR, typical for page-based memory management as used in Linux. This effect can be observed with any non-fully-associative cache (including regular CPU caches), although it will be the worst for one-way-associative (direct-mapped) one.

For the most predictable, and generally best, performance, it may make sense to consider rebooting nodes between each job on Theta, or at least make node reboot a user-accessible option at job submission time...

Kamil _______________________________________________ Intel-nda mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/intel-nda