Unfortunately, a reboot is 15 mins. That's a lot of time so I think we don't want to reboot for each job. We've asked Cray and Intel to analyze and document where the time is spent during the boot to see if there is a way to reduce it, but for now we are stuck with 15 min boots. I do like the idea of allowing the user to request a reboot even if they are running in the default mode. That seems like it should be doable.

-Susan


Kamil Iskra wrote:
On Tue, Jun 28, 2016 at 19:42:29 -0500, Hal Finkel wrote:


Description:        Application Performance can degrade over time when
                    MCDRAM is used in cache mode
Impact:             The direct-mapped nature of MCDRAM-as-cache is by
                    design. Real applications that need more than 16GB
                    of memory will not exhibit this time-based
                    degradation; it is really only a problem for
                    benchmarking when running smaller datasets.
Mitigation:         Workaround is to reboot the node

That seems scary. I believe we were considering booting the nodes in cache mode by default on Theta. Maybe that's a bad idea.


If I understand it correctly then, this is due to physical memory
fragmentation of DDR, typical for page-based memory management as used in
Linux.  This effect can be observed with any non-fully-associative cache
(including regular CPU caches), although it will be the worst for
one-way-associative (direct-mapped) one.

For the most predictable, and generally best, performance, it may make
sense to consider rebooting nodes between each job on Theta, or at least
make node reboot a user-accessible option at job submission time...

Kamil
_______________________________________________
Intel-nda mailing list
Intel-nda@lists.jlse.anl.gov
https://lists.jlse.anl.gov/mailman/listinfo/intel-nda