Alternatively, we may partition the machine with a partition of all nodes in cache mode and a partition of all nodes in flat mode. The user job should specify where it wants to run and it gets scheduled there. The question on how many nodes we need in each partition is open. Vitali On 06/29/2016 11:32 AM, Scott Parker wrote:

That 15 minute boot time could be a real efficiency loss. The only way we have to change the memory mode between cache and flat is a reboot and I think we may see a mix of flat and cache mode jobs on the system given the roughly 30% memory bandwidth reduction seen in cache mode. Applications that can be made to run in flat mode could see a real performance advantage for doing so. So we may see a need for frequent reboots if even a moderate fraction of our applications opt to use flat mode.

-Scott

On 6/29/16 11:15 AM, Susan Coghlan wrote:

...

Unfortunately, a reboot is 15 mins. That's a lot of time so I think we don't want to reboot for each job. We've asked Cray and Intel to analyze and document where the time is spent during the boot to see if there is a way to reduce it, but for now we are stuck with 15 min boots. I do like the idea of allowing the user to request a reboot even if they are running in the default mode. That seems like it should be doable.

-Susan

Kamil Iskra wrote:

...

On Tue, Jun 28, 2016 at 19:42:29 -0500, Hal Finkel wrote:

...

Description: Application Performance can degrade over time when MCDRAM is used in cache mode Impact: The direct-mapped nature of MCDRAM-as-cache is by design. Real applications that need more than 16GB of memory will not exhibit this time-based degradation; it is really only a problem for benchmarking when running smaller datasets. Mitigation: Workaround is to reboot the node

That seems scary. I believe we were considering booting the nodes in cache mode by default on Theta. Maybe that's a bad idea.

If I understand it correctly then, this is due to physical memory fragmentation of DDR, typical for page-based memory management as used in Linux. This effect can be observed with any non-fully-associative cache (including regular CPU caches), although it will be the worst for one-way-associative (direct-mapped) one.

For the most predictable, and generally best, performance, it may make sense to consider rebooting nodes between each job on Theta, or at least make node reboot a user-accessible option at job submission time...

Kamil _______________________________________________ Intel-nda mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

_______________________________________________ Intel-nda mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

_______________________________________________ Intel-nda mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/intel-nda