I don't want to rely upon work by PNNL to drive FT-RMA; clearly there are issues there. The FT WG has clearly made a lot of progress on non-RMA so far and I don't want RMA users to be left out. Let's work together to solve the problem of non-Byzantine fault-tolerance in RMA by leveraging the existing work you have done and the knowledge that RMA was designed such that it could be implemented on top of Send-Recv (recall that the Forum refuses to require progress in passive target RMA...). The first step should be to make RMA symmetric w.r.t. Isend regarding the implementation not being able to modify buffers outside of what is require to provide the RMA semantics in question (i.e. MPI_Get does not modify target window, MPI_Put does not modify input buffer, etc.). I'll read stuff over the holiday and be prepared to work on this with you in January. Best, Jeff On Fri, Dec 12, 2014 at 12:43 AM, George Bosilca <[email protected]> wrote:

On Thu, Dec 11, 2014 at 1:26 AM, Jeff Hammond <[email protected]> wrote:

...

http://pubs.acs.org/doi/abs/10.1021/ct100439u is the paper I was implicitly referencing. They do RAID inside of GA.

I can only do this sanely with MPI RMA (ie without resorting to nproc times as many windows as necessary) if and only iff I can continue to use data after process failure if I know it could not have been corrupted.

GA folks understood that minimal (and potentially intrusive) changes were required from the GA underlying runtime and communication library in order to support highly effective application specific fault tolerance methods. The paper you pointed out unfortunately leaves such discussions out, but it does prove a point similar to ULFM, that building upon a fault tolerant communication library could have drastic benefits for applications in case of faults, while minimizing the impact of the failure-free execution.

Now, for the sake of the discussion let's dig a little deeper. To be extremely pedantic, while there is a 2-level RAID inside GA, this cannot be compared with MPI (which is more like ARMCI). As you might notice, there is a subtle difference here, ARMCI do not guarantee the correctness in the traditional sense for one-sided operations (except obviously for get-based protocols). Instead, they use the GA-level data redundancy, together with write-based one-sided communication primitives and fences to ensure the existence of a consistent state. Brilliantly simplistic approach, at the opposite spectrum of the MPI Forum who seem to require data integrity on windows in all memory models.

Anyway, the good thing is that in the context of the FT WG, we are way past the point where GA seems to be (for everything but one-sided). We do have a clear description of the expected behavior for all communications (except for one-sided), with a well described API (except for one-sided), and now 2 widely available implementations (except for one-sided). Over the last 2 years, many large scale applications have shown that taking advantage of these extensions, drastic improvements in the time-to-solution for these applications can be achieved in faulty environments. This is proof that instead of providing a limited-scope fault management model, ULFM expose a portable API, allowing application/library developers to design and implement highly efficient application/domain-specific fault tolerant models.

Hopefully with the help of interested folks and with the support of the RMA WG, we could settle on an approach similar to FT-ARMCI, and start building something constructive from there. It is about time.

...

It is possible that the paper doesn't adequately explain things for this context, in which case I will provide them later.

We should not mix application/domain-specific with communication library-level fault tolerance. Most of these papers do a great job as exposing high-level strategies, domain-specific or application-level, to handle faults. They seems to imply some level of resilience from the underlying runtime and communication library, but unfortunately such details are extremely scarce.

There are 2 questions I would appreciate to get more details on. In the light on the point raised in the discussion regarding the memory reuse in RMA, how are the GA folks dealing with this case? My understanding is that they leverage the GA-level data redundancy to ensure consistency. But if we suppose lingering messages in the network generated by the failed process, how do they ensure the viability of the shadow data and especially how do they maintain the data consistency across multiple subsequent failures?

Sorry for the long email. George.

PS: Reading through these papers I noticed an unsettling thing. In the HIPC'10 paper that talk about FT-ARMCI, performance numbers are presented using a fault-tolerant ARMCI/GA. However, all the other papers, especially those published after the HIPC'10 paper, state that no fault-tolerant GA implementation exists, and present instead results obtained using the original GA implementation (convenient ...). I wonder what the reaction of the MPI Forum would have been if the FT WG would have dared to present fault tolerance related results using a stock MPI library.

...

Other stuff that may or may matter:

http://hpc.pnl.gov/people/vishnu/public/vishnu_overdecomposition.pdf http://hpc.pnl.gov/people/vishnu/public/vishnu_hipc10.pdf http://dx.doi.org/10.1109/PDP.2011.72 http://link.springer.com/chapter/10.1007/978-3-642-23397-5_34

I assume someone from Argonne has presented GVR to the WG?

Jeff

Sent from my iPhone

On Dec 10, 2014, at 10:12 PM, George Bosilca <[email protected]> wrote:

Jeff,

I was trying to find some references to the GA FT work you mentioned during the plenary discussion today.

The only reference I could find about the FT capabilities of GA is [1] but it is getting dusty. A more recent reference [2] addresses NWCHEM in particular, but represents an application-specific user-level checkpoint/restart strategy, requiring minimal support from the communication library and that has little in common with the ongoing discussion in the WG.

I would really appreciate if you could provide a reference.

Thanks, George.

[1] V. Tipparaju, M. Krishnan, B. Palmer, F. Petrini, and J. Nieplocha, “Towards fault resilient Global Arrays.” in International Conference on Parallel Computing, vol. 15, 2007, pp. 339–345. [2] Nawab Ali, Sriram Krishnamoorthy, Niranjan Govind, Bruce Palmer, "A Redundant Communication Approach to Scalable Fault Tolerance in PGAS Programming Models", in PDP'11

On Wed, Dec 10, 2014 at 5:14 PM, Wesley Bland <[email protected]> wrote:

...

I've posted notes from today's plenary session on the wiki page:

https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ftwg2014-12-10

I'm also attaching the slides to this email and I believe they'll be posted on the forum website by Martin at some point.

Thanks, Wesley

_______________________________________________ mpiwg-ft mailing list [email protected] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft

-- Jeff Hammond [email protected] http://jeffhammond.github.io/