June 2008 - mpiwg-ft - mailman3.cels.anl.gov

[Mpi3-ft] Error handling proposal
by Greg Bronevetsky 14 Jun '08

14 Jun '08

Following our discussion at the last telecon I've drafted a proposal for handling faults in MPI and uploaded it to the Wiki. The proposal is available in both Word and pdf formats and I can upload a .txt version if people want. The proposal leaves several areas unspecified (highlighted in blue) because I wasn't sure how to organize some functionality that I think we should have. Any comments about the document are welcome. I also notice that Alexander Supovalov uploaded an alternate fault handling proposal to the Wiki a while ago. Between these two drafts we should have a decent start towards putting something useful together. Please voice your initial thoughts over email so that we can handle more concrete issues at our face-to-face. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov

1 0

Re: [Mpi3-ft] docs on CP/R with RDMA fabrics
by Mike Heffner 06 Jun '08

06 Jun '08

Josh, On the last teleconf we primarily gave a brief background on our Availability Services and the fabrics we support today. We described briefly how we integrate with MVAPICH to support CP/R over zero-copy RDMA with low overhead. As our goal is to remain transparent from an application standpoint our plan right now is to present how we integrate below the MPI API. We are not currently proposing any new MPI interfaces, but as a company we are interested in following the MPI FT discussions and providing feedback/insight into the proposals. Our interest would be in following the discussions to see how applications that were written to take advantage of the new MPI FT interfaces could leverage our CP/R solution and how that may best serve the FT design decisions. As a company we'll also be pursuing various partner relationships with the popular MPI vendors to discuss integration opportunities. I'd imagine most work would go on below the hood at first, but hopefully we could form some type of formal integration design that would assist in future MPI stack integration. Cheers, Mike Josh Hursey wrote: > Mike, > > I was not on the last teleconf so this might have been covered there, > but just to make sure I understand the later two proposals you sent. > You are not proposing any MPI interfaces for checkpoint/restart, but > just describing how you implemented a transparent solution inside or > below an MPI implementation. Is that correct? > > -- Josh > > On Jun 5, 2008, at 5:44 PM, Mike Heffner wrote: > >> [3rd try around, mailing list bounce] >> >> All, >> >> Attached are the documents I promised during the previous conference >> call. >> >> The IPDPS 2007 paper gives a technical description of the >> Availability Services product (known academically as "DejaVu") >> including the online logging algorithm used for BSD socket >> applications. It also includes some performance numbers from >> previous experiments with our RDMA MVAPICH implementation. >> >> The avs_mpi_integration.pdf document provides a brief description of >> the interfaces that were required to integrate a userlevel CP/R >> framework with MVAPICH/RDMA. This provides an insight into what we >> required to integrate with a real-world RDMA MPI stack to provide CP/ >> R with very little overhead. >> >> The third document is one I wrote this afternoon to propose an >> asynchronous, quiescence interface. It is similar to Joshua's >> proposal on the wiki, but provides an asynchronous, driver-level >> version that our solution requires for application transparency. >> >> I will try to get some of these onto the wiki as well. >> >> >> Cheers, >> >> Mike >> >> -- >> >> Mike Heffner <mike.heffner(a)evergrid.com> >> EverGrid Software >> Blacksburg, VA USA >> >> Voice: (540) 443-3500 x603 >> < >> ipdps07 >> .pdf >> < >> avs_mpi_integration >> .pdf >> < >> evergrid_async_quiesce >> .pdf>_______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- Mike Heffner <mike.heffner(a)evergrid.com> EverGrid Software Blacksburg, VA USA Voice: (540) 443-3500 x603

1 0

Re: [Mpi3-ft] docs on CP/R with RDMA fabrics
by Josh Hursey 06 Jun '08

06 Jun '08

Mike, I was not on the last teleconf so this might have been covered there, but just to make sure I understand the later two proposals you sent. You are not proposing any MPI interfaces for checkpoint/restart, but just describing how you implemented a transparent solution inside or below an MPI implementation. Is that correct? -- Josh On Jun 5, 2008, at 5:44 PM, Mike Heffner wrote: > [3rd try around, mailing list bounce] > > All, > > Attached are the documents I promised during the previous conference > call. > > The IPDPS 2007 paper gives a technical description of the > Availability Services product (known academically as "DejaVu") > including the online logging algorithm used for BSD socket > applications. It also includes some performance numbers from > previous experiments with our RDMA MVAPICH implementation. > > The avs_mpi_integration.pdf document provides a brief description of > the interfaces that were required to integrate a userlevel CP/R > framework with MVAPICH/RDMA. This provides an insight into what we > required to integrate with a real-world RDMA MPI stack to provide CP/ > R with very little overhead. > > The third document is one I wrote this afternoon to propose an > asynchronous, quiescence interface. It is similar to Joshua's > proposal on the wiki, but provides an asynchronous, driver-level > version that our solution requires for application transparency. > > I will try to get some of these onto the wiki as well. > > > Cheers, > > Mike > > -- > > Mike Heffner <mike.heffner(a)evergrid.com> > EverGrid Software > Blacksburg, VA USA > > Voice: (540) 443-3500 x603 > < > ipdps07 > .pdf > > > < > avs_mpi_integration > .pdf > > > < > evergrid_async_quiesce > .pdf>_______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

[Mpi3-ft] docs on CP/R with RDMA fabrics
by Mike Heffner 05 Jun '08

05 Jun '08

[3rd try around, mailing list bounce] All, Attached are the documents I promised during the previous conference call. The IPDPS 2007 paper gives a technical description of the Availability Services product (known academically as "DejaVu") including the online logging algorithm used for BSD socket applications. It also includes some performance numbers from previous experiments with our RDMA MVAPICH implementation. The avs_mpi_integration.pdf document provides a brief description of the interfaces that were required to integrate a userlevel CP/R framework with MVAPICH/RDMA. This provides an insight into what we required to integrate with a real-world RDMA MPI stack to provide CP/R with very little overhead. The third document is one I wrote this afternoon to propose an asynchronous, quiescence interface. It is similar to Joshua's proposal on the wiki, but provides an asynchronous, driver-level version that our solution requires for application transparency. I will try to get some of these onto the wiki as well. Cheers, Mike -- Mike Heffner <mike.heffner(a)evergrid.com> EverGrid Software Blacksburg, VA USA Voice: (540) 443-3500 x603

1 0

[Mpi3-ft] docs on CP/R with RDMA fabrics
by Mike Heffner 05 Jun '08

05 Jun '08

All, Attached are the documents I promised during the previous conference call. The IPDPS 2007 paper gives a technical description of the Availability Services product (known academically as "DejaVu") including the online logging algorithm used for BSD socket applications. It also includes some performance numbers from previous experiments with our RDMA MVAPICH implementation. The avs_mpi_integration.pdf document provides a brief description of the interfaces that were required to integrate a userlevel CP/R framework with MVAPICH/RDMA. This provides an insight into what we required to integrate with a real-world RDMA MPI stack to provide CP/R with very little overhead. The third document is one I wrote this afternoon to propose an asynchronous, quiescence interface. It is similar to Joshua's proposal on the wiki, but provides an asynchronous, driver-level version that our solution requires for application transparency. I will try to get some of these onto the wiki as well. Cheers, -- Mike Heffner <mike.heffner(a)evergrid.com> EverGrid Software Blacksburg, VA USA Voice: (540) 443-3500 x603

1 0

Re: [Mpi3-ft] Asynchronous error handling
by Richard Graham 02 Jun '08

02 Jun '08

Greg, I like your suggestion how about if we adopt what the CIFTS project is doing as our model (s) ? These are methods already in use in other contexts, and have been proven to be useful. Seems like there are several items that would need to be addressed such as: - Is reliable delivery guaranteed ? - Is notification unique i.e., can we have an error code returned from MPI_Isend() and a callback also be generated, both for the (subscribed) error condition. - What happens with errors that impact correctness, but that have not been subscribed to ? Lets schedule a long chunk of time (about 4 hours) for our working group to meet and talk at our next meeting we will follow up on what we discuss this coming Friday. Rich On 5/26/08 11:51 AM, "Greg Bronevetsky" <bronevetsky1(a)llnl.gov> wrote: > > >> > On the telecon today we agreed to have our next telecon on 6/6 focus on >> how >> >we may handle asynchronous error notification within MPI. The working >> >assumption is that we will still have return error codes, but also make use >> >of asynchronous notification. We need to >> > - Clearly define the boundary between these two different error >> >notification mechanisms, i.e., when we use one and when the other >> > - Define the precise mechanism for asynchronous error notification >> >This e-mail is intended to jump start discussion in preparation for the next >> >telecon. > > I'll throw something out here. I wasn't around for the initial > discussions, so some of this may fly in the face of something that > people have already decided is obviously wrong. Either way, its a > start. You may commence with the tomato throwing. > > The idea for this proposal is a publish-subscribe model where the > spec defines the default publish-subscribe relations but allows MPI > implementations to define new events and default and allows > applications to cancel/add new event subscriptions. I like this model > mostly because it is the one being used by the CIFTS project, which I > suspect will have an important role to play in MPI application fault > tolerance. Since we won't be able to list all the possible errors > that may occur, we'll need to define the possible error types and > describe describe the error notification properties of these broad > types, rather than individual events. Implementations may then put > each real error into any type that is deemed appropriate. > > Every error will have a defined detection set, which is the set of > processes that by default subscribe to being notified of this event. > For example, if a given process fails, any process that tries to > receive a message from this process is definitely within its > detection radius. However, if the failed process is a receiver in a > broadcast, we may or may not choose to include the other broadcast > receivers in the detection radius (probably not). Each process is > subscribed to all error events that happen in the process, as long as > the errors don't cause the process itself to fail. > > For each failure event type we will define the latest point in time > when each process within the event's detection set will be notified. > For example, if process p fails, all other processes must be notified > no later than their next receive call that must receive from p (i.e. > receives with MPI_ANY_SOURCE don't qualify). For errors that cause > process state to be corrupted, we may want to inform other processes > no later than the first point in time when their state becomes > dependent on the corruption. The MPI implementation may deliver the > event at this latest point using the synchronous error API or at any > earlier point in time using the asynchronous API. > > The synchronous API will be a direct extension of the current error > reporting API. The asynchronous API will take the form of an events > queue that may be explicitly polled by the application to see if > there are any pending events. Applications will also be able to > register a callback function that will automatically be called by MPI > whenever a new event arrives. Furthermore, processes may subscribe to > events emanating from other processes as they see fit. For example, > the application may designate one or more processes as error monitors > and these processes would register themselves to listen to all other > processes and take appropriate corrective measures if something goes wrong. > > Greg Bronevetsky > Post-Doctoral Researcher > 1028 Building 451 > Lawrence Livermore National Lab > (925) 424-5756 > bronevetsky1(a)llnl.gov > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >

1 0