December 2011 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] RTS Proposal Overview Slides
by Bronevetsky, Greg 21 Dec '11

21 Dec '11

Josh, since I've been out of the loop I took the chance to catch up using the slides. They look good! Thanks for the great work! Greg Bronevetsky Lawrence Livermore National Lab (925) 424-5756 bronevetsky(a)llnl.gov http://greg.bronevetsky.com > -----Original Message----- > From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- > bounces(a)lists.mpi-forum.org] On Behalf Of Josh Hursey > Sent: Wednesday, December 21, 2011 8:28 AM > To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group > Subject: [Mpi3-ft] RTS Proposal Overview Slides > > I just uploaded the first draft of the overview slides to the FT WG wiki page. > These slides are intended to be presented during the plenary session on > Monday, Jan 9. > > These slides will probably be expanded in the coming weeks to include more > examples, and refine the text a bit. Once the we are satisfied with the slides, > I will send the link out to the full forum so that people can look over them > before the meeting. > > The slides can be found at the links below: > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/rts_proposal_main > https://svn.mpi-forum.org/trac/mpi-forum- > web/attachment/wiki/ft/rts_proposal_main/RTS-Overview-2012-01-v1.pdf > > Let me know what you think. > > Thanks, > Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

[Mpi3-ft] RTS Proposal Overview Slides
by Josh Hursey 21 Dec '11

21 Dec '11

I just uploaded the first draft of the overview slides to the FT WG wiki page. These slides are intended to be presented during the plenary session on Monday, Jan 9. These slides will probably be expanded in the coming weeks to include more examples, and refine the text a bit. Once the we are satisfied with the slides, I will send the link out to the full forum so that people can look over them before the meeting. The slides can be found at the links below: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/rts_proposal_main https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/ft/rts_proposa… Let me know what you think. Thanks, Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] New revision of RTS proposal
by Josh Hursey 20 Dec '11

20 Dec '11

Updated document now available on the website. I'm planning on sending this around to the forum at 5 pm eastern. -- Josh On Mon, Dec 19, 2011 at 1:44 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > A new version of the document is available on the following wiki page: > ï¿½https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/rts_proposal_main > I think this one is ready to go. If you have time, take a look over it > and send notes to the list. I am planning on sending this out to the > MPI Forum mailing list (and attaching it to the ticket) tonight or > tomorrow morning. > > I'm working on the slides for the meeting. I hope to have a version to > circulate tomorrow afternoon. > > Thanks, > Josh > Change Log:-----------ï¿½* Minor wording touch up clarificationsï¿½* 17.6: > Added clarification to MPI_Comm_reenable_any_source() that is ok to > pass MPI_GROUP_IGNORE.ï¿½* 17.6: Added clarification that MPI_Comm_drain > will match across process failure notification. Similar to > MPI_Comm_validate.ï¿½* 17.7.1: Added matching clarification to > MPI_Comm_validate.ï¿½* 17.10: Clarify that MPI_Comm_disconnect can be > passed a collectively inactive communicator. > > On Fri, Dec 16, 2011 at 3:36 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: >> A new version of the document is available on the following wiki page: >> ï¿½https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/rts_proposal_main >> >> We have to have this document finalized by COB Monday if at all >> possible. So take a look at it and send feedback to the list. >> >> -- Josh >> >> Change Log: >> ----------- >> ï¿½* Fix a few typos >> ï¿½* 17.5.1: Update the Process Failure handler section per discussions >> ï¿½* 17.6: Touch up the MPI_ANY_SOURCE wording >> ï¿½* 17.6: Clarify that if MPI_Comm_reenable_any_source is called with >> an intercommunicator then it returns the group of failed processes in >> the remote group. >> ï¿½* 17.6: Advice to implementors that MPI_Comm_drain maybe a locally >> collective operation. >> ï¿½* 17.6: Extend advice to users to note that MPI_FAILHANDLER_MODE_ALL >> does not guarantee that the failure handler will be called the same >> number of times unless they call a validation operation to synchronize >> the handlers. So they need to take care when using MPI_Comm_drain() in >> this operating mode. >> ï¿½* 17.6.1: Add a sentence allowing MPI_ERR_IN_STATUS to be returned >> from test and completion operations even if it is just the >> MPI_ERR_ANY_SOURCE_DISABLED warning. >> ï¿½* 17.8.2: Communicator creation must have a collectively active input >> communicator, and return uniformly at all processes. >> ï¿½* 17.8.2: Communicator construction operations will match across >> process failure. So they match similar to MPI_Comm_validate() and not >> like other collectives. >> ï¿½* 17.8.3: Inter-communicator creation operations have the same >> constraints as communicator creation (Previous 2 points). >> ï¿½* 17.8.4: Added example section with the communicator creation loop example. >> ï¿½* 17.9: Topology creation operations match semantics of communicator creation. >> ï¿½* 17.10: Dynamic creation operations (spawn and friends) match >> semantics of communicator creation. >> ï¿½* 17.11: Window creation operations match semantics of communicator creation. >> ï¿½* 17.12.2: File_open/close match semantics of communicator creation. >> ï¿½* A.1.1: Added the MPI_FAILHANDLER_MODE_*'s to the appendix >> >> Open Discussion Items: >> ---------------------- >> ï¿½* 17.6: Rename MPI_Comm_reenable_any_source to *_validate_* (?) >> ï¿½* 3.10 & 17.6.2 : Do these sections conflict? Should the status only >> be associated with the 'source' since MPI_Recv would have returned the >> status value if the operations were called separately? >> >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] MPI_Comm_validate() protection
by Josh Hursey 20 Dec '11

20 Dec '11

Thanks for the wording modification. I just updated the document in SVN. I'll post a new draft on the website momentarily. I'm planning on sending this out at 5 pm eastern, unless someone raises a red flag. Thanks, Josh On Tue, Dec 20, 2011 at 2:43 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: > I think this is OK. ï¿½Some suggestions inline. > > -d > > On Dec 20, 2011, at 12:15 PM, Josh Hursey wrote: > >> For the resetting of the matching, how about we replace the >> problematic sentence in 17.7.1 with: >> ï¿½ "Upon successful re-activation of collective communication by a >> validation operation the matching of collective operations is reset, >> except where exempt (e.g., drain and communication object creation > ï¿½^^^^^^^^^^^^^^^^^^^^ > ï¿½unless otherwise specified >> operations). Therefore collective operations do not match across the > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½^^^^^^^^^^ > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½In other words, >> notification of a process failure. Validation operations will match >> across the notification of a process failure." > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] MPI_Comm_validate() protection
by Darius Buntinas 20 Dec '11

20 Dec '11

I think this is OK. Some suggestions inline. -d On Dec 20, 2011, at 12:15 PM, Josh Hursey wrote: > For the resetting of the matching, how about we replace the > problematic sentence in 17.7.1 with: > "Upon successful re-activation of collective communication by a > validation operation the matching of collective operations is reset, > except where exempt (e.g., drain and communication object creation ^^^^^^^^^^^^^^^^^^^^ unless otherwise specified > operations). Therefore collective operations do not match across the ^^^^^^^^^^ In other words, > notification of a process failure. Validation operations will match > across the notification of a process failure."

1 0

Re: [Mpi3-ft] MPI_Comm_validate() protection
by Josh Hursey 20 Dec '11

20 Dec '11

So posted collectives are covered by the statement in 17.7 of: ï¿½ "If a collectively inactive communicator is used in a collective operation (except where explicitly permitted, like for MPI_COMM_VALIDATE) the operation will complete and raise an error code of the class MPI_ERR_PROC_FAIL_STOP." That overs posting of a collectives on an inactive communicator. They are all supposed to complete either with error or success. For the resetting of the matching, how about we replace the problematic sentence in 17.7.1 with: ï¿½ "Upon successful re-activation of collective communication by a validation operation the matching of collective operations is reset, except where exempt (e.g., drain and communication object creation operations). Therefore collective operations do not match across the notification of a process failure. Validation operations will match across the notification of a process failure." How about that? -- Josh On Tue, Dec 20, 2011 at 12:21 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: > Yeah, this might be OK. ï¿½How do we word that in the standard? > > -d > > On Dec 20, 2011, at 10:44 AM, Josh Hursey wrote: > >> How about: >> When calling validate the epoch should only change when there is a new >> failure detected. So when switching from an inactive to active >> collective mode. >> >> If there is no new failure on the communicator, then, in your example, >> the Ibcast would complete successfully at all processes. Since without >> a new failure the epoch should not change on the communicator. >> >> If there is a new failure detected before or in the validate then a >> new 'cut' would be made (increments a global epoch on the >> communicator). The Ibcast in P0 would either complete successfully or >> with an error (depending on how the collective is implemented). The >> Ibcast in P1 would match after the 'cut'. So P1 would be waiting for >> P0 to post a new Ibcast to match. >> >> The user should be programing around such scenarios, so that >> collectives match correctly after a validate. So I would say that the >> example is correct, but unsafe without additional process failure >> checking logic. >> >> So only when a process failure occurs the validate forces the >> cancelation of outstanding collective operations, and effectively >> resets matching. >> >> Thoughts? >> >> -- Josh >> >> On Tue, Dec 20, 2011 at 11:16 AM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: >>> Hmm. ï¿½There was a reason we added that. >>> >>> I think this had to do with separating collectives from one "epoch" from those of another, so that no one does anything like this: >>> >>> P0 ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ P1 >>> ------------------- ï¿½ ï¿½------------------- >>> MPI_Ibcast() >>> MPI_Comm_validate() ï¿½ ï¿½MPI_Comm_validate() >>> . ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Ibcast() >>> MPI_Wait() ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ MPI_Wait() >>> >>> Maybe we should say that validate completes-with-error all outstanding collective operations. ï¿½But in that case if the user spans a collective across a validate the collective may complete successfully at one process (e.g., root of bcast) and complete-with-an-error on another, but validate() would show that no processes have failed (no process failures would normally indicate that the bcast completed successfully). ï¿½I guess we can call this a user error, and just say don't do that. >>> >>> -d >>> >>> On Dec 20, 2011, at 9:03 AM, Josh Hursey wrote: >>> >>>> In 17.7.1 the proposal states: >>>> ï¿½"All collective communication operations initiated before the call >>>> to MPI_COMM_VALIDATE must also complete before it is called, and no >>>> collective calls may be initiated until it has completed." >>>> >>>> Considering the case where FailHandlers are used in the 'ALL' >>>> operating mode. In this mode, a user may want to call validate in all >>>> of the FailHanlders to synchronize them. But if the FailHandler was >>>> triggered out of a collective operation over a communicator that does >>>> -not- include the failed process then the user cannot write a correct >>>> program (since they cannot cancel, and may not be able to complete the >>>> collective operation). >>>> >>>> My suggestion is that we remove this sentence. Do folks have a problem >>>> with this? >>>> >>>> Thanks, >>>> Josh >>>> >>>> -- >>>> Joshua Hursey >>>> Postdoctoral Research Associate >>>> Oak Ridge National Laboratory >>>> http://users.nccs.gov/~jjhursey >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >> >> >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] MPI_Comm_validate() protection
by Darius Buntinas 20 Dec '11

20 Dec '11

Yeah, this might be OK. How do we word that in the standard? -d On Dec 20, 2011, at 10:44 AM, Josh Hursey wrote: > How about: > When calling validate the epoch should only change when there is a new > failure detected. So when switching from an inactive to active > collective mode. > > If there is no new failure on the communicator, then, in your example, > the Ibcast would complete successfully at all processes. Since without > a new failure the epoch should not change on the communicator. > > If there is a new failure detected before or in the validate then a > new 'cut' would be made (increments a global epoch on the > communicator). The Ibcast in P0 would either complete successfully or > with an error (depending on how the collective is implemented). The > Ibcast in P1 would match after the 'cut'. So P1 would be waiting for > P0 to post a new Ibcast to match. > > The user should be programing around such scenarios, so that > collectives match correctly after a validate. So I would say that the > example is correct, but unsafe without additional process failure > checking logic. > > So only when a process failure occurs the validate forces the > cancelation of outstanding collective operations, and effectively > resets matching. > > Thoughts? > > -- Josh > > On Tue, Dec 20, 2011 at 11:16 AM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: >> Hmm. There was a reason we added that. >> >> I think this had to do with separating collectives from one "epoch" from those of another, so that no one does anything like this: >> >> P0 P1 >> ------------------- ------------------- >> MPI_Ibcast() >> MPI_Comm_validate() MPI_Comm_validate() >> . MPI_Ibcast() >> MPI_Wait() MPI_Wait() >> >> Maybe we should say that validate completes-with-error all outstanding collective operations. But in that case if the user spans a collective across a validate the collective may complete successfully at one process (e.g., root of bcast) and complete-with-an-error on another, but validate() would show that no processes have failed (no process failures would normally indicate that the bcast completed successfully). I guess we can call this a user error, and just say don't do that. >> >> -d >> >> On Dec 20, 2011, at 9:03 AM, Josh Hursey wrote: >> >>> In 17.7.1 the proposal states: >>> "All collective communication operations initiated before the call >>> to MPI_COMM_VALIDATE must also complete before it is called, and no >>> collective calls may be initiated until it has completed." >>> >>> Considering the case where FailHandlers are used in the 'ALL' >>> operating mode. In this mode, a user may want to call validate in all >>> of the FailHanlders to synchronize them. But if the FailHandler was >>> triggered out of a collective operation over a communicator that does >>> -not- include the failed process then the user cannot write a correct >>> program (since they cannot cancel, and may not be able to complete the >>> collective operation). >>> >>> My suggestion is that we remove this sentence. Do folks have a problem >>> with this? >>> >>> Thanks, >>> Josh >>> >>> -- >>> Joshua Hursey >>> Postdoctoral Research Associate >>> Oak Ridge National Laboratory >>> http://users.nccs.gov/~jjhursey >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] MPI_Comm_validate() protection
by Josh Hursey 20 Dec '11

20 Dec '11

How about: When calling validate the epoch should only change when there is a new failure detected. So when switching from an inactive to active collective mode. If there is no new failure on the communicator, then, in your example, the Ibcast would complete successfully at all processes. Since without a new failure the epoch should not change on the communicator. If there is a new failure detected before or in the validate then a new 'cut' would be made (increments a global epoch on the communicator). The Ibcast in P0 would either complete successfully or with an error (depending on how the collective is implemented). The Ibcast in P1 would match after the 'cut'. So P1 would be waiting for P0 to post a new Ibcast to match. The user should be programing around such scenarios, so that collectives match correctly after a validate. So I would say that the example is correct, but unsafe without additional process failure checking logic. So only when a process failure occurs the validate forces the cancelation of outstanding collective operations, and effectively resets matching. Thoughts? -- Josh On Tue, Dec 20, 2011 at 11:16 AM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: > Hmm. ï¿½There was a reason we added that. > > I think this had to do with separating collectives from one "epoch" from those of another, so that no one does anything like this: > > P0 ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ P1 > ------------------- ï¿½ ï¿½------------------- > MPI_Ibcast() > MPI_Comm_validate() ï¿½ ï¿½MPI_Comm_validate() > . ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Ibcast() > MPI_Wait() ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ MPI_Wait() > > Maybe we should say that validate completes-with-error all outstanding collective operations. ï¿½But in that case if the user spans a collective across a validate the collective may complete successfully at one process (e.g., root of bcast) and complete-with-an-error on another, but validate() would show that no processes have failed (no process failures would normally indicate that the bcast completed successfully). ï¿½I guess we can call this a user error, and just say don't do that. > > -d > > On Dec 20, 2011, at 9:03 AM, Josh Hursey wrote: > >> In 17.7.1 the proposal states: >> ï¿½"All collective communication operations initiated before the call >> to MPI_COMM_VALIDATE must also complete before it is called, and no >> collective calls may be initiated until it has completed." >> >> Considering the case where FailHandlers are used in the 'ALL' >> operating mode. In this mode, a user may want to call validate in all >> of the FailHanlders to synchronize them. But if the FailHandler was >> triggered out of a collective operation over a communicator that does >> -not- include the failed process then the user cannot write a correct >> program (since they cannot cancel, and may not be able to complete the >> collective operation). >> >> My suggestion is that we remove this sentence. Do folks have a problem >> with this? >> >> Thanks, >> Josh >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] MPI_Comm_validate() protection
by Darius Buntinas 20 Dec '11

20 Dec '11

Hmm. There was a reason we added that. I think this had to do with separating collectives from one "epoch" from those of another, so that no one does anything like this: P0 P1 ------------------- ------------------- MPI_Ibcast() MPI_Comm_validate() MPI_Comm_validate() . MPI_Ibcast() MPI_Wait() MPI_Wait() Maybe we should say that validate completes-with-error all outstanding collective operations. But in that case if the user spans a collective across a validate the collective may complete successfully at one process (e.g., root of bcast) and complete-with-an-error on another, but validate() would show that no processes have failed (no process failures would normally indicate that the bcast completed successfully). I guess we can call this a user error, and just say don't do that. -d On Dec 20, 2011, at 9:03 AM, Josh Hursey wrote: > In 17.7.1 the proposal states: > "All collective communication operations initiated before the call > to MPI_COMM_VALIDATE must also complete before it is called, and no > collective calls may be initiated until it has completed." > > Considering the case where FailHandlers are used in the 'ALL' > operating mode. In this mode, a user may want to call validate in all > of the FailHanlders to synchronize them. But if the FailHandler was > triggered out of a collective operation over a communicator that does > -not- include the failed process then the user cannot write a correct > program (since they cannot cancel, and may not be able to complete the > collective operation). > > My suggestion is that we remove this sentence. Do folks have a problem > with this? > > Thanks, > Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

[Mpi3-ft] MPI_Comm_validate() protection
by Josh Hursey 20 Dec '11

20 Dec '11

In 17.7.1 the proposal states: "All collective communication operations initiated before the call to MPI_COMM_VALIDATE must also complete before it is called, and no collective calls may be initiated until it has completed." Considering the case where FailHandlers are used in the 'ALL' operating mode. In this mode, a user may want to call validate in all of the FailHanlders to synchronize them. But if the FailHandler was triggered out of a collective operation over a communicator that does -not- include the failed process then the user cannot write a correct program (since they cannot cancel, and may not be able to complete the collective operation). My suggestion is that we remove this sentence. Do folks have a problem with this? Thanks, Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0