August 2013 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] ULFM Slides for Madrid
by Wesley Bland 19 Aug '13

19 Aug '13

FYI: I'll send out another version of the slides with any more edits that we come up with today with the announcement for the call later. Wesley On Aug 19, 2013, at 1:50 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: > > On Aug 19, 2013, at 12:57 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > >> Hi Wesley, >> >> Overall, this is going in a great direction. Thanks for all of the effort you and others have been putting into the presentation. >> >> Slide 4: "Only requirement is that failure are eventually reported to all processes which communicate with the failed process." I am wondering if this statement, to be precise, applies only to point-to-point communication? Is it possible for me to never find out about a failure of another process with whom I make collective calls? > > I've updated the slide to say this: > > Only requirement is that failure are eventually reported if they prevent correct completion of another operation. > > The intent of the text is to say that whenever the failure of a process will keep the correct thing from happening from my point of view, you must tell me about it. That means that if I'm doing pt2pt with the dead process, obviously I need to know about it because there is no way to get the right answer (unless it was sent before the process died). If I'm doing a collective that can complete correctly without the failed process, for instance a broadcast where one of the leaves failed, then I don't need to know about the failure. Basically, as long as all of the buffers are correct, then we don't need to report an error. > >> >> Slide 5: I haven't heard the term "algorithm completion" before. Would it be better to call this something like fault tolerant consensus? > > That makes more sense. Updated accordingly. > >> >> Slide 11: You should point out that we have no way to free a communicator, if it is not valid at all correct processes. Processes that noticed the failure wouldn't be able to call revoke, since they don't have a valid handle to the communicator. > > I didn't point out anywhere that we've modified the semantics of MPI_Comm_free to allow the implementation to free local resources if it's no longer possible to carry out the rest of the semantics for MPI_Comm_free. Said another way, if your communicator has been broken because of failures so collectives no longer work, then you can still call MPI_Comm_free and free all of the local resources to prevent a memory leak. I've added a slide to clarify this. > > (I don't see this text in the document anywhere. We need to fix this. I'm not sure where this text went.) > >> >> Slide 12: Should you clean up the communicator somehow, if the creation failed? Do you revoke or free it? Is this semantic defined -- seems like the same issue as on slide 11? > > Fixed on the slide per above text. > > Thanks, > Wesley > >> >> ~Jim. >> >> >> On Fri, Aug 16, 2013 at 1:17 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: >> I've put together a first draft of some slides that give an overview of ULFM for the forum meeting in Madrid for Rich to present. I think I captured most of the discussion we had on the last call relating to rationale, but if I missed something, feel free to add that to this deck or send me edits. >> >> I think the plan of action, as I understand it from Rich and Geoffroy, is to iterate on these slides until the next call on Tuesday and then we'll go over them as a group to make sure we're all on the same page. Rich, will you be able to attend the call this week (Tuesday, 3:00 PM EST)? If not, we can adjust it this week to make sure you can be there. >> >> Just to be clear, the goal of this presentation is to provide an overview of ULFM for the European crown that can't usually attend the forum meetings. This will probably be a review for many of the people who attend regularly, but there is some new rationale that we haven't included in the past when we've been putting these presentations together. I'd imagine that this meeting will have some confusion from the attendees where they might remember parts of the previous proposals and mix them, but if we can tell them to do a memory wipe ahead of time, that would help. >> >> Let me know what I've missed. >> >> Thanks, >> Wesley >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >

1 0

Re: [Mpi3-ft] ULFM Slides for Madrid
by Wesley Bland 19 Aug '13

19 Aug '13

On Aug 19, 2013, at 12:57 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > Hi Wesley, > > Overall, this is going in a great direction. Thanks for all of the effort you and others have been putting into the presentation. > > Slide 4: "Only requirement is that failure are eventually reported to all processes which communicate with the failed process." I am wondering if this statement, to be precise, applies only to point-to-point communication? Is it possible for me to never find out about a failure of another process with whom I make collective calls? I've updated the slide to say this: Only requirement is that failure are eventually reported if they prevent correct completion of another operation. The intent of the text is to say that whenever the failure of a process will keep the correct thing from happening from my point of view, you must tell me about it. That means that if I'm doing pt2pt with the dead process, obviously I need to know about it because there is no way to get the right answer (unless it was sent before the process died). If I'm doing a collective that can complete correctly without the failed process, for instance a broadcast where one of the leaves failed, then I don't need to know about the failure. Basically, as long as all of the buffers are correct, then we don't need to report an error. > > Slide 5: I haven't heard the term "algorithm completion" before. Would it be better to call this something like fault tolerant consensus? That makes more sense. Updated accordingly. > > Slide 11: You should point out that we have no way to free a communicator, if it is not valid at all correct processes. Processes that noticed the failure wouldn't be able to call revoke, since they don't have a valid handle to the communicator. I didn't point out anywhere that we've modified the semantics of MPI_Comm_free to allow the implementation to free local resources if it's no longer possible to carry out the rest of the semantics for MPI_Comm_free. Said another way, if your communicator has been broken because of failures so collectives no longer work, then you can still call MPI_Comm_free and free all of the local resources to prevent a memory leak. I've added a slide to clarify this. (I don't see this text in the document anywhere. We need to fix this. I'm not sure where this text went.) > > Slide 12: Should you clean up the communicator somehow, if the creation failed? Do you revoke or free it? Is this semantic defined -- seems like the same issue as on slide 11? Fixed on the slide per above text. Thanks, Wesley > > ~Jim. > > > On Fri, Aug 16, 2013 at 1:17 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: > I've put together a first draft of some slides that give an overview of ULFM for the forum meeting in Madrid for Rich to present. I think I captured most of the discussion we had on the last call relating to rationale, but if I missed something, feel free to add that to this deck or send me edits. > > I think the plan of action, as I understand it from Rich and Geoffroy, is to iterate on these slides until the next call on Tuesday and then we'll go over them as a group to make sure we're all on the same page. Rich, will you be able to attend the call this week (Tuesday, 3:00 PM EST)? If not, we can adjust it this week to make sure you can be there. > > Just to be clear, the goal of this presentation is to provide an overview of ULFM for the European crown that can't usually attend the forum meetings. This will probably be a review for many of the people who attend regularly, but there is some new rationale that we haven't included in the past when we've been putting these presentations together. I'd imagine that this meeting will have some confusion from the attendees where they might remember parts of the previous proposals and mix them, but if we can tell them to do a memory wipe ahead of time, that would help. > > Let me know what I've missed. > > Thanks, > Wesley > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] ULFM Slides for Madrid
by Jim Dinan 19 Aug '13

19 Aug '13

Hi Wesley, Overall, this is going in a great direction. Thanks for all of the effort you and others have been putting into the presentation. Slide 4: "Only requirement is that failure are eventually reported to all processes which communicate with the failed process." I am wondering if this statement, to be precise, applies only to point-to-point communication? Is it possible for me to never find out about a failure of another process with whom I make collective calls? Slide 5: I haven't heard the term "algorithm completion" before. Would it be better to call this something like fault tolerant consensus? Slide 11: You should point out that we have no way to free a communicator, if it is not valid at all correct processes. Processes that noticed the failure wouldn't be able to call revoke, since they don't have a valid handle to the communicator. Slide 12: Should you clean up the communicator somehow, if the creation failed? Do you revoke or free it? Is this semantic defined -- seems like the same issue as on slide 11? ~Jim. On Fri, Aug 16, 2013 at 1:17 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: > I've put together a first draft of some slides that give an overview of > ULFM for the forum meeting in Madrid for Rich to present. I think I > captured most of the discussion we had on the last call relating to > rationale, but if I missed something, feel free to add that to this deck or > send me edits. > > I think the plan of action, as I understand it from Rich and Geoffroy, is > to iterate on these slides until the next call on Tuesday and then we'll go > over them as a group to make sure we're all on the same page. Rich, will > you be able to attend the call this week (Tuesday, 3:00 PM EST)? If not, we > can adjust it this week to make sure you can be there. > > Just to be clear, the goal of this presentation is to provide an overview > of ULFM for the European crown that can't usually attend the forum > meetings. This will probably be a review for many of the people who attend > regularly, but there is some new rationale that we haven't included in the > past when we've been putting these presentations together. I'd imagine that > this meeting will have some confusion from the attendees where they might > remember parts of the previous proposals and mix them, but if we can tell > them to do a memory wipe ahead of time, that would help. > > Let me know what I've missed. > > Thanks, > Wesley > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >

1 0

Re: [Mpi3-ft] ULFM Slides for Madrid
by Sur, Sayantan 16 Aug '13

16 Aug '13

Ah, gotcha. Sayantan From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Wesley Bland Sent: Friday, August 16, 2013 1:55 PM To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group Subject: Re: [Mpi3-ft] ULFM Slides for Madrid I think my slide was unclear. The case I meant was if a process failed before the Allreduce. In that case, the Allreduce would always fail.. If the failure occurs during the algorithm, as you pointed out, it wouldn't necessarily fail everywhere. Thanks, Wesley On Friday, August 16, 2013 at 3:51 PM, Sur, Sayantan wrote: Hi Wesley, Thanks for sending the slides around. Does the assertion on Slide 6 and example on Slide 12 that “Allreduce would always fail” (in the case of failure of one of the participants) hold true? For example, an MPI implementation might have a terrible implementation of allreduce, where participating ranks send their buffer to a root, which does the reduction. The root then sends the results back to the participants one after the other. One of these p2p sends then fails. In this case, isn’t it possible that one rank gets MPI_ERR_PROC_FAILED, whereas the others get MPI_SUCCESS? Thanks, Sayantan From: mpi3-ft-bounces(a)lists.mpi-forum.org<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Wesley Bland Sent: Friday, August 16, 2013 10:17 AM To: MPI3-FT Working Group Subject: [Mpi3-ft] ULFM Slides for Madrid I've put together a first draft of some slides that give an overview of ULFM for the forum meeting in Madrid for Rich to present. I think I captured most of the discussion we had on the last call relating to rationale, but if I missed something, feel free to add that to this deck or send me edits. I think the plan of action, as I understand it from Rich and Geoffroy, is to iterate on these slides until the next call on Tuesday and then we'll go over them as a group to make sure we're all on the same page. Rich, will you be able to attend the call this week (Tuesday, 3:00 PM EST)? If not, we can adjust it this week to make sure you can be there. Just to be clear, the goal of this presentation is to provide an overview of ULFM for the European crown that can't usually attend the forum meetings. This will probably be a review for many of the people who attend regularly, but there is some new rationale that we haven't included in the past when we've been putting these presentations together. I'd imagine that this meeting will have some confusion from the attendees where they might remember parts of the previous proposals and mix them, but if we can tell them to do a memory wipe ahead of time, that would help. Let me know what I've missed. Thanks, Wesley _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org<mailto:[email protected]> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] ULFM Slides for Madrid
by Wesley Bland 16 Aug '13

16 Aug '13

I think my slide was unclear. The case I meant was if a process failed before the Allreduce. In that case, the Allreduce would always fail.. If the failure occurs during the algorithm, as you pointed out, it wouldn't necessarily fail everywhere. Thanks, Wesley On Friday, August 16, 2013 at 3:51 PM, Sur, Sayantan wrote: > Hi Wesley, > > Thanks for sending the slides around. Does the assertion on Slide 6 and example on Slide 12 that “Allreduce would always fail” (in the case of failure of one of the participants) hold true? > > For example, an MPI implementation might have a terrible implementation of allreduce, where participating ranks send their buffer to a root, which does the reduction. The root then sends the results back to the participants one after the other. One of these p2p sends then fails. In this case, isn’t it possible that one rank gets MPI_ERR_PROC_FAILED, whereas the others get MPI_SUCCESS? > > Thanks, > Sayantan > > From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Wesley Bland > Sent: Friday, August 16, 2013 10:17 AM > To: MPI3-FT Working Group > Subject: [Mpi3-ft] ULFM Slides for Madrid > > I've put together a first draft of some slides that give an overview of ULFM for the forum meeting in Madrid for Rich to present. I think I captured most of the discussion we had on the last call relating to rationale, but if I missed something, feel free to add that to this deck or send me edits. > > > > I think the plan of action, as I understand it from Rich and Geoffroy, is to iterate on these slides until the next call on Tuesday and then we'll go over them as a group to make sure we're all on the same page. Rich, will you be able to attend the call this week (Tuesday, 3:00 PM EST)? If not, we can adjust it this week to make sure you can be there. > > > > Just to be clear, the goal of this presentation is to provide an overview of ULFM for the European crown that can't usually attend the forum meetings. This will probably be a review for many of the people who attend regularly, but there is some new rationale that we haven't included in the past when we've been putting these presentations together. I'd imagine that this meeting will have some confusion from the attendees where they might remember parts of the previous proposals and mix them, but if we can tell them to do a memory wipe ahead of time, that would help. > > > > Let me know what I've missed. > > > > Thanks, > > Wesley > > > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org (mailto:[email protected]) > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >

1 0

Re: [Mpi3-ft] ULFM Slides for Madrid
by Sur, Sayantan 16 Aug '13

16 Aug '13

Hi Wesley, Thanks for sending the slides around. Does the assertion on Slide 6 and example on Slide 12 that “Allreduce would always fail” (in the case of failure of one of the participants) hold true? For example, an MPI implementation might have a terrible implementation of allreduce, where participating ranks send their buffer to a root, which does the reduction. The root then sends the results back to the participants one after the other. One of these p2p sends then fails. In this case, isn’t it possible that one rank gets MPI_ERR_PROC_FAILED, whereas the others get MPI_SUCCESS? Thanks, Sayantan From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Wesley Bland Sent: Friday, August 16, 2013 10:17 AM To: MPI3-FT Working Group Subject: [Mpi3-ft] ULFM Slides for Madrid I've put together a first draft of some slides that give an overview of ULFM for the forum meeting in Madrid for Rich to present. I think I captured most of the discussion we had on the last call relating to rationale, but if I missed something, feel free to add that to this deck or send me edits. I think the plan of action, as I understand it from Rich and Geoffroy, is to iterate on these slides until the next call on Tuesday and then we'll go over them as a group to make sure we're all on the same page. Rich, will you be able to attend the call this week (Tuesday, 3:00 PM EST)? If not, we can adjust it this week to make sure you can be there. Just to be clear, the goal of this presentation is to provide an overview of ULFM for the European crown that can't usually attend the forum meetings. This will probably be a review for many of the people who attend regularly, but there is some new rationale that we haven't included in the past when we've been putting these presentations together. I'd imagine that this meeting will have some confusion from the attendees where they might remember parts of the previous proposals and mix them, but if we can tell them to do a memory wipe ahead of time, that would help. Let me know what I've missed. Thanks, Wesley

1 0

[Mpi3-ft] ULFM Slides for Madrid
by Wesley Bland 16 Aug '13

16 Aug '13

I've put together a first draft of some slides that give an overview of ULFM for the forum meeting in Madrid for Rich to present. I think I captured most of the discussion we had on the last call relating to rationale, but if I missed something, feel free to add that to this deck or send me edits. I think the plan of action, as I understand it from Rich and Geoffroy, is to iterate on these slides until the next call on Tuesday and then we'll go over them as a group to make sure we're all on the same page. Rich, will you be able to attend the call this week (Tuesday, 3:00 PM EST)? If not, we can adjust it this week to make sure you can be there. Just to be clear, the goal of this presentation is to provide an overview of ULFM for the European crown that can't usually attend the forum meetings. This will probably be a review for many of the people who attend regularly, but there is some new rationale that we haven't included in the past when we've been putting these presentations together. I'd imagine that this meeting will have some confusion from the attendees where they might remember parts of the previous proposals and mix them, but if we can tell them to do a memory wipe ahead of time, that would help. Let me know what I've missed. Thanks, Wesley

1 0

Re: [Mpi3-ft] ULFM Rationale Slides
by Thomas Naughton 06 Aug '13

06 Aug '13

Attached is a version of the nodes from whiteboard to help with current discussion on teleconf. --tjn _________________________________________________________________________ Thomas Naughton naughtont(a)ornl.gov Research Associate (865) 576-4184 On Tue, 6 Aug 2013, Wesley Bland wrote: > As promised on the call last week, here's some slides discussing the > rationale that we developed at the face-to-face meeting @ UTK. We'll discuss > this more on the call this afternoon. I'm attaching a Dropbox link this time > instead of sending an attachment as those seem to be filtered by the > listserv. > > Some of these slides will be a quick overview for those who might have > forgotten the basic ideas of ULFM. > > ULFM Rationale Slides.pdf > > Thanks, > Wesley > >

1 0

Re: [Mpi3-ft] ULFM Rationale Slides
by Wesley Bland 06 Aug '13

06 Aug '13

I agree. This was essentially a beginning for what will eventually become the slides for the plenary when we're ready to do the reading. There's certainly more to add and I'll make sure that stuff gets in there. On Tuesday, August 6, 2013 at 12:00 PM, Jim Dinan wrote: > Hi Wesley, > > This slide deck is missing 10-15 minutes of introductory material that describes the failure model, illustrate how these failures affect MPI / enumerate the problems that need to be solved (e.g. wildcards, etc), and how the proposed solution addresses these problems at a high level. This material will give people context they need to understand why you have chosen this interface, what the constraints are, and how well it addresses the critical problems. > > ~Jim. > > > On Tue, Aug 6, 2013 at 10:10 AM, Wesley Bland <wbland(a)mcs.anl.gov (mailto:[email protected])> wrote: > > As promised on the call last week, here's some slides discussing the rationale that we developed at the face-to-face meeting @ UTK. We'll discuss this more on the call this afternoon. I'm attaching a Dropbox link this time instead of sending an attachment as those seem to be filtered by the listserv. > > > > Some of these slides will be a quick overview for those who might have forgotten the basic ideas of ULFM. > > > > ULFM Rationale Slides.pdf (http://db.tt/Zbjrl4bM) > > > > Thanks, > > Wesley > > > > _______________________________________________ > > mpi3-ft mailing list > > mpi3-ft(a)lists.mpi-forum.org (mailto:[email protected]) > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org (mailto:[email protected]) > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >

1 0

Re: [Mpi3-ft] ULFM Rationale Slides
by Jim Dinan 06 Aug '13

06 Aug '13

Hi Wesley, This slide deck is missing 10-15 minutes of introductory material that describes the failure model, illustrate how these failures affect MPI / enumerate the problems that need to be solved (e.g. wildcards, etc), and how the proposed solution addresses these problems at a high level. This material will give people context they need to understand why you have chosen this interface, what the constraints are, and how well it addresses the critical problems. ~Jim. On Tue, Aug 6, 2013 at 10:10 AM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: > As promised on the call last week, here's some slides discussing the > rationale that we developed at the face-to-face meeting @ UTK. We'll > discuss this more on the call this afternoon. I'm attaching a Dropbox link > this time instead of sending an attachment as those seem to be filtered by > the listserv. > > Some of these slides will be a quick overview for those who might have > forgotten the basic ideas of ULFM. > > ULFM Rationale Slides.pdf <http://db.tt/Zbjrl4bM> > > Thanks, > Wesley > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >

1 0