September 2013 - mpiwg-ft - mailman3.cels.anl.gov

FTWG Con Call 2013-10-01
by Wesley Bland 30 Sep '13

30 Sep '13

Dear WG members, This is a reminder that according to our planning, we are having our regular phone meeting tomorrow at 3pm EDT. NOTE THE NEW CALL-IN NUMBER. This a permanent change from the old number. Date: October 1, Time: 3pm EDT/New York Dial-in information: 712-432-0360 Code: 623998# Agenda: * Wrap up from Madrid meeting * Plans moving forward Next Meetings: * October 15, 2013 * October 29, 2013

1 0

Re: [mpiwg-ft] Madrid Report
by Aurélien Bouteiller 26 Sep '13

26 Sep '13

Le 25 sept. 2013 à 18:51, Bronis R. de Supinski <bronis(a)llnl.gov> a écrit : > > Wesley: > > I disagree vehemently with your statement that "either one > individually would be useless." While I was not at the > meeting (and thus did not raise the objections or make this > suggestion -- this time), I consider this statement to be > a continued example of how those working on the proposal > have failed to understand what users most desire from MPI > in terms of a first step for fault tolerance. The biggest > concern is simply that the MPI standard states on p28:24-26: > > This document does not specify the state of a computation > after an erroneous MPI call has occurred. The desired > behavior is that a relevant error code be returned, and > the effect of the error be localized to the greatest > possible extent. > > I believe at least one other example exists of this sort of > text but I will not look for it now. The point is that the > user has no guarantees about the state of MPI, the computation > or anything else after an error. While that may be unavoidable, > the user also has no why to query the implementation about > what the most likely state is. Thus, the user must make the > most pessimistic assumption and catching an error is pretty > much useless other than to implement some very basic message - > or if the user is feeling frisky, some more complex action - > prior to shutdown soon after the error is observed. > Bronis, You fail to understand that beside Shrink, the entire proposal is focused on defining exactly this piece: how the application can understand and resolve the consequence of failures. This proposal has 2 parts, one is giving the opportunity for applications to get that a failure impacted them. This can be can be very simple for some applications, but it can also be rather complex for some with transitive communication patterns, hence the need for some extra bullets like Revoke and Agree, beyond simple error codes. If you stay at the error code level, some application patterns will remain deadlocked (granted, some other application pattern do not need this, good for them, we split the logic in error codes and separate calls so that they can invoke the appropriate level of consistency only when needed). I invite you to read the papers we published that explain all the rationale. I'm sure you'll see the point more clearly then. The second one is Shrink, which is very simple, has no cost on performance, and is of undisputed usefulness to numerous users, if not most. So in essence, you want to have 2 proposals, one that contains everything, the other one with shrink only. There's no point in this. > I think the working group has listened to a small set of users No, we have not. We do have users doing coordinated Checkpoint/restart, bag of tasks, or coordinated algorithmic recovery. Our point is to define MPI state and how to make it communicative again, nothing more. Thereby our users are every code that wants to resume communicating after a failure. I have the impression that you believe that we can do ABFT only (or that it is a favored citizen), which is as far from truth as it can get. I reiterate, our goal is to enable applications (whatever FT method they use) to communicate again, nothing more, but nothing less (meaning all applications, even those with complex communication patterns that would deadlock irremediably with your truncated error reporting model). > and missed that the vast majority of users would be happy (at > least initially) if the standard provided such a capability. > I have spoken to numerous users for whom this statement is > true. Further, such a capability would be fairly easy to make > low cost. While some details would need to be worked out, Devil is in the details. If you actually tried to figure them out, you would have found the difficulties along the path and you would know that what you propose doesn't work. Best, Aurelien > which would take time, you would find wide acceptance of > that approach -- as opposed to trying to ram your complex, > unintuitive interface down the throats of those who object. > > That brings me to my other major objection to your email, > You state "I think we all expect that this proposal won't > be passing unanimously." The point of deferring the FT > proposal was not to have it pushed with increase urgency. > The point was to have objections addressed and to make > the adopted interface broadly accepted. You still seem > to think that a narrowly passed proposal would be OK. > > Bronis > > > > > > > > > On Wed, 25 Sep 2013, Wesley Bland wrote: > >> Let me add my two cents here as well. >> >> As Jim mentioned, this wasn't a rehashed slide deck. This was a new one. The confusion may have come from the fact that there haven't been any changes to the proposal recently and the talk that Rich gave was designed to be informational for those who hadn't attended the US meetings recently. >> >> As Jim mentioned, this was really supposed to be a justification talk to give background and rationale. It sounds like we may not have achieved that goal. It's unfortunate that more of us couldn't be there to help the conversation, but I'd like to try to find out more about the specific concerns that were raised as we've been working very hard to respond to these sorts of things in the past. >> >> I don't know if you actually meant that there was a desire to see the proposal split in two or not, but I'd really be against that. It would present a lot of work and without both pieces, either one individually would be useless. We've gone through the text many times to be sure that we're presenting the least intrusive interface possible that still accomplishes all of the needs of FT. As you mentioned, it isn't the most user friendly interface and we don't expect it to be. However, the real user friendly interfaces (C/R, replication, migration, strong consistency, etc.) can be built on top of the proposal, which is what we intended in our design. If that's one of the objections, then we many need to rethink things here or do a better job of convincing the forum that this is a good idea. FWIW, I believe that was always the original intent of MPI anyway. >> >> If it's going to be a problem to bring this to the forum for a reading in December, we can push things out, but we've been targeting that date and I'd like to have a better idea of what the objections are before we decide to delay again. If it's just the same arguments again, then it may not do any good to push this for another 3-6 months. I think we all expect that this proposal won't be passing unanimously. >> >> Thanks for the feedback Martin. I know that you probably weren't the one objecting and those who were probably aren't on this list. It's too bad that more of us couldn't be at the meeting, but I think we're all familiar with the funding situation in the US right now for international travel. >> >> Thanks, >> Wesley >> >> On Sep 25, 2013, at 8:23 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: >> >>> Hi Martin, >>> >>> The slides that were presented in Madrid were actually a new deck. We had tried to address feedback that there is a need to do a better job of covering the core issues, potential approaches, and justifications for design decisions, beyond just presenting the proposal. It sounds like more discussion around these topics is needed, and we can work with that for the Chicago meeting. >>> >>> It sounds like we're being asked to break the proposal up into two sets of changes -- one that defines failures in MPI and MPI's state after a failure; and second that adds a recovery API. Is that right? I'm not sure if it's possible to tease these things apart, but we can look into it. >>> >>> Cheers, >>> ~Jim. >>> >>> >>> On Tue, Sep 24, 2013 at 2:48 PM, Martin Schulz <schulzm(a)llnl.gov> wrote: >>>> >>>> Hi all, >>>> >>>> This is more from the viewpoint of an external observer, since I haven't had the time to really participate lately. >>>> >>>> Rich gave a presentation covering the current proposal, but he also said that this was basically the old slide deck that had been shown before and didn't contain any of the new work that the FT group has been doing in coordinating with the other WGs. There were also several questions that people asked and that we couldn't answer since Rich was the only FT group member at the meeting. >>>> >>>> Based on this I got the feeling that many in the forum were still concerned about this overarching proposal and would like to hear more. In particular, a few meetings back after the MPI 3.0 vote we had a discussion about what was missing to make such concerns go away and one thing that we agreed on was to first go through the standard and clean things up to make it compatible with an FT proposal. From what I can tell, a lot of this has happened and it may help to explicitly present that before even going into the API approach. This (plus an updated talk with a broader discussion with more FT group members being present) may be better for the Chicago meeting than doing an actual formal reading (which may be too early since there doesn't seem to be consensus about the approach, yet). >>>> >>>> From the application side, I have seen a few people starting to use it. One of our PDs is using it for a large MD application and there is also the following work I saw at a recent conference: >>>> >>>> ftp://ftp.inf.ethz.ch/pub/publications/tech-reports/7xx/793.pdf >>>> >>>> Both said that the interface is overly complex and has a very (too?) high of an impact on applications, which makes me worried. Not sure how we can address this, though. >>>> >>>> Martin >>>> >>>> >>>> >>>> >>>> On Sep 19, 2013, at 6:43 PM, Aurélien Bouteiller <bouteill(a)icl.utk.edu> wrote: >>>> >>>> I am currently in Europe and busy. My schedule will get back to normal in october. >>>> >>>> I'm also interested in a summary of events though. >>>> >>>> Aurelien >>>> >>>> >>>> Le 20 sept. 2013 à 00:42, Wesley Bland <wbland(a)mcs.anl.gov> a écrit : >>>> >>>> Hi WG, >>>> >>>> I've been out if the loop for a bit since my wife had our first baby last week (everyone is doing great). I was wondering how things went at the forum meeting last week. Was there any feedback from those in attendance? >>>> >>>> Also, I didn't see an email about this week's con call. I assume that it didn't happen, but if it did can anyone mention what was discussed? >>>> >>>> Thanks, >>>> Wesley >>>> _______________________________________________ >>>> mpiwg-ft mailing list >>>> mpiwg-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>>> >>>> >>>> -- >>>> * Dr. Aurélien Bouteiller >>>> * Researcher at Innovative Computing Laboratory >>>> * University of Tennessee >>>> * 1122 Volunteer Boulevard, suite 309b >>>> * Knoxville, TN 37996 >>>> * 865 974 9375 >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> mpiwg-ft mailing list >>>> mpiwg-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>>> >>>> >>>> ________________________________________________________________________ >>>> Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm >>>> CASC @ Lawrence Livermore National Laboratory, Livermore, USA >>>> >>>> >>>> _______________________________________________ >>>> mpiwg-ft mailing list >>>> mpiwg-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>> _______________________________________________ >>> mpiwg-ft mailing list >>> mpiwg-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >> > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft -- * Dr. Aurélien Bouteiller * Researcher at Innovative Computing Laboratory * University of Tennessee * 1122 Volunteer Boulevard, suite 309b * Knoxville, TN 37996 * 865 974 9375

1 0

Re: [mpiwg-ft] Madrid Report
by Aurélien Bouteiller 26 Sep '13

26 Sep '13

Le 24 sept. 2013 à 20:48, Martin Schulz <schulzm(a)llnl.gov> a écrit : > Hi all, > > This is more from the viewpoint of an external observer, since I haven't had the time to really participate lately. > > Rich gave a presentation covering the current proposal, but he also said that this was basically the old slide deck that had been shown before and didn't contain any of the new work that the FT group has been doing in coordinating with the other WGs. There were also several questions that people asked and that we couldn't answer since Rich was the only FT group member at the meeting. > > Based on this I got the feeling that many in the forum were still concerned about this overarching proposal and would like to hear more. In particular, a few meetings back after the MPI 3.0 vote we had a discussion about what was missing to make such concerns go away and one thing that we agreed on was to first go through the standard and clean things up to make it compatible with an FT proposal. From what I can tell, a lot of this has happened and it may help to explicitly present that before even going into the API approach. This (plus an updated talk with a broader discussion with more FT group members being present) may be better for the Chicago meeting than doing an actual formal reading (which may be too early since there doesn't seem to be consensus about the approach, yet). > > From the application side, I have seen a few people starting to use it. One of our PDs is using it for a large MD application and there is also the following work I saw at a recent conference: > > ftp://ftp.inf.ethz.ch/pub/publications/tech-reports/7xx/793.pdf > > Both said that the interface is overly complex and has a very (too?) high of an impact on applications, which > makes me worried. Not sure how we can address this, though. > Martin, thanks for the good feedback. I read the linked report and found no such reference to difficulties in using ULFM interface (on the contrary they state that "In the presence of failure some overhead is unavoidable because communicators have to be revoked and shrinked. Our results indicate that these effects are negligible.", which is extremely positive!). However they state difficulties with a bug in the implementation that we have recently fixed (errors in agreement). I would be very eager to hear the details of what was found difficult to use with the spec itself (not implementation bugs, which are obviously annoying). We still have time to account for good user feedback! Aurelien > Martin > > > > > On Sep 19, 2013, at 6:43 PM, Aurélien Bouteiller <bouteill(a)icl.utk.edu> wrote: > >> I am currently in Europe and busy. My schedule will get back to normal in october. >> >> I'm also interested in a summary of events though. >> >> Aurelien >> >> >> Le 20 sept. 2013 à 00:42, Wesley Bland <wbland(a)mcs.anl.gov> a écrit : >> >>> Hi WG, >>> >>> I've been out if the loop for a bit since my wife had our first baby last week (everyone is doing great). I was wondering how things went at the forum meeting last week. Was there any feedback from those in attendance? >>> >>> Also, I didn't see an email about this week's con call. I assume that it didn't happen, but if it did can anyone mention what was discussed? >>> >>> Thanks, >>> Wesley >>> _______________________________________________ >>> mpiwg-ft mailing list >>> mpiwg-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >> >> -- >> * Dr. Aurélien Bouteiller >> * Researcher at Innovative Computing Laboratory >> * University of Tennessee >> * 1122 Volunteer Boulevard, suite 309b >> * Knoxville, TN 37996 >> * 865 974 9375 >> >> >> >> >> >> >> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > > ________________________________________________________________________ > Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm > CASC @ Lawrence Livermore National Laboratory, Livermore, USA > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft -- * Dr. Aurélien Bouteiller * Researcher at Innovative Computing Laboratory * University of Tennessee * 1122 Volunteer Boulevard, suite 309b * Knoxville, TN 37996 * 865 974 9375

1 0

Re: [mpiwg-ft] Madrid Report
by Aurélien Bouteiller 26 Sep '13

26 Sep '13

Le 26 sept. 2013 à 00:10, Bronis R. de Supinski <bronis(a)llnl.gov> a écrit : > > Wesley: > > An approach similar to Init_Thread seems essential. > > As to what I was suggesting, it would not require any > polling or (additional) remote communication. It would > ideally be something along the lines of "MPI returned > this error code to me. If I make the same call again > will I get the same error code." The interface would > require some way to identify the call as well as the > error code. The answer would be easy for many types > of errors. For example, if the error corresponds to > bad arguments then the implementation can say that > "Yes, you will always get that error." However, the > implementation should be able to say that calls > with correct arguments will work. As the standard > currently is specified, one call with bad arguments > means all calls are is hosed, which is clearly unnecessary. > > Other error codes could provide different results. However, > the standard does not need to mandate them. For example, > if you have a process fail then you could indicate any > communication involving that process would fail. You seem > to be claiming that is basically the current proposed > interface. However, under a best effort query interface, > the user must actively request information for each error > code. Thus, the implementation could validly choose any of: > > 1. No machinery to detect remote process failures so > the best guess is that it will continue to work; > 2. A heuristic to detect remote process failure (e.g., > the last N messages to that process failed); > 3. A more complicated solution, likely corresponding > to what the current proposal requires. > Bronis, the current proposal can be upgraded to accommodate such supplementary fuzzy error codes. I encourage you to further define this behavior if you know of credible use cases, so it can be considered in future iterations of the forum. Aurelien > The second option is probably the best choice without > inexpensive detection that the failure has occurred. > It might be pessimistic at times but most users > would be OK with that. Most users would strongly > prefer that occasional pessimism in exchange for > higher performance. > > Bronis > > > > > On Wed, 25 Sep 2013, Wesley Bland wrote: > >> Hi Bronis, >> >> Thanks for the feedback. As Christian mentioned, most of what this proposal does is failure notification (as opposed to recovery). The biggest change (in terms of implementation) is found in the text in section 17.2.2: >> >> "In all other cases, the operation raises an exception of class MPI_ERR_PROC_FAILED to indicate that the failure prevents the operation from following its failure-free specification. If there is a request identifying the point-to-point communication, it is completed. Future point-to-point communication with the same process on this communicator must also raise MPI_ERR_PROC_FAILED." >> >> This (along with some other similar passages for things like collectives, startup/finalize, dynamic processes, etc. found in 17.2) specify how MPI should behave after a failure. There is other text sprinkled around the other chapters to clean up the sections you mentioned that obviously need to be updated to mention that we will now handle failure, but the other new additions are relatively minor from an implementation perspective. Also, it's our intent to combine ticket 336 (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/336) into this proposal to allow the user to query whether the implementation supports FT (this might show up in the form of an INIT_THREADS style request/supported system, but we still have to discuss it further). So if the user doesn't want/need FT, they're welcome to ignore it. >> >> As far as querying whether an error will persist, that requires an entirely different set of semantics. That would necessitate standardizing transient errors and specifying how an implementation should handle them. We're trying to make things as simple as possible. Either a process has failed and will always be failed, or it is still correct. As Christian mentioned, this can and should probably be handled with an exception system like the MPI Errhandler callbacks. >> >> I may be misunderstanding your meaning, but I think periodically polling the system to check if some process or set of processes is still alive would be rather cumbersome. Instead, the library can alert you when the failure occurs and do the hard work for you. Everything else we provide is a follow on from that, but it's designed to be minimally intrusive. >> >> To your last point, I'm certainly not rooting for a narrow passage, and after the most recent voting rules update, I don't think such a thing is possible anymore. We're not trying to ram this down anyone's throat at all. That's why we've been socializing it so much to get as much feedback as possible. We want to take into consideration all of the feedback that we're getting and provide rationale whenever we can. The reason we have been targeting December is that the feedback that I've been aware of as far back as the last vote in Japan was not that the proposal was technically deficient, but that it was too young and unstable. We've given it another year and a half to acquire more users, implementations, feedback, etc. and time to stabilize. We believe we've done that and are ready to give it another shot. The text is largely unchanged from the previous reading other than a few clarifications and some additions to the RMA and I/O sections. More feedback is encouraged and we > lcomed and if there's something that we can fix, we'll do so. >> >> Thanks, >> Wesley >> >> On Sep 25, 2013, at 1:43 PM, "Bronis R. de Supinski" <bronis(a)llnl.gov> wrote: >> >>> >>> Christian: >>> >>> My suggestion was not to remove a functipon from the existing >>> proposal in order to provide a query capability. My proposal >>> was to re-evaluate the decision to jump to a complex solution >>> prior to addressing the primary issue for most users. >>> >>> I am aware of the lineage of the paragraph that I cited. I >>> do not agree that its age implies it cannot be fixed. In >>> fact, no FT proposal has any value if it does not address >>> that paragraph. My suggestion was to take a simpler approach >>> that allows a user to query whether an error will persist >>> and, if so, does it imply other failures for various >>> classes of operations. Such an interface would be best >>> effort but would generally allow a simple "I can keep >>> trying (possibly ina slightly modified form until I think >>> the implementation does not know it is hosed" approach. >>> More complex approaches could follow but they would be >>> imprved by having such a query interface available. >>> >>> Bronis >>> >>> >>> >>> >>> >>> >>> >>> On Wed, 25 Sep 2013, Christian Engelmann wrote: >>> >>>> >>>> Bronis, the problem is that the paragraph you pointed out (and quoted below for reference) has been part of MPI since version 1.1. MPI was simply not conceived with fault tolerance or fault awareness in mind. Moreover, MPI was designed with a simplistic fault model, i.e., the state after any error is undefined. Rectifying this almost 20 years later is a difficult task. >>>> >>>> I agree that the proposed solution is not easy to comprehend from an application or library developer's point of view. I do think that fault tolerant applications that demonstrate the capabilities and usefulness of the proposed enhancements would help a lot. >>>> >>>> I also think that there is a general misunderstanding on how this interface is supposed to be used. Looping around every MPI call to catch and recover from potential errors is certainly a quite nonsensical approach. Instead, the more practical approach is transaction- or exception-based programming. This requires programming templates atop the proposed MPI enhancements. >>>> >>>> Once again, demonstrations using fault tolerant applications would really help. Martin already pointed to some. I know of other work by UT (Jack's group), UCR (Zizhong Chen), and UoH (Edgar Gabriel), all based on UT's FT-MPI from 2003. Moving those to the proposed MPI enhancements would help greatly. >>>> >>>> The only failure recovery function proposed is MPI_Comm_shrink(). Removing this, simply means that only point-to-point communication will work after a process fault, but not collectives. A point-to-point only MPI is pretty useless in my opinion. >>>> >>>> Christian >>>> >>>> On Sep 25, 2013, at 12:51 PM, Bronis R. de Supinski <bronis(a)llnl.gov> wrote: >>>> >>>>> This document does not specify the state of a computation >>>>> after an erroneous MPI call has occurred. The desired >>>>> behavior is that a relevant error code be returned, and >>>>> the effect of the error be localized to the greatest >>>>> possible extent. >>>> >>>> -- >>>> >>>> Christian Engelmann, Ph.D. >>>> >>>> System Software Team Task Lead / R&D Staff Scientist >>>> Computer Science Research Group >>>> Computer Science and Mathematics Division >>>> Oak Ridge National Laboratory >>>> >>>> Mail: P.O. Box 2008, Oak Ridge, TN 37831-6173, USA >>>> Phone: +1 (865) 574-3132 / Fax: +1 (865) 576-5491 >>>> e-Mail: engelmannc(a)ornl.gov / Home: www.christian-engelmann.info >>>> >>>> >>> _______________________________________________ >>> mpiwg-ft mailing list >>> mpiwg-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >> > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft -- * Dr. Aurélien Bouteiller * Researcher at Innovative Computing Laboratory * University of Tennessee * 1122 Volunteer Boulevard, suite 309b * Knoxville, TN 37996 * 865 974 9375

1 0

Re: [mpiwg-ft] Madrid Report
by Bronis R. de Supinski 25 Sep '13

25 Sep '13

Wesley: An approach similar to Init_Thread seems essential. As to what I was suggesting, it would not require any polling or (additional) remote communication. It would ideally be something along the lines of "MPI returned this error code to me. If I make the same call again will I get the same error code." The interface would require some way to identify the call as well as the error code. The answer would be easy for many types of errors. For example, if the error corresponds to bad arguments then the implementation can say that "Yes, you will always get that error." However, the implementation should be able to say that calls with correct arguments will work. As the standard currently is specified, one call with bad arguments means all calls are is hosed, which is clearly unnecessary. Other error codes could provide different results. However, the standard does not need to mandate them. For example, if you have a process fail then you could indicate any communication involving that process would fail. You seem to be claiming that is basically the current proposed interface. However, under a best effort query interface, the user must actively request information for each error code. Thus, the implementation could validly choose any of: 1. No machinery to detect remote process failures so the best guess is that it will continue to work; 2. A heuristic to detect remote process failure (e.g., the last N messages to that process failed); 3. A more complicated solution, likely corresponding to what the current proposal requires. The second option is probably the best choice without inexpensive detection that the failure has occurred. It might be pessimistic at times but most users would be OK with that. Most users would strongly prefer that occasional pessimism in exchange for higher performance. Bronis On Wed, 25 Sep 2013, Wesley Bland wrote: > Hi Bronis, > > Thanks for the feedback. As Christian mentioned, most of what this proposal does is failure notification (as opposed to recovery). The biggest change (in terms of implementation) is found in the text in section 17.2.2: > > "In all other cases, the operation raises an exception of class MPI_ERR_PROC_FAILED to indicate that the failure prevents the operation from following its failure-free specification. If there is a request identifying the point-to-point communication, it is completed. Future point-to-point communication with the same process on this communicator must also raise MPI_ERR_PROC_FAILED." > > This (along with some other similar passages for things like collectives, startup/finalize, dynamic processes, etc. found in 17.2) specify how MPI should behave after a failure. There is other text sprinkled around the other chapters to clean up the sections you mentioned that obviously need to be updated to mention that we will now handle failure, but the other new additions are relatively minor from an implementation perspective. Also, it's our intent to combine ticket 336 (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/336) into this proposal to allow the user to query whether the implementation supports FT (this might show up in the form of an INIT_THREADS style request/supported system, but we still have to discuss it further). So if the user doesn't want/need FT, they're welcome to ignore it. > > As far as querying whether an error will persist, that requires an entirely different set of semantics. That would necessitate standardizing transient errors and specifying how an implementation should handle them. We're trying to make things as simple as possible. Either a process has failed and will always be failed, or it is still correct. As Christian mentioned, this can and should probably be handled with an exception system like the MPI Errhandler callbacks. > > I may be misunderstanding your meaning, but I think periodically polling the system to check if some process or set of processes is still alive would be rather cumbersome. Instead, the library can alert you when the failure occurs and do the hard work for you. Everything else we provide is a follow on from that, but it's designed to be minimally intrusive. > > To your last point, I'm certainly not rooting for a narrow passage, and after the most recent voting rules update, I don't think such a thing is possible anymore. We're not trying to ram this down anyone's throat at all. That's why we've been socializing it so much to get as much feedback as possible. We want to take into consideration all of the feedback that we're getting and provide rationale whenever we can. The reason we have been targeting December is that the feedback that I've been aware of as far back as the last vote in Japan was not that the proposal was technically deficient, but that it was too young and unstable. We've given it another year and a half to acquire more users, implementations, feedback, etc. and time to stabilize. We believe we've done that and are ready to give it another shot. The text is largely unchanged from the previous reading other than a few clarifications and some additions to the RMA and I/O sections. More feedback is encouraged and welcomed and if there's something that we can fix, we'll do so. > > Thanks, > Wesley > > On Sep 25, 2013, at 1:43 PM, "Bronis R. de Supinski" <bronis(a)llnl.gov> wrote: > >> >> Christian: >> >> My suggestion was not to remove a functipon from the existing >> proposal in order to provide a query capability. My proposal >> was to re-evaluate the decision to jump to a complex solution >> prior to addressing the primary issue for most users. >> >> I am aware of the lineage of the paragraph that I cited. I >> do not agree that its age implies it cannot be fixed. In >> fact, no FT proposal has any value if it does not address >> that paragraph. My suggestion was to take a simpler approach >> that allows a user to query whether an error will persist >> and, if so, does it imply other failures for various >> classes of operations. Such an interface would be best >> effort but would generally allow a simple "I can keep >> trying (possibly ina slightly modified form until I think >> the implementation does not know it is hosed" approach. >> More complex approaches could follow but they would be >> imprved by having such a query interface available. >> >> Bronis >> >> >> >> >> >> >> >> On Wed, 25 Sep 2013, Christian Engelmann wrote: >> >>> >>> Bronis, the problem is that the paragraph you pointed out (and quoted below for reference) has been part of MPI since version 1.1. MPI was simply not conceived with fault tolerance or fault awareness in mind. Moreover, MPI was designed with a simplistic fault model, i.e., the state after any error is undefined. Rectifying this almost 20 years later is a difficult task. >>> >>> I agree that the proposed solution is not easy to comprehend from an application or library developer's point of view. I do think that fault tolerant applications that demonstrate the capabilities and usefulness of the proposed enhancements would help a lot. >>> >>> I also think that there is a general misunderstanding on how this interface is supposed to be used. Looping around every MPI call to catch and recover from potential errors is certainly a quite nonsensical approach. Instead, the more practical approach is transaction- or exception-based programming. This requires programming templates atop the proposed MPI enhancements. >>> >>> Once again, demonstrations using fault tolerant applications would really help. Martin already pointed to some. I know of other work by UT (Jack's group), UCR (Zizhong Chen), and UoH (Edgar Gabriel), all based on UT's FT-MPI from 2003. Moving those to the proposed MPI enhancements would help greatly. >>> >>> The only failure recovery function proposed is MPI_Comm_shrink(). Removing this, simply means that only point-to-point communication will work after a process fault, but not collectives. A point-to-point only MPI is pretty useless in my opinion. >>> >>> Christian >>> >>> On Sep 25, 2013, at 12:51 PM, Bronis R. de Supinski <bronis(a)llnl.gov> wrote: >>> >>>> This document does not specify the state of a computation >>>> after an erroneous MPI call has occurred. The desired >>>> behavior is that a relevant error code be returned, and >>>> the effect of the error be localized to the greatest >>>> possible extent. >>> >>> -- >>> >>> Christian Engelmann, Ph.D. >>> >>> System Software Team Task Lead / R&D Staff Scientist >>> Computer Science Research Group >>> Computer Science and Mathematics Division >>> Oak Ridge National Laboratory >>> >>> Mail: P.O. Box 2008, Oak Ridge, TN 37831-6173, USA >>> Phone: +1 (865) 574-3132 / Fax: +1 (865) 576-5491 >>> e-Mail: engelmannc(a)ornl.gov / Home: www.christian-engelmann.info >>> >>> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > >

1 0

Re: [mpiwg-ft] Madrid Report
by Wesley Bland 25 Sep '13

25 Sep '13

Hi Bronis, Thanks for the feedback. As Christian mentioned, most of what this proposal does is failure notification (as opposed to recovery). The biggest change (in terms of implementation) is found in the text in section 17.2.2: "In all other cases, the operation raises an exception of class MPI_ERR_PROC_FAILED to indicate that the failure prevents the operation from following its failure-free specification. If there is a request identifying the point-to-point communication, it is completed. Future point-to-point communication with the same process on this communicator must also raise MPI_ERR_PROC_FAILED." This (along with some other similar passages for things like collectives, startup/finalize, dynamic processes, etc. found in 17.2) specify how MPI should behave after a failure. There is other text sprinkled around the other chapters to clean up the sections you mentioned that obviously need to be updated to mention that we will now handle failure, but the other new additions are relatively minor from an implementation perspective. Also, it's our intent to combine ticket 336 (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/336) into this proposal to allow the user to query whether the implementation supports FT (this might show up in the form of an INIT_THREADS style request/supported system, but we still have to discuss it further). So if the user doesn't want/need FT, they're welcome to ignore it. As far as querying whether an error will persist, that requires an entirely different set of semantics. That would necessitate standardizing transient errors and specifying how an implementation should handle them. We're trying to make things as simple as possible. Either a process has failed and will always be failed, or it is still correct. As Christian mentioned, this can and should probably be handled with an exception system like the MPI Errhandler callbacks. I may be misunderstanding your meaning, but I think periodically polling the system to check if some process or set of processes is still alive would be rather cumbersome. Instead, the library can alert you when the failure occurs and do the hard work for you. Everything else we provide is a follow on from that, but it's designed to be minimally intrusive. To your last point, I'm certainly not rooting for a narrow passage, and after the most recent voting rules update, I don't think such a thing is possible anymore. We're not trying to ram this down anyone's throat at all. That's why we've been socializing it so much to get as much feedback as possible. We want to take into consideration all of the feedback that we're getting and provide rationale whenever we can. The reason we have been targeting December is that the feedback that I've been aware of as far back as the last vote in Japan was not that the proposal was technically deficient, but that it was too young and unstable. We've given it another year and a half to acquire more users, implementations, feedback, etc. and time to stabilize. We believe we've done that and are ready to give it another shot. The text is largely unchanged from the previous reading other than a few clarifications and some additions to the RMA and I/O sections. More feedback is encouraged and welcomed and if there's something that we can fix, we'll do so. Thanks, Wesley On Sep 25, 2013, at 1:43 PM, "Bronis R. de Supinski" <bronis(a)llnl.gov> wrote: > > Christian: > > My suggestion was not to remove a functipon from the existing > proposal in order to provide a query capability. My proposal > was to re-evaluate the decision to jump to a complex solution > prior to addressing the primary issue for most users. > > I am aware of the lineage of the paragraph that I cited. I > do not agree that its age implies it cannot be fixed. In > fact, no FT proposal has any value if it does not address > that paragraph. My suggestion was to take a simpler approach > that allows a user to query whether an error will persist > and, if so, does it imply other failures for various > classes of operations. Such an interface would be best > effort but would generally allow a simple "I can keep > trying (possibly ina slightly modified form until I think > the implementation does not know it is hosed" approach. > More complex approaches could follow but they would be > imprved by having such a query interface available. > > Bronis > > > > > > > > On Wed, 25 Sep 2013, Christian Engelmann wrote: > >> >> Bronis, the problem is that the paragraph you pointed out (and quoted below for reference) has been part of MPI since version 1.1. MPI was simply not conceived with fault tolerance or fault awareness in mind. Moreover, MPI was designed with a simplistic fault model, i.e., the state after any error is undefined. Rectifying this almost 20 years later is a difficult task. >> >> I agree that the proposed solution is not easy to comprehend from an application or library developer's point of view. I do think that fault tolerant applications that demonstrate the capabilities and usefulness of the proposed enhancements would help a lot. >> >> I also think that there is a general misunderstanding on how this interface is supposed to be used. Looping around every MPI call to catch and recover from potential errors is certainly a quite nonsensical approach. Instead, the more practical approach is transaction- or exception-based programming. This requires programming templates atop the proposed MPI enhancements. >> >> Once again, demonstrations using fault tolerant applications would really help. Martin already pointed to some. I know of other work by UT (Jack's group), UCR (Zizhong Chen), and UoH (Edgar Gabriel), all based on UT's FT-MPI from 2003. Moving those to the proposed MPI enhancements would help greatly. >> >> The only failure recovery function proposed is MPI_Comm_shrink(). Removing this, simply means that only point-to-point communication will work after a process fault, but not collectives. A point-to-point only MPI is pretty useless in my opinion. >> >> Christian >> >> On Sep 25, 2013, at 12:51 PM, Bronis R. de Supinski <bronis(a)llnl.gov> wrote: >> >>> This document does not specify the state of a computation >>> after an erroneous MPI call has occurred. The desired >>> behavior is that a relevant error code be returned, and >>> the effect of the error be localized to the greatest >>> possible extent. >> >> -- >> >> Christian Engelmann, Ph.D. >> >> System Software Team Task Lead / R&D Staff Scientist >> Computer Science Research Group >> Computer Science and Mathematics Division >> Oak Ridge National Laboratory >> >> Mail: P.O. Box 2008, Oak Ridge, TN 37831-6173, USA >> Phone: +1 (865) 574-3132 / Fax: +1 (865) 576-5491 >> e-Mail: engelmannc(a)ornl.gov / Home: www.christian-engelmann.info >> >> > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft

1 0

Re: [mpiwg-ft] Madrid Report
by Bronis R. de Supinski 25 Sep '13

25 Sep '13

Christian: My suggestion was not to remove a functipon from the existing proposal in order to provide a query capability. My proposal was to re-evaluate the decision to jump to a complex solution prior to addressing the primary issue for most users. I am aware of the lineage of the paragraph that I cited. I do not agree that its age implies it cannot be fixed. In fact, no FT proposal has any value if it does not address that paragraph. My suggestion was to take a simpler approach that allows a user to query whether an error will persist and, if so, does it imply other failures for various classes of operations. Such an interface would be best effort but would generally allow a simple "I can keep trying (possibly ina slightly modified form until I think the implementation does not know it is hosed" approach. More complex approaches could follow but they would be imprved by having such a query interface available. Bronis On Wed, 25 Sep 2013, Christian Engelmann wrote: > > Bronis, the problem is that the paragraph you pointed out (and quoted below for reference) has been part of MPI since version 1.1. MPI was simply not conceived with fault tolerance or fault awareness in mind. Moreover, MPI was designed with a simplistic fault model, i.e., the state after any error is undefined. Rectifying this almost 20 years later is a difficult task. > > I agree that the proposed solution is not easy to comprehend from an application or library developer's point of view. I do think that fault tolerant applications that demonstrate the capabilities and usefulness of the proposed enhancements would help a lot. > > I also think that there is a general misunderstanding on how this interface is supposed to be used. Looping around every MPI call to catch and recover from potential errors is certainly a quite nonsensical approach. Instead, the more practical approach is transaction- or exception-based programming. This requires programming templates atop the proposed MPI enhancements. > > Once again, demonstrations using fault tolerant applications would really help. Martin already pointed to some. I know of other work by UT (Jack's group), UCR (Zizhong Chen), and UoH (Edgar Gabriel), all based on UT's FT-MPI from 2003. Moving those to the proposed MPI enhancements would help greatly. > > The only failure recovery function proposed is MPI_Comm_shrink(). Removing this, simply means that only point-to-point communication will work after a process fault, but not collectives. A point-to-point only MPI is pretty useless in my opinion. > > Christian > > On Sep 25, 2013, at 12:51 PM, Bronis R. de Supinski <bronis(a)llnl.gov> wrote: > >> This document does not specify the state of a computation >> after an erroneous MPI call has occurred. The desired >> behavior is that a relevant error code be returned, and >> the effect of the error be localized to the greatest >> possible extent. > > -- > > Christian Engelmann, Ph.D. > > System Software Team Task Lead / R&D Staff Scientist > Computer Science Research Group > Computer Science and Mathematics Division > Oak Ridge National Laboratory > > Mail: P.O. Box 2008, Oak Ridge, TN 37831-6173, USA > Phone: +1 (865) 574-3132 / Fax: +1 (865) 576-5491 > e-Mail: engelmannc(a)ornl.gov / Home: www.christian-engelmann.info > >

1 0

Re: [mpiwg-ft] Madrid Report
by Christian Engelmann 25 Sep '13

25 Sep '13

Bronis, the problem is that the paragraph you pointed out (and quoted below for reference) has been part of MPI since version 1.1. MPI was simply not conceived with fault tolerance or fault awareness in mind. Moreover, MPI was designed with a simplistic fault model, i.e., the state after any error is undefined. Rectifying this almost 20 years later is a difficult task. I agree that the proposed solution is not easy to comprehend from an application or library developer's point of view. I do think that fault tolerant applications that demonstrate the capabilities and usefulness of the proposed enhancements would help a lot. I also think that there is a general misunderstanding on how this interface is supposed to be used. Looping around every MPI call to catch and recover from potential errors is certainly a quite nonsensical approach. Instead, the more practical approach is transaction- or exception-based programming. This requires programming templates atop the proposed MPI enhancements. Once again, demonstrations using fault tolerant applications would really help. Martin already pointed to some. I know of other work by UT (Jack's group), UCR (Zizhong Chen), and UoH (Edgar Gabriel), all based on UT's FT-MPI from 2003. Moving those to the proposed MPI enhancements would help greatly. The only failure recovery function proposed is MPI_Comm_shrink(). Removing this, simply means that only point-to-point communication will work after a process fault, but not collectives. A point-to-point only MPI is pretty useless in my opinion. Christian On Sep 25, 2013, at 12:51 PM, Bronis R. de Supinski <bronis(a)llnl.gov> wrote: > This document does not specify the state of a computation > after an erroneous MPI call has occurred. The desired > behavior is that a relevant error code be returned, and > the effect of the error be localized to the greatest > possible extent. -- Christian Engelmann, Ph.D. System Software Team Task Lead / R&D Staff Scientist Computer Science Research Group Computer Science and Mathematics Division Oak Ridge National Laboratory Mail: P.O. Box 2008, Oak Ridge, TN 37831-6173, USA Phone: +1 (865) 574-3132 / Fax: +1 (865) 576-5491 e-Mail: engelmannc(a)ornl.gov / Home: www.christian-engelmann.info

1 0

Re: [mpiwg-ft] Madrid Report
by Richard Graham 25 Sep '13

25 Sep '13

Here is a list of the feedback I have from the presentation in Madrid: - Bill had issues with the error codes, rather than relying on error classes. Seems like his e-mail today clarified what he meant (which was not what I understood at the time) - Need to remove slide 8 (I should have done this but did not) - Slide 12 just confuses the issues, and should be removed. - Slide 13: seems to be a typo, seems like the barrier should be on newcomm - Slide 15: really should have a high level description of what this function behaves like (will clarify a lot). This is basically a gather followed by a broadcast from a "root". - slide 17: many folks did not know what an MCS lock is, so just confused the issues. - Library composition example: indicates that the status object can be used maintain status between library calls, but really does not indicate how. A few more details would really help in getting the gist of the approach. Rich -----Original Message----- From: mpiwg-ft [mailto:[email protected]] On Behalf Of Bronis R. de Supinski Sent: Wednesday, September 25, 2013 12:52 PM To: MPI WG Fault Tolerance and Dynamic Process Control working Group Subject: Re: [mpiwg-ft] Madrid Report Wesley: I disagree vehemently with your statement that "either one individually would be useless." While I was not at the meeting (and thus did not raise the objections or make this suggestion -- this time), I consider this statement to be a continued example of how those working on the proposal have failed to understand what users most desire from MPI in terms of a first step for fault tolerance. The biggest concern is simply that the MPI standard states on p28:24-26: This document does not specify the state of a computation after an erroneous MPI call has occurred. The desired behavior is that a relevant error code be returned, and the effect of the error be localized to the greatest possible extent. I believe at least one other example exists of this sort of text but I will not look for it now. The point is that the user has no guarantees about the state of MPI, the computation or anything else after an error. While that may be unavoidable, the user also has no why to query the implementation about what the most likely state is. Thus, the user must make the most pessimistic assumption and catching an error is pretty much useless other than to implement some very basic message - or if the user is feeling frisky, some more complex action - prior to shutdown soon after the error is observed. I think the working group has listened to a small set of users and missed that the vast majority of users would be happy (at least initially) if the standard provided such a capability. I have spoken to numerous users for whom this statement is true. Further, such a capability would be fairly easy to make low cost. While some details would need to be worked out, which would take time, you would find wide acceptance of that approach -- as opposed to trying to ram your complex, unintuitive interface down the throats of those who object. That brings me to my other major objection to your email, You state "I think we all expect that this proposal won't be passing unanimously." The point of deferring the FT proposal was not to have it pushed with increase urgency. The point was to have objections addressed and to make the adopted interface broadly accepted. You still seem to think that a narrowly passed proposal would be OK. Bronis On Wed, 25 Sep 2013, Wesley Bland wrote: > Let me add my two cents here as well. > > As Jim mentioned, this wasn't a rehashed slide deck. This was a new one. The confusion may have come from the fact that there haven't been any changes to the proposal recently and the talk that Rich gave was designed to be informational for those who hadn't attended the US meetings recently. > > As Jim mentioned, this was really supposed to be a justification talk to give background and rationale. It sounds like we may not have achieved that goal. It's unfortunate that more of us couldn't be there to help the conversation, but I'd like to try to find out more about the specific concerns that were raised as we've been working very hard to respond to these sorts of things in the past. > > I don't know if you actually meant that there was a desire to see the proposal split in two or not, but I'd really be against that. It would present a lot of work and without both pieces, either one individually would be useless. We've gone through the text many times to be sure that we're presenting the least intrusive interface possible that still accomplishes all of the needs of FT. As you mentioned, it isn't the most user friendly interface and we don't expect it to be. However, the real user friendly interfaces (C/R, replication, migration, strong consistency, etc.) can be built on top of the proposal, which is what we intended in our design. If that's one of the objections, then we many need to rethink things here or do a better job of convincing the forum that this is a good idea. FWIW, I believe that was always the original intent of MPI anyway. > > If it's going to be a problem to bring this to the forum for a reading in December, we can push things out, but we've been targeting that date and I'd like to have a better idea of what the objections are before we decide to delay again. If it's just the same arguments again, then it may not do any good to push this for another 3-6 months. I think we all expect that this proposal won't be passing unanimously. > > Thanks for the feedback Martin. I know that you probably weren't the one objecting and those who were probably aren't on this list. It's too bad that more of us couldn't be at the meeting, but I think we're all familiar with the funding situation in the US right now for international travel. > > Thanks, > Wesley > > On Sep 25, 2013, at 8:23 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: > >> Hi Martin, >> >> The slides that were presented in Madrid were actually a new deck. We had tried to address feedback that there is a need to do a better job of covering the core issues, potential approaches, and justifications for design decisions, beyond just presenting the proposal. It sounds like more discussion around these topics is needed, and we can work with that for the Chicago meeting. >> >> It sounds like we're being asked to break the proposal up into two sets of changes -- one that defines failures in MPI and MPI's state after a failure; and second that adds a recovery API. Is that right? I'm not sure if it's possible to tease these things apart, but we can look into it. >> >> Cheers, >> ~Jim. >> >> >> On Tue, Sep 24, 2013 at 2:48 PM, Martin Schulz <schulzm(a)llnl.gov> wrote: >>> >>> Hi all, >>> >>> This is more from the viewpoint of an external observer, since I haven't had the time to really participate lately. >>> >>> Rich gave a presentation covering the current proposal, but he also said that this was basically the old slide deck that had been shown before and didn't contain any of the new work that the FT group has been doing in coordinating with the other WGs. There were also several questions that people asked and that we couldn't answer since Rich was the only FT group member at the meeting. >>> >>> Based on this I got the feeling that many in the forum were still concerned about this overarching proposal and would like to hear more. In particular, a few meetings back after the MPI 3.0 vote we had a discussion about what was missing to make such concerns go away and one thing that we agreed on was to first go through the standard and clean things up to make it compatible with an FT proposal. From what I can tell, a lot of this has happened and it may help to explicitly present that before even going into the API approach. This (plus an updated talk with a broader discussion with more FT group members being present) may be better for the Chicago meeting than doing an actual formal reading (which may be too early since there doesn't seem to be consensus about the approach, yet). >>> >>> From the application side, I have seen a few people starting to use it. One of our PDs is using it for a large MD application and there is also the following work I saw at a recent conference: >>> >>> ftp://ftp.inf.ethz.ch/pub/publications/tech-reports/7xx/793.pdf >>> >>> Both said that the interface is overly complex and has a very (too?) high of an impact on applications, which makes me worried. Not sure how we can address this, though. >>> >>> Martin >>> >>> >>> >>> >>> On Sep 19, 2013, at 6:43 PM, Aurélien Bouteiller <bouteill(a)icl.utk.edu> wrote: >>> >>> I am currently in Europe and busy. My schedule will get back to normal in october. >>> >>> I'm also interested in a summary of events though. >>> >>> Aurelien >>> >>> >>> Le 20 sept. 2013 à 00:42, Wesley Bland <wbland(a)mcs.anl.gov> a écrit : >>> >>> Hi WG, >>> >>> I've been out if the loop for a bit since my wife had our first baby last week (everyone is doing great). I was wondering how things went at the forum meeting last week. Was there any feedback from those in attendance? >>> >>> Also, I didn't see an email about this week's con call. I assume that it didn't happen, but if it did can anyone mention what was discussed? >>> >>> Thanks, >>> Wesley >>> _______________________________________________ >>> mpiwg-ft mailing list >>> mpiwg-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>> >>> >>> -- >>> * Dr. Aurélien Bouteiller >>> * Researcher at Innovative Computing Laboratory >>> * University of Tennessee >>> * 1122 Volunteer Boulevard, suite 309b >>> * Knoxville, TN 37996 >>> * 865 974 9375 >>> >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> mpiwg-ft mailing list >>> mpiwg-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>> >>> >>> ____________________________________________________________________ >>> ____ Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm >>> CASC @ Lawrence Livermore National Laboratory, Livermore, USA >>> >>> >>> _______________________________________________ >>> mpiwg-ft mailing list >>> mpiwg-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > >

1 0

Re: [mpiwg-ft] Madrid Report
by Bronis R. de Supinski 25 Sep '13

25 Sep '13

Wesley: I disagree vehemently with your statement that "either one individually would be useless." While I was not at the meeting (and thus did not raise the objections or make this suggestion -- this time), I consider this statement to be a continued example of how those working on the proposal have failed to understand what users most desire from MPI in terms of a first step for fault tolerance. The biggest concern is simply that the MPI standard states on p28:24-26: This document does not specify the state of a computation after an erroneous MPI call has occurred. The desired behavior is that a relevant error code be returned, and the effect of the error be localized to the greatest possible extent. I believe at least one other example exists of this sort of text but I will not look for it now. The point is that the user has no guarantees about the state of MPI, the computation or anything else after an error. While that may be unavoidable, the user also has no why to query the implementation about what the most likely state is. Thus, the user must make the most pessimistic assumption and catching an error is pretty much useless other than to implement some very basic message - or if the user is feeling frisky, some more complex action - prior to shutdown soon after the error is observed. I think the working group has listened to a small set of users and missed that the vast majority of users would be happy (at least initially) if the standard provided such a capability. I have spoken to numerous users for whom this statement is true. Further, such a capability would be fairly easy to make low cost. While some details would need to be worked out, which would take time, you would find wide acceptance of that approach -- as opposed to trying to ram your complex, unintuitive interface down the throats of those who object. That brings me to my other major objection to your email, You state "I think we all expect that this proposal won't be passing unanimously." The point of deferring the FT proposal was not to have it pushed with increase urgency. The point was to have objections addressed and to make the adopted interface broadly accepted. You still seem to think that a narrowly passed proposal would be OK. Bronis On Wed, 25 Sep 2013, Wesley Bland wrote: > Let me add my two cents here as well. > > As Jim mentioned, this wasn't a rehashed slide deck. This was a new one. The confusion may have come from the fact that there haven't been any changes to the proposal recently and the talk that Rich gave was designed to be informational for those who hadn't attended the US meetings recently. > > As Jim mentioned, this was really supposed to be a justification talk to give background and rationale. It sounds like we may not have achieved that goal. It's unfortunate that more of us couldn't be there to help the conversation, but I'd like to try to find out more about the specific concerns that were raised as we've been working very hard to respond to these sorts of things in the past. > > I don't know if you actually meant that there was a desire to see the proposal split in two or not, but I'd really be against that. It would present a lot of work and without both pieces, either one individually would be useless. We've gone through the text many times to be sure that we're presenting the least intrusive interface possible that still accomplishes all of the needs of FT. As you mentioned, it isn't the most user friendly interface and we don't expect it to be. However, the real user friendly interfaces (C/R, replication, migration, strong consistency, etc.) can be built on top of the proposal, which is what we intended in our design. If that's one of the objections, then we many need to rethink things here or do a better job of convincing the forum that this is a good idea. FWIW, I believe that was always the original intent of MPI anyway. > > If it's going to be a problem to bring this to the forum for a reading in December, we can push things out, but we've been targeting that date and I'd like to have a better idea of what the objections are before we decide to delay again. If it's just the same arguments again, then it may not do any good to push this for another 3-6 months. I think we all expect that this proposal won't be passing unanimously. > > Thanks for the feedback Martin. I know that you probably weren't the one objecting and those who were probably aren't on this list. It's too bad that more of us couldn't be at the meeting, but I think we're all familiar with the funding situation in the US right now for international travel. > > Thanks, > Wesley > > On Sep 25, 2013, at 8:23 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: > >> Hi Martin, >> >> The slides that were presented in Madrid were actually a new deck. We had tried to address feedback that there is a need to do a better job of covering the core issues, potential approaches, and justifications for design decisions, beyond just presenting the proposal. It sounds like more discussion around these topics is needed, and we can work with that for the Chicago meeting. >> >> It sounds like we're being asked to break the proposal up into two sets of changes -- one that defines failures in MPI and MPI's state after a failure; and second that adds a recovery API. Is that right? I'm not sure if it's possible to tease these things apart, but we can look into it. >> >> Cheers, >> ~Jim. >> >> >> On Tue, Sep 24, 2013 at 2:48 PM, Martin Schulz <schulzm(a)llnl.gov> wrote: >>> >>> Hi all, >>> >>> This is more from the viewpoint of an external observer, since I haven't had the time to really participate lately. >>> >>> Rich gave a presentation covering the current proposal, but he also said that this was basically the old slide deck that had been shown before and didn't contain any of the new work that the FT group has been doing in coordinating with the other WGs. There were also several questions that people asked and that we couldn't answer since Rich was the only FT group member at the meeting. >>> >>> Based on this I got the feeling that many in the forum were still concerned about this overarching proposal and would like to hear more. In particular, a few meetings back after the MPI 3.0 vote we had a discussion about what was missing to make such concerns go away and one thing that we agreed on was to first go through the standard and clean things up to make it compatible with an FT proposal. From what I can tell, a lot of this has happened and it may help to explicitly present that before even going into the API approach. This (plus an updated talk with a broader discussion with more FT group members being present) may be better for the Chicago meeting than doing an actual formal reading (which may be too early since there doesn't seem to be consensus about the approach, yet). >>> >>> From the application side, I have seen a few people starting to use it. One of our PDs is using it for a large MD application and there is also the following work I saw at a recent conference: >>> >>> ftp://ftp.inf.ethz.ch/pub/publications/tech-reports/7xx/793.pdf >>> >>> Both said that the interface is overly complex and has a very (too?) high of an impact on applications, which makes me worried. Not sure how we can address this, though. >>> >>> Martin >>> >>> >>> >>> >>> On Sep 19, 2013, at 6:43 PM, Aurï¿œlien Bouteiller <bouteill(a)icl.utk.edu> wrote: >>> >>> I am currently in Europe and busy. My schedule will get back to normal in october. >>> >>> I'm also interested in a summary of events though. >>> >>> Aurelien >>> >>> >>> Le 20 sept. 2013 ï¿œ 00:42, Wesley Bland <wbland(a)mcs.anl.gov> a ï¿œcrit : >>> >>> Hi WG, >>> >>> I've been out if the loop for a bit since my wife had our first baby last week (everyone is doing great). I was wondering how things went at the forum meeting last week. Was there any feedback from those in attendance? >>> >>> Also, I didn't see an email about this week's con call. I assume that it didn't happen, but if it did can anyone mention what was discussed? >>> >>> Thanks, >>> Wesley >>> _______________________________________________ >>> mpiwg-ft mailing list >>> mpiwg-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>> >>> >>> -- >>> * Dr. Aurï¿œlien Bouteiller >>> * Researcher at Innovative Computing Laboratory >>> * University of Tennessee >>> * 1122 Volunteer Boulevard, suite 309b >>> * Knoxville, TN 37996 >>> * 865 974 9375 >>> >>> >>> >>> >>> >>> >>> >>> _______________________________________________ >>> mpiwg-ft mailing list >>> mpiwg-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>> >>> >>> ________________________________________________________________________ >>> Martin Schulz, schulzm(a)llnl.gov, http://people.llnl.gov/schulzm >>> CASC @ Lawrence Livermore National Laboratory, Livermore, USA >>> >>> >>> _______________________________________________ >>> mpiwg-ft mailing list >>> mpiwg-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > >

1 0