February 2011 - mpiwg-ft - mailman3.cels.anl.gov

[Mpi3-ft] Exit Code from 'mpirun' upon failure recovery
by Joshua Hursey 04 Feb '11

04 Feb '11

So this is not really appropriate for the MPI standard language, but more of a user experience question. In fact this is a much larger question that implementations have to struggle with already. If a process fails in the application (either by external causes or by calling MPI_Abort), what should 'mpirun' return as its exit status? If the application intends to handle the failure and continue running after recovering then they may expect that as long as MPI_Finalize is called in all remaining processes that 'mpirun' return '0' or success. But if no process calls MPI_Finalize (because they either called MPI_Abort or terminated abnormally) that it return a non-zero value - probably one of the values that they set in MPI_Abort, if possible. Of course there is the case where the failure occurs during MPI_Finalize, to which the MPI implementation may or may not be able to act consistently depending on the timing of the failure notification. I was mucking around in this code in the Open MPI prototype, and thought I would get the opinions of the group as I move forward. Thanks, Josh ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] FT Working Group Teleconf (Special Date/Time: Monday, Feb. 7)
by Joshua Hursey 03 Feb '11

03 Feb '11

There is an MPI Forum meeting next week, and since some of us cannot attend (including me) I have asked Jeff Squyres to setup a WebEx teleconf session for the Fault Tolerance Working Group session. According to the agenda, we are scheduled to meet from 1-3 pm Pacific Time on Monday, Feb. 7, 2011: http://meetings.mpi-forum.org/secretary/2011/02/agenda.php I would recommend joining the meeting online, if you can, so we can share slides/desktops/browsers, if necessary. Date: Feb., 7, 2011 Time: 1-3 pm PST/San Jose, CA Dial-in information: <See at Bottom> Agenda: - Piggybacking (Martin S.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Piggybacking - Run-through stabilization proposal (Josh H.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… - other items? Next Meetings: * Feb 23, 2010 (Teleconf) * March 9, 2010 (Teleconf) * March 23, 2010 (Teleconf) * March 28, 2010 (MPI Forum Meeting) * April 13, 2010 (Teleconf) * April 27, 2010 (Teleconf) * May 9, 2010 (MPI Forum Meeting) Respond to the list if you have any additional topics that you would like to discuss. -- Josh ----------------------- ---- WebEx Invite ----- ----------------------- Meeting Number: 204318040 Password: mpi3 ------------------------ To join the meeting online ------------------------ 1. Go to https://cisco.webex.com/cisco/j.php?ED=158898517&UID=0&PW=NN2JhMmQ3MTEw&RT=… 2. Enter your name and email address. 3. Enter the meeting password: mpi3 4. Click "Join". 5. If the meeting includes a teleconference, follow the instructions that appear on your screen. ------------------------ To join the teleconference only ------------------------ 1. Dial into Cisco WebEx - view all Global Access Numbers at http://cisco.com/en/US/about/doing_business/conferencing/index.html 2. Follow the prompts to enter the Meeting Number (listed above) or Access Code followed by the # sign. San Jose, CA: +1.408.525.6800 RTP: +1.919.392.3330 US/Canada: +1.866.432.9903 United Kingdom: +44.20.8824.0117 India: +91.80.4350.1111 Germany: +49.619.6773.9002 Japan: +81.3.5763.9394 China: +86.10.8515.5666 Access code: 204318040 Sign up for a free trial of WebEx http://www.webex.com/go/mcemfreetrial IMPORTANT NOTICE: This WebEx service includes a feature that allows audio and any documents and other materials exchanged or viewed during the session to be recorded. By joining this session, you automatically consent to such recordings. If you do not consent to the recording, do not join the session. ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Run-Through Stabilization Users Guide
by Joshua Hursey 01 Feb '11

01 Feb '11

On Feb 1, 2011, at 11:03 AM, Bronevetsky, Greg wrote: >> I do see the point though. I wonder if an additional, combination function >> would be useful for this case. Something like: >> MPI_COMM_VALIDATE_ALL_FULL_REPORT(comm, incount, outcount, totalcount, >> rank_infos) >> comm: communicator >> incount: size of rank_infos array >> outcount: number of rank_info entries filled >> totalcount: total number of failures known >> rank_infos: array of MPI_Rank_info types >> >> This would allow the user to determine if they are getting a report of all >> the failures (outcount == totalcount), or just a subset because they did >> not supply a sufficiently allocated buffer (outcount < totalcount). This >> does force the user to allocate the rank_infos buffer before it may be >> needed, but if the type of consistency that you cite is needed then maybe >> this is not a problem. >> > This would do the job but would have the memory allocation problem that you describe. Another was to do this would be to provide something like an MPI_Status object to MPI_COMM_VALIDATE_ALL and provide functions to allow the user to further interrogate it. This object would not keep a list of all the failed ranks but instead would correspond to a point in time where some failures occur before this point and others after this point. This would allow the object to stay small. But then how does the user communicate that they don't need this information any longer? Since the MPI implementation will have to save the state associated with every call MPI_Comm_validate_all in case the user wants the information from one or more invocations. We may get away with making the rule that it is only associated with the last call, but apps might want the ability to essentially do a diff of the lists from one or more previous invocations. An alternative solution with the above example would be to use MPI_Comm_validate_all to get the count, then use MPI_Comm_validate_all_full_report to get a consistent list. This would remove the inconsistent in-between state, but at the cost of two collectives over the same communicator, which I like less than the memory allocation solution. > > One larger question about memory: Are we worried that MPI's internal storage to keep track of failed processes will grow out of proportion? In the limit, this is certainly a problem (e.g. half the ranks are dead) but in practice I'm not sure if we care. If MPI is running out of space to store the ranks of dead processes, are we comfortable with the idea of MPI actively killing ranks so that it can compress its internal representation? I'm not so worried about the memory requirements at the moment (and in practice it is not a big issue right now). I think the more pressing problem is how an MPI implementation is going to track a very large number of processes in general. Once we implementors figure out a way to track membership in a very large communicator (where membership cannot be fully known locally), adding a small amount of extra state to the distributed store to determine if the process is OK/FAILED/NULL should be considered in such a solution. But I don't think the proposed interface requires the MPI implementation to do anything too exhaustive unless the application asks for a full list of a large number of failures. I think we should steer clear of even hinting to the application that we are going to kill processes so that we can save memory. Now the process could fail because MPI tried to allocate memory and ran out - at which point the MPI should return an error, and shutdown services - Maybe a good use case for CANNOT_CONTINUE. That situation is slightly different that proactively killing processes because we don't want to/cannot track their state. That sounds like a sure fire way to make users flee from the MPI to something else. As far as the application is concerned, it only needs to allocate an array of MPI_Rank_info objects the size of that retuned by MPI_Comm_validate_all. Which should be small, but is inclusive of all the known failures (recognized and unrecognized). We could add a flag to the local MPI_Comm_validate that would return a list of the unrecognized failures, instead of all failures. Or, a bit more specifically, a 'state' key argument to the function, and it will return only those processes that match that state. This would allow the application to manage smaller sets of failures without having to iterate over previously recognized failures. What do you think about that? > >>> Would it be better to explicitly have separate states for failures that >> have been recognized locally and collectively? >> >> I don't think so. I think this starts to become confusing to the user, and >> muddles the semantics a bit. If global/collective recognition does not >> imply local recognition, then do we require that the user both locally and >> globally recognize a failure before they can create a new communicator? >> What if the communicator has a global recognition of failures, but not >> locally? In that case collectives will succeed, but only some point-to- >> point operations. This seems to be adding more work for the application >> without a clear use case on when it would be required. >> > You can separate the two concepts while still ensuring that collective recognition implies local recognition. It would just be a hierarchy of recognition levels (none < local < collective on individual communicators < collective on all communicators), where a given rank's position is always clearly defined. Ah I see what you are saying now. So add one more state in order to the taxonomy: MPI_RANK_STATE_OK Normal running state MPI_RANK_STATE_FAILED Failed, has not been recognized/cleared MPI_RANK_STATE_LOCAL_NULL Failed, has been recognized/cleared locally MPI_RANK_STATE_NULL Failed, has been recognized/cleared globally OK -> FAILED : When process failure is detected FAILED -> LOCAL_NULL: When locally validated LOCAL_NULL -> NULL : When collectively validated FAILED -> NULL : When collectively validated NULL -> LOCAL_NULL: A new failure occurs in the communicator (?) When a new failure occurs in a communicator do all the ranks that were previously in MPI_RANK_STATE_NULL (globally recognized) move to MPI_RANK_STATE_LOCAL_NULL (locally recognized only)? So the transition diagram could have a loop between the LOCAL_NULL and NULL states. Something like: OK -> FAILED -> LOCAL_NULL <-> NULL or OK -> FAILED -> NULL <-> LOCAL_NULL It makes the state transitions a little more complex, but does allow the user to determine if the process has been locally or globally cleared. The question really is, is it useful to know that an individual rank is globally cleared? Globally cleared really only has meaning at the full communicator level (used to determine if collectives are enabled/disabled). So wouldn't it be better to have a way to query the communicator to see if it was globally cleared or not, in order for the application to decide if collectives can be used or not. We talked a while back about adding an attribute to the communicator to indicate this, but I don't think it ever made it into the proposal. I'll make a note about this, and maybe we can discuss what it should look like next week. > >>> In the broadcast example it would be easier if both examples used the >> same bcast algorithm. You don't really explain how the algorithm works, so >> it'll be easier to understand if you don't switch them. >> >> I can see that. I think explaining the way that different implementations >> may cause things to go wonky might be useful. I cleaned up the language a >> bit to describe the two algorithms and why we discuss them here. I would be >> fine with dropping one of the bcast illustrations if you all think it is >> still too confusing. >> > I don't think it's a big deal. Our developers are fairly familiar with communication protocols so I doubt that it will really confuse them. Cool. Thanks, Josh > > Greg Bronevetsky > Lawrence Livermore National Lab > (925) 424-5756 > bronevetsky(a)llnl.gov > http://greg.bronevetsky.com > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Run-Through Stabilization Users Guide
by Bronevetsky, Greg 01 Feb '11

01 Feb '11

> I do see the point though. I wonder if an additional, combination function > would be useful for this case. Something like: > MPI_COMM_VALIDATE_ALL_FULL_REPORT(comm, incount, outcount, totalcount, > rank_infos) > comm: communicator > incount: size of rank_infos array > outcount: number of rank_info entries filled > totalcount: total number of failures known > rank_infos: array of MPI_Rank_info types > > This would allow the user to determine if they are getting a report of all > the failures (outcount == totalcount), or just a subset because they did > not supply a sufficiently allocated buffer (outcount < totalcount). This > does force the user to allocate the rank_infos buffer before it may be > needed, but if the type of consistency that you cite is needed then maybe > this is not a problem. > This would do the job but would have the memory allocation problem that you describe. Another was to do this would be to provide something like an MPI_Status object to MPI_COMM_VALIDATE_ALL and provide functions to allow the user to further interrogate it. This object would not keep a list of all the failed ranks but instead would correspond to a point in time where some failures occur before this point and others after this point. This would allow the object to stay small. One larger question about memory: Are we worried that MPI's internal storage to keep track of failed processes will grow out of proportion? In the limit, this is certainly a problem (e.g. half the ranks are dead) but in practice I'm not sure if we care. If MPI is running out of space to store the ranks of dead processes, are we comfortable with the idea of MPI actively killing ranks so that it can compress its internal representation? > > Would it be better to explicitly have separate states for failures that > have been recognized locally and collectively? > > I don't think so. I think this starts to become confusing to the user, and > muddles the semantics a bit. If global/collective recognition does not > imply local recognition, then do we require that the user both locally and > globally recognize a failure before they can create a new communicator? > What if the communicator has a global recognition of failures, but not > locally? In that case collectives will succeed, but only some point-to- > point operations. This seems to be adding more work for the application > without a clear use case on when it would be required. > You can separate the two concepts while still ensuring that collective recognition implies local recognition. It would just be a hierarchy of recognition levels (none < local < collective on individual communicators < collective on all communicators), where a given rank's position is always clearly defined. > > In the broadcast example it would be easier if both examples used the > same bcast algorithm. You don't really explain how the algorithm works, so > it'll be easier to understand if you don't switch them. > > I can see that. I think explaining the way that different implementations > may cause things to go wonky might be useful. I cleaned up the language a > bit to describe the two algorithms and why we discuss them here. I would be > fine with dropping one of the bcast illustrations if you all think it is > still too confusing. > I don't think it's a big deal. Our developers are fairly familiar with communication protocols so I doubt that it will really confuse them. Greg Bronevetsky Lawrence Livermore National Lab (925) 424-5756 bronevetsky(a)llnl.gov http://greg.bronevetsky.com

1 0

Re: [Mpi3-ft] Run-Through Stabilization Users Guide
by Joshua Hursey 01 Feb '11

01 Feb '11

On Jan 31, 2011, at 11:05 AM, Bronevetsky, Greg wrote: > Josh, the document looks very nice. I'll pass it around LLNL. > > Some comments: > > There should be some discussion in the intro about how communication relates to failures (i.e. you can't talk to failed processes but you can talk to non-failed processes). Good idea. I'll add some language for that in the first section. > > MPI_COMM_VALIDATE is supposed to be used to examine the list of failed ranks in detail after a call to MPI_COMM_VALIDATE_ALL. If there is a failure between the start of MPI_COMM_VALIDATE_ALL and the time MPI_COMM_VALIDATE is called, this list of failed ranks may be larger than the number returned by MPI_COMM_VALIDATE_ALL. There should be a way to remove this inconsistency by allowing the application to carry some kind of connection from MPI_COMM_VALIDATE_ALL to MPI_COMM_VALIDATE to make sure that the application knows which failures have been globally recognized and which only locally. That is a common way to use the MPI_COMM_VALIDATE function. By tethering the two functions, does this mislead the application by reporting less than the full set of known failures? The user could supply an 'incount' of something larger than what was reported by MPI_COMM_VALIDATE_ALL, then inspect the 'outcount' of MPI_COMM_VALIDATE to determine if they agree. If they do not then further action may need to be taken. The only reason you would need to call MPI_COMM_VALIDATE is if you cared about which individual ranks failed. If you only care about creating recovery blocks, then you only need the count from MPI_COMM_VALIDATE_ALL, and don't have to worry about sufficient buffer space. If there is only a subset of the ranks that are critical, then you can use MPI_COMM_VALIDATE_RANK to inspect just those ranks. I do see the point though. I wonder if an additional, combination function would be useful for this case. Something like: MPI_COMM_VALIDATE_ALL_FULL_REPORT(comm, incount, outcount, totalcount, rank_infos) comm: communicator incount: size of rank_infos array outcount: number of rank_info entries filled totalcount: total number of failures known rank_infos: array of MPI_Rank_info types This would allow the user to determine if they are getting a report of all the failures (outcount == totalcount), or just a subset because they did not supply a sufficiently allocated buffer (outcount < totalcount). This does force the user to allocate the rank_infos buffer before it may be needed, but if the type of consistency that you cite is needed then maybe this is not a problem. > > Does collective failure recognition imply local recognition? Yes. I added a line about this under the validation section for clarity. > > Would it be better to explicitly have separate states for failures that have been recognized locally and collectively? I don't think so. I think this starts to become confusing to the user, and muddles the semantics a bit. If global/collective recognition does not imply local recognition, then do we require that the user both locally and globally recognize a failure before they can create a new communicator? What if the communicator has a global recognition of failures, but not locally? In that case collectives will succeed, but only some point-to-point operations. This seems to be adding more work for the application without a clear use case on when it would be required. > If nothing else, you should change the language to clearly specify whether local or collective recognition is required. For example, some might get confused and think that getting an error from a collective constitutes recognition or that calling MPI_Comm_validate can make it possible to use collectives after a failure. These constraints are specified earlier in the text but its good to repeat the concept throughout. Good point. I'll clarify the text in the collective section. > > In the broadcast example it would be easier if both examples used the same bcast algorithm. You don't really explain how the algorithm works, so it'll be easier to understand if you don't switch them. I can see that. I think explaining the way that different implementations may cause things to go wonky might be useful. I cleaned up the language a bit to describe the two algorithms and why we discuss them here. I would be fine with dropping one of the bcast illustrations if you all think it is still too confusing. > > For the Gather example it would be useful to have a picture. I'll see what I can do. I updated the wiki page with the notes above, minus the Gather image which will come later. Thanks for the feedback. -- Josh > > Greg Bronevetsky > Lawrence Livermore National Lab > (925) 424-5756 > bronevetsky(a)llnl.gov > http://greg.bronevetsky.com > > >> -----Original Message----- >> From: mpi3-ft-bounces(a)lists.mpi-forum.org [mailto:mpi3-ft- >> bounces(a)lists.mpi-forum.org] On Behalf Of Joshua Hursey >> Sent: Friday, January 28, 2011 11:44 AM >> To: MPI 3.0 Fault Tolerance and Dynamic Process Control working Group >> Subject: [Mpi3-ft] Run-Through Stabilization Users Guide >> >> As mentioned on the teleconf, I created a Run-Through Stabilization Users >> Guide wiki page. >> https://svn.mpi-forum.org/trac/mpi-forum- >> web/wiki/ft/run_through_users_guide >> >> The purpose of this page is to introduce application developers to the high >> level philosophy of the run-through stabilization proposal, and some of the >> most commonly used interfaces and semantics. I included some example code >> snippets to help new users start programming against the emerging >> prototypes. >> >> The page is not meant to be comprehensive, but just enough to get an >> application developer or new group member started with the ideas contained >> in the proposal. This is an alternative to the existing proposal which is >> structured to help the group form it into a formal proposal for the MPI >> forum, which can be confusing for those not steeped in the the MPI standard >> structure and language. >> >> Let me know what you think, and if you have any suggestions on how to make >> the page more useful. >> >> -- Josh >> >> ------------------------------------ >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0