February 2008 - mpiwg-ft - mailman3.cels.anl.gov

[Mpi3-ft] Con call
by Richard Graham 27 Feb '08

27 Feb '08

Lets not have the call this week, and focus on getting done with writing up a first cut at the docs we signed up to start writing for our face-to-face meeting in 2 weeks. To date I think Greg is the only one that has done this, and there are a few more first drafts that need to be written. Any objections ? Rich

1 0

[Mpi3-ft] "Fault handling = error handling" proposal added
by Supalov, Alexander 27 Feb '08

27 Feb '08

Hi everybody, I've added to the Wiki a proposal that suggests reuse of the existing MPI error handling mechanism for fault notification (see http://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Fault%20Handling) Best regards. Alexander -- Dr Alexander Supalov Intel GmbH Hermuelheimer Strasse 8a 50321 Bruehl, Germany Phone: +49 2232 209034 Mobile: +49 173 511 8735 Fax: +49 2232 209029 --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [Mpi3-ft] Transactional Messages
by Josh Hursey 25 Feb '08

25 Feb '08

On Feb 21, 2008, at 5:15 PM, Greg Bronevetsky wrote: > BTW, I don't know how to use > this Wiki, so I've just uploaded my .txt file as a page attachment. > If there's a cleaner way to do this, someone please let me know. I took the liberty of converting the 2 txt files to wiki pages. This will allow us to better use the features of the wiki, like change logs. They are linked off of the main FT wiki page: http://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ FaultToleranceWikiPage If you click on the "Edit this page" button at the bottom of any Wiki page you can edit that page. Wiki formatting is quite simple once you get the basic feel of it. There is a whole page describing the particulars here: http://svn.mpi-forum.org/trac/mpi-forum-web/wiki/WikiFormatting It is many times easier to learn by example, so if you just follow the formatting examples in these pages it will show you basically all you need to know. I suggest that we remove the two .txt files from the wiki head so we do not become confused as to which document we are talking about. -- Josh

1 0

Re: [Mpi3-ft] Transactional Messages
by Greg Bronevetsky 25 Feb '08

25 Feb '08

>So are you really talking about adding a whole new set of pt2pt and >collectives APIs specifically to handle transactional communications? Or >are you looking to extend the current APIs to have a mode that a user >could set the error detection? I have to admit I still find the >difference between the two modes that Rich describes not that far apart >from each other. I don't think that we're that far ahead on this but my personal opinion is that we should not add new sets of MPI calls unless it is truly necessary. I think that in this case we can get all the functionality we need by adding - a new function that the user would call at initialization time to set the error detection/notification policy and - communicating error events via new return values and error handler calls. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov

1 0

Re: [Mpi3-ft] Transactional Messages
by Terry Dontje 25 Feb '08

25 Feb '08

Greg Bronevetsky wrote: > At 09:38 AM 2/23/2008, Richard Graham wrote: >> So I think we are some what talking past each other. I think that >> what you really care about >> with respect to communications errors is information on messages that >> have not completed, >> and, more important, canï¿½t complete ? Is this correct ? >> I have focused more on errors that occur, but the low-level can >> handle them and does not need >> to pass information about them back up to the user. I believe this is >> where we said that we >> may want to be able to give the app some indication on performance >> degradation, at their >> request. Is this correct ? > > Exactly. My point was that the former belongs in the transactional > memory API, while the latter belongs in the QoS API. Kannan and I are > tasked with drafting something for the latter. One thing to note > though is that identical low-level events may fall under either API. > In particular, a single-bitflip may result in a message drop in one > MPI implementation and a seamless recovery with minor performance > degradation in another. The MPI implementation gets to choose the > mapping between low-level events and their high-level manifestations. > Right now we are just trying to define a reasonable interface for the > high-level manifestations. So are you really talking about adding a whole new set of pt2pt and collectives APIs specifically to handle transactional communications? Or are you looking to extend the current APIs to have a mode that a user could set the error detection? I have to admit I still find the difference between the two modes that Rich describes not that far apart from each other. --td

1 0

Re: [Mpi3-ft] Transactional Messages
by Greg Bronevetsky 23 Feb '08

23 Feb '08

At 09:38 AM 2/23/2008, Richard Graham wrote: >So I think we are some what talking past each other. I think that >what you really care about > with respect to communications errors is information on messages > that have not completed, > and, more important, can't complete ? Is this correct ? >I have focused more on errors that occur, but the low-level can >handle them and does not need > to pass information about them back up to the user. I believe > this is where we said that we > may want to be able to give the app some indication on performance > degradation, at their > request. Is this correct ? Exactly. My point was that the former belongs in the transactional memory API, while the latter belongs in the QoS API. Kannan and I are tasked with drafting something for the latter. One thing to note though is that identical low-level events may fall under either API. In particular, a single-bitflip may result in a message drop in one MPI implementation and a seamless recovery with minor performance degradation in another. The MPI implementation gets to choose the mapping between low-level events and their high-level manifestations. Right now we are just trying to define a reasonable interface for the high-level manifestations. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov

1 0

Re: [Mpi3-ft] Transactional Messages
by Richard Graham 23 Feb '08

23 Feb '08

So I think we are some what talking past each other. I think that what you really care about with respect to communications errors is information on messages that have not completed, and, more important, can¹t complete ? Is this correct ? I have focused more on errors that occur, but the low-level can handle them and does not need to pass information about them back up to the user. I believe this is where we said that we may want to be able to give the app some indication on performance degradation, at their request. Is this correct ? Rich On 2/23/08 8:47 AM, "Greg Bronevetsky" <bronevetsky1(a)llnl.gov> wrote: >>>> >> There is not new API, would just need to pass some sort of parameter to >>>> MPI_Init(). This >>>> >> is not the only place this is needed. If you want to take a look at >>>> the implementation, take >>>> >> a loot at http:// <http://public.lanl.gov/lampi> public.lanl.gov/lampi >>>> <http://public.lanl.gov/lampi> >> > > Right, LAMPI! Yeah, its not a transactional API but rather a more reliable > implementation of MPI. This is also good but its probably not something > related to transactional messages since the transactional messages API is > meant to define how message errors are exported to the application. LAMPI and > similar stuff would be more related to a QoS API that allows the application > to trade off performance for reliability. > > Greg Bronevetsky > Post-Doctoral Researcher > 1028 Building 451 > Lawrence Livermore National Lab > (925) 424-5756 > bronevetsky1(a)llnl.gov > > _______________________________________________ > Mpi3-ft mailing list > Mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Transactional Messages
by Greg Bronevetsky 23 Feb '08

23 Feb '08

> >> There is not new API, would just need to pass some sort of > parameter to MPI_Init(). This > >> is not the only place this is needed. If you want to take a > look at the implementation, take > >> a loot at http://public.lanl.gov/lampi > Right, LAMPI! Yeah, its not a transactional API but rather a more reliable implementation of MPI. This is also good but its probably not something related to transactional messages since the transactional messages API is meant to define how message errors are exported to the application. LAMPI and similar stuff would be more related to a QoS API that allows the application to trade off performance for reliability. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov

1 0

Re: [Mpi3-ft] Transactional Messages
by Richard Graham 23 Feb '08

23 Feb '08

On 2/23/08 1:18 AM, "Greg Bronevetsky" <bronevetsky1(a)llnl.gov> wrote: >> Just to follow up, I think that the ³right² thing to do with respect to some >> sort of >> transactional model is to have some sort of standard way to request such >> communications take place probably at init time. We have had such an MPI >> implementation running in production for several years on a multi-thousand >> process cluster, and the only thing that needs to be exposed to the users is >> the >> ability to turn on/off the functionality all the rest is taken care of >> just fine within >> the context of the MPI 2.0 standard, and is 100% standard compliant. > Cool! Can you post the API for this thing? Might be a useful thing to start > from. Also, its a prototype implementation. > >>> >> There is not new API, would just need to pass some sort of parameter to >>> MPI_Init(). This >>> >> is not the only place this is needed. If you want to take a look at the >>> implementation, take >>> >> a loot at http://public.lanl.gov/lampi > > Rich > >> This does not deal with hints on the network ³state². > Yeah, I'm not sure if such hints are valuable enough to be included in the > spec. It really falls under the quality-of-service ideas what we've tossed > around, so we should write up a proposal for those and see how much QoS people > really want. > > Greg Bronevetsky > Post-Doctoral Researcher > 1028 Building 451 > Lawrence Livermore National Lab > (925) 424-5756 > bronevetsky1(a)llnl.gov > > _______________________________________________ > Mpi3-ft mailing list > Mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Transactional Messages
by Greg Bronevetsky 23 Feb '08

23 Feb '08

If dropped messages are exported by MPI to the application, any recovery mechanism by the application would have no need to place the lost message back into the original sequence order. The application will know that the message is dropped and none of the APIs in the transactional messages proposal allow the application to put a message into some specific spot in the message order. It is informed of the drop and it may do whatever it wants. Importantly, if MPI notices that a message is corrupted and tells the application that it is dropped, it will also be able to perform correct matching with that gap in the sequence number. In fact, that requirement must be included in the spec in order to make sure that we don't have any ambiguities. If a message is dropped, it should be counted as delivered when the application is informed of the drop. As for high-level recovery techniques, I can see a number of uses for message drop information. It may be that the message is some sort of periodic notification from the master to the slaves and dropped messages are irrelevant. It may also be that the application is in fact using some sort of protocol to overcome such failures. The point is that such notifications can be useful and the main question for us is the appropriate balance between clean semantics and the performance hit. The fully transactional option is going to be expensive but very convenient. However, we have a number of other options that will be quite useful for use with higher-level recovery protocols and quite cheap. I think that we should gravitate towards those but I thought it important to include a variety of options for people to toss around. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov >Actually, it is most likely that MPI >implementations that dont try to deal with > dropped messages, cant even detect that such event have occurred. For > such implementation I would expect them to be able to detect a problem with > failed communications only if the low-level library they use to implement > the communications, such as some OS bypass library, returns an error when > trying to post some sort of communications, or if the run-time used by MPI > detects a fail process, and propagates this information to the rest of the > processes in the application. > >The ONLY layer that can handle any sort of >recovery from a live communications failure - > i.e. w/o some sort of check-point restart with > or with out message logging is the > MPI implementation itself. The application > reposting a send cant take get around the > lost data, because of the MPI message ordering > requirements, unless the implementation > totally relies on another library to satisfy > the MPI ordering requirements (i.e. it does not > generate some sort of message sequence number) > and the message lost is the last one > that was sent. MPI is not allowed to attempt > any matching if there is a gap in the > sequence number.

1 0