December 2013 - mpiwg-ft - mailman3.cels.anl.gov

Re: [mpiwg-ft] FTWG Action Items
by Aurélien Bouteiller 19 Dec '13

19 Dec '13

Wesley, This is all good ideas, I encourage you to create branches for new features like query, new error codes, etc (anything that’s not wording fix for current functions) so that it is easier to track what changes pertain to what and to merge (or not) when we take the final decision. I have added some more comments inline. Aurelien Le 19 déc. 2013 à 11:06, Wesley Bland <wbland(a)mcs.anl.gov> a écrit : > Hi all, > > Since we’re not going to have a call next week due to the break, I thought it would be good to try to get the action items from the previous meeting and surrounding discussions going. The first two are things that we should all discuss, the second two are more minor things that I’ve been taking care of, the last one is important, but will require more work on our part. Please take a look and provide feedback. > > Thanks, > Wesley > > --- > > * Add another error code to replace the use of MPI_ERR_PENDING (MPI_ERR_FAILURE_PENDING?) to prevent the requirement of scanning all return codes to find out if there has been a failure that left a request pending. > * This is probably a good idea as it’s become clear that overloading MPI_ERR_PENDING is more confusing and adds overhead to what the user has to do to determine if their request is still valid or not. Coming up with the name is the only challenge here. > * For now, I’ll change the text to say MPI_ERR_FAILURE_PENDING (23 characters < 30 character limit), but if we come up with something better, we can change it again. Lets think a bit more, I’m for it on principle, but only after all side effects have been considered carefully (in particular with respect to multiple any-source pending in the same wait). I feel this will not be an issue, but better safe than sorry. > > * Function to query the status of a communicator (is it revoked or not). > * If this were done via a communicator attribute, it would be the only one I’m aware of that provides information to the user that is set by the library. All others are stored by the user and retrieved by the user. > * Do we want to add a new API to take care of this, or can we confirm that it’s ok to do this via an attribute? > It really depends on what is the status-quo with respect to attributes. I’d rather avoid having an extra API just for that. > * I believe the intention of the description of MPI_COMM_AGREE is to ensure that after the agreement, all failures are propagated to all processes in the communicator, which will result in the next MPI_COMM_FAILURE_ACK/GET_ACKED returning a complete group of failed processes as of the last call to AGREE. However, the text specifically say that all failures have been detected. I think this might be the wrong choice of words since we don’t talk about detection anywhere else in the chapter other than to say that we don’t specify anything about failure detection. The text here should be tweaked. > *I’ve made changes to the text that I think make this more clear. > Good, I’ll take a look. > * We should add an example of a way to get a uniform group of failed processes via MPI_COMM_AGREE and FAILURE_ACK/GET_ACKED. I think this would be a good example of the relationship between the functions and it’s been a common question that we’ve received from users. > * I’ll work on drafting one of these. > I have one already, I’ll dump it in there. > * We need to make sure that we are compatible with the endpoints and init/finalize proposals. > * The next step for this is to probably read through their proposals and come up with a rough outline of what our text would need to add to cover these new concepts (if anything). > * The endpoints text is here: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/380 We may need to clarify what revoke does when called on one of the shadow handles. My take is that it should revoke all communicators with sharing that same cid, so as to remain global for the communicator at all ranks/endpoints. It doesn’t seem excruciatingly difficult to come up with something simple and workable on that front. > * I don’t think we have anything to look at for init/finalize. That proposal seems to have morphed a few times, but the current version is not about re-entrant MPI, but about a thread-safe Init/Finalize, which is completely orthogonal to our work. Probably nothing to do here. I think this is a correct assessment. Aurelien -- * Dr. Aurélien Bouteiller * Researcher at Innovative Computing Laboratory * University of Tennessee * 1122 Volunteer Boulevard, suite 309b * Knoxville, TN 37996 * 865 974 9375

1 0

Re: [mpiwg-ft] FTWG Action Items
by Wesley Bland 19 Dec '13

19 Dec '13

That’s what I was afraid of. AFAIK, both attributes and info keys aren’t really appropriate for this so the only way we could provide this information to the user is with a new API call: MPI_COMM_IS_REVOKED(). This wouldn’t be hard to add, but the question is whether it’s worth the trouble of adding it. The same thing could be accomplished with an MPI_SENDRECV to yourself, but that’s not really an elegant solution to put out there. On Dec 19, 2013, at 10:16 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: > On Thu, Dec 19, 2013 at 11:06 AM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: > * Function to query the status of a communicator (is it revoked or not). > * If this were done via a communicator attribute, it would be the only one I’m aware of that provides information to the user that is set by the library. All others are stored by the user and retrieved by the user. > * Do we want to add a new API to take care of this, or can we confirm that it’s ok to do this via an attribute? > > Sorry, I guess I was thinking of RMA when I suggested this; I don't know if it's feasible to use communicator attributes in this way. MPI_Win_get_attr() has several predefined keys for querying window properties, e.g. MPI_WIN_BASE, etc. Described in section 11.2.6 in MPI 3.0. > > ~Jim. > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft

1 0

Re: [mpiwg-ft] FTWG Action Items
by Jim Dinan 19 Dec '13

19 Dec '13

On Thu, Dec 19, 2013 at 11:06 AM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: > * Function to query the status of a communicator (is it revoked or not). > * If this were done via a communicator attribute, it would be the > only one I’m aware of that provides information to the user that is set by > the library. All others are stored by the user and retrieved by the user. > * Do we want to add a new API to take care of this, or can we > confirm that it’s ok to do this via an attribute? > Sorry, I guess I was thinking of RMA when I suggested this; I don't know if it's feasible to use communicator attributes in this way. MPI_Win_get_attr() has several predefined keys for querying window properties, e.g. MPI_WIN_BASE, etc. Described in section 11.2.6 in MPI 3.0. ~Jim.

1 0

FTWG Action Items
by Wesley Bland 19 Dec '13

19 Dec '13

Hi all, Since we’re not going to have a call next week due to the break, I thought it would be good to try to get the action items from the previous meeting and surrounding discussions going. The first two are things that we should all discuss, the second two are more minor things that I’ve been taking care of, the last one is important, but will require more work on our part. Please take a look and provide feedback. Thanks, Wesley --- * Add another error code to replace the use of MPI_ERR_PENDING (MPI_ERR_FAILURE_PENDING?) to prevent the requirement of scanning all return codes to find out if there has been a failure that left a request pending. * This is probably a good idea as it’s become clear that overloading MPI_ERR_PENDING is more confusing and adds overhead to what the user has to do to determine if their request is still valid or not. Coming up with the name is the only challenge here. * For now, I’ll change the text to say MPI_ERR_FAILURE_PENDING (23 characters < 30 character limit), but if we come up with something better, we can change it again. * Function to query the status of a communicator (is it revoked or not). * If this were done via a communicator attribute, it would be the only one I’m aware of that provides information to the user that is set by the library. All others are stored by the user and retrieved by the user. * Do we want to add a new API to take care of this, or can we confirm that it’s ok to do this via an attribute? * I believe the intention of the description of MPI_COMM_AGREE is to ensure that after the agreement, all failures are propagated to all processes in the communicator, which will result in the next MPI_COMM_FAILURE_ACK/GET_ACKED returning a complete group of failed processes as of the last call to AGREE. However, the text specifically say that all failures have been detected. I think this might be the wrong choice of words since we don’t talk about detection anywhere else in the chapter other than to say that we don’t specify anything about failure detection. The text here should be tweaked. *I’ve made changes to the text that I think make this more clear. * We should add an example of a way to get a uniform group of failed processes via MPI_COMM_AGREE and FAILURE_ACK/GET_ACKED. I think this would be a good example of the relationship between the functions and it’s been a common question that we’ve received from users. * I’ll work on drafting one of these. * We need to make sure that we are compatible with the endpoints and init/finalize proposals. * The next step for this is to probably read through their proposals and come up with a rough outline of what our text would need to add to cover these new concepts (if anything). * The endpoints text is here: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/380 * I don’t think we have anything to look at for init/finalize. That proposal seems to have morphed a few times, but the current version is not about re-entrant MPI, but about a thread-safe Init/Finalize, which is completely orthogonal to our work. Probably nothing to do here.

1 0

Re: [mpiwg-ft] 2013-12 MPI Forum Wrapup
by Wesley Bland 13 Dec '13

13 Dec '13

I agree here. This isn’t a major change, but I can see how it would be difficult to use since we’ve overloaded the meaning of MPI_ERR_PENDING. Wesley On Dec 13, 2013, at 2:40 PM, Aurélien Bouteiller <bouteill(a)icl.utk.edu> wrote: > Jim, > > Indeed, I was referring to the 2 subitems under Shrink as being dismissed already. > > The ERR_PENDING item is however on the table. It is not really necessary, but may help clarity, so I would not regard it as highly controversial. > > Aurelien > > > Le 13 déc. 2013 à 15:28, Jim Dinan <james.dinan(a)gmail.com> a écrit : > >> Hi Aurelien, >> >> Just to make sure there isn't confusion because of where comments were inlined: >> >>> * Do we need a new error code in place of MPI_ERR_PENDING? >>> * Right now, if there is an ANY_SOURCE receive in a request array passed to e.g. MPI_Waitall, you need to scan the list of requests to see if they are all MPI_ERR_PENDING in order to determine if a process failure may have occurred. >> >> IIRC, this item was one that we are supposed to look into. There was concern that one has to deduce that a failure occurred causing MPI_Wait/test_some/any/all by scanning the statuses to see if all are ERR_PENDING. Something more definite, like setting status errors to a new value (e.g. MPI_ERR_PENDING_FAILURE) would be clearer, but there is concern on the FTWG's side about fitting this cleanly into the standard. >> >> ~Jim. >> >> >> On Fri, Dec 13, 2013 at 2:40 PM, Aurélien Bouteiller <bouteill(a)icl.utk.edu> wrote: >> >> Le 13 déc. 2013 à 14:29, Jim Dinan <james.dinan(a)gmail.com> a écrit : >> >>> Hi All, >>> >>> +1, this is the most successful FT presentation that we've had so far, and we got lots of positive feedback. Here are few notes I took during the FT plenary, primarily items that were discussed or questions that were asked: >>> >> Jim, thanks for this summary of the key followup items. >> >>> * Do we need a new error code in place of MPI_ERR_PENDING? >>> * Right now, if there is an ANY_SOURCE receive in a request array passed to e.g. MPI_Waitall, you need to scan the list of requests to see if they are all MPI_ERR_PENDING in order to determine if a process failure may have occurred. >>> >>> * MPI_Comm_shrink should specify the process ordering (may already be covered in the spec) >>> * Should there be a key argument to MPI_Comm_shrink to allow the user to specify ordering? (Martin) >>> >> These 2 discussions items have been resolved in the room already: the spec is already well specified for rank ordering in shrink (same as split with well defined parameters); the key argument requires ranks to agree on something meaningful=> agree,shrink, so no performance advantage compared to shrink,split. >> >>> * Can we query whether a communicator has been revoked? Perhaps through a communicator attribute? (Jim) >>> >>> * Discussed getting the failed group uniformly at all processes >>> * Two protocols, shrink and agree >>> * Agree is faster when there are no intervening failures, otherwise shrink is faster >>> * Might be worthwhile to add a function in the future to achieve this, (Jim) >>> >> Additionally, it has been noted that there may be some performance advantages to standardize the “MPIX_comm_replace” example. Another long-term item. >> >>> * Need to verify sane interaction between endpoints, init/finalize, and FT proposals (Martin) >>> >>> ~Jim. >>> >>> >>> On Fri, Dec 13, 2013 at 12:01 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: >>> Now that the forum meeting has finished, I wanted to send a wrap-up email about how things went for those who couldn’t be there and to continue the discussion with those who were. >>> >>> We had a productive meeting on Monday and Tuesday within the working group where we discussed some of the concerns raised by some of our outside collaborators. I won’t go into all of the details as those were captured in the wiki page (https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/2013-12-10) but the general result was that we seem to have addressed all of the concerns that have been raised so far. The biggest challenge is to continue to have discussions with people and evangelize the proposal. >>> >>> On Thursday, we had a plenary time with the full forum where we presented the latest version of the proposal. There were minor changes since the last time this was presented to the forum, but there was one more important change to MPI_COMM_AGREE that provides new functionality. Again, I won’t go into detail about how it works as that was covered in the talk and text, but it does solve one of the use cases raised by Rich that some users want to be able to continue using a communicator without revoking and reordering ranks. Now it is possible to use MPI_COMM_AGREE as a transactional style function to periodically agree on the remaining processes. The slides for the talks given by Aurelien and me should be posted on the web site soon (http://meetings.mpi-forum.org/secretary/2013/12/slides.php) >>> >>> The reaction from the forum was quite positive. There were plenty of questions, but from what we could tell, it seems like most attendees were largely receptive to the current version. The major contributing factors to this that we heard from the people we talked to at the end of the plenary were that they like the ability to “turn off” FT for systems where it is not needed (smaller scale, reliable hardware, etc.) and we also provided more concrete examples of how to use the proposal. There had been concern about the performance impact of this proposal on systems where it was not needed, but the ability to compile it out should make that better. Many people said they still need to take this back to their users now that they have a better understanding of what’s going on in the proposal. We’ll hopefully hear back from them before March if there are concerns on their end. I don’t think there were any major issues with any of the technical content of the presentation. >>> >>> Our current plan is still to bring this for a reading at the next meeting in March in San Jose and pursue votes at the following two meetings. One of the most requested things to show at that meeting is to have performance numbers, so we will try to have something ready by then. These will be easier if we have some application partners that we can use to generate these numbers so if you have some “real” apps that you can run with ULFM (even if it’s failure-free runs), that would be very helpful. The other thing we can all do is to talk to our collaborators and see if there are any concerns that they didn’t raise during the full meeting that might hinder passing the proposal >>> >>> Thanks for all of your work! >>> Wesley >>> >>> _______________________________________________ >>> mpiwg-ft mailing list >>> mpiwg-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >>> >>> _______________________________________________ >>> mpiwg-ft mailing list >>> mpiwg-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >> >> -- >> * Dr. Aurélien Bouteiller >> * Researcher at Innovative Computing Laboratory >> * University of Tennessee >> * 1122 Volunteer Boulevard, suite 309b >> * Knoxville, TN 37996 >> * 865 974 9375 >> >> >> >> >> >> >> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > > -- > * Dr. Aurélien Bouteiller > * Researcher at Innovative Computing Laboratory > * University of Tennessee > * 1122 Volunteer Boulevard, suite 309b > * Knoxville, TN 37996 > * 865 974 9375 > > > > > > > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft

1 0

Re: [mpiwg-ft] 2013-12 MPI Forum Wrapup
by Aurélien Bouteiller 13 Dec '13

13 Dec '13

Jim, Indeed, I was referring to the 2 subitems under Shrink as being dismissed already. The ERR_PENDING item is however on the table. It is not really necessary, but may help clarity, so I would not regard it as highly controversial. Aurelien Le 13 déc. 2013 à 15:28, Jim Dinan <james.dinan(a)gmail.com> a écrit : > Hi Aurelien, > > Just to make sure there isn't confusion because of where comments were inlined: > > > * Do we need a new error code in place of MPI_ERR_PENDING? > > * Right now, if there is an ANY_SOURCE receive in a request array passed to e.g. MPI_Waitall, you need to scan the list of requests to see if they are all MPI_ERR_PENDING in order to determine if a process failure may have occurred. > > IIRC, this item was one that we are supposed to look into. There was concern that one has to deduce that a failure occurred causing MPI_Wait/test_some/any/all by scanning the statuses to see if all are ERR_PENDING. Something more definite, like setting status errors to a new value (e.g. MPI_ERR_PENDING_FAILURE) would be clearer, but there is concern on the FTWG's side about fitting this cleanly into the standard. > > ~Jim. > > > On Fri, Dec 13, 2013 at 2:40 PM, Aurélien Bouteiller <bouteill(a)icl.utk.edu> wrote: > > Le 13 déc. 2013 à 14:29, Jim Dinan <james.dinan(a)gmail.com> a écrit : > > > Hi All, > > > > +1, this is the most successful FT presentation that we've had so far, and we got lots of positive feedback. Here are few notes I took during the FT plenary, primarily items that were discussed or questions that were asked: > > > Jim, thanks for this summary of the key followup items. > > > * Do we need a new error code in place of MPI_ERR_PENDING? > > * Right now, if there is an ANY_SOURCE receive in a request array passed to e.g. MPI_Waitall, you need to scan the list of requests to see if they are all MPI_ERR_PENDING in order to determine if a process failure may have occurred. > > > > * MPI_Comm_shrink should specify the process ordering (may already be covered in the spec) > > * Should there be a key argument to MPI_Comm_shrink to allow the user to specify ordering? (Martin) > > > These 2 discussions items have been resolved in the room already: the spec is already well specified for rank ordering in shrink (same as split with well defined parameters); the key argument requires ranks to agree on something meaningful=> agree,shrink, so no performance advantage compared to shrink,split. > > > * Can we query whether a communicator has been revoked? Perhaps through a communicator attribute? (Jim) > > > > * Discussed getting the failed group uniformly at all processes > > * Two protocols, shrink and agree > > * Agree is faster when there are no intervening failures, otherwise shrink is faster > > * Might be worthwhile to add a function in the future to achieve this, (Jim) > > > Additionally, it has been noted that there may be some performance advantages to standardize the “MPIX_comm_replace” example. Another long-term item. > > > * Need to verify sane interaction between endpoints, init/finalize, and FT proposals (Martin) > > > > ~Jim. > > > > > > On Fri, Dec 13, 2013 at 12:01 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: > > Now that the forum meeting has finished, I wanted to send a wrap-up email about how things went for those who couldn’t be there and to continue the discussion with those who were. > > > > We had a productive meeting on Monday and Tuesday within the working group where we discussed some of the concerns raised by some of our outside collaborators. I won’t go into all of the details as those were captured in the wiki page (https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/2013-12-10) but the general result was that we seem to have addressed all of the concerns that have been raised so far. The biggest challenge is to continue to have discussions with people and evangelize the proposal. > > > > On Thursday, we had a plenary time with the full forum where we presented the latest version of the proposal. There were minor changes since the last time this was presented to the forum, but there was one more important change to MPI_COMM_AGREE that provides new functionality. Again, I won’t go into detail about how it works as that was covered in the talk and text, but it does solve one of the use cases raised by Rich that some users want to be able to continue using a communicator without revoking and reordering ranks. Now it is possible to use MPI_COMM_AGREE as a transactional style function to periodically agree on the remaining processes. The slides for the talks given by Aurelien and me should be posted on the web site soon (http://meetings.mpi-forum.org/secretary/2013/12/slides.php) > > > > The reaction from the forum was quite positive. There were plenty of questions, but from what we could tell, it seems like most attendees were largely receptive to the current version. The major contributing factors to this that we heard from the people we talked to at the end of the plenary were that they like the ability to “turn off” FT for systems where it is not needed (smaller scale, reliable hardware, etc.) and we also provided more concrete examples of how to use the proposal. There had been concern about the performance impact of this proposal on systems where it was not needed, but the ability to compile it out should make that better. Many people said they still need to take this back to their users now that they have a better understanding of what’s going on in the proposal. We’ll hopefully hear back from them before March if there are concerns on their end. I don’t think there were any major issues with any of the technical content of the presentation. > > > > Our current plan is still to bring this for a reading at the next meeting in March in San Jose and pursue votes at the following two meetings. One of the most requested things to show at that meeting is to have performance numbers, so we will try to have something ready by then. These will be easier if we have some application partners that we can use to generate these numbers so if you have some “real” apps that you can run with ULFM (even if it’s failure-free runs), that would be very helpful. The other thing we can all do is to talk to our collaborators and see if there are any concerns that they didn’t raise during the full meeting that might hinder passing the proposal > > > > Thanks for all of your work! > > Wesley > > > > _______________________________________________ > > mpiwg-ft mailing list > > mpiwg-ft(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > > > > _______________________________________________ > > mpiwg-ft mailing list > > mpiwg-ft(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > > -- > * Dr. Aurélien Bouteiller > * Researcher at Innovative Computing Laboratory > * University of Tennessee > * 1122 Volunteer Boulevard, suite 309b > * Knoxville, TN 37996 > * 865 974 9375 > > > > > > > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft -- * Dr. Aurélien Bouteiller * Researcher at Innovative Computing Laboratory * University of Tennessee * 1122 Volunteer Boulevard, suite 309b * Knoxville, TN 37996 * 865 974 9375

1 0

Re: [mpiwg-ft] 2013-12 MPI Forum Wrapup
by Jim Dinan 13 Dec '13

13 Dec '13

Hi Aurelien, Just to make sure there isn't confusion because of where comments were inlined: > * Do we need a new error code in place of MPI_ERR_PENDING? > * Right now, if there is an ANY_SOURCE receive in a request array passed to e.g. MPI_Waitall, you need to scan the list of requests to see if they are all MPI_ERR_PENDING in order to determine if a process failure may have occurred. IIRC, this item was one that we are supposed to look into. There was concern that one has to deduce that a failure occurred causing MPI_Wait/test_some/any/all by scanning the statuses to see if all are ERR_PENDING. Something more definite, like setting status errors to a new value (e.g. MPI_ERR_PENDING_FAILURE) would be clearer, but there is concern on the FTWG's side about fitting this cleanly into the standard. ~Jim. On Fri, Dec 13, 2013 at 2:40 PM, Aurélien Bouteiller <bouteill(a)icl.utk.edu>wrote: > > Le 13 déc. 2013 à 14:29, Jim Dinan <james.dinan(a)gmail.com> a écrit : > > > Hi All, > > > > +1, this is the most successful FT presentation that we've had so far, > and we got lots of positive feedback. Here are few notes I took during the > FT plenary, primarily items that were discussed or questions that were > asked: > > > Jim, thanks for this summary of the key followup items. > > > * Do we need a new error code in place of MPI_ERR_PENDING? > > * Right now, if there is an ANY_SOURCE receive in a request array > passed to e.g. MPI_Waitall, you need to scan the list of requests to see if > they are all MPI_ERR_PENDING in order to determine if a process failure may > have occurred. > > > > * MPI_Comm_shrink should specify the process ordering (may already be > covered in the spec) > > * Should there be a key argument to MPI_Comm_shrink to allow the > user to specify ordering? (Martin) > > > These 2 discussions items have been resolved in the room already: the spec > is already well specified for rank ordering in shrink (same as split with > well defined parameters); the key argument requires ranks to agree on > something meaningful=> agree,shrink, so no performance advantage compared > to shrink,split. > > > * Can we query whether a communicator has been revoked? Perhaps through > a communicator attribute? (Jim) > > > > * Discussed getting the failed group uniformly at all processes > > * Two protocols, shrink and agree > > * Agree is faster when there are no intervening failures, otherwise > shrink is faster > > * Might be worthwhile to add a function in the future to achieve > this, (Jim) > > > Additionally, it has been noted that there may be some performance > advantages to standardize the “MPIX_comm_replace” example. Another > long-term item. > > > * Need to verify sane interaction between endpoints, init/finalize, and > FT proposals (Martin) > > > > ~Jim. > > > > > > On Fri, Dec 13, 2013 at 12:01 PM, Wesley Bland <wbland(a)mcs.anl.gov> > wrote: > > Now that the forum meeting has finished, I wanted to send a wrap-up > email about how things went for those who couldn’t be there and to continue > the discussion with those who were. > > > > We had a productive meeting on Monday and Tuesday within the working > group where we discussed some of the concerns raised by some of our outside > collaborators. I won’t go into all of the details as those were captured in > the wiki page ( > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/2013-12-10) but the > general result was that we seem to have addressed all of the concerns that > have been raised so far. The biggest challenge is to continue to have > discussions with people and evangelize the proposal. > > > > On Thursday, we had a plenary time with the full forum where we > presented the latest version of the proposal. There were minor changes > since the last time this was presented to the forum, but there was one more > important change to MPI_COMM_AGREE that provides new functionality. Again, > I won’t go into detail about how it works as that was covered in the talk > and text, but it does solve one of the use cases raised by Rich that some > users want to be able to continue using a communicator without revoking and > reordering ranks. Now it is possible to use MPI_COMM_AGREE as a > transactional style function to periodically agree on the remaining > processes. The slides for the talks given by Aurelien and me should be > posted on the web site soon ( > http://meetings.mpi-forum.org/secretary/2013/12/slides.php) > > > > The reaction from the forum was quite positive. There were plenty of > questions, but from what we could tell, it seems like most attendees were > largely receptive to the current version. The major contributing factors to > this that we heard from the people we talked to at the end of the plenary > were that they like the ability to “turn off” FT for systems where it is > not needed (smaller scale, reliable hardware, etc.) and we also provided > more concrete examples of how to use the proposal. There had been concern > about the performance impact of this proposal on systems where it was not > needed, but the ability to compile it out should make that better. Many > people said they still need to take this back to their users now that they > have a better understanding of what’s going on in the proposal. We’ll > hopefully hear back from them before March if there are concerns on their > end. I don’t think there were any major issues with any of the technical > content of the presentation. > > > > Our current plan is still to bring this for a reading at the next > meeting in March in San Jose and pursue votes at the following two > meetings. One of the most requested things to show at that meeting is to > have performance numbers, so we will try to have something ready by then. > These will be easier if we have some application partners that we can use > to generate these numbers so if you have some “real” apps that you can run > with ULFM (even if it’s failure-free runs), that would be very helpful. The > other thing we can all do is to talk to our collaborators and see if there > are any concerns that they didn’t raise during the full meeting that might > hinder passing the proposal > > > > Thanks for all of your work! > > Wesley > > > > _______________________________________________ > > mpiwg-ft mailing list > > mpiwg-ft(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > > > > _______________________________________________ > > mpiwg-ft mailing list > > mpiwg-ft(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > > -- > * Dr. Aurélien Bouteiller > * Researcher at Innovative Computing Laboratory > * University of Tennessee > * 1122 Volunteer Boulevard, suite 309b > * Knoxville, TN 37996 > * 865 974 9375 > > > > > > > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >

1 0

Re: [mpiwg-ft] 2013-12 MPI Forum Wrapup
by Aurélien Bouteiller 13 Dec '13

13 Dec '13

Le 13 déc. 2013 à 14:29, Jim Dinan <james.dinan(a)gmail.com> a écrit : > Hi All, > > +1, this is the most successful FT presentation that we've had so far, and we got lots of positive feedback. Here are few notes I took during the FT plenary, primarily items that were discussed or questions that were asked: > Jim, thanks for this summary of the key followup items. > * Do we need a new error code in place of MPI_ERR_PENDING? > * Right now, if there is an ANY_SOURCE receive in a request array passed to e.g. MPI_Waitall, you need to scan the list of requests to see if they are all MPI_ERR_PENDING in order to determine if a process failure may have occurred. > > * MPI_Comm_shrink should specify the process ordering (may already be covered in the spec) > * Should there be a key argument to MPI_Comm_shrink to allow the user to specify ordering? (Martin) > These 2 discussions items have been resolved in the room already: the spec is already well specified for rank ordering in shrink (same as split with well defined parameters); the key argument requires ranks to agree on something meaningful=> agree,shrink, so no performance advantage compared to shrink,split. > * Can we query whether a communicator has been revoked? Perhaps through a communicator attribute? (Jim) > > * Discussed getting the failed group uniformly at all processes > * Two protocols, shrink and agree > * Agree is faster when there are no intervening failures, otherwise shrink is faster > * Might be worthwhile to add a function in the future to achieve this, (Jim) > Additionally, it has been noted that there may be some performance advantages to standardize the “MPIX_comm_replace” example. Another long-term item. > * Need to verify sane interaction between endpoints, init/finalize, and FT proposals (Martin) > > ~Jim. > > > On Fri, Dec 13, 2013 at 12:01 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: > Now that the forum meeting has finished, I wanted to send a wrap-up email about how things went for those who couldn’t be there and to continue the discussion with those who were. > > We had a productive meeting on Monday and Tuesday within the working group where we discussed some of the concerns raised by some of our outside collaborators. I won’t go into all of the details as those were captured in the wiki page (https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/2013-12-10) but the general result was that we seem to have addressed all of the concerns that have been raised so far. The biggest challenge is to continue to have discussions with people and evangelize the proposal. > > On Thursday, we had a plenary time with the full forum where we presented the latest version of the proposal. There were minor changes since the last time this was presented to the forum, but there was one more important change to MPI_COMM_AGREE that provides new functionality. Again, I won’t go into detail about how it works as that was covered in the talk and text, but it does solve one of the use cases raised by Rich that some users want to be able to continue using a communicator without revoking and reordering ranks. Now it is possible to use MPI_COMM_AGREE as a transactional style function to periodically agree on the remaining processes. The slides for the talks given by Aurelien and me should be posted on the web site soon (http://meetings.mpi-forum.org/secretary/2013/12/slides.php) > > The reaction from the forum was quite positive. There were plenty of questions, but from what we could tell, it seems like most attendees were largely receptive to the current version. The major contributing factors to this that we heard from the people we talked to at the end of the plenary were that they like the ability to “turn off” FT for systems where it is not needed (smaller scale, reliable hardware, etc.) and we also provided more concrete examples of how to use the proposal. There had been concern about the performance impact of this proposal on systems where it was not needed, but the ability to compile it out should make that better. Many people said they still need to take this back to their users now that they have a better understanding of what’s going on in the proposal. We’ll hopefully hear back from them before March if there are concerns on their end. I don’t think there were any major issues with any of the technical content of the presentation. > > Our current plan is still to bring this for a reading at the next meeting in March in San Jose and pursue votes at the following two meetings. One of the most requested things to show at that meeting is to have performance numbers, so we will try to have something ready by then. These will be easier if we have some application partners that we can use to generate these numbers so if you have some “real” apps that you can run with ULFM (even if it’s failure-free runs), that would be very helpful. The other thing we can all do is to talk to our collaborators and see if there are any concerns that they didn’t raise during the full meeting that might hinder passing the proposal > > Thanks for all of your work! > Wesley > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft -- * Dr. Aurélien Bouteiller * Researcher at Innovative Computing Laboratory * University of Tennessee * 1122 Volunteer Boulevard, suite 309b * Knoxville, TN 37996 * 865 974 9375

1 0

Re: [mpiwg-ft] 2013-12 MPI Forum Wrapup
by Jim Dinan 13 Dec '13

13 Dec '13

Hi All, +1, this is the most successful FT presentation that we've had so far, and we got lots of positive feedback. Here are few notes I took during the FT plenary, primarily items that were discussed or questions that were asked: * Do we need a new error code in place of MPI_ERR_PENDING? * Right now, if there is an ANY_SOURCE receive in a request array passed to e.g. MPI_Waitall, you need to scan the list of requests to see if they are all MPI_ERR_PENDING in order to determine if a process failure may have occurred. * MPI_Comm_shrink should specify the process ordering (may already be covered in the spec) * Should there be a key argument to MPI_Comm_shrink to allow the user to specify ordering? (Martin) * Can we query whether a communicator has been revoked? Perhaps through a communicator attribute? (Jim) * Discussed getting the failed group uniformly at all processes * Two protocols, shrink and agree * Agree is faster when there are no intervening failures, otherwise shrink is faster * Might be worthwhile to add a function in the future to achieve this, (Jim) * Need to verify sane interaction between endpoints, init/finalize, and FT proposals (Martin) ~Jim. On Fri, Dec 13, 2013 at 12:01 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: > Now that the forum meeting has finished, I wanted to send a wrap-up email > about how things went for those who couldn’t be there and to continue the > discussion with those who were. > > We had a productive meeting on Monday and Tuesday within the working group > where we discussed some of the concerns raised by some of our outside > collaborators. I won’t go into all of the details as those were captured in > the wiki page ( > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/2013-12-10) but the > general result was that we seem to have addressed all of the concerns that > have been raised so far. The biggest challenge is to continue to have > discussions with people and evangelize the proposal. > > On Thursday, we had a plenary time with the full forum where we presented > the latest version of the proposal. There were minor changes since the last > time this was presented to the forum, but there was one more important > change to MPI_COMM_AGREE that provides new functionality. Again, I won’t go > into detail about how it works as that was covered in the talk and text, > but it does solve one of the use cases raised by Rich that some users want > to be able to continue using a communicator without revoking and reordering > ranks. Now it is possible to use MPI_COMM_AGREE as a transactional style > function to periodically agree on the remaining processes. The slides for > the talks given by Aurelien and me should be posted on the web site soon ( > http://meetings.mpi-forum.org/secretary/2013/12/slides.php) > > The reaction from the forum was quite positive. There were plenty of > questions, but from what we could tell, it seems like most attendees were > largely receptive to the current version. The major contributing factors to > this that we heard from the people we talked to at the end of the plenary > were that they like the ability to “turn off” FT for systems where it is > not needed (smaller scale, reliable hardware, etc.) and we also provided > more concrete examples of how to use the proposal. There had been concern > about the performance impact of this proposal on systems where it was not > needed, but the ability to compile it out should make that better. Many > people said they still need to take this back to their users now that they > have a better understanding of what’s going on in the proposal. We’ll > hopefully hear back from them before March if there are concerns on their > end. I don’t think there were any major issues with any of the technical > content of the presentation. > > Our current plan is still to bring this for a reading at the next meeting > in March in San Jose and pursue votes at the following two meetings. One of > the most requested things to show at that meeting is to have performance > numbers, so we will try to have something ready by then. These will be > easier if we have some application partners that we can use to generate > these numbers so if you have some “real” apps that you can run with ULFM > (even if it’s failure-free runs), that would be very helpful. The other > thing we can all do is to talk to our collaborators and see if there are > any concerns that they didn’t raise during the full meeting that might > hinder passing the proposal > > Thanks for all of your work! > Wesley > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-ft >

1 0

2013-12 MPI Forum Wrapup
by Wesley Bland 13 Dec '13

13 Dec '13

Now that the forum meeting has finished, I wanted to send a wrap-up email about how things went for those who couldn’t be there and to continue the discussion with those who were. We had a productive meeting on Monday and Tuesday within the working group where we discussed some of the concerns raised by some of our outside collaborators. I won’t go into all of the details as those were captured in the wiki page (https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/2013-12-10) but the general result was that we seem to have addressed all of the concerns that have been raised so far. The biggest challenge is to continue to have discussions with people and evangelize the proposal. On Thursday, we had a plenary time with the full forum where we presented the latest version of the proposal. There were minor changes since the last time this was presented to the forum, but there was one more important change to MPI_COMM_AGREE that provides new functionality. Again, I won’t go into detail about how it works as that was covered in the talk and text, but it does solve one of the use cases raised by Rich that some users want to be able to continue using a communicator without revoking and reordering ranks. Now it is possible to use MPI_COMM_AGREE as a transactional style function to periodically agree on the remaining processes. The slides for the talks given by Aurelien and me should be posted on the web site soon (http://meetings.mpi-forum.org/secretary/2013/12/slides.php) The reaction from the forum was quite positive. There were plenty of questions, but from what we could tell, it seems like most attendees were largely receptive to the current version. The major contributing factors to this that we heard from the people we talked to at the end of the plenary were that they like the ability to “turn off” FT for systems where it is not needed (smaller scale, reliable hardware, etc.) and we also provided more concrete examples of how to use the proposal. There had been concern about the performance impact of this proposal on systems where it was not needed, but the ability to compile it out should make that better. Many people said they still need to take this back to their users now that they have a better understanding of what’s going on in the proposal. We’ll hopefully hear back from them before March if there are concerns on their end. I don’t think there were any major issues with any of the technical content of the presentation. Our current plan is still to bring this for a reading at the next meeting in March in San Jose and pursue votes at the following two meetings. One of the most requested things to show at that meeting is to have performance numbers, so we will try to have something ready by then. These will be easier if we have some application partners that we can use to generate these numbers so if you have some “real” apps that you can run with ULFM (even if it’s failure-free runs), that would be very helpful. The other thing we can all do is to talk to our collaborators and see if there are any concerns that they didn’t raise during the full meeting that might hinder passing the proposal Thanks for all of your work! Wesley

1 0