July 2013 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] WG call on tuesday aug. 9, 3pm est
by George Bosilca 15 Jul '13

15 Jul '13

Jim, let me try to articulate a comprehensible answer without going too deep into technicalities. On Jul 10, 2013, at 16:07 , Jim Dinan <james.dinan(a)gmail.com> wrote: > Hi George, > >> Re: comm_create and friends -- I'd be interested in hearing the result of the F2F discussion. I've been the person complaining about this, so if it would be helpful for me to join you guys over the phone next Friday, let me know. > > I did not have the opportunity to hear your complaint about this. Can you please summarize it here on the mailing list to make sure 1) that we are all at the same level of understanding; 2) the complaint reach all the interested audience and 3) we have the opportunity to address it as accurately as possible. > > I'm concerned that the current spec is missing a few features that are needed for the usage model where users continue to use a communicator with "holes" in it. In order for this model to work well, users must be able to query which processes are known to have failed in a given communicator. The proposed FT framework was designed to address a wide spectrum of possible approaches, by covering two extreme cases, on one side where global knowledge and global actions are required and on the other side where only local knowledge and local actions are required, while providing building blocks for all other intermediary approaches. Looking specifically at your concern it seems that you want to continue to use a communicator with holes while avoiding the revoke, but expect to need global knowledge about the failures. Thus you propose to have a special call GET_ALL_FAILED that provides you the list of failed processes. Interesting idea but unfortunately it has pitfalls. - In order to be able to use the list of failed processes to create new communicators using the COMM_CREATE_GROUP this list should be identical on all processes. Therefore the call to GET_ALL_FAILED is similar to an agreement (collective and must have a consensus meaning). Then the obvious question will be how do you synchronize all the living processes to call this blocking function together? This is why you need an atomic broadcast (guaranteed delivery despite failure) that can interrupt the normal behavior of the application. And this is typically the goal of the revoke functionality. You could implement your own protocol of failure knowledge propagation inside your communication scheme, but imagine how much your code will be impacted by this: at any point where you do a communication, you must be able to receive either a normal message of your application, or a notification message. >> without the revoke functionality (the knowledge about the fault is only propagated to processes that directly communicate with the dead nodes)? Do you really intend to implement your own protocol of failure knowledge propagation integrated in the application? >> >>> Given the current interface, users are not able to query >>> the set of failed processes without creating a new communicator and >>> translating ranks, via MPI_Comm_shrink. This requires the user to first >>> revoke the parent communicator, which is something we want to avoid. >> >> Again the revoke is an optional step. If you have a special way to have a consensus over all still alive processes about the list of dead processes, you just have to use it. However, for the sake of performance and portability we strongly believe that MPI should propose such a functionality for the users, that unlike you, don't want to implement their own consensus. >> >>> Along these same lines, we should also ensure that MPI_Comm_create_group >>> will work as expected when a communicator with holes is used, provided >>> the output group excludes failed processes (i.e., the operation is not >>> collective over any failed processes). >> >> I fail to see your concern here as I can't imagine which of the part of the current proposal prevent you from using such an approach. So from a technical point of view this approach should work in the context of the current proposal >> >> Now from a performance point of view the story is slightly different. While in the case of MPI_Comm_shrink highly optimized implementations have the opportunity for merge all the stage of the operation (consensus over the dead processes and creation of a new communicator) in a single step, your approach will have to validate (via an agreement) that the newly created communicator is indeed valid (aka, no new failure has been discovered during the communicator creation). > Given the current interface, users are not able to query the set of failed processes without creating a new communicator and translating ranks, via MPI_Comm_shrink. This requires the user to first revoke the parent communicator, which is something we want to avoid. Again the revoke is an optional step. If you have a special way to have a consensus over all still alive processes about the list of dead processes, you just have to use it. However, for the sake of performance and portability we strongly believe that MPI should propose such a functionality for the users, that unlike you, don't want to implement their own consensus. > Along these same lines, we should also ensure that MPI_Comm_create_group will work as expected when a communicator with holes is used, provided the output group excludes failed processes (i.e., the operation is not collective over any failed processes). I fail to see your concern here as I can't imagine which of the part of the current proposal prevent you from using such an approach. So from a technical point of view this approach should work in the context of the current proposal Now from a performance point of view the story is slightly different. While in the case of MPI_Comm_shrink highly optimized implementations have the opportunity for merge all the stage of the operation (consensus over the dead processes and creation of a new communicator) in a single step, your approach will have to validate (via an agreement) that the newly created communicator is indeed valid (aka, no new failure has been discovered during the communicator creation). And to do so, you will have to have a communicator without holes, because agree will not provide you appropriate information on communicators with holes. > This might not require any text changes, unless we want to allow this operation on revoked communicators. At this point I get confused as I was under the impression that your goal was to avoid revoking the communicator? Anyway, we do not want original meaning of COMM_CREATE_GROUP to be tainted by FT semantic. Especially true for comm_create_group which is supposed to be super scalable. See above for why it serves little purpose anyway, if it is not to become an expensive agreement operation. George. >> Re: Roadmap -- Before a reading, it might be helpful to give a brief presentation to the Forum again giving the high level ideas and justifications for each new addition to the FT proposal. I think it's been long enough that people have forgotten the details and this might help them feel more comfortable that the proposal is complete and self-consistent. > > There was at least one [more or less] "brief" presentation of the FT proposal at every meeting for the last two years. I would even emphasize the fact that over the last year no major modification of the proposal has been put forward, fact that might indicate a certain level of completeness and self-consistency. > > I'm just trying to convey the temperature in the room, as I felt it. I think a 15 minute, very high level warm-up immediately before the reading on the usage models, big ideas, and conventions would go a long way toward prepping the audience. > > ~Jim. > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] WG call on tuesday aug. 9, 3pm est
by Jim Dinan 10 Jul '13

10 Jul '13

Hi George, Re: comm_create and friends -- I'd be interested in hearing the result of > the F2F discussion. I've been the person complaining about this, so if it > would be helpful for me to join you guys over the phone next Friday, let me > know. > > > I did not have the opportunity to hear your complaint about this. Can you > please summarize it here on the mailing list to make sure 1) that we are > all at the same level of understanding; 2) the complaint reach all the > interested audience and 3) we have the opportunity to address it as > accurately as possible. > I'm concerned that the current spec is missing a few features that are needed for the usage model where users continue to use a communicator with "holes" in it. In order for this model to work well, users must be able to query which processes are known to have failed in a given communicator. Given the current interface, users are not able to query the set of failed processes without creating a new communicator and translating ranks, via MPI_Comm_shrink. This requires the user to first revoke the parent communicator, which is something we want to avoid. Along these same lines, we should also ensure that MPI_Comm_create_group will work as expected when a communicator with holes is used, provided the output group excludes failed processes (i.e., the operation is not collective over any failed processes). This might not require any text changes, unless we want to allow this operation on revoked communicators. Re: Roadmap -- Before a reading, it might be helpful to give a brief > presentation to the Forum again giving the high level ideas and > justifications for each new addition to the FT proposal. I think it's been > long enough that people have forgotten the details and this might help them > feel more comfortable that the proposal is complete and self-consistent. > > > There was at least one [more or less] "brief" presentation of the FT > proposal at every meeting for the last two years. I would even emphasize > the fact that over the last year no major modification of the proposal has > been put forward, fact that might indicate a certain level of completeness > and self-consistency. > I'm just trying to convey the temperature in the room, as I felt it. I think a 15 minute, very high level warm-up immediately before the reading on the usage models, big ideas, and conventions would go a long way toward prepping the audience. ~Jim.

1 0

Re: [Mpi3-ft] WG call on tuesday aug. 9, 3pm est
by George Bosilca 09 Jul '13

09 Jul '13

Jim, Thanks for your remarks. On Jul 9, 2013, at 23:43 , Jim Dinan <james.dinan(a)gmail.com> wrote: > Hi Aurelien, > > Re: comm_create and friends -- I'd be interested in hearing the result of the F2F discussion. I've been the person complaining about this, so if it would be helpful for me to join you guys over the phone next Friday, let me know. I did not have the opportunity to hear your complaint about this. Can you please summarize it here on the mailing list to make sure 1) that we are all at the same level of understanding; 2) the complaint reach all the interested audience and 3) we have the opportunity to address it as accurately as possible. > Re: Roadmap -- Before a reading, it might be helpful to give a brief presentation to the Forum again giving the high level ideas and justifications for each new addition to the FT proposal. I think it's been long enough that people have forgotten the details and this might help them feel more comfortable that the proposal is complete and self-consistent. There was at least one [more or less] "brief" presentation of the FT proposal at every meeting for the last two years. I would even emphasize the fact that over the last year no major modification of the proposal has been put forward, fact that might indicate a certain level of completeness and self-consistency. Thanks, George. > > ~Jim. > > > On Tue, Jul 9, 2013 at 4:13 PM, Aurélien Bouteiller <bouteill(a)icl.utk.edu> wrote: > The minutes of the meeting are summarized here: > > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/2013-07-09 > > > > > Le 8 juil. 2013 à 09:01, Aurélien Bouteiller <bouteill(a)icl.utk.edu> a écrit : > > > Dear WG members, > > > > This is a reminder that according to our planning, we are having our regular phone meeting tomorrow at 3pm EDT. > > > > > > Date: July 9, > > Time: 3pm EDT/New York > > Dial-in information: 218-339-4600 > > Code: 623998# > > > > > > Next Meetings: > > * July. 23, 2013 > > * August. 6, 2013 > > > > > > -- > > * Dr. Aurélien Bouteiller > > * Researcher at Innovative Computing Laboratory > > * University of Tennessee > > * 1122 Volunteer Boulevard, suite 309b > > * Knoxville, TN 37996 > > * 865 974 9375 > > > > > > > > > > > > > > > > -- > * Dr. Aurélien Bouteiller > * Researcher at Innovative Computing Laboratory > * University of Tennessee > * 1122 Volunteer Boulevard, suite 309b > * Knoxville, TN 37996 > * 865 974 9375 > > > > > > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] WG call on tuesday aug. 9, 3pm est
by Jim Dinan 09 Jul '13

09 Jul '13

Hi Aurelien, Re: comm_create and friends -- I'd be interested in hearing the result of the F2F discussion. I've been the person complaining about this, so if it would be helpful for me to join you guys over the phone next Friday, let me know. Re: Roadmap -- Before a reading, it might be helpful to give a brief presentation to the Forum again giving the high level ideas and justifications for each new addition to the FT proposal. I think it's been long enough that people have forgotten the details and this might help them feel more comfortable that the proposal is complete and self-consistent. ~Jim. On Tue, Jul 9, 2013 at 4:13 PM, Aurélien Bouteiller <bouteill(a)icl.utk.edu>wrote: > The minutes of the meeting are summarized here: > > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/2013-07-09 > > > > > Le 8 juil. 2013 à 09:01, Aurélien Bouteiller <bouteill(a)icl.utk.edu> a > écrit : > > > Dear WG members, > > > > This is a reminder that according to our planning, we are having our > regular phone meeting tomorrow at 3pm EDT. > > > > > > Date: July 9, > > Time: 3pm EDT/New York > > Dial-in information: 218-339-4600 > > Code: 623998# > > > > > > Next Meetings: > > * July. 23, 2013 > > * August. 6, 2013 > > > > > > -- > > * Dr. Aurélien Bouteiller > > * Researcher at Innovative Computing Laboratory > > * University of Tennessee > > * 1122 Volunteer Boulevard, suite 309b > > * Knoxville, TN 37996 > > * 865 974 9375 > > > > > > > > > > > > > > > > -- > * Dr. Aurélien Bouteiller > * Researcher at Innovative Computing Laboratory > * University of Tennessee > * 1122 Volunteer Boulevard, suite 309b > * Knoxville, TN 37996 > * 865 974 9375 > > > > > > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >

1 0

Re: [Mpi3-ft] WG call on tuesday aug. 9, 3pm est
by Aurélien Bouteiller 09 Jul '13

09 Jul '13

The minutes of the meeting are summarized here: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/2013-07-09 Le 8 juil. 2013 ï¿½ 09:01, Aurï¿½lien Bouteiller <bouteill(a)icl.utk.edu> a ï¿½crit : > Dear WG members, > > This is a reminder that according to our planning, we are having our regular phone meeting tomorrow at 3pm EDT. > > > Date: July 9, > Time: 3pm EDT/New York > Dial-in information: 218-339-4600 > Code: 623998# > > > Next Meetings: > * July. 23, 2013 > * August. 6, 2013 > > > -- > * Dr. Aurï¿½lien Bouteiller > * Researcher at Innovative Computing Laboratory > * University of Tennessee > * 1122 Volunteer Boulevard, suite 309b > * Knoxville, TN 37996 > * 865 974 9375 > > > > > > > -- * Dr. Aurï¿½lien Bouteiller * Researcher at Innovative Computing Laboratory * University of Tennessee * 1122 Volunteer Boulevard, suite 309b * Knoxville, TN 37996 * 865 974 9375

1 0

Re: [Mpi3-ft] WG call on tuesday aug. 9, 3pm est
by Aurélien Bouteiller 08 Jul '13

08 Jul '13

As food for thoughts, here are the initial findings we have come to at UTK, regarding the items discussed at the forum: MPI_COMM_ALLGET_FAILED (or whatever name it may come about): - Can be implemented by users (tedious, but doable: could be provided as an helper support lib, or standardized on its own later if deemed useful; bottom line, missing this API does not reduce features or limit expressivity) - Has no use case that we can think of, so it maybe is a hammer looking for a nail? + Can be used to do the COMM_CREATE_GROUP trick (but it is unclear what the advantage is with regard to SHRINK, as this ALLGET_FAILED will cost the same as the expensive part of SHRINK). We are wary of feature creep. If it is not absolutely necessary, we'd advocate not having it. We'd like to see more realistic use cases where shrink is not the appropriate answer before looking further. I/O etc: * It is very difficult to define a clear meaning for what is the status of a file pointer when multiple async I/O are ongoing. The I/O do not have to necessarily be in-order, so it might end-up that some later posted I/O will succeed while early one fail, during the same completion call. The status of the file pointer is very difficult to define in such a case (and the implementation could turn into a nightmare). * The file pointer can be reset by FILE_SEEK. This is a local function, so the end-user can keep using it to restore the value of the file pointer to something meaningful, corresponding to the status of its file operations. Shared pointers: impossible to tackle in any meaningful way. These are collective operations in the true sense, it should not be expected to be able to continue collective operations on an object (here the file) which has reported failures. Summary: there is no real problem. The pointer being undefined is not a blocker, it can be reset with seek, and (non-collective) operations continued, without much trouble. Lets discuss this tomorrow, Aurelien Le 8 juil. 2013 ï¿½ 09:01, Aurï¿½lien Bouteiller <bouteill(a)icl.utk.edu> a ï¿½crit : > Dear WG members, > > This is a reminder that according to our planning, we are having our regular phone meeting tomorrow at 3pm EDT. > > > Date: July 9, > Time: 3pm EDT/New York > Dial-in information: 218-339-4600 > Code: 623998# > > > Next Meetings: > * July. 23, 2013 > * August. 6, 2013 > > > -- > * Dr. Aurï¿½lien Bouteiller > * Researcher at Innovative Computing Laboratory > * University of Tennessee > * 1122 Volunteer Boulevard, suite 309b > * Knoxville, TN 37996 > * 865 974 9375 > > > > > > > -- * Dr. Aurï¿½lien Bouteiller * Researcher at Innovative Computing Laboratory * University of Tennessee * 1122 Volunteer Boulevard, suite 309b * Knoxville, TN 37996 * 865 974 9375

1 0

[Mpi3-ft] WG call on tuesday aug. 9, 3pm est
by Aurélien Bouteiller 08 Jul '13

08 Jul '13

Dear WG members, This is a reminder that according to our planning, we are having our regular phone meeting tomorrow at 3pm EDT. Date: July 9, Time: 3pm EDT/New York Dial-in information: 218-339-4600 Code: 623998# Next Meetings: * July. 23, 2013 * August. 6, 2013 -- * Dr. Aurï¿½lien Bouteiller * Researcher at Innovative Computing Laboratory * University of Tennessee * 1122 Volunteer Boulevard, suite 309b * Knoxville, TN 37996 * 865 974 9375

1 0