April 2012 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] FT WG Teleconference Wednesday @ Noon EDT
by Josh Hursey 11 Apr '12

11 Apr '12

During the teleconf today I noticed the the 'Next Meetings' listing was missing a week. The following is the correct list of upcoming meetings. * April 18, 2012 (Teleconf - If needed) * April 25, 2012 (Teleconf) * May 2, 2012 (Teleconf - If needed) * May 9, 2012 (Teleconf) * May 16, 2012 (Teleconf - If needed) * May 23, 2012 (Teleconf) * May 28, 2012 (MPI Forum Meeting) Sorry for the confusion. -- Josh On Wed, Apr 11, 2012 at 9:10 AM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > Reminder that we have a teleconf Wednesday at Noon EDT. > > Date: April 11, 2012 > Time: Noon EDT/New York > Dial-in information: 877-801-8130 > Code: 1044056 > > > Agenda: > - Misc. > ï¿½- Meeting next week? > ï¿½- Teleconf with RMA group > ï¿½- Teleconf with IO group > - User-Level Failure Mitigation Proposal > ï¿½ https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/323 > ï¿½- Fortran interfaces > ï¿½- Implementation progress > ï¿½- New name for 'invalidate' > ï¿½ ï¿½http://lists.mpi-forum.org/mpi3-ft/2012/03/1080.php > ï¿½- RMA sub-ticket > ï¿½ ï¿½https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/325 > ï¿½- I/O sub-ticket > ï¿½ ï¿½https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/326 > ï¿½- Dynamics sub-ticket > ï¿½ ï¿½https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/327 > - FT Query Interface (Josh) > ï¿½ https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/336 > - FATAL scope clarification (Josh) > ï¿½ https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/324 > - Other items? > > > Additional Agenda Items [as time permits]: > - FA-MPI Alternative Proposal (Tony) > ï¿½ http://lists.mpi-forum.org/mpi3-ft/2012/02/1048.php > - Piggybacking (Martin S.) > ï¿½ https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Piggybacking > - Process Recovery (Josh H.) > ï¿½ https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/process_recovery_2 > - Error to Fault changes proposal (Darius B.) > ï¿½ http://lists.mpi-forum.org/mpi3-ft/2011/06/0773.php > - Network Channel Failure > ï¿½ http://lists.mpi-forum.org/mpi3-ft/2011/06/0730.php > > > Next Meetings: > * April 11, 2012 (Teleconf) > * April 18, 2012 (Teleconf - If needed) > * May ï¿½ ï¿½2, 2012 (Teleconf) > * May ï¿½ ï¿½9, 2012 (Teleconf - If needed) > * May ï¿½ 16, 2012 (Teleconf) > * May ï¿½ 23, 2012 (Teleconf - If needed) > * May ï¿½ 28, 2012 (MPI Forum Meeting) > > Respond to the list if you have any additional topics that you would > like to discuss. > > -- Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] FT WG Teleconference Wednesday @ Noon EDT
by Josh Hursey 11 Apr '12

11 Apr '12

Reminder that we have a teleconf Wednesday at Noon EDT. Date: April 11, 2012 Time: Noon EDT/New York Dial-in information: 877-801-8130 Code: 1044056 Agenda: - Misc. - Meeting next week? - Teleconf with RMA group - Teleconf with IO group - User-Level Failure Mitigation Proposal https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/323 - Fortran interfaces - Implementation progress - New name for 'invalidate' http://lists.mpi-forum.org/mpi3-ft/2012/03/1080.php - RMA sub-ticket https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/325 - I/O sub-ticket https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/326 - Dynamics sub-ticket https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/327 - FT Query Interface (Josh) https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/336 - FATAL scope clarification (Josh) https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/324 - Other items? Additional Agenda Items [as time permits]: - FA-MPI Alternative Proposal (Tony) http://lists.mpi-forum.org/mpi3-ft/2012/02/1048.php - Piggybacking (Martin S.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/Piggybacking - Process Recovery (Josh H.) https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/process_recovery_2 - Error to Fault changes proposal (Darius B.) http://lists.mpi-forum.org/mpi3-ft/2011/06/0773.php - Network Channel Failure http://lists.mpi-forum.org/mpi3-ft/2011/06/0730.php Next Meetings: * April 11, 2012 (Teleconf) * April 18, 2012 (Teleconf - If needed) * May 2, 2012 (Teleconf) * May 9, 2012 (Teleconf - If needed) * May 16, 2012 (Teleconf) * May 23, 2012 (Teleconf - If needed) * May 28, 2012 (MPI Forum Meeting) Respond to the list if you have any additional topics that you would like to discuss. -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Using MPI_Comm_shrink and MPI_Comm_agreement in the same application
by Josh Hursey 10 Apr '12

10 Apr '12

Those are some good general rules. You can definitely write a fault tolerant code that does not require the agreement operation, but it does make things easier to reason about at times. Often you will want the agreement as part of a termination algorithm ("Do we all agree that it is time to finish and call finalize/move to the next phase of the computation?"). In the error handler callback, I think the rule-of-thumb is to not call shrink/agreement in them, and that if you need to do something to the communicator consider invalidate/revoke. -- Josh On Mon, Apr 9, 2012 at 5:01 PM, David Solt <dsolt(a)us.ibm.com> wrote: > Thanks for the feedback from everyone. ï¿½ ï¿½It is interesting to read over the > various approaches. ï¿½I was attempting to see if I could distill any general > rules from the examples. ï¿½ I like the guideline of "Don't use > MPI_Comm_agreement in the non-error case", though even George's example will > call MPI_Comm_agreement when there are no errors. ï¿½ It is clearly possible > to use MPI_Comm_agreement in non-failures cases, but it still seems like a > good guideline. ï¿½ ï¿½An obvious rule is "Don't write code where one rank can > possibly call MPI_Comm_shrink while another calls MPI_Comm_agreement". > ï¿½I'm still wrestling with whether MPI_Comm_agreement is necessary to write > nontrivial FT code or if it is simply a convenience that makes writing FT > code simpler. ï¿½ It seems to me that you can write some simple infinite code > loops that error check MPI calls but do not use MPI_Comm_agreement, but > before a rank makes any significant transition within the code (for example, > calling MPI_Finalize or leaving a library), a call to MPI_Comm_agreement > seems necessary. ï¿½ I'm assuming in all these cases that the application > calls collectives. ï¿½ Anyhow, thanks again for the code/comments. ï¿½ ï¿½ As to > Josh's question of can you ever call shrink or agreement in a callback, I > believe that would be very difficult to do safely, especially if the > application is known to make direct calls to either of those routines. > > Thanks, > Dave > > > > From: ï¿½ ï¿½ ï¿½ ï¿½George Bosilca <bosilca(a)eecs.utk.edu> > To: ï¿½ ï¿½ ï¿½ ï¿½"MPI 3.0 Fault Tolerance and Dynamic Process Control working > Group" <mpi3-ft(a)lists.mpi-forum.org> > Date: ï¿½ ï¿½ ï¿½ ï¿½04/09/2012 11:41 AM > Subject: ï¿½ ï¿½ ï¿½ ï¿½Re: [Mpi3-ft] Using MPI_Comm_shrink and MPI_Comm_agreement > in the ï¿½ ï¿½ ï¿½ ï¿½same application > Sent by: ï¿½ ï¿½ ï¿½ ï¿½mpi3-ft-bounces(a)lists.mpi-forum.org > ________________________________ > > > > Dave, > > The MPI_Comm_agree is meant to be used in case of failure. It has a > significant cost, large enough not to force it on users in __any__ case. > > Below you will find the FT version of your example. We started from the non > fault tolerant version, and added what was required to make it fault > tolerant. > > ï¿½george. > > > > ï¿½ ï¿½success = false; > ï¿½ ï¿½do { > ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_size(comm, &size); > ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_rank(comm, &rank); > ï¿½ ï¿½ ï¿½ ï¿½root = (0 == rank); > ï¿½ ï¿½ ï¿½ ï¿½do { > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½if (root) read_some_data_from_a_file(buffer); > > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½rc = MPI_Bcast(buffer, .... ,root, comm); > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½if( MPI_SUCCESS != rc ) { ï¿½/* check only for FT type of errors */ > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_revoke(comm); > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½break; > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½} > > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½done = do_computation(buffer, size); > > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½rc = MPI_Allreduce( &done, &success, ... MPI_OP_AND, comm ); > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½if( MPI_SUCCESS != rc ) { ï¿½/* check only for FT type of errors */ > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½success = false; ï¿½/* not defined if MPI_Allreduce failed */ > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_revoke(comm); > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½break; > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½} > ï¿½ ï¿½ ï¿½ ï¿½} while(false == success); > ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_agree( comm, &success ); > ï¿½ ï¿½ ï¿½ ï¿½if( false == success ) { > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_revoke(comm); > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_shrink(comm, &newcomm); > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_free(comm); > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½comm = newcomm; > ï¿½ ï¿½ ï¿½ ï¿½} > ï¿½ ï¿½} while (false == success); > > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Using MPI_Comm_shrink and MPI_Comm_agreement in the same application
by David Solt 09 Apr '12

09 Apr '12

Thanks for the feedback from everyone. It is interesting to read over the various approaches. I was attempting to see if I could distill any general rules from the examples. I like the guideline of "Don't use MPI_Comm_agreement in the non-error case", though even George's example will call MPI_Comm_agreement when there are no errors. It is clearly possible to use MPI_Comm_agreement in non-failures cases, but it still seems like a good guideline. An obvious rule is "Don't write code where one rank can possibly call MPI_Comm_shrink while another calls MPI_Comm_agreement". I'm still wrestling with whether MPI_Comm_agreement is necessary to write nontrivial FT code or if it is simply a convenience that makes writing FT code simpler. It seems to me that you can write some simple infinite code loops that error check MPI calls but do not use MPI_Comm_agreement, but before a rank makes any significant transition within the code (for example, calling MPI_Finalize or leaving a library), a call to MPI_Comm_agreement seems necessary. I'm assuming in all these cases that the application calls collectives. Anyhow, thanks again for the code/comments. As to Josh's question of can you ever call shrink or agreement in a callback, I believe that would be very difficult to do safely, especially if the application is known to make direct calls to either of those routines. Thanks, Dave From: George Bosilca <bosilca(a)eecs.utk.edu> To: "MPI 3.0 Fault Tolerance and Dynamic Process Control working Group" <mpi3-ft(a)lists.mpi-forum.org> Date: 04/09/2012 11:41 AM Subject: Re: [Mpi3-ft] Using MPI_Comm_shrink and MPI_Comm_agreement in the same application Sent by: mpi3-ft-bounces(a)lists.mpi-forum.org Dave, The MPI_Comm_agree is meant to be used in case of failure. It has a significant cost, large enough not to force it on users in __any__ case. Below you will find the FT version of your example. We started from the non fault tolerant version, and added what was required to make it fault tolerant. george. success = false; do { MPI_Comm_size(comm, &size); MPI_Comm_rank(comm, &rank); root = (0 == rank); do { if (root) read_some_data_from_a_file(buffer); rc = MPI_Bcast(buffer, .... ,root, comm); if( MPI_SUCCESS != rc ) { /* check only for FT type of errors */ MPI_Comm_revoke(comm); break; } done = do_computation(buffer, size); rc = MPI_Allreduce( &done, &success, ... MPI_OP_AND, comm ); if( MPI_SUCCESS != rc ) { /* check only for FT type of errors */ success = false; /* not defined if MPI_Allreduce failed */ MPI_Comm_revoke(comm); break; } } while(false == success); MPI_Comm_agree( comm, &success ); if( false == success ) { MPI_Comm_revoke(comm); MPI_Comm_shrink(comm, &newcomm); MPI_Comm_free(comm); comm = newcomm; } } while (false == success); _______________________________________________ mpi3-ft mailing list mpi3-ft(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Using MPI_Comm_shrink and MPI_Comm_agreement in the same application
by George Bosilca 09 Apr '12

09 Apr '12

Dave, The MPI_Comm_agree is meant to be used in case of failure. It has a significant cost, large enough not to force it on users in __any__ case. Below you will find the FT version of your example. We started from the non fault tolerant version, and added what was required to make it fault tolerant. george. success = false; do { MPI_Comm_size(comm, &size); MPI_Comm_rank(comm, &rank); root = (0 == rank); do { if (root) read_some_data_from_a_file(buffer); rc = MPI_Bcast(buffer, .... ,root, comm); if( MPI_SUCCESS != rc ) { /* check only for FT type of errors */ MPI_Comm_revoke(comm); break; } done = do_computation(buffer, size); rc = MPI_Allreduce( &done, &success, ... MPI_OP_AND, comm ); if( MPI_SUCCESS != rc ) { /* check only for FT type of errors */ success = false; /* not defined if MPI_Allreduce failed */ MPI_Comm_revoke(comm); break; } } while(false == success); MPI_Comm_agree( comm, &success ); if( false == success ) { MPI_Comm_revoke(comm); MPI_Comm_shrink(comm, &newcomm); MPI_Comm_free(comm); comm = newcomm; } } while (false == success);

1 0

Re: [Mpi3-ft] Using MPI_Comm_shrink and MPI_Comm_agreement in the same application
by Josh Hursey 09 Apr '12

09 Apr '12

That is one way to do it. The core problem is two fold: 1) the collective operations do not return uniformly in the presence of process failure 2) shrink() and agreement() cannot be invalidated, and must return uniformly. So you can either not branch on the return code from a collective, or branch on the return code and code around the potential bifurcation in the program execution flow when an error occurs. An alternative to your solution would be: ------------------------- do { if (root) read_some_data_from_a_file(buffer); err = MPI_Bcast(buffer, .... ,root, comm); if (err) { /* Display a message, but take no collective action */ } /* Let's agree for sure if we are done with the computation */ done = (MPI_SUCCESS == err); MPI_Comm_agreement(comm, &done); if( !done ) { MPI_Comm_invalidate(comm); MPI_Comm_shrink(comm, &newcomm); MPI_Comm_free(&comm); comm = newcomm; continue; /* try Bcast again */ } MPI_Comm_size(comm, size); done = do_computation(buffer, size); MPI_Comm_agreement(comm, &done); if( !done ) { MPI_Comm_invalidate(comm); MPI_Comm_shrink(comm, &newcomm); MPI_Comm_free(&comm); comm = newcomm; continue; /* try Bcast again */ } } while (!done); MPI_Finalize(); ---------------------- or, to reduce the number of agreements, don't block after an error in Bcast. ------------------------- do { if (root) read_some_data_from_a_file(buffer); err = MPI_Bcast(buffer, .... ,root, comm); if( err ) { done = (MPI_SUCCESS == err ); } else { /* Local success, so do my work until I finish it or hit a process failure error */ MPI_Comm_size(comm, size); done = do_computation(buffer, size); } /* Alternatively, if do_computation() has communication in it * over the communicator 'comm' then you can call invalidate here * to ensure it will unblock pt2pt operations. But if do_computation() * is purely local, then we know it will eventually emerge and find its * way to the agreement */ MPI_Comm_agreement(comm, &done); if( !done ) { /* Oops. either the bcast failed or the do_computation * either way go back and try again */ MPI_Comm_invalidate(comm); MPI_Comm_shrink(comm, &newcomm); MPI_Comm_free(&comm); comm = newcomm; continue; /* try Bcast again */ } } while (!done); MPI_Finalize(); ---------------------- Both your solution and the variations above are correct. It is just a matter of which way works best with your code. This does raise the subtle question of 'what if you tried to do this in an error handler callback function' - is it ever safe to call 'shrink' or 'agreement' in the callback? -- Josh On Fri, Apr 6, 2012 at 12:32 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: > Ignore the last email there were bugs, and the bcast should be in a while loop. > > -d > > ï¿½ ï¿½ ï¿½ do { > > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ if (root) read_some_data_from_a_file(buffer); > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ do { > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ int val; > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ err = MPI_Bcast(buffer, .... ,root, comm); > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ if (err) { > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ val = FALSE; > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ MPI_Comm_agreement(comm, &val); > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ MPI_Comm_invalidate(comm); > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ MPI_Comm_shrink(comm, &newcomm); > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ MPI_Comm_free(&comm); > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ comm = newcomm; > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ } else { > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ val = TRUE; > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ MPI_Comm_agreement(comm, &val); > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ if (!val) { > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ MPI_Comm_shrink(comm, &newcomm); > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ MPI_Comm_free(&comm); > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ comm = newcomm; > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ } > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ } while (!val); > > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ MPI_Comm_size(comm, size); > > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ done = do_computation(buffer, size); > > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ /* Let's agree for sure if we are done with the computation */ > > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ MPI_Comm_agreement(comm, &done); > > ï¿½ ï¿½ ï¿½ while (!done); > > ï¿½ ï¿½ ï¿½ MPI_Finalize(); > > On Apr 6, 2012, at 11:28 AM, Darius Buntinas wrote: > >> >> Wouldn't you need to do something like this? >> >> -d >> >> ï¿½ ï¿½ ï¿½ ï¿½do { >> >> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½if (root) read_some_data_from_a_file(buffer); >> >> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½err = MPI_Bcast(buffer, .... ,root, comm); >> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½if (err) { >> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½int val = FALSE; >> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_invalidate(comm); >> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_shrink(comm, &newcomm); >> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_free(&comm); >> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½comm = newcomm; >> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_agreement(comm, &val); >> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½} else { >> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½int val = TRUE; >> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_agreement(comm, &val); >> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½if (!val) { >> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_shrink(comm, &newcomm); >> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_free(&comm); >> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½comm = newcomm; >> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½} >> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_size(comm, size); >> >> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½done = do_computation(buffer, size); >> >> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½/* Let's agree for sure if we are done with the computation */ >> >> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_agreement(comm, &done); >> >> ï¿½ ï¿½ ï¿½ ï¿½while (!done); >> >> ï¿½ ï¿½ ï¿½ ï¿½MPI_Finalize(); >> >> On Apr 5, 2012, at 4:50 PM, David Solt wrote: >> >>> I have another question about MPI_Comm_agreement: >>> >>> If I want to do this: >>> >>> ï¿½ ï¿½ ï¿½ ï¿½.... >>> ï¿½ ï¿½ ï¿½ ï¿½do { >>> >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½if (root) read_some_data_from_a_file(buffer); >>> >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½err = MPI_Bcast(buffer, .... ,root, comm); >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½if (err) { >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_invalidate(comm); >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_shrink(comm, &newcomm); >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_free(&comm); >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½comm = newcomm; >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½} >>> >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_size(comm, size); >>> >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½done = do_computation(buffer, size); >>> >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½/* Let's agree for sure if we are done with the computation */ >>> >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_agreement(comm, &done); >>> >>> ï¿½ ï¿½ ï¿½ ï¿½while (!done); >>> >>> ï¿½ ï¿½ ï¿½ ï¿½MPI_Finalize(); >>> >>> This code can deadlock because some ranks may enter MPI_Comm_agreement while others detect an error in MPI_Bcast and call MPI_Comm_invalidate followed by MPI_Comm_shrink (assume that do_computation is really, really fast). ï¿½ The call to MPI_Comm_invalidate will not allow the processes that have already entered MPI_Comm_agreement to leave that call (P543L45: "Advice to users. MPI_COMM_AGREEMENT maintains its collective behavior even >>> if the comm is invalidated. (End of advice to users.)" ) and MPI_Comm_agreement cannot return an error due to the call to MPI_Comm_invalidate (P545L38: ï¿½"This function must not return an error due to process failure (error classes MPI_ERR_PROC_FAILED and MPI_ERR_INVALIDATED)...") . >>> >>> This would not work: >>> ï¿½ ï¿½ ï¿½ ï¿½.... >>> ï¿½ ï¿½ ï¿½ ï¿½do { >>> >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½if (root) read_some_data_from_a_file(buffer); >>> >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½err = MPI_Bcast(buffer, .... ,root, comm); >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½if (err) { >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_invalidate(comm); >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_shrink(comm, &newcomm); >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_free(&comm); >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½comm = newcomm; >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½} >>> >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_size(comm, size); >>> >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½done = do_computation(buffer, size); >>> >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½/* Let's agree for sure if we are done with the computation */ >>> >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Barrier(comm); ï¿½ // don't check the error code, this is just to "catch" invalidate messages >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_agreement(comm, &done); >>> >>> ï¿½ ï¿½ ï¿½ ï¿½while (!done); >>> >>> ï¿½ ï¿½ ï¿½ ï¿½MPI_Finalize(); >>> >>> because a rank may enter the barrier, get knocked out by the call to invalidate and then go on to call MPI_Comm_agreement anyway. ï¿½So we can try the following: >>> >>> ï¿½ ï¿½ ï¿½ ï¿½do { >>> >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½if (root) read_some_data_from_a_file(buffer); >>> >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½err = MPI_Bcast(buffer, .... ,root, comm); >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½if (err) { >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_invalidate(comm); >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_shrink(comm, &newcomm); >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_free(&comm); >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½comm = newcomm; >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½} >>> >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_size(comm, size); >>> >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½done = do_computation(buffer, size); >>> >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½/* Let's agree for sure if we are done with the computation */ >>> >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½err ï¿½= MPI_Barrier(comm); >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½if (err) { >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_invalidate(comm); >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_shrink(comm, &newcomm); >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_free(&comm); >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½comm = newcomm; >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½} >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_agreement(comm, &done); >>> >>> ï¿½ ï¿½ ï¿½ ï¿½while (!done); >>> >>> ï¿½ ï¿½ ï¿½ ï¿½MPI_Finalize(); >>> >>> But now we have done nothing more than move the problem down a few lines. ï¿½Some ranks may succeed the MPI_Barrier and go on to MPI_Comm_agreement while others attempt to invalidate/shrink. ï¿½ Is there a solution to this problem? ï¿½ How can one safely use MPI_Comm_agreement and MPI_Comm_shrink in the same application? >>> >>> Thanks, >>> Dave ï¿½_______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Using MPI_Comm_shrink and MPI_Comm_agreement in the same application
by Darius Buntinas 06 Apr '12

06 Apr '12

Ignore the last email there were bugs, and the bcast should be in a while loop. -d do { if (root) read_some_data_from_a_file(buffer); do { int val; err = MPI_Bcast(buffer, .... ,root, comm); if (err) { val = FALSE; MPI_Comm_agreement(comm, &val); MPI_Comm_invalidate(comm); MPI_Comm_shrink(comm, &newcomm); MPI_Comm_free(&comm); comm = newcomm; } else { val = TRUE; MPI_Comm_agreement(comm, &val); if (!val) { MPI_Comm_shrink(comm, &newcomm); MPI_Comm_free(&comm); comm = newcomm; } } while (!val); MPI_Comm_size(comm, size); done = do_computation(buffer, size); /* Let's agree for sure if we are done with the computation */ MPI_Comm_agreement(comm, &done); while (!done); MPI_Finalize(); On Apr 6, 2012, at 11:28 AM, Darius Buntinas wrote: > > Wouldn't you need to do something like this? > > -d > > do { > > if (root) read_some_data_from_a_file(buffer); > > err = MPI_Bcast(buffer, .... ,root, comm); > if (err) { > int val = FALSE; > MPI_Comm_invalidate(comm); > MPI_Comm_shrink(comm, &newcomm); > MPI_Comm_free(&comm); > comm = newcomm; > MPI_Comm_agreement(comm, &val); > } else { > int val = TRUE; > MPI_Comm_agreement(comm, &val); > if (!val) { > MPI_Comm_shrink(comm, &newcomm); > MPI_Comm_free(&comm); > comm = newcomm; > } > MPI_Comm_size(comm, size); > > done = do_computation(buffer, size); > > /* Let's agree for sure if we are done with the computation */ > > MPI_Comm_agreement(comm, &done); > > while (!done); > > MPI_Finalize(); > > On Apr 5, 2012, at 4:50 PM, David Solt wrote: > >> I have another question about MPI_Comm_agreement: >> >> If I want to do this: >> >> .... >> do { >> >> if (root) read_some_data_from_a_file(buffer); >> >> err = MPI_Bcast(buffer, .... ,root, comm); >> if (err) { >> MPI_Comm_invalidate(comm); >> MPI_Comm_shrink(comm, &newcomm); >> MPI_Comm_free(&comm); >> comm = newcomm; >> } >> >> MPI_Comm_size(comm, size); >> >> done = do_computation(buffer, size); >> >> /* Let's agree for sure if we are done with the computation */ >> >> MPI_Comm_agreement(comm, &done); >> >> while (!done); >> >> MPI_Finalize(); >> >> This code can deadlock because some ranks may enter MPI_Comm_agreement while others detect an error in MPI_Bcast and call MPI_Comm_invalidate followed by MPI_Comm_shrink (assume that do_computation is really, really fast). The call to MPI_Comm_invalidate will not allow the processes that have already entered MPI_Comm_agreement to leave that call (P543L45: "Advice to users. MPI_COMM_AGREEMENT maintains its collective behavior even >> if the comm is invalidated. (End of advice to users.)" ) and MPI_Comm_agreement cannot return an error due to the call to MPI_Comm_invalidate (P545L38: "This function must not return an error due to process failure (error classes MPI_ERR_PROC_FAILED and MPI_ERR_INVALIDATED)...") . >> >> This would not work: >> .... >> do { >> >> if (root) read_some_data_from_a_file(buffer); >> >> err = MPI_Bcast(buffer, .... ,root, comm); >> if (err) { >> MPI_Comm_invalidate(comm); >> MPI_Comm_shrink(comm, &newcomm); >> MPI_Comm_free(&comm); >> comm = newcomm; >> } >> >> MPI_Comm_size(comm, size); >> >> done = do_computation(buffer, size); >> >> /* Let's agree for sure if we are done with the computation */ >> >> MPI_Barrier(comm); // don't check the error code, this is just to "catch" invalidate messages >> MPI_Comm_agreement(comm, &done); >> >> while (!done); >> >> MPI_Finalize(); >> >> because a rank may enter the barrier, get knocked out by the call to invalidate and then go on to call MPI_Comm_agreement anyway. So we can try the following: >> >> do { >> >> if (root) read_some_data_from_a_file(buffer); >> >> err = MPI_Bcast(buffer, .... ,root, comm); >> if (err) { >> MPI_Comm_invalidate(comm); >> MPI_Comm_shrink(comm, &newcomm); >> MPI_Comm_free(&comm); >> comm = newcomm; >> } >> >> MPI_Comm_size(comm, size); >> >> done = do_computation(buffer, size); >> >> /* Let's agree for sure if we are done with the computation */ >> >> err = MPI_Barrier(comm); >> if (err) { >> MPI_Comm_invalidate(comm); >> MPI_Comm_shrink(comm, &newcomm); >> MPI_Comm_free(&comm); >> comm = newcomm; >> } >> MPI_Comm_agreement(comm, &done); >> >> while (!done); >> >> MPI_Finalize(); >> >> But now we have done nothing more than move the problem down a few lines. Some ranks may succeed the MPI_Barrier and go on to MPI_Comm_agreement while others attempt to invalidate/shrink. Is there a solution to this problem? How can one safely use MPI_Comm_agreement and MPI_Comm_shrink in the same application? >> >> Thanks, >> Dave _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >

1 0

Re: [Mpi3-ft] Using MPI_Comm_shrink and MPI_Comm_agreement in the same application
by Darius Buntinas 06 Apr '12

06 Apr '12

Wouldn't you need to do something like this? -d do { if (root) read_some_data_from_a_file(buffer); err = MPI_Bcast(buffer, .... ,root, comm); if (err) { int val = FALSE; MPI_Comm_invalidate(comm); MPI_Comm_shrink(comm, &newcomm); MPI_Comm_free(&comm); comm = newcomm; MPI_Comm_agreement(comm, &val); } else { int val = TRUE; MPI_Comm_agreement(comm, &val); if (!val) { MPI_Comm_shrink(comm, &newcomm); MPI_Comm_free(&comm); comm = newcomm; } MPI_Comm_size(comm, size); done = do_computation(buffer, size); /* Let's agree for sure if we are done with the computation */ MPI_Comm_agreement(comm, &done); while (!done); MPI_Finalize(); On Apr 5, 2012, at 4:50 PM, David Solt wrote: > I have another question about MPI_Comm_agreement: > > If I want to do this: > > .... > do { > > if (root) read_some_data_from_a_file(buffer); > > err = MPI_Bcast(buffer, .... ,root, comm); > if (err) { > MPI_Comm_invalidate(comm); > MPI_Comm_shrink(comm, &newcomm); > MPI_Comm_free(&comm); > comm = newcomm; > } > > MPI_Comm_size(comm, size); > > done = do_computation(buffer, size); > > /* Let's agree for sure if we are done with the computation */ > > MPI_Comm_agreement(comm, &done); > > while (!done); > > MPI_Finalize(); > > This code can deadlock because some ranks may enter MPI_Comm_agreement while others detect an error in MPI_Bcast and call MPI_Comm_invalidate followed by MPI_Comm_shrink (assume that do_computation is really, really fast). The call to MPI_Comm_invalidate will not allow the processes that have already entered MPI_Comm_agreement to leave that call (P543L45: "Advice to users. MPI_COMM_AGREEMENT maintains its collective behavior even > if the comm is invalidated. (End of advice to users.)" ) and MPI_Comm_agreement cannot return an error due to the call to MPI_Comm_invalidate (P545L38: "This function must not return an error due to process failure (error classes MPI_ERR_PROC_FAILED and MPI_ERR_INVALIDATED)...") . > > This would not work: > .... > do { > > if (root) read_some_data_from_a_file(buffer); > > err = MPI_Bcast(buffer, .... ,root, comm); > if (err) { > MPI_Comm_invalidate(comm); > MPI_Comm_shrink(comm, &newcomm); > MPI_Comm_free(&comm); > comm = newcomm; > } > > MPI_Comm_size(comm, size); > > done = do_computation(buffer, size); > > /* Let's agree for sure if we are done with the computation */ > > MPI_Barrier(comm); // don't check the error code, this is just to "catch" invalidate messages > MPI_Comm_agreement(comm, &done); > > while (!done); > > MPI_Finalize(); > > because a rank may enter the barrier, get knocked out by the call to invalidate and then go on to call MPI_Comm_agreement anyway. So we can try the following: > > do { > > if (root) read_some_data_from_a_file(buffer); > > err = MPI_Bcast(buffer, .... ,root, comm); > if (err) { > MPI_Comm_invalidate(comm); > MPI_Comm_shrink(comm, &newcomm); > MPI_Comm_free(&comm); > comm = newcomm; > } > > MPI_Comm_size(comm, size); > > done = do_computation(buffer, size); > > /* Let's agree for sure if we are done with the computation */ > > err = MPI_Barrier(comm); > if (err) { > MPI_Comm_invalidate(comm); > MPI_Comm_shrink(comm, &newcomm); > MPI_Comm_free(&comm); > comm = newcomm; > } > MPI_Comm_agreement(comm, &done); > > while (!done); > > MPI_Finalize(); > > But now we have done nothing more than move the problem down a few lines. Some ranks may succeed the MPI_Barrier and go on to MPI_Comm_agreement while others attempt to invalidate/shrink. Is there a solution to this problem? How can one safely use MPI_Comm_agreement and MPI_Comm_shrink in the same application? > > Thanks, > Dave _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Using MPI_Comm_shrink and MPI_Comm_agreement in the same application
by David Solt 05 Apr '12

05 Apr '12

I have another question about MPI_Comm_agreement: If I want to do this: .... do { if (root) read_some_data_from_a_file(buffer); err = MPI_Bcast(buffer, .... ,root, comm); if (err) { MPI_Comm_invalidate(comm); MPI_Comm_shrink(comm, &newcomm); MPI_Comm_free(&comm); comm = newcomm; } MPI_Comm_size(comm, size); done = do_computation(buffer, size); /* Let's agree for sure if we are done with the computation */ MPI_Comm_agreement(comm, &done); while (!done); MPI_Finalize(); This code can deadlock because some ranks may enter MPI_Comm_agreement while others detect an error in MPI_Bcast and call MPI_Comm_invalidate followed by MPI_Comm_shrink (assume that do_computation is really, really fast). The call to MPI_Comm_invalidate will not allow the processes that have already entered MPI_Comm_agreement to leave that call (P543L45: " Advice to users. MPI_COMM_AGREEMENT maintains its collective behavior even if the comm is invalidated. (End of advice to users.)" ) and MPI_Comm_agreement cannot return an error due to the call to MPI_Comm_invalidate (P545L38: "This function must not return an error due to process failure (error classes MPI_ERR_PROC_FAILED and MPI_ERR_ INVALIDATED)...") . This would not work: .... do { if (root) read_some_data_from_a_file(buffer); err = MPI_Bcast(buffer, .... ,root, comm); if (err) { MPI_Comm_invalidate(comm); MPI_Comm_shrink(comm, &newcomm); MPI_Comm_free(&comm); comm = newcomm; } MPI_Comm_size(comm, size); done = do_computation(buffer, size); /* Let's agree for sure if we are done with the computation */ MPI_Barrier(comm); // don't check the error code, this is just to "catch" invalidate messages MPI_Comm_agreement(comm, &done); while (!done); MPI_Finalize(); because a rank may enter the barrier, get knocked out by the call to invalidate and then go on to call MPI_Comm_agreement anyway. So we can try the following: do { if (root) read_some_data_from_a_file(buffer); err = MPI_Bcast(buffer, .... ,root, comm); if (err) { MPI_Comm_invalidate(comm); MPI_Comm_shrink(comm, &newcomm); MPI_Comm_free(&comm); comm = newcomm; } MPI_Comm_size(comm, size); done = do_computation(buffer, size); /* Let's agree for sure if we are done with the computation */ err = MPI_Barrier(comm); if (err) { MPI_Comm_invalidate(comm); MPI_Comm_shrink(comm, &newcomm); MPI_Comm_free(&comm); comm = newcomm; } MPI_Comm_agreement(comm, &done); while (!done); MPI_Finalize(); But now we have done nothing more than move the problem down a few lines. Some ranks may succeed the MPI_Barrier and go on to MPI_Comm_agreement while others attempt to invalidate/shrink. Is there a solution to this problem? How can one safely use MPI_Comm_agreement and MPI_Comm_shrink in the same application? Thanks, Dave

1 0

[Mpi3-ft] No Teleconf Today
by Josh Hursey 04 Apr '12

04 Apr '12

No teleconf today. We'll meet next Wednesday, April 11 at the usual time. Some things to think about/discuss before then: - New name for 'invalidate' - MPI_Comm_agreement -> MPI_Commm_agree - Ticket 336: Text for FT Query Interface -- Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0