April 2010 - mpi-forum - mailman3.cels.anl.gov

Re: [Mpi-forum] MPI_Abort - meaning
by Jeff Squyres 06 Apr '10

06 Apr '10

On Apr 5, 2010, at 4:42 PM, N.M. Maclaren wrote: > I agree with Richard's reservations, incidentally. About the best that > can be done is to specify an MPI_Quit that stops the calling process > cleanly (in the language sense) and has the effect of MPI_Abort on the > others. But, as I said, even that is tricky to implement on some systems > and may not be reliable. You mentioned this in an earlier post -- can you explain some issues? For example, you mention it might be difficult with systems that use sysv shared memory. Why? I ask with the *assumption* that the MPI implementation can/will have some kind of helper run-time system around that can do whatever cleanup is required (MPI-2 dynamics more-or-less require this anyway). Keep in mind that I'm not saying that the run-time system has to be part of the MPI implementation -- it may simply be provided on the system. For example: if sysv memory segments are left lying around because MPI processes are killed by the run-time system, a high-quality run-time system should be able to clean up the shared memory as well (or at least provide hooks such that MPI processes can tell the run-time system about resources that would need to be cleaned up in the event of untimely MPI process deaths, such as sysv shared memory segments). Is that a bad assumption? -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [Mpi-forum] MPI_Abort - meaning
by Richard Treumann 05 Apr '10

05 Apr '10

Maybe - the standard does not say anything specific about the meaning of MPI_Abort with a value of zero. The "best effort" part and the fact that the value passed to MPI_Abort is not required as a job RC both make MPI_Abort unsatisfactory as currently described. The name "Abort" also sounds like an undesired outcome. You do not abort something when you have gotten what you wanted. If the Forum chose to be explicit about the meaning of MPI_Abort with zero, it would answer the need but it would still feel sort of ugly to me. In part, I bring this up because now that it has been pointed out to me, I am a bit surprised that I have never come across the issue before. That made me wonder if wanting to end a parallel application this way is rare or if people have just been calling ABORT and being content it seemed to work OK. Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 From: Fab Tillier <ftillier(a)microsoft.com> To: Main MPI Forum mailing list <mpi-forum(a)lists.mpi-forum.org> Date: 04/05/2010 03:48 PM Subject: Re: [Mpi-forum] MPI_Abort - meaning Sent by: mpi-forum-bounces(a)lists.mpi-forum.org Richard Treumann wrote on Mon, 5 Apr 2010 at 10:45:00 > A call to MPI_Quit would let the application give a return code of 0 so > a script or wrapper that checks the mpiexec return code would regard the > application as a success. How is this different than calling MPI_Abort with an error code of 0? Is it just that MPI_Quit would do more than 'best effort'? -Fab _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum

1 0

Re: [Mpi-forum] MPI_Abort - meaning
by N.M. Maclaren 05 Apr '10

05 Apr '10

On Apr 5 2010, Fab Tillier wrote: >Richard Treumann wrote on Mon, 5 Apr 2010 at 10:45:00 > >> A call to MPI_Quit would let the application give a return code of 0 so >> a script or wrapper that checks the mpiexec return code would regard the >> application as a success. > >How is this different than calling MPI_Abort with an error code of 0? For example, MPI_Quit might be required to go through normal language- specified termination (and hence close files cleanly). MPI_Abort does not require that. I agree with Richard's reservations, incidentally. About the best that can be done is to specify an MPI_Quit that stops the calling process cleanly (in the language sense) and has the effect of MPI_Abort on the others. But, as I said, even that is tricky to implement on some systems and may not be reliable. Regards, Nick Maclaren.

1 0

Re: [Mpi-forum] MPI_Abort - meaning
by Fab Tillier 05 Apr '10

05 Apr '10

Richard Treumann wrote on Mon, 5 Apr 2010 at 10:45:00 > A call to MPI_Quit would let the application give a return code of 0 so > a script or wrapper that checks the mpiexec return code would regard the > application as a success. How is this different than calling MPI_Abort with an error code of 0? Is it just that MPI_Quit would do more than 'best effort'? -Fab

1 0

Re: [Mpi-forum] MPI_Abort - meaning
by Richard Treumann 05 Apr '10

05 Apr '10

I should add to my mention of MPI_Quit that I am not at all confident that a clean & portable semantic is possible. I am not suggesting anything that provides subset synchronizations or application notifications. No promise of message delivery among other tasks. No promise that other tasks reach a standards defined state before being ended. If MPI_Quit means anything more than a promise by libmpi that all tasks will be terminated and all resources recovered, it becomes a much more difficult problem. A call to MPI_Quit would let the application give a return code of 0 so a script or wrapper that checks the mpiexec return code would regard the application as a success. Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363

1 0

Re: [Mpi-forum] MPI_Abort - meaning
by N.M. Maclaren 05 Apr '10

05 Apr '10

On Apr 5 2010, Jeff Hammond wrote: > On Mon, Apr 5, 2010 at 10:18 AM, Richard Treumann <treumann(a)us.ibm.com> > wrote: >> It has come to my attention that there is at least one MPI user who >> considers an MPI_Abort call to be a legitimate way to terminate an >> application that has reached a correct result. I gather the situation is >> that as soon as any task has a result it decides is satisfactory, >> whatever the other tasks are working on becomes instantly irrelevant. >> >> Is this a situation other members of the Forum consider as common or at >> least legitimate? I can see the approach as legitimate but question the >> use of MPI_Abort. The approach is reasonable; using MPI_Abort for that is demented. >> Would you regard it as legitimate to consider an application that ended >> with a call to MPI_Abort to be successful? If so, should we say >> something explicit in the MPI_Abort description? With implementation and portability hats on, No Way, Jose! >> Should there be something new like MPI_Quit which is defined as a >> "correct" single task termination of an MPI application? Grrk. It could be done, but the consequences are non-trivial. For example, consider an implementation of MPI on a system like zOS. Or one that uses "System V shared memory" for communication. >> We currently say that MPI_Abort makes a "best attempt" which to me >> implies it is not guaranteed by the standard that it will really leave >> the system as good as new. I am not aware of anybody having a problem >> making MPI_Abort a total termination of processes and recovery of >> resources but the escape hatch is there for an MPI implementation that >> was unable to do a perfectly clean Abort. That's because the users report it to people like me, who tell them it's unavoidable, according to specification, and to stop using MPI_Abort that way. So the failures don't get reported to people like you. And, yes, I have had poe fail that way, as well as most other MPIs I have used or supported heavily. >That's actually a pretty good idea for a number of applications. I >wouldn't feel comfortable using a function called MPI_Abort for this, >but since MPI_Finalize is collective, any alternative is going to have >some unnecessary overhead. MPI needs either a new function call or >modification of MPI_Finalize to allow for asynchronous termination. I fully agree. >Can we have MPI_Quit take various arguments specifying termination >behavior such as NO_WAIT (a cleaner single-rank MPI_Abort) and >possibly other options which make sense in the context of MPI >endpoints and other threading uses? There might be times when one >would want to sync up the node or a subset of the processes before >terminating. Yes, but that is related to another deficiency of MPI (in the sense of a missing facility, not a fault). There isn't any way to enquire what the state of the local message queues are, even though it would be possible to specify cleanly. Without that, one would have to spoecify what synchronising means fairly carefully. It should be feasible. >If anyone thinks this usage of MPI_Abort is crazy, I'll pseudo-code up >something that would require it to perform efficiently. Never mind the efficiency - let's see something that is semantically well-defined and feasible to implement on any system that current MPI supports! Regards, Nick Maclaren.

1 0

Re: [Mpi-forum] MPI_Abort - meaning
by Jeff Hammond 05 Apr '10

05 Apr '10

Dick, That's actually a pretty good idea for a number of applications. I wouldn't feel comfortable using a function called MPI_Abort for this, but since MPI_Finalize is collective, any alternative is going to have some unnecessary overhead. MPI needs either a new function call or modification of MPI_Finalize to allow for asynchronous termination. Can we have MPI_Quit take various arguments specifying termination behavior such as NO_WAIT (a cleaner single-rank MPI_Abort) and possibly other options which make sense in the context of MPI endpoints and other threading uses? There might be times when one would want to sync up the node or a subset of the processes before terminating. For example, a multi-level parallel implementation of an algorithm which motivates the use of MPI_Abort would want to terminate MPI as soon as a collection of processes collectively reached the termination condition but not wait for any other subsets of processes. I guess what I'm saying is that MPI_Quit needs to be sufficiently general to support all uses of the type you describe, not just a single rank dumping the job. If anyone thinks this usage of MPI_Abort is crazy, I'll pseudo-code up something that would require it to perform efficiently. Best, Jeff On Mon, Apr 5, 2010 at 10:18 AM, Richard Treumann <treumann(a)us.ibm.com> wrote: > It has come to my attention that there is at least one MPI user who > considers an MPI_Abort call to be a legitimate way to terminate an > application that has reached a correct result. I gather the situation is > that as soon as any task has a result it decides is satisfactory, whatever > the other tasks are working on becomes instantly irrelevant. > > Is this a situation other members of the Forum consider as common or at > least legitimate? I can see the approach as legitimate but question the use > of MPI_Abort. > > Would you regard it as legitimate to consider an application that ended with > a call to MPI_Abort to be successful? If so, should we say something > explicit in the MPI_Abort description? > > Should there be something new like MPI_Quit which is defined as a "correct" > single task termination of an MPI application? > > We currently say that MPI_Abort makes a "best attempt" which to me implies > it is not guaranteed by the standard that it will really leave the system as > good as new. I am not aware of anybody having a problem making MPI_Abort a > total termination of processes and recovery of resources but the escape > hatch is there for an MPI implementation that was unable to do a perfectly > clean Abort. > > > Dick > > > Dick Treumann - MPI Team > IBM Systems & Technology Group > Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 > Tele (845) 433-7846 Fax (845) 433-8363 > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > > -- Jeff Hammond Argonne Leadership Computing Facility jhammond(a)mcs.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond

1 0

MPI_Abort - meaning
by Richard Treumann 05 Apr '10

05 Apr '10

It has come to my attention that there is at least one MPI user who considers an MPI_Abort call to be a legitimate way to terminate an application that has reached a correct result. I gather the situation is that as soon as any task has a result it decides is satisfactory, whatever the other tasks are working on becomes instantly irrelevant. Is this a situation other members of the Forum consider as common or at least legitimate? I can see the approach as legitimate but question the use of MPI_Abort. Would you regard it as legitimate to consider an application that ended with a call to MPI_Abort to be successful? If so, should we say something explicit in the MPI_Abort description? Should there be something new like MPI_Quit which is defined as a "correct" single task termination of an MPI application? We currently say that MPI_Abort makes a "best attempt" which to me implies it is not guaranteed by the standard that it will really leave the system as good as new. I am not aware of anybody having a problem making MPI_Abort a total termination of processes and recovery of resources but the escape hatch is there for an MPI implementation that was unable to do a perfectly clean Abort. Dick Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363

1 0