October 2011 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] MPI_ANY_SOURCE
by Josh Hursey 13 Oct '11

13 Oct '11

yeah you're right. We only need to lock around the wait for the nonblocking case. One thing that slipped my mind was that MPI_Irecv will return regardless of the state of other processes - since it is nonblocking. So eventhough it is ANY_SOURCE, it will still return MPI_SUCCESS once a request is generated. The request might be marked as failed, but the call to MPI_Irecv will not return the error, the subsequent MPI_Wait will return the error. So it is only really necessary to lock around the Wait. To tweak your example a bit: -------------------------- /* Assume the user will cancel the request outside of this function, * so we do not need to cancel it for them inside the function. */ int My_AS_MPI_Wait(MPI_Request *req, MPI_Status *status) { while(1) { reader_lock(); if (my_cnt != recognize_cnt) { /* New failures were detected */ /* check failed_group and decide if ok to continue */ if (ok_to_continue(req, failed_group) == FALSE) { reader_unlock(); /* caller responsible for canceling the request */ return MPI_ERR_PROC_FAIL_STOP; } my_cnt == recognize_cnt; } err = MPI_Wait(req, status); if (err == MPI_WARN_PROC_FAIL_STOP) { /* Failure case */ reader_unlock(); writer_lock(); if (my_cnt != recognize_cnt) { /* another thread has already re-enabled wildcards */ writer_unlock(); continue; } MPI_Comm_reenable_any_source(comm, &failed_group); ++recognize_cnt; writer_unlock(); continue; } else if (MPI_SUCCESS != err) { reader_unlock(); /* caller responsible for canceling the request */ return MPI_ERR_PROC_FAIL_STOP; } /* Successful completion of the MPI_Wait */ reader_unlock(); return MPI_SUCCESS; } } -------------------------- On Tue, Oct 11, 2011 at 4:16 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: > > I think you're right Josh, the blocking version wouldn't need to be changed. > > For the nonblocking version, wouldn't we only need to lock around the Wait, not between the Recv and Wait? ï¿½If we're worried about hanging in a blocking Wait, I think we just need to check for all-clients-failed before calling Wait. ï¿½If anysources are reenabled by another thread before this thread calls Wait, that's OK, so long as the thread checks first. > > Here's a function a user could implement to use whenever waiting on an anysource: > > int My_AS_MPI_Wait(MPI_Request *req, MPI_Status *status) > { > ï¿½ ï¿½while(1) { > ï¿½ ï¿½ ï¿½ ï¿½reader_lock(); > ï¿½ ï¿½ ï¿½ ï¿½if (my_cnt != recognize_cnt) { > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½/* New failures were detected */ > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½/* check failed_group and decide if ok to continue */ > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½if (ok_to_continue(req, failed_group) == FALSE) { > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½reader_unlock(); > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½return MPI_ERR_PROC_FAIL_STOP; > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½} > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½my_cnt == recognize_cnt; > ï¿½ ï¿½ ï¿½ ï¿½} > ï¿½ ï¿½ ï¿½ ï¿½err = MPI_Wait(req, status); > ï¿½ ï¿½ ï¿½ ï¿½if (err == MPI_WARN_PROC_FAIL_STOP) { > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½/* Failure case */ > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½reader_unlock(); > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½writer_lock(); > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½if (my_cnt != recognize_cnt) { > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½/* another thread has already re-enabled wildcards */ > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½writer_unlock(); > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½continue; > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½} > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½MPI_Comm_reenable_any_source(comm, &failed_group); > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½++recognize_cnt; > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½writer_unlock(); > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½continue; > ï¿½ ï¿½ ï¿½ ï¿½} else { > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½reader_unlock(); > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½return MPI_ERR_PROC_FAIL_STOP; > ï¿½ ï¿½ ï¿½ ï¿½} > ï¿½ ï¿½ ï¿½ ï¿½reader_unlock(); > ï¿½ ï¿½} > } > > -d > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[Mpi3-ft] Teleconf yesterday
by Josh Hursey 13 Oct '11

13 Oct '11

Sorry I did not have the chance to send around the teleconf announcement yesterday. Did you all meet anyway? Our next teleconf is next Wednesday. On or before that teleconf we should probably try to finish the MPI_ANY_SOURCE semantics, and work it into the PDF - so we can circulate it before the MPI Forum meeting. I mentioned the Warning concept to some developer/apps folks over the past couple of days, and they seemed ok with it. Thanks, Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Eventually Perfect Failure Detector
by Josh Hursey 13 Oct '11

13 Oct '11

We still have the part about excluding/terminating the mistakenly identified process. We say that the process is excluded from MPI, and the MPI should try to terminate it (we cannot require termination since some environment cannot support it). In the advice to users we have a line stating: ----------- Additionally, if a process was mistakenly reported as failed it is possible that for some period of time a subset of processes interact with the process normally, while others see it as failed. Eventually all processes in the MPI universe will become aware of the process failure. ------------ Do you think we should extend this a bit to include a warning about the process participating via a communication mechanism outside of MPI? Can you suggest some wording? Thanks, Josh On Wed, Oct 12, 2011 at 8:19 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: > Hi Adam, > > I believe we have the part about killing a transient process, but I'm not sure whether the part about external interaction with the process. We should make sure that's in there. > > -d > > > On Oct 12, 2011, at 7:12 PM, Adam T. Moody wrote: > >> Hi Josh, >> At one time, I think we had text also suggesting to implementors to try to kill a transient process that is marked as failed. And I think we had a note to warn users that even though a process is marked as failed w/ respect to MPI, it may not actually be dead, and applications should expect that the process may be interacting with the job through external mechanisms, e.g., a common file system or sockets. Are these items still listed somewhere? >> -Adam >> >> Aurélien Bouteiller wrote: >> >>> Josh, >>> I think this is a much better position to consider eventually perfect. I'm happy to see that these discussions we had have been fruitful. >>> Aurelien >>> Le 1 sept. 2011 ą 11:34, Josh Hursey a écrit : >>> >>> >>>> So we have been working under a modified definition of a 'Perfect >>>> Failure Detector' for a while. Recently we have been encouraged to >>>> revisit and clarify this assumption and definition (1) to make it >>>> clearer without the academic jargon (2) to be more precise as far as >>>> the detector we actually semantically require. >>>> >>>> It should be noted that a truly Perfect Detector can only be >>>> implemented in a synchronous system, which makes it an unsafe >>>> assumption for the MPI standard. In the RTS proposal we weakened the >>>> definition slightly to handle transient failures (an therefor a nod to >>>> partially synchronous systems), but we still talk about it as a >>>> Perfect Detector which has been getting us in trouble. >>>> >>>> After a few conversations with folks and some additional (re-)reading >>>> I started to reformulate the text in the RTS proposal. At bottom I >>>> included the new text that I am suggesting to replace the 'Perfect >>>> Detector' language in the RTS proposal. >>>> >>>> At its core is the movement to what is called an 'Eventually Perfect' >>>> failure detector with a terminate-upon-mistake constraint for >>>> mistakenly identified failed processes (previously we called them >>>> transitent failures). So to the local process the failure detector >>>> seems perfect, but in the system as a whole it is only eventually so. >>>> >>>> The front part of the text below is meant to be the clearer language, >>>> and I pushed the more precise stuff into the rationale. >>>> >>>> Take a read through the text below and let me know what you think. I >>>> suspect it will require some iteration to get just right, but >>>> hopefully it starts us out on better footing going forward. >>>> >>>> Thanks, >>>> Josh >>>> >>>> >>>> --------------------------------------------- >>>> 17.2 MPI Terms and Conventions >>>> >>>> Fail-Stop process failure is one in which the process stops executing, >>>> and its internal state is lost [5]. >>>> >>>> 17.3 Detection of Process Failure >>>> >>>> MPI will provide the ability to detect process failures. MPI will >>>> guarantee that eventually all alive processes will be able to know >>>> about the failure. The state management and query operations defined >>>> in this chapter allow the application to query for the failed set of >>>> processes in a communication group. Additional semantics regarding >>>> communication involving failed processes are defined later in this >>>> chapter. >>>> >>>> It is possible that MPI mistakenly identifies a process as failed when >>>> it is not failed. In this situation the MPI library will exclude the >>>> mistakenly identified failed process from the MPI universe, and >>>> eventually all alive processes will see this process as failed. >>>> >>>> ------------------ >>>> Rationale. >>>> MPI provides an eventually perfect failure detector for fail-stop >>>> process failures [1]. An eventually perfect failure detector is both >>>> strongly complete and eventually strongly accurate. >>>> >>>> Strong completeness is defined as: "Eventually every process that >>>> crashes is permanently suspected by every correct process" [1]. In >>>> essence this means that eventually every failed process will be able >>>> to be known to all alive processes. Without strong completeness some >>>> alive process that depends on a failed process is not guaranteed to >>>> ever receive an error, which can lead to a deadlock. >>>> >>>> Eventual strong accuracy is defined as: "There is a time after which >>>> correct processes are not suspected by any correct process" [1]. >>>> Depending on the system architecture, it may be impossible to >>>> correctly determine if a process is failed or slow [3]. Eventual >>>> strong accuracy allows for unreliable failure detectors that may >>>> mistakenly suspect a process as failed when it is not failed [1]. >>>> >>>> If a process failure was reported to the application and the process >>>> is later found to be alive then MPI will exclude the process from the >>>> MPI universe. Resolving the mistake by excluding the process from the >>>> MPI universe is similar to the technique used by the group membership >>>> protocol in [4]. This additional constraint allows for consistent >>>> reporting of error states to the local process. Without this >>>> constraint the application would not be able to trust the MPI >>>> implementation when it reports process failure errors. Once an alive >>>> process receives notification of a failed peer process, then it may >>>> continue under the assumption that the process is failed. >>>> >>>> End of rationale. >>>> ------------------ >>>> >>>> ------------------ >>>> Advice to users. >>>> The strong completeness condition of the failure detector allows the >>>> MPI implementation some flexibility in managing the performance costs >>>> involved with process failure detection and notification. As such, it >>>> is possible that some alive processes in the MPI universe know of >>>> process failures that other alive processes do not for a period of >>>> time. Additionally, if a process was mistakenly reported as failed it >>>> is possible that for some period of time a subset of processes >>>> interact with the process normally, while others see it as failed. >>>> Eventually all processes in the MPI universe will be able to be aware >>>> of the process failure. >>>> >>>> End of advice to users. >>>> ------------------ >>>> >>>> ------------------ >>>> Advice to implementors. >>>> An MPI implementation may choose to provide a stronger failure >>>> detector (i.e., perfect failure detector), but is not required to do >>>> so. This may be possible for MPI implementations targeted at >>>> synchronous systems [2]. >>>> >>>> End of advice to implementors. >>>> ------------------ >>>> >>>> Citations: >>>> ---------- >>>> [1] Chandra, T. and Toueg, S. Unreliable Failure Detectors for >>>> Reliable Distributed Systems. Journal of the ACM (1996). >>>> [2] Dwork, C., Lynch, N., and Stockmeyer, L. Consensus in the Presence >>>> of Partial Synchrony. Journal of the ACM (1988). >>>> [3] Fischer, M., Lynch, N., and Paterson, M. Impossibility of >>>> distributed consensus with one faulty process. Journal of the ACM >>>> (1985). >>>> [4] Ricciardi, A. and Birman, K. Using process groups to implement >>>> failure detection in asynchronous environments. In Proceedings of the >>>> Eleventh ACM Symposium on Principles of Distributed Computing (1991). >>>> [5] Schlichting, R.D. and Schneider, F.B. Fail-stop processors: An >>>> approach to designing fault-tolerant computing systems. ACM >>>> Transactions on Computer Systems (1983). >>>> >>>> --------------------------------------------- >>>> >>>> -- >>>> Joshua Hursey >>>> Postdoctoral Research Associate >>>> Oak Ridge National Laboratory >>>> http://users.nccs.gov/~jjhursey >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] Eventually Perfect Failure Detector
by Darius Buntinas 12 Oct '11

12 Oct '11

Hi Adam, I believe we have the part about killing a transient process, but I'm not sure whether the part about external interaction with the process. We should make sure that's in there. -d On Oct 12, 2011, at 7:12 PM, Adam T. Moody wrote: > Hi Josh, > At one time, I think we had text also suggesting to implementors to try to kill a transient process that is marked as failed. And I think we had a note to warn users that even though a process is marked as failed w/ respect to MPI, it may not actually be dead, and applications should expect that the process may be interacting with the job through external mechanisms, e.g., a common file system or sockets. Are these items still listed somewhere? > -Adam > > Aurï¿½lien Bouteiller wrote: > >> Josh, >> I think this is a much better position to consider eventually perfect. I'm happy to see that these discussions we had have been fruitful. >> Aurelien >> Le 1 sept. 2011 ï¿½ 11:34, Josh Hursey a ï¿½crit : >> >> >>> So we have been working under a modified definition of a 'Perfect >>> Failure Detector' for a while. Recently we have been encouraged to >>> revisit and clarify this assumption and definition (1) to make it >>> clearer without the academic jargon (2) to be more precise as far as >>> the detector we actually semantically require. >>> >>> It should be noted that a truly Perfect Detector can only be >>> implemented in a synchronous system, which makes it an unsafe >>> assumption for the MPI standard. In the RTS proposal we weakened the >>> definition slightly to handle transient failures (an therefor a nod to >>> partially synchronous systems), but we still talk about it as a >>> Perfect Detector which has been getting us in trouble. >>> >>> After a few conversations with folks and some additional (re-)reading >>> I started to reformulate the text in the RTS proposal. At bottom I >>> included the new text that I am suggesting to replace the 'Perfect >>> Detector' language in the RTS proposal. >>> >>> At its core is the movement to what is called an 'Eventually Perfect' >>> failure detector with a terminate-upon-mistake constraint for >>> mistakenly identified failed processes (previously we called them >>> transitent failures). So to the local process the failure detector >>> seems perfect, but in the system as a whole it is only eventually so. >>> >>> The front part of the text below is meant to be the clearer language, >>> and I pushed the more precise stuff into the rationale. >>> >>> Take a read through the text below and let me know what you think. I >>> suspect it will require some iteration to get just right, but >>> hopefully it starts us out on better footing going forward. >>> >>> Thanks, >>> Josh >>> >>> >>> --------------------------------------------- >>> 17.2 MPI Terms and Conventions >>> >>> Fail-Stop process failure is one in which the process stops executing, >>> and its internal state is lost [5]. >>> >>> 17.3 Detection of Process Failure >>> >>> MPI will provide the ability to detect process failures. MPI will >>> guarantee that eventually all alive processes will be able to know >>> about the failure. The state management and query operations defined >>> in this chapter allow the application to query for the failed set of >>> processes in a communication group. Additional semantics regarding >>> communication involving failed processes are defined later in this >>> chapter. >>> >>> It is possible that MPI mistakenly identifies a process as failed when >>> it is not failed. In this situation the MPI library will exclude the >>> mistakenly identified failed process from the MPI universe, and >>> eventually all alive processes will see this process as failed. >>> >>> ------------------ >>> Rationale. >>> MPI provides an eventually perfect failure detector for fail-stop >>> process failures [1]. An eventually perfect failure detector is both >>> strongly complete and eventually strongly accurate. >>> >>> Strong completeness is defined as: "Eventually every process that >>> crashes is permanently suspected by every correct process" [1]. In >>> essence this means that eventually every failed process will be able >>> to be known to all alive processes. Without strong completeness some >>> alive process that depends on a failed process is not guaranteed to >>> ever receive an error, which can lead to a deadlock. >>> >>> Eventual strong accuracy is defined as: "There is a time after which >>> correct processes are not suspected by any correct process" [1]. >>> Depending on the system architecture, it may be impossible to >>> correctly determine if a process is failed or slow [3]. Eventual >>> strong accuracy allows for unreliable failure detectors that may >>> mistakenly suspect a process as failed when it is not failed [1]. >>> >>> If a process failure was reported to the application and the process >>> is later found to be alive then MPI will exclude the process from the >>> MPI universe. Resolving the mistake by excluding the process from the >>> MPI universe is similar to the technique used by the group membership >>> protocol in [4]. This additional constraint allows for consistent >>> reporting of error states to the local process. Without this >>> constraint the application would not be able to trust the MPI >>> implementation when it reports process failure errors. Once an alive >>> process receives notification of a failed peer process, then it may >>> continue under the assumption that the process is failed. >>> >>> End of rationale. >>> ------------------ >>> >>> ------------------ >>> Advice to users. >>> The strong completeness condition of the failure detector allows the >>> MPI implementation some flexibility in managing the performance costs >>> involved with process failure detection and notification. As such, it >>> is possible that some alive processes in the MPI universe know of >>> process failures that other alive processes do not for a period of >>> time. Additionally, if a process was mistakenly reported as failed it >>> is possible that for some period of time a subset of processes >>> interact with the process normally, while others see it as failed. >>> Eventually all processes in the MPI universe will be able to be aware >>> of the process failure. >>> >>> End of advice to users. >>> ------------------ >>> >>> ------------------ >>> Advice to implementors. >>> An MPI implementation may choose to provide a stronger failure >>> detector (i.e., perfect failure detector), but is not required to do >>> so. This may be possible for MPI implementations targeted at >>> synchronous systems [2]. >>> >>> End of advice to implementors. >>> ------------------ >>> >>> Citations: >>> ---------- >>> [1] Chandra, T. and Toueg, S. Unreliable Failure Detectors for >>> Reliable Distributed Systems. Journal of the ACM (1996). >>> [2] Dwork, C., Lynch, N., and Stockmeyer, L. Consensus in the Presence >>> of Partial Synchrony. Journal of the ACM (1988). >>> [3] Fischer, M., Lynch, N., and Paterson, M. Impossibility of >>> distributed consensus with one faulty process. Journal of the ACM >>> (1985). >>> [4] Ricciardi, A. and Birman, K. Using process groups to implement >>> failure detection in asynchronous environments. In Proceedings of the >>> Eleventh ACM Symposium on Principles of Distributed Computing (1991). >>> [5] Schlichting, R.D. and Schneider, F.B. Fail-stop processors: An >>> approach to designing fault-tolerant computing systems. ACM >>> Transactions on Computer Systems (1983). >>> >>> --------------------------------------------- >>> >>> -- >>> Joshua Hursey >>> Postdoctoral Research Associate >>> Oak Ridge National Laboratory >>> http://users.nccs.gov/~jjhursey >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] Eventually Perfect Failure Detector
by Adam T. Moody 12 Oct '11

12 Oct '11

Hi Josh, At one time, I think we had text also suggesting to implementors to try to kill a transient process that is marked as failed. And I think we had a note to warn users that even though a process is marked as failed w/ respect to MPI, it may not actually be dead, and applications should expect that the process may be interacting with the job through external mechanisms, e.g., a common file system or sockets. Are these items still listed somewhere? -Adam Aurï¿½lien Bouteiller wrote: >Josh, > >I think this is a much better position to consider eventually perfect. I'm happy to see that these discussions we had have been fruitful. > >Aurelien > >Le 1 sept. 2011 ï¿½ 11:34, Josh Hursey a ï¿½crit : > > > >>So we have been working under a modified definition of a 'Perfect >>Failure Detector' for a while. Recently we have been encouraged to >>revisit and clarify this assumption and definition (1) to make it >>clearer without the academic jargon (2) to be more precise as far as >>the detector we actually semantically require. >> >>It should be noted that a truly Perfect Detector can only be >>implemented in a synchronous system, which makes it an unsafe >>assumption for the MPI standard. In the RTS proposal we weakened the >>definition slightly to handle transient failures (an therefor a nod to >>partially synchronous systems), but we still talk about it as a >>Perfect Detector which has been getting us in trouble. >> >>After a few conversations with folks and some additional (re-)reading >>I started to reformulate the text in the RTS proposal. At bottom I >>included the new text that I am suggesting to replace the 'Perfect >>Detector' language in the RTS proposal. >> >>At its core is the movement to what is called an 'Eventually Perfect' >>failure detector with a terminate-upon-mistake constraint for >>mistakenly identified failed processes (previously we called them >>transitent failures). So to the local process the failure detector >>seems perfect, but in the system as a whole it is only eventually so. >> >>The front part of the text below is meant to be the clearer language, >>and I pushed the more precise stuff into the rationale. >> >>Take a read through the text below and let me know what you think. I >>suspect it will require some iteration to get just right, but >>hopefully it starts us out on better footing going forward. >> >>Thanks, >>Josh >> >> >>--------------------------------------------- >>17.2 MPI Terms and Conventions >> >>Fail-Stop process failure is one in which the process stops executing, >>and its internal state is lost [5]. >> >>17.3 Detection of Process Failure >> >>MPI will provide the ability to detect process failures. MPI will >>guarantee that eventually all alive processes will be able to know >>about the failure. The state management and query operations defined >>in this chapter allow the application to query for the failed set of >>processes in a communication group. Additional semantics regarding >>communication involving failed processes are defined later in this >>chapter. >> >>It is possible that MPI mistakenly identifies a process as failed when >>it is not failed. In this situation the MPI library will exclude the >>mistakenly identified failed process from the MPI universe, and >>eventually all alive processes will see this process as failed. >> >>------------------ >>Rationale. >>MPI provides an eventually perfect failure detector for fail-stop >>process failures [1]. An eventually perfect failure detector is both >>strongly complete and eventually strongly accurate. >> >>Strong completeness is defined as: "Eventually every process that >>crashes is permanently suspected by every correct process" [1]. In >>essence this means that eventually every failed process will be able >>to be known to all alive processes. Without strong completeness some >>alive process that depends on a failed process is not guaranteed to >>ever receive an error, which can lead to a deadlock. >> >>Eventual strong accuracy is defined as: "There is a time after which >>correct processes are not suspected by any correct process" [1]. >>Depending on the system architecture, it may be impossible to >>correctly determine if a process is failed or slow [3]. Eventual >>strong accuracy allows for unreliable failure detectors that may >>mistakenly suspect a process as failed when it is not failed [1]. >> >>If a process failure was reported to the application and the process >>is later found to be alive then MPI will exclude the process from the >>MPI universe. Resolving the mistake by excluding the process from the >>MPI universe is similar to the technique used by the group membership >>protocol in [4]. This additional constraint allows for consistent >>reporting of error states to the local process. Without this >>constraint the application would not be able to trust the MPI >>implementation when it reports process failure errors. Once an alive >>process receives notification of a failed peer process, then it may >>continue under the assumption that the process is failed. >> >>End of rationale. >>------------------ >> >>------------------ >>Advice to users. >>The strong completeness condition of the failure detector allows the >>MPI implementation some flexibility in managing the performance costs >>involved with process failure detection and notification. As such, it >>is possible that some alive processes in the MPI universe know of >>process failures that other alive processes do not for a period of >>time. Additionally, if a process was mistakenly reported as failed it >>is possible that for some period of time a subset of processes >>interact with the process normally, while others see it as failed. >>Eventually all processes in the MPI universe will be able to be aware >>of the process failure. >> >>End of advice to users. >>------------------ >> >>------------------ >>Advice to implementors. >>An MPI implementation may choose to provide a stronger failure >>detector (i.e., perfect failure detector), but is not required to do >>so. This may be possible for MPI implementations targeted at >>synchronous systems [2]. >> >>End of advice to implementors. >>------------------ >> >>Citations: >>---------- >>[1] Chandra, T. and Toueg, S. Unreliable Failure Detectors for >>Reliable Distributed Systems. Journal of the ACM (1996). >>[2] Dwork, C., Lynch, N., and Stockmeyer, L. Consensus in the Presence >>of Partial Synchrony. Journal of the ACM (1988). >>[3] Fischer, M., Lynch, N., and Paterson, M. Impossibility of >>distributed consensus with one faulty process. Journal of the ACM >>(1985). >>[4] Ricciardi, A. and Birman, K. Using process groups to implement >>failure detection in asynchronous environments. In Proceedings of the >>Eleventh ACM Symposium on Principles of Distributed Computing (1991). >>[5] Schlichting, R.D. and Schneider, F.B. Fail-stop processors: An >>approach to designing fault-tolerant computing systems. ACM >>Transactions on Computer Systems (1983). >> >>--------------------------------------------- >> >>-- >>Joshua Hursey >>Postdoctoral Research Associate >>Oak Ridge National Laboratory >>http://users.nccs.gov/~jjhursey >>_______________________________________________ >>mpi3-ft mailing list >>mpi3-ft(a)lists.mpi-forum.org >>http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> > > >_______________________________________________ >mpi3-ft mailing list >mpi3-ft(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >

1 0

Re: [Mpi3-ft] Eventually Perfect Failure Detector
by Aurélien Bouteiller 12 Oct '11

12 Oct '11

Josh, I think this is a much better position to consider eventually perfect. I'm happy to see that these discussions we had have been fruitful. Aurelien Le 1 sept. 2011 ï¿½ 11:34, Josh Hursey a ï¿½crit : > So we have been working under a modified definition of a 'Perfect > Failure Detector' for a while. Recently we have been encouraged to > revisit and clarify this assumption and definition (1) to make it > clearer without the academic jargon (2) to be more precise as far as > the detector we actually semantically require. > > It should be noted that a truly Perfect Detector can only be > implemented in a synchronous system, which makes it an unsafe > assumption for the MPI standard. In the RTS proposal we weakened the > definition slightly to handle transient failures (an therefor a nod to > partially synchronous systems), but we still talk about it as a > Perfect Detector which has been getting us in trouble. > > After a few conversations with folks and some additional (re-)reading > I started to reformulate the text in the RTS proposal. At bottom I > included the new text that I am suggesting to replace the 'Perfect > Detector' language in the RTS proposal. > > At its core is the movement to what is called an 'Eventually Perfect' > failure detector with a terminate-upon-mistake constraint for > mistakenly identified failed processes (previously we called them > transitent failures). So to the local process the failure detector > seems perfect, but in the system as a whole it is only eventually so. > > The front part of the text below is meant to be the clearer language, > and I pushed the more precise stuff into the rationale. > > Take a read through the text below and let me know what you think. I > suspect it will require some iteration to get just right, but > hopefully it starts us out on better footing going forward. > > Thanks, > Josh > > > --------------------------------------------- > 17.2 MPI Terms and Conventions > > Fail-Stop process failure is one in which the process stops executing, > and its internal state is lost [5]. > > 17.3 Detection of Process Failure > > MPI will provide the ability to detect process failures. MPI will > guarantee that eventually all alive processes will be able to know > about the failure. The state management and query operations defined > in this chapter allow the application to query for the failed set of > processes in a communication group. Additional semantics regarding > communication involving failed processes are defined later in this > chapter. > > It is possible that MPI mistakenly identifies a process as failed when > it is not failed. In this situation the MPI library will exclude the > mistakenly identified failed process from the MPI universe, and > eventually all alive processes will see this process as failed. > > ------------------ > Rationale. > MPI provides an eventually perfect failure detector for fail-stop > process failures [1]. An eventually perfect failure detector is both > strongly complete and eventually strongly accurate. > > Strong completeness is defined as: "Eventually every process that > crashes is permanently suspected by every correct process" [1]. In > essence this means that eventually every failed process will be able > to be known to all alive processes. Without strong completeness some > alive process that depends on a failed process is not guaranteed to > ever receive an error, which can lead to a deadlock. > > Eventual strong accuracy is defined as: "There is a time after which > correct processes are not suspected by any correct process" [1]. > Depending on the system architecture, it may be impossible to > correctly determine if a process is failed or slow [3]. Eventual > strong accuracy allows for unreliable failure detectors that may > mistakenly suspect a process as failed when it is not failed [1]. > > If a process failure was reported to the application and the process > is later found to be alive then MPI will exclude the process from the > MPI universe. Resolving the mistake by excluding the process from the > MPI universe is similar to the technique used by the group membership > protocol in [4]. This additional constraint allows for consistent > reporting of error states to the local process. Without this > constraint the application would not be able to trust the MPI > implementation when it reports process failure errors. Once an alive > process receives notification of a failed peer process, then it may > continue under the assumption that the process is failed. > > End of rationale. > ------------------ > > ------------------ > Advice to users. > The strong completeness condition of the failure detector allows the > MPI implementation some flexibility in managing the performance costs > involved with process failure detection and notification. As such, it > is possible that some alive processes in the MPI universe know of > process failures that other alive processes do not for a period of > time. Additionally, if a process was mistakenly reported as failed it > is possible that for some period of time a subset of processes > interact with the process normally, while others see it as failed. > Eventually all processes in the MPI universe will be able to be aware > of the process failure. > > End of advice to users. > ------------------ > > ------------------ > Advice to implementors. > An MPI implementation may choose to provide a stronger failure > detector (i.e., perfect failure detector), but is not required to do > so. This may be possible for MPI implementations targeted at > synchronous systems [2]. > > End of advice to implementors. > ------------------ > > Citations: > ---------- > [1] Chandra, T. and Toueg, S. Unreliable Failure Detectors for > Reliable Distributed Systems. Journal of the ACM (1996). > [2] Dwork, C., Lynch, N., and Stockmeyer, L. Consensus in the Presence > of Partial Synchrony. Journal of the ACM (1988). > [3] Fischer, M., Lynch, N., and Paterson, M. Impossibility of > distributed consensus with one faulty process. Journal of the ACM > (1985). > [4] Ricciardi, A. and Birman, K. Using process groups to implement > failure detection in asynchronous environments. In Proceedings of the > Eleventh ACM Symposium on Principles of Distributed Computing (1991). > [5] Schlichting, R.D. and Schneider, F.B. Fail-stop processors: An > approach to designing fault-tolerant computing systems. ACM > Transactions on Computer Systems (1983). > > --------------------------------------------- > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [Mpi3-ft] MPI_ANY_SOURCE
by Darius Buntinas 11 Oct '11

11 Oct '11

I think we can do this without a warning, just by using an error: If a wait (or wait_all, test, etc) is called with an anysource request while the req's communicator is anysource disabled, then the function will return an error (maybe something like MPI_ERR_ANYSOURCE_DISABLED). Of course in the wait_all case the app needs to be able to determine which req is the anysource on a disabled communicator. We may be able to help the user here by using the status array. -d On Oct 11, 2011, at 3:16 PM, Darius Buntinas wrote: > > I think you're right Josh, the blocking version wouldn't need to be changed. > > For the nonblocking version, wouldn't we only need to lock around the Wait, not between the Recv and Wait? If we're worried about hanging in a blocking Wait, I think we just need to check for all-clients-failed before calling Wait. If anysources are reenabled by another thread before this thread calls Wait, that's OK, so long as the thread checks first. > > Here's a function a user could implement to use whenever waiting on an anysource: > > int My_AS_MPI_Wait(MPI_Request *req, MPI_Status *status) > { > while(1) { > reader_lock(); > if (my_cnt != recognize_cnt) { > /* New failures were detected */ > /* check failed_group and decide if ok to continue */ > if (ok_to_continue(req, failed_group) == FALSE) { > reader_unlock(); > return MPI_ERR_PROC_FAIL_STOP; > } > my_cnt == recognize_cnt; > } > err = MPI_Wait(req, status); > if (err == MPI_WARN_PROC_FAIL_STOP) { > /* Failure case */ > reader_unlock(); > writer_lock(); > if (my_cnt != recognize_cnt) { > /* another thread has already re-enabled wildcards */ > writer_unlock(); > continue; > } > MPI_Comm_reenable_any_source(comm, &failed_group); > ++recognize_cnt; > writer_unlock(); > continue; > } else { > reader_unlock(); > return MPI_ERR_PROC_FAIL_STOP; > } > reader_unlock(); > } > } > > -d >

1 0

Re: [Mpi3-ft] MPI_ANY_SOURCE
by Darius Buntinas 11 Oct '11

11 Oct '11

I think you're right Josh, the blocking version wouldn't need to be changed. For the nonblocking version, wouldn't we only need to lock around the Wait, not between the Recv and Wait? If we're worried about hanging in a blocking Wait, I think we just need to check for all-clients-failed before calling Wait. If anysources are reenabled by another thread before this thread calls Wait, that's OK, so long as the thread checks first. Here's a function a user could implement to use whenever waiting on an anysource: int My_AS_MPI_Wait(MPI_Request *req, MPI_Status *status) { while(1) { reader_lock(); if (my_cnt != recognize_cnt) { /* New failures were detected */ /* check failed_group and decide if ok to continue */ if (ok_to_continue(req, failed_group) == FALSE) { reader_unlock(); return MPI_ERR_PROC_FAIL_STOP; } my_cnt == recognize_cnt; } err = MPI_Wait(req, status); if (err == MPI_WARN_PROC_FAIL_STOP) { /* Failure case */ reader_unlock(); writer_lock(); if (my_cnt != recognize_cnt) { /* another thread has already re-enabled wildcards */ writer_unlock(); continue; } MPI_Comm_reenable_any_source(comm, &failed_group); ++recognize_cnt; writer_unlock(); continue; } else { reader_unlock(); return MPI_ERR_PROC_FAIL_STOP; } reader_unlock(); } } -d

1 0

Re: [Mpi3-ft] MPI_ANY_SOURCE
by Josh Hursey 06 Oct '11

06 Oct '11

The blocking MPI_Recv(ANY_SOURCE) example uses readers/writers locks: ------------------------------ while(!done) { reader_lock(); if (my_cnt != recognize_cnt) { /* New failures were detected */ /* check failed_group and decide if ok to continue */ if (ok_to_continue(failed_group) == FALSE) { reader_unlock(); break; } my_cnt == recognize_cnt; } err = MPI_Recv(..., MPI_ANY_SOURCE, ..., comm, ...); if (err == MPI_ERR_PROC_FAIL_STOP) { /* Failure case */ reader_unlock(); writer_lock(); if (my_cnt != recognize_cnt) { /* another thread has already re-enabled wildcards */ writer_unlock(); continue; } MPI_Comm_reenable_any_source(comm, &failed_group); ++recognize_cnt; writer_unlock(); continue; } reader_unlock(); /* Process the received message */ } ------------------------------ I think we can do the same with the nonblocking version. So in the example: Thread 0 Thread 1 -------------------------------------- Irecv(AS, TagB)[B] Irecv(AS,TagA)[A] /******** Process N fails ***********/ Wait() ->Warn ->Reenable() ->Wait() Irecv(AS, TagC)[C] Waitall() --> ??? -------------------------------------- When process N fails, both [A] and [B] are marked to 'warn'. The call to Reenable() from Thread 0 only affects -newly- posted receives following it. So [B] would still produce a warning at the waitall() even though ANY_SOURCE is enabled for the communicator. [C] would block as normal since it occurred after the Reenable. The user should use a synchronization mechanism like readers/writers to make sure that the posting of [C] is ok or not. So a modified version of the example might look like: ------------------------------ while(!done) { reader_lock(); if (my_cnt != recognize_cnt) { /* New failures were detected */ /* check failed_group and decide if ok to continue */ if (ok_to_continue(failed_group) == FALSE) { reader_unlock(); break; } my_cnt == recognize_cnt; } MPI_Irecv(..., MPI_ANY_SOURCE, ..., comm, ..., &req); do { err = MPI_Wait(req, status) if (err == MPI_WARN_PROC_FAIL_STOP) { /* Failure case */ reader_unlock(); writer_lock(); /* We could use the following call to check if * some other thread handled the warning */ MPI_Comm_any_source_enabled(comm, &enabled); if( my_cnt != recognize_cnt) { /* another thread has already re-enabled wildcards */ writer_unlock(); reader_lock(); continue; } MPI_Comm_reenable_any_source(comm, &failed_group); /* Decide if we should keep waiting or not */ if (ok_to_continue(failed_group) == FALSE) { /* must cancel the request if we do not intend to wait */ MPI_Cancel(req); writer_unlock(); reader_lock(); break; } /* Otherwise keep waiting */ ++recognize_cnt; writer_unlock(); reader_lock(); continue; } } while(err != MPI_SUCCESS); reader_unlock(); /* Process the received message */ } ------------------------------ Does that address the threading problem? We still need to hold the lock through the completion (just like the blocking), but we do not restrict the number of MPI_Recv()s that can be posted, just that only one can update ANY_SOURCE. And it can only do so when all of the processes are waiting on the reader_lock() or writer_lock() - so no outstanding MPI_Irecv(ANY) calls. What do you all think? -- Josh On Thu, Oct 6, 2011 at 9:30 AM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > Yeah that's a tough one... > > Thread 0 ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ Thread 1 > -------------------------------------- > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ Irecv(AS, TagB)[B] > Irecv(AS,TagA)[A] > /******** Process N fails ***********/ > Wait() > ->Warn > ->Reenable() > ->Wait() > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ Irecv(AS, TagC)[C] > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ Waitall() > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ --> ??? > -------------------------------------- > > So [A] gets a warning, and Thread 0 fixes the communicator. What > should happen to [B] and [C]? > ï¿½- Should [B] and [C] both wait normally, since the warning was handled? > ï¿½- Should [B] return a warning (since it was logically before the > Reenable()), and [C] wait normally? > ï¿½- Should [B] and [C] both return a warning since it was not in there > thread context in which the Reenable was called? > > The first solution seems to be what we have at the moment. The second > seems to imply that Reenable() needs to iterate over all pending > requests to mark them appropriately as 'to warn'. I don't think the > third option is viable since I'm not sure the concept of 'thread > context' is well founded in the standard - except for the 'main' > thread. > > One solution is to say that the user is responsible for handling this > in their application though some complex locking scheme. I'm not 100% > convinced that this is a sound statement at the moment. > > Another is what Darius suggested, that we Reenable each AS receive > request. We would still need a Reenable for the communicator so > blocking Recv(AS) could complete successfully. So we may have to > differentiate Reenable_blocking_AS(comm) and > Reenable_nonblocking_AS(request). Seems like this could get messy. > > > > I've been thinking about two alternative designs that might be > something to consider (not abandoning the warning solution, but just a > thought experiment to try to and find something more elegant). > > Alternative A: > -------------- > Do nothing. ANY_SOURCE receives will wait for successful completion > without being interrupted by emerging failures. > > The user is then responsible for canceling (or locally completing) the > AS receive operation when they determine that it cannot complete. They > would need to determine this out of band, possibly by polling in a > separate thread. We would need to think through an example > demonstrating how the user could do this. > > With this we also lose our mechanism for general notification. > Previously we could use an MPI_Probe(ANY_SOURCE) to receive > notification of new failure without modifying the matching semantics. > With this we would need to introduce something new. > > I like this only because it makes the MPI specification/implementation > simplier/easier. I don't like that it pushes the burden on the end > user. > > > Alternative B: > -------------- > Per message callbacks. The user registers a special ANY_SOURCE warning > callback. If there are failures in the communicator, the callback is > fired and the user has to decide on a per-receive basis whether to > cancel or continue waiting. > > The callback could be applied to both blocking and nonblocking receive > operations, and have a signature of something like: > ï¿½MPI_ANY_SOURCE_FAULT_CALLBACK(tag, comm, failed_grp) > The user would return 0 to continue waiting, and -1 to cancel the > operation and complete it with an error. > > The problem with this is that the callback is that it may be difficult > to associate with a specific receive and line in the code. So it may > be difficult to figure out if it is ok to keep waiting from just the > matching bits and the group of failed processes. We also need to > specify what MPI calls can be used in the callback - I would say it is > restricted to something like MPI_Op, only local operations. > > > > On Wed, Oct 5, 2011 at 3:10 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: >> >> I suspect that we'd get thread issues with the warning approach: >> >> ï¿½ ï¿½Two threads post AS receives. >> ï¿½ ï¿½Thread A does a Wait and gets the warning, then does a reenable_AS >> ï¿½ ï¿½Thread B does a Wait and hangs because all of its senders are dead >> >> We didn't have this problem before because thread B's receives would have completed with an error and so it would have gotten an error when it tried to wait on it. >> >> We handled this case by using reader-writer locks to prevent another thread from reenabling anysources between when the user checks that potential senders are alive and when it posts the receive. ï¿½Now we'd need to hold the lock between when we check for senders and when the request actually completes. >> >> Does that mean we need to explicitly reenable each AS receive? >> >> -d >> >> >> On Oct 5, 2011, at 1:34 PM, Josh Hursey wrote: >> >>> Currently we state: >>> ---------------------- >>> 17.6 Point-to-Point Communication >>> ... >>> When a process detects a new process failure, the ability to perform >>> wildcard receives (i.e., receives where MPI_ANY_SOURCE has been >>> specified for the source parameter) will be disabled on all >>> communicators that contain the failed process. When wildcard receives >>> are disabled on a communicator, all pending wildcard receive >>> operations on that communicator are completed and an error with class >>> MPI_ERR_PROC_FAIL_STOP will be returned for those operations. Any new >>> wildcard receive operations posted to a communicator with disabled >>> wildcard receives will be immediately completed and return an error >>> code of the class MPI_ERR_PROC_FAIL_STOP. >>> Wildcard receives can be re-enabled with the >>> MPI_COMM_REENABLE_ANY_SOURCE function described below. >>> ---------------------- >>> >>> The problem is that by completing the pending wildcard receives with >>> an error, we may cause unintended matching of concurrent receives. For >>> example: >>> ------------------- >>> Proc 0 ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½Proc 1 >>> Irecv(ANY,TAGX); [A] >>> Irecv(1, TAGX); [B] >>> Waitall() >>> /******* Proc 2 fails *********/ >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½Send(0, TAGX) [C] >>> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½Send(0, TAGX) [D] >>> ------------------- >>> The intention is that Irecv[A] will match Send[C] and Irecv[B] will >>> match Send[D]. But if another process fails (proc 2 in this example), >>> then Irecv[A] will complete in error. Then Irecv[B] will match >>> Send[C], and Send[D] will remain unmatched. >>> >>> >>> We need a mechanism that provides notification to the application >>> waiting on an ANY_SOURCE receive that some process failed in this >>> communicator while still providing the necessary matching guarantees. >>> The user can then choose if they want to cancel the operation or >>> continue waiting. >>> >>> >>> On the teleconference today, we were thinking through a warning based >>> concept. Instead of completing the pending nonblocking ANY_SOURCE >>> receives, we would return a special warning code (say >>> MPI_WARN_PROC_FAIL_STOP). The message would -not- be completed, but >>> return to the user to decide if they wish to continue waiting or >>> cancel the offending receive operation. >>> >>> For blocking ANY_SOURCE receives, the user would be returned an error >>> (MPI_ERR_PROC_FAIL_STOP), and the operation would be completed in >>> error. Since we do not have a request handle, there is no way to keep >>> this receive active while the situation is resolved. So we complete >>> the receive with an error, in a sense canceling the receive operation. >>> >>> For nonblocking ANY_SOURCE receives, the user would be returned a >>> warning (MPI_WARN_PROC_FAIL_STOP), and the request would remain >>> active. The application would then be able to either re-enable >>> ANY_SOURCE (via the API call) or cancel the offending receive >>> operation. The user is in control over the matching guarantees since >>> the request remains active, but cannot complete without action from >>> the user. >>> >>> >>> In the example above, Proc 0 would return MPI_ERR_IN_STATUS from >>> MPI_WAITALL. The status of the active requests would be >>> MPI_ERR_PENDING (per section 3.7.3), except for the offending >>> ANY_SOURCE operations which will have the error field set to >>> MPI_WARN_PROC_FAIL_STOP. So Irecv[A] would be MPI_ERR_PENDING, and >>> Irecv[B] would be MPI_WARN_PROC_FAIL_STOP. The user can then associate >>> the error with a specific request. Once a request is identified the >>> user can either cancel it, or re-enable ANY_SOURCE on the communicator >>> and continue waiting. >>> >>> >>> So I think this fixes the matching issue with the interface, but I'm >>> not sure if it addresses the concerns of the hardware matching folks. >>> Additionally, this proposal adds a new state to the request concept >>> 'active but not completable without action by the user' and we need to >>> think if this requires semantic qualification elsewhere in the >>> standard. >>> >>> >>> What to folks think about this solution? >>> >>> -- Josh >>> >>> -- >>> Joshua Hursey >>> Postdoctoral Research Associate >>> Oak Ridge National Laboratory >>> http://users.nccs.gov/~jjhursey >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [Mpi3-ft] MPI_ANY_SOURCE
by Josh Hursey 06 Oct '11

06 Oct '11

Yeah that's a tough one... Thread 0 Thread 1 -------------------------------------- Irecv(AS, TagB)[B] Irecv(AS,TagA)[A] /******** Process N fails ***********/ Wait() ->Warn ->Reenable() ->Wait() Irecv(AS, TagC)[C] Waitall() --> ??? -------------------------------------- So [A] gets a warning, and Thread 0 fixes the communicator. What should happen to [B] and [C]? - Should [B] and [C] both wait normally, since the warning was handled? - Should [B] return a warning (since it was logically before the Reenable()), and [C] wait normally? - Should [B] and [C] both return a warning since it was not in there thread context in which the Reenable was called? The first solution seems to be what we have at the moment. The second seems to imply that Reenable() needs to iterate over all pending requests to mark them appropriately as 'to warn'. I don't think the third option is viable since I'm not sure the concept of 'thread context' is well founded in the standard - except for the 'main' thread. One solution is to say that the user is responsible for handling this in their application though some complex locking scheme. I'm not 100% convinced that this is a sound statement at the moment. Another is what Darius suggested, that we Reenable each AS receive request. We would still need a Reenable for the communicator so blocking Recv(AS) could complete successfully. So we may have to differentiate Reenable_blocking_AS(comm) and Reenable_nonblocking_AS(request). Seems like this could get messy. I've been thinking about two alternative designs that might be something to consider (not abandoning the warning solution, but just a thought experiment to try to and find something more elegant). Alternative A: -------------- Do nothing. ANY_SOURCE receives will wait for successful completion without being interrupted by emerging failures. The user is then responsible for canceling (or locally completing) the AS receive operation when they determine that it cannot complete. They would need to determine this out of band, possibly by polling in a separate thread. We would need to think through an example demonstrating how the user could do this. With this we also lose our mechanism for general notification. Previously we could use an MPI_Probe(ANY_SOURCE) to receive notification of new failure without modifying the matching semantics. With this we would need to introduce something new. I like this only because it makes the MPI specification/implementation simplier/easier. I don't like that it pushes the burden on the end user. Alternative B: -------------- Per message callbacks. The user registers a special ANY_SOURCE warning callback. If there are failures in the communicator, the callback is fired and the user has to decide on a per-receive basis whether to cancel or continue waiting. The callback could be applied to both blocking and nonblocking receive operations, and have a signature of something like: MPI_ANY_SOURCE_FAULT_CALLBACK(tag, comm, failed_grp) The user would return 0 to continue waiting, and -1 to cancel the operation and complete it with an error. The problem with this is that the callback is that it may be difficult to associate with a specific receive and line in the code. So it may be difficult to figure out if it is ok to keep waiting from just the matching bits and the group of failed processes. We also need to specify what MPI calls can be used in the callback - I would say it is restricted to something like MPI_Op, only local operations. On Wed, Oct 5, 2011 at 3:10 PM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: > > I suspect that we'd get thread issues with the warning approach: > > ï¿½ ï¿½Two threads post AS receives. > ï¿½ ï¿½Thread A does a Wait and gets the warning, then does a reenable_AS > ï¿½ ï¿½Thread B does a Wait and hangs because all of its senders are dead > > We didn't have this problem before because thread B's receives would have completed with an error and so it would have gotten an error when it tried to wait on it. > > We handled this case by using reader-writer locks to prevent another thread from reenabling anysources between when the user checks that potential senders are alive and when it posts the receive. ï¿½Now we'd need to hold the lock between when we check for senders and when the request actually completes. > > Does that mean we need to explicitly reenable each AS receive? > > -d > > > On Oct 5, 2011, at 1:34 PM, Josh Hursey wrote: > >> Currently we state: >> ---------------------- >> 17.6 Point-to-Point Communication >> ... >> When a process detects a new process failure, the ability to perform >> wildcard receives (i.e., receives where MPI_ANY_SOURCE has been >> specified for the source parameter) will be disabled on all >> communicators that contain the failed process. When wildcard receives >> are disabled on a communicator, all pending wildcard receive >> operations on that communicator are completed and an error with class >> MPI_ERR_PROC_FAIL_STOP will be returned for those operations. Any new >> wildcard receive operations posted to a communicator with disabled >> wildcard receives will be immediately completed and return an error >> code of the class MPI_ERR_PROC_FAIL_STOP. >> Wildcard receives can be re-enabled with the >> MPI_COMM_REENABLE_ANY_SOURCE function described below. >> ---------------------- >> >> The problem is that by completing the pending wildcard receives with >> an error, we may cause unintended matching of concurrent receives. For >> example: >> ------------------- >> Proc 0 ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½Proc 1 >> Irecv(ANY,TAGX); [A] >> Irecv(1, TAGX); [B] >> Waitall() >> /******* Proc 2 fails *********/ >> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½Send(0, TAGX) [C] >> ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½Send(0, TAGX) [D] >> ------------------- >> The intention is that Irecv[A] will match Send[C] and Irecv[B] will >> match Send[D]. But if another process fails (proc 2 in this example), >> then Irecv[A] will complete in error. Then Irecv[B] will match >> Send[C], and Send[D] will remain unmatched. >> >> >> We need a mechanism that provides notification to the application >> waiting on an ANY_SOURCE receive that some process failed in this >> communicator while still providing the necessary matching guarantees. >> The user can then choose if they want to cancel the operation or >> continue waiting. >> >> >> On the teleconference today, we were thinking through a warning based >> concept. Instead of completing the pending nonblocking ANY_SOURCE >> receives, we would return a special warning code (say >> MPI_WARN_PROC_FAIL_STOP). The message would -not- be completed, but >> return to the user to decide if they wish to continue waiting or >> cancel the offending receive operation. >> >> For blocking ANY_SOURCE receives, the user would be returned an error >> (MPI_ERR_PROC_FAIL_STOP), and the operation would be completed in >> error. Since we do not have a request handle, there is no way to keep >> this receive active while the situation is resolved. So we complete >> the receive with an error, in a sense canceling the receive operation. >> >> For nonblocking ANY_SOURCE receives, the user would be returned a >> warning (MPI_WARN_PROC_FAIL_STOP), and the request would remain >> active. The application would then be able to either re-enable >> ANY_SOURCE (via the API call) or cancel the offending receive >> operation. The user is in control over the matching guarantees since >> the request remains active, but cannot complete without action from >> the user. >> >> >> In the example above, Proc 0 would return MPI_ERR_IN_STATUS from >> MPI_WAITALL. The status of the active requests would be >> MPI_ERR_PENDING (per section 3.7.3), except for the offending >> ANY_SOURCE operations which will have the error field set to >> MPI_WARN_PROC_FAIL_STOP. So Irecv[A] would be MPI_ERR_PENDING, and >> Irecv[B] would be MPI_WARN_PROC_FAIL_STOP. The user can then associate >> the error with a specific request. Once a request is identified the >> user can either cancel it, or re-enable ANY_SOURCE on the communicator >> and continue waiting. >> >> >> So I think this fixes the matching issue with the interface, but I'm >> not sure if it addresses the concerns of the hardware matching folks. >> Additionally, this proposal adds a new state to the request concept >> 'active but not completable without action by the user' and we need to >> think if this requires semantic qualification elsewhere in the >> standard. >> >> >> What to folks think about this solution? >> >> -- Josh >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0