August 2010 - mpiwg-ft - mailman3.cels.anl.gov

Re: [Mpi3-ft] Persistent Communication & Sendrecv
by Fab Tillier 31 Aug '10

31 Aug '10

Joshua Hursey wrote on Tue, 31 Aug 2010 at 12:59:58 > > I think I see what is getting confused. We are not changing > MPI_Recv_init or MPI_Irecv from the current standard. So the status > object is only available on a call to one of the completion functions. > > I was running with the assumption that if a process is known to be > failed at the call to either of these functions then they will return > MPI_ERR_IN_STATUS (even though these functions don't have a status > object, this tells the user to call a completion call to access the > status object). Further, if the process is not known to be failed until > the call to MPI_Start{all}, then the start function would return > MPI_ERR_IN_STATUS. Then the application should use one of the completion > calls to access the error code from the returned status object. I'd much rather see the functions return MPI_SUCCESS here - the user needs to call the completion function to get actual completion status, and at that point the error can be reported. Returning MPI_ERR_IN_STATUS when no status is returned and requiring a user to call a completion function to get the actual error is the exact same code flow as the MPI_SUCCESS case, so not sure the MPI_ERR_IN_STATUS return value buys you anything. > Thinking about this some more and rereading the standard, both > MPI_Irecv, MPI_Recv_init, MPI_Start{all} should return immediately > since they are local operations. Their return code should -not- be > determined by the other processes. I say this after reading the > following sentence in the introduction to section 3.7: > "In all cases, the send start call is local: it returns immediately, > irrespective of the status of other processes." > > So for MPI_Startall, it will start all of the requests and return > MPI_SUCCESS (unless one of the arguments is bad). It will not be until > the user calls a completion function that they will be notified of an > error, even if it was known at the time of the call to MPI_Startall. > > How do people feel about this? I like this much better, but this should be applied to MPI_Irecv and MPI_Recv_init too. It wasn't clear if you intended this to apply only to MPI_Startall or not. -Fab

1 0

Re: [Mpi3-ft] Persistent Communication & Sendrecv
by Joshua Hursey 31 Aug '10

31 Aug '10

I updated the wiki with some examples, and cleaned up the persistent discussion per this thread. https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… -- Josh On Aug 31, 2010, at 3:59 PM, Joshua Hursey wrote: > > I think I see what is getting confused. We are not changing MPI_Recv_init or MPI_Irecv from the current standard. So the status object is only available on a call to one of the completion functions. > > I was running with the assumption that if a process is known to be failed at the call to either of these functions then they will return MPI_ERR_IN_STATUS (even though these functions don't have a status object, this tells the user to call a completion call to access the status object). Further, if the process is not known to be failed until the call to MPI_Start{all}, then the start function would return MPI_ERR_IN_STATUS. Then the application should use one of the completion calls to access the error code from the returned status object. > > Thinking about this some more and rereading the standard, both MPI_Irecv, MPI_Recv_init, MPI_Start{all} should return immediately since they are local operations. Their return code should -not- be determined by the other processes. I say this after reading the following sentence in the introduction to section 3.7: > "In all cases, the send start call is local: it returns immediately, irrespective of the status of other processes." > > So for MPI_Startall, it will start all of the requests and return MPI_SUCCESS (unless one of the arguments is bad). It will not be until the user calls a completion function that they will be notified of an error, even if it was known at the time of the call to MPI_Startall. > > How do people feel about this? > > I'll sketch out some examples on the wiki that we can use to help the discussion. > > -- Josh > > On Aug 31, 2010, at 3:15 PM, Bronis R. de Supinski wrote: > >> >> Josh: >> >> OK, so I am not fully up to date on the FT plans. If you >> have MPI_Recv_init or MPI_Irecv return a status (which >> is how I read the idea) then it CANNOT return what was >> matched. If, as this post reads, you mean the completion >> associated with the request then I am not clear on how >> it is relevant whether it was an immediate or persistent >> request. Perhaps we are talking across each other... >> >> Bronis >> >> >> >> On Tue, 31 Aug 2010, Joshua Hursey wrote: >> >>> I agree that the count=0 is odd (section 3.2.2 in case anyone is interested). Maybe it is for control messages where the buffer does not matter, just that you received a message (e.g., Barrier). >>> >>> I don't know if I follow the rest. So when I call: >>> ------------------ >>> MPI_Irecv(MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, &req); >>> MPI_Wait(req, status); >>> ------------------ >>> The field values in 'status' will match the incoming received message. This way the user can figure out from whom and with what tag this recv matched. Or with what error the request failed. >>> >>> If the user were to write: >>> ------------------ >>> MPI_Irecv(MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, &req); >>> MPI_Wait(req, status); >>> MPI_Wait(req, status); >>> ------------------ >>> The second wait would return MPI_SUCCESS with an 'empty' status since the first wait destroys the request. >>> >>> For the persistent situation, the MPI_Wait does not destroy the request, but just sets it inactive. So in: >>> ------------------ >>> MPI_Recv_init(MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, &req); >>> MPI_Start(req); >>> MPI_Request_get_status(req, flag, status); /* A */ >>> MPI_Wait(req, status); >>> MPI_Request_get_status(req, flag, status); /* B */ >>> MPI_Wait(req, status); >>> ------------------ >>> The second MPI_Wait would still return an 'empty' status since the request is inactive. The MPI_Request_get_status in (A), if the request is finished, will set the flag to true and return a status with the various fields set to match the incoming message. If the request is still active then it will return false, and the 'empty' status. The MPI_Request_get_status in (B) will set the flag to true (the request is not active or null) and return 'empty' status. >>> >>> So in the Startall case the user can figure out which requests were started, which failed, and which were not started by checking the arguments returned by MPI_Request_get_status one at a time: >>> forall requests call MPI_Request_get_status(req, flag, status) >>> if( flag ) { >>> if( status.error != success ) // Failed >>> if( status == empty ) // Not started >>> if( status != empty ) // Started, and now complete >>> } else { >>> // Started, and not yet complete >>> } >>> >>> What am I missing? >>> >>> -- Josh >>> >>> >>> On Aug 31, 2010, at 1:20 PM, Bronis R. de Supinski wrote: >>> >>>> >>>> Huh? No, the MPI_Recv_init creates a persistent request with >>>> MPI_ANY_SOURCE and MPI_ANY_TAG for those fields. Combined with >>>> the MPI_Startall, it is roughly the same as this: >>>> >>>> MPI_Irecv (MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, &req[1]); >>>> >>>> (I'll admit the "count=0" arg is a bit odd). A wait on the >>>> started request can match any message (again, the count is >>>> a bit strange and Josh omitted the type arg but it is >>>> pseudocode so no big deal). >>>> >>>> On Tue, 31 Aug 2010, Joshua Hursey wrote: >>>> >>>>> >>>>> On Aug 31, 2010, at 1:09 PM, Fab Tillier wrote: >>>>> >>>>>> Joshua Hursey wrote on Tue, 31 Aug 2010 at 08:43:24 >>>>>> >>>>>>> I was thinking more about MPI_Startall() this morning and found a >>>>>>> situation where this technique would not work. >>>>>>> >>>>>>> If the application does: >>>>>>> -------------------- >>>>>>> MPI_Send_init(rank=1, tag=123, req[0]); >>>>>>> MPI_Recv_init(MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, req[1]); >>>>>>> MPI_Send_init(rank=2, tag=123, req[2]); >>>>>>> >>>>>>> MPI_Startall(3, req) // Fails with MPI_ERR_IN_STATUS >>>>>>> if( failed ) { >>>>>>> for(i=0; i<3; ++i) { >>>>>>> MPI_Request_get_status(req[i], flag, status); >>>>>>> if( flag && status.error != success ) // Failed >>>>>>> if( flag && status == empty ) // Not started >>>>>>> if( flag && status != empty ) // Complete >>>>>>> } >>>>>>> } >>>>>>> -------------------- >>>>>>> >>>>>>> The problem is with the definition of an 'empty' status which has >>>>>>> (section 3.7.3): >>>>>>> --------------------- >>>>>>> source = MPI_ANY_SOURCE >>>>>>> tag = MPI_ANY_TAG >>>>>>> error = MPI_SUCCESS >>>>>>> MPI_Get_count = 0 >>>>>>> MPI_test_cancelled = false >>>>>>> --------------------- >>>>>>> So the successful completion of the MPI_Recv_init() call would be >>>>>>> indistinguishable from the 'not started' or inactive state of the call. >>>>>> >>>>>> Wouldn't a successful completion of the MPI_Recv_init() return a specific source and tag for the message actually received? The source and tag fields of the receive are for filtering incoming sends, but when the receive completes, it was matched to exactly one send, with a specific tag and source. >>>>>> >>>>>> What am I missing? >>>>> >>>>> Ah yes. You are correct. So this is not a problem then, and the MPI_Request_get_status() technique would be a good way to check the state of all of the requests. >>>>> >>>>> Thanks, >>>>> Josh >>>>> >>>>>> >>>>>> -Fab >>>>>> >>>>>> _______________________________________________ >>>>>> mpi3-ft mailing list >>>>>> mpi3-ft(a)lists.mpi-forum.org >>>>>> http://**lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>>> >>>>> >>>>> ------------------------------------ >>>>> Joshua Hursey >>>>> Postdoctoral Research Associate >>>>> Oak Ridge National Laboratory >>>>> http://**www.**cs.indiana.edu/~jjhursey >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org >>>>> http://**lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>> >>>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://*lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>> >>> ------------------------------------ >>> Joshua Hursey >>> Postdoctoral Research Associate >>> Oak Ridge National Laboratory >>> http://*www.*cs.indiana.edu/~jjhursey >>> >>> >>> >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://*lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> > > ------------------------------------ > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://www.cs.indiana.edu/~jjhursey > > > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://www.cs.indiana.edu/~jjhursey

1 0

Re: [Mpi3-ft] Persistent Communication & Sendrecv
by Joshua Hursey 31 Aug '10

31 Aug '10

I think I see what is getting confused. We are not changing MPI_Recv_init or MPI_Irecv from the current standard. So the status object is only available on a call to one of the completion functions. I was running with the assumption that if a process is known to be failed at the call to either of these functions then they will return MPI_ERR_IN_STATUS (even though these functions don't have a status object, this tells the user to call a completion call to access the status object). Further, if the process is not known to be failed until the call to MPI_Start{all}, then the start function would return MPI_ERR_IN_STATUS. Then the application should use one of the completion calls to access the error code from the returned status object. Thinking about this some more and rereading the standard, both MPI_Irecv, MPI_Recv_init, MPI_Start{all} should return immediately since they are local operations. Their return code should -not- be determined by the other processes. I say this after reading the following sentence in the introduction to section 3.7: "In all cases, the send start call is local: it returns immediately, irrespective of the status of other processes." So for MPI_Startall, it will start all of the requests and return MPI_SUCCESS (unless one of the arguments is bad). It will not be until the user calls a completion function that they will be notified of an error, even if it was known at the time of the call to MPI_Startall. How do people feel about this? I'll sketch out some examples on the wiki that we can use to help the discussion. -- Josh On Aug 31, 2010, at 3:15 PM, Bronis R. de Supinski wrote: > > Josh: > > OK, so I am not fully up to date on the FT plans. If you > have MPI_Recv_init or MPI_Irecv return a status (which > is how I read the idea) then it CANNOT return what was > matched. If, as this post reads, you mean the completion > associated with the request then I am not clear on how > it is relevant whether it was an immediate or persistent > request. Perhaps we are talking across each other... > > Bronis > > > > On Tue, 31 Aug 2010, Joshua Hursey wrote: > >> I agree that the count=0 is odd (section 3.2.2 in case anyone is interested). Maybe it is for control messages where the buffer does not matter, just that you received a message (e.g., Barrier). >> >> I don't know if I follow the rest. So when I call: >> ------------------ >> MPI_Irecv(MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, &req); >> MPI_Wait(req, status); >> ------------------ >> The field values in 'status' will match the incoming received message. This way the user can figure out from whom and with what tag this recv matched. Or with what error the request failed. >> >> If the user were to write: >> ------------------ >> MPI_Irecv(MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, &req); >> MPI_Wait(req, status); >> MPI_Wait(req, status); >> ------------------ >> The second wait would return MPI_SUCCESS with an 'empty' status since the first wait destroys the request. >> >> For the persistent situation, the MPI_Wait does not destroy the request, but just sets it inactive. So in: >> ------------------ >> MPI_Recv_init(MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, &req); >> MPI_Start(req); >> MPI_Request_get_status(req, flag, status); /* A */ >> MPI_Wait(req, status); >> MPI_Request_get_status(req, flag, status); /* B */ >> MPI_Wait(req, status); >> ------------------ >> The second MPI_Wait would still return an 'empty' status since the request is inactive. The MPI_Request_get_status in (A), if the request is finished, will set the flag to true and return a status with the various fields set to match the incoming message. If the request is still active then it will return false, and the 'empty' status. The MPI_Request_get_status in (B) will set the flag to true (the request is not active or null) and return 'empty' status. >> >> So in the Startall case the user can figure out which requests were started, which failed, and which were not started by checking the arguments returned by MPI_Request_get_status one at a time: >> forall requests call MPI_Request_get_status(req, flag, status) >> if( flag ) { >> if( status.error != success ) // Failed >> if( status == empty ) // Not started >> if( status != empty ) // Started, and now complete >> } else { >> // Started, and not yet complete >> } >> >> What am I missing? >> >> -- Josh >> >> >> On Aug 31, 2010, at 1:20 PM, Bronis R. de Supinski wrote: >> >>> >>> Huh? No, the MPI_Recv_init creates a persistent request with >>> MPI_ANY_SOURCE and MPI_ANY_TAG for those fields. Combined with >>> the MPI_Startall, it is roughly the same as this: >>> >>> MPI_Irecv (MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, &req[1]); >>> >>> (I'll admit the "count=0" arg is a bit odd). A wait on the >>> started request can match any message (again, the count is >>> a bit strange and Josh omitted the type arg but it is >>> pseudocode so no big deal). >>> >>> On Tue, 31 Aug 2010, Joshua Hursey wrote: >>> >>>> >>>> On Aug 31, 2010, at 1:09 PM, Fab Tillier wrote: >>>> >>>>> Joshua Hursey wrote on Tue, 31 Aug 2010 at 08:43:24 >>>>> >>>>>> I was thinking more about MPI_Startall() this morning and found a >>>>>> situation where this technique would not work. >>>>>> >>>>>> If the application does: >>>>>> -------------------- >>>>>> MPI_Send_init(rank=1, tag=123, req[0]); >>>>>> MPI_Recv_init(MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, req[1]); >>>>>> MPI_Send_init(rank=2, tag=123, req[2]); >>>>>> >>>>>> MPI_Startall(3, req) // Fails with MPI_ERR_IN_STATUS >>>>>> if( failed ) { >>>>>> for(i=0; i<3; ++i) { >>>>>> MPI_Request_get_status(req[i], flag, status); >>>>>> if( flag && status.error != success ) // Failed >>>>>> if( flag && status == empty ) // Not started >>>>>> if( flag && status != empty ) // Complete >>>>>> } >>>>>> } >>>>>> -------------------- >>>>>> >>>>>> The problem is with the definition of an 'empty' status which has >>>>>> (section 3.7.3): >>>>>> --------------------- >>>>>> source = MPI_ANY_SOURCE >>>>>> tag = MPI_ANY_TAG >>>>>> error = MPI_SUCCESS >>>>>> MPI_Get_count = 0 >>>>>> MPI_test_cancelled = false >>>>>> --------------------- >>>>>> So the successful completion of the MPI_Recv_init() call would be >>>>>> indistinguishable from the 'not started' or inactive state of the call. >>>>> >>>>> Wouldn't a successful completion of the MPI_Recv_init() return a specific source and tag for the message actually received? The source and tag fields of the receive are for filtering incoming sends, but when the receive completes, it was matched to exactly one send, with a specific tag and source. >>>>> >>>>> What am I missing? >>>> >>>> Ah yes. You are correct. So this is not a problem then, and the MPI_Request_get_status() technique would be a good way to check the state of all of the requests. >>>> >>>> Thanks, >>>> Josh >>>> >>>>> >>>>> -Fab >>>>> >>>>> _______________________________________________ >>>>> mpi3-ft mailing list >>>>> mpi3-ft(a)lists.mpi-forum.org >>>>> http://**lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>>> >>>> >>>> ------------------------------------ >>>> Joshua Hursey >>>> Postdoctoral Research Associate >>>> Oak Ridge National Laboratory >>>> http://**www.**cs.indiana.edu/~jjhursey >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://**lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://*lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >> >> ------------------------------------ >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://*www.*cs.indiana.edu/~jjhursey >> >> >> >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://*lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://www.cs.indiana.edu/~jjhursey

1 0

Re: [Mpi3-ft] Persistent Communication & Sendrecv
by Bronis R. de Supinski 31 Aug '10

31 Aug '10

Josh: OK, so I am not fully up to date on the FT plans. If you have MPI_Recv_init or MPI_Irecv return a status (which is how I read the idea) then it CANNOT return what was matched. If, as this post reads, you mean the completion associated with the request then I am not clear on how it is relevant whether it was an immediate or persistent request. Perhaps we are talking across each other... Bronis On Tue, 31 Aug 2010, Joshua Hursey wrote: > I agree that the count=0 is odd (section 3.2.2 in case anyone is interested). Maybe it is for control messages where the buffer does not matter, just that you received a message (e.g., Barrier). > > I don't know if I follow the rest. So when I call: > ------------------ > MPI_Irecv(MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, &req); > MPI_Wait(req, status); > ------------------ > The field values in 'status' will match the incoming received message. This way the user can figure out from whom and with what tag this recv matched. Or with what error the request failed. > > If the user were to write: > ------------------ > MPI_Irecv(MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, &req); > MPI_Wait(req, status); > MPI_Wait(req, status); > ------------------ > The second wait would return MPI_SUCCESS with an 'empty' status since the first wait destroys the request. > > For the persistent situation, the MPI_Wait does not destroy the request, but just sets it inactive. So in: > ------------------ > MPI_Recv_init(MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, &req); > MPI_Start(req); > MPI_Request_get_status(req, flag, status); /* A */ > MPI_Wait(req, status); > MPI_Request_get_status(req, flag, status); /* B */ > MPI_Wait(req, status); > ------------------ > The second MPI_Wait would still return an 'empty' status since the request is inactive. The MPI_Request_get_status in (A), if the request is finished, will set the flag to true and return a status with the various fields set to match the incoming message. If the request is still active then it will return false, and the 'empty' status. The MPI_Request_get_status in (B) will set the flag to true (the request is not active or null) and return 'empty' status. > > So in the Startall case the user can figure out which requests were started, which failed, and which were not started by checking the arguments returned by MPI_Request_get_status one at a time: > forall requests call MPI_Request_get_status(req, flag, status) > if( flag ) { > if( status.error != success ) // Failed > if( status == empty ) // Not started > if( status != empty ) // Started, and now complete > } else { > // Started, and not yet complete > } > > What am I missing? > > -- Josh > > > On Aug 31, 2010, at 1:20 PM, Bronis R. de Supinski wrote: > >> >> Huh? No, the MPI_Recv_init creates a persistent request with >> MPI_ANY_SOURCE and MPI_ANY_TAG for those fields. Combined with >> the MPI_Startall, it is roughly the same as this: >> >> MPI_Irecv (MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, &req[1]); >> >> (I'll admit the "count=0" arg is a bit odd). A wait on the >> started request can match any message (again, the count is >> a bit strange and Josh omitted the type arg but it is >> pseudocode so no big deal). >> >> On Tue, 31 Aug 2010, Joshua Hursey wrote: >> >>> >>> On Aug 31, 2010, at 1:09 PM, Fab Tillier wrote: >>> >>>> Joshua Hursey wrote on Tue, 31 Aug 2010 at 08:43:24 >>>> >>>>> I was thinking more about MPI_Startall() this morning and found a >>>>> situation where this technique would not work. >>>>> >>>>> If the application does: >>>>> -------------------- >>>>> MPI_Send_init(rank=1, tag=123, req[0]); >>>>> MPI_Recv_init(MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, req[1]); >>>>> MPI_Send_init(rank=2, tag=123, req[2]); >>>>> >>>>> MPI_Startall(3, req) // Fails with MPI_ERR_IN_STATUS >>>>> if( failed ) { >>>>> for(i=0; i<3; ++i) { >>>>> MPI_Request_get_status(req[i], flag, status); >>>>> if( flag && status.error != success ) // Failed >>>>> if( flag && status == empty ) // Not started >>>>> if( flag && status != empty ) // Complete >>>>> } >>>>> } >>>>> -------------------- >>>>> >>>>> The problem is with the definition of an 'empty' status which has >>>>> (section 3.7.3): >>>>> --------------------- >>>>> source = MPI_ANY_SOURCE >>>>> tag = MPI_ANY_TAG >>>>> error = MPI_SUCCESS >>>>> MPI_Get_count = 0 >>>>> MPI_test_cancelled = false >>>>> --------------------- >>>>> So the successful completion of the MPI_Recv_init() call would be >>>>> indistinguishable from the 'not started' or inactive state of the call. >>>> >>>> Wouldn't a successful completion of the MPI_Recv_init() return a specific source and tag for the message actually received? The source and tag fields of the receive are for filtering incoming sends, but when the receive completes, it was matched to exactly one send, with a specific tag and source. >>>> >>>> What am I missing? >>> >>> Ah yes. You are correct. So this is not a problem then, and the MPI_Request_get_status() technique would be a good way to check the state of all of the requests. >>> >>> Thanks, >>> Josh >>> >>>> >>>> -Fab >>>> >>>> _______________________________________________ >>>> mpi3-ft mailing list >>>> mpi3-ft(a)lists.mpi-forum.org >>>> http://**lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>>> >>> >>> ------------------------------------ >>> Joshua Hursey >>> Postdoctoral Research Associate >>> Oak Ridge National Laboratory >>> http://**www.**cs.indiana.edu/~jjhursey >>> >>> >>> >>> >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://**lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >>> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://*lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> > > ------------------------------------ > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://*www.*cs.indiana.edu/~jjhursey > > > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://*lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >

1 0

Re: [Mpi3-ft] Persistent Communication & Sendrecv
by Joshua Hursey 31 Aug '10

31 Aug '10

I agree that the count=0 is odd (section 3.2.2 in case anyone is interested). Maybe it is for control messages where the buffer does not matter, just that you received a message (e.g., Barrier). I don't know if I follow the rest. So when I call: ------------------ MPI_Irecv(MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, &req); MPI_Wait(req, status); ------------------ The field values in 'status' will match the incoming received message. This way the user can figure out from whom and with what tag this recv matched. Or with what error the request failed. If the user were to write: ------------------ MPI_Irecv(MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, &req); MPI_Wait(req, status); MPI_Wait(req, status); ------------------ The second wait would return MPI_SUCCESS with an 'empty' status since the first wait destroys the request. For the persistent situation, the MPI_Wait does not destroy the request, but just sets it inactive. So in: ------------------ MPI_Recv_init(MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, &req); MPI_Start(req); MPI_Request_get_status(req, flag, status); /* A */ MPI_Wait(req, status); MPI_Request_get_status(req, flag, status); /* B */ MPI_Wait(req, status); ------------------ The second MPI_Wait would still return an 'empty' status since the request is inactive. The MPI_Request_get_status in (A), if the request is finished, will set the flag to true and return a status with the various fields set to match the incoming message. If the request is still active then it will return false, and the 'empty' status. The MPI_Request_get_status in (B) will set the flag to true (the request is not active or null) and return 'empty' status. So in the Startall case the user can figure out which requests were started, which failed, and which were not started by checking the arguments returned by MPI_Request_get_status one at a time: forall requests call MPI_Request_get_status(req, flag, status) if( flag ) { if( status.error != success ) // Failed if( status == empty ) // Not started if( status != empty ) // Started, and now complete } else { // Started, and not yet complete } What am I missing? -- Josh On Aug 31, 2010, at 1:20 PM, Bronis R. de Supinski wrote: > > Huh? No, the MPI_Recv_init creates a persistent request with > MPI_ANY_SOURCE and MPI_ANY_TAG for those fields. Combined with > the MPI_Startall, it is roughly the same as this: > > MPI_Irecv (MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, &req[1]); > > (I'll admit the "count=0" arg is a bit odd). A wait on the > started request can match any message (again, the count is > a bit strange and Josh omitted the type arg but it is > pseudocode so no big deal). > > On Tue, 31 Aug 2010, Joshua Hursey wrote: > >> >> On Aug 31, 2010, at 1:09 PM, Fab Tillier wrote: >> >>> Joshua Hursey wrote on Tue, 31 Aug 2010 at 08:43:24 >>> >>>> I was thinking more about MPI_Startall() this morning and found a >>>> situation where this technique would not work. >>>> >>>> If the application does: >>>> -------------------- >>>> MPI_Send_init(rank=1, tag=123, req[0]); >>>> MPI_Recv_init(MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, req[1]); >>>> MPI_Send_init(rank=2, tag=123, req[2]); >>>> >>>> MPI_Startall(3, req) // Fails with MPI_ERR_IN_STATUS >>>> if( failed ) { >>>> for(i=0; i<3; ++i) { >>>> MPI_Request_get_status(req[i], flag, status); >>>> if( flag && status.error != success ) // Failed >>>> if( flag && status == empty ) // Not started >>>> if( flag && status != empty ) // Complete >>>> } >>>> } >>>> -------------------- >>>> >>>> The problem is with the definition of an 'empty' status which has >>>> (section 3.7.3): >>>> --------------------- >>>> source = MPI_ANY_SOURCE >>>> tag = MPI_ANY_TAG >>>> error = MPI_SUCCESS >>>> MPI_Get_count = 0 >>>> MPI_test_cancelled = false >>>> --------------------- >>>> So the successful completion of the MPI_Recv_init() call would be >>>> indistinguishable from the 'not started' or inactive state of the call. >>> >>> Wouldn't a successful completion of the MPI_Recv_init() return a specific source and tag for the message actually received? The source and tag fields of the receive are for filtering incoming sends, but when the receive completes, it was matched to exactly one send, with a specific tag and source. >>> >>> What am I missing? >> >> Ah yes. You are correct. So this is not a problem then, and the MPI_Request_get_status() technique would be a good way to check the state of all of the requests. >> >> Thanks, >> Josh >> >>> >>> -Fab >>> >>> _______________________________________________ >>> mpi3-ft mailing list >>> mpi3-ft(a)lists.mpi-forum.org >>> http://*lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >>> >> >> ------------------------------------ >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://*www.*cs.indiana.edu/~jjhursey >> >> >> >> >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://*lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> >> > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://www.cs.indiana.edu/~jjhursey

1 0

Re: [Mpi3-ft] Persistent Communication & Sendrecv
by Bronis R. de Supinski 31 Aug '10

31 Aug '10

Huh? No, the MPI_Recv_init creates a persistent request with MPI_ANY_SOURCE and MPI_ANY_TAG for those fields. Combined with the MPI_Startall, it is roughly the same as this: MPI_Irecv (MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, &req[1]); (I'll admit the "count=0" arg is a bit odd). A wait on the started request can match any message (again, the count is a bit strange and Josh omitted the type arg but it is pseudocode so no big deal). On Tue, 31 Aug 2010, Joshua Hursey wrote: > > On Aug 31, 2010, at 1:09 PM, Fab Tillier wrote: > >> Joshua Hursey wrote on Tue, 31 Aug 2010 at 08:43:24 >> >>> I was thinking more about MPI_Startall() this morning and found a >>> situation where this technique would not work. >>> >>> If the application does: >>> -------------------- >>> MPI_Send_init(rank=1, tag=123, req[0]); >>> MPI_Recv_init(MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, req[1]); >>> MPI_Send_init(rank=2, tag=123, req[2]); >>> >>> MPI_Startall(3, req) // Fails with MPI_ERR_IN_STATUS >>> if( failed ) { >>> for(i=0; i<3; ++i) { >>> MPI_Request_get_status(req[i], flag, status); >>> if( flag && status.error != success ) // Failed >>> if( flag && status == empty ) // Not started >>> if( flag && status != empty ) // Complete >>> } >>> } >>> -------------------- >>> >>> The problem is with the definition of an 'empty' status which has >>> (section 3.7.3): >>> --------------------- >>> source = MPI_ANY_SOURCE >>> tag = MPI_ANY_TAG >>> error = MPI_SUCCESS >>> MPI_Get_count = 0 >>> MPI_test_cancelled = false >>> --------------------- >>> So the successful completion of the MPI_Recv_init() call would be >>> indistinguishable from the 'not started' or inactive state of the call. >> >> Wouldn't a successful completion of the MPI_Recv_init() return a specific source and tag for the message actually received? The source and tag fields of the receive are for filtering incoming sends, but when the receive completes, it was matched to exactly one send, with a specific tag and source. >> >> What am I missing? > > Ah yes. You are correct. So this is not a problem then, and the MPI_Request_get_status() technique would be a good way to check the state of all of the requests. > > Thanks, > Josh > >> >> -Fab >> >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://*lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft >> > > ------------------------------------ > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://*www.*cs.indiana.edu/~jjhursey > > > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://*lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > >

1 0

Re: [Mpi3-ft] Persistent Communication & Sendrecv
by Joshua Hursey 31 Aug '10

31 Aug '10

On Aug 31, 2010, at 1:09 PM, Fab Tillier wrote: > Joshua Hursey wrote on Tue, 31 Aug 2010 at 08:43:24 > >> I was thinking more about MPI_Startall() this morning and found a >> situation where this technique would not work. >> >> If the application does: >> -------------------- >> MPI_Send_init(rank=1, tag=123, req[0]); >> MPI_Recv_init(MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, req[1]); >> MPI_Send_init(rank=2, tag=123, req[2]); >> >> MPI_Startall(3, req) // Fails with MPI_ERR_IN_STATUS >> if( failed ) { >> for(i=0; i<3; ++i) { >> MPI_Request_get_status(req[i], flag, status); >> if( flag && status.error != success ) // Failed >> if( flag && status == empty ) // Not started >> if( flag && status != empty ) // Complete >> } >> } >> -------------------- >> >> The problem is with the definition of an 'empty' status which has >> (section 3.7.3): >> --------------------- >> source = MPI_ANY_SOURCE >> tag = MPI_ANY_TAG >> error = MPI_SUCCESS >> MPI_Get_count = 0 >> MPI_test_cancelled = false >> --------------------- >> So the successful completion of the MPI_Recv_init() call would be >> indistinguishable from the 'not started' or inactive state of the call. > > Wouldn't a successful completion of the MPI_Recv_init() return a specific source and tag for the message actually received? The source and tag fields of the receive are for filtering incoming sends, but when the receive completes, it was matched to exactly one send, with a specific tag and source. > > What am I missing? Ah yes. You are correct. So this is not a problem then, and the MPI_Request_get_status() technique would be a good way to check the state of all of the requests. Thanks, Josh > > -Fab > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://www.cs.indiana.edu/~jjhursey

1 0

Re: [Mpi3-ft] Persistent Communication & Sendrecv
by Fab Tillier 31 Aug '10

31 Aug '10

Joshua Hursey wrote on Tue, 31 Aug 2010 at 08:43:24 > I was thinking more about MPI_Startall() this morning and found a > situation where this technique would not work. > > If the application does: > -------------------- > MPI_Send_init(rank=1, tag=123, req[0]); > MPI_Recv_init(MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, req[1]); > MPI_Send_init(rank=2, tag=123, req[2]); > > MPI_Startall(3, req) // Fails with MPI_ERR_IN_STATUS > if( failed ) { > for(i=0; i<3; ++i) { > MPI_Request_get_status(req[i], flag, status); > if( flag && status.error != success ) // Failed > if( flag && status == empty ) // Not started > if( flag && status != empty ) // Complete > } > } > -------------------- > > The problem is with the definition of an 'empty' status which has > (section 3.7.3): > --------------------- > source = MPI_ANY_SOURCE > tag = MPI_ANY_TAG > error = MPI_SUCCESS > MPI_Get_count = 0 > MPI_test_cancelled = false > --------------------- > So the successful completion of the MPI_Recv_init() call would be > indistinguishable from the 'not started' or inactive state of the call. Wouldn't a successful completion of the MPI_Recv_init() return a specific source and tag for the message actually received? The source and tag fields of the receive are for filtering incoming sends, but when the receive completes, it was matched to exactly one send, with a specific tag and source. What am I missing? -Fab

1 0

Re: [Mpi3-ft] Persistent Communication & Sendrecv
by Joshua Hursey 31 Aug '10

31 Aug '10

On Aug 30, 2010, at 5:35 PM, Joshua Hursey wrote: > I was reviewing the point-to-point chapter today looking for wording/interfaces that need to be addressed in the FT proposal. I came across two sets of interfaces that probably deserve to be brought up on the list. > > Examples and collective stuff coming soon :) > > Persistent Communication: > ------------------------- > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… > > None of the persistent calls have a 'status' object associated with them, so I had to add some wording about failures during MPI_{*Send|Recv}_init and MPI_Start. The interesting case is if MPI_Startall() fails. What should it return (MPI_ERR_IN_STATUS)? How does an application determine which of the requests: > - Started > - Failed > - Were not started > > I think I devised a strategy for the application to check which started and which did not. But I wanted a few more eyes on this to see if there is a better way to deal with this situation. I was thinking more about MPI_Startall() this morning and found a situation where this technique would not work. If the application does: -------------------- MPI_Send_init(rank=1, tag=123, req[0]); MPI_Recv_init(MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, req[1]); MPI_Send_init(rank=2, tag=123, req[2]); MPI_Startall(3, req) // Fails with MPI_ERR_IN_STATUS if( failed ) { for(i=0; i<3; ++i) { MPI_Request_get_status(req[i], flag, status); if( flag && status.error != success ) // Failed if( flag && status == empty ) // Not started if( flag && status != empty ) // Complete } } -------------------- The problem is with the definition of an 'empty' status which has (section 3.7.3): --------------------- source = MPI_ANY_SOURCE tag = MPI_ANY_TAG error = MPI_SUCCESS MPI_Get_count = 0 MPI_test_cancelled = false --------------------- So the successful completion of the MPI_Recv_init() call would be indistinguishable from the 'not started' or inactive state of the call. I see three options (without changing the function interface): 1) Add a field to the MPI_Status object to determine if the request is active or not. MPI_Status.MPI_IS_ACTIVE returns false if inactive, true if active. 2) Change the definition of an 'empty' status to allow the user to distinguish it from the completed MPI_Recv_init request above. (Backwards compatibility issues here). 3) Since MPI_Start and MPI_Startall do not specify what happens when an active request is passed to them (one that has been started, but not completed) we could add some language that works in our benefit. > If an active request is passed to MPI_Start{all} the function will immediately return MPI_ERR_PENDING without starting any of the requests. After an error from MPI_Startall the user could use MPI_Start to distinguish the successful completion from the not started. The problem being that it would automatically start the request that was previously not started. Is this a problem? So the above code would be changed to: -------------------- MPI_Send_init(rank=1, tag=123, req[0]); MPI_Recv_init(MPI_ANY_SOURCE, MPI_ANY_TAG, count=0, req[1]); MPI_Send_init(rank=2, tag=123, req[2]); MPI_Startall(3, req) // Fails with MPI_ERR_IN_STATUS if( failed ) { for(i=0; i<3; ++i) { MPI_Request_get_status(req[i], flag, status); if( flag && status.error != success ) // Failed if( flag ) { ret = MPI_Start(req[i]) if( ret == MPI_ERR_PENDING ) // Complete if( ret == MPI_SUCCESS ) // Previously not started, now active } } } -------------------- Thoughts? Preferences? -- Josh > > > Sendrecv{_replace} > ------------------ > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/ft/run_through_stabilizat… > > These interfaces only have one status object associated with them. So if both ranks fail, only one of them can be indicated in the status object. So I wrote up some text to add to this section for the application to determine what happened. Another set of eyes here would be appreciated just to make sure I am not missing anything. > > > Thanks, > Josh > > ------------------------------------ > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://www.cs.indiana.edu/~jjhursey > > > > > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > ------------------------------------ Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://www.cs.indiana.edu/~jjhursey

1 0

[Mpi3-ft] AUTO: Richard Treumann/Poughkeepsie/IBM is out of the office until 01/02/2001. (returning 09/07/2010)
by Richard Treumann 30 Aug '10

30 Aug '10

I am out of the office until 09/07/2010. I will be out of the office on vacation the week before Labor Day. I will not see any email. Note: This is an automated response to your message "Re: [Mpi3-ft] MPI_Init / MPI_Finalize" sent on 8/30/10 17:35:39. This is the only notification you will receive while this person is away.

1 0