July 2008 - Mpi-22 - mailman3.cels.anl.gov

Re: [Mpi-22] MPI-2.1 ambiguity?
by unknown＠example.com 16 Jul '08

16 Jul '08

On Jul 16, 2008, at 11:24 AM, William Gropp wrote: > One piece of relevant text in MPI 2.0 is in section 4.12.6 (in > MPI-2.0) says that "All predefined datatypes can be used in datatype > constructors in any language." However, the example uses > MPI_Type_f2c ton convert the Fortran handle to the C version before > passing it into the MPI_Type_create_struct routine, so this text > could be interpreted as allowing the use of those datatypes through > the handle conversion mechanism without requiring them to be defined > in each languages header file. Presumably the requirement is that > they be predefined with the value that you'd get from the handle > conversion function? The text in Annex B is stronger than that, though. It specifically states "they are therefore defined for all three language bindings" -- is that a mistake? >>> 2. Section 3.2.2 on page 27, Section 16.1.6 on page 453, and Annex >>> A.1 >>> on page 491. >>> MPI_LONG_LONG_INT, MPI_LONG_LONG (as synonym), >>> MPI_UNSIGNED_LONG_LONG, >>> MPI_SIGNED_CHAR, and MPI_WCHAR are moved from optional to oï¬cial >>> and they >>> are therefore deï¬ned for all three language bindings. -- Jeff Squyres Cisco Systems

1 0

Re: [Mpi-22] [Mpi-forum] MPI 2.2 proposal:resolving MPI_Request_free issues
by unknown＠example.com 16 Jul '08

16 Jul '08

On Jul 16, 2008, at 7:31 PM, Underwood, Keith D wrote: >>> [erezh] Correct; this is the scenario I was describing with the RDMA > write. > > It would be interesting to see exactly what the error mode here is. > Retransmitting corrupted data should be ok, since a correctly > delivered > message means that the retransmit must be dropped. I suppose that if > the NIC speaks virtual addresses and the free actually results in a > trap > to the kernel that unmaps the pages, then the NIC could retransmit and > find that there isn't a valid page table entry... 'zactly. And then the local completion entry would be a failure -- so the sender would [erroneously] think that the message had failed to be delivered. > Solution #5: Change the advice to users - "...the arrival of the > reply > informs the sender that the send has completed and the send buffer can > be overwritten. If the buffer will ever be freed, the application > should call MPI_Wait or MPI_Cancel instead of MPI_Request_free." That seems like a good compromise. -- Jeff Squyres Cisco Systems

1 0

Re: [Mpi-22] [Mpi-forum] MPI 2.2 proposal:resolving MPI_Request_free issues
by unknown＠example.com 16 Jul '08

16 Jul '08

>> I don't quite understand examples 1 and 2 (how would they cause segv's >> in the TCP stack). It is permissible to (pseudocode): >> >> while (bytes_to_send > 0) { >> rc = write(fd, buffer, bytes_to_send); >> if (rc > 0) { >> buffer += rc; >> bytes_to_send -= rc; >> } else { >> ...error... >> } >> } >> free(buffer); >> >> regardless of what the receiver does. I'm not a kernel guy; does >> updating TCP sequence numbers also interact with the payload buffer? > >[erezh] it will never happen with your code above; but you are not using >async zcopy. >The pattern in windows is to use overlapped send (write) which is still >active when the function returns, and is the most efficient way to send >your buffer. I know it's possible with Linux but I don't' have the exact >pattern. So, how does this work for other users of zero copy? How do they know when the send buffer is truly free? >> FWIW: I can see the RMA interconnect example much easier. You can >> imagine a scenario where a sender successfully sends and the receiver >> successfully receives, but the hardware ACK from the receiver gets >> lost. The receiver then sends an MPI message back to the sender, but >> the sender is still in the middle of a retransmit timeout (while >> waiting for the hardware ACK that was lost). In this case, the user >> app may free the buffer too soon, resulting in a segv (or some other >> lion, tiger, or bear) when the sending hardware tries to retransmit. >> > >[erezh] Correct; this is the scenario I was describing with the RDMA write. It would be interesting to see exactly what the error mode here is. Retransmitting corrupted data should be ok, since a correctly delivered message means that the retransmit must be dropped. I suppose that if the NIC speaks virtual addresses and the free actually results in a trap to the kernel that unmaps the pages, then the NIC could retransmit and find that there isn't a valid page table entry... >> MPI_Request_free() is used in application programs. For example, it is >> the easiest (and portable) way to send a non-blocking acknowledge >> to a destination process. > >[erezh] See Adam Moody proposal for MPI_REQUEST_IGNORE; I think it's safer >than the current pattern > >> >> MPI_Isend (buf, 0, MPI_INT, dest, TAG_ACK, comm, lrequest) >> MPI_Request_free (lrequest) Hrm. This example doesn't seem to trigger Erez's concern... That's a zero length buffer, right? Solution #5: Change the advice to users - "...the arrival of the reply informs the sender that the send has completed and the send buffer can be overwritten. If the buffer will ever be freed, the application should call MPI_Wait or MPI_Cancel instead of MPI_Request_free." If you have received your application level ack from the remote side, MPI_Cancel should give you the equivalent functionality of MPI_Request_free - allowing you to release the request without having to wait for local completion. I'm not entirely sure WHY you would want to do this, but... ;-) Keith

1 0

Re: [Mpi-22] MPI-2.1 ambiguity?
by unknown＠example.com 16 Jul '08

16 Jul '08

What's about the C++ datatypes for complex ? MPI_COMPLEX and MPI_Double_Complex are already defined for Fortran datatypes. The complex Fortran datatypes within C++ have a special name in order to avoid a name collision. Hubert William Gropp wrote: > One piece of relevant text in MPI 2.0 is in section 4.12.6 (in > MPI-2.0) says that "All predefined datatypes can be used in datatype > constructors in any language." However, the example uses MPI_Type_f2c > ton convert the Fortran handle to the C version before passing it into > the MPI_Type_create_struct routine, so this text could be interpreted > as allowing the use of those datatypes through the handle conversion > mechanism without requiring them to be defined in each languages > header file. Presumably the requirement is that they be predefined > with the value that you'd get from the handle conversion function? > > Bill > > On Jul 16, 2008, at 8:18 AM, Hubert Ritzdorf wrote: > >> Hi, >> >> when I remember correctly, it was decided to provide the C MPI datatypes >> in Fortran since >> the Fortran datatypes are supported in C and C++. >> >> Hubert >> >> Jeff Squyres wrote: >>> I will file a proposal about this if there is a problem, but I wanted >>> to ask the group before I did so. In the changelog for MPI-2.1: >>> >>> 2. Section 3.2.2 on page 27, Section 16.1.6 on page 453, and Annex A.1 >>> on page 491. >>> MPI_LONG_LONG_INT, MPI_LONG_LONG (as synonym), MPI_UNSIGNED_LONG_LONG, >>> MPI_SIGNED_CHAR, and MPI_WCHAR are moved from optional to oï¬cial and >>> they >>> are therefore deï¬ned for all three language bindings. >>> >>> Note that it says that these C datatype names must be in all three >>> language bindings. >>> >>> Are C MPI datatypes supposed to be available in Fortran? I looked in >>> mpif.h for Open MPI, MPICH2, HP MPI, and Intel MPI, and I didn't see C >>> MPI datatypes in there (e.g., MPI_INT and MPI_FLOAT). >>> >>> So: >>> >>> - is there language somewhere in MPI that says that C datatypes are >>> not supposed to be in mpif.h? (and therefore the MPI-2.1 changelog is >>> wrong) >>> >>> or >>> >>> - are all 4 of the above MPI's wrong for not including the C datatypes >>> in mpif.h? >>> >>> --Jeff Squyres >>> Cisco Systems >>> >>> >>> _______________________________________________ >>> mpi-22 mailing list >>> mpi-22_at_[hidden] <mailto:mpi-22_at_[hidden]> >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 >>> >>> <smime.p7s><ATT00001.txt> > > William Gropp > Paul and Cynthia Saylor Professor of Computer Science > University of Illinois Urbana-Champaign > > > > ------------------------------------------------------------------------ > > _______________________________________________ > mpi-22 mailing list > mpi-22_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 > *

1 0

Re: [Mpi-22] MPI-2.1 ambiguity?
by unknown＠example.com 16 Jul '08

16 Jul '08

One piece of relevant text in MPI 2.0 is in section 4.12.6 (in MPI-2.0) says that "All predefined datatypes can be used in datatype constructors in any language." However, the example uses MPI_Type_f2c ton convert the Fortran handle to the C version before passing it into the MPI_Type_create_struct routine, so this text could be interpreted as allowing the use of those datatypes through the handle conversion mechanism without requiring them to be defined in each languages header file. Presumably the requirement is that they be predefined with the value that you'd get from the handle conversion function? Bill On Jul 16, 2008, at 8:18 AM, Hubert Ritzdorf wrote: > Hi, > > when I remember correctly, it was decided to provide the C MPI > datatypes > in Fortran since > the Fortran datatypes are supported in C and C++. > > Hubert > > Jeff Squyres wrote: >> I will file a proposal about this if there is a problem, but I wanted >> to ask the group before I did so. In the changelog for MPI-2.1: >> >> 2. Section 3.2.2 on page 27, Section 16.1.6 on page 453, and Annex >> A.1 >> on page 491. >> MPI_LONG_LONG_INT, MPI_LONG_LONG (as synonym), >> MPI_UNSIGNED_LONG_LONG, >> MPI_SIGNED_CHAR, and MPI_WCHAR are moved from optional to oï¬cial >> and they >> are therefore deï¬ned for all three language bindings. >> >> Note that it says that these C datatype names must be in all three >> language bindings. >> >> Are C MPI datatypes supposed to be available in Fortran? I looked in >> mpif.h for Open MPI, MPICH2, HP MPI, and Intel MPI, and I didn't >> see C >> MPI datatypes in there (e.g., MPI_INT and MPI_FLOAT). >> >> So: >> >> - is there language somewhere in MPI that says that C datatypes are >> not supposed to be in mpif.h? (and therefore the MPI-2.1 changelog is >> wrong) >> >> or >> >> - are all 4 of the above MPI's wrong for not including the C >> datatypes >> in mpif.h? >> >> --Jeff Squyres >> Cisco Systems >> >> >> _______________________________________________ >> mpi-22 mailing list >> mpi-22_at_[hidden] >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 >> >> <smime.p7s><ATT00001.txt> William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign *

1 0

Re: [Mpi-22] MPI-2.1 ambiguity?
by unknown＠example.com 16 Jul '08

16 Jul '08

Hi, when I remember correctly, it was decided to provide the C MPI datatypes in Fortran since the Fortran datatypes are supported in C and C++. Hubert Jeff Squyres wrote: > I will file a proposal about this if there is a problem, but I wanted > to ask the group before I did so. In the changelog for MPI-2.1: > > 2. Section 3.2.2 on page 27, Section 16.1.6 on page 453, and Annex A.1 > on page 491. > MPI_LONG_LONG_INT, MPI_LONG_LONG (as synonym), MPI_UNSIGNED_LONG_LONG, > MPI_SIGNED_CHAR, and MPI_WCHAR are moved from optional to oï¬cial and they > are therefore deï¬ned for all three language bindings. > > Note that it says that these C datatype names must be in all three > language bindings. > > Are C MPI datatypes supposed to be available in Fortran? I looked in > mpif.h for Open MPI, MPICH2, HP MPI, and Intel MPI, and I didn't see C > MPI datatypes in there (e.g., MPI_INT and MPI_FLOAT). > > So: > > - is there language somewhere in MPI that says that C datatypes are > not supposed to be in mpif.h? (and therefore the MPI-2.1 changelog is > wrong) > > or > > - are all 4 of the above MPI's wrong for not including the C datatypes > in mpif.h? > > --Jeff Squyres > Cisco Systems > > > _______________________________________________ > mpi-22 mailing list > mpi-22_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 > *

1 0

MPI-2.1 ambiguity?
by unknown＠example.com 15 Jul '08

15 Jul '08

I will file a proposal about this if there is a problem, but I wanted to ask the group before I did so. In the changelog for MPI-2.1: 2. Section 3.2.2 on page 27, Section 16.1.6 on page 453, and Annex A.1 on page 491. MPI_LONG_LONG_INT, MPI_LONG_LONG (as synonym), MPI_UNSIGNED_LONG_LONG, MPI_SIGNED_CHAR, and MPI_WCHAR are moved from optional to oï¬cial and they are therefore deï¬ned for all three language bindings. Note that it says that these C datatype names must be in all three language bindings. Are C MPI datatypes supposed to be available in Fortran? I looked in mpif.h for Open MPI, MPICH2, HP MPI, and Intel MPI, and I didn't see C MPI datatypes in there (e.g., MPI_INT and MPI_FLOAT). So: - is there language somewhere in MPI that says that C datatypes are not supposed to be in mpif.h? (and therefore the MPI-2.1 changelog is wrong) or - are all 4 of the above MPI's wrong for not including the C datatypes in mpif.h? -- Jeff Squyres Cisco Systems

1 0

Re: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues
by unknown＠example.com 15 Jul '08

15 Jul '08

See inline -----Original Message----- From: mpi-22-bounces_at_[hidden] [mailto:mpi-22-bounces_at_[hidden]] On Behalf Of Hubert Ritzdorf Sent: Tuesday, July 15, 2008 3:11 AM To: MPI 2.2 Subject: Re: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues Jeff Squyres wrote: > On Jul 14, 2008, at 5:50 PM, Erez Haba wrote: > >> Issue #1: >> Advice to user quote: >> >> "Once a request is freed by a call to MPI_REQUEST_FREE, it is not >> possible to check for the successful completion of the associated >> communication with calls to MPI_WAIT or MPI_TEST. Also, if an error >> occurs subsequently during the communication, an error code cannot be >> returned to the user - such an error must be treated as fatal." >> >> This is the only place in the MPI standard that mandates an error to >> be FATAL, regardless of the user settings. This is truly >> unrecoverable because the user can not associate the error with the >> failed send and cannot recover after MPI_Request_free was called. >> This poses a problem for a fault-tolerance implementation as it must >> handle this failure without the ability to notify the user for the >> specific error for the lack of context. > > I'm not sure I agree with this premise. If you need this > functionality, then you shouldn't be using MPI_REQUEST_FREE. > Logically speaking, if you want to associate a specific error with a > specific communication request, then you must have something to tie > the error *to*. In this case, it's a request -- but the application > has explicitly stated that it no longer cares about the request. > > Therefore: if you care about the request, don't free it. > [erezh] The points I am making are. 1. This is the only location in the standard where it mandates the error to be FATAL 2. using this function creates a problem for fault-tolerance implementations. (i.e., how would the FT code would know if to make the MPI_Isend FT? the request free can happen any time in the future) >> Issue #2: >> Advice to user quote: >> >> "Questions arise as to how one knows when the operations have >> completed when using [snip] >> causes the send to fail. > > I don't quite understand examples 1 and 2 (how would they cause segv's > in the TCP stack). It is permissible to (pseudocode): > > while (bytes_to_send > 0) { > rc = write(fd, buffer, bytes_to_send); > if (rc > 0) { > buffer += rc; > bytes_to_send -= rc; > } else { > ...error... > } > } > free(buffer); > > regardless of what the receiver does. I'm not a kernel guy; does > updating TCP sequence numbers also interact with the payload buffer? [erezh] it will never happen with your code above; but you are not using async zcopy. The pattern in windows is to use overlapped send (write) which is still active when the function returns, and is the most efficient way to send your buffer. I know it's possible with Linux but I don't' have the exact pattern. > > FWIW: I can see the RMA interconnect example much easier. You can > imagine a scenario where a sender successfully sends and the receiver > successfully receives, but the hardware ACK from the receiver gets > lost. The receiver then sends an MPI message back to the sender, but > the sender is still in the middle of a retransmit timeout (while > waiting for the hardware ACK that was lost). In this case, the user > app may free the buffer too soon, resulting in a segv (or some other > lion, tiger, or bear) when the sending hardware tries to retransmit. > [erezh] Correct; this is the scenario I was describing with the RDMA write. > Don't get me wrong; I'm not a fan of MPI_REQUEST_FREE either. :-) > >> Proposed Solution >> 3 proposals from the least restrictive to the most restrictive: >> >> Solution #1: >> Remove the advice to user to reuse the buffer once a reply has >> arrived. There is no safe way to reuse the buffer (free), overwrite >> is somewhat safer. >> >> Solution #2: >> Remove the advice to user altogether, disallow the usage pattern of >> freeing active requests. Only inactive requests are allowed to be >> freed. (i.e., not started). >> >> Solution #3: >> Deprecate MPI_Request_free. Users can always use MPI_Wait to complete >> the request. >> >> Recommendation: >> Use solution #2, as users still need to free requests if they are not >> used; e.g., the app called MPI_Send_init but never got the start that >> request; hence the request still need to be freed. > > > I'm not an apps guy, but I thought there were real world apps out > there that use MPI_REQUEST_FREE. So #2 would break real apps -- but I > have no idea how many. [erezh] correct, it possibly can break existing applications as mentioned in the "impact on user" section. > MPI_Request_free() is used in application programs. For example, it is > the easiest (and portable) way to send a non-blocking acknowledge > to a destination process. [erezh] See Adam Moody proposal for MPI_REQUEST_IGNORE; I think it's safer than the current pattern > > MPI_Isend (buf, 0, MPI_INT, dest, TAG_ACK, comm, lrequest) > MPI_Request_free (lrequest) > > I know applications which are transferring (long living) buffers to > destination processes in broker/request parts of application > and using MPI_Request_free(). [erezh] longed lived buffers is okay (at least they don't free it); how do they know when it's safe to update the send buffer? (there is an issue of correctness here) > By the way, I think that Issue 2 is independent on MPI_Request_free(). > "/the arrival of the reply informs the sender that the send has > completed and the send buffer can be reused"/ is not necessarily related > to the usage of MPI_Request_free(). > // [erezh] I agree; however this pattern is suggested only in the MPI_Request_free() advice to users. > > Perhaps an alternative solution would be: > > Solution #4: Deprecate MPI_REQUEST_FREE so that it can actually be > removed someday. State that requests that are freed can never know > when it is safe to free the corresponding buffer (thus making > MPI_REQUEST_FREE so unattractive that its use tapers off, and also > making it perfectly permissible for an MPI implementation to segv if a > user frees a buffer associated with a REQUEST_FREE'd request --- such > as the scenarios described above -- because that would be an erroneous > program :-) ). > > MPI_Request_free() is used, provides a required functionality and should > not be deprecated. > > Hubert

1 0

Re: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues
by unknown＠example.com 15 Jul '08

15 Jul '08

On Jul 15, 2008, at 6:11 AM, Hubert Ritzdorf wrote: > Freeing user buffers which are still in use by an interconnect is > another topic. What's about the receive buffer ? The receive buffer > could also already be freed or re-used in another way directly after > the receive was finished. That should be fine. > Does the retransmit to the receive buffer not cause corresponding > problems ? I would think that the hardware should recognize it as a duplicate message and drop it (therefore not causing a problem with the receive buffer). After all, in my example, only the receiver's ACK was lost -- the message was otherwise already successfully received. > I think that the receive cannot return before the transfer is > finished and the user buffers can be reused. Agreed. -- Jeff Squyres Cisco Systems

1 0

Re: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues
by unknown＠example.com 15 Jul '08

15 Jul '08

Jeff Squyres wrote: > On Jul 14, 2008, at 5:50 PM, Erez Haba wrote: > >> Issue #1: >> Advice to user quote: >> >> Once a request is freed by a call to MPI_REQUEST_FREE, it is not >> possible to check for the successful completion of the associated >> communication with calls to MPI_WAIT or MPI_TEST. Also, if an error >> occurs subsequently during the communication, an error code cannot be >> returned to the user  such an error must be treated as fatal. >> >> This is the only place in the MPI standard that mandates an error to >> be FATAL, regardless of the user settings. This is truly >> unrecoverable because the user can not associate the error with the >> failed send and cannot recover after MPI_Request_free was called. >> This poses a problem for a fault-tolerance implementation as it must >> handle this failure without the ability to notify the user for the >> specific error for the lack of context. > > I'm not sure I agree with this premise. If you need this > functionality, then you shouldn't be using MPI_REQUEST_FREE. > Logically speaking, if you want to associate a specific error with a > specific communication request, then you must have something to tie > the error *to*. In this case, it's a request -- but the application > has explicitly stated that it no longer cares about the request. > > Therefore: if you care about the request, don't free it. > >> Issue #2: >> Advice to user quote: >> >> Questions arise as to how one knows when the operations have >> completed when using [snip] >> causes the send to fail. > > I don't quite understand examples 1 and 2 (how would they cause segv's > in the TCP stack). It is permissible to (pseudocode): > > while (bytes_to_send > 0) { > rc = write(fd, buffer, bytes_to_send); > if (rc > 0) { > buffer += rc; > bytes_to_send -= rc; > } else { > ...error... > } > } > free(buffer); > > regardless of what the receiver does. I'm not a kernel guy; does > updating TCP sequence numbers also interact with the payload buffer? > > FWIW: I can see the RMA interconnect example much easier. You can > imagine a scenario where a sender successfully sends and the receiver > successfully receives, but the hardware ACK from the receiver gets > lost. The receiver then sends an MPI message back to the sender, but > the sender is still in the middle of a retransmit timeout (while > waiting for the hardware ACK that was lost). In this case, the user > app may free the buffer too soon, resulting in a segv (or some other > lion, tiger, or bear) when the sending hardware tries to retransmit. > > Don't get me wrong; I'm not a fan of MPI_REQUEST_FREE either. :-) > >> Proposed Solution >> 3 proposals from the least restrictive to the most restrictive: >> >> Solution #1: >> Remove the advice to user to reuse the buffer once a reply has >> arrived. There is no safe way to reuse the buffer (free), overwrite >> is somewhat safer. >> >> Solution #2: >> Remove the advice to user altogether, disallow the usage pattern of >> freeing active requests. Only inactive requests are allowed to be >> freed. (i.e., not started). >> >> Solution #3: >> Deprecate MPI_Request_free. Users can always use MPI_Wait to complete >> the request. >> >> Recommendation: >> Use solution #2, as users still need to free requests if they are not >> used; e.g., the app called MPI_Send_init but never got the start that >> request; hence the request still need to be freed. > > > I'm not an apps guy, but I thought there were real world apps out > there that use MPI_REQUEST_FREE. So #2 would break real apps -- but I > have no idea how many. MPI_Request_free() is used in application programs. For example, it is the easiest (and portable) way to send a non-blocking acknowledge to a destination process. MPI_Isend (buf, 0, MPI_INT, dest, TAG_ACK, comm, lrequest) MPI_Request_free (lrequest) I know applications which are transferring (long living) buffers to destination processes in broker/request parts of application and using MPI_Request_free(). By the way, I think that Issue 2 is independent on MPI_Request_free(). "/the arrival of the reply informs the sender that the send has completed and the send buffer can be reused"/ is not necessarily related to the usage of MPI_Request_free(). // > > Perhaps an alternative solution would be: > > Solution #4: Deprecate MPI_REQUEST_FREE so that it can actually be > removed someday. State that requests that are freed can never know > when it is safe to free the corresponding buffer (thus making > MPI_REQUEST_FREE so unattractive that its use tapers off, and also > making it perfectly permissible for an MPI implementation to segv if a > user frees a buffer associated with a REQUEST_FREE'd request --- such > as the scenarios described above -- because that would be an erroneous > program :-) ). > MPI_Request_free() is used, provides a required functionality and should not be deprecated. Freeing user buffers which are still in use by an interconnect is another topic. What's about the receive buffer ? The receive buffer could also already be freed or re-used in another way directly after the receive was finished. Does the retransmit to the receive buffer not cause corresponding problems ? I think that the receive cannot return before the transfer is finished and the user buffers can be reused. Hubert *

1 0