February 2009 - mpiwg-coll - mailman3.cels.anl.gov

Re: [mpi3-coll] Simplify "local completion" to just "completion"
by Adam Moody 02 Feb '09

02 Feb '09

Ok, just a bit more and then I'll back down ;-) From what I've read, I think things are different in point-to-point because there are three different send modes (standard, buffered, or synchronous) that have different guarantees on whether completion is local or not (up to the MPI implementation, guaranteed local, guaranteed non-local, respectively). So far with collectives, we only have one completion mode (up to the MPI implementation). In fact, this very difference is why I find the phrase "local completion" to be confusing with collectives. As a reader, it sounds like there may be a mechanism for non-local completion (like MPI_Isbcast or something), but there is not (at least not yet). We have only one completion mode. I understand the intent here is to help the reader by stressing that when a collective completes at a process, there are no guarantess about the completion status of other processes. However, I think this is quite clear with our current text: p 2, lines 20-23 "The completion of a collective operation indicates that the caller is free to access locations in the communication buffer. It does not indicate that other processes in the group have completed or even started the operation (unless otherwise implied by the description of the operation)." and restated on p 50, lines 4-9 "Similarly to the blocking case, collective operations are considered to be complete when the local part of the operation is finished, i.e., the semantics of the operation are guaranteed and all buffers can be safely accessed and modified. Completion does not indicate that other processes have completed or even started the operation (unless otherwise implied by the description of the operation)." Personally, I just find the extra "local" keywords to be distracting. -Adam Bronis R. de Supinski wrote: >Adam: > >Local completion is the term used in the point-to-point >discussions. In particular, even MPI_Send has "local >completion semantics", which means that the other side >of the operation may not have completed. I think we should >retain it since otherwise one can take it that the >entire operation must complete (i.e., in all tasks). > >Bronis > > >On Mon, 2 Feb 2009, Adam Moody wrote: > > > >>I think we could reduce "local completion" to just "completion" >>throughout the text. I don't think the adjective "local" is necessary >>and in some ways I find it confusing. We define precisely what >>"completion" means in a collective, and we also explicitly note that >>completion at a process does not indicate the status of other >>processes. Whenever we say "local completion", we really mean >>"completion" by our definition, so "local" is not necessary here. And >>by saying "local" completion, it sounds as a reader that there may be >>such a thing as "non-local" completion, but we don't define it. There >>is only one definition for collective completion. >>-Adam >> >>p 2, line 17 in "Collective operations can (but are not required to) >>complete locally as soon as the caller's participation in the collective >>communication is finished." >> "complete locally" --> "complete" >> >>p 2, line 20 in "The local completion of a collective operation >>indicates that the caller is free to access locations in the >>communication buffer." >> "The local completion" --> "The completion" >> >>p 49, line 48 & p 50, line 1 in "Once initiated, all associated send >>buffers should not be modified and all associated receive buffers should >>not be accessed until the collective operation completes locally." >> "until the collective operation completes locally." --> "until the >>collective operation completes." >> >>p 51, line 32 in "On intracommunicators, the operation completes locally >>after every process in the communicator called MPI_IBARRIER." >> "the operation completes locally" --> "the operation completes" >>and (to preserve verb tense) >> "after every process in the communicator called" --> "after every >>process in the communicator calls" >>_______________________________________________ >>mpi3-coll mailing list >>mpi3-coll(a)lists.mpi-forum.org >>http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll >> >> >> >> >_______________________________________________ >mpi3-coll mailing list >mpi3-coll(a)lists.mpi-forum.org >http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll > > > >

1 0

Re: [mpi3-coll] Simplify "local completion" to just "completion"
by Bronis R. de Supinski 02 Feb '09

02 Feb '09

Adam: Local completion is the term used in the point-to-point discussions. In particular, even MPI_Send has "local completion semantics", which means that the other side of the operation may not have completed. I think we should retain it since otherwise one can take it that the entire operation must complete (i.e., in all tasks). Bronis On Mon, 2 Feb 2009, Adam Moody wrote: > I think we could reduce "local completion" to just "completion" > throughout the text. I don't think the adjective "local" is necessary > and in some ways I find it confusing. We define precisely what > "completion" means in a collective, and we also explicitly note that > completion at a process does not indicate the status of other > processes. Whenever we say "local completion", we really mean > "completion" by our definition, so "local" is not necessary here. And > by saying "local" completion, it sounds as a reader that there may be > such a thing as "non-local" completion, but we don't define it. There > is only one definition for collective completion. > -Adam > > p 2, line 17 in "Collective operations can (but are not required to) > complete locally as soon as the caller's participation in the collective > communication is finished." > "complete locally" --> "complete" > > p 2, line 20 in "The local completion of a collective operation > indicates that the caller is free to access locations in the > communication buffer." > "The local completion" --> "The completion" > > p 49, line 48 & p 50, line 1 in "Once initiated, all associated send > buffers should not be modified and all associated receive buffers should > not be accessed until the collective operation completes locally." > "until the collective operation completes locally." --> "until the > collective operation completes." > > p 51, line 32 in "On intracommunicators, the operation completes locally > after every process in the communicator called MPI_IBARRIER." > "the operation completes locally" --> "the operation completes" > and (to preserve verb tense) > "after every process in the communicator called" --> "after every > process in the communicator calls" > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll > >

1 0

[mpi3-coll] specified in or implied by
by Adam Moody 02 Feb '09

02 Feb '09

Several folks mentioned that this line in the NBC intro is wordy: "Completion does not imply that other processes have completed or even started the operation unless otherwise specified in, or implied by, the description of the operation." I think a good solution here is to borrow from the existing collectives intro (see below). -Adam From p 2, lines 22-24 we have "It does not indicate that other processes in the group have completed or even started the operation (unless otherwise implied by in the description of the operation)." First, drop the extra preposition in this text (typo from original) "implied by in" --> "implied by" Second, borrow this wording for NBC on p 50, lines 7-8. Change: "Completion does not imply that other processes have completed or even started the operation unless otherwise specified in, or implied by, the description of the operation." To: "Completion does not indicate that other processes have completed or even started the operation (unless otherwise implied by the description of the operation.)"

1 0

[mpi3-coll] Simplify "local completion" to just "completion"
by Adam Moody 02 Feb '09

02 Feb '09

I think we could reduce "local completion" to just "completion" throughout the text. I don't think the adjective "local" is necessary and in some ways I find it confusing. We define precisely what "completion" means in a collective, and we also explicitly note that completion at a process does not indicate the status of other processes. Whenever we say "local completion", we really mean "completion" by our definition, so "local" is not necessary here. And by saying "local" completion, it sounds as a reader that there may be such a thing as "non-local" completion, but we don't define it. There is only one definition for collective completion. -Adam p 2, line 17 in "Collective operations can (but are not required to) complete locally as soon as the caller's participation in the collective communication is finished." "complete locally" --> "complete" p 2, line 20 in "The local completion of a collective operation indicates that the caller is free to access locations in the communication buffer." "The local completion" --> "The completion" p 49, line 48 & p 50, line 1 in "Once initiated, all associated send buffers should not be modified and all associated receive buffers should not be accessed until the collective operation completes locally." "until the collective operation completes locally." --> "until the collective operation completes." p 51, line 32 in "On intracommunicators, the operation completes locally after every process in the communicator called MPI_IBARRIER." "the operation completes locally" --> "the operation completes" and (to preserve verb tense) "after every process in the communicator called" --> "after every process in the communicator calls"

1 0

[mpi3-coll] More on in-place
by Adam Moody 02 Feb '09

02 Feb '09

I think we need another statement about the in-place option in the text regarding buffer access. Something along the lines of: When using the "in-place" option, message buffers may function as both send and receive buffers. Such buffers should not be modified or accessed until the operation completes. This is somewhat unique to NBC. However, this statement is necessary for blocking collectives in threaded environments, as well. Did the "Send buffer access" ticket resolve this? Either way, it might not hurt to reiterate this point in the NBC intro, since it is more likely to come up there. -Adam

1 0

Re: [mpi3-coll] New nonblocking collective intro text
by Adam Moody 02 Feb '09

02 Feb '09

Thanks for your review, Christian. I personally had trouble parsing the whole picture in the original text. I know it's a fairly big change, but it's mostly just reordering of the existing content. I felt it was important to do since this introduces new readers to the NBC concept. -Adam Christian Siebert wrote: > Dear Adam and all, > > I very much appreciate many of your textual improvements. However, it > is very time consuming for us as reviewers to keep track of such > substantial changes. It would have been much easier, if you had > contributed your changes in an earlier stage. Anyway, I waded through > the changes to revision 3 and will comment only on the things that I > don't like (consider everything else as excellent): > > I second Torsten's opinion on p. 49, l. 35-37: This reformulation > introduces three times the term "complete". Besides this drawback, I > also don't see any benefit from this reformulation compared to the > previous version. It's a bit repetitive, I know. To make a direct parallel, I copied the same sentence structure used in the nonblocking point-to-point section (see the 2.1 document, page 47, lines 46-48). We could drop the "but does not complete it" clause to shorten this some (saves one "complete"): "A nonblocking start call initiates a collective operation, which must be completed in a separate completion call." > > p. 50, l. 1-2: Similar problem (i.e. 2x"complete"). May I propose: > "The start call returns a request handle, which is eventually passed > to a completion call." I'd be ok with dropping a "complete" here, as well, although I prefer "which must be passed" over "which is eventually passed": "The start call returns a request handle, which must be passed to a completion call." > > > p. 50, l. 8: The part "unless otherwise specified in, or implied by, > the description of the operation" has been added. Seem ok to me at > first glance. However, we might re-check if this is really necessary > and drop it otherwise. I added the "implied by" to address cases like MPI_Alltoall, where there is no specified synchronization as in MPI_Barrier, but clearly a process can't complete until it has received data from every other process and thus knows all other processes have started the call. I have no objections to dropping one of these two phrases. Going with just "implied by" leads to a cyclic-sounding statement. "Completion does not imply that other processes have completed or even started the operation unless otherwise implied by the description of the operation." We could just go with "specified in": "Completion does not imply that other processes have completed or even started the operation unless otherwise specified in the description of the operation." > > p. 50, l. 16: This looks quite redundant to me. I'd propose sth. like > the following restatement: "Upon completion of a nonblocking > collective operation, the MPI_ERROR field...". Yes, that's fine. > > p. 50, l. 17: The added postfix "to indicate any errors" seems wrong > to me because MPI_SUCCESS is also a valid entry for the MPI_ERROR > field. I'd propose to simply omit this postfix. However, I'm not 100% > sure because the section about the MPI Status object looks quite > confusing in this respect. Btw. there exists also an empty status > (which sets all 3 fields) - so we might need to reconsider our > decision about setting only the error field for nonblocking collective > operations. Yes, I agree, but I'll leave this alone as it is a bigger discussion for the group. > > p. 50, l. 29: "nonblocking collective communications" => "nonblocking > collective operations" Sure. > > p. 50, l. 44: You changed "running time" to "minimal time to > completion" - this is _not_ the same thing! I'd strongly prefer to > stay with the previous terminology. I meant to go with whatever we had after the telecon here. This statement is only trying to make a point to the reader that there are reasonably two different algorithms possible here. It does not demand what an MPI implemenation must do. I'm happy with either wording. > > p. 62, l. 36/37: now we have two consecutive "the"s Yep, nice catch. > > > > Many thanks! > > Christian > >

1 0

Re: [mpi3-coll] New nonblocking collective intro text
by Christian Siebert 02 Feb '09

02 Feb '09

Dear Adam and all, I very much appreciate many of your textual improvements. However, it is very time consuming for us as reviewers to keep track of such substantial changes. It would have been much easier, if you had contributed your changes in an earlier stage. Anyway, I waded through the changes to revision 3 and will comment only on the things that I don't like (consider everything else as excellent): I second Torsten's opinion on p. 49, l. 35-37: This reformulation introduces three times the term "complete". Besides this drawback, I also don't see any benefit from this reformulation compared to the previous version. p. 50, l. 1-2: Similar problem (i.e. 2x"complete"). May I propose: "The start call returns a request handle, which is eventually passed to a completion call." p. 50, l. 8: The part "unless otherwise specified in, or implied by, the description of the operation" has been added. Seem ok to me at first glance. However, we might re-check if this is really necessary and drop it otherwise. p. 50, l. 16: This looks quite redundant to me. I'd propose sth. like the following restatement: "Upon completion of a nonblocking collective operation, the MPI_ERROR field...". p. 50, l. 17: The added postfix "to indicate any errors" seems wrong to me because MPI_SUCCESS is also a valid entry for the MPI_ERROR field. I'd propose to simply omit this postfix. However, I'm not 100% sure because the section about the MPI Status object looks quite confusing in this respect. Btw. there exists also an empty status (which sets all 3 fields) - so we might need to reconsider our decision about setting only the error field for nonblocking collective operations. p. 50, l. 29: "nonblocking collective communications" => "nonblocking collective operations" p. 50, l. 44: You changed "running time" to "minimal time to completion" - this is _not_ the same thing! I'd strongly prefer to stay with the previous terminology. p. 62, l. 36/37: now we have two consecutive "the"s Many thanks! Christian -- Christian Siebert, Dipl.-Inf. Research Associate NEC Laboratories Europe, NEC Europe Ltd. Rathausallee 10, D-53757 Sankt Augustin, Germany Phone: +49 (0) 2241 - 92 52 44 Fax: +49 (0) 2241 - 92 52 99 (Registered Office: 1 Victoria Road, London W3 6BL, 2832014)

1 0

Re: [mpi3-coll] New nonblocking collective intro text
by Jesper Larsson Traeff 02 Feb '09

02 Feb '09

Dear Adam, I think it reads well (have two comments below), but not sure how much it improves over the current text, and did not yet find time to compare and think sorry for this useless remark Jesper On Thu, Jan 29, 2009 at 11:39:12AM -0800, Adam Moody wrote: > Here is the updated text after incorporating the fixes Bronis caught. > Thanks, Bronis. > -Adam > > > As described in Section ?? (Section 3.7), performance on many systems > can be improved by overlapping communication and computation. > Nonblocking collectives combine the potential benefits of nonblocking > point-to-point operations to exploit overlap and to avoid > synchronization with the optimized implementation and message scheduling I'm not sure "optimized implementation" here really conveyes to the user the advantage of NBCs > provided by collective operations [1,4]. One way of doing this would be > to perform a blocking collective operation in a separate thread. An > alternative mechanism that often leads to better performance (e.g., > avoids context switching, scheduler overheads, and thread management) is > to use nonblocking collective communication [2]. "the use of" > > The nonblocking collective communication model is similar to the model > used in nonblocking point-to-point communication. A nonblocking start > call initiates the collective operation, but does not complete it. A > separate completion call is needed to complete the operation. Once > initiated, the operation may progress independently of any computation > or other communication at participating processes. In this manner, > nonblocking collectives can mitigate synchronizing effects of collective > operations by running them in the "background." In addition to enabling > communication-computation overlap, nonblocking collectives can perform > collective operations on overlapping communicators that would lead to > deadlock with blocking operations. The semantic advantages of > nonblocking collectives can also be useful in combination with > point-to-point communication. > > As in the nonblocking point-to-point case, all start calls are local and > return immediately irrespective of the status of other processes. The > start call initiates the operation which indicates that the system may > start to copy data out of the send buffer and into the receive buffer. > Once intiated, all associated send buffers should not be modified and > all associated receive buffers should not be accessed until the > collective operation completes. The start call returns a request handle, > which must be passed to a completion call to complete the operation. > > All completion calls (e.g., MPI_WAIT) described in Section ?? (Section > 3.7.3) are supported for nonblocking collective operations. Similarly to > the blocking case, collective operations are considered to be complete > when the local part of the operation is finished, i.e., the semantics of > the operation are guaranteed and all buffers can be safely accessed and > modified. Completion does not imply that other processes have completed > or even started the operation unless otherwise specified in or implied > by the description of the operation. Completion of a particular > nonblocking collective operation also does not imply completion of any > other posted nonblocking collective (or send-receive) operations, > whether they are posted before or after the completed operation. > > Advice to users. Some implementations may have the effect of > synchronizing processes during the completion of a nonblocking > collective. A correct, portable program cannot rely on such > synchronization side-effects, however, one must program so as to allow > them. (End of advice to users.) > > Upon returning from a completion call in which a nonblocking colletive > completes, the MPI_ERROR field in the associated status object is set > appropriately to indicate any errors. The values of the MPI_SOURCE and > MPI_TAG fields are undefined. It is valid to mix different request types > (i.e., any combination of collective requests, I/O requests, generalized > requests, or point-to-point requests) in functions that enable multiple > completions (e.g., MPI_WAITALL). It is erroneous to call > MPI_REQUEST_FREE or MPI_CANCEL with a request for a nonblocking > collective operation. > > Rationale. Freeing an active nonblocking collective request could cause > similar problems as discussed for point-to-point requests (see Section > ?? (3.7.3)). Cancelling a request is not supported because the semantics > of this operation are not well-defined. (End of rationale.) > > Multiple nonblocking collective operations can be outstanding on a > single communicator. If the nonblocking collective causes some system > resource to be exhausted, then it will fail and generate an MPI > exception. Quality implementations of MPI should ensure that this > happens only in pathological cases. That is, an MPI implementation > should be able to support a large number of pending nonblocking > collective operations. > > Unlike point-to-point operations, nonblocking collective operations do > not match with blocking collectives, and collective operations do not > have a tag argument. All processes must call collective operations > (blocking and nonblocking) in the same order per communicator. In > particular, once a process calls a collective operation, all other > processes in the communicator must eventually call the same collective > operation, and no other collective operation in between. This is > consistent with the ordering rules for blocking collective operations in > threaded environments. > > Rationale. Matching blocking and nonblocking collectives is not allowed > because an implementation might use different communication algorithms > for the two cases. Blocking collectives may be optimized for minimal > time to completion, while nonblocking collectives may balance time to > completion with CPU overhead and asynchronous progression. > > The use of tags for collective operations can prevent certain hardware > optimizations. (End of rationale.) > > Advice to users. If program semantics require matching blocking and > nonblocking collectives, then a nonblocking collective operation can be > initiated and immediately completed with a blocking wait to emulate > blocking behavior. (End of advice to users.) very good! > > In terms of data movements, each nonblocking collective operation has > the same effect as its blocking counterpart for intracommunicators and > intercommunicators after completion. The use of the “in place” option > is allowed exactly as described for the corresponding blocking collective > operations. Likewise, upon completion, nonblocking collective reduction > operations have the same effect as their blocking counterparts, and the > same restrictions and recommendations on reduction orders apply. > > Progression rules for nonblocking collectives are similar to progression > of nonblocking point-to-point operations, refer to Section ?? (Section > 3.7.4). > > Advice to implementors. Nonblocking collective operations can be > implemented with local execution schedules [3] using nonblocking > point-to-point communication and a reserved tag-space. (End of advice to > implementors.) > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll

1 0