February 2009 - mpiwg-coll - mailman3.cels.anl.gov

[mpi3-coll] Additional edits 2
by Adam Moody 04 Feb '09

04 Feb '09

Also against revision 3. I'd like to make the language in the descriptions for the data movement collectives consistent across the different collectives. The text skips around with several different flavors. Personally, I like this template best: "MPI_Ixxx starts a nonblocking variant of MPI_Xxx. The data movement after MPI_Ixxx completes is identical to the data movement after MPI_Xxx completes (see Section 5.Y)." This would apply to bcast, gather, gatherv, scatter, scatterv, allgather, allgatherv, alltoall, alltoallv, and alltoallw. We could add mention of inter- and intra-communicators here if folks would like to, but I like the simplicity of the above template myself -- leave all description up to the corresponding blocking section and provide a back reference. Of couse, for the reductions: reduce, allreduce, reduce_scatter, scan, and exscan; we still need more discussion on what "delivers the same results" mean in the current description since the blocking and nonblocking operations may use different algorithms and we have the associativity issue. -Adam

1 0

[mpi3-coll] Additional edits
by Adam Moody 03 Feb '09

03 Feb '09

Here are a few more edits -- against revision 3. -Adam I don't know if we want to list *every* section for the reduction collectives. (We don't do this for other collectives like alltoall, alltoallv, alltoalw) I think we could simplify a couple: p 1, lines 41-42 Simplify: "where the result is returned to all members of a group (Section 5.9.6 and Section 5.12.8) and a variation where the result is returned to only one member (Section 5.9 and Section 5.12.7)." to: "where the result is returned to all members of a group and a variation where the result is returned to only one member (Section 5.9, Section 5.12.7, and Section 5.12.8)." p 1, lines 47-48 Simplify: "(Section 5.11, Section 5.11.2, Section 5.12.10, and Section 5.12.11)" to: "(Section 5.11, Section 5.12.10, and Section 5.12.11)" Mixing call and operation p 2, lines 18-20 Change: "A blocking operation is complete as soon as the call returns. A nonblocking (immediate) call requires a separate completion operation" to: "A blocking operation is complete as soon as the call returns. A nonblocking operation requires a separate completion call" Missing a "nonblocking": p 50, line 5 "Similarly to the blocking case, collective operations ..." --> "Similarly to the blocking case, nonblocking collective operations ..." We should also stress local-ness of completion in the "i.e." phrase p 5, line 7 "i.e., the semantics of the operation are guaranteed" --> "i.e., for the caller, the semantics of the operation are guaranteed" Since we are now using the word "indicate" in the first sentence, it'd be good to use this same word in the following "also" sentence. p 50, line 10 "does not imply" --> "does not indicate" I think it'd read better to mention the "in place" buffer restriction following the other sentence about "in place" near the end. p 50, line 1-2, remove sentences "When using the "in place" option, ... Such buffers ... until the operation completes." p 51, lines 9-14: Split this paragraph into two and add the removed sentences from above (I also added "Once an operation is initiated"). The result paragraphs read: In terms of data movements, each nonblocking collective operation has the same effect as its blocking counterpart for the intracommunicators and intercommunicators after completion. Likewise, upon completion, nonblocking collective reduction operations have the same effect as their blocking counterparts, and the same restrictions and recommendations on reduction orders apply. The use of the "in place" option is allowed exactly as described for the corresponding blocking collective operations. When using the "in place" option, message buffers function as both send and receive buffers. Once an operation is initiated, such buffers should not be modified or accessed until the operation completes. Verb tense of "call" and keep the "only" keyword from blocking description: p 51, line 36 "the operation completes after every process in the communicator called MPI_IBARRIER" --> "the operation completes only after every process in the communicator calls MPI_IBARRIER" Expand intercommunicator sentence for MPI_Ibarrier p 51, lines 36-37 Replace: "On intercommunicators, the operation completes when the MPI_BARRIER call would return." with: "On intercommunicators, the operation completes for processes in one group (group A) of the intercommunicator only after all processes in the other group (group B) call MPI_IBARRIER."

1 0

Re: [mpi3-coll] specified in or implied by
by Jesper Larsson Traeff 03 Feb '09

03 Feb '09

I'm also form these Jesper On Mon, Feb 02, 2009 at 12:17:06PM -0800, Adam Moody wrote: > Several folks mentioned that this line in the NBC intro is wordy: > "Completion does not imply that other processes have completed or > even started the operation unless otherwise specified in, or implied by, > the description of the operation." > I think a good solution here is to borrow from the existing collectives > intro (see below). > -Adam > > From p 2, lines 22-24 we have > "It does not indicate that other processes in the group have > completed or even started the operation (unless otherwise implied by in > the description of the operation)." > > First, drop the extra preposition in this text (typo from original) > "implied by in" --> "implied by" > > Second, borrow this wording for NBC on p 50, lines 7-8. > Change: > "Completion does not imply that other processes have completed or > even started the operation unless otherwise specified in, or implied by, > the description of the operation." > To: > "Completion does not indicate that other processes have completed or > even started the operation (unless otherwise implied by the description > of the operation.)" > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll

1 0

Re: [mpi3-coll] Simplify "local completion" to just "completion"
by Jesper Larsson Traeff 03 Feb '09

03 Feb '09

I'd agree with that jesper On Mon, Feb 02, 2009 at 12:07:24PM -0800, Adam Moody wrote: > I think we could reduce "local completion" to just "completion" > throughout the text. I don't think the adjective "local" is necessary > and in some ways I find it confusing. We define precisely what > "completion" means in a collective, and we also explicitly note that > completion at a process does not indicate the status of other > processes. Whenever we say "local completion", we really mean > "completion" by our definition, so "local" is not necessary here. And > by saying "local" completion, it sounds as a reader that there may be > such a thing as "non-local" completion, but we don't define it. There > is only one definition for collective completion. > -Adam > > p 2, line 17 in "Collective operations can (but are not required to) > complete locally as soon as the caller's participation in the collective > communication is finished." > "complete locally" --> "complete" > > p 2, line 20 in "The local completion of a collective operation > indicates that the caller is free to access locations in the > communication buffer." > "The local completion" --> "The completion" > > p 49, line 48 & p 50, line 1 in "Once initiated, all associated send > buffers should not be modified and all associated receive buffers should > not be accessed until the collective operation completes locally." > "until the collective operation completes locally." --> "until the > collective operation completes." > > p 51, line 32 in "On intracommunicators, the operation completes locally > after every process in the communicator called MPI_IBARRIER." > "the operation completes locally" --> "the operation completes" > and (to preserve verb tense) > "after every process in the communicator called" --> "after every > process in the communicator calls" > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll

1 0

Re: [mpi3-coll] More on in-place
by Torsten Hoefler 02 Feb '09

02 Feb '09

On Mon, Feb 02, 2009 at 11:31:32AM -0800, Adam Moody wrote: > I think we need another statement about the in-place option in the text > regarding buffer access. Something along the lines of: > > When using the "in-place" option, message buffers may function as both > send and receive buffers. Such buffers should not be modified or > accessed until the operation completes. good point! I added: "When using the ``in place'' option, message buffers function as both send and receive buffers. Such buffers should not be modified or accessed until the operation completes." > This is somewhat unique to NBC. However, this statement is necessary > for blocking collectives in threaded environments, as well. Did the > "Send buffer access" ticket resolve this? Either way, it might not hurt > to reiterate this point in the NBC intro, since it is more likely to > come up there. uh well - collectives need more discussion anyway. Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Simplify "local completion" to just "completion"
by Torsten Hoefler 02 Feb '09

02 Feb '09

On Mon, Feb 02, 2009 at 03:06:50PM -0800, Adam Moody wrote: > Ok, just a bit more and then I'll back down ;-) I think Adam has a point and I thought about exactly the same thing while merging. I would also say that it could be confusing and that completion is clearly defined. I removed the four occurrences of "local(ly)" in rev. 3. We can discuss this very minor thing during the next Forum. Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] specified in or implied by
by Torsten Hoefler 02 Feb '09

02 Feb '09

On Mon, Feb 02, 2009 at 12:17:06PM -0800, Adam Moody wrote: > Several folks mentioned that this line in the NBC intro is wordy: > "Completion does not imply that other processes have completed or > even started the operation unless otherwise specified in, or implied by, > the description of the operation." > I think a good solution here is to borrow from the existing collectives > intro (see below). > -Adam > > From p 2, lines 22-24 we have > "It does not indicate that other processes in the group have > completed or even started the operation (unless otherwise implied by in > the description of the operation)." > > First, drop the extra preposition in this text (typo from original) > "implied by in" --> "implied by" > > Second, borrow this wording for NBC on p 50, lines 7-8. > Change: > "Completion does not imply that other processes have completed or > even started the operation unless otherwise specified in, or implied by, > the description of the operation." > To: > "Completion does not indicate that other processes have completed or > even started the operation (unless otherwise implied by the description > of the operation.)" that's fine with me too - I updated rev. 3. All three versions say the same thing. Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] New nonblocking collective intro text
by Torsten Hoefler 02 Feb '09

02 Feb '09

Jesper, > > As described in Section ?? (Section 3.7), performance on many systems > > can be improved by overlapping communication and computation. > > Nonblocking collectives combine the potential benefits of nonblocking > > point-to-point operations to exploit overlap and to avoid > > synchronization with the optimized implementation and message scheduling > I'm not sure "optimized implementation" here really conveyes to the > user the advantage of NBCs hmm, this seems fine to me - you can propose a better wording, but I think it's of marginal importance. > > provided by collective operations [1,4]. One way of doing this would be > > to perform a blocking collective operation in a separate thread. An > > alternative mechanism that often leads to better performance (e.g., > > avoids context switching, scheduler overheads, and thread management) is > > to use nonblocking collective communication [2]. > "the use of" I'd prefer to stay consistent to p47:46 Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] New nonblocking collective intro text
by Torsten Hoefler 02 Feb '09

02 Feb '09

Adam, Christian, oh - this mail was delayed at my mailserver, ignore my reply to the previous one. >> I very much appreciate many of your textual improvements. However, it >> is very time consuming for us as reviewers to keep track of such >> substantial changes. It would have been much easier, if you had >> contributed your changes in an earlier stage. Anyway, I waded through >> the changes to revision 3 and will comment only on the things that I >> don't like (consider everything else as excellent): >> >> I second Torsten's opinion on p. 49, l. 35-37: This reformulation >> introduces three times the term "complete". Besides this drawback, I >> also don't see any benefit from this reformulation compared to the >> previous version. > > It's a bit repetitive, I know. To make a direct parallel, I copied the > same sentence structure used in the nonblocking point-to-point section > (see the 2.1 document, page 47, lines 46-48). We could drop the "but > does not complete it" clause to shorten this some (saves one "complete"): > "A nonblocking start call initiates a collective operation, which > must be completed in a separate completion call." ok, I revert my "might" if the point-to-point chapter mandates a completion function. I implemented your proposal from above. >> p. 50, l. 1-2: Similar problem (i.e. 2x"complete"). May I propose: >> "The start call returns a request handle, which is eventually passed >> to a completion call." > > I'd be ok with dropping a "complete" here, as well, although I prefer > "which must be passed" over "which is eventually passed": > "The start call returns a request handle, which must be passed to a > completion call." sounds good - implemented this. >> >> p. 50, l. 8: The part "unless otherwise specified in, or implied by, >> the description of the operation" has been added. Seem ok to me at >> first glance. However, we might re-check if this is really necessary >> and drop it otherwise. > > I added the "implied by" to address cases like MPI_Alltoall, where there > is no specified synchronization as in MPI_Barrier, but clearly a process > can't complete until it has received data from every other process and > thus knows all other processes have started the call. I have no > objections to dropping one of these two phrases. Going with just > "implied by" leads to a cyclic-sounding statement. > "Completion does not imply that other processes have completed or > even started the operation unless otherwise implied by the description > of the operation." > We could just go with "specified in": > "Completion does not imply that other processes have completed or > even started the operation unless otherwise specified in the description > of the operation." I'm fine either way. The existing text in MPI-2.1 never mentiones this implied by, so I'll drop it (it should be intuitive). >> p. 50, l. 17: The added postfix "to indicate any errors" seems wrong >> to me because MPI_SUCCESS is also a valid entry for the MPI_ERROR >> field. I'd propose to simply omit this postfix. However, I'm not 100% >> sure because the section about the MPI Status object looks quite >> confusing in this respect. Btw. there exists also an empty status >> (which sets all 3 fields) - so we might need to reconsider our >> decision about setting only the error field for nonblocking collective >> operations. > > Yes, I agree, but I'll leave this alone as it is a bigger discussion for > the group. we discussed that setting this would cause unnecessary overheads. Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] New nonblocking collective intro text
by Torsten Hoefler 02 Feb '09

02 Feb '09

Hi Christian, > I very much appreciate many of your textual improvements. However, it is > very time consuming for us as reviewers to keep track of such > substantial changes. It would have been much easier, if you had > contributed your changes in an earlier stage. Anyway, I waded through > the changes to revision 3 and will comment only on the things that I > don't like (consider everything else as excellent): > > I second Torsten's opinion on p. 49, l. 35-37: This reformulation > introduces three times the term "complete". Besides this drawback, I > also don't see any benefit from this reformulation compared to the > previous version. ok, so I reformulated "A nonblocking start call initiates a collective operation, but does not complete it. A separate completion call is needed to complete the operation." to "A nonblocking start call initiates a collective operation and a completion call might be needed to finish the operation." I introduced "might" because the initialization is allowed to finish the operation (semantically) - think of MPI_COMM_SELF. > p. 50, l. 1-2: Similar problem (i.e. 2x"complete"). May I propose: "The > start call returns a request handle, which is eventually passed to a > completion call." we should stick to imperative voice in the standard. I think we're not writing a novel and this repetition is acceptable in a technical document. > p. 50, l. 8: The part "unless otherwise specified in, or implied by, the > description of the operation" has been added. Seem ok to me at first > glance. However, we might re-check if this is really necessary and drop > it otherwise. MPI_Ibarrier() specifies this > p. 50, l. 16: This looks quite redundant to me. I'd propose sth. like > the following restatement: "Upon completion of a nonblocking collective > operation, the MPI_ERROR field...". ah, there is a slight semantic difference between "returning from a completion call" and "completion". I think the current wording is less abiguous. > p. 50, l. 17: The added postfix "to indicate any errors" seems wrong to > me because MPI_SUCCESS is also a valid entry for the MPI_ERROR field. > I'd propose to simply omit this postfix. However, I'm not 100% sure > because the section about the MPI Status object looks quite confusing in > this respect. Btw. there exists also an empty status (which sets all 3 > fields) - so we might need to reconsider our decision about setting only > the error field for nonblocking collective operations. omitting it is a safe way of avoiding mistakes. It's gone now and just says "is set appropriately" which is defined elsewhere. > p. 50, l. 29: "nonblocking collective communications" => "nonblocking > collective operations" fixed > p. 50, l. 44: You changed "running time" to "minimal time to completion" > - this is _not_ the same thing! I'd strongly prefer to stay with the > previous terminology. why? I'll leave it as is for now. > p. 62, l. 36/37: now we have two consecutive "the"s fixed Updates are in rev. 3. Thanks, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0