January 2009 - mpiwg-coll - mailman3.cels.anl.gov

[mpi3-coll] NBC Proposal Revision 2
by Torsten Hoefler 27 Jan '09

27 Jan '09

Hello, I finished revision 2 of the NBC proposal based on discussions in today's telecon and several reviews. The document is attached to ticket #109: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/109 and for fast access temporarily: http://www.unixer.de/sec/nbc-proposal-rev-2.pdf the diff-file is: http://www.unixer.de/sec/nbc-proposal-rev-2.diff Please review the changes carefully and send me any comments until the deadline on Friday, Jan 30th! I also added a list of attendees to the wiki-page. I did not ask this time, so please let me know if I forgot somebody. Thanks & Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Very minor comments on nonblocking collectives section
by Torsten Hoefler 27 Jan '09

27 Jan '09

Hi, > Two very minor issues that can probably be ignored...but I could not > help but send an email :) > > 1. The use of quotation marks with "in place" everywhere in the document > seems strange. Do we really need to quote this everywhere? My thought > was just in-place, but I assume that "in place" is legacy. > > I did some research and found a bunch of stuff, but nothing to really > tell me whether using the quotes were right or wrong. I think it's legacy - but we can propose to change it in MPI-2.2 (but I don't find it critical). > I found this at http://owl.english.purdue.edu/handouts/grammar/g_quote.html > Use quotation marks to indicate words used ironically, > with reservations, or in some unusual way. > > > 2. Somewhat similarly, on page 50, why do we quote "pathological"? I > think in this case we should just drop the quotes? I removed this one in rev. 2 Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Comments on nonblocking collectives section
by Torsten Hoefler 27 Jan '09

27 Jan '09

Hi Christian, thanks for the review! > p 4, l 26: old typo? "indentifier" -> "identifier" fixed > p 49, l 36: question for "the communication may progress independently > of the computations at all participating processes": What about all the > Reduce operations which involve both communication and computation? > Should we replace "the communication may progress" with the more general > "the operation may progress"? changed > p 50, l 14: same argument as above; replace "collective communication" > with "collective operation" changed > p 50, l 31: "using normal point-to-point" is not accurate/sufficient > enough => "using nonblocking point-to-point" ah, that's minor - changed anyway > p 65, 5.13: The example is good but the current text is bad. It is > simply not allowed to call NBC in a different order. If a certain > implementation runs into a deadlock does not matter. This would even be > on of the better scenarios (i.e. an easy-to-noticeable error). Other > implementations could complete both bcasts (because it thinks they > match) but provide the wrong data (i.e. the data from the other bcast or > even a mixture?) - which is a much harder failure to detect. which example are you referring to? Only Ex 5.24 is on this page and doesn't use NBC (and the phrasing "deadlock" is used correctly there. > p 66, example 5.25: should we add sth like the following at the end? > "Note: Using nonblocking collectives can fix such > problems, see example 5.34." added > p 69, l 32: maybe better "However, communicator construction might > induce overheads and consume resources so that this should be used > economically." not changed (decided on telecon) > p 70, example 5.31: To me this example is not "per se" illegal. Both > operations simply don't match - that's fine. However, it is possible to > extent the example so that the final program is still a valid MPI > program. Maybe just add sth. like "in this form" or "without additional > collectives that match". no, it's not possible to make this correct if both ranks call it at the same time. Remember, blocking and nonblocking colls have the same matching "namespace". All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Comments on nonblocking collectives section
by Adam Moody 27 Jan '09

27 Jan '09

For the most part, everyone's comments fit in fine with my reordering which I'll propose after Torsten integrates the comments from today. I have just one conflict with one of Jesper's suggestions: > > >>p. 49, line 26: >>"the performance of many systems..." -> >>"the performance of many applications can be improved by overlapping communication >>and computation, and many systems enable this." >> >> >fixed > > I'd just simplify this to: "As described in Section 3.7, performance on many systems can be improved by overlapping communication and computation."

1 0

Re: [mpi3-coll] Comments on nonblocking collectives section
by Bronis R. de Supinski 27 Jan '09

27 Jan '09

Torsten and Jesper: Re: > > sorry for my late input. I've had time to read the document, and have found > > something. I refer to the version of 21st January. > Thanks for your review - I'll comment below! I'm clipping items on which I have no comments. > > p. 49, line 26: > > "the performance of many systems..." -> > > "the performance of many applications can be improved by overlapping communication > > and computation, and many systems enable this." > fixed > > > p. 49, line 28: > > "utilize overlap -> "exploit overlap" > fixed Reasonable. I see a split infinitive there also: to utilize overlap and avoid synchronization => to exploit overlap and to avoid synchronization > > p. 49, line 32: > > drop "the use of" (grammar: "the use of" is no mechanism) > nb collective communication doesn't increase performance by itself, only > the correct *use* of them leads to higher performance, cf. MPI-2.1 > p47:45. I changed "the use of" to "to use" which might be better > English. That does sound better. You could delete "mechanism" in "An alternative mechanism" earlier in the sentence since "to use collective communication" no longer seems like a mechanism but an implementation choice but it is not critical. > > p.49, line 35: > > "a collective communication, which..." -> "a collective communication > > operation, that..." ("that" better than "which" here?) > don't know - I'll leave it as is since Bronis didn't comment on this > before Proper English uses "that" with no comma for required clauses -- clauses without which the sentence does not make sense -- and "which" preceded by a comma for all others (usually apository but can have other functions). This clause does not seem required to me so I would go with ", which" but I could see an argument for "that". In any event, ", that" would be incorrect. > > p.50, line 35: > > "In particular," -> "This means that" > fixed Hmm. I think at this point neither of these phrases really are needed. I would just use a colon instead of breaking the two sentences: Blocking and nonblocking collective operations are different collective operations: if some process calls a collective operation, all other processes in the communicator must eventually call the same collective operation, and no other collective operation in between. You might insert "blocking or nonblocking" between "same" and "collective" but I don't think it is critical. > > p.50, line 48: > > "collective operations is required" -> "collective operation semantics > > is required" > I don't understand this proposal and I think the current text is > precise. I agree with Torsten. Bronis

1 0

[mpi3-coll] Very minor comments on nonblocking collectives section
by Rolf Vandevaart 27 Jan '09

27 Jan '09

Hi Torsten: Two very minor issues that can probably be ignored...but I could not help but send an email :) 1. The use of quotation marks with "in place" everywhere in the document seems strange. Do we really need to quote this everywhere? My thought was just in-place, but I assume that "in place" is legacy. I did some research and found a bunch of stuff, but nothing to really tell me whether using the quotes were right or wrong. I found this at http://owl.english.purdue.edu/handouts/grammar/g_quote.html Use quotation marks to indicate words used ironically, with reservations, or in some unusual way. 2. Somewhat similarly, on page 50, why do we quote "pathological"? I think in this case we should just drop the quotes? Rolf vandeVaart -- ========================= rolf.vandevaart(a)sun.com 781-442-3043 =========================

1 0

Re: [mpi3-coll] Comments on nonblocking collectives section
by Christian Siebert 27 Jan '09

27 Jan '09

Dear all, here are a couple of comments from my site (referring to the new version that Torsten sent out an hour ago): p 4, l 26: old typo? "indentifier" -> "identifier" p 49, l 36: question for "the communication may progress independently of the computations at all participating processes": What about all the Reduce operations which involve both communication and computation? Should we replace "the communication may progress" with the more general "the operation may progress"? p 50, l 14: same argument as above; replace "collective communication" with "collective operation" p 50, l 31: "using normal point-to-point" is not accurate/sufficient enough => "using nonblocking point-to-point" p 65, 5.13: The example is good but the current text is bad. It is simply not allowed to call NBC in a different order. If a certain implementation runs into a deadlock does not matter. This would even be on of the better scenarios (i.e. an easy-to-noticeable error). Other implementations could complete both bcasts (because it thinks they match) but provide the wrong data (i.e. the data from the other bcast or even a mixture?) - which is a much harder failure to detect. p 66, example 5.25: should we add sth like the following at the end? "Note: Using nonblocking collectives can fix such problems, see example 5.34." p 69, l 32: maybe better "However, communicator construction might induce overheads and consume resources so that this should be used economically." p 70, example 5.31: To me this example is not "per se" illegal. Both operations simply don't match - that's fine. However, it is possible to extent the example so that the final program is still a valid MPI program. Maybe just add sth. like "in this form" or "without additional collectives that match". Best regards, Christian -- Christian Siebert, Dipl.-Inf. Research Associate NEC Laboratories Europe, NEC Europe Ltd. Rathausallee 10, D-53757 Sankt Augustin, Germany Phone: +49 (0) 2241 - 92 52 44 Fax: +49 (0) 2241 - 92 52 99 (Registered Office: 1 Victoria Road, London W3 6BL, 2832014)

1 0

Re: [mpi3-coll] Comments on nonblocking collectives section
by Torsten Hoefler 27 Jan '09

27 Jan '09

Hi Jesper, > > > p.50, line 20 (or elsewhere), add: > > > "Completion of a particular nonblocking collective operations does > > > \emph{not} imply completion of any other posted nonblocking collective > > > (or send-receive) operations, whether they are posted before or after > > > the operation in case." > > hmm, I think this clarification is unnecessary because similar things > > would apply to nonblocking point-to-point and are not mentioned there > > (and the semantics are clearly defined). We have Example 5.34 to make > > this clear, but I'd rather not introduce this to the introductory text. > > > I think it is a fairly important semantic issue, and should be said somewhere. > The (good) example comes at the end of the chapter. ok, I put it on the agenda for today's meeting. > > I'd propose to stay with the reviewed phrasing. > > > Maybe Bronis look it over? Again, the "identical" phrase is in my opinion > bad (but you changed that). hmm, I'd still argue for its correctness. > > > p.68, line 21: does a race condition "cause" non-determinism, or is it > > > the other way round? > > I think it's correct because it might match correctly, however, if the > > race is done in the wrong order, then it's wrong. The casual chain is > > here: programming mistake -> race condition -> nondeterminism > > > and that's not really what the caption says. I suggest a rewording, will > try and come up with something hmm, we can discuss this as well on the telco -> added it to agenda Thanks & Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Comments on nonblocking collectives section
by Jesper Larsson Traeff 27 Jan '09

27 Jan '09

Dear all, On Tue, Jan 27, 2009 at 09:28:05AM -0500, Torsten Hoefler wrote: > ... MPI_REQUEST_FREE > Please bring it up again if you think it needs more discussion! > It doesn't, all has been said... > > p.50, line 20 (or elsewhere), add: > > "Completion of a particular nonblocking collective operations does > > \emph{not} imply completion of any other posted nonblocking collective > > (or send-receive) operations, whether they are posted before or after > > the operation in case." > hmm, I think this clarification is unnecessary because similar things > would apply to nonblocking point-to-point and are not mentioned there > (and the semantics are clearly defined). We have Example 5.34 to make > this clear, but I'd rather not introduce this to the introductory text. > I think it is a fairly important semantic issue, and should be said somewhere. The (good) example comes at the end of the chapter. > > p.50, line 24-25: > > "cannot be defined clearly." -> "is not well-defined." > I think that "cannot be defined" is stronger than "is not well-defined". > yes, that's why I suggest the slightly, perhaps, weaker text > > p.50, line 48: > > "collective operations is required" -> "collective operation semantics > > is required" > I don't understand this proposal and I think the current text is > precise. > I didn't think so, will try to come up with another wording > > p.53, line 37: > > "are identical as after a call" - bad Engish, should rather be: > > "are the same as after a call". > fixed (twice) > > > I would strongly suggest using exactly the same phrase for all of the > > data moving collectives! This is not a document being read as a whole, > > so we don't need to fear "repetition". > maybe, I think the wording is clear as it is and has been reviewed > multiple times. Thus, it should be fine. Feel free to propose a unified > wording that does not invalidate all previous reviews ;). > I don't think it has been reviewed multiple times... ;-) it's the sort of thing you just skip. The English (use of "identical") is in many cases not correct... > I'd propose to stay with the reviewed phrasing. > Maybe Bronis look it over? Again, the "identical" phrase is in my opinion bad (but you changed that). > > p.68, line 21: does a race condition "cause" non-determinism, or is it > > the other way round? > I think it's correct because it might match correctly, however, if the > race is done in the wrong order, then it's wrong. The casual chain is > here: programming mistake -> race condition -> nondeterminism > and that's not really what the caption says. I suggest a rewording, will try and come up with something > > p. 68, line 26: > > move "outstanding" to "even if there is an outstanding nonblocking..." > that's a matter of taste > yes > > p. 69, after line 34: > > Would suggest an example 5.31 that shows that e.g. MPI_IAlltoall does > > not match MPI_Alltoall. > > > > "Blocking and nonblocking collective operations do not match. The > > following is illegal:" > added (this should be clear after Ex 5.29, but more examples can't harm) > exactly! > I will attach it to ticket #109 after the telecon. > great! Jesper

1 0

Re: [mpi3-coll] Comments on nonblocking collectives section
by Torsten Hoefler 27 Jan '09

27 Jan '09

Hi Jesper, > sorry for my late input. I've had time to read the document, and have found > something. I refer to the version of 21st January. Thanks for your review - I'll comment below! > p. 49, line 26: > "the performance of many systems..." -> > "the performance of many applications can be improved by overlapping communication > and computation, and many systems enable this." fixed > p. 49, line 28: > "utilize overlap -> "exploit overlap" fixed > p. 49, line 32: > drop "the use of" (grammar: "the use of" is no mechanism) nb collective communication doesn't increase performance by itself, only the correct *use* of them leads to higher performance, cf. MPI-2.1 p47:45. I changed "the use of" to "to use" which might be better English. > p.49, line 35: > "a collective communication, which..." -> "a collective communication > operation, that..." ("that" better than "which" here?) don't know - I'll leave it as is since Bronis didn't comment on this before > p.49, line 36: > "the communication may progress" -> "the collective communication may > then progress" no, there is no temporal description, because we talk about initiation *and* completion in the previous sentence. This sentence is clear(er) without this change. > p.49, line 38: > "mitigate synchronizing effects" -> "mitigate synchronization effects" no, "synchronization effects" sounds more like they're supposed to synchronize (like barrier) to me while "synchronizing effects" sounds more like it's meant - nasty effects that synchronize which has negative impact to the app. That's my personal interpretation, we can discuss it. > p. 50, line 9-11: > drop sentence "Collective operations...", allready said in line 4-7? fixed, added the buffer access rule to 4-7 > p.50, after line 18, add: > "Nonblocking collective requests are not persistent, and > MPI_REQUEST_NULL is returned after successful completion." I added "Nonblocking collective requests are not persistent." Returning REQUEST_NULL is part of the semantics of Test/Wait and friends. I would rather not scatter them throughout the document. > p. 50, line 18: > I still think that MPI_Request_free should be allowed. we had this question asked 5 times in the woring group and two times to the whole Forum. We reached the same consensus everytime. > The rationale also still reads contrived. I think the rationale is perfectly fine. Please point at particular weaknesses or propose an alternative. If you think that it's contrived, then give us an example of how to check the completion of an MPI_Ialltoall where the request was freed in a portable way (yes, with all buffering TCP implementations, buffering out-of-order UDP (or whatever) implementations, and without any memory barrier semantics). Some things work (Bcast), but most just don't, thus we should not encourage the use of such nonportable techniques (cf. Erez' original ticket/discussion). Please bring it up again if you think it needs more discussion! > p.50, line 20 (or elsewhere), add: > "Completion of a particular nonblocking collective operations does > \emph{not} imply completion of any other posted nonblocking collective > (or send-receive) operations, whether they are posted before or after > the operation in case." hmm, I think this clarification is unnecessary because similar things would apply to nonblocking point-to-point and are not mentioned there (and the semantics are clearly defined). We have Example 5.34 to make this clear, but I'd rather not introduce this to the introductory text. > p.50, line 24: > "Canceling a request" -> "Canceling a nonblocking request" hmm, aren't all requests nonblocking? And it's clear (from the context) that it refers to requests associated to nonblocking collectives. > p.50, line 24-25: > "cannot be defined clearly." -> "is not well-defined." I think that "cannot be defined" is stronger than "is not well-defined". > p.50, line 33: > "In this sense"?? What does this refer to? Drop, just say "Blocking > and nonblocking..." fixed > p.50, line 35: > "In particular," -> "This means that" fixed > p.50, line 44: > "could find an equilibrium between" -> "could attempt to balance" fixed (also added a missing comma before "and") > p. 50, line 47-48: > "than for point-to-point operations." -> > "than for point-to-point operations, where for instance a blocking > send can be matched by a nonblocking receive." added > p.50, line 48: > "collective operations is required" -> "collective operation semantics > is required" I don't understand this proposal and I think the current text is precise. > p.51, line 6 (or elsewhere), add: > "In terms of data movements each nonblocking collective operation upon > completion has the same effect as its blocking counterpart, both for > intracommunicators and intercommunicators. The use of the "in place" > option is allowed exactly as described for the corresponding blocking > collective operations. Likewise, the nonblocking collective reduction > operations upon completion have the same effect as their blocking > counterparts, and the same restrictions and recommendations on > reduction orders apply." excellent proposal! I re-phrased it slightly. The sentence about reductions might be what Rolf wanted. > I think this is quite important for the semantic definition of the > nonblocking collectives. The individual remarks for each of the > operations could be dropped, although I think it is good to have them > there (for people just checking the one operation they are interested > in). However, the English is in most of the individual cases not good, > see below. yes, I would not drop them (they're short anyway). > p.51, line 26-30: > Advice reads a bit awkward, but have no suggestion. I think it's clear. > p.53, line 37: > "are identical as after a call" - bad Engish, should rather be: > "are the same as after a call". fixed (twice) > I would strongly suggest using exactly the same phrase for all of the > data moving collectives! This is not a document being read as a whole, > so we don't need to fear "repetition". maybe, I think the wording is clear as it is and has been reviewed multiple times. Thus, it should be fine. Feel free to propose a unified wording that does not invalidate all previous reviews ;). > Perhaps add: > "The "in place" option is allowed for intracommunicators, with the same > effect as for MPI_Gather." we would have to add this to every collective. I think it's clear with the paragraph we added above. > p, 54, line 41: Language! hmm, sounds correct to me > p. 55, line 35: Same > p. 56, line 34: Same > p. 57, line 32: Same > p. 58, line 26: Same > p. 59, line 32: Same > p. 60, line 28: Same > p. 61, line 47: Reads sort of ok here, would still be in favor of using the same phrase > everywhere. I'd propose to stay with the reviewed phrasing. > p. 62, line 35-39: > Drop advice to users, but add "The same rules and restrictions as for > MPI_Reduce apply." (consistency is no hobgoblin here, and I think it's > important that the nonblocking collectives "does" the same as their > blocking counterparts) there is no advice to users? We will discuss the changes to the advice to implementors today. > p. 63, line 26: Language. > p. 64, line 11: Sort of OK. > p. 65, line 25: Sort of OK. seems clear to me > p.68, line 21: does a race condition "cause" non-determinism, or is it > the other way round? I think it's correct because it might match correctly, however, if the race is done in the wrong order, then it's wrong. The casual chain is here: programming mistake -> race condition -> nondeterminism > p. 68, line 26: > move "outstanding" to "even if there is an outstanding nonblocking..." that's a matter of taste > p. 69, after line 34: > Would suggest an example 5.31 that shows that e.g. MPI_IAlltoall does > not match MPI_Alltoall. > > "Blocking and nonblocking collective operations do not match. The > following is illegal:" added (this should be clear after Ex 5.29, but more examples can't harm) > p.70, line 6: > "the "MPI_Recv." -> "the MPI_Recv call." fixed > p. 70, line 42: > "lower performance" (sounds too colloqial) -> "impact performance negatively." or > "have a negative effect on performance." fixed All changes have been applied to Rev. 2. We will discuss those (and possibly more) changes in today's telecon. THe document is available here: pdf: http://www.unixer.de/sec/nbc-proposal-rev-2.pdf diff: http://www.unixer.de/sec/nbc-proposal-rev-2.diff I will attach it to ticket #109 after the telecon. Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0