November 2008 - mpiwg-coll - mailman3.cels.anl.gov

Re: [mpi3-coll] Nonblocking collectives standard draft
by Torsten Hoefler 24 Nov '08

24 Nov '08

On Mon, Nov 24, 2008 at 09:34:09PM -0800, Greg Bronevetsky wrote: > > >- another argument _for_ MPI_REQUEST_FREE is the existence of > >intercommunicators where all processes (i.e., in group A and in > >group B) have to call the collective but only part of them might be > >involved (e.g., for MPI_IBCAST all but the root process in group A > >don't really participate in the operation - they pass MPI_PROC_NULL > >as root argument). So having a lightweight MPI_Wait alternative > >might not be too bad ... > > I would argue against MPI_Request_free in general for both > point-to-point and collective operations. For one thing, if there is > an error during the operation, MPI has no logical place to deliver > this notification. Another reason is that on single-threaded > architectures (BG/L CNK, Cray Catamount) it is completely possible > for the non-blocking to not complete until MPI_Finalize because the > application is not required to issue any more MPI calls during which > progress may be made on the communication. At least if we have no > MPI_Request_free the user is eventually required to call MPI_Test or > MPI_Wait to ensure progress. I can go on but MPI_Request_free is bad. yes, we should probably bring this up again in the discussion of MPI-2.2. Let's talk about this on the telecon (I just put it on the agenda). > Having said that, I object to the text that explains the reasoning > for removing MPI_Request_free on page 50 for exactly the reason > above. We shouldn't have MPI_Request_free because its generally bad. yes, I agree. I just wanted to stay consistent with the current standard with the text, but I don't like it either. I think we can't say that Request_free is generally bad, thu I chose this phrasing. However, I'm very open to changes. Can you propose a better wording? Thanks, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Nonblocking collectives standard draft
by Greg Bronevetsky 24 Nov '08

24 Nov '08

>- another argument _for_ MPI_REQUEST_FREE is the existence of >intercommunicators where all processes (i.e., in group A and in >group B) have to call the collective but only part of them might be >involved (e.g., for MPI_IBCAST all but the root process in group A >don't really participate in the operation - they pass MPI_PROC_NULL >as root argument). So having a lightweight MPI_Wait alternative >might not be too bad ... I would argue against MPI_Request_free in general for both point-to-point and collective operations. For one thing, if there is an error during the operation, MPI has no logical place to deliver this notification. Another reason is that on single-threaded architectures (BG/L CNK, Cray Catamount) it is completely possible for the non-blocking to not complete until MPI_Finalize because the application is not required to issue any more MPI calls during which progress may be made on the communication. At least if we have no MPI_Request_free the user is eventually required to call MPI_Test or MPI_Wait to ensure progress. I can go on but MPI_Request_free is bad. Having said that, I object to the text that explains the reasoning for removing MPI_Request_free on page 50 for exactly the reason above. We shouldn't have MPI_Request_free because its generally bad. Greg Bronevetsky Post-Doctoral Researcher 1028 Building 451 Lawrence Livermore National Lab (925) 424-5756 bronevetsky1(a)llnl.gov

1 0

Re: [mpi3-coll] Nonblocking collectives standard draft
by Torsten Hoefler 24 Nov '08

24 Nov '08

Hi Christian, > I did a second reading and found several but only minor issues with the > text. If you send me the TeX sources, I can send you my proposed > corrections/improvements directly as a patch file (diff). I will send it via mail. > Some more things for discussion: > > - p. 49 l. 41: MPI has the concept of error handlers. I don't see a > reason why we should forbid ("return an error code") its usage. My > suggestion is to simply delete the two sentences (from "If the call ..." > to "... cases."). This is actually consistent with MPI-2.1 p 48:17. I would agree to omit the "return an error code", but since it's already written in this style for point-to-point, we should keep it consistent. > - p. 61 l 33-38: Should we really keep this Advice for implementors > (same result for MPI_IREDUCE when called with same arguments)? For the > blocking variant, there stands this argument of reproducibility vs. > performance. However, the non-blocking variant could be optimized much > better (e.g., might handle bad process skew) and we should consider to > weaken this semantic restriction to allow this better performance. Yes, this is a bigger issue than we assume. Reproducability is very important for real applications. > - another argument _for_ MPI_REQUEST_FREE is the existence of > intercommunicators where all processes (i.e., in group A and in group B) > have to call the collective but only part of them might be involved > (e.g., for MPI_IBCAST all but the root process in group A don't really > participate in the operation - they pass MPI_PROC_NULL as root > argument). So having a lightweight MPI_Wait alternative might not be too > bad ... I disagree because an MPI_Wait() on a request that doesn't involve anything is as cheap as Request_free. > The NBC examples are quite buggy (missing MPI_STATUS in MPI_Recv, > missing semicolons, duplicate usage of same req-argument -> bad Waitall, > ...) - but as with the text, I can send you corrected versions in the diff. yes, it's version one and I have to admit that I did not try to compile them yet. Please go ahead and send corrections! Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Next teleconference
by Torsten Hoefler 24 Nov '08

24 Nov '08

Hello, it looks like Wednesday 11/26 would be the best for all if we move it to 11am. Thus, I reserved a slot for 11am this Wednesday. The call-in information is: Conf Id: 151988 Date: 11/26/2008 Begin Time: 11:00:00 AM Eastern Time (US & Canada) Phone Number: (812) 856-3600 PIN 002118# I hope you can make it. We will mostly be discussing the nonblocking collectives proposal. Please read the current draft. http://www.unixer.de/sec/nbc-proposal-11-14-2008.pdf We will also have time to talk about the sparse collective aspects. As usual, all other topics are welcome and highly encouraged! All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Nonblocking collectives standard draft
by Christian Siebert 24 Nov '08

24 Nov '08

Hi Torsten et al., > the new version is in the wiki [1] and at [2]. Please check! Much better, thanks! I did a second reading and found several but only minor issues with the text. If you send me the TeX sources, I can send you my proposed corrections/improvements directly as a patch file (diff). Some more things for discussion: - p. 49 l. 41: MPI has the concept of error handlers. I don't see a reason why we should forbid ("return an error code") its usage. My suggestion is to simply delete the two sentences (from "If the call ..." to "... cases."). - p. 61 l 33-38: Should we really keep this Advice for implementors (same result for MPI_IREDUCE when called with same arguments)? For the blocking variant, there stands this argument of reproducibility vs. performance. However, the non-blocking variant could be optimized much better (e.g., might handle bad process skew) and we should consider to weaken this semantic restriction to allow this better performance. - another argument _for_ MPI_REQUEST_FREE is the existence of intercommunicators where all processes (i.e., in group A and in group B) have to call the collective but only part of them might be involved (e.g., for MPI_IBCAST all but the root process in group A don't really participate in the operation - they pass MPI_PROC_NULL as root argument). So having a lightweight MPI_Wait alternative might not be too bad ... The NBC examples are quite buggy (missing MPI_STATUS in MPI_Recv, missing semicolons, duplicate usage of same req-argument -> bad Waitall, ...) - but as with the text, I can send you corrected versions in the diff. Best regards, Christian -- Christian Siebert, Dipl.-Inf. Research Associate NEC Laboratories Europe, NEC Europe Ltd. Rathausallee 10, D-53757 Sankt Augustin, Germany Phone: +49 (0) 2241 - 92 52 44 Fax: +49 (0) 2241 - 92 52 99 (Registered Office: 1 Victoria Road, London W3 6BL, 2832014)

1 0

Re: [mpi3-coll] Nonblocking collectives standard draft
by Torsten Hoefler 14 Nov '08

14 Nov '08

On Fri, Nov 14, 2008 at 11:55:08AM -0500, Torsten Hoefler wrote: > I will upload a new version later today. the new version is in the wiki [1] and at [2]. Please check! All the Best, Torsten [1]: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/NBColl [2]: http://www.unixer.de/sec/nbc-proposal-11-14-2008.pdf -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Nonblocking collectives standard draft
by Torsten Hoefler 14 Nov '08

14 Nov '08

Hi Christian, > >>p 50, l 7: we should discuss MPI_REQUEST_FREE (again) - I talked to some > >> colleagues and we see application scenarios where this might be > >> useful (e.g., if only a subgroup participates in an irregular > >> collective); because this is a local operation, we also don't > >> see any problems by generally allowing MPI_REQUEST_FREE for > >> nonblocking collective operations > >I personally don't like request free > I also don't like MPI_REQUEST_FREE - but nevertheless this function is > already in the MPI standard. Purely subjective matters should not be > used for such decisions as long as there are good objective ones. So as > long as this function will not be deprecated for the whole MPI standard, > we are primarily forced to extend its definition also for nonblocking > collective operations. I never decided anything subjective. We had this discussion three times on our telecons and nobody spoke up for it. I also aked it two times to the Forum and nobody wanted to "rescue" request-free. > >because it makes thread-safe implementations really hard. > Can you please elaborate on this issue. I currently don't see any > problems related to threads... I will explain offline (something higher bandwidth - e.g., phone). If you're really interested look at this year's EuroPVM paper about thread safety. > >Also, I don't see a good portable way to check/ensure that the > >operation completed on the receiver side (we had this > >discussion before on the MPI-2.1 ML I think). Do you know one? > Just because we can't see a way at this time does not mean that no way > exists. that sounds rather random :). Would this not also be an argument for MPI_Flush_toilet :). > Message passing typically consists of two parts: data > transmission and synchronization. If the synchronization is already done > through a side channel, then we don't need to duplicate this work. Yes, but MPI is supposed to be portable, also to machines that are not cache coherent and/or use heavy message buffering. See the previous discussion for examples that break request-free. > An example is a scenario where the master broadcasts some tasks to the > slaves and then gathers the final results. We don't necessarily need > the synchronization part (i.e. MPI_Wait) for the broadcast because > once the master receives all results it is clear that all tasks have > been transmitted to the slaves. ok, that could/would work I think. Savings to avoid the local completion check would be minimal. Note that the synchronization is only local because the gather will synchronize anyway. > Another example includes a barrier where only > half of the processes want to participate - so instead of creating a new > communicator (expensive), we could simply call MPI_REQUEST_FREE (likely > much cheaper) at the uninterested ranks. no, they would still need to participate before the barrier completes, i.e., you would introduce unnecessary synchronization. And the programming style seems awful. We should discourage such a usage. > The other examples with the irregular collectives where some processes > supply zero-size entries still hold. They can also free the request > without waiting for any completion. same as the barrier - but yes, it could be valid. I don't want to think of the effects of missing progress (i.e., if one of those "freed" nodes does not call MPI for another 30 minutes, all others might wait). > I don't recommend to do it like in these examples, but they show that > there _might_ be certain use-cases. Programmers will use everything that you give them. The design of a good interface also means that pitfalls and bad side-effects are avoided. > I don't like to forbid something that is already defined without any > good reason - especially since this would increase the number of > exceptions in MPI which the user must be aware of. that's a good argument. I think we should reconsider to deprecate request-free unless we have a really good use-case and not just "there might be people who use it" - that does not mean anything. I'm sure there are still people who use goto to implement for loops :). I added a revisit of the discussions about request-free to the next meeting agenda. > >> The sentence in l 20 ("The behavior...") can and should be deleted. > >> "Nbc operations and bc operations [+are totally unrelated and ] > >> do not match each other." > >addition not made ("totally unrelated") - I think we shouldn't say this > >sinde they perform very similar operations and can be implemented on top > >of each other. > maybe - however, it might be a good idea to provide a hint to the reader > _why_ we decided that bc and nbc don't match each other. That they can > (and for performance reasons should) be implemented differently is such > a (hopefully) understandable hint. ok, that sounds reasonable. I added a rationale and an advice to users to the draft. > >>several locations: "memory movement" should be replaced by "data movement" > >> (but "movement" sounds bad anyway because you say > >> "after completion" - maybe "data placement" would be > >> more appropriate) > >the MPI standard says for example "The ``in place'' operations are > >provided to reduce unnecessary memory motion". I agree that it sounds > >strange, but it seems to be the used like this in MPI terms. But I am > >open to change it. > ``in place'' seems like something else to me because it specifies purely > local semantics whereas collectives define remote correlations. But this > seems more like a question for language experts or at least native > speakers - so I'll leave this open from my side and hope that someone > else jumps in ... I added it to the agende for the next meeting > >>p 61, l 31.8 & p 62, l 27.5: "after it completion" -> "after completion"? > >I don't see a mistake there. It says "after it completed" in the text. > yes, sorry for the typo - it just sounds "bad" could you elaborate or propose a rephrasing? > >>p 69, l 41: "in SPMD style programs" seems superflous to me > >> but "double buffering" could also be added as an example ... > >it's not superflous. One can also implement task-parallel or other > >programming paradigms with MPI. > I agree that there are other programming paradigms. But why can > pipelining techniques only be used in SPMD style programs? I know of at > least one application (similar to POP) that uses multiple programs but > nevertheless uses MPI collective operations. And it's not unlikely that > it could also benefit from NBC by applying pipelining/double buffering > techniques. the text does not exclude other uses. It just serves as an easy to understand example I think. > >>Example 5.33 seems to be very dangerous for at least two reasons: > >> 2) this doesn't make it a "global" allreduce because > >> (depending how sbuf and rbuf are initialized) all > >> seem to calculate different results > >> (0+0+1+2 vs. 0+1+1+2 vs. 1+2+0+2) > >exactly - that's what they want to do. > Then I didn't understand the example. To me the text suggested that this > approach might be used to simulate a global collective operation > without building a combined communicator... no, that would be useless. I just want to demonstrate collectives on overlapping communicators. This was requested by Rolf during the last meeting, and I like the idea. But we can work on the example if you have issues with it. > >This tiny example is probably not very useful, > that was the problem ;-) feel free to propose something more useful :). > > but easy to understand. > If an example doesn't make any sense, then it can also be anything but > easy to understand. The example could be more helpful if it would be > really useful instead of confusing. see above > >>What do we want to have (either or): > >> a) the same ordering over both blocking and nonblocking collectives > >> (suggested by Example 5.29) > >> b) blocking and nonblocking collectives are completely independent > >> (suggested by "reserved tag-space" p 50, l 15) > >>Upon a decision, the related text needs some clarification. > >we decided to go with a). The advice to implementors talks only about > >non-blocking point-to-point when it mentions the reserved tag space. > ok (this important point should be stated more clearly somewhere) I extended lines 17-22 on page 50. > I also noted that there is currently no statement that describes how an > implementation has to deal with the MPI_STATUS object. yes, this is on the agenda for the next meeting/telecon. I will upload a new version later today. Thanks & Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Nonblocking collectives standard draft
by Christian Siebert 14 Nov '08

14 Nov '08

Hi Torsten, > Hello Christian, > thanks for the review of the first draft. My pleasure! > All comments that I deleted are fixed, great > but I'd like to open the discussion of others up. of course >> btw. all line numbers within the MPI standard are not really line >> numbers (they have equal distances and as soon as there is another font >> size - like for the interface definitions - then you end up anywhere...) >> It's like "look at page X at _centimetre_ Y". > yes, I sent a note about this to the MPI-2.1 mailinglist a while ago, > but saw no reaction. too bad :-( >> p 49, l 46: it is not enough to state "receive buffer must not be read" >> it should be forbidden to access the receive buffer in any way >> (imagine an implementation that uses the buffer to forwards sth.) > Right, I changed this to be similar to MPI-2.1 p52 line 4-7. Now it > reads "must not be accessed". much better >> p 50, l 7: we should discuss MPI_REQUEST_FREE (again) - I talked to some >> colleagues and we see application scenarios where this might be >> useful (e.g., if only a subgroup participates in an irregular >> collective); because this is a local operation, we also don't >> see any problems by generally allowing MPI_REQUEST_FREE for >> nonblocking collective operations > I personally don't like request free I also don't like MPI_REQUEST_FREE - but nevertheless this function is already in the MPI standard. Purely subjective matters should not be used for such decisions as long as there are good objective ones. So as long as this function will not be deprecated for the whole MPI standard, we are primarily forced to extend its definition also for nonblocking collective operations. > because it makes thread-safe implementations really hard. Can you please elaborate on this issue. I currently don't see any problems related to threads... > Also, I don't see a good portable way to check/ensure that the > operation completed on the receiver side (we had this > discussion before on the MPI-2.1 ML I think). Do you know one? Just because we can't see a way at this time does not mean that no way exists. Message passing typically consists of two parts: data transmission and synchronization. If the synchronization is already done through a side channel, then we don't need to duplicate this work. An example is a scenario where the master broadcasts some tasks to the slaves and then gathers the final results. We don't necessarily need the synchronization part (i.e. MPI_Wait) for the broadcast because once the master receives all results it is clear that all tasks have been transmitted to the slaves. Another example includes a barrier where only half of the processes want to participate - so instead of creating a new communicator (expensive), we could simply call MPI_REQUEST_FREE (likely much cheaper) at the uninterested ranks. The other examples with the irregular collectives where some processes supply zero-size entries still hold. They can also free the request without waiting for any completion. I don't recommend to do it like in these examples, but they show that there _might_ be certain use-cases. I don't like to forbid something that is already defined without any good reason - especially since this would increase the number of exceptions in MPI which the user must be aware of. >> p 50, l 19: maybe we should be even more precise and add after >> "environments." something like "It is therefore erroneous to start a set >> of nbc in different logical orders on the participating ranks." > we didn't really define "different logical order" anywhere. This is why > we decided to go by examples. ok >> The sentence in l 20 ("The behavior...") can and should be deleted. >> "Nbc operations and bc operations [+are totally unrelated and ] >> do not match each other." > addition not made ("totally unrelated") - I think we shouldn't say this > sinde they perform very similar operations and can be implemented on top > of each other. maybe - however, it might be a good idea to provide a hint to the reader _why_ we decided that bc and nbc don't match each other. That they can (and for performance reasons should) be implemented differently is such a (hopefully) understandable hint. >> several locations: "memory movement" should be replaced by "data movement" >> (but "movement" sounds bad anyway because you say >> "after completion" - maybe "data placement" would be >> more appropriate) > the MPI standard says for example "The ``in place'' operations are > provided to reduce unnecessary memory motion". I agree that it sounds > strange, but it seems to be the used like this in MPI terms. But I am > open to change it. ``in place'' seems like something else to me because it specifies purely local semantics whereas collectives define remote correlations. But this seems more like a question for language experts or at least native speakers - so I'll leave this open from my side and hope that someone else jumps in ... >> p 61, l 31.8 & p 62, l 27.5: "after it completion" -> "after completion"? > I don't see a mistake there. It says "after it completed" in the text. yes, sorry for the typo - it just sounds "bad" >> btw. there seems to be a potential naming problem for beginners: >> "Exclusive Scan -> Exscan" && "Inclusive Scan -> Iscan"??? > Hmm, I wouldn't think so, but we can discuss this. This is just a minor issue and can possibly be neglected. >> p 69, l 41: "in SPMD style programs" seems superflous to me >> but "double buffering" could also be added as an example ... > it's not superflous. One can also implement task-parallel or other > programming paradigms with MPI. I agree that there are other programming paradigms. But why can pipelining techniques only be used in SPMD style programs? I know of at least one application (similar to POP) that uses multiple programs but nevertheless uses MPI collective operations. And it's not unlikely that it could also benefit from NBC by applying pipelining/double buffering techniques. > I added double-bufferng (even though > this is just a weaker form of pipelining). good point >> Example 5.33 seems to be very dangerous for at least two reasons: >> 2) this doesn't make it a "global" allreduce because >> (depending how sbuf and rbuf are initialized) all >> seem to calculate different results >> (0+0+1+2 vs. 0+1+1+2 vs. 1+2+0+2) > exactly - that's what they want to do. Then I didn't understand the example. To me the text suggested that this approach might be used to simulate a global collective operation without building a combined communicator... > Everybody wants to reduce in two communicators. ... ok, then it's fine > This tiny example is probably not very useful, that was the problem ;-) > but easy to understand. If an example doesn't make any sense, then it can also be anything but easy to understand. The example could be more helpful if it would be really useful instead of confusing. >> What do we want to have (either or): >> a) the same ordering over both blocking and nonblocking collectives >> (suggested by Example 5.29) >> b) blocking and nonblocking collectives are completely independent >> (suggested by "reserved tag-space" p 50, l 15) >> Upon a decision, the related text needs some clarification. > we decided to go with a). The advice to implementors talks only about > non-blocking point-to-point when it mentions the reserved tag space. ok (this important point should be stated more clearly somewhere) I also noted that there is currently no statement that describes how an implementation has to deal with the MPI_STATUS object. > Thank you very much for your review! Thanks for your draft! Have a nice weekend. Christian -- Christian Siebert, Dipl.-Inf. Research Associate NEC Laboratories Europe, NEC Europe Ltd. Rathausallee 10, D-53757 Sankt Augustin, Germany Phone: +49 (0) 2241 - 92 52 44 Fax: +49 (0) 2241 - 92 52 99 (Registered Office: 1 Victoria Road, London W3 6BL, 2832014)

1 0

Re: [mpi3-coll] Next teleconference
by Plum, Hans-Joachim 13 Nov '08

13 Nov '08

either would work for me -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Torsten Hoefler Sent: Donnerstag, 13. November 2008 03:35 To: mpi3-coll(a)lists.mpi-forum.org Subject: Re: [mpi3-coll] Next teleconference On Wed, Nov 12, 2008 at 08:43:08PM -0500, Torsten Hoefler wrote: > Can we please do another poll for the week from 11/24 to 11/28? It looks > like noon EST works pretty well, thus, I would propose: > > Wednesday, 11/19 > Friday, 11/21 sorry - I actually meant: Wednesday, 11/26 Friday, 11/28 All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608 _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

1 0

Re: [mpi3-coll] Nonblocking collectives standard draft
by Torsten Hoefler 13 Nov '08

13 Nov '08

Hello Christian, thanks for the review of the first draft. All comments that I deleted are fixed, but I'd like to open the discussion of others up. > btw. all line numbers within the MPI standard are not really line > numbers (they have equal distances and as soon as there is another font > size - like for the interface definitions - then you end up anywhere...) > It's like "look at page X at _centimetre_ Y". yes, I sent a note about this to the MPI-2.1 mailinglist a while ago, but saw no reaction. > p 49, l 46: it is not enough to state "receive buffer must not be read" > it should be forbidden to access the receive buffer in any way > (imagine an implementation that uses the buffer to forwards sth.) Right, I changed this to be similar to MPI-2.1 p52 line 4-7. Now it reads "must not be accessed". > p 50, l 7: we should discuss MPI_REQUEST_FREE (again) - I talked to some > colleagues and we see application scenarios where this might be > useful (e.g., if only a subgroup participates in an irregular > collective); because this is a local operation, we also don't > see any problems by generally allowing MPI_REQUEST_FREE for > nonblocking collective operations I personally don't like request free because it makes thread-safe implementations really hard. Also, I don't see a good portable way to check/ensure that the operation completed on the receiver side (we had this discussion before on the MPI-2.1 ML I think). Do you know one? > p 50, l 19: maybe we should be even more precise and add after > "environments." something like "It is therefore erroneous to start a set > of nbc in different logical orders on the participating ranks." we didn't really define "different logical order" anywhere. This is why we decided to go by examples. > The sentence in l 20 ("The behavior...") can and should be deleted. > "Nbc operations and bc operations [+are totally unrelated and ] > do not match each other." addition not made ("totally unrelated") - I think we shouldn't say this sinde they perform very similar operations and can be implemented on top of each other. > several locations: "memory movement" should be replaced by "data movement" > (but "movement" sounds bad anyway because you say > "after completion" - maybe "data placement" would be > more appropriate) the MPI standard says for example "The ``in place'' operations are provided to reduce unnecessary memory motion". I agree that it sounds strange, but it seems to be the used like this in MPI terms. But I am open to change it. > p 61, l 31.8 & p 62, l 27.5: "after it completion" -> "after completion"? I don't see a mistake there. It says "after it completed" in the text. > btw. there seems to be a potential naming problem for beginners: > "Exclusive Scan -> Exscan" && "Inclusive Scan -> Iscan"??? Hmm, I wouldn't think so, but we can discuss this. > p 69, l 41: "in SPMD style programs" seems superflous to me > but "double buffering" could also be added as an example ... it's not superflous. One can also implement task-parallel or other programming paradigms with MPI. I added double-bufferng (even though this is just a weaker form of pipelining). > Example 5.33 seems to be very dangerous for at least two reasons: > 2) this doesn't make it a "global" allreduce because > (depending how sbuf and rbuf are initialized) all > seem to calculate different results > (0+0+1+2 vs. 0+1+1+2 vs. 1+2+0+2) exactly - that's what they want to do. Everybody wants to reduce in two communicators. This tiny example is probably not very useful, but easy to understand. Think of nearest neighbor reductions. > This example is a good idea but needs some clarifications... > > What do we want to have (either or): > a) the same ordering over both blocking and nonblocking collectives > (suggested by Example 5.29) > b) blocking and nonblocking collectives are completely independent > (suggested by "reserved tag-space" p 50, l 15) > Upon a decision, the related text needs some clarification. we decided to go with a). The advice to implementors talks only about non-blocking point-to-point when it mentions the reserved tag space. > All in all a big step towards the standardization. However, there seem > to be quite a couple of mistakes in the copy-and-paste parts and I'm > quite sure that I didn't catch all in my first go... so please be > especially careful in those places. yes, this is the first draft to be circulated only in the workgroup and far away from an official reading. I attached an updated document to the wiki page [1] and put another convenience copy here [2]. Thank you very much for your review! All the Best, Torsten [1]: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/NBColl [2]: http://www.unixer.de/sec/nbc-proposal-11-13-2008.pdf -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0