November 2008 - mpiwg-coll - mailman3.cels.anl.gov

Re: [mpi3-coll] Nonblocking collectives standard draft
by Christian Siebert 13 Nov '08

13 Nov '08

Hello Torsten, excellent! However, there are some remarks and corrections from my (our) side: p 1: "Non-Blocking" -> "Nonblocking" ;-) btw. all line numbers within the MPI standard are not really line numbers (they have equal distances and as soon as there is another font size - like for the interface definitions - then you end up anywhere...) It's like "look at page X at _centimetre_ Y". p 49, l 46: it is not enough to state "receive buffer must not be read" it should be forbidden to access the receive buffer in any way (imagine an implementation that uses the buffer to forwards sth.) p 50, l 7: we should discuss MPI_REQUEST_FREE (again) - I talked to some colleagues and we see application scenarios where this might be useful (e.g., if only a subgroup participates in an irregular collective); because this is a local operation, we also don't see any problems by generally allowing MPI_REQUEST_FREE for nonblocking collective operations p 50, l 13: "a simple local execution schedule" -> "local execution schedules" (alternatively "A nonblocking collective operation..."; however, please omit this subjective "simple") p 50, l 19: maybe we should be even more precise and add after "environments." something like "It is therefore erroneous to start a set of nbc in different logical orders on the participating ranks." The sentence in l 20 ("The behavior...") can and should be deleted. "Nbc operations and bc operations [+are totally unrelated and ] do not match each other." p 52, l 41.5: the C++ interface name is for the blocking variant l 45: "non-blocking" -> "nonblocking" several locations: "memory movement" should be replaced by "data movement" (but "movement" sounds bad anyway because you say "after completion" - maybe "data placement" would be more appropriate) p 58, l 23.6 and 28.4: interface name for C++ & Fortran is for blocking var. p 59, l 17.2: Fortran interface name... p 61, l 31.8 & p 62, l 27.5: "after it completion" -> "after completion"? p 63, l 11: "reduction-to-all" for MPI_REDUCE_SCATTER? p 63, l 39-41 ... btw. there seems to be a potential naming problem for beginners: "Exclusive Scan -> Exscan" && "Inclusive Scan -> Iscan"??? p 64, l 24-27 ... p 68, l 46.3: "req[1]" -> "req" p 69, l 20: "req[1]" -> "req[0]" p 69, l 41: "in SPMD style programs" seems superflous to me but "double buffering" could also be added as an example ... p 70, l 7: "t" -> "to" Example 5.33 seems to be very dangerous for at least two reasons: 1) sbuf and rbuf must be carefully initialized (using the same for both operations seems nonsense to me) 2) this doesn't make it a "global" allreduce because (depending how sbuf and rbuf are initialized) all seem to calculate different results (0+0+1+2 vs. 0+1+1+2 vs. 1+2+0+2) This example is a good idea but needs some clarifications... What do we want to have (either or): a) the same ordering over both blocking and nonblocking collectives (suggested by Example 5.29) b) blocking and nonblocking collectives are completely independent (suggested by "reserved tag-space" p 50, l 15) Upon a decision, the related text needs some clarification. All in all a big step towards the standardization. However, there seem to be quite a couple of mistakes in the copy-and-paste parts and I'm quite sure that I didn't catch all in my first go... so please be especially careful in those places. Many thanks! Best regards, Christian Torsten Hoefler wrote: > Hello, > as promised in the last forum, I finished the draft of our standard > proposal. I don't want to post it to the mailinglist, so I uploaded it > to the wiki pages [1] and put a copy at [2] for convenience. > > The document includes the whole collective chapter and the additions > (some references to other chapters are thus broken). All changes are > clearly marked: > - Striked text has been removed and underlined text has been added. > - Section 5.12 is new (not underlined) > - Examples 5.28-5.33 are new (not underlined) > > Please read the additions and send comments to me! > > I hope to be able to discuss the proposal during the next teleconference > (which will be announced in a separate mail). > > Thanks & All the Best, > Torsten > > [1]: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/NBColl > [2]: http://www.unixer.de/sec/nbc-proposal-11-12-2008.pdf -- Christian Siebert, Dipl.-Inf. Research Associate NEC Laboratories Europe, NEC Europe Ltd. Rathausallee 10, D-53757 Sankt Augustin, Germany Phone: +49 (0) 2241 - 92 52 44 Fax: +49 (0) 2241 - 92 52 99 (Registered Office: 1 Victoria Road, London W3 6BL, 2832014)

1 0

[mpi3-coll] Design issues/possibilities for Sparse Collectives
by Jesper Larsson Traeff 13 Nov '08

13 Nov '08

Dear coll WG, Torsten and I have tried to write up more systematically some of the issues and possibilities that arise in the design of sparse collectives for MPI, and made a concrete proposal for a small set of sparse collectives. We hope this will generate some serious discussion; all sorts of comments, suggestions, criticisms of course most welcome. Leaving aside improvements and changes to the interface, the more interesting question to the Forum is of course whether this kind of functionality helps enough users and/or library builders enough that it should be included in a future MPI standard The discussion/proposal has been put on the Wiki, Collectives WG under III. Sparse/Topological collectives best regards Jesper

1 0

Re: [mpi3-coll] Next teleconference
by Jesper Larsson Traeff 13 Nov '08

13 Nov '08

oops, sorry: Wednesday, 11/26 would be best for me Jesper On Thu, Nov 13, 2008 at 10:14:12AM +0100, Jesper Larsson Traeff wrote: > > dear all, > > both days, should work for me, strong preference for > > Wednesday, 11/28 > > though! > > Jesper > > On Wed, Nov 12, 2008 at 09:35:20PM -0500, Torsten Hoefler wrote: > > On Wed, Nov 12, 2008 at 08:43:08PM -0500, Torsten Hoefler wrote: > > > Can we please do another poll for the week from 11/24 to 11/28? It looks > > > like noon EST works pretty well, thus, I would propose: > > > > > > Wednesday, 11/19 > > > Friday, 11/21 > > sorry - I actually meant: > > > > Wednesday, 11/26 > > Friday, 11/28 > > > > All the Best, > > Torsten > > > > -- > > bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- > > Torsten Hoefler | Postdoctoral Researcher > > Open Systems Lab | Indiana University > > 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA > > Lindley Hall Room 135 | +01 (812) 855-3608 > > _______________________________________________ > > mpi3-coll mailing list > > mpi3-coll(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll

1 0

Re: [mpi3-coll] Next teleconference
by Jesper Larsson Traeff 13 Nov '08

13 Nov '08

dear all, both days, should work for me, strong preference for Wednesday, 11/28 though! Jesper On Wed, Nov 12, 2008 at 09:35:20PM -0500, Torsten Hoefler wrote: > On Wed, Nov 12, 2008 at 08:43:08PM -0500, Torsten Hoefler wrote: > > Can we please do another poll for the week from 11/24 to 11/28? It looks > > like noon EST works pretty well, thus, I would propose: > > > > Wednesday, 11/19 > > Friday, 11/21 > sorry - I actually meant: > > Wednesday, 11/26 > Friday, 11/28 > > All the Best, > Torsten > > -- > bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- > Torsten Hoefler | Postdoctoral Researcher > Open Systems Lab | Indiana University > 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA > Lindley Hall Room 135 | +01 (812) 855-3608 > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll

1 0

Re: [mpi3-coll] Next teleconference
by Richard Graham 12 Nov '08

12 Nov '08

11/26 might work 11/28 won¹t work Rich On 11/12/08 9:35 PM, "Torsten Hoefler" <htor(a)cs.indiana.edu> wrote: > On Wed, Nov 12, 2008 at 08:43:08PM -0500, Torsten Hoefler wrote: >> > Can we please do another poll for the week from 11/24 to 11/28? It looks >> > like noon EST works pretty well, thus, I would propose: >> > >> > Wednesday, 11/19 >> > Friday, 11/21 > sorry - I actually meant: > > Wednesday, 11/26 > Friday, 11/28 > > All the Best, > Torsten > > -- > bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- > Torsten Hoefler | Postdoctoral Researcher > Open Systems Lab | Indiana University > 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA > Lindley Hall Room 135 | +01 (812) 855-3608 > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll >

1 0

Re: [mpi3-coll] Next teleconference
by Torsten Hoefler 12 Nov '08

12 Nov '08

On Wed, Nov 12, 2008 at 08:43:08PM -0500, Torsten Hoefler wrote: > Can we please do another poll for the week from 11/24 to 11/28? It looks > like noon EST works pretty well, thus, I would propose: > > Wednesday, 11/19 > Friday, 11/21 sorry - I actually meant: Wednesday, 11/26 Friday, 11/28 All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

[mpi3-coll] Next teleconference
by Torsten Hoefler 12 Nov '08

12 Nov '08

Hello, I leared from Rich that the Thursday after SC is Thanksgiving. I guess we won't have too many participants if we do it at noon. Can we please do another poll for the week from 11/24 to 11/28? It looks like noon EST works pretty well, thus, I would propose: Wednesday, 11/19 Friday, 11/21 What do you think? New proposals (time,date) are welcome! Thanks & All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

[mpi3-coll] Nonblocking collectives standard draft
by Torsten Hoefler 12 Nov '08

12 Nov '08

Hello, as promised in the last forum, I finished the draft of our standard proposal. I don't want to post it to the mailinglist, so I uploaded it to the wiki pages [1] and put a copy at [2] for convenience. The document includes the whole collective chapter and the additions (some references to other chapters are thus broken). All changes are clearly marked: - Striked text has been removed and underlined text has been added. - Section 5.12 is new (not underlined) - Examples 5.28-5.33 are new (not underlined) Please read the additions and send comments to me! I hope to be able to discuss the proposal during the next teleconference (which will be announced in a separate mail). Thanks & All the Best, Torsten [1]: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/NBColl [2]: http://www.unixer.de/sec/nbc-proposal-11-12-2008.pdf -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Non-blocking Collectives Proposal Draft
by Torsten Hoefler 09 Nov '08

09 Nov '08

Hi Rich, thanks for your comments! I will comment below: > Section 1.2 paragraph 2: The matching of those operations a is ruled by > the order.... This is a bit confusing, as the paragraph mentions both > point-to-point operations and blocking collective operations. I would > suggest changing this to "The matching of those blocking collective > operations is rules by the order ...." fixed > Section 2: "High-quality ..." - In general I am against these sorts of > comments, even though they are strewn throughout the standard. There are > many tradeoffs in implementing a communications library, and what may be > good in one instance, may not be appropriate in another. It may be more > appropriate to state "The enables the application to take advantage of > asynchronous progress, if the implementation implements such a > capability..." While I agree with the sentiment, I disagree with the > categorization. yes/no: I'm hesitant to change this before we actually clarify the intent of those statements in the remaining standard. I will bring this up at the next meeting. > Section 2.1: Instead of using the term nested collectives, multiple > outstanding non-blocking collectives seems clearer to me. fixed > The comment that calling MPI_Request_free() is not > useful on the send side is not quite clear to me. Why is the send side > different from the receive side ? In either case, I do not think that we > should allow freeing a request in the middle of a collective (like one can > for ptp communications) The text actually states: "Freeing a request is only useful at the sender side and not on the receiver side." What I meant refers to MPI-2.1 page 55 lines 22-23: An active receive request should never be freed as the receiver will have no way to verify that the receive has completed and the receive buffer can be reused." I think request_free makes so many problems and is really not very useful. We should think again about deprecating it. But we should certainly not allow it for collectives! > The paragraph on the bottom of page 3 is confusing. After mentioning > that we can have multiple outstanding collectives, the last sentence seems > to imply that in such a case, all would have to be of the same type (such > as ibcast), which I do not think is the intent. this will be completely rephrased and improved with the examples from the last meeting in the new version. > Section 2.3: As I mentioned before, I do not believe we should specify > that an implementation should support more than a minimum of 1 (i.e must > provide support for this). Especially as the system sizes increase > markedly, we need to be careful on what sort of resource requirements we > place on an implementation. Hmm, I tend to disagree because if we don't guarantee anything, the it's really hard to write portable programs (i.e., all codes I know that use NBC would not be portable). Also, as we define it right now, it would be really easy to fall back to a software implementation which is available anytime. I would also claim that a software installation can be built with resource requirements that scale with O(1) with the number of processors (this implementation would be slow which is certainly bettar than not working at all). The interesting question is, how MPI-2.1 handles this issue with nonblocking point-to-point operations. Phrased differently: is an implementation that fails for every Isend/Irecv with "out of resources" a legal MPI-2.1 implementation? If yes, I would agree to the rephrasing, if no, then I would like to reconsider this topic. I know that we said in the Forum that this is outside the scope of MPI. But I think we should define a minimal quality (as we do for the tag space) in order to enable portable programming. Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

[mpi3-coll] Our MPI-2.2 Items
by Torsten Hoefler 09 Nov '08

09 Nov '08

Hello workgroup, our WG has several open MPI-2.2 items. I am listing them here for discussion. We should make progress on all of them (i.e., bring them to e review-ready state or request more feedback from the forum). We should discuss the remaining open items in the mailinglist and/or during our next teleconference (the Thursday after SC08). Here are the items: #23 (Generalized v-collectives: allow unknown recvcounts / rdispls): - seems to have several problems and the benefits seem to be very limited (main concerns: benefits only exist for small messages and an upper bound to the cumulative message size has to be known before the operation. So the user could as well do a non-vector operations with slightly larger buffers and achieve higher performance with negligible memory overhead. It is also unclear how to implement this performing (e.g., with a tree algorithm) without excessive memory copies and forcing serial receives.) #24 (Add a local Reduction Function): - seems to be ready for first reading - Jesper volunteered to add required changes to document, I will review #27 (Regular (non-vector) version of MPI_Reduce_scatter): - is missing a use-case, implementation and a clear benefit #30 (Clarification to intercomm MPI_Barrier): - awaiting reviews #31 (Add MPI_IN_PLACE option to Alltoall): - it's still unclear whether we want MPI_IN_PLACE or MPI_Alltoall_replace() - we should have a straw-vote during the next telecon (after SC) and update the proposal #33 (Fix Scalability Issues in Graph Topology Interface): - issue if we want weights (and what weights) or not is open - otherwise it seems ready for a first reading - we should come to a conclusion with the weight issues before proceeding and update the proposal #38 (Matched probe/receive): - seems to fit better in p2p WG - will move over #44 (Non blocking versus non-blocking versus nonblocking): - trivial, ready for reviews by chapter authors #26 (Add a callback function if a request completes): - relatively clear "no" for 2.2 - will move to p2p WG New additions are still welcome! All the best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0