- mpiwg-coll - mailman3.cels.anl.gov

Re: [mpi3-coll] NBC Draft Revision 5
by Jesper Larsson Traeff 26 Feb '09

26 Feb '09

I'm also for option 2 Jesper On Wed, Feb 25, 2009 at 11:29:17AM -0800, Adam Moody wrote: > Hello all, > I opened a ticket to add the text to the collective intro regarding > access restrictions on MPI_IN_PLACE like we have in the NBC proposal: > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/131 > Erez reviewed it and pointed out that it's not really necessary. All > collectives specify MPI_IN_PLACE should be applied to the receive buffer > which has the stronger constraint, except for the root in scatter and > scatterv which does not receive anything. Then, after I looked over > things again, I realized that my current statements are actually more > restrictive than they need to be for scatter{v}. > > "When using the "in place" option, message buffers function as both > send and receive buffers. Such buffers should not be modified or > accessed until the operation completes." > > Since nothing is received at the root in scatter{v}, it could in fact > issue several NBC scatter{v} calls on different communicators (reusing > an active send buffer). To fix this, we have two options; > 1) Change "When using the 'in place' option, message buffers > function" to "When using the 'in place' option, receive message > buffers may function" > 2) Close ticket 131 and strike the similar lines in the NBC > proposal -- just rely on the implicit send/receive buffer constraints > I'm in favor of option #2 myself, since I'm not sure how much info #1 adds. > > If we go with option #2, it would leave a single-sentence paragraph on > page 51, line 11 in the NBCv5 text. We could append this sentence to > the paragraph above it. > > Any opinions? > -Adam > > Torsten Hoefler wrote: > > >Hello workgroup, > >I just posted revision 5 of the NBC draft to ticket #109. > > > >The draft is also available at [1] and a diff to revision 4 at [2]. > > > >[1]: http:// www. unixer.de/sec/nbc-proposal-rev-5.pdf > >[2]: http:// www. unixer.de/sec/nbc-proposal-rev-5.diff > > > >Please review and comment! > > > >All the Best, > > Torsten > > > > > > > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll

1 0

Re: [mpi3-coll] NBC Draft Revision 5
by Bronis R. de Supinski 25 Feb '09

25 Feb '09

I prefer option 2. On Wed, 25 Feb 2009, Adam Moody wrote: > Hello all, > I opened a ticket to add the text to the collective intro regarding > access restrictions on MPI_IN_PLACE like we have in the NBC proposal: > https:// svn.mpi-forum.org/trac/mpi-forum-web/ticket/131 > Erez reviewed it and pointed out that it's not really necessary. All > collectives specify MPI_IN_PLACE should be applied to the receive buffer > which has the stronger constraint, except for the root in scatter and > scatterv which does not receive anything. Then, after I looked over > things again, I realized that my current statements are actually more > restrictive than they need to be for scatter{v}. > > "When using the "in place" option, message buffers function as both > send and receive buffers. Such buffers should not be modified or > accessed until the operation completes." > > Since nothing is received at the root in scatter{v}, it could in fact > issue several NBC scatter{v} calls on different communicators (reusing > an active send buffer). To fix this, we have two options; > 1) Change "When using the 'in place' option, message buffers > function" to "When using the 'in place' option, receive message > buffers may function" > 2) Close ticket 131 and strike the similar lines in the NBC > proposal -- just rely on the implicit send/receive buffer constraints > I'm in favor of option #2 myself, since I'm not sure how much info #1 adds. > > If we go with option #2, it would leave a single-sentence paragraph on > page 51, line 11 in the NBCv5 text. We could append this sentence to > the paragraph above it. > > Any opinions? > -Adam > > Torsten Hoefler wrote: > > >Hello workgroup, > >I just posted revision 5 of the NBC draft to ticket #109. > > > >The draft is also available at [1] and a diff to revision 4 at [2]. > > > >[1]: http:// www. unixer.de/sec/nbc-proposal-rev-5.pdf > >[2]: http:// www. unixer.de/sec/nbc-proposal-rev-5.diff > > > >Please review and comment! > > > >All the Best, > > Torsten > > > > > > > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http:// lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll > >

1 0

Re: [mpi3-coll] NBC Draft Revision 5
by Adam Moody 25 Feb '09

25 Feb '09

Hello all, I opened a ticket to add the text to the collective intro regarding access restrictions on MPI_IN_PLACE like we have in the NBC proposal: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/131 Erez reviewed it and pointed out that it's not really necessary. All collectives specify MPI_IN_PLACE should be applied to the receive buffer which has the stronger constraint, except for the root in scatter and scatterv which does not receive anything. Then, after I looked over things again, I realized that my current statements are actually more restrictive than they need to be for scatter{v}. "When using the "in place" option, message buffers function as both send and receive buffers. Such buffers should not be modified or accessed until the operation completes." Since nothing is received at the root in scatter{v}, it could in fact issue several NBC scatter{v} calls on different communicators (reusing an active send buffer). To fix this, we have two options; 1) Change "When using the 'in place' option, message buffers function" to "When using the 'in place' option, receive message buffers may function" 2) Close ticket 131 and strike the similar lines in the NBC proposal -- just rely on the implicit send/receive buffer constraints I'm in favor of option #2 myself, since I'm not sure how much info #1 adds. If we go with option #2, it would leave a single-sentence paragraph on page 51, line 11 in the NBCv5 text. We could append this sentence to the paragraph above it. Any opinions? -Adam Torsten Hoefler wrote: >Hello workgroup, >I just posted revision 5 of the NBC draft to ticket #109. > >The draft is also available at [1] and a diff to revision 4 at [2]. > >[1]: http:// www. unixer.de/sec/nbc-proposal-rev-5.pdf >[2]: http:// www. unixer.de/sec/nbc-proposal-rev-5.diff > >Please review and comment! > >All the Best, > Torsten > > >

1 0

[mpi3-coll] NBC Draft Revision 5
by Torsten Hoefler 20 Feb '09

20 Feb '09

Hello workgroup, I just posted revision 5 of the NBC draft to ticket #109. The draft is also available at [1] and a diff to revision 4 at [2]. [1]: http://www.unixer.de/sec/nbc-proposal-rev-5.pdf [2]: http://www.unixer.de/sec/nbc-proposal-rev-5.diff Please review and comment! All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

[mpi3-coll] Question on reduction operation in intercommunicators
by Adam Moody 13 Feb '09

13 Feb '09

Hi everyone, The spec is a bit fuzzy regarding the op parameter for reduction-type collectives on intercommunicators. There are a few lines in the intracommunicator text under MPI_REDUCE, which says: page 161, lines 11-15: "Furthermore, the datatype and op given for predefined operators must be the same on all processes." "Note, that it is possible for users to supply different user-defined operations to MPI_REDUCE in each process. MPI does not define which operations are used on which operands in this case." However, nothing is explicitly said along such lines for intercommunicators. I assume by default the above statements must extend to intercommunicators as well. I have two questions. 1) For intracommunicators, I understand the restriction on having the same op specified by all processes for predefined operators -- performance. I could see a similar argument being made that all processes (from both groups) in an intercomm also specify the same op for predefined ops. However, I also see utility in forcing all processes within a group to specify the same op, but allowing each group to specify different ops (e.g., one may want the maximum of values from group A and the minimum of values from group B). The downside in this case is that processes of one group don't know the operation used by processes in the other group without some communication. Also, we'd need to be precise in specifying what data the two ops apply to (i.e., do I apply the op specified in group A to the data from group A or the data from group B?). 2) A similar question arises for user-defined operators, since here the current spec says that different processes may specify different ops. It says MPI can apply which ever ops to which ever operands in such a case for intracommunicators. Again, I could see value in enabling the user to specify the same user-defined op within a group, but different ops across groups. Any thoughts? -Adam

1 0

Re: [mpi3-coll] Additional edits 2
by Torsten Hoefler 11 Feb '09

11 Feb '09

Hello working group, I applied some of the small fixes proposed by Adam and others. Revision 4 is now attached to ticket #109 and at [1]. The diff file is available at [2]. [1]: http://www.unixer.de/sec/nbc-proposal-rev-4.pdf [2]: http://www.unixer.de/sec/nbc-proposal-rev-4.diff Rev 4 will be read/discussed tomorrow. Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Additional edits 2
by Torsten Hoefler 11 Feb '09

11 Feb '09

On Wed, Feb 04, 2009 at 10:19:23AM -0800, Adam Moody wrote: > Also against revision 3. > > I'd like to make the language in the descriptions for the data movement > collectives consistent across the different collectives. The text skips > around with several different flavors. Personally, I like this template > best: > > "MPI_Ixxx starts a nonblocking variant of MPI_Xxx. The data movement > after MPI_Ixxx completes is identical to the data movement after MPI_Xxx > completes (see Section 5.Y)." > > This would apply to bcast, gather, gatherv, scatter, scatterv, > allgather, allgatherv, alltoall, alltoallv, and alltoallw. We could add > mention of inter- and intra-communicators here if folks would like to, > but I like the simplicity of the above template myself -- leave all > description up to the corresponding blocking section and provide a back > reference. yes, we will certainly find a single template - however, I'd like to discuss this tomorrow. > Of couse, for the reductions: reduce, allreduce, reduce_scatter, scan, > and exscan; we still need more discussion on what "delivers the same > results" mean in the current description since the blocking and > nonblocking operations may use different algorithms and we have the > associativity issue. yes, clearly - also up for discussion tomorrow! -> updated agenda Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Additional edits
by Torsten Hoefler 11 Feb '09

11 Feb '09

Hi, > I don't know if we want to list *every* section for the reduction > collectives. (We don't do this for other collectives like alltoall, > alltoallv, alltoalw) I think we could simplify a couple: > p 1, lines 41-42 > Simplify: > "where the result is returned to all members of a group (Section > 5.9.6 and Section 5.12.8) and a variation where the result is returned > to only one member (Section 5.9 and Section 5.12.7)." > to: > "where the result is returned to all members of a group and a > variation where the result is returned to only one member (Section 5.9, > Section 5.12.7, and Section 5.12.8)." nope, we actually do this for Alltoall > p 1, lines 47-48 > Simplify: > "(Section 5.11, Section 5.11.2, Section 5.12.10, and Section 5.12.11)" > to: > "(Section 5.11, Section 5.12.10, and Section 5.12.11)" same, I think it's important to either list all sections or none. I am in favor of listing the sections right now (not strongly, but it's consistent with 2.2). > Mixing call and operation > p 2, lines 18-20 > Change: > "A blocking operation is complete as soon as the call returns. A > nonblocking (immediate) call requires a separate completion operation" > to: > "A blocking operation is complete as soon as the call returns. A > nonblocking operation requires a separate completion call" nope, that's actually meant as written, i.e., if I start an operation (with a nonblocking call), then I *must* complete it. > Missing a "nonblocking": > p 50, line 5 > "Similarly to the blocking case, collective operations ..." --> > "Similarly to the blocking case, nonblocking collective operations ..." I think it's correct as stated (this sentence applies to blocking and nonblocking colls) but sounds awkward -> changed > We should also stress local-ness of completion in the "i.e." phrase > p 5, line 7 > "i.e., the semantics of the operation are guaranteed" --> "i.e., > for the caller, the semantics of the operation are guaranteed" hmm, that doesn't change anything - who's the caller? Everybody has to call the collective, so everybody is the caller. We could introduce the notion of local completion again here, but we just removed it because we defined "completion" as generally local. > Since we are now using the word "indicate" in the first sentence, it'd > be good to use this same word in the following "also" sentence. > p 50, line 10 > "does not imply" --> "does not indicate" ok, I don't care > I think it'd read better to mention the "in place" buffer restriction > following the other sentence about "in place" near the end. > p 50, line 1-2, remove sentences > "When using the "in place" option, ... Such buffers ... until the > operation completes." > p 51, lines 9-14: Split this paragraph into two and add the removed > sentences from above (I also added "Once an operation is initiated"). > The result paragraphs read: > In terms of data movements, each nonblocking collective operation has > the same effect as its blocking counterpart for the intracommunicators > and intercommunicators after completion. Likewise, upon completion, > nonblocking collective reduction operations have the same effect as their > blocking counterparts, and the same restrictions and recommendations on > reduction orders apply. > The use of the "in place" option is allowed exactly as described for > the corresponding blocking collective operations. When using the "in > place" option, message buffers function as both send and receive > buffers. Once an operation is initiated, such buffers should not be > modified or accessed until the operation completes. ok, I don't mind - changed > Verb tense of "call" and keep the "only" keyword from blocking description: > p 51, line 36 > "the operation completes after every process in the communicator > called MPI_IBARRIER" --> "the operation completes only after every > process in the communicator calls MPI_IBARRIER" hmm, I don't mind - but I used "have called" as tense because that's consistent with the blocking description. > Expand intercommunicator sentence for MPI_Ibarrier > p 51, lines 36-37 > Replace: > "On intercommunicators, the operation completes when the MPI_BARRIER > call would return." > with: > "On intercommunicators, the operation completes for processes in one > group (group A) of the intercommunicator only after all processes in the > other group (group B) call MPI_IBARRIER." ok, we need to merge all other MPI-2.2. items later also ;). Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

[mpi3-coll] Additional edits 2
by Adam Moody 04 Feb '09

04 Feb '09

Also against revision 3. I'd like to make the language in the descriptions for the data movement collectives consistent across the different collectives. The text skips around with several different flavors. Personally, I like this template best: "MPI_Ixxx starts a nonblocking variant of MPI_Xxx. The data movement after MPI_Ixxx completes is identical to the data movement after MPI_Xxx completes (see Section 5.Y)." This would apply to bcast, gather, gatherv, scatter, scatterv, allgather, allgatherv, alltoall, alltoallv, and alltoallw. We could add mention of inter- and intra-communicators here if folks would like to, but I like the simplicity of the above template myself -- leave all description up to the corresponding blocking section and provide a back reference. Of couse, for the reductions: reduce, allreduce, reduce_scatter, scan, and exscan; we still need more discussion on what "delivers the same results" mean in the current description since the blocking and nonblocking operations may use different algorithms and we have the associativity issue. -Adam

1 0

[mpi3-coll] Additional edits
by Adam Moody 03 Feb '09

03 Feb '09

Here are a few more edits -- against revision 3. -Adam I don't know if we want to list *every* section for the reduction collectives. (We don't do this for other collectives like alltoall, alltoallv, alltoalw) I think we could simplify a couple: p 1, lines 41-42 Simplify: "where the result is returned to all members of a group (Section 5.9.6 and Section 5.12.8) and a variation where the result is returned to only one member (Section 5.9 and Section 5.12.7)." to: "where the result is returned to all members of a group and a variation where the result is returned to only one member (Section 5.9, Section 5.12.7, and Section 5.12.8)." p 1, lines 47-48 Simplify: "(Section 5.11, Section 5.11.2, Section 5.12.10, and Section 5.12.11)" to: "(Section 5.11, Section 5.12.10, and Section 5.12.11)" Mixing call and operation p 2, lines 18-20 Change: "A blocking operation is complete as soon as the call returns. A nonblocking (immediate) call requires a separate completion operation" to: "A blocking operation is complete as soon as the call returns. A nonblocking operation requires a separate completion call" Missing a "nonblocking": p 50, line 5 "Similarly to the blocking case, collective operations ..." --> "Similarly to the blocking case, nonblocking collective operations ..." We should also stress local-ness of completion in the "i.e." phrase p 5, line 7 "i.e., the semantics of the operation are guaranteed" --> "i.e., for the caller, the semantics of the operation are guaranteed" Since we are now using the word "indicate" in the first sentence, it'd be good to use this same word in the following "also" sentence. p 50, line 10 "does not imply" --> "does not indicate" I think it'd read better to mention the "in place" buffer restriction following the other sentence about "in place" near the end. p 50, line 1-2, remove sentences "When using the "in place" option, ... Such buffers ... until the operation completes." p 51, lines 9-14: Split this paragraph into two and add the removed sentences from above (I also added "Once an operation is initiated"). The result paragraphs read: In terms of data movements, each nonblocking collective operation has the same effect as its blocking counterpart for the intracommunicators and intercommunicators after completion. Likewise, upon completion, nonblocking collective reduction operations have the same effect as their blocking counterparts, and the same restrictions and recommendations on reduction orders apply. The use of the "in place" option is allowed exactly as described for the corresponding blocking collective operations. When using the "in place" option, message buffers function as both send and receive buffers. Once an operation is initiated, such buffers should not be modified or accessed until the operation completes. Verb tense of "call" and keep the "only" keyword from blocking description: p 51, line 36 "the operation completes after every process in the communicator called MPI_IBARRIER" --> "the operation completes only after every process in the communicator calls MPI_IBARRIER" Expand intercommunicator sentence for MPI_Ibarrier p 51, lines 36-37 Replace: "On intercommunicators, the operation completes when the MPI_BARRIER call would return." with: "On intercommunicators, the operation completes for processes in one group (group A) of the intercommunicator only after all processes in the other group (group B) call MPI_IBARRIER."

1 0