December 2008 - mpiwg-coll - mailman3.cels.anl.gov

Re: [mpi3-coll] Updates after first reading
by Bronis R. de Supinski 31 Dec '08

31 Dec '08

Torsten: Re: > A small textual update as described in Ticket #93 (applied to > nonblocking collectives). As you might have guessed, I have been reading through the NBC proposal in detail. I created tickets for things that weren't (strictly) related to it -- thanks for catching that the #93 also applied to a portion of it so I don't need to report that here. I do have several other changes to suggest. I will put them in terms of the 12-31 version you just announced. Oops you just announced another. I'll try to catch any changes to pages or line numbers... I have two other things that I don't report here. One is related to ticket #86. I will open a ticket for the other. I think it looks like more than it really is. Let me know if you have any questions. Thanks, Bronis On page 1, we need to include the cross references for the immediate operations as well as the blocking ones. This leads to several small changes: Page 1, line 20: "(Section 5.3)" => "(Sections 5.3 and 5.12.1)" Page 1, line 22: "(Section 5.4)" => "(Sections 5.4 and 5.12.2)" Page 1, line 25: "(Section 5.5)" => "(Sections 5.5 and 5.12.3)" Page 1, line 29: "(Section 5.6)" => "(Sections 5.6 and 5.12.4)" Page 1, line 33: "(Section 5.7)" => "(Sections 5.7 and 5.12.5)" Page 1, line 37: "(Section 5.8)" => "(Sections 5.8 and 5.12.6)" Page 1, line 43: "(Section 5.9)" => "(Sections 5.9, 5.12.7, and 5.12.8)" Page 1, line 45: "(Section 5.10)" => "(Sections 5.10 and 5.12.9)" Page 1, line 48: "(Section 5.11)" => "(Sections 5.11, 5.12.10, and 5.12.11)" I have a few slightly more complicated changes to suggest for page 3. The sentence you added on line 16 ("The collective operations do not have a message tag argument.") does not have anything to do with type matching so it is misplaced in that location. It fits much better with the paragraph on lines 27-30. I think it should appear between lines 29 and 30, which would make that paragraph: Collective communication calls may use the same communicators as point-to-point communication; MPI guarantees that messages generated on behalf of collective communication calls will not be confused with messages generated by point-to-point communication. The collective operations do not have a message tag argument. A more detailed discussion of correct use of collective routines is found in Section 5.13. I think the intervening paragraph on page 3 (lines 17-26) also needs a little rewording and reordering. With the deletions omitted for clarity, it currently reads: Collective routine operations can (but are not required to) complete locally as soon as the caller's participation in the collective communication is finished. The local completion of a collective operation indicates that the caller is now free to access locations in the communication buffer. It does not indicate that other processes in the group have completed or even started the operation (unless otherwise implied by in the description of the operation). A blocking operation is complete as soon as the call returns. A nonblocking (immediate) call requires a separate completion operation, cf. ?? (Section 3.7). Thus, a collective communication operation may, or may not, have the effect of synchronizing all calling processes. This statement excludes, of course, the barrier operation. The sentences about when operations complete need to appear earlier in the paragraph. Where they are currently interferes with the goal of stating that completion does not imply synchronization. I also suggest a little rewording here: Collective routine operations can (but are not required to) complete locally as soon as the caller's participation in the collective communication is finished. A blocking operation completes locally as soon as the call returns. A nonblocking (immediate) call requires a separate completion operation, similarly to nonblocking point-to-point operations as discussed in Section 3.7. The local completion of a collective operation indicates that the caller is now free to access locations in the communication buffer. It does not indicate that other processes in the group have completed or even started the operation (unless otherwise implied by in the description of the operation). Thus, a collective communication operation may, or may not, have the effect of synchronizing all calling processes. This statement excludes, of course, the barrier operation. I have a few changes for pages 49-50, most of which may just be related to how you are setting up for eventual cross-references: Page 49, line 26: "As described in Chapter ?? (Section 3.7)," => "As described in Section 3.7," (use "Section", not "Chapter") Page 49, line 34: "?? (Section 3)" => "(Section 3.7)" (include the word "Section" and the parentheses as well as the subnumber) Page 49, line 40: "communicators which would" => "communicators that would" (the phrase is required so "that" is correct) Page 49, line 44: "?? (Section 3.7.1)," => "(Section 3.7.1)," (include the parentheses and the word "Section") Page 50, line 16: "?? (Section 3.7.5)," => "(Section 3.7.5)," (include the parentheses and the word "Section") Page 50, lines 39-40: "refer to ?? (Section 3.7.4)." => "refer to Section 3.7.4." (include the word "Section" but not the parentheses) A general observation is that the descriptions of the operations does not discuss intercommunicators. I think something needs to be said about how the use of intracommunicators and intercommunicators affect the results of the operation for each one. Most of the time, this can be handled by adding this phrase: ",whether on an intracommunicator or an intercommunicator" at the end of the sentences that state "data placements are identical after the operation completes" or "memory movement after completion is identical" or "equivalent to" the blocking operation. These sentences specifically occur on: page 52, lines 7-8; page 53, lines 36-37; page 54, lines 32-33; page 55, lines 35-36; page 56, lines 33-34; page 57, lines 32-33; page 58, lines 26-27; page 59, lines 32-33; page 61, lines 46-48; page 62, lines 31-32; page 63, lines 25-27; page 64, lines 9-11; page 64, lines 40-41; and page 65, lines 24-25. The statement for MPI_IALLTOALLV is terms of MPI_ALLTOALL so it is probably OK as it stands, although perhaps it would be better to reword the statement to be in terms of MPI_ALLTOALLV. In fact, I suggest adding the following on page 60 at the end of line 30: The data movement after an MPI_IALLTOALLALLV completes is identical to MPI_ALLTOALLV, whether on intracommunicator or an intercommunicator. The change for intercommunicators is more complicated for MPI_IBARRIER and relates to the outcome of ticket #30 so I think we should just make a note of the need for it here; maybe add something in parentheses on page 51 The sentence on page 52, lines 26-27 ("As in many of our example code fragments, we assume that some of the variables (such as comm in the above) have been assigned appropriate values.") already appears earlier in the chapter. There is no need to repeat it here so it should be deleted. Page 63, line 26: "operation which delivers" => "operation that delivers" Page 64, line 10: "operation which delivers" => "operation that delivers" Page 64, line 41: "operation which delivers" => "operation that delivers" Page 65, line 25: "operation which delivers" => "operation that delivers" On page 66, line 19, you use "cyclic dependency" and on line 23 you use "cyclic dependences". It is never clear to me which term is correct. I think either is in this case. However, you should be consistent. I have a slight preference for "cyclic dependency" so I suggest that you change the one on line 23 to "cyclic dependencies" although changing the one on line 19 to "cyclic dependence" would also be OK with me.

1 0

Re: [mpi3-coll] Updates after first reading
by Torsten Hoefler 31 Dec '08

31 Dec '08

Hello, Bronis' comments to Ticket #83 are now also incorporated into the newest proposal. The new version is available at [1] and the wiki [2]. The diff is at [3]. [1]: http://www.unixer.de/sec/nbc-proposal-12-31-2008-2.pdf [2]: https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/NBColl/nbc-pro… [3]: http://www.unixer.de/sec/nbc-proposal-12-31-2008-2.diff Please review all changes! Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Updates after first reading
by Torsten Hoefler 31 Dec '08

31 Dec '08

Hello, A small textual update as described in Ticket #93 (applied to nonblocking collectives). The new version is available at [1] and the wiki [2]. The diff is at [3]. [1]: http://www.unixer.de/sec/nbc-proposal-12-31-2008.pdf [2]: https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/NBColl/ [3]: http://www.unixer.de/sec/nbc-proposal-12-31-2008.diff Please review all changes! Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

[mpi3-coll] AUTO: Bin Jia/Poughkeepsie/IBM is out of the office. (returning 01/05/2009)
by Bin Jia 29 Dec '08

29 Dec '08

I am out of the office until 01/05/2009. I will respond to your note when I return. Note: This is an automated response to your message "[mpi3-coll] Updates after first reading" sent on 12/29/08 19:42:01. This is the only notification you will receive while this person is away.

1 0

[mpi3-coll] Updates after first reading
by Torsten Hoefler 29 Dec '08

29 Dec '08

Hello, I updated the document with all comments that I received at the last meeting. I created tickets for the more compplicated issues. I assigned four reviewers to each ticket and ask them to drive the discussion (the people who were most active in the discussion at the last meetings). However, everybody should review the changes and review the tickets. I am open for comments and suggestions. The new document is at [1] and in the wiki [2] and the LaTeX diff [3]. [1]: http://www.unixer.de/sec/nbc-proposal-12-29-2008.pdf [2]: https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/NBColl/nbc-pro… [3]: http://www.unixer.de/sec/nbc-proposal-12-29-2008.diff The detailed list of changes: 1) p1:18-41 - added all MPI_Ixxx calls to 5.1 2) p3:11 - "callers" changed to caller's 3) p3:22 - added new sentence (proposed by Jesper): "A blocking operation is complete as soon as the call returns. a Nonblocking call requires a separate completion operation" -> ref. to Sec 3.7 4) p3:23 - "call" changed to "operation" 5) p3:26 - "function" changed to "operation" 6) p5-7:all - added non-blocking versions (also to the next paragraphs) 7) p49:38-42 - added semantic advantages 8) p50:4-6 - removed send buffer restrictions! "All associated send buffers should not be modified and all associated receive buffers should not be accessed between init and completion" -> opened separate ticket (need to clarify send and receive buffers!?) #81 - reviewers: Erez, Darius, Jeff, Keith 9) p49:34 - added comma: communication"," which is eventually 10) p49:41-42 - system resources: asked Jeff for details (change P2P too)! #82 - reviewers: Jeff, Rich, Bronis, Keith 11) p50:7-10 - dropped "Advice to implementors" completely 12) p50:11-12 - replaced regexp by "all request test and completion functions (e.g., MPI_Wait)" 13) p50:13-16 - mention that mixing with other requests as possible (I/O, NB P2P) 14) p50:19-21 - rationale against MPI_Request_free, offline to ML (I updated it) #83 - reviewers: Jesper, Rolf, Bill, Erez 15) p50:19-28 - combine rationales and put them after the paragraph 16) p51:22 - add rational why cancel is not supported -> semantically unclear 17) p50:22 - point-to-point -> collective =) 18) p50:17 - new paragraph before "Coll operations" 19) p50:22-23 - rationale: "consistent for blocking call + additional benefit is HW support" 20) p50:31 - "in the same order per communicator" 21) p50:37 - have to -> could 22) p50:36 - only need -> would be 23) p50:40 - advice for users: "this is different than for P2P" 24) p51:1 - "will be ignored" -> ask Jeff (change text and advice to users) #84 - Jeff, Jesper, Rolf, Bronis 25) p51:9 - extra space "est )" (removed all spaces before ")" and after "(" in the whole chapter 26) p51:21-22 - "every process called MPI_Ibarrier" 27) p51:14 - operation -> call 28) pxx:xx - operation -> call! everywhere -> the word operation is overloaded (MPI Operation and Collective Operation) - do we care? #85 - Bronis, Jeff, Rolf, Bill 29) p67:24 - mixed -> blocking calls can be interleaved 30) p67:34 - after -> until 31) p68:33 - dropped "huge" 32) p69:44-48 - made advice weaker 33) p69:45 - should -> could 34) p69:44 - can potentially -> may 35) p69:44 - dropped "huge" 36) p69:44-48 - rephrase advice to implementors: Keith, Jeff, Bronis, Rolf #86 - Keith, Jeff, Bronis, Rolf 37) p70:11-24 - add picture 38) p70:27 - remove "very" Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Official reading of NBC - Re: Notes from Telecon
by Torsten Hoefler 17 Dec '08

17 Dec '08

I have to announce another (nonfundamental wording) update based on Rolf's third partial review and Jeff's review that I got this morning (thanks!)! The PDF is at [1] and [2]. The diff is at [3]. [1]: https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/NBColl/nbc-pro… [2]: http://www.unixer.de/sec/nbc-proposal-12-17-2008-2.pdf [3]: http://www.unixer.de/sec/nbc-proposal-12-17-2008-2.diff -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Official reading of NBC - Re: Notes from Telecon
by Torsten Hoefler 17 Dec '08

17 Dec '08

I have to announce another (nonfundamental wording) update based on Rolf's second partial review that I got this morning (thanks!)! The PDF is at [1] and [2]. The diff is at [3]. [1]: https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/NBColl/nbc-pro… [2]: http://www.unixer.de/sec/nbc-proposal-12-17-2008.pdf [3]: http://www.unixer.de/sec/nbc-proposal-12-17-2008.diff -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Official reading of NBC - Re: Notes from Telecon
by Torsten Hoefler 16 Dec '08

16 Dec '08

I have to announce another (nonfundamental wording) update based on Adam's review (thanks!)! The PDF is at [1] and [2]. The diff is at [3]. [1]: https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/NBColl/nbc-pro… [2]: http://www.unixer.de/sec/nbc-proposal-12-16-2008-2.pdf [3]: http://www.unixer.de/sec/nbc-proposal-12-16-2008-2.diff -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Official reading of NBC - Re: Notes from Telecon
by Torsten Hoefler 16 Dec '08

16 Dec '08

I have to announce another (nonfundamental wording) update based on Rolf's partial review (thanks!)! The PDF is at [1] and [2]. The diff is at [3]. [1]: https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/NBColl/nbc-pro… [2]: http://www.unixer.de/sec/nbc-proposal-12-16-2008.pdf [3]: http://www.unixer.de/sec/nbc-proposal-12-16-2008.diff Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Official reading of NBC - Re: Notes from Telecon
by Torsten Hoefler 15 Dec '08

15 Dec '08

Another (slight wording) update! The PDF is at [1] and [2]. The diff is at [3]. [1]: https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/NBColl/nbc-pro… [2]: http://www.unixer.de/sec/nbc-proposal-11-26-2008-2.pdf [3]: http://www.unixer.de/sec/nbc-proposal-12-15-2008-2.diff Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0