- mpi-comments - mailman3.cels.anl.gov

Re: [Mpi-comments] Comments on MPI 3.0 draft 2 [part 2]
by William Gropp 19 Aug '12

19 Aug '12

Jeremiah, Thanks for your very detailed review; we'll go over these carefully. On your first point about the style of the standard, this has been a long-term problem, as the audience for the document includes both implementors and users. It is something to remember, as the most important role for the document is as a standard, not a user guide. Bill William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign On Aug 20, 2012, at 7:06 AM, Jeremiah Willcock wrote: > These are in reference to the draft at > <URL:http://meetings.mpi-forum.org/mpi3.0_draft_2.pdf>. Thank you for considering them. > > -- Jeremiah Willcock > > General: > > [post 3.0] Overall, the document reads like a tutorial rather than a reference document or standard (for example, going from simpler topics to more advanced ones, forward references for datatypes and communicators in the point-to-point chapter, ignoring MPI_PROC_NULL when talking about point-to-point operations then introducing it later in the chapter, and the wording on page 225 lines 13-16). > > It would be nice to know, for each function, constant, etc., which MPI version introduced it. This would allow developers to write MPI-3.0-compliant programs that would also work on older MPI versions. > > Chapter 5: > > Why are nonblocking collectives in their own section rather than mixed in with the normal collectives (i.e., put each collective in both blocking and nonblocking versions together)? > > Page 141: MPI_REDUCE_LOCAL is not mentioned here at all (it isn't a communication operation, but it is part of the contents of this chapter). > > Page 141 lines 33 and 36: I don't think "gather" and "scatter" need to be capitalized. > > Page 141 lines 40-43: The reduction paragraph here looks very different from the others (has the ALL* version first, groups them together with single-root reduce as a variant version, and points to a subsection for allreduce while pointing to its whole containing section for reduce). > > Given the pattern of the other sections, allreduce and reduce should be in separate sections, with reduce first followed by allreduce. > > Reduce_scatter_block and reduce_scatter should be in the same subsection for the nonblocking versions, just like allgather and allgatherv (for example) are. > > Page 142 lines 1-3: It's odd to point to section 5.11 then subsection 5.11.2; these should probably be combined. > > Page 144 line 9: What are the "auxiliary functions" referring to? Do they require any access to the communication library outside that provided by MPI point-to-point operations? > > Page 144 lines 25-27: Are intercommunicator and intracommunicator hyphenated or not? They are used both ways in these three lines. Chapter 6 uses hyphenated versions. > > Page 145 line 45: The list here appears to be every collective from the list starting at line 9, except for the scan collectives. It might be easier to just say that. > > [post 3.0] Why does bcast not have separate send and receive buffers (with an MPI_IN_PLACE option) like the other one-to-all collectives? > > Page 148 line 33: What does "itself included" mean here? The root does not do any writing to its buffer, so that text seems to be meaningless, and conflicts with the "other" on line 35. > > Page 148 line 33: "is" should be "must be". > > Page 149 line 6: The sentence should be singular since there is only one example. > > Page 160 lines 19-21: It seems odd to have an unnecessary restriction just for symmetry; why not have the call be as general as possible? > > Given that multiple writes are now allowed using MPI_Put with undefined results, might it make sense to apply the same rule to multiple MPI_Recvs and collective recvbufs that write the same location multiple times? > > Page 162 lines 10-11: This restriction is unnecessary as well. > > Page 162 line 42: It might be interesting to have the same example written using MPI_Scatter (no v) and a datatype with an adjusted extent, or at least not claim that MPI_Scatterv is required. > > Page 165 lines 40-43: This description is incorrect when MPI_IN_PLACE is used, I think. This comment also applies to page 167 lines 15-20, page 168 lines 42-48, page 171 lines 5-10, and page 173 lines 13-18. > > Page 165 line 45: There is an extra space after "0" in this line. > > Can MPI_BOTTOM be used as the buffer location for collectives when count=0 for the corresponding send or receive (depending on the collective)? Can NULL be used, or some random invalid pointer? That might be worth stating explicitly. > > In v and w versions of collectives, if the count is 0 for some entry of the array of counts, is the displacement for that entry guaranteed to be ignored (and thus doesn't need to point to valid memory)? > > For rooted collectives, is root allowed to be MPI_PROC_NULL? It is probably not, but this should probably be stated explicitly. > > Page 170 lines 43-45: Not only can the displacements be different, but the sizes can be as well. > > Page 171 lines 14-15, 20-21: What is the reason for MPI_IN_PLACE needing to be given on all processes? > > Page 171 lines 32-41: This text seems like it should be after the description of MPI_Alltoallw, since most of it applies to all three functions. > > Page 171 lines 34-35: The restriction to sequential storage does not apply to MPI_Alltoallv (unless that's meant to say that they need to be offsets in the same buffer, unless MPI_Alltoallw). > > Page 172: As I commented yesterday, MPI_Alltoallw should have a version taking arrays of MPI_Aint for sdispls and rdispls to allow the use of MPI_BOTTOM for sendbuf and recvbuf. Note that MPI_Neighbor_alltoallw does use MPI_Aint (probably since it does not need to be backwards-compatible); if a different name is chosen for MPI_Alltoallw with MPI_Aint, MPI_Neighbor_alltoallw might need to be renamed to match it. > > Page 173 line 35: MPI_SCATTERW should be in sans-serif font rather than code, I think; BTW, why aren't there w versions of all of the v collectives? > > Page 173 line 40: The first "and" on this line should be "or". > > Page 174 lines 36-37: "count" should have a comma after it on both lines. > > Page 174 line 39: "root" should have a comma after it. > > Page 174 line 45: Remove the comma here. > > Page 175 lines 28-29: This conflicts with page 174 line 38. > > Page 176: Are the logical reduction ops allowed to return any nonzero value for "true"? The definition of Boolean value earlier in the standard says that the test for bool arguments is zero/nonzero; are reduction ops given that flexibility in their outputs as well? > > Page 177 line 29: Why are logical operations not allowed on Fortran integer types? > > Page 178: Why are the functions named PAR_BLAS* and not something like PAR_DOT*? > > Page 178 line 20: Does "array" mean "matrix" here? > > Page 179 lines 47-48: It might be worth mentioning that MPI_MAXLOC does not compute a lexicographic maximum. > > Page 180: Would it be possible to generalize the *loc routines to allow wider integer types as the second element? > > Page 180 line 39: This computation ignores alignment issues (in particular, if int is larger than float). > > Page 182: Example 5.19 breaks if the total data size is greater than INT_MAX, which is increasingly common on large systems, so it can't be used as the basis of scalable code. > > Page 183 line 40: There should be a comma after "len". > > [post 3.0] Reduction operations should allow user-defined ops that change the data size (this is important for bindings to higher-level languages such as C++). > > Page 184 line 15: There should be more space around the circle operator (like on line 20). There are extra spaces after "0" and the ellipsis. > > Page 184 lines 36-45: What does this text mean? There are functions in MPI to decode datatypes. > > Page 184 line 48: "convention" should be "conventions". > > Page 185 line 4: "naive" should be "na\"\i{}ve". > > Page 185 line 41: Are user-defined ops required to be freed at some point? What is the reason for having MPI_Op_free? > > Page 186 line 9: "it is time" sounds very informal. > > Page 187 line 31: How "identical" do the results need to be? Reasonable algorithms may output non-bit-identical results on different ranks for not-truly-associative floating-point operations. > > Page 187 lines 28-: What are the requirements about argument consistency between processes for MPI_Allreduce? > > Page 188 line 28: "array" should probably be "matrix". > > Page 189 line 6: This sentence should be singular. > > Page 189 lines 11 and 44, page 190 line 21, page 192 line 1: There is an extra space after the left parenthesis on each of these lines. > > Page 191 line 7: There should be a comma after "op". > > Page 191: I think "n", and maybe "i", here should be in sans-serif font. > > Page 192 lines 32-39: "n" and "i" are in different fonts in different places in this paragraph. > > Page 192 line 34: There should be a comma after "count". > > Page 192 line 39: There should be a comma after "recvcounts[i]". > > Page 193 line 46: It seems like the code here should be in sans-serif font, plus with spaces after the commas near the end of the line. > > Page 193 lines 47-48: The variable names here are in the wrong font, and there should be a comma after "op" on line 48; both of these issues also apply to page 194 lines 33-34. > > Page 194 lines 38-39: Are the contents of recvbuf completely undefined, or are they just left unmodified from what they were before MPI_Exscan was called? Page 195 line 3 says that MPI_IN_PLACE means that recvbuf is left unchanged; is that true even without MPI_IN_PLACE? > > Page 194 lines 39-40: It is unnecessary to special-case rank 1 here. > > Page 196: The spacing inside the parentheses of the function calls in the code here is inconsistent. > > Page 197 lines 15-16: Does "passed to a completion call" mean "completed" in the sense of point-to-point (i.e., not just having MPI_Test called once, but calling a sequence of test and wait operations that end up moving the request to the completed, inactive state)? > > Page 197 lines 31-32: Why do nonblocking collectives fill in MPI_ERROR rather than returning an error from the completion call like nonblocking point-to-point calls do? > > Page 197 line 46: "MPI exception" should refer to the definition in section 8.3. > > Page 198 lines 1-7: Which processes need to complete (in the MPI_Wait, etc. sense) a nonblocking collective for it to finish executing (as in, allow MPI_Wait to return) on other processes? > > Page 198 lines 31-32: What does it mean that the progression rules are just "similar"? Do they differ in some way from the point-to-point rules? > > Page 198 line 41: There is an extra space after "comm". > > Section 5.12 (in general): The argument name "request" in the annotated argument list for each function does not line up with the other argument names above it. > > Page 216 lines 1-10: Would this example be safe if an ibarrier but blocking point-to-point calls were used? > > Page 217 lines 24-27: This rule should be a lot earlier; the previous pages seem to imply that you can pipeline nonblocking collectives on the same communicator as long as you start them in the same order on all ranks. Example 5.29 violates this rule as well, and example 5.34 explicitly states the opposite of page 217. > > Page 220 line 20: "completed" should be "complete". > > Page 220 line 46: The example header should be on the same page as the text. > > [post 3.0] Dynamic sparse data exchange (http://www.unixer.de/publications/img/hoefler-dsde-protocols.pdf) in some form might be worthwhile to add as a collective in the future, at least for the case where all data is sent at once using an alltoallv-like call but the receiver does not know the data quantity from each sender up front. Something like http://www.tu-chemnitz.de/informatik/PI/forschung/pub/download/HR_europvmmp… would also be very useful. > > Chapter 6: > > Page 224 line 20: "on a par" should probably be "on par". > > Page 229 line 1: There is an extra space before the left parenthesis. > > Page 231 line 45: "first" should be "group1". > > Page 231 line 48: Add "the" before "first group". > > Page 233 line 13: There should be a comma after the ellipsis. > > Page 233 line 16: Does that mean newgroup == group (as handles), or just that the groups themselves are MPI_IDENT? > > Page 233 line 39: "consist" should be "consists". > > Page 235 line 44: There is extra space before "comm". > > Page 236 lines 11 and 40: The comma is unnecessary here. > > Page 236 lines 14-15: This conflicts with the wording on page 227 lines 44-45. > > Page 237 line 7: There should be a space before the second "MPI_Comm". > > Page 237 line 42: The terms "group A" and "group B" are used for "left" and "right" (in some order) in chapter 5; consistent terms should be used. > > Page 238 line 15: "duplicates" should be lowercase. > > Page 238 line 23: Remove "call". > > Page 238 line 26: Does that mean that pending collective communications are not allowed? > > Pages 238-239: MPI_COMM_IDUP and MPI_COMM_DUP_WITH_INFO are in the opposite order when being introduced at the beginning of this section. > > Page 240 line 4: "group" should be lowercase. > > Page 240 line 20: Add "MPI_COMM_CREATE" after "call". > > Page 240 line 24: Add a comma after "otherwise", or merge this sentence with the previous one. > > Page 240 line 44: "sometimes to avoid" sounds awkward. > > Page 241 line 3: Add a comma after "MPI_COMM_CREATE". > > Page 241 lines 4-5: "further to subdivide" should be "to further subdivide". > > Page 241 line 17: "should" should be "must". > > Page 241 lines 21-28: What ends up as the remote group of the new inter-communicator? > > Page 242 line 22: "create" should be "creation". > > Page 243 line 44: "comm" is in an incorrect font. > > Page 244 line 30: Add "or MPI_UNDEFINED" to the end of this sentence. > > Page 245 line 10: "doesn't" should be "does not". > > Page 245 lines 30-48 and page 247 lines 2-16: The spaces in the function calls here are inconsistent with most other examples in the specification. > > Page 250 lines 17-: The style used for examples here is different from in the rest of the specification. What are these examples (especially the first few that do not do anything "interesting" with groups or communicators) trying to show? > > Page 256 line 42-page 257 line 6: "backmasking" is spelled three different ways in these two paragraphs. > > Page 257 lines 42-43: Doesn't MPI_COMM_IDUP work on intercommunicators? Why are there not non-blocking versions of the other {intra,inter}-communicator constructors? Lack of interest/need? > > Page 259 line 11: What is an "additional safe context"? > > Page 260 line 1: "and" should be "or". > > Page 260 line 4: The colon probably should be a period since the list does not immediately follow. > > Page 261 lines 4-6: "Dual membership" seems to be describing a non-standard behavior that would likely cause problems for many inter-communicator collectives (especially with MPI_IN_PLACE). > > Page 262 line 44: There should be a comma after "MPI_COMM_FREE". > > Page 265 line 45: Add a comma after "window". > > Page 266 line 12: "attributing" should be "attribute". > > Page 270 line 22: The last comma in this line should be a semicolon. > > Page 272 line 44: What does "new" refer to here? > > Page 279 lines 29-30: "The last three" should be "the last four", but it seems odd that a communicator with an invalid keyval associated with it would ever get to the point that it could be copied. > > Page 279 lines 34-35: What does this sentence mean? > > Page 283 line 9: Should "got" be "gotten" or "retrieved"? > > Page 285 line 37: The term "loosely synchronous model" is not used anywhere else that I can tell except in the context of one-sided operations. It seems like it's some property of MPI-based libraries, but this is not explained. If this topic is going to be discussed, it should also be mentioned in the context of library routines that are non-blocking internally (and maybe suggest the use of generalized requests for that), as well as why MPI_COMM_IDUP is necessary for some libraries. > > Page 287 line 1: "well-ordering" should probably be "non-overtaking" to match standard MPI terminology. > > Page 287 line 15: "then" should be "that". > > Chapter 7: > > Page 290 line 2: There should not be a comma before "process-naming". > > Page 291 line 11: There should be a comma after "therefore". > > Page 292: Which arguments must be consistent across all processes? > > Page 292: Is the created topology when reorder=false row-major or column-major? (The distinction does not matter when reorder=true). > > Page 293: Is MPI_DIMS_CREATE a local call? > > Page 294 line 37: There should be a comma after "index". > > Page 297 lines 46-47 and page 300 lines 2-4: Why not allow MPI_UNWEIGHTED as a special case for some or all isolated nodes (even if it is not passed on non-isolated nodes)? That would avoid the issues discussed on page 298 lines 6-16 and page 300 lines 9-19, and make this call more consistent with others that allow NULL for empty arrays. In particular, having MPI_UNWEIGHTED be possibly NULL and yet forbidden for degree-zero nodes prevents using something like an std::vector that could be empty as a source for weights (since its data pointer might be NULL when it is empty). > > Page 302 line 48, page 303 line 26: The variable names should be sans-serif. > > Page 307 line 2: The variable names should be sans-serif. > > Page 307 line 3: "array" should be "arrays". > > Page 312: Which arguments must be consistent across processes? Is this function collective? > > Page 312 line 46: "capability" should be "capabilities". > > Pages 313-314: Are the *_MAP functions local or collective? If they are collective, which arguments must be consistent? > > Pages 314-330: Why are these functions not in chapter 5 like the other collectives? If the issue is that they apply to topologies only, you can generalize them by assuming a non-topology-based communicator is a one-dimensional ring or similar. > > Page 320 lines 42: This line ends abruptly. > > Page 321 lines 3-21: "indegree" and "outdegree" should be in sans-serif font when referring to arguments. > > Page 324 line 28: This line ends abruptly. > > Page 331 lines 29-30: The results of these calls are only used in commented-out code on page 332. > > Page 333: This code seems like it would be clearer if it used MPI_Neighbor_alltoallv and removed the variable sizeofreal. Also, is "all-to-all-w" the official name of the operation to use in text? > > Chapter 8: > > Page 335 lines 28-29: Are these constants required to be macros (rather than const global variables), and do they need to be defined exactly as shown? Otherwise, someone could write "#define MPI_VERSION 1+2" or (in C++) "static const int MPI_VERSION = 3;". > > Page 336 line 46: "value" should be "values". > > Page 337 line 8: What is "myrank"? > > Page 337 line 12: "implementation specific" should be "implementation-specific". > > Page 337 line 23: There should be a comma before "inclusive". > > Page 337 line 41: There should not be a comma before "I/O". > > Page 341 line 26: "pointer" should be "pointers". > > Page 341 line 48: This text should end with a period, and be on the same page as the code. > > Page 342: What is the return value of an error handler used for? Is it meaningful at all? Is it guaranteed that, if an MPI function returns at all after an error, its return value is the same (implementation-defined) error code that was passed to the handler? > > Page 342 lines 35-36: "Thus, if the user chooses not to control error handling, every error that MPI handles is treated as fatal." is not true for I/O. > > Page 342 lines 40-41: "non trivial" should be "non-trivial". > > Page 343 line 29: "implementation" should be "implementations". > > Page 344 line 16: "stdargs" is not a standard term. > > Page 349 lines 44-45: Line 45 appears to be redundant with the last part of line 44. > > Page 357 line 6: "high-resolution" should be "high resolution". Also, this text might want to repeat the wording from Annex A that MPI_{Wtime,Wtick} can be macros and/or not go through the profiling interface. > > Page 357 line 18: The discussion of MPI_WTIME_IS_GLOBAL should refer to the section where it is defined. > > Page 357 lines 34-: MPI_INIT_THREAD is not mentioned here; this seems like the logical place to describe it. > > Page 359 line 38: There is extra space in the middle of the line. > > Page 359 line 45: There should be a comma after "MPI_REQUEST_FREE". > > Page 359 line 48: "operations" should be "operation". > > Page 360 lines 1-2: Are there any requirements that particular objects be freed before MPI_FINALIZE? > > Page 361 line 31: Who/what does "they" refer to? > > Page 362: Both functions on this page have extra whitespace in their abstract signatures. > > Page 362 line 41: There should be a comma after "accepted". > > Page 363 lines 1 and 7: "errorcode" should be in sans-serif. > _______________________________________________ > mpi-comments mailing list > mpi-comments(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-comments

1 0

Comments on MPI 3.0 draft 2 [part 2]
by Jeremiah Willcock 19 Aug '12

19 Aug '12

These are in reference to the draft at <URL:http://meetings.mpi-forum.org/mpi3.0_draft_2.pdf>. Thank you for considering them. -- Jeremiah Willcock General: [post 3.0] Overall, the document reads like a tutorial rather than a reference document or standard (for example, going from simpler topics to more advanced ones, forward references for datatypes and communicators in the point-to-point chapter, ignoring MPI_PROC_NULL when talking about point-to-point operations then introducing it later in the chapter, and the wording on page 225 lines 13-16). It would be nice to know, for each function, constant, etc., which MPI version introduced it. This would allow developers to write MPI-3.0-compliant programs that would also work on older MPI versions. Chapter 5: Why are nonblocking collectives in their own section rather than mixed in with the normal collectives (i.e., put each collective in both blocking and nonblocking versions together)? Page 141: MPI_REDUCE_LOCAL is not mentioned here at all (it isn't a communication operation, but it is part of the contents of this chapter). Page 141 lines 33 and 36: I don't think "gather" and "scatter" need to be capitalized. Page 141 lines 40-43: The reduction paragraph here looks very different from the others (has the ALL* version first, groups them together with single-root reduce as a variant version, and points to a subsection for allreduce while pointing to its whole containing section for reduce). Given the pattern of the other sections, allreduce and reduce should be in separate sections, with reduce first followed by allreduce. Reduce_scatter_block and reduce_scatter should be in the same subsection for the nonblocking versions, just like allgather and allgatherv (for example) are. Page 142 lines 1-3: It's odd to point to section 5.11 then subsection 5.11.2; these should probably be combined. Page 144 line 9: What are the "auxiliary functions" referring to? Do they require any access to the communication library outside that provided by MPI point-to-point operations? Page 144 lines 25-27: Are intercommunicator and intracommunicator hyphenated or not? They are used both ways in these three lines. Chapter 6 uses hyphenated versions. Page 145 line 45: The list here appears to be every collective from the list starting at line 9, except for the scan collectives. It might be easier to just say that. [post 3.0] Why does bcast not have separate send and receive buffers (with an MPI_IN_PLACE option) like the other one-to-all collectives? Page 148 line 33: What does "itself included" mean here? The root does not do any writing to its buffer, so that text seems to be meaningless, and conflicts with the "other" on line 35. Page 148 line 33: "is" should be "must be". Page 149 line 6: The sentence should be singular since there is only one example. Page 160 lines 19-21: It seems odd to have an unnecessary restriction just for symmetry; why not have the call be as general as possible? Given that multiple writes are now allowed using MPI_Put with undefined results, might it make sense to apply the same rule to multiple MPI_Recvs and collective recvbufs that write the same location multiple times? Page 162 lines 10-11: This restriction is unnecessary as well. Page 162 line 42: It might be interesting to have the same example written using MPI_Scatter (no v) and a datatype with an adjusted extent, or at least not claim that MPI_Scatterv is required. Page 165 lines 40-43: This description is incorrect when MPI_IN_PLACE is used, I think. This comment also applies to page 167 lines 15-20, page 168 lines 42-48, page 171 lines 5-10, and page 173 lines 13-18. Page 165 line 45: There is an extra space after "0" in this line. Can MPI_BOTTOM be used as the buffer location for collectives when count=0 for the corresponding send or receive (depending on the collective)? Can NULL be used, or some random invalid pointer? That might be worth stating explicitly. In v and w versions of collectives, if the count is 0 for some entry of the array of counts, is the displacement for that entry guaranteed to be ignored (and thus doesn't need to point to valid memory)? For rooted collectives, is root allowed to be MPI_PROC_NULL? It is probably not, but this should probably be stated explicitly. Page 170 lines 43-45: Not only can the displacements be different, but the sizes can be as well. Page 171 lines 14-15, 20-21: What is the reason for MPI_IN_PLACE needing to be given on all processes? Page 171 lines 32-41: This text seems like it should be after the description of MPI_Alltoallw, since most of it applies to all three functions. Page 171 lines 34-35: The restriction to sequential storage does not apply to MPI_Alltoallv (unless that's meant to say that they need to be offsets in the same buffer, unless MPI_Alltoallw). Page 172: As I commented yesterday, MPI_Alltoallw should have a version taking arrays of MPI_Aint for sdispls and rdispls to allow the use of MPI_BOTTOM for sendbuf and recvbuf. Note that MPI_Neighbor_alltoallw does use MPI_Aint (probably since it does not need to be backwards-compatible); if a different name is chosen for MPI_Alltoallw with MPI_Aint, MPI_Neighbor_alltoallw might need to be renamed to match it. Page 173 line 35: MPI_SCATTERW should be in sans-serif font rather than code, I think; BTW, why aren't there w versions of all of the v collectives? Page 173 line 40: The first "and" on this line should be "or". Page 174 lines 36-37: "count" should have a comma after it on both lines. Page 174 line 39: "root" should have a comma after it. Page 174 line 45: Remove the comma here. Page 175 lines 28-29: This conflicts with page 174 line 38. Page 176: Are the logical reduction ops allowed to return any nonzero value for "true"? The definition of Boolean value earlier in the standard says that the test for bool arguments is zero/nonzero; are reduction ops given that flexibility in their outputs as well? Page 177 line 29: Why are logical operations not allowed on Fortran integer types? Page 178: Why are the functions named PAR_BLAS* and not something like PAR_DOT*? Page 178 line 20: Does "array" mean "matrix" here? Page 179 lines 47-48: It might be worth mentioning that MPI_MAXLOC does not compute a lexicographic maximum. Page 180: Would it be possible to generalize the *loc routines to allow wider integer types as the second element? Page 180 line 39: This computation ignores alignment issues (in particular, if int is larger than float). Page 182: Example 5.19 breaks if the total data size is greater than INT_MAX, which is increasingly common on large systems, so it can't be used as the basis of scalable code. Page 183 line 40: There should be a comma after "len". [post 3.0] Reduction operations should allow user-defined ops that change the data size (this is important for bindings to higher-level languages such as C++). Page 184 line 15: There should be more space around the circle operator (like on line 20). There are extra spaces after "0" and the ellipsis. Page 184 lines 36-45: What does this text mean? There are functions in MPI to decode datatypes. Page 184 line 48: "convention" should be "conventions". Page 185 line 4: "naive" should be "na\"\i{}ve". Page 185 line 41: Are user-defined ops required to be freed at some point? What is the reason for having MPI_Op_free? Page 186 line 9: "it is time" sounds very informal. Page 187 line 31: How "identical" do the results need to be? Reasonable algorithms may output non-bit-identical results on different ranks for not-truly-associative floating-point operations. Page 187 lines 28-: What are the requirements about argument consistency between processes for MPI_Allreduce? Page 188 line 28: "array" should probably be "matrix". Page 189 line 6: This sentence should be singular. Page 189 lines 11 and 44, page 190 line 21, page 192 line 1: There is an extra space after the left parenthesis on each of these lines. Page 191 line 7: There should be a comma after "op". Page 191: I think "n", and maybe "i", here should be in sans-serif font. Page 192 lines 32-39: "n" and "i" are in different fonts in different places in this paragraph. Page 192 line 34: There should be a comma after "count". Page 192 line 39: There should be a comma after "recvcounts[i]". Page 193 line 46: It seems like the code here should be in sans-serif font, plus with spaces after the commas near the end of the line. Page 193 lines 47-48: The variable names here are in the wrong font, and there should be a comma after "op" on line 48; both of these issues also apply to page 194 lines 33-34. Page 194 lines 38-39: Are the contents of recvbuf completely undefined, or are they just left unmodified from what they were before MPI_Exscan was called? Page 195 line 3 says that MPI_IN_PLACE means that recvbuf is left unchanged; is that true even without MPI_IN_PLACE? Page 194 lines 39-40: It is unnecessary to special-case rank 1 here. Page 196: The spacing inside the parentheses of the function calls in the code here is inconsistent. Page 197 lines 15-16: Does "passed to a completion call" mean "completed" in the sense of point-to-point (i.e., not just having MPI_Test called once, but calling a sequence of test and wait operations that end up moving the request to the completed, inactive state)? Page 197 lines 31-32: Why do nonblocking collectives fill in MPI_ERROR rather than returning an error from the completion call like nonblocking point-to-point calls do? Page 197 line 46: "MPI exception" should refer to the definition in section 8.3. Page 198 lines 1-7: Which processes need to complete (in the MPI_Wait, etc. sense) a nonblocking collective for it to finish executing (as in, allow MPI_Wait to return) on other processes? Page 198 lines 31-32: What does it mean that the progression rules are just "similar"? Do they differ in some way from the point-to-point rules? Page 198 line 41: There is an extra space after "comm". Section 5.12 (in general): The argument name "request" in the annotated argument list for each function does not line up with the other argument names above it. Page 216 lines 1-10: Would this example be safe if an ibarrier but blocking point-to-point calls were used? Page 217 lines 24-27: This rule should be a lot earlier; the previous pages seem to imply that you can pipeline nonblocking collectives on the same communicator as long as you start them in the same order on all ranks. Example 5.29 violates this rule as well, and example 5.34 explicitly states the opposite of page 217. Page 220 line 20: "completed" should be "complete". Page 220 line 46: The example header should be on the same page as the text. [post 3.0] Dynamic sparse data exchange (http://www.unixer.de/publications/img/hoefler-dsde-protocols.pdf) in some form might be worthwhile to add as a collective in the future, at least for the case where all data is sent at once using an alltoallv-like call but the receiver does not know the data quantity from each sender up front. Something like http://www.tu-chemnitz.de/informatik/PI/forschung/pub/download/HR_europvmmp… would also be very useful. Chapter 6: Page 224 line 20: "on a par" should probably be "on par". Page 229 line 1: There is an extra space before the left parenthesis. Page 231 line 45: "first" should be "group1". Page 231 line 48: Add "the" before "first group". Page 233 line 13: There should be a comma after the ellipsis. Page 233 line 16: Does that mean newgroup == group (as handles), or just that the groups themselves are MPI_IDENT? Page 233 line 39: "consist" should be "consists". Page 235 line 44: There is extra space before "comm". Page 236 lines 11 and 40: The comma is unnecessary here. Page 236 lines 14-15: This conflicts with the wording on page 227 lines 44-45. Page 237 line 7: There should be a space before the second "MPI_Comm". Page 237 line 42: The terms "group A" and "group B" are used for "left" and "right" (in some order) in chapter 5; consistent terms should be used. Page 238 line 15: "duplicates" should be lowercase. Page 238 line 23: Remove "call". Page 238 line 26: Does that mean that pending collective communications are not allowed? Pages 238-239: MPI_COMM_IDUP and MPI_COMM_DUP_WITH_INFO are in the opposite order when being introduced at the beginning of this section. Page 240 line 4: "group" should be lowercase. Page 240 line 20: Add "MPI_COMM_CREATE" after "call". Page 240 line 24: Add a comma after "otherwise", or merge this sentence with the previous one. Page 240 line 44: "sometimes to avoid" sounds awkward. Page 241 line 3: Add a comma after "MPI_COMM_CREATE". Page 241 lines 4-5: "further to subdivide" should be "to further subdivide". Page 241 line 17: "should" should be "must". Page 241 lines 21-28: What ends up as the remote group of the new inter-communicator? Page 242 line 22: "create" should be "creation". Page 243 line 44: "comm" is in an incorrect font. Page 244 line 30: Add "or MPI_UNDEFINED" to the end of this sentence. Page 245 line 10: "doesn't" should be "does not". Page 245 lines 30-48 and page 247 lines 2-16: The spaces in the function calls here are inconsistent with most other examples in the specification. Page 250 lines 17-: The style used for examples here is different from in the rest of the specification. What are these examples (especially the first few that do not do anything "interesting" with groups or communicators) trying to show? Page 256 line 42-page 257 line 6: "backmasking" is spelled three different ways in these two paragraphs. Page 257 lines 42-43: Doesn't MPI_COMM_IDUP work on intercommunicators? Why are there not non-blocking versions of the other {intra,inter}-communicator constructors? Lack of interest/need? Page 259 line 11: What is an "additional safe context"? Page 260 line 1: "and" should be "or". Page 260 line 4: The colon probably should be a period since the list does not immediately follow. Page 261 lines 4-6: "Dual membership" seems to be describing a non-standard behavior that would likely cause problems for many inter-communicator collectives (especially with MPI_IN_PLACE). Page 262 line 44: There should be a comma after "MPI_COMM_FREE". Page 265 line 45: Add a comma after "window". Page 266 line 12: "attributing" should be "attribute". Page 270 line 22: The last comma in this line should be a semicolon. Page 272 line 44: What does "new" refer to here? Page 279 lines 29-30: "The last three" should be "the last four", but it seems odd that a communicator with an invalid keyval associated with it would ever get to the point that it could be copied. Page 279 lines 34-35: What does this sentence mean? Page 283 line 9: Should "got" be "gotten" or "retrieved"? Page 285 line 37: The term "loosely synchronous model" is not used anywhere else that I can tell except in the context of one-sided operations. It seems like it's some property of MPI-based libraries, but this is not explained. If this topic is going to be discussed, it should also be mentioned in the context of library routines that are non-blocking internally (and maybe suggest the use of generalized requests for that), as well as why MPI_COMM_IDUP is necessary for some libraries. Page 287 line 1: "well-ordering" should probably be "non-overtaking" to match standard MPI terminology. Page 287 line 15: "then" should be "that". Chapter 7: Page 290 line 2: There should not be a comma before "process-naming". Page 291 line 11: There should be a comma after "therefore". Page 292: Which arguments must be consistent across all processes? Page 292: Is the created topology when reorder=false row-major or column-major? (The distinction does not matter when reorder=true). Page 293: Is MPI_DIMS_CREATE a local call? Page 294 line 37: There should be a comma after "index". Page 297 lines 46-47 and page 300 lines 2-4: Why not allow MPI_UNWEIGHTED as a special case for some or all isolated nodes (even if it is not passed on non-isolated nodes)? That would avoid the issues discussed on page 298 lines 6-16 and page 300 lines 9-19, and make this call more consistent with others that allow NULL for empty arrays. In particular, having MPI_UNWEIGHTED be possibly NULL and yet forbidden for degree-zero nodes prevents using something like an std::vector that could be empty as a source for weights (since its data pointer might be NULL when it is empty). Page 302 line 48, page 303 line 26: The variable names should be sans-serif. Page 307 line 2: The variable names should be sans-serif. Page 307 line 3: "array" should be "arrays". Page 312: Which arguments must be consistent across processes? Is this function collective? Page 312 line 46: "capability" should be "capabilities". Pages 313-314: Are the *_MAP functions local or collective? If they are collective, which arguments must be consistent? Pages 314-330: Why are these functions not in chapter 5 like the other collectives? If the issue is that they apply to topologies only, you can generalize them by assuming a non-topology-based communicator is a one-dimensional ring or similar. Page 320 lines 42: This line ends abruptly. Page 321 lines 3-21: "indegree" and "outdegree" should be in sans-serif font when referring to arguments. Page 324 line 28: This line ends abruptly. Page 331 lines 29-30: The results of these calls are only used in commented-out code on page 332. Page 333: This code seems like it would be clearer if it used MPI_Neighbor_alltoallv and removed the variable sizeofreal. Also, is "all-to-all-w" the official name of the operation to use in text? Chapter 8: Page 335 lines 28-29: Are these constants required to be macros (rather than const global variables), and do they need to be defined exactly as shown? Otherwise, someone could write "#define MPI_VERSION 1+2" or (in C++) "static const int MPI_VERSION = 3;". Page 336 line 46: "value" should be "values". Page 337 line 8: What is "myrank"? Page 337 line 12: "implementation specific" should be "implementation-specific". Page 337 line 23: There should be a comma before "inclusive". Page 337 line 41: There should not be a comma before "I/O". Page 341 line 26: "pointer" should be "pointers". Page 341 line 48: This text should end with a period, and be on the same page as the code. Page 342: What is the return value of an error handler used for? Is it meaningful at all? Is it guaranteed that, if an MPI function returns at all after an error, its return value is the same (implementation-defined) error code that was passed to the handler? Page 342 lines 35-36: "Thus, if the user chooses not to control error handling, every error that MPI handles is treated as fatal." is not true for I/O. Page 342 lines 40-41: "non trivial" should be "non-trivial". Page 343 line 29: "implementation" should be "implementations". Page 344 line 16: "stdargs" is not a standard term. Page 349 lines 44-45: Line 45 appears to be redundant with the last part of line 44. Page 357 line 6: "high-resolution" should be "high resolution". Also, this text might want to repeat the wording from Annex A that MPI_{Wtime,Wtick} can be macros and/or not go through the profiling interface. Page 357 line 18: The discussion of MPI_WTIME_IS_GLOBAL should refer to the section where it is defined. Page 357 lines 34-: MPI_INIT_THREAD is not mentioned here; this seems like the logical place to describe it. Page 359 line 38: There is extra space in the middle of the line. Page 359 line 45: There should be a comma after "MPI_REQUEST_FREE". Page 359 line 48: "operations" should be "operation". Page 360 lines 1-2: Are there any requirements that particular objects be freed before MPI_FINALIZE? Page 361 line 31: Who/what does "they" refer to? Page 362: Both functions on this page have extra whitespace in their abstract signatures. Page 362 line 41: There should be a comma after "accepted". Page 363 lines 1 and 7: "errorcode" should be in sans-serif.

1 0

Re: [Mpi-comments] Comments on MPI 3.0 draft 2
by Rolf Rabenseifner 19 Aug '12

19 Aug '12

Dear Jeremiah Willcock, thank you very much for your detailed comments, especially in the chapters for which I'm responsible or co-auther, i.e., 2. Terms, 17. Fortran and Interop (Jeff together with Craig and me), Annex A Language Binding Summary, and Annex B (Change-log), and partially the Index. The typos will be corrected. I'll come back with answers for your questions in "my" chapters. Best regards Rolf ----- Original Message ----- > From: "Jeremiah Willcock" <jewillco(a)osl.iu.edu> > To: mpi-comments(a)mpi-forum.org > Sent: Sunday, August 19, 2012 2:13:41 AM > Subject: [Mpi-comments] Comments on MPI 3.0 draft 2 > > These are in reference to the draft at > <URL:http://meetings.mpi-forum.org/mpi3.0_draft_2.pdf>. Thank you for > considering them. > > -- Jeremiah Willcock > ... -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

Comments on MPI 3.0 draft 2
by Jeremiah Willcock 18 Aug '12

18 Aug '12

These are in reference to the draft at <URL:http://meetings.mpi-forum.org/mpi3.0_draft_2.pdf>. Thank you for considering them. -- Jeremiah Willcock General: The line numbers do not match the lines of text; my comments are best estimates of which line each line of text is on. [post 3.0] It might be worthwhile to create a document of "performance expectations," basing it on papers such as <URL:http://www.mcs.anl.gov/~thakur/papers/qualityofmpi.pdf>, <URL:http://www.unixer.de/publications/img/datatype-performance.pdf>, and <URL:http://www.mcs.anl.gov/uploads/cels/papers/2008/P1500.pdf>. This may or may not be normative, but maybe you can have a defined "self-consistently performing" status for MPI implementations that meet the standard and satisfy these extra requirements. [cosmetic] In many code samples, non-matching curly quotes are used and look unusual. A solution for that is at <URL:http://stackoverflow.com/questions/1662037/how-to-write-programming-cod…>; it may be easy to apply globally without needing to change each individual quotation mark. Front matter: The change log starting on page iii does not have a title. How much of that material is redundant with sections 1.2-1.6? On page iii line 18, there should be a comma after "Operations". On page iv lines 6-8, that text seems very obsolete (not correct for the current standard). On page xix line 11, part of the table title is black, while the rest are blue. On page xxviii line 20, "award" should be "awards" and the numbers do not usually have number signs before them. The same issue appears on page xxx lines 36-37. On page xxviii line 29, "frontmatter" should be "front matter". Chapter 1: On page 1 line 24, the comma should be after "which" instead of before. Also, "methods" being in that list may no longer be relevant. On page 1 line 44, "coprocessor" should be "coprocessors". On page 3 line 16, there should be a comma after "e.g.". On page 3 line 46, the comma should be before the closing quotation mark. On page 5 line 35, "Info" should be lowercase. Chapter 2: On page 9 line 16, why does the MPI 3.0 document still compare itself to MPI 1? On page 9 line 18, "vs" needs a period. On page 10 line 34, there should be a comma after "OUT". On page 11 line 30, there should be a comma after "i.e.". On page 12 line 9, there should not be a comma after "portable". On page 13 line 43, there should not be a comma after "C". On page 15, should the MPI version constants be in the first list (of compile-time constants)? On page 18, the font in the table is smaller than the rest of the text or other tables in the standard. On page 19 line 14, the restriction on use of MPI_* names should also apply to preprocessor macros, namespaces in C++, types, and other things. On page 19 line 44, "executing their own code" might be better as "each executing its own code". Is this text trying to say that MPI is not a SIMD model? That is not clear from those two sentences (that the executables may or may not be identical, but the processes always have different control flow). Chapter 3: The top of page 24 refers to "parameters" to mean what chapter 1 said would be referred to as "arguments" to avoid confusion with Fortran. On page 25 lines 36-38, there need to be commas after "MPI_REAL4", "MPI_INTEGER1", and "MPI_INTEGER2". On page 27 line 43, there should not be a space after the ellipsis. On page 29 lines 35-36, the list of valid values omits "MPI_PROC_NULL" (so does the corresponding list for sends). On page 31 line 32, should "receive" be "send"? On page 31 lines 38-39, it might be good to have a similar thread safety warning about probe and iprobe that is on line 33 about other message passing libraries. On page 32 line 9, there should be a comma after "comm". On page 33 line 22, the last comma should be a colon. On page 33 line 48, the example header should be on the same page as the code. On page 34 line 32, "bound" should be "bounds". On page 36 line 3, "logical or" should be "logical, and". Is the buffer required to be a valid pointer (not NULL, for example) when using count=0 for sends and/or receives? On page 40 line 39, did you want the "should" to be "must", at least under MPI_THREAD_MULTIPLE? Or is that too strong of a constraint on user-mode threading implementations? On page 44 line 43, "C++" should be removed. On page 44 line 45, the comma should be before the closing single quote. On page 46 line 1, doesn't MPI_BSEND_OVERHEAD come into play here as well? This paragraph suggests that MPI must be able to use the full user buffer, while that is contradicted by page 46 line 40. Page 47 lines 46-48 seem unclear to me -- is that paragraph an "advice to users" that says that an implementation might delay completing standard sends until the corresponding receives are posted but it isn't required to? The "may not" wording can also be read to mean that the send is not allowed to complete until the receive is finished, which is not what was intended. On page 56 line 30, the text seems to require background progress (sends progressing even if MPI is not called). I know that has been a disputed issue in the past, but is that the desired behavior? On page 57 line 11 and further on, when count = 0 for a multiple completion operation, are the arrays passed in guaranteed to not be touched by the MPI implementation (and thus they can be NULL)? I think this was implied before, but isn't explicit. On page 57 line 46, there is an extra space after "request". On page 60 line 31, it might be good to put "in the same manner" after "handled". On page 62 lines 30-34 (and example 3.16 on page 63), doesn't that text contradict the discussion on the mailing list about fairness issues (and the lack of fairness guarantees in MPI)? Also, that text looks like it should be under MPI_WAITSOME, not MPI_TESTSOME. Are the handles passed in to the multiple-completion routines required to be distinct? I.e., can I have the same active handle at two different positions in the input array, and if so, what happens? Does MPI_*PROBE have the same non-overtaking rules as receives (in terms of matching the first message sent of those that could match the probe operation)? Page 67 lines 27-38 says that (although possibly not normatively since it's in "advice to implementors"), but the non-overtaking section doesn't. Does that rule apply to MPI_IMPROBE as well (if I do two MPI_IMPROBES that each match two messages from the same sender, the order the probes get the messages must be the same as the send order even if I don't do MPI_MRECV before doing the next improbe)? How do those rules interact with MPI_CANCEL on MPI_IMRECV-created requests? Is there a reason that there are not non-blocking versions of MPI_SENDRECV and MPI_SENDRECV_REPLACE? The second one, in particular, is not trivial to emulate using only a single isend/irecv pair. Chapter 4: On page 83 line 23, "communication" should be "communications". On page 83 line 34, "shape and size" should be "shapes and sizes". On page 84 line 42, "sizeof" should be inside \mathit. On page 84 line 45, where does "i" come from? Is the extent rounded to a multiple of k_i, or is the upper bound rounded (these differ when lb is not aligned)? If extent is rounded, the +eps term should be moved to the computation of extent, not ub, unless that definition of ub is used somewhere else. On page 95, why does MPI_TYPE_CREATE_SUBARRAY use zero-based indexing for Fortran; I believe many other places in MPI use one-based array indices when called from Fortran, even if they use zero-based indices from C. Also, is there a version of MPI_TYPE_CREATE_SUBARRAY that allows strided indexing in the dimensions (MPI_TYPE_CREATE_DARRAY looks more limited)? On page 103 line 18, MPI_ADDRESS has now been removed, not just deprecated. Is offsetof a valid way to get displacements within C structs to use with MPI (rather than creating an object of that type and calling MPI_GET_ADDRESS on it and its individual fields)? Wouldn't MPI_TYPE_SIZE and a way to "flatten" an MPI datatype into a datatype with the same primitive elements but sequential in memory (or using MPI_PACKED) be better than MPI_TYPE_TRUE_EXTENT for the example on page 108 line 29? On page 110 line 24, there should be a comma after "e.g.". On page 111 line 42, there is some odd space at the end of the line. On page 112 line 7, there shouldn't be a space after "datatype". On page 112 lines 11-14, should there be an MPI_TYPE_FREE added as well? On page 115 line 44, it seems odd that adding MPI_BOTTOM to a valid address gives a valid address, since that allows me to create a valid address MPI_BOTTOM * n + v where n is nonnegative and v is a valid address. On page 128 line 32, example 4.18 uses "class" as a field name, which makes it impossible to compile with a C++ compiler. Chapter 5: If there are going to be MPI_Count versions of datatype functions, would it at least be possible to have a version of MPI_Alltoallw that uses MPI_Aint for displacements? It seems like that was an oversight before, since byte offsets from MPI_BOTTOM are intended as arbitrary addresses. Chapter 12: Did the extended generalized request proposal from "Extending the MPI-2 Generalized Request Interface" by Latham et al not get accepted into MPI 3.0? Chapter 16: On page 597 lines 20-21, "language neutral" should be hyphenated. On page 597, line 43, "ion" should be "on". On page 598 line 22, "prototyes" should be "prototypes" in the subsection title. On page 598, tables 16.2 and 16.3 still list C++ bindings. Chapter 17: It may be too late to change this, but "typeclass" is an unfortunate name to use in MPI since it refers to a specific language feature in Haskell and some other functional languages. On page 600 line 18, "INCLUDE mpif.h" is missing the single quotes that are used in other places. On page 600 line 27, do you want to still mention "C/C++ header files" now that the C++ bindings have been removed? On page 600 line 47, do you want to require Fortran 2008 support for all MPI implementations that have any Fortran support at all? What does that mean on platforms that have older Fortran compilers but not Fortran 2008 support at the compiler level? On page 601 line 32, should "feature" be "features"? On page 603 line 2, the period should be before the closing double quote (in American English). On page 604 lines 14-15, should the single quotes be double quotes? On page 604 line 18, "constaints" should be "constraints". On page 604 line 36, "With MPI_SUBARRAYS_SUPPORTED==.TRUE." might be better as "When MPI_SUBARRAYS_SUPPORTED is defined as .TRUE.". On page 605 lines 21-23, the italicization of the quotation seems incorrect (although I don't know what the source of the quote had italicized). On page 606 lines 16 and 18, "wrong" might read better as "incorrect". On page 606 line 41, the equation should be in math mode. On page 607 line 29 and on page 608 line 6, "BIND(C)" should be in the code font. On page 608 line 19, "and" should be removed. On page 608 lines 24 and 25, "MPI" and "PMPI" use different fonts; this may be a typo. On page 608 line 35, "achieved" should be "available". On page 608 line 38, "implement" should be "implementation". On page 609 lines 1-7, the advice to implementors is hard to read (the first sentence in particular is very long). On page 609 line 10, "notifying" should be "indicating". On page 609 line 33, "MPI_Address" is one of the functions marked as removed in MPI 3.0, so it should not be listed there. On page 610 line 36, should the linker names be in the code font? On page 611 line 3, "non optional" should be "non-optional". On page 611 lines 9-10, "compiler" should be "compilers", and the "the" before "C" can be removed. On page 611 lines 16-17, the formatting is strange (extra spaces between names). On page 611 line 44, "b" should be "by". On page 611 line 48, "MPI_Address" has been removed in MPI 3.0. On page 612 line 14, the hyphen should be an en dash (denoting a range). On page 612 lines 37 and 42, "S1", "S2", and "S3" are not in sans serif font like they are elsewhere on the page. On page 613 line 30, the hyphen between the MPI version numbers should be an en dash. Also, why is the MPI 3.0 standard discussing implementing older versions? On page 614 lines 14-15, isn't the second line redundant with the first one? On page 615 line 29, the single quotes should be double quotes. On page 616 line 14, "IO" should be "I/O". On page 616 line 45, the subsection title should not have a hyphen before "synchronization". On page 617 lines 13-14, why not just say that the routine must be written in an implementation-specific way that achieves the listed behavior, rather than specifying a particular requirement about the compiler's knowledge of the routine's body? On page 617 lines 31-32, the single quotes should be double quotes. On page 618 line 2, there should be a comma after "LOGICAL". On page 618 line 8, there should be a comma after "COMPLEX". On page 618 lines 21-22, should the section title referred to be in quotes? On page 619 line 33 and other places, should "external32" be in the code font? On page 621 line 37, there should be commas after "Double" and "8". On page 622 line 14, there should be a comma after "MPI_UNPACK_EXTERNAL". On page 624 line 3, "sizeof" is built in to the language as syntax; it is not a function. On page 626 line 13, there is extra space after "90". On page 626 line 40, the hyphen between the MPI versions should be an en dash. On page 627 line 4, should "KIND" be in the code font? On page 627 line 30, "mpif.h" may need to be in single quotes. On page 627 line 39, "module" should be "modules". On page 628 lines 34-42, does that mean that users are allowed to modify send buffers during Fortran nonblocking communications (since the implementation is required to act as if it made a copy of the user data when the operation began)? The "Displacements in" sentence on lines 38-39 is also confusing; does that mean that datatypes need to be created using different displacements in Fortran than in C (even if they are structurally identical)? This reading conflicts with page 653, but is reasonable from the text on 628. On page 629 line 2, "comminication" should be "communication". On page 629 line 21, there should not be a space after the ellipsis. On page 629 lines 44 and 45, the single quotes should be double quotes. On page 632 line 27, should the comment be indented to match the one two lines later? On page 632 line 42, "extensions" should be "extension". On page 633 line 9, there should be a comma before "e.g.". On page 633 line 12, "as it may fit" sounds odd. On page 633 lines 47 and 48, the section titles should be in the normal text font. On page 634 line 17, the subsection title has extra space after "and". On page 634 line 20, "block" should be in the text font. On page 634 line 41, "be" should be "have been". On page 635 line 7, there should be spaces after the second two "copy"s. On page 635 line 9, should "send" be "CALL send"? On page 635 lines 13 and 16, there should not be spaces around the slashes. On page 635 line 26, there should be a comma after "MPI_RECV". On page 636 line 36, because of interprocedural optimization, it may not be possible for a user routine to have the same behavior as MPI_F_SYNC_REG without resorting to compiler-specific techniques, and thus it may not be good to recommend doing. The same thing applies to page 640 lines 15-36 and page 642 line 48. On page 637 line 39, should the "ASYNCHRONOUS" in the title be in the code font? On page 639 line 47, the headings should be moved to the same page as the code. On page 640 line 33, "posibilities" should be "possibilities". On page 643 line 38, "Automatic garbage collection implementation" should be "An implementation with automatic garbage collection". On page 644 line 5, "&" should be in the code font. On page 647 line 4, remove "applications" from before "library". On page 648 line 22, is there some reason that the conversion functions still take integers on the Fortran side and not the handle types? Is that just a compatibility issue? On page 648 line 26, "int" should be in the code font. On page 648 lines 38-39, shouldn't passing in MPI_COMM_NULL in Fortran return the C value of MPI_COMM_NULL (and never something else), and the behavior be unspecified for passing in an invalid communicator? The same applies to lines 43-44. Do other predefined communicators need to map to the C versions of their predefined values (MPI_COMM_SELF -> MPI_COMM_SELF, not something else equivalent to it)? On page 655 line 3, "stdargs" is not a standard term; use "variable-length argument list" instead. On page 655 line 21, the sentence there does not end with a period (the one on "etc." does not count). On page 655, "address valued" and "integer valued" should have hyphens when used as adjectives. On page 656 line 9, "identical" should be "identically". On page 657 line 48, the header of the example should be on the same page as the code. On page 658 line 47, "val" should be either sans-serif or in the code font. Some other things in that paragraph might need to be changed as well. On page 659 line 22, "extra state" should be hyphenated in the subsection title if it is in the text. On page 661 line 16, "int" should be in the code font. Annex A: On page 674 lines 41-47, don't those constants have particular, standard-specified values (not just arbitrary definitions)? On page 676 lines 10-15, are those values required to be defined as macros that expand to integral values? Thus, they can't have "const int" or "unnamed enum" types, but must be macros. On page 678 line 48 and some places on 679, "module" should be "modules". On page 685, what are the language bindings for the info keys and values supposed to be? The C prototypes for functions (throughout the standard, but especially here) don't end with semicolons like prototypes actually do in C. On page 693 lines 45-46, shouldn't MPI_COMM_DUP_FN (and the others in A.2.4 that are just signatures but not defined by MPI) be moved to section A.1.3 with the other user-defined function prototypes? Annex B: On page 789 line 28, "errata" is plural, so it should probably be "erratum". There seem to be other places in Annex B with this error as well. On page 789 line 39, the opening quotation mark is backwards. Section B.1 mentions various C++ bindings to things as having been corrected in MPI 3.0, while actually all of those bindings were removed (see page 790 lines 5-6 for an example). On page 790 line 7, the second comma should be inside the quotation marks. On page 792 line 40, "one-sided" should be lowercase. On page 793 line 7, there should be a comma after "==". On page 794 line 17, "a with" should be "with a". On page 794 line 26, "solution" should be "solutions". Index: Page 811 lists now-removed C++ bindings. The function index starting on page 818 sometimes lists the same function twice on the same page (see MPI_ALLTOALLV for an example with page 170). On page 824, should individual PMPI functions be on the function list? Should mpiexec and mpirun?

1 0

Re: [Mpi-comments] One comment on MPI-3.0 Draft 2, August 2012
by George Bosilca 15 Aug '12

15 Aug '12

In other terms it depends on what you prefer to understand by fairness. geeks: http://en.wikipedia.org/wiki/Unbounded_nondeterminism otherwise: http://dictionary.reference.com/browse/fairness george. On Aug 15, 2012, at 21:29 , William Gropp wrote: > This is a good point. Using the Testsome or Waitsome interface permits an application to respond fairly without burdening the implementation with any particular fairness definition. Not every code can be conveniently written this way, but it addresses many needs. > > As has been noted, specifying the exact meaning of fairness (e.g., round robin) may have serious performance impact, as an implementation may make fair progress but in a different, possibly non-deterministic, order. Even if "fairness" is provided, it should be defined in terms of the essential properties, not a particular implementation that happens to have those properties. > > Bill > > William Gropp > Director, Parallel Computing Institute > Deputy Director for Research > Institute for Advanced Computing Applications and Technologies > Paul and Cynthia Saylor Professor of Computer Science > University of Illinois Urbana-Champaign > > > > On Aug 15, 2012, at 8:54 AM, Jim Dinan wrote: > >> Hi Sebastien, >> >> Something to consider in addition to probe is that fairness semantics are also relevant to test/wait operations that take arrays of requests. It may make the most sense to attach a fairness attribute to communication contexts (communicators and windows) to provide more flexibility. This could already be done by an MPI implementation through an info key on a communicator/window, however implementations need not support all info keys and fairness could be a requirement for applications that rely on it. >> >> Cheers, >> ~Jim. >> >> On 8/15/12 10:19 AM, Sï¿½bastien Boisvert wrote: >>> Maybe there could be something like MPI_Init_fairness which could either >>> be MPI_FAIRNESS_NO (the current behavior) or MPI_FAIRNESS_YES (which >>> would do round-robin reception). >> _______________________________________________ >> mpi-comments mailing list >> mpi-comments(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-comments > > > _______________________________________________ > mpi-comments mailing list > mpi-comments(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-comments

1 0

Re: [Mpi-comments] One comment on MPI-3.0 Draft 2, August 2012
by William Gropp 15 Aug '12

15 Aug '12

This is a good point. Using the Testsome or Waitsome interface permits an application to respond fairly without burdening the implementation with any particular fairness definition. Not every code can be conveniently written this way, but it addresses many needs. As has been noted, specifying the exact meaning of fairness (e.g., round robin) may have serious performance impact, as an implementation may make fair progress but in a different, possibly non-deterministic, order. Even if "fairness" is provided, it should be defined in terms of the essential properties, not a particular implementation that happens to have those properties. Bill William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign On Aug 15, 2012, at 8:54 AM, Jim Dinan wrote: > Hi Sebastien, > > Something to consider in addition to probe is that fairness semantics are also relevant to test/wait operations that take arrays of requests. It may make the most sense to attach a fairness attribute to communication contexts (communicators and windows) to provide more flexibility. This could already be done by an MPI implementation through an info key on a communicator/window, however implementations need not support all info keys and fairness could be a requirement for applications that rely on it. > > Cheers, > ~Jim. > > On 8/15/12 10:19 AM, Sï¿½bastien Boisvert wrote: >> Maybe there could be something like MPI_Init_fairness which could either >> be MPI_FAIRNESS_NO (the current behavior) or MPI_FAIRNESS_YES (which >> would do round-robin reception). > _______________________________________________ > mpi-comments mailing list > mpi-comments(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-comments

1 0

Re: [Mpi-comments] One comment on MPI-3.0 Draft 2, August 2012
by Jim Dinan 15 Aug '12

15 Aug '12

Hi Sebastien, Something to consider in addition to probe is that fairness semantics are also relevant to test/wait operations that take arrays of requests. It may make the most sense to attach a fairness attribute to communication contexts (communicators and windows) to provide more flexibility. This could already be done by an MPI implementation through an info key on a communicator/window, however implementations need not support all info keys and fairness could be a requirement for applications that rely on it. Cheers, ~Jim. On 8/15/12 10:19 AM, Sï¿½bastien Boisvert wrote: > Maybe there could be something like MPI_Init_fairness which could either > be MPI_FAIRNESS_NO (the current behavior) or MPI_FAIRNESS_YES (which > would do round-robin reception).

1 0

Re: [Mpi-comments] One comment on MPI-3.0 Draft 2, August 2012
by Richard Graham 15 Aug '12

15 Aug '12

Sï¿½bastien, To get involved, see meetings.mpi-forum.org, where there are mailing lists you can join, as well as meeting details. The most effective way to participate it to get involved in person. Rich -----Original Message----- From: mpi-comments-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Sï¿½bastien Boisvert Sent: Wednesday, August 15, 2012 11:20 AM To: mpi-comments(a)lists.mpi-forum.org Subject: Re: [Mpi-comments] One comment on MPI-3.0 Draft 2, August 2012 Hello Richard, I really agree that this is a big change. And I am aware that there is no fairness by design [1]. In Open-MPI, George Bosilca provided a patch [2], which I modified [3] for a round-robin policy and tested [4]. For threading, there is MPI_Init_thread that tells MPI if one or more thread will be spawned and if one or more will make MPI calls. Maybe there could be something like MPI_Init_fairness which could either be MPI_FAIRNESS_NO (the current behavior) or MPI_FAIRNESS_YES (which would do round-robin reception). How can I get involved ? [1] http://www.mcs.anl.gov/research/projects/mpi/tutorial/gropp/node92.html [2] http://www.open-mpi.org/community/lists/devel/att-9769/iprobe_starvation.pa… [3] http://www.open-mpi.org/community/lists/devel/att-9770/iprobe_starvation_fi… [4] http://www.open-mpi.org/community/lists/devel/2011/09/9770.php Richard Graham a ï¿½crit : > Sï¿½bastien, > Thanks you for sending in this request, this is truly appreciated. > The propose change is a large change to the standard, even though it may not seem so at a first glance. Today the standard is explicit about lack of fairness, and we would have to think through in detail on the broad impact of such a change. So, this does not the sort of item we can address for MPI 3.0. > However, if you would like to push a change like this into MPI, this can be addressed for the following version of the standard. What I would suggest, if this is something that you really think belongs in the standard, that you either get involved directly in the process (the standardization effort is open to all), or find someone the is involved to actively push such a request. As I said, this is a non-trivial change, so that both API changes (if any) and implementation issues need to be addressed. Please let me know if you need help with figuring out how to get involved. > > Thanks again, > Rich > > -----Original Message----- > From: Sï¿½bastien Boisvert [mailto:[email protected]] > Sent: Tuesday, August 14, 2012 9:47 AM > To: mpi-comments(a)mpi-forum.org > Subject: [Mpi-comments] One comment on MPI-3.0 Draft 2, August 2012 > > Dear MPI Forum committee members, > > I would like to submit a comment on the MPI-3.0 Draft 2, August 2012 for your consideration. > > Version: MPI-3.0 Draft 2, August 2012. > > The URL of the version of the MPI standard: > http://meetings.mpi-forum.org/draft_standard/mpi3.0_draft_2.pdf > > Page: 65 > > Line number: 28 > > Section: 3.8.1 > > In: > > 3. Point-to-Point Communication > 3.8 Probe and Cancel > 3.8.1 Probe > > Comment: > > It says that the source argument of MPI_Iprobe can be MPI_ANY_SOURCE, but it does say anything about fairness. Therefore MPI_ANY_SOURCE can lead to resource starvation. > > I think it would be better if probing would be done in a round-robin fashion when the source is MPI_ANY_SOURCE so that any MPI rank has an equal chance of having its message probed and received. > > Presently, the MPI standard contains nothing about which source should be probed when MPI_ANY_SOURCE is provided. > > I hope you will consider my comment. > > > Sincerely, > > > Sï¿½bastien Boisvert > PhD student > Universitï¿½ Laval > > _______________________________________________ > mpi-comments mailing list > mpi-comments(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-comments > > _______________________________________________ > mpi-comments mailing list > mpi-comments(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-comments _______________________________________________ mpi-comments mailing list mpi-comments(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-comments

1 0

Re: [Mpi-comments] One comment on MPI-3.0 Draft 2, August 2012
by Sébastien Boisvert 15 Aug '12

15 Aug '12

Hello Richard, I really agree that this is a big change. And I am aware that there is no fairness by design [1]. In Open-MPI, George Bosilca provided a patch [2], which I modified [3] for a round-robin policy and tested [4]. For threading, there is MPI_Init_thread that tells MPI if one or more thread will be spawned and if one or more will make MPI calls. Maybe there could be something like MPI_Init_fairness which could either be MPI_FAIRNESS_NO (the current behavior) or MPI_FAIRNESS_YES (which would do round-robin reception). How can I get involved ? [1] http://www.mcs.anl.gov/research/projects/mpi/tutorial/gropp/node92.html [2] http://www.open-mpi.org/community/lists/devel/att-9769/iprobe_starvation.pa… [3] http://www.open-mpi.org/community/lists/devel/att-9770/iprobe_starvation_fi… [4] http://www.open-mpi.org/community/lists/devel/2011/09/9770.php Richard Graham a ï¿½crit : > Sï¿½bastien, > Thanks you for sending in this request, this is truly appreciated. > The propose change is a large change to the standard, even though it may not seem so at a first glance. Today the standard is explicit about lack of fairness, and we would have to think through in detail on the broad impact of such a change. So, this does not the sort of item we can address for MPI 3.0. > However, if you would like to push a change like this into MPI, this can be addressed for the following version of the standard. What I would suggest, if this is something that you really think belongs in the standard, that you either get involved directly in the process (the standardization effort is open to all), or find someone the is involved to actively push such a request. As I said, this is a non-trivial change, so that both API changes (if any) and implementation issues need to be addressed. Please let me know if you need help with figuring out how to get involved. > > Thanks again, > Rich > > -----Original Message----- > From: Sï¿½bastien Boisvert [mailto:[email protected]] > Sent: Tuesday, August 14, 2012 9:47 AM > To: mpi-comments(a)mpi-forum.org > Subject: [Mpi-comments] One comment on MPI-3.0 Draft 2, August 2012 > > Dear MPI Forum committee members, > > I would like to submit a comment on the MPI-3.0 Draft 2, August 2012 for your consideration. > > Version: MPI-3.0 Draft 2, August 2012. > > The URL of the version of the MPI standard: > http://meetings.mpi-forum.org/draft_standard/mpi3.0_draft_2.pdf > > Page: 65 > > Line number: 28 > > Section: 3.8.1 > > In: > > 3. Point-to-Point Communication > 3.8 Probe and Cancel > 3.8.1 Probe > > Comment: > > It says that the source argument of MPI_Iprobe can be MPI_ANY_SOURCE, but it does say anything about fairness. Therefore MPI_ANY_SOURCE can lead to resource starvation. > > I think it would be better if probing would be done in a round-robin fashion when the source is MPI_ANY_SOURCE so that any MPI rank has an equal chance of having its message probed and received. > > Presently, the MPI standard contains nothing about which source should be probed when MPI_ANY_SOURCE is provided. > > I hope you will consider my comment. > > > Sincerely, > > > Sï¿½bastien Boisvert > PhD student > Universitï¿½ Laval > > _______________________________________________ > mpi-comments mailing list > mpi-comments(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-comments > > _______________________________________________ > mpi-comments mailing list > mpi-comments(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-comments

1 0

Post MPI 3.0 suggestion
by Stephen Booth 15 Aug '12

15 Aug '12

This comment comes under the following: > - Wishes for additional enhancements or new functionality > to be taken into account after 3.0 is published. I think there are some cases where persistent collective requests would make sense. Specifically as an opportunity to allow auto-tuning of the implementation based on the call parameters. The most compelling justification for this is to optimise AlltoAllv operations. Unlike the AlltoAll operation these cannot easily use tree based optimisations for small message sizes as each process only knows that part of the communication pattern they are directly involved with. However if the same communication pattern is to be re-used multiple times as a persistent request then the init call (which should itself be collective) could take the time to auto-tune a communication schedule best suited to the specified parameters. Stephen -- ====================================================================== |epcc| Dr Stephen P Booth Principal Architect |epcc| |epcc| s.booth(a)epcc.ed.ac.uk Phone 0131 650 5746 |epcc| ====================================================================== -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0