August 2012 - mpi-comments - mailman3.cels.anl.gov

Re: [Mpi-comments] Comments on MPI 3.0 draft 2
by George Bosilca 23 Aug '12

23 Aug '12

On Aug 23, 2012, at 20:56 , Jeremiah Willcock <jewillco(a)osl.iu.edu> wrote: > On Thu, 23 Aug 2012, George Bosilca wrote: > >> >> On Aug 18, 2012, at 20:13 , Jeremiah Willcock <jewillco(a)osl.iu.edu> wrote: >> >>> On page 84 line 42, "sizeof" should be inside \mathit. >> >> ? > > "sizeof" appears to be in math mode, but since it's a multi-letter variable name, it needs to be wrapped in \mathit so that LaTeX doesn't think it's a multiplication. "buf" on line 16 of that page has the same issue, while the version on line 14 doesn't. One is inside and the other outside a math environment. Fixed. >>> On page 84 line 45, where does "i" come from? Is the extent rounded to a multiple of k_i, or is the upper bound rounded (these differ when lb is not aligned)? If extent is rounded, the +eps term should be moved to the computation of extent, not ub, unless that definition of ub is used somewhere else. >> >> Which i? The type_i or k_i? > > Both -- nothing ever says what "i" means. Right. We used j above so it make sense to keep referring to them as _j and not _i. >> The upper bound is increased in such a way that the extent is a multiple of the biggest k_i (to solve all alignment issues in the case multiple instances of the same datatype are considered). > > That's for i = 0 ... n-1, right? That is the part that is never explained in the text. It is expressed on the definition of the Typemap few lines above. >>> On page 95, why does MPI_TYPE_CREATE_SUBARRAY use zero-based indexing for Fortran; I believe many other places in MPI use one-based array indices when called from Fortran, even if they use zero-based indices from C. >> >> I can't find any text that correspond to this remark. > > See page 96 lines 7-9 -- why is the function defined that way (requiring the n-1 modification to calls) for Fortran? Answered by Rajeev. >>> Also, is there a version of MPI_TYPE_CREATE_SUBARRAY that allows strided indexing in the dimensions (MPI_TYPE_CREATE_DARRAY looks more limited)? >> >> One can use a carefully hand-crafted datatype to achieve this. > > That is a suggestion to put on the list for the future. > >>> On page 115 line 44, it seems odd that adding MPI_BOTTOM to a valid address gives a valid address, since that allows me to create a valid address MPI_BOTTOM * n + v where n is nonnegative and v is a valid address. >> >> Too late to change now. We realized this few weeks ago, and we amended the example 4.18. > > This seems like it would be a serious problem for implementations that have MPI_BOTTOM != 0; why is it too late to fix? After reconsideration removing the last bullet in the list should be enough. george. > > -- Jeremiah Willcock > _______________________________________________ > mpi-comments mailing list > mpi-comments(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-comments

1 0

Re: [Mpi-comments] Comments on MPI 3.0 draft 2
by Rajeev Thakur 23 Aug '12

23 Aug '12

>>> On page 95, why does MPI_TYPE_CREATE_SUBARRAY use zero-based indexing for Fortran; I believe many other places in MPI use one-based array indices when called from Fortran, even if they use zero-based indices from C. >> >> I can't find any text that correspond to this remark. > > See page 96 lines 7-9 -- why is the function defined that way (requiring the n-1 modification to calls) for Fortran? It is so that the Fortran binding can simply pass the array_of_starts to the C interface without having to allocate an array and copy over everything while subtracting by 1. For the user, the array_of_starts is often filled as a calculation. He can easily subtract 1 in the calculation. It has been like that since MPI-2 (1997). Rajeev

1 0

Re: [Mpi-comments] Comments on MPI 3.0 draft 2
by Jeremiah Willcock 23 Aug '12

23 Aug '12

On Thu, 23 Aug 2012, George Bosilca wrote: > > On Aug 18, 2012, at 20:13 , Jeremiah Willcock <jewillco(a)osl.iu.edu> wrote: > >> On page 84 line 42, "sizeof" should be inside \mathit. > > ? "sizeof" appears to be in math mode, but since it's a multi-letter variable name, it needs to be wrapped in \mathit so that LaTeX doesn't think it's a multiplication. "buf" on line 16 of that page has the same issue, while the version on line 14 doesn't. >> On page 84 line 45, where does "i" come from? Is the extent rounded to >> a multiple of k_i, or is the upper bound rounded (these differ when lb >> is not aligned)? If extent is rounded, the +eps term should be moved >> to the computation of extent, not ub, unless that definition of ub is >> used somewhere else. > > Which i? The type_i or k_i? Both -- nothing ever says what "i" means. > The upper bound is increased in such a way that the extent is a multiple > of the biggest k_i (to solve all alignment issues in the case multiple > instances of the same datatype are considered). That's for i = 0 ... n-1, right? That is the part that is never explained in the text. >> On page 95, why does MPI_TYPE_CREATE_SUBARRAY use zero-based indexing >> for Fortran; I believe many other places in MPI use one-based array >> indices when called from Fortran, even if they use zero-based indices >> from C. > > I can't find any text that correspond to this remark. See page 96 lines 7-9 -- why is the function defined that way (requiring the n-1 modification to calls) for Fortran? >> Also, is there a version of MPI_TYPE_CREATE_SUBARRAY that allows >> strided indexing in the dimensions (MPI_TYPE_CREATE_DARRAY looks more >> limited)? > > One can use a carefully hand-crafted datatype to achieve this. That is a suggestion to put on the list for the future. >> On page 115 line 44, it seems odd that adding MPI_BOTTOM to a valid >> address gives a valid address, since that allows me to create a valid >> address MPI_BOTTOM * n + v where n is nonnegative and v is a valid >> address. > > Too late to change now. We realized this few weeks ago, and we amended > the example 4.18. This seems like it would be a serious problem for implementations that have MPI_BOTTOM != 0; why is it too late to fix? -- Jeremiah Willcock

1 0

Re: [Mpi-comments] Comments on MPI 3.0 draft 2
by George Bosilca 23 Aug '12

23 Aug '12

On Aug 18, 2012, at 20:13 , Jeremiah Willcock <jewillco(a)osl.iu.edu> wrote: > Chapter 4: > > On page 83 line 23, "communication" should be "communications". > > On page 83 line 34, "shape and size" should be "shapes and sizes". Done. > On page 84 line 42, "sizeof" should be inside \mathit. ? > On page 84 line 45, where does "i" come from? Is the extent rounded to a multiple of k_i, or is the upper bound rounded (these differ when lb is not aligned)? If extent is rounded, the +eps term should be moved to the computation of extent, not ub, unless that definition of ub is used somewhere else. Which i? The type_i or k_i? The upper bound is increased in such a way that the extent is a multiple of the biggest k_i (to solve all alignment issues in the case multiple instances of the same datatype are considered). > On page 95, why does MPI_TYPE_CREATE_SUBARRAY use zero-based indexing for Fortran; I believe many other places in MPI use one-based array indices when called from Fortran, even if they use zero-based indices from C. I can't find any text that correspond to this remark. > Also, is there a version of MPI_TYPE_CREATE_SUBARRAY that allows strided indexing in the dimensions (MPI_TYPE_CREATE_DARRAY looks more limited)? One can use a carefully hand-crafted datatype to achieve this. > On page 103 line 18, MPI_ADDRESS has now been removed, not just deprecated. Indeed. > Is offset of a valid way to get displacements within C structs to use with MPI (rather than creating an object of that type and calling MPI_GET_ADDRESS on it and its individual fields)? Absolutely. > Wouldn't MPI_TYPE_SIZE and a way to "flatten" an MPI datatype into a datatype with the same primitive elements but sequential in memory (or using MPI_PACKED) be better than MPI_TYPE_TRUE_EXTENT for the example on page 108 line 29? There are multiple ways to achieve the goal depicted in the example. MPI_Type_get_true_extent is a valid one, even if not the most optimal in terms of memory. > On page 110 line 24, there should be a comma after "e.g.". > > On page 111 line 42, there is some odd space at the end of the line. Weird, I can't figure out how to get rid of this one. I'll investigate more. > On page 112 line 7, there shouldn't be a space after "datatype". > > On page 112 lines 11-14, should there be an MPI_TYPE_FREE added as well? Pedantic(!). > On page 115 line 44, it seems odd that adding MPI_BOTTOM to a valid address gives a valid address, since that allows me to create a valid address MPI_BOTTOM * n + v where n is nonnegative and v is a valid address. Too late to change now. We realized this few weeks ago, and we amended the example 4.18. > On page 128 line 32, example 4.18 uses "class" as a field name, which makes it impossible to compile with a C++ compiler. Fixed. Thanks, george.

1 0

Comments on MPI 3.0 draft 2 [part 4]
by Jeremiah Willcock 23 Aug '12

23 Aug '12

These are in reference to the draft at <URL:http://meetings.mpi-forum.org/mpi3.0_draft_2.pdf>. Thank you for considering them. -- Jeremiah Willcock Chapter 11: Why are the request-based one-sided functions named MPI_R* rather than MPI_I* like nonblocking point-to-point and collective functions? There are no request-based versions of fetch-and-op or compare-and-swap. Page 403 line 31: Add a comma after the second capital "B". Page 403 line 34: The semicolon here can be changed to a comma, and probably removed entirely. Page 404 line 15: "distributed memory" should be hyphenated. Page 404 line 22: The first comma should be a colon. Page 404 line 23: There should be a comma after "MPI_WIN_ALLOCATE_SHARED". Page 404 line 25: There is extra space around "to specify". Page 404 line 30: Should MPI be in sans-serif font? Page 405 line 24: Remove "and C++". Page 405 line 32: "address sized" should be hyphenated. Page 405 lines 42-43; page 408, lines 26-27; page 409, line 47: Which of these info keys must be set consistently between different processes? Page 406 lines 4-9: Can accumulate_ops be validly and portably set to anything other than the two values listed there? Page 406 line 17: There should be a comma after "units". Page 406 lines 42-43: "implementation specific" should be hyphenated. Page 409 line 44: The function name on the next line looks like it might fit on this one. Page 411 lines 4-5: Window flavors have not yet been introduced. In particular, the allocation functions should probably say which flavor of window they each produce. Page 411 lines 7-8: Some of these argument names might need to be in sans-serif. Page 411 lines 30-: "RMA" was written in sans-serif earlier in the chapter. Page 412 line 34: "MPI" should be in sans-serif. Page 413 line 21: "win" should be in sans-serif. Page 413 lines 21-22: Can the same memory be attached to different windows, just not the same window? Page 414 line 25: The "should" might need to be "must" to avoid issues similar to <URL:https://upc-bugs.lbl.gov/bugzilla/show_bug.cgi?id=495>. Page 415 line 12: "called" should be "call". Page 416 line 6: There should be commas after "disp_unit" and "create_kind". Page 416 line 20: There should be a comma after "returned". What if the user modifies the pointees of the pointers that are the values of these attributes in C? Page 416 line 28: Why return the group rather than the communicator itself? The group returned is listed as a "duplicate" -- does that mean the user is responsible for freeing it? Page 417 line 2: "MPI" should be in sans-serif. Page 417 line 12: "this" should be "the". Page 418 line 17: There should be a comma after "MPI_RGET_ACCUMULATE". Page 418 line 27: Make sure to note that "locally completed" does not mean the same thing as "completed" (I think). Will those be defined in detail later? Page 418 line 30: There should be a comma after "get call". Page 418 line 35: "; namely," can be changed to just a colon. Page 418 line 36: Which calls count as "accumulate calls"? Page 420 line 2: There should be a comma after "origin node". Page 420 line 31: "shared memory" should be hyphenated. Page 420 line 40: "occurred" should be "occurs". Page 422 line 7: There should be a comma after "B". Page 423 lines 31 and 37, page 424 lines 18 and 24: Would it be helpful to users to show the correct hints ("assert" arguments) to use in MPI_WIN_FENCE to express each fence's semantics to the implementation? Page 423 line 47: The example header should be on the same page as the example text. Page 424 line 33: I think "contribution" should be "contributions". Page 425 line 40: There should be a comma after "origin_count". Page 426 lines 9-11: Is there a valid reason that MPI_REPLACE (and MPI_NO_OP) are not allowed for MPI_REDUCE? They are not useful, most likely, but orthogonality seems to suggest that they be allowed anyway. Page 426 line 18: There should be a comma after "B". Page 426 line 46: Remove "then". Page 428 line 15: There is a line break (and thus probably a space in the source code) after the left parenthesis here. Page 428 line 26: "and MPI_NO_OP or" should be "MPI_NO_OP, or". Page 429: Is there a reason that there is not a similar special-case version of MPI_ACCUMULATE? Page 430: What happens if an MPI implementation does not support, for example, compare-and-swap on MPI_INTEGER16 values? Page 430 line 21: Please put compare_addr as its own variable declaration to avoid a line break between two variables. Page 430 line 48: "passive-target epoch" has not been defined yet as far as I know, and this use of the term does not reference a definition. Page 431 line 10: There should be a comma after "MPI_WIN_FLUSH_LOCAL". Page 432 line 15: There should be a comma after "MPI_WIN_UNLOCK". Page 433: Is it possible/likely that an rget does not complete locally until after a flush, fence, or similar call? That might be worth mentioning if true. Page 436 lines 24-25: "RMA" here should be in sans-serif (plus italic). Page 436 lines 26-34: This seems to imply sequential consistency between get and put accesses on the same window, even if they are not consistent with loads and stores? Or is that issue removed by the rules about overlapping get and put accesses in the same epoch? Page 436 lines 35-39: Can different processes see the updates in a different order (than each other's view of them, not just from the order the updates actually occurred)? Page 437: "passive target" and "active target" should be hyphenated. Page 437 line 19: "shared memory" should be hyphenated. Page 438 line 7: There should be a comma after "MPI_WIN_POST". Page 441: Does MPI_WIN_FENCE start and/or end active-target or passive-target epochs? This is answered on page 438, but the answer should be here as well. Page 443 line 13: There should be a space between the comma and "win". Page 443 line 22: "occurred" should be "occurs". Page 443 line 25: Insert "has" before "called". Does it make sense to have a nonblocking version of MPI_WIN_START? What about MPI_WIN_FENCE or the other synchronization calls that could block (MPI_WIN_WAIT already has a nonblocking version, although it does not use an MPI_Request). Page 445 line 21: "call" should be "calls". Page 445 lines 25 and 31: The "no need to wait" part is not a complete sentence. Page 445 lines 24-34: "initiate" should be either "initiates" or capitalized in each of these. Page 446 line 2: The tuple representing the graph should probably use \langle and \rangle rather than < and >; this will probably fix the spacing around them as well. Page 446 lines 6 and 10: "group" in each of these lines might need to be sans-serif. Page 447 line 48: There is extra space before "load". Page 448 line 1: "call" should be "calls". Page 448 line 3: "lock protected" should be hyphenated. Page 450 lines 16-18: How does this affect whether rget, rput, ... accesses have "locally completed"? Is this sentence true about those kinds of accesses until MPI_Wait is called on them? Is MPI_Wait on such a request guaranteed not to block after MPI_WIN_FLUSH_LOCAL returns? Page 450 line 46: "copy" should be "copies"? Page 451 line 15: "implementation" should be "implementation's". Page 451 line 17: "assertion specific" should be hyphenated. Page 451 line 25: There should be a comma after "MPI_MODE_NOPRECEDE". Page 452 lines 1 and 9: There is extra space before "stores" on each of these lines. Page 452 line 22: There should be a comma after "acquire". Page 452 line 32: "argument" should be "arguments". Page 452 line 43: "comm" should be in sans-serif. Page 453 lines 33-36: Is this text defining "complete", or just repeating a definition stated officially elsewhere? If it is the definition, the wording doesn't make that obvious. Page 454 lines 14 and 21: "becomes visible" means "becomes visible eventually" (in an implementation-defined amount of time); that may be worth stating explicitly here. Page 454 line 18: There is a space missing before "MPI_WIN_LOCK". Page 454 line 29: Insert "a" or "the" before "private". Which process does "the process memory" refer to? Page 454 lines 35-36: What is the motivation for putting "even" before "flush"? Page 454 line 47: This needs to be qualified with "within the same access epoch" or similar, like the next sentence is. Page 455 line 4: "MPI" should be sans-serif. Page 455 lines 26 and 29: Does "update" mean remote update here? Page 456 line 1: "will" should be "would". Page 456 line 18: "a remote process" should be plural. Page 456 line 23: "magnitude" should be "magnitudes". Page 456 line 38: There is extra space before "updates". Page 456 line 41: Add "elsewhere" before "in". Page 456 line 45: Remove "in the case". Page 457 line 14: There is extra space before "stores". Page 457 line 20: Should "being posted" be "posted" instead? The comma after "posted" can be removed. Page 457 line 47: "implementation dependent" should be hyphenated. Page 458 line 5: "X" (and probably "A" and "B" where they appear) should be in code font. This applies to the rest of the examples as well. Page 458 lines 11 and 17, page 459 lines 4 and 10, page 460 line 35, and page 461 line 9: "EXCLUSIVE" should be "MPI_LOCK_EXCLUSIVE". Page 458 line 23: There is extra space after "combining". Page 458 line 26: "the" should be "an". Page 458 line 38: There is extra space after "not". Page 458 line 39: "compiler and hardware specific" should be "compiler- and hardware-specific". Page 461 line 27: How are get-accumulate and similar operations incorporated into these restrictions? It would be very useful to be able to use fetch-and-add to get unique counter values, for example, although this operation will return different answers based on message interleavings (contradicting lines 36 and 37). How does compare-and-swap fit into the rules about accumulate_ops? Is compare-and-swap considered to be another reduction op for the purposes of the concurrent update rules, or is it a special case? Page 461 line 30: The comma should be a semicolon or colon. Page 462: Is it possible to express the different ordering models in terms of shared-memory consistency models (sequential, cache-coherent, etc.)? Those names are easier to look up than MPI-specific terms for accumulate_ordering values. Page 462 lines 31-32: Is this true for both memory models, or is there a stronger rule for the unified memory model? Page 463 lines 28 and 31: Remove the comma after "reversed". Page 463 line 30: Remove the comma after "deadlock". [may not be fixable] Page 464 lines 29-30: It looks odd to have "end of advice to users" after something saying that the entire section is advice to users. Page 464 line 32: "value" should be "values". Page 465 line 8: There is extra whitespace in this line. Page 465 lines 11-19: Section titles are in sans-serif here, while they are normal text in the rest of the standard. Page 465 line 43: "use" and "provide" should be "uses" and "provides", respectively. Page 467 line 26: There should be a space before "*/". Page 468 line 4: The semicolon should be a comma. Page 468 line 7: "naive" might need to be 'na\"\ive'. Page 469 lines 16 and 19: "A" as the variable name might need to be in the code font. Page 469 lines 38-39: "NSTEPS" and "M" should be in the code font, or else "NSTEPS" needs to be inside \mathit. Page 470 line 25: "N" might need to be in the code font or math mode. Chapter 12: Page 476 line 42: There is extra space after "request". There are a few places where "thread safe" is used as an adjective; in that context, it should be spelled "thread-safe". Page 487: I think MPI_INIT_THREAD should be moved to be near MPI_INIT in the standard, since many high-performance applications use threads and any threading at all (even OpenMP in non-MPI-using regions) is forbidden when MPI_INIT is used. [post 3.0] In the future, it might make sense for MPI_INIT to choose a thread mode such as MPI_THREAD_FUNNELED (or for implementations to be allowed to choose one) as the default when, for example, OpenMP is enabled by compiler settings. Many people are probably using MPI_INIT in OpenMP applications without knowing that MPI makes doing that erroneous. Page 488 lines 37-41: The text in lines 39-41 seems to contradict the behavior of MPI_INIT required by 37-38. Page 489 line 22: "is specified at link time." is repeated twice. It might be worthwhile to discuss MPI_MPROBE and MPI_IMPROBE and why they are necessary in threaded contexts in "advice to users" or similar non-normative text somewhere in this chapter.

1 0

Comments on MPI 3.0 draft 2 [part 3]
by Jeremiah Willcock 20 Aug '12

20 Aug '12

These are in reference to the draft at <URL:http://meetings.mpi-forum.org/mpi3.0_draft_2.pdf>. Thank you for considering them. Chapter 9: Page 367 line 21: There is an extra space before "value" that is not in other occurrences of that pair name. Page 367 line 29: Why is there a standard-defined upper limit on key length, rather than just saying it's up to implementations? Page 367 line 30: There is no lower bound given on MPI_MAX_INFO_VAL, allowing an implementation to have an absurdly low limit such as one character. Page 368 line 2: Are all strings consisting only of an (optional) sign and decimal digits considered to be decimal (and not octal if they begin with a leading zero)? This issue has come up in practice. Page 368 line 3: There is an extra space before "integer". Page 368 line 6: What is a "valid element"? Is that just a value of the correct type for that particular info key? Is it possible to have lists of lists? Is there a way to escape a comma in a list element? Page 369 line 16: What if the key is too long? Page 370 line 29: Why does this particular case of the key being too long not return a defined error class, while the other functions do? Chapter 10: The chapter uses "intercommunicator" as one word, which is inconsistent with the chapter on communicators. Page 373 line 19: "allow" should be "allows". Page 373 line 24: "later" should be "latter". Page 373 line 32: There is extra space before and after "MPI". Page 374 line 31: There is extra space before "group". Page 376 line 1: There is extra space after "MPI_UNIVERSE_SIZE". Page 376 lines 11-12: "(group,rank)" was written without spaces before this, but with spaces here. Page 376 line 21: "starting first" should be "first starting". Page 377 line 1 and page 382: Is there a reason that argv is not "const char* const argv[]" or similar? Compatibility? Page 378 lines 13-18: This is written very informally. Page 378 lines 22-23: Is there a way to pass an empty argument in Fortran without terminating the argument list? Page 379 line 36: "soft" should be in italics (the one on line 37 should not be). Page 380 line 3: The 9 here refers to a chapter, not a section. Page 380 line 16: Must the root argument be consistent across the entire communicator? Page 380 line 24: Must the same error code be used for all the failed processes? If not, the "error code" part needs to be plural. Page 380 lines 30-32: This refers to the C++ binding which have been removed. Page 381 lines 13-14: Is calling MPI_COMM_FREE on the parent intercommunicator an error? If not, why not? Page 382 lines 35-36: These argument declarations should be separate so the list of variables is not split onto two lines. Page 383 line 25: Why can't MPI_ARGV_NULL be used for individual commands? Page 384 lines 7-8: The command name and arguments should be in the code font. Page 384 line 17: This text is very informal. Page 384 lines 37-: Lines 2+ of each of these paragraphs does not line up with the text on the first line. Page 384 line 45: "path" should be in sans-serif. Page 385 line 15: Is c allowed to be zero? Is b allowed to be equal to a? Page 385 lines 20 and 23: Add "a" (on 20) and "an" (on 23) after "allows". Page 385 line 27: There is extra space before "Using". For spawned processes, what happens if one or more of the child processes calls MPI_Finalize? MPI_Abort? What about one or more of the parents? Which processes can continue in those cases, and what happens to the various communicators? Page 400 mentions it for connected processes, but are the rules for children different? Page 387 line 27: "client server" should be "client/server". Page 387 line 44: There should be a comma after "simple". Page 387 line 45 and page 389 lines 19 and 29: "internet" may or may not need to be capitalized depending on which meaning is intended. Page 387 line 47: The comma should be a semicolon. Page 389 line 19 and others: Some or all of "(host:port)" might need to be in sans-serif or code font. Page 390 line 33 and page 391 line 26: Must root be the same on all processes in the communicator? Page 391 line 37: "high quality" should be "high-quality". Page 391 line 40: There should be a comma after "MPI_COMM_ACCEPT". Page 392 line 2: Some parts of this line should be in sans-serif or code font. Page 392 line 42: Should the behavior be implementation-dependent (with a defined error class if the implementation chooses to return an error) rather than completely undefined? Page 394 line 33: IP does not have port numbers; they are added by higher-level protocols such as TCP and UDP. Page 395 line 9: "stdin" should be in the code font. Page 395 lines 22-34: Do some of these calls need to execute on only a single rank if the program is parallel? Page 396 line 17: Doesn't the client need to be disconnected explicitly in this case as well? Page 399 lines 35-36 seem to say that. Page 399 lines 11-12: Having only a single "appnum" info argument for MPI_COMM_SPAWN_MULTIPLE breaks the usual mechanism in which each command spawned gets a different MPI_APPNUM. Is there a way to specify a list of appnums instead of a single one for MPI_COMM_SPAWN_MULTIPLE? Page 399 line 46: There should be a comma before "may". Page 400: What does MPI_COMM_DISCONNECT do to/on spawned processes? Page 400 line 36: There should be a comma after "MPI_WIN_FREE". Page 400 line 38: "Notes" should be "Note". Page 402 line 12: Does "concurrently" mean "concurrently on the same socket"? -- Jeremiah Willcock

1 0

Re: [Mpi-comments] Comments on MPI 3.0 draft 2 [part 2]
by William Gropp 19 Aug '12

19 Aug '12

Jeremiah, Thanks for your very detailed review; we'll go over these carefully. On your first point about the style of the standard, this has been a long-term problem, as the audience for the document includes both implementors and users. It is something to remember, as the most important role for the document is as a standard, not a user guide. Bill William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign On Aug 20, 2012, at 7:06 AM, Jeremiah Willcock wrote: > These are in reference to the draft at > <URL:http://meetings.mpi-forum.org/mpi3.0_draft_2.pdf>. Thank you for considering them. > > -- Jeremiah Willcock > > General: > > [post 3.0] Overall, the document reads like a tutorial rather than a reference document or standard (for example, going from simpler topics to more advanced ones, forward references for datatypes and communicators in the point-to-point chapter, ignoring MPI_PROC_NULL when talking about point-to-point operations then introducing it later in the chapter, and the wording on page 225 lines 13-16). > > It would be nice to know, for each function, constant, etc., which MPI version introduced it. This would allow developers to write MPI-3.0-compliant programs that would also work on older MPI versions. > > Chapter 5: > > Why are nonblocking collectives in their own section rather than mixed in with the normal collectives (i.e., put each collective in both blocking and nonblocking versions together)? > > Page 141: MPI_REDUCE_LOCAL is not mentioned here at all (it isn't a communication operation, but it is part of the contents of this chapter). > > Page 141 lines 33 and 36: I don't think "gather" and "scatter" need to be capitalized. > > Page 141 lines 40-43: The reduction paragraph here looks very different from the others (has the ALL* version first, groups them together with single-root reduce as a variant version, and points to a subsection for allreduce while pointing to its whole containing section for reduce). > > Given the pattern of the other sections, allreduce and reduce should be in separate sections, with reduce first followed by allreduce. > > Reduce_scatter_block and reduce_scatter should be in the same subsection for the nonblocking versions, just like allgather and allgatherv (for example) are. > > Page 142 lines 1-3: It's odd to point to section 5.11 then subsection 5.11.2; these should probably be combined. > > Page 144 line 9: What are the "auxiliary functions" referring to? Do they require any access to the communication library outside that provided by MPI point-to-point operations? > > Page 144 lines 25-27: Are intercommunicator and intracommunicator hyphenated or not? They are used both ways in these three lines. Chapter 6 uses hyphenated versions. > > Page 145 line 45: The list here appears to be every collective from the list starting at line 9, except for the scan collectives. It might be easier to just say that. > > [post 3.0] Why does bcast not have separate send and receive buffers (with an MPI_IN_PLACE option) like the other one-to-all collectives? > > Page 148 line 33: What does "itself included" mean here? The root does not do any writing to its buffer, so that text seems to be meaningless, and conflicts with the "other" on line 35. > > Page 148 line 33: "is" should be "must be". > > Page 149 line 6: The sentence should be singular since there is only one example. > > Page 160 lines 19-21: It seems odd to have an unnecessary restriction just for symmetry; why not have the call be as general as possible? > > Given that multiple writes are now allowed using MPI_Put with undefined results, might it make sense to apply the same rule to multiple MPI_Recvs and collective recvbufs that write the same location multiple times? > > Page 162 lines 10-11: This restriction is unnecessary as well. > > Page 162 line 42: It might be interesting to have the same example written using MPI_Scatter (no v) and a datatype with an adjusted extent, or at least not claim that MPI_Scatterv is required. > > Page 165 lines 40-43: This description is incorrect when MPI_IN_PLACE is used, I think. This comment also applies to page 167 lines 15-20, page 168 lines 42-48, page 171 lines 5-10, and page 173 lines 13-18. > > Page 165 line 45: There is an extra space after "0" in this line. > > Can MPI_BOTTOM be used as the buffer location for collectives when count=0 for the corresponding send or receive (depending on the collective)? Can NULL be used, or some random invalid pointer? That might be worth stating explicitly. > > In v and w versions of collectives, if the count is 0 for some entry of the array of counts, is the displacement for that entry guaranteed to be ignored (and thus doesn't need to point to valid memory)? > > For rooted collectives, is root allowed to be MPI_PROC_NULL? It is probably not, but this should probably be stated explicitly. > > Page 170 lines 43-45: Not only can the displacements be different, but the sizes can be as well. > > Page 171 lines 14-15, 20-21: What is the reason for MPI_IN_PLACE needing to be given on all processes? > > Page 171 lines 32-41: This text seems like it should be after the description of MPI_Alltoallw, since most of it applies to all three functions. > > Page 171 lines 34-35: The restriction to sequential storage does not apply to MPI_Alltoallv (unless that's meant to say that they need to be offsets in the same buffer, unless MPI_Alltoallw). > > Page 172: As I commented yesterday, MPI_Alltoallw should have a version taking arrays of MPI_Aint for sdispls and rdispls to allow the use of MPI_BOTTOM for sendbuf and recvbuf. Note that MPI_Neighbor_alltoallw does use MPI_Aint (probably since it does not need to be backwards-compatible); if a different name is chosen for MPI_Alltoallw with MPI_Aint, MPI_Neighbor_alltoallw might need to be renamed to match it. > > Page 173 line 35: MPI_SCATTERW should be in sans-serif font rather than code, I think; BTW, why aren't there w versions of all of the v collectives? > > Page 173 line 40: The first "and" on this line should be "or". > > Page 174 lines 36-37: "count" should have a comma after it on both lines. > > Page 174 line 39: "root" should have a comma after it. > > Page 174 line 45: Remove the comma here. > > Page 175 lines 28-29: This conflicts with page 174 line 38. > > Page 176: Are the logical reduction ops allowed to return any nonzero value for "true"? The definition of Boolean value earlier in the standard says that the test for bool arguments is zero/nonzero; are reduction ops given that flexibility in their outputs as well? > > Page 177 line 29: Why are logical operations not allowed on Fortran integer types? > > Page 178: Why are the functions named PAR_BLAS* and not something like PAR_DOT*? > > Page 178 line 20: Does "array" mean "matrix" here? > > Page 179 lines 47-48: It might be worth mentioning that MPI_MAXLOC does not compute a lexicographic maximum. > > Page 180: Would it be possible to generalize the *loc routines to allow wider integer types as the second element? > > Page 180 line 39: This computation ignores alignment issues (in particular, if int is larger than float). > > Page 182: Example 5.19 breaks if the total data size is greater than INT_MAX, which is increasingly common on large systems, so it can't be used as the basis of scalable code. > > Page 183 line 40: There should be a comma after "len". > > [post 3.0] Reduction operations should allow user-defined ops that change the data size (this is important for bindings to higher-level languages such as C++). > > Page 184 line 15: There should be more space around the circle operator (like on line 20). There are extra spaces after "0" and the ellipsis. > > Page 184 lines 36-45: What does this text mean? There are functions in MPI to decode datatypes. > > Page 184 line 48: "convention" should be "conventions". > > Page 185 line 4: "naive" should be "na\"\i{}ve". > > Page 185 line 41: Are user-defined ops required to be freed at some point? What is the reason for having MPI_Op_free? > > Page 186 line 9: "it is time" sounds very informal. > > Page 187 line 31: How "identical" do the results need to be? Reasonable algorithms may output non-bit-identical results on different ranks for not-truly-associative floating-point operations. > > Page 187 lines 28-: What are the requirements about argument consistency between processes for MPI_Allreduce? > > Page 188 line 28: "array" should probably be "matrix". > > Page 189 line 6: This sentence should be singular. > > Page 189 lines 11 and 44, page 190 line 21, page 192 line 1: There is an extra space after the left parenthesis on each of these lines. > > Page 191 line 7: There should be a comma after "op". > > Page 191: I think "n", and maybe "i", here should be in sans-serif font. > > Page 192 lines 32-39: "n" and "i" are in different fonts in different places in this paragraph. > > Page 192 line 34: There should be a comma after "count". > > Page 192 line 39: There should be a comma after "recvcounts[i]". > > Page 193 line 46: It seems like the code here should be in sans-serif font, plus with spaces after the commas near the end of the line. > > Page 193 lines 47-48: The variable names here are in the wrong font, and there should be a comma after "op" on line 48; both of these issues also apply to page 194 lines 33-34. > > Page 194 lines 38-39: Are the contents of recvbuf completely undefined, or are they just left unmodified from what they were before MPI_Exscan was called? Page 195 line 3 says that MPI_IN_PLACE means that recvbuf is left unchanged; is that true even without MPI_IN_PLACE? > > Page 194 lines 39-40: It is unnecessary to special-case rank 1 here. > > Page 196: The spacing inside the parentheses of the function calls in the code here is inconsistent. > > Page 197 lines 15-16: Does "passed to a completion call" mean "completed" in the sense of point-to-point (i.e., not just having MPI_Test called once, but calling a sequence of test and wait operations that end up moving the request to the completed, inactive state)? > > Page 197 lines 31-32: Why do nonblocking collectives fill in MPI_ERROR rather than returning an error from the completion call like nonblocking point-to-point calls do? > > Page 197 line 46: "MPI exception" should refer to the definition in section 8.3. > > Page 198 lines 1-7: Which processes need to complete (in the MPI_Wait, etc. sense) a nonblocking collective for it to finish executing (as in, allow MPI_Wait to return) on other processes? > > Page 198 lines 31-32: What does it mean that the progression rules are just "similar"? Do they differ in some way from the point-to-point rules? > > Page 198 line 41: There is an extra space after "comm". > > Section 5.12 (in general): The argument name "request" in the annotated argument list for each function does not line up with the other argument names above it. > > Page 216 lines 1-10: Would this example be safe if an ibarrier but blocking point-to-point calls were used? > > Page 217 lines 24-27: This rule should be a lot earlier; the previous pages seem to imply that you can pipeline nonblocking collectives on the same communicator as long as you start them in the same order on all ranks. Example 5.29 violates this rule as well, and example 5.34 explicitly states the opposite of page 217. > > Page 220 line 20: "completed" should be "complete". > > Page 220 line 46: The example header should be on the same page as the text. > > [post 3.0] Dynamic sparse data exchange (http://www.unixer.de/publications/img/hoefler-dsde-protocols.pdf) in some form might be worthwhile to add as a collective in the future, at least for the case where all data is sent at once using an alltoallv-like call but the receiver does not know the data quantity from each sender up front. Something like http://www.tu-chemnitz.de/informatik/PI/forschung/pub/download/HR_europvmmp… would also be very useful. > > Chapter 6: > > Page 224 line 20: "on a par" should probably be "on par". > > Page 229 line 1: There is an extra space before the left parenthesis. > > Page 231 line 45: "first" should be "group1". > > Page 231 line 48: Add "the" before "first group". > > Page 233 line 13: There should be a comma after the ellipsis. > > Page 233 line 16: Does that mean newgroup == group (as handles), or just that the groups themselves are MPI_IDENT? > > Page 233 line 39: "consist" should be "consists". > > Page 235 line 44: There is extra space before "comm". > > Page 236 lines 11 and 40: The comma is unnecessary here. > > Page 236 lines 14-15: This conflicts with the wording on page 227 lines 44-45. > > Page 237 line 7: There should be a space before the second "MPI_Comm". > > Page 237 line 42: The terms "group A" and "group B" are used for "left" and "right" (in some order) in chapter 5; consistent terms should be used. > > Page 238 line 15: "duplicates" should be lowercase. > > Page 238 line 23: Remove "call". > > Page 238 line 26: Does that mean that pending collective communications are not allowed? > > Pages 238-239: MPI_COMM_IDUP and MPI_COMM_DUP_WITH_INFO are in the opposite order when being introduced at the beginning of this section. > > Page 240 line 4: "group" should be lowercase. > > Page 240 line 20: Add "MPI_COMM_CREATE" after "call". > > Page 240 line 24: Add a comma after "otherwise", or merge this sentence with the previous one. > > Page 240 line 44: "sometimes to avoid" sounds awkward. > > Page 241 line 3: Add a comma after "MPI_COMM_CREATE". > > Page 241 lines 4-5: "further to subdivide" should be "to further subdivide". > > Page 241 line 17: "should" should be "must". > > Page 241 lines 21-28: What ends up as the remote group of the new inter-communicator? > > Page 242 line 22: "create" should be "creation". > > Page 243 line 44: "comm" is in an incorrect font. > > Page 244 line 30: Add "or MPI_UNDEFINED" to the end of this sentence. > > Page 245 line 10: "doesn't" should be "does not". > > Page 245 lines 30-48 and page 247 lines 2-16: The spaces in the function calls here are inconsistent with most other examples in the specification. > > Page 250 lines 17-: The style used for examples here is different from in the rest of the specification. What are these examples (especially the first few that do not do anything "interesting" with groups or communicators) trying to show? > > Page 256 line 42-page 257 line 6: "backmasking" is spelled three different ways in these two paragraphs. > > Page 257 lines 42-43: Doesn't MPI_COMM_IDUP work on intercommunicators? Why are there not non-blocking versions of the other {intra,inter}-communicator constructors? Lack of interest/need? > > Page 259 line 11: What is an "additional safe context"? > > Page 260 line 1: "and" should be "or". > > Page 260 line 4: The colon probably should be a period since the list does not immediately follow. > > Page 261 lines 4-6: "Dual membership" seems to be describing a non-standard behavior that would likely cause problems for many inter-communicator collectives (especially with MPI_IN_PLACE). > > Page 262 line 44: There should be a comma after "MPI_COMM_FREE". > > Page 265 line 45: Add a comma after "window". > > Page 266 line 12: "attributing" should be "attribute". > > Page 270 line 22: The last comma in this line should be a semicolon. > > Page 272 line 44: What does "new" refer to here? > > Page 279 lines 29-30: "The last three" should be "the last four", but it seems odd that a communicator with an invalid keyval associated with it would ever get to the point that it could be copied. > > Page 279 lines 34-35: What does this sentence mean? > > Page 283 line 9: Should "got" be "gotten" or "retrieved"? > > Page 285 line 37: The term "loosely synchronous model" is not used anywhere else that I can tell except in the context of one-sided operations. It seems like it's some property of MPI-based libraries, but this is not explained. If this topic is going to be discussed, it should also be mentioned in the context of library routines that are non-blocking internally (and maybe suggest the use of generalized requests for that), as well as why MPI_COMM_IDUP is necessary for some libraries. > > Page 287 line 1: "well-ordering" should probably be "non-overtaking" to match standard MPI terminology. > > Page 287 line 15: "then" should be "that". > > Chapter 7: > > Page 290 line 2: There should not be a comma before "process-naming". > > Page 291 line 11: There should be a comma after "therefore". > > Page 292: Which arguments must be consistent across all processes? > > Page 292: Is the created topology when reorder=false row-major or column-major? (The distinction does not matter when reorder=true). > > Page 293: Is MPI_DIMS_CREATE a local call? > > Page 294 line 37: There should be a comma after "index". > > Page 297 lines 46-47 and page 300 lines 2-4: Why not allow MPI_UNWEIGHTED as a special case for some or all isolated nodes (even if it is not passed on non-isolated nodes)? That would avoid the issues discussed on page 298 lines 6-16 and page 300 lines 9-19, and make this call more consistent with others that allow NULL for empty arrays. In particular, having MPI_UNWEIGHTED be possibly NULL and yet forbidden for degree-zero nodes prevents using something like an std::vector that could be empty as a source for weights (since its data pointer might be NULL when it is empty). > > Page 302 line 48, page 303 line 26: The variable names should be sans-serif. > > Page 307 line 2: The variable names should be sans-serif. > > Page 307 line 3: "array" should be "arrays". > > Page 312: Which arguments must be consistent across processes? Is this function collective? > > Page 312 line 46: "capability" should be "capabilities". > > Pages 313-314: Are the *_MAP functions local or collective? If they are collective, which arguments must be consistent? > > Pages 314-330: Why are these functions not in chapter 5 like the other collectives? If the issue is that they apply to topologies only, you can generalize them by assuming a non-topology-based communicator is a one-dimensional ring or similar. > > Page 320 lines 42: This line ends abruptly. > > Page 321 lines 3-21: "indegree" and "outdegree" should be in sans-serif font when referring to arguments. > > Page 324 line 28: This line ends abruptly. > > Page 331 lines 29-30: The results of these calls are only used in commented-out code on page 332. > > Page 333: This code seems like it would be clearer if it used MPI_Neighbor_alltoallv and removed the variable sizeofreal. Also, is "all-to-all-w" the official name of the operation to use in text? > > Chapter 8: > > Page 335 lines 28-29: Are these constants required to be macros (rather than const global variables), and do they need to be defined exactly as shown? Otherwise, someone could write "#define MPI_VERSION 1+2" or (in C++) "static const int MPI_VERSION = 3;". > > Page 336 line 46: "value" should be "values". > > Page 337 line 8: What is "myrank"? > > Page 337 line 12: "implementation specific" should be "implementation-specific". > > Page 337 line 23: There should be a comma before "inclusive". > > Page 337 line 41: There should not be a comma before "I/O". > > Page 341 line 26: "pointer" should be "pointers". > > Page 341 line 48: This text should end with a period, and be on the same page as the code. > > Page 342: What is the return value of an error handler used for? Is it meaningful at all? Is it guaranteed that, if an MPI function returns at all after an error, its return value is the same (implementation-defined) error code that was passed to the handler? > > Page 342 lines 35-36: "Thus, if the user chooses not to control error handling, every error that MPI handles is treated as fatal." is not true for I/O. > > Page 342 lines 40-41: "non trivial" should be "non-trivial". > > Page 343 line 29: "implementation" should be "implementations". > > Page 344 line 16: "stdargs" is not a standard term. > > Page 349 lines 44-45: Line 45 appears to be redundant with the last part of line 44. > > Page 357 line 6: "high-resolution" should be "high resolution". Also, this text might want to repeat the wording from Annex A that MPI_{Wtime,Wtick} can be macros and/or not go through the profiling interface. > > Page 357 line 18: The discussion of MPI_WTIME_IS_GLOBAL should refer to the section where it is defined. > > Page 357 lines 34-: MPI_INIT_THREAD is not mentioned here; this seems like the logical place to describe it. > > Page 359 line 38: There is extra space in the middle of the line. > > Page 359 line 45: There should be a comma after "MPI_REQUEST_FREE". > > Page 359 line 48: "operations" should be "operation". > > Page 360 lines 1-2: Are there any requirements that particular objects be freed before MPI_FINALIZE? > > Page 361 line 31: Who/what does "they" refer to? > > Page 362: Both functions on this page have extra whitespace in their abstract signatures. > > Page 362 line 41: There should be a comma after "accepted". > > Page 363 lines 1 and 7: "errorcode" should be in sans-serif. > _______________________________________________ > mpi-comments mailing list > mpi-comments(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-comments

1 0

Comments on MPI 3.0 draft 2 [part 2]
by Jeremiah Willcock 19 Aug '12

19 Aug '12

These are in reference to the draft at <URL:http://meetings.mpi-forum.org/mpi3.0_draft_2.pdf>. Thank you for considering them. -- Jeremiah Willcock General: [post 3.0] Overall, the document reads like a tutorial rather than a reference document or standard (for example, going from simpler topics to more advanced ones, forward references for datatypes and communicators in the point-to-point chapter, ignoring MPI_PROC_NULL when talking about point-to-point operations then introducing it later in the chapter, and the wording on page 225 lines 13-16). It would be nice to know, for each function, constant, etc., which MPI version introduced it. This would allow developers to write MPI-3.0-compliant programs that would also work on older MPI versions. Chapter 5: Why are nonblocking collectives in their own section rather than mixed in with the normal collectives (i.e., put each collective in both blocking and nonblocking versions together)? Page 141: MPI_REDUCE_LOCAL is not mentioned here at all (it isn't a communication operation, but it is part of the contents of this chapter). Page 141 lines 33 and 36: I don't think "gather" and "scatter" need to be capitalized. Page 141 lines 40-43: The reduction paragraph here looks very different from the others (has the ALL* version first, groups them together with single-root reduce as a variant version, and points to a subsection for allreduce while pointing to its whole containing section for reduce). Given the pattern of the other sections, allreduce and reduce should be in separate sections, with reduce first followed by allreduce. Reduce_scatter_block and reduce_scatter should be in the same subsection for the nonblocking versions, just like allgather and allgatherv (for example) are. Page 142 lines 1-3: It's odd to point to section 5.11 then subsection 5.11.2; these should probably be combined. Page 144 line 9: What are the "auxiliary functions" referring to? Do they require any access to the communication library outside that provided by MPI point-to-point operations? Page 144 lines 25-27: Are intercommunicator and intracommunicator hyphenated or not? They are used both ways in these three lines. Chapter 6 uses hyphenated versions. Page 145 line 45: The list here appears to be every collective from the list starting at line 9, except for the scan collectives. It might be easier to just say that. [post 3.0] Why does bcast not have separate send and receive buffers (with an MPI_IN_PLACE option) like the other one-to-all collectives? Page 148 line 33: What does "itself included" mean here? The root does not do any writing to its buffer, so that text seems to be meaningless, and conflicts with the "other" on line 35. Page 148 line 33: "is" should be "must be". Page 149 line 6: The sentence should be singular since there is only one example. Page 160 lines 19-21: It seems odd to have an unnecessary restriction just for symmetry; why not have the call be as general as possible? Given that multiple writes are now allowed using MPI_Put with undefined results, might it make sense to apply the same rule to multiple MPI_Recvs and collective recvbufs that write the same location multiple times? Page 162 lines 10-11: This restriction is unnecessary as well. Page 162 line 42: It might be interesting to have the same example written using MPI_Scatter (no v) and a datatype with an adjusted extent, or at least not claim that MPI_Scatterv is required. Page 165 lines 40-43: This description is incorrect when MPI_IN_PLACE is used, I think. This comment also applies to page 167 lines 15-20, page 168 lines 42-48, page 171 lines 5-10, and page 173 lines 13-18. Page 165 line 45: There is an extra space after "0" in this line. Can MPI_BOTTOM be used as the buffer location for collectives when count=0 for the corresponding send or receive (depending on the collective)? Can NULL be used, or some random invalid pointer? That might be worth stating explicitly. In v and w versions of collectives, if the count is 0 for some entry of the array of counts, is the displacement for that entry guaranteed to be ignored (and thus doesn't need to point to valid memory)? For rooted collectives, is root allowed to be MPI_PROC_NULL? It is probably not, but this should probably be stated explicitly. Page 170 lines 43-45: Not only can the displacements be different, but the sizes can be as well. Page 171 lines 14-15, 20-21: What is the reason for MPI_IN_PLACE needing to be given on all processes? Page 171 lines 32-41: This text seems like it should be after the description of MPI_Alltoallw, since most of it applies to all three functions. Page 171 lines 34-35: The restriction to sequential storage does not apply to MPI_Alltoallv (unless that's meant to say that they need to be offsets in the same buffer, unless MPI_Alltoallw). Page 172: As I commented yesterday, MPI_Alltoallw should have a version taking arrays of MPI_Aint for sdispls and rdispls to allow the use of MPI_BOTTOM for sendbuf and recvbuf. Note that MPI_Neighbor_alltoallw does use MPI_Aint (probably since it does not need to be backwards-compatible); if a different name is chosen for MPI_Alltoallw with MPI_Aint, MPI_Neighbor_alltoallw might need to be renamed to match it. Page 173 line 35: MPI_SCATTERW should be in sans-serif font rather than code, I think; BTW, why aren't there w versions of all of the v collectives? Page 173 line 40: The first "and" on this line should be "or". Page 174 lines 36-37: "count" should have a comma after it on both lines. Page 174 line 39: "root" should have a comma after it. Page 174 line 45: Remove the comma here. Page 175 lines 28-29: This conflicts with page 174 line 38. Page 176: Are the logical reduction ops allowed to return any nonzero value for "true"? The definition of Boolean value earlier in the standard says that the test for bool arguments is zero/nonzero; are reduction ops given that flexibility in their outputs as well? Page 177 line 29: Why are logical operations not allowed on Fortran integer types? Page 178: Why are the functions named PAR_BLAS* and not something like PAR_DOT*? Page 178 line 20: Does "array" mean "matrix" here? Page 179 lines 47-48: It might be worth mentioning that MPI_MAXLOC does not compute a lexicographic maximum. Page 180: Would it be possible to generalize the *loc routines to allow wider integer types as the second element? Page 180 line 39: This computation ignores alignment issues (in particular, if int is larger than float). Page 182: Example 5.19 breaks if the total data size is greater than INT_MAX, which is increasingly common on large systems, so it can't be used as the basis of scalable code. Page 183 line 40: There should be a comma after "len". [post 3.0] Reduction operations should allow user-defined ops that change the data size (this is important for bindings to higher-level languages such as C++). Page 184 line 15: There should be more space around the circle operator (like on line 20). There are extra spaces after "0" and the ellipsis. Page 184 lines 36-45: What does this text mean? There are functions in MPI to decode datatypes. Page 184 line 48: "convention" should be "conventions". Page 185 line 4: "naive" should be "na\"\i{}ve". Page 185 line 41: Are user-defined ops required to be freed at some point? What is the reason for having MPI_Op_free? Page 186 line 9: "it is time" sounds very informal. Page 187 line 31: How "identical" do the results need to be? Reasonable algorithms may output non-bit-identical results on different ranks for not-truly-associative floating-point operations. Page 187 lines 28-: What are the requirements about argument consistency between processes for MPI_Allreduce? Page 188 line 28: "array" should probably be "matrix". Page 189 line 6: This sentence should be singular. Page 189 lines 11 and 44, page 190 line 21, page 192 line 1: There is an extra space after the left parenthesis on each of these lines. Page 191 line 7: There should be a comma after "op". Page 191: I think "n", and maybe "i", here should be in sans-serif font. Page 192 lines 32-39: "n" and "i" are in different fonts in different places in this paragraph. Page 192 line 34: There should be a comma after "count". Page 192 line 39: There should be a comma after "recvcounts[i]". Page 193 line 46: It seems like the code here should be in sans-serif font, plus with spaces after the commas near the end of the line. Page 193 lines 47-48: The variable names here are in the wrong font, and there should be a comma after "op" on line 48; both of these issues also apply to page 194 lines 33-34. Page 194 lines 38-39: Are the contents of recvbuf completely undefined, or are they just left unmodified from what they were before MPI_Exscan was called? Page 195 line 3 says that MPI_IN_PLACE means that recvbuf is left unchanged; is that true even without MPI_IN_PLACE? Page 194 lines 39-40: It is unnecessary to special-case rank 1 here. Page 196: The spacing inside the parentheses of the function calls in the code here is inconsistent. Page 197 lines 15-16: Does "passed to a completion call" mean "completed" in the sense of point-to-point (i.e., not just having MPI_Test called once, but calling a sequence of test and wait operations that end up moving the request to the completed, inactive state)? Page 197 lines 31-32: Why do nonblocking collectives fill in MPI_ERROR rather than returning an error from the completion call like nonblocking point-to-point calls do? Page 197 line 46: "MPI exception" should refer to the definition in section 8.3. Page 198 lines 1-7: Which processes need to complete (in the MPI_Wait, etc. sense) a nonblocking collective for it to finish executing (as in, allow MPI_Wait to return) on other processes? Page 198 lines 31-32: What does it mean that the progression rules are just "similar"? Do they differ in some way from the point-to-point rules? Page 198 line 41: There is an extra space after "comm". Section 5.12 (in general): The argument name "request" in the annotated argument list for each function does not line up with the other argument names above it. Page 216 lines 1-10: Would this example be safe if an ibarrier but blocking point-to-point calls were used? Page 217 lines 24-27: This rule should be a lot earlier; the previous pages seem to imply that you can pipeline nonblocking collectives on the same communicator as long as you start them in the same order on all ranks. Example 5.29 violates this rule as well, and example 5.34 explicitly states the opposite of page 217. Page 220 line 20: "completed" should be "complete". Page 220 line 46: The example header should be on the same page as the text. [post 3.0] Dynamic sparse data exchange (http://www.unixer.de/publications/img/hoefler-dsde-protocols.pdf) in some form might be worthwhile to add as a collective in the future, at least for the case where all data is sent at once using an alltoallv-like call but the receiver does not know the data quantity from each sender up front. Something like http://www.tu-chemnitz.de/informatik/PI/forschung/pub/download/HR_europvmmp… would also be very useful. Chapter 6: Page 224 line 20: "on a par" should probably be "on par". Page 229 line 1: There is an extra space before the left parenthesis. Page 231 line 45: "first" should be "group1". Page 231 line 48: Add "the" before "first group". Page 233 line 13: There should be a comma after the ellipsis. Page 233 line 16: Does that mean newgroup == group (as handles), or just that the groups themselves are MPI_IDENT? Page 233 line 39: "consist" should be "consists". Page 235 line 44: There is extra space before "comm". Page 236 lines 11 and 40: The comma is unnecessary here. Page 236 lines 14-15: This conflicts with the wording on page 227 lines 44-45. Page 237 line 7: There should be a space before the second "MPI_Comm". Page 237 line 42: The terms "group A" and "group B" are used for "left" and "right" (in some order) in chapter 5; consistent terms should be used. Page 238 line 15: "duplicates" should be lowercase. Page 238 line 23: Remove "call". Page 238 line 26: Does that mean that pending collective communications are not allowed? Pages 238-239: MPI_COMM_IDUP and MPI_COMM_DUP_WITH_INFO are in the opposite order when being introduced at the beginning of this section. Page 240 line 4: "group" should be lowercase. Page 240 line 20: Add "MPI_COMM_CREATE" after "call". Page 240 line 24: Add a comma after "otherwise", or merge this sentence with the previous one. Page 240 line 44: "sometimes to avoid" sounds awkward. Page 241 line 3: Add a comma after "MPI_COMM_CREATE". Page 241 lines 4-5: "further to subdivide" should be "to further subdivide". Page 241 line 17: "should" should be "must". Page 241 lines 21-28: What ends up as the remote group of the new inter-communicator? Page 242 line 22: "create" should be "creation". Page 243 line 44: "comm" is in an incorrect font. Page 244 line 30: Add "or MPI_UNDEFINED" to the end of this sentence. Page 245 line 10: "doesn't" should be "does not". Page 245 lines 30-48 and page 247 lines 2-16: The spaces in the function calls here are inconsistent with most other examples in the specification. Page 250 lines 17-: The style used for examples here is different from in the rest of the specification. What are these examples (especially the first few that do not do anything "interesting" with groups or communicators) trying to show? Page 256 line 42-page 257 line 6: "backmasking" is spelled three different ways in these two paragraphs. Page 257 lines 42-43: Doesn't MPI_COMM_IDUP work on intercommunicators? Why are there not non-blocking versions of the other {intra,inter}-communicator constructors? Lack of interest/need? Page 259 line 11: What is an "additional safe context"? Page 260 line 1: "and" should be "or". Page 260 line 4: The colon probably should be a period since the list does not immediately follow. Page 261 lines 4-6: "Dual membership" seems to be describing a non-standard behavior that would likely cause problems for many inter-communicator collectives (especially with MPI_IN_PLACE). Page 262 line 44: There should be a comma after "MPI_COMM_FREE". Page 265 line 45: Add a comma after "window". Page 266 line 12: "attributing" should be "attribute". Page 270 line 22: The last comma in this line should be a semicolon. Page 272 line 44: What does "new" refer to here? Page 279 lines 29-30: "The last three" should be "the last four", but it seems odd that a communicator with an invalid keyval associated with it would ever get to the point that it could be copied. Page 279 lines 34-35: What does this sentence mean? Page 283 line 9: Should "got" be "gotten" or "retrieved"? Page 285 line 37: The term "loosely synchronous model" is not used anywhere else that I can tell except in the context of one-sided operations. It seems like it's some property of MPI-based libraries, but this is not explained. If this topic is going to be discussed, it should also be mentioned in the context of library routines that are non-blocking internally (and maybe suggest the use of generalized requests for that), as well as why MPI_COMM_IDUP is necessary for some libraries. Page 287 line 1: "well-ordering" should probably be "non-overtaking" to match standard MPI terminology. Page 287 line 15: "then" should be "that". Chapter 7: Page 290 line 2: There should not be a comma before "process-naming". Page 291 line 11: There should be a comma after "therefore". Page 292: Which arguments must be consistent across all processes? Page 292: Is the created topology when reorder=false row-major or column-major? (The distinction does not matter when reorder=true). Page 293: Is MPI_DIMS_CREATE a local call? Page 294 line 37: There should be a comma after "index". Page 297 lines 46-47 and page 300 lines 2-4: Why not allow MPI_UNWEIGHTED as a special case for some or all isolated nodes (even if it is not passed on non-isolated nodes)? That would avoid the issues discussed on page 298 lines 6-16 and page 300 lines 9-19, and make this call more consistent with others that allow NULL for empty arrays. In particular, having MPI_UNWEIGHTED be possibly NULL and yet forbidden for degree-zero nodes prevents using something like an std::vector that could be empty as a source for weights (since its data pointer might be NULL when it is empty). Page 302 line 48, page 303 line 26: The variable names should be sans-serif. Page 307 line 2: The variable names should be sans-serif. Page 307 line 3: "array" should be "arrays". Page 312: Which arguments must be consistent across processes? Is this function collective? Page 312 line 46: "capability" should be "capabilities". Pages 313-314: Are the *_MAP functions local or collective? If they are collective, which arguments must be consistent? Pages 314-330: Why are these functions not in chapter 5 like the other collectives? If the issue is that they apply to topologies only, you can generalize them by assuming a non-topology-based communicator is a one-dimensional ring or similar. Page 320 lines 42: This line ends abruptly. Page 321 lines 3-21: "indegree" and "outdegree" should be in sans-serif font when referring to arguments. Page 324 line 28: This line ends abruptly. Page 331 lines 29-30: The results of these calls are only used in commented-out code on page 332. Page 333: This code seems like it would be clearer if it used MPI_Neighbor_alltoallv and removed the variable sizeofreal. Also, is "all-to-all-w" the official name of the operation to use in text? Chapter 8: Page 335 lines 28-29: Are these constants required to be macros (rather than const global variables), and do they need to be defined exactly as shown? Otherwise, someone could write "#define MPI_VERSION 1+2" or (in C++) "static const int MPI_VERSION = 3;". Page 336 line 46: "value" should be "values". Page 337 line 8: What is "myrank"? Page 337 line 12: "implementation specific" should be "implementation-specific". Page 337 line 23: There should be a comma before "inclusive". Page 337 line 41: There should not be a comma before "I/O". Page 341 line 26: "pointer" should be "pointers". Page 341 line 48: This text should end with a period, and be on the same page as the code. Page 342: What is the return value of an error handler used for? Is it meaningful at all? Is it guaranteed that, if an MPI function returns at all after an error, its return value is the same (implementation-defined) error code that was passed to the handler? Page 342 lines 35-36: "Thus, if the user chooses not to control error handling, every error that MPI handles is treated as fatal." is not true for I/O. Page 342 lines 40-41: "non trivial" should be "non-trivial". Page 343 line 29: "implementation" should be "implementations". Page 344 line 16: "stdargs" is not a standard term. Page 349 lines 44-45: Line 45 appears to be redundant with the last part of line 44. Page 357 line 6: "high-resolution" should be "high resolution". Also, this text might want to repeat the wording from Annex A that MPI_{Wtime,Wtick} can be macros and/or not go through the profiling interface. Page 357 line 18: The discussion of MPI_WTIME_IS_GLOBAL should refer to the section where it is defined. Page 357 lines 34-: MPI_INIT_THREAD is not mentioned here; this seems like the logical place to describe it. Page 359 line 38: There is extra space in the middle of the line. Page 359 line 45: There should be a comma after "MPI_REQUEST_FREE". Page 359 line 48: "operations" should be "operation". Page 360 lines 1-2: Are there any requirements that particular objects be freed before MPI_FINALIZE? Page 361 line 31: Who/what does "they" refer to? Page 362: Both functions on this page have extra whitespace in their abstract signatures. Page 362 line 41: There should be a comma after "accepted". Page 363 lines 1 and 7: "errorcode" should be in sans-serif.

1 0

Re: [Mpi-comments] Comments on MPI 3.0 draft 2
by Rolf Rabenseifner 19 Aug '12

19 Aug '12

Dear Jeremiah Willcock, thank you very much for your detailed comments, especially in the chapters for which I'm responsible or co-auther, i.e., 2. Terms, 17. Fortran and Interop (Jeff together with Craig and me), Annex A Language Binding Summary, and Annex B (Change-log), and partially the Index. The typos will be corrected. I'll come back with answers for your questions in "my" chapters. Best regards Rolf ----- Original Message ----- > From: "Jeremiah Willcock" <jewillco(a)osl.iu.edu> > To: mpi-comments(a)mpi-forum.org > Sent: Sunday, August 19, 2012 2:13:41 AM > Subject: [Mpi-comments] Comments on MPI 3.0 draft 2 > > These are in reference to the draft at > <URL:http://meetings.mpi-forum.org/mpi3.0_draft_2.pdf>. Thank you for > considering them. > > -- Jeremiah Willcock > ... -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

Comments on MPI 3.0 draft 2
by Jeremiah Willcock 18 Aug '12

18 Aug '12

These are in reference to the draft at <URL:http://meetings.mpi-forum.org/mpi3.0_draft_2.pdf>. Thank you for considering them. -- Jeremiah Willcock General: The line numbers do not match the lines of text; my comments are best estimates of which line each line of text is on. [post 3.0] It might be worthwhile to create a document of "performance expectations," basing it on papers such as <URL:http://www.mcs.anl.gov/~thakur/papers/qualityofmpi.pdf>, <URL:http://www.unixer.de/publications/img/datatype-performance.pdf>, and <URL:http://www.mcs.anl.gov/uploads/cels/papers/2008/P1500.pdf>. This may or may not be normative, but maybe you can have a defined "self-consistently performing" status for MPI implementations that meet the standard and satisfy these extra requirements. [cosmetic] In many code samples, non-matching curly quotes are used and look unusual. A solution for that is at <URL:http://stackoverflow.com/questions/1662037/how-to-write-programming-cod…>; it may be easy to apply globally without needing to change each individual quotation mark. Front matter: The change log starting on page iii does not have a title. How much of that material is redundant with sections 1.2-1.6? On page iii line 18, there should be a comma after "Operations". On page iv lines 6-8, that text seems very obsolete (not correct for the current standard). On page xix line 11, part of the table title is black, while the rest are blue. On page xxviii line 20, "award" should be "awards" and the numbers do not usually have number signs before them. The same issue appears on page xxx lines 36-37. On page xxviii line 29, "frontmatter" should be "front matter". Chapter 1: On page 1 line 24, the comma should be after "which" instead of before. Also, "methods" being in that list may no longer be relevant. On page 1 line 44, "coprocessor" should be "coprocessors". On page 3 line 16, there should be a comma after "e.g.". On page 3 line 46, the comma should be before the closing quotation mark. On page 5 line 35, "Info" should be lowercase. Chapter 2: On page 9 line 16, why does the MPI 3.0 document still compare itself to MPI 1? On page 9 line 18, "vs" needs a period. On page 10 line 34, there should be a comma after "OUT". On page 11 line 30, there should be a comma after "i.e.". On page 12 line 9, there should not be a comma after "portable". On page 13 line 43, there should not be a comma after "C". On page 15, should the MPI version constants be in the first list (of compile-time constants)? On page 18, the font in the table is smaller than the rest of the text or other tables in the standard. On page 19 line 14, the restriction on use of MPI_* names should also apply to preprocessor macros, namespaces in C++, types, and other things. On page 19 line 44, "executing their own code" might be better as "each executing its own code". Is this text trying to say that MPI is not a SIMD model? That is not clear from those two sentences (that the executables may or may not be identical, but the processes always have different control flow). Chapter 3: The top of page 24 refers to "parameters" to mean what chapter 1 said would be referred to as "arguments" to avoid confusion with Fortran. On page 25 lines 36-38, there need to be commas after "MPI_REAL4", "MPI_INTEGER1", and "MPI_INTEGER2". On page 27 line 43, there should not be a space after the ellipsis. On page 29 lines 35-36, the list of valid values omits "MPI_PROC_NULL" (so does the corresponding list for sends). On page 31 line 32, should "receive" be "send"? On page 31 lines 38-39, it might be good to have a similar thread safety warning about probe and iprobe that is on line 33 about other message passing libraries. On page 32 line 9, there should be a comma after "comm". On page 33 line 22, the last comma should be a colon. On page 33 line 48, the example header should be on the same page as the code. On page 34 line 32, "bound" should be "bounds". On page 36 line 3, "logical or" should be "logical, and". Is the buffer required to be a valid pointer (not NULL, for example) when using count=0 for sends and/or receives? On page 40 line 39, did you want the "should" to be "must", at least under MPI_THREAD_MULTIPLE? Or is that too strong of a constraint on user-mode threading implementations? On page 44 line 43, "C++" should be removed. On page 44 line 45, the comma should be before the closing single quote. On page 46 line 1, doesn't MPI_BSEND_OVERHEAD come into play here as well? This paragraph suggests that MPI must be able to use the full user buffer, while that is contradicted by page 46 line 40. Page 47 lines 46-48 seem unclear to me -- is that paragraph an "advice to users" that says that an implementation might delay completing standard sends until the corresponding receives are posted but it isn't required to? The "may not" wording can also be read to mean that the send is not allowed to complete until the receive is finished, which is not what was intended. On page 56 line 30, the text seems to require background progress (sends progressing even if MPI is not called). I know that has been a disputed issue in the past, but is that the desired behavior? On page 57 line 11 and further on, when count = 0 for a multiple completion operation, are the arrays passed in guaranteed to not be touched by the MPI implementation (and thus they can be NULL)? I think this was implied before, but isn't explicit. On page 57 line 46, there is an extra space after "request". On page 60 line 31, it might be good to put "in the same manner" after "handled". On page 62 lines 30-34 (and example 3.16 on page 63), doesn't that text contradict the discussion on the mailing list about fairness issues (and the lack of fairness guarantees in MPI)? Also, that text looks like it should be under MPI_WAITSOME, not MPI_TESTSOME. Are the handles passed in to the multiple-completion routines required to be distinct? I.e., can I have the same active handle at two different positions in the input array, and if so, what happens? Does MPI_*PROBE have the same non-overtaking rules as receives (in terms of matching the first message sent of those that could match the probe operation)? Page 67 lines 27-38 says that (although possibly not normatively since it's in "advice to implementors"), but the non-overtaking section doesn't. Does that rule apply to MPI_IMPROBE as well (if I do two MPI_IMPROBES that each match two messages from the same sender, the order the probes get the messages must be the same as the send order even if I don't do MPI_MRECV before doing the next improbe)? How do those rules interact with MPI_CANCEL on MPI_IMRECV-created requests? Is there a reason that there are not non-blocking versions of MPI_SENDRECV and MPI_SENDRECV_REPLACE? The second one, in particular, is not trivial to emulate using only a single isend/irecv pair. Chapter 4: On page 83 line 23, "communication" should be "communications". On page 83 line 34, "shape and size" should be "shapes and sizes". On page 84 line 42, "sizeof" should be inside \mathit. On page 84 line 45, where does "i" come from? Is the extent rounded to a multiple of k_i, or is the upper bound rounded (these differ when lb is not aligned)? If extent is rounded, the +eps term should be moved to the computation of extent, not ub, unless that definition of ub is used somewhere else. On page 95, why does MPI_TYPE_CREATE_SUBARRAY use zero-based indexing for Fortran; I believe many other places in MPI use one-based array indices when called from Fortran, even if they use zero-based indices from C. Also, is there a version of MPI_TYPE_CREATE_SUBARRAY that allows strided indexing in the dimensions (MPI_TYPE_CREATE_DARRAY looks more limited)? On page 103 line 18, MPI_ADDRESS has now been removed, not just deprecated. Is offsetof a valid way to get displacements within C structs to use with MPI (rather than creating an object of that type and calling MPI_GET_ADDRESS on it and its individual fields)? Wouldn't MPI_TYPE_SIZE and a way to "flatten" an MPI datatype into a datatype with the same primitive elements but sequential in memory (or using MPI_PACKED) be better than MPI_TYPE_TRUE_EXTENT for the example on page 108 line 29? On page 110 line 24, there should be a comma after "e.g.". On page 111 line 42, there is some odd space at the end of the line. On page 112 line 7, there shouldn't be a space after "datatype". On page 112 lines 11-14, should there be an MPI_TYPE_FREE added as well? On page 115 line 44, it seems odd that adding MPI_BOTTOM to a valid address gives a valid address, since that allows me to create a valid address MPI_BOTTOM * n + v where n is nonnegative and v is a valid address. On page 128 line 32, example 4.18 uses "class" as a field name, which makes it impossible to compile with a C++ compiler. Chapter 5: If there are going to be MPI_Count versions of datatype functions, would it at least be possible to have a version of MPI_Alltoallw that uses MPI_Aint for displacements? It seems like that was an oversight before, since byte offsets from MPI_BOTTOM are intended as arbitrary addresses. Chapter 12: Did the extended generalized request proposal from "Extending the MPI-2 Generalized Request Interface" by Latham et al not get accepted into MPI 3.0? Chapter 16: On page 597 lines 20-21, "language neutral" should be hyphenated. On page 597, line 43, "ion" should be "on". On page 598 line 22, "prototyes" should be "prototypes" in the subsection title. On page 598, tables 16.2 and 16.3 still list C++ bindings. Chapter 17: It may be too late to change this, but "typeclass" is an unfortunate name to use in MPI since it refers to a specific language feature in Haskell and some other functional languages. On page 600 line 18, "INCLUDE mpif.h" is missing the single quotes that are used in other places. On page 600 line 27, do you want to still mention "C/C++ header files" now that the C++ bindings have been removed? On page 600 line 47, do you want to require Fortran 2008 support for all MPI implementations that have any Fortran support at all? What does that mean on platforms that have older Fortran compilers but not Fortran 2008 support at the compiler level? On page 601 line 32, should "feature" be "features"? On page 603 line 2, the period should be before the closing double quote (in American English). On page 604 lines 14-15, should the single quotes be double quotes? On page 604 line 18, "constaints" should be "constraints". On page 604 line 36, "With MPI_SUBARRAYS_SUPPORTED==.TRUE." might be better as "When MPI_SUBARRAYS_SUPPORTED is defined as .TRUE.". On page 605 lines 21-23, the italicization of the quotation seems incorrect (although I don't know what the source of the quote had italicized). On page 606 lines 16 and 18, "wrong" might read better as "incorrect". On page 606 line 41, the equation should be in math mode. On page 607 line 29 and on page 608 line 6, "BIND(C)" should be in the code font. On page 608 line 19, "and" should be removed. On page 608 lines 24 and 25, "MPI" and "PMPI" use different fonts; this may be a typo. On page 608 line 35, "achieved" should be "available". On page 608 line 38, "implement" should be "implementation". On page 609 lines 1-7, the advice to implementors is hard to read (the first sentence in particular is very long). On page 609 line 10, "notifying" should be "indicating". On page 609 line 33, "MPI_Address" is one of the functions marked as removed in MPI 3.0, so it should not be listed there. On page 610 line 36, should the linker names be in the code font? On page 611 line 3, "non optional" should be "non-optional". On page 611 lines 9-10, "compiler" should be "compilers", and the "the" before "C" can be removed. On page 611 lines 16-17, the formatting is strange (extra spaces between names). On page 611 line 44, "b" should be "by". On page 611 line 48, "MPI_Address" has been removed in MPI 3.0. On page 612 line 14, the hyphen should be an en dash (denoting a range). On page 612 lines 37 and 42, "S1", "S2", and "S3" are not in sans serif font like they are elsewhere on the page. On page 613 line 30, the hyphen between the MPI version numbers should be an en dash. Also, why is the MPI 3.0 standard discussing implementing older versions? On page 614 lines 14-15, isn't the second line redundant with the first one? On page 615 line 29, the single quotes should be double quotes. On page 616 line 14, "IO" should be "I/O". On page 616 line 45, the subsection title should not have a hyphen before "synchronization". On page 617 lines 13-14, why not just say that the routine must be written in an implementation-specific way that achieves the listed behavior, rather than specifying a particular requirement about the compiler's knowledge of the routine's body? On page 617 lines 31-32, the single quotes should be double quotes. On page 618 line 2, there should be a comma after "LOGICAL". On page 618 line 8, there should be a comma after "COMPLEX". On page 618 lines 21-22, should the section title referred to be in quotes? On page 619 line 33 and other places, should "external32" be in the code font? On page 621 line 37, there should be commas after "Double" and "8". On page 622 line 14, there should be a comma after "MPI_UNPACK_EXTERNAL". On page 624 line 3, "sizeof" is built in to the language as syntax; it is not a function. On page 626 line 13, there is extra space after "90". On page 626 line 40, the hyphen between the MPI versions should be an en dash. On page 627 line 4, should "KIND" be in the code font? On page 627 line 30, "mpif.h" may need to be in single quotes. On page 627 line 39, "module" should be "modules". On page 628 lines 34-42, does that mean that users are allowed to modify send buffers during Fortran nonblocking communications (since the implementation is required to act as if it made a copy of the user data when the operation began)? The "Displacements in" sentence on lines 38-39 is also confusing; does that mean that datatypes need to be created using different displacements in Fortran than in C (even if they are structurally identical)? This reading conflicts with page 653, but is reasonable from the text on 628. On page 629 line 2, "comminication" should be "communication". On page 629 line 21, there should not be a space after the ellipsis. On page 629 lines 44 and 45, the single quotes should be double quotes. On page 632 line 27, should the comment be indented to match the one two lines later? On page 632 line 42, "extensions" should be "extension". On page 633 line 9, there should be a comma before "e.g.". On page 633 line 12, "as it may fit" sounds odd. On page 633 lines 47 and 48, the section titles should be in the normal text font. On page 634 line 17, the subsection title has extra space after "and". On page 634 line 20, "block" should be in the text font. On page 634 line 41, "be" should be "have been". On page 635 line 7, there should be spaces after the second two "copy"s. On page 635 line 9, should "send" be "CALL send"? On page 635 lines 13 and 16, there should not be spaces around the slashes. On page 635 line 26, there should be a comma after "MPI_RECV". On page 636 line 36, because of interprocedural optimization, it may not be possible for a user routine to have the same behavior as MPI_F_SYNC_REG without resorting to compiler-specific techniques, and thus it may not be good to recommend doing. The same thing applies to page 640 lines 15-36 and page 642 line 48. On page 637 line 39, should the "ASYNCHRONOUS" in the title be in the code font? On page 639 line 47, the headings should be moved to the same page as the code. On page 640 line 33, "posibilities" should be "possibilities". On page 643 line 38, "Automatic garbage collection implementation" should be "An implementation with automatic garbage collection". On page 644 line 5, "&" should be in the code font. On page 647 line 4, remove "applications" from before "library". On page 648 line 22, is there some reason that the conversion functions still take integers on the Fortran side and not the handle types? Is that just a compatibility issue? On page 648 line 26, "int" should be in the code font. On page 648 lines 38-39, shouldn't passing in MPI_COMM_NULL in Fortran return the C value of MPI_COMM_NULL (and never something else), and the behavior be unspecified for passing in an invalid communicator? The same applies to lines 43-44. Do other predefined communicators need to map to the C versions of their predefined values (MPI_COMM_SELF -> MPI_COMM_SELF, not something else equivalent to it)? On page 655 line 3, "stdargs" is not a standard term; use "variable-length argument list" instead. On page 655 line 21, the sentence there does not end with a period (the one on "etc." does not count). On page 655, "address valued" and "integer valued" should have hyphens when used as adjectives. On page 656 line 9, "identical" should be "identically". On page 657 line 48, the header of the example should be on the same page as the code. On page 658 line 47, "val" should be either sans-serif or in the code font. Some other things in that paragraph might need to be changed as well. On page 659 line 22, "extra state" should be hyphenated in the subsection title if it is in the text. On page 661 line 16, "int" should be in the code font. Annex A: On page 674 lines 41-47, don't those constants have particular, standard-specified values (not just arbitrary definitions)? On page 676 lines 10-15, are those values required to be defined as macros that expand to integral values? Thus, they can't have "const int" or "unnamed enum" types, but must be macros. On page 678 line 48 and some places on 679, "module" should be "modules". On page 685, what are the language bindings for the info keys and values supposed to be? The C prototypes for functions (throughout the standard, but especially here) don't end with semicolons like prototypes actually do in C. On page 693 lines 45-46, shouldn't MPI_COMM_DUP_FN (and the others in A.2.4 that are just signatures but not defined by MPI) be moved to section A.1.3 with the other user-defined function prototypes? Annex B: On page 789 line 28, "errata" is plural, so it should probably be "erratum". There seem to be other places in Annex B with this error as well. On page 789 line 39, the opening quotation mark is backwards. Section B.1 mentions various C++ bindings to things as having been corrected in MPI 3.0, while actually all of those bindings were removed (see page 790 lines 5-6 for an example). On page 790 line 7, the second comma should be inside the quotation marks. On page 792 line 40, "one-sided" should be lowercase. On page 793 line 7, there should be a comma after "==". On page 794 line 17, "a with" should be "with a". On page 794 line 26, "solution" should be "solutions". Index: Page 811 lists now-removed C++ bindings. The function index starting on page 818 sometimes lists the same function twice on the same page (see MPI_ALLTOALLV for an example with page 170). On page 824, should individual PMPI functions be on the function list? Should mpiexec and mpirun?

1 0