Hi Jeremiah, I am not a chapter author but a member of the RMA working group. Thanks for your very thorough comments. I will comment on the few non-trivial ones that I am qualified to discuss. I'm sure others will comment once they've read your exhausted list and cross-checked with the standard. Best, Jeff
Why are the request-based one-sided functions named MPI_R* rather than MPI_I* like nonblocking point-to-point and collective functions?
The "I" in MPI_Isend is in reference to these functions returning "immediately", not their use of requests. MPI RMA communications operations are "nonblocking" (there was some discussion about when they could block, but I believe that is only in lock/unlock; I could be wrong) and therefore return "immediately" whether or not they involve requests, so the "immediate" name attribute is not appropriate as a distinguishing feature of the request-based operations. In any case, the MPI standard does not seek to come up with perfect names. There are plenty of functions that have unfortunate names. Users are always welcome to define their own wrappers to MPI with names they like :-)
There are no request-based versions of fetch-and-op or compare-and-swap.
There was no use-case for this. I had to fight pretty hard to get request-based operations into the RMA chapter and I did so by pointing to a very reasonable use-case. If you have a use-case for request-based RMW operations, then I'm sure the Forum would be happy to consider them for MPI-Next. There's nothing stopping us from defining these operations.
Page 414 line 25: The "should" might need to be "must" to avoid issues similar to <URL:https://upc-bugs.lbl.gov/bugzilla/show_bug.cgi?id=495>.
I completely agree with the general point that the word "should" is useless. It is not binding, which is why legal documents use "shall" instead. I don't know if "must" has a legal meaning. The MPI standard is, of course, not a legal document, but it never hurts to use language in such a way that would stand up in court :-)
Page 433: Is it possible/likely that an rget does not complete locally until after a flush, fence, or similar call? That might be worth mentioning if true.
I am reading this as "rget does not complete locally until a flush, fence, or similar call" since it is not clear to me what you mean by "after". I assume the meaning is "at" instead. MPI requires the user to query completion explicitly before the local buffer can be queried in a meaningful way (there is an exception for MPI_Isend and read-only access, but that's not relevant here). Polling on bytes is E.V.I.L. and not standard-compliant. Perhaps I have misunderstood your comment.
Page 487: I think MPI_INIT_THREAD should be moved to be near MPI_INIT in the standard, since many high-performance applications use threads and any threading at all (even OpenMP in non-MPI-using regions) is forbidden when MPI_INIT is used.
I agree with this 100%. The reason things are the way they are is presumably historical. I find it really obnoxious to have these two functions in different chapters. I'd love to see them relocated in MPI-Next.
[post 3.0] In the future, it might make sense for MPI_INIT to choose a thread mode such as MPI_THREAD_FUNNELED (or for implementations to be allowed to choose one) as the default when, for example, OpenMP is enabled by compiler settings. Many people are probably using MPI_INIT in OpenMP applications without knowing that MPI makes doing that erroneous.
This would require MPI implementations to interact with the compiler, which is not something anyone I know desires. Furthermore, it is possible that an application will use MPI and OpenMP in a disjoint set of source files. How will the MPI implementation know that e.g. foo() invokes OpenMP? Do you want the linker to do something crazy with symbol redefinition to make this happen? I find many MPI users violated the standard when using OpenMP. I rebuke them whenever I observe this. I've not experienced anyone refusing to change their code to comply with the standard. I'll note that some implementations, which I will not name here to protect the guilty, support threads in a very poor way such that MPI_THREAD_FUNNELED is implemented as MPI_THREAD_MULTIPLE, meaning that it is better for uses to violate the MPI standard and take it on faith the implementation won't explode when OpenMP's Pthreads are cloned than to introduce the locking overhead of MPI_THREAD_MULTIPLE when it is entirely unnecessary. -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute [email protected] / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond