January 2015 - mpiwg-rma - mailman3.cels.anl.gov

Re: [mpiwg-rma] RMA Errata
by William Gropp 30 Jan '15

30 Jan '15

This is incorrect, and an example of why trying to put the shared memory issues together with the MPI-2 RMA concepts is a mistake. The local window makes sense for MPI RMA windows created with MPI_Win_create and with the MPI-3 routines MPI_Win_allocate and MPI_Win_create_dynamic. In those three cases, the memory is part of the process that created the MPI_Win and (at least as far as the MPI standard is concerned) is not visible to other processes. The term “local” as used in the MPI standard in the RMA context (not counting shared memory) refers to access to that memory from outside of MPI. Further, the sentence on page 410, lines 17-19 says nothing about “local and remote accesses” to shared memory. In the shared memory case, the situation is much more complicated, and it is made worse by the fact that the standard almost certainly has *ERRORS* in it with respect to shared memory (One of these errors might be page 410, lines 17-20). We should be trying to fix the errors, not compounding them by trying to warp the discussion to be compatible with the errors. Moving on to the use of the term “local”, the issue is that once MPI exposes memory that is shared between processes, any of those processes may use language-specific mechanisms to access the memory. We could call this “local”, but that would be dangerous and misleading, since there is no locality involved (I am not counting the various memory consistency effects, since these are not really local/remote but me/you effects). #456 is not acceptable in its current form, for reasons that have been discussed at length, and documented in the discussion with that ticket and at the Forum. If we are serious about fixing this issue, we need to start fresh, being careful not to fall into the two traps of (a) a simplistic and incorrect model of shared memory and (b) mandating overly strong memory consistency semantics that will slow down MPI programs. I personally prefer moving the shared memory consistency control to the language, where it can be properly handled. C11 appears to have what we need (caveat - see below), and the Forum, with the Fortran interface, has demonstrated that it is willing to take advantage of brand new language features to provide a better interface. Note that this also tells the compiler about the memory fence, which impacts the code transformations that are valid for the compiler, something that *is not possible* with a purely library based approach. We could also, using terms from C11, require that some small set of MPI routines (such as MPI_Win_fence, MPI_Win_lock/unlock, MPI_Win_post/start/complete/wait/test) act as an atomic_thread_fence(memory_order_seq_cst). (sequentially consistent acquire and release fence). But note that C defines several weaker fences, and fast applications make use of those. Insisting on the strongest is an acceptance of extra overhead. On the up side, this extra overhead would only apply to MPI RMA on shared memory windows. A similar solution is to have MPI_Win_sync (and only MPI_Win_sync) imply this memory fence. Or we could add an MPI function with the flexibility of the C11 atomic_thread_fence (and to partially solve the problem in Fortran, which doesn’t have a counter part to the C11 features). Variants of these let the user provide MPI_Info values when the window is created to control the level of fence (e.g., an info value that says the user will take responsibility for all memory consistency outside of the MPI RMA routines). If any of these are adopted, it will be important (and in contrast to the usual approach) to explicitly enumerate every single routine that may imply a memory fence. (The caveat on C11: Yes, these are for thread programming, and strictly speaking, might not be sufficient. But we can use the same definitions and semantics, and should not define our own terms). Bill On Jan 29, 2015, at 11:54 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: >> local part of a shared memory window and >> remote parts of the shared memory, > > For that there is no such "accepted definitions" in the literature. > MPI-3.0 is an exception. In the MPI one-sided chapter, it is defined > in detail what is presented Figures 11.2 and 11.3 on pages 439 and 440: > local window accesses, RMA accesses and synchronizations. > And the sentence page 410 lines 17-19 generalizes this for > local and remote accesses to shared memory

1 0

Re: [mpiwg-rma] RMA Errata
by Rolf Rabenseifner 29 Jan '15

29 Jan '15

> local part of a shared memory window and > remote parts of the shared memory, For that there is no such "accepted definitions" in the literature. MPI-3.0 is an exception. In the MPI one-sided chapter, it is defined in detail what is presented Figures 11.2 and 11.3 on pages 439 and 440: local window accesses, RMA accesses and synchronizations. And the sentence page 410 lines 17-19 generalizes this for local and remote accesses to shared memory. For me, this is strange. The solution in ticket #456 does not has this differentiation between "local" and "remote". The proposed new wording in new Section 11.5.4A does not contain any "local" or "remote" wording. The solution in #456 is based on: Paul E. McKenney? (ed.). Is Parallel Programming Hard, And, If So, What Can You Do About It? First Edition, Linux Technology Center, IBM Beaverton, March 10, 2014. http://kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook-e1.pdf Best regards Rolf ----- Original Message ----- > From: "William Gropp" <wgropp(a)illinois.edu> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Thursday, January 29, 2015 10:38:59 AM > Subject: Re: [mpiwg-rma] RMA Errata > > What does this mean? Please provide a reference into the accepted > definitions in the shared memory literature. > > Bill > > On Jan 29, 2015, at 10:12 AM, Rolf Rabenseifner < rabenseifner(a)hlrs.de > > wrote: > > > > local part of a shared memory window and > remote parts of the shared memory, > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] RMA Errata
by William Gropp 29 Jan '15

29 Jan '15

What does this mean? Please provide a reference into the accepted definitions in the shared memory literature. Bill On Jan 29, 2015, at 10:12 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > local part of a shared memory window and > remote parts of the shared memory,

1 0

Re: [mpiwg-rma] RMA Errata
by Rolf Rabenseifner 29 Jan '15

29 Jan '15

As far as I see, the Forum went into the direction to keep the sentence that (MPI-3.0 page 410 lines 17-19) "A consistent view can be created in the unified memory model (see Section 11.4) by utilizing the window synchronization functions (see Section 11.5) or explicitly completing outstanding store accesses (e.g., by calling MPI_WIN_FLUSH)." For the PSCW synchronization, this definition seems for me to restrictive, because it looks like that wording about local and remote accesses would imply a differentiation between a local part of a shared memory window and remote parts of the shared memory, which is unnatural for a shared Memory. To overcome this unnatural definition of PSCW synchronization (and also for the other one-sided synchronization methods) I wrote the solution in ticket #456 with a generalized definition that - has identical semantics with the existing text (cited above) for the case that one makes such unnatural Differentiation between local and remote shared memory parts, - but additional describes the synchronization without this differentiation, i.e., by defining the semantics of the synchronization functions for accesses to the same shared memory location by two processes. As far as I see, this more general definition does not require any implementation changes in the MPI libraries because all what is needed for the cited sentence implies the semantics defined in ticket #456. By the way, using only C11 functionality means that the user has to program its own process to process synchronization and all needed memory fences. It may make a code faster but not really easier to program. There is a high risk to make bugs or to use more fences than necessary. The definitions in #456 by using general access patterns is easy to program because the synchronization and memory fences are combined inside of the one-sided synchronization routines. Rolf ----- Original Message ----- > From: "William Gropp" <wgropp(a)illinois.edu> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Thursday, January 29, 2015 9:01:25 AM > Subject: Re: [mpiwg-rma] RMA Errata > > I believe that this is not only a pragmatic approach, but the correct one. > You should be able to use the C11 atomic memory features to have a > portable shared memory code (once we have full C11 compilers). > > Bill > > On Jan 28, 2015, at 11:22 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > > Thanks. I understand that this isn't an easy issue. We are trying to > > use MPI-3 shared memory windows internally with PSCW-like > > synchronization and I am inclined to just give up and use x86 > > intrinsics to do what I know is sufficient for our processors and > > compilers, just to make it easy (and fast). > > > > Jeff > > > > On Wed, Jan 28, 2015 at 1:43 PM, William Gropp <wgropp(a)illinois.edu> wrote: > >> That’s a harder topic and I wanted to think more about it. Here are the > >> two issues with which I am most concerned: > >> > >> 1) We are defining the interaction of MPI with stuff that happens outside of > >> MPI, in the programming language. Yes, we always sort of did, but this is > >> in an area where many skilled people have made mistakes, and the likelyhood > >> of an error is much higher. The lack of precision in these discussions > >> reinforces my concern. > >> > >> 2) We are not considering the potential overhead that requiring other than a > >> small set of RMA synchronization routines to guarantee some shared memory > >> synchronization. The sense that I get is that the Forum is unconcerned > >> about this, which I think is a serious mistake, and one for which I have > >> yet to see a compelling use case. > >> > >> Bill > >> > >> On Jan 28, 2015, at 7:46 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > >> > >> Fine, but it would be helpful if you would respond to my comments > >> about allowing point-to-point _synchronization_ on shared memory using > >> PSCW, since those were the more germane ones anyways. > >> > >> > >> > >> _______________________________________________ > >> mpiwg-rma mailing list > >> mpiwg-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > > > > > -- > > Jeff Hammond > > jeff.science(a)gmail.com > > http://jeffhammond.github.io/ > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] RMA Errata
by William Gropp 29 Jan '15

29 Jan '15

I believe that this is not only a pragmatic approach, but the correct one. You should be able to use the C11 atomic memory features to have a portable shared memory code (once we have full C11 compilers). Bill On Jan 28, 2015, at 11:22 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > Thanks. I understand that this isn't an easy issue. We are trying to > use MPI-3 shared memory windows internally with PSCW-like > synchronization and I am inclined to just give up and use x86 > intrinsics to do what I know is sufficient for our processors and > compilers, just to make it easy (and fast). > > Jeff > > On Wed, Jan 28, 2015 at 1:43 PM, William Gropp <wgropp(a)illinois.edu> wrote: >> That’s a harder topic and I wanted to think more about it. Here are the two >> issues with which I am most concerned: >> >> 1) We are defining the interaction of MPI with stuff that happens outside of >> MPI, in the programming language. Yes, we always sort of did, but this is >> in an area where many skilled people have made mistakes, and the likelyhood >> of an error is much higher. The lack of precision in these discussions >> reinforces my concern. >> >> 2) We are not considering the potential overhead that requiring other than a >> small set of RMA synchronization routines to guarantee some shared memory >> synchronization. The sense that I get is that the Forum is unconcerned >> about this, which I think is a serious mistake, and one for which I have yet >> to see a compelling use case. >> >> Bill >> >> On Jan 28, 2015, at 7:46 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >> >> Fine, but it would be helpful if you would respond to my comments >> about allowing point-to-point _synchronization_ on shared memory using >> PSCW, since those were the more germane ones anyways. >> >> >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] RMA Errata
by Jeff Hammond 28 Jan '15

28 Jan '15

Thanks. I understand that this isn't an easy issue. We are trying to use MPI-3 shared memory windows internally with PSCW-like synchronization and I am inclined to just give up and use x86 intrinsics to do what I know is sufficient for our processors and compilers, just to make it easy (and fast). Jeff On Wed, Jan 28, 2015 at 1:43 PM, William Gropp <wgropp(a)illinois.edu> wrote: > That’s a harder topic and I wanted to think more about it. Here are the two > issues with which I am most concerned: > > 1) We are defining the interaction of MPI with stuff that happens outside of > MPI, in the programming language. Yes, we always sort of did, but this is > in an area where many skilled people have made mistakes, and the likelyhood > of an error is much higher. The lack of precision in these discussions > reinforces my concern. > > 2) We are not considering the potential overhead that requiring other than a > small set of RMA synchronization routines to guarantee some shared memory > synchronization. The sense that I get is that the Forum is unconcerned > about this, which I think is a serious mistake, and one for which I have yet > to see a compelling use case. > > Bill > > On Jan 28, 2015, at 7:46 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > Fine, but it would be helpful if you would respond to my comments > about allowing point-to-point _synchronization_ on shared memory using > PSCW, since those were the more germane ones anyways. > > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpiwg-rma] RMA Errata
by William Gropp 28 Jan '15

28 Jan '15

That’s a harder topic and I wanted to think more about it. Here are the two issues with which I am most concerned: 1) We are defining the interaction of MPI with stuff that happens outside of MPI, in the programming language. Yes, we always sort of did, but this is in an area where many skilled people have made mistakes, and the likelyhood of an error is much higher. The lack of precision in these discussions reinforces my concern. 2) We are not considering the potential overhead that requiring other than a small set of RMA synchronization routines to guarantee some shared memory synchronization. The sense that I get is that the Forum is unconcerned about this, which I think is a serious mistake, and one for which I have yet to see a compelling use case. Bill On Jan 28, 2015, at 7:46 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > Fine, but it would be helpful if you would respond to my comments > about allowing point-to-point _synchronization_ on shared memory using > PSCW, since those were the more germane ones anyways.

1 0

Re: [mpiwg-rma] RMA Errata
by Jeff Hammond 28 Jan '15

28 Jan '15

Fine, but it would be helpful if you would respond to my comments about allowing point-to-point _synchronization_ on shared memory using PSCW, since those were the more germane ones anyways. Thanks, Jeff On Tue, Jan 27, 2015 at 11:12 PM, William Gropp <wgropp(a)illinois.edu> wrote: > No, this routine only tells you what address a particular process (specified > by rank) uses as the base address of a particular segment of shared memory. > It doesn’t say anything about which process “owns” the shared memory or is > “associated” with it. > > The issue is really one of care and precision in terminology, as well as > consistency with the language and definitions used by others in describing > shared memory. MPI shared memory RMA will not be credible if we make up own > own vague terms. > > Bill > > On Jan 28, 2015, at 6:45 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > And it is absolutely clear in MPI-3 that shared memory is associated > with a particular process. See the rank argument in > http://www.mpich.org/static/docs/v3.1/www3/MPI_Win_shared_query.html. > This function takes process rank as an _input_, which means that there > is a well-defined notion of memory associated with a particular > process. > > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpiwg-rma] RMA Errata
by William Gropp 28 Jan '15

28 Jan '15

No, this routine only tells you what address a particular process (specified by rank) uses as the base address of a particular segment of shared memory. It doesn’t say anything about which process “owns” the shared memory or is “associated” with it. The issue is really one of care and precision in terminology, as well as consistency with the language and definitions used by others in describing shared memory. MPI shared memory RMA will not be credible if we make up own own vague terms. Bill On Jan 28, 2015, at 6:45 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > And it is absolutely clear in MPI-3 that shared memory is associated > with a particular process. See the rank argument in > http://www.mpich.org/static/docs/v3.1/www3/MPI_Win_shared_query.html. > This function takes process rank as an _input_, which means that there > is a well-defined notion of memory associated with a particular > process.

1 0

Re: [mpiwg-rma] RMA Errata
by Jeff Hammond 27 Jan '15

27 Jan '15

I don't see why it matters which process allocated the memory. These are synchronization operations we are talking about and the notion of synchronization on the memory in a shared memory window should be valid in all the synchronization modes we have specified. I think it's reasonable for Rolf to want PSCW to make sense for shared memory windows. Imagine there is hardware with a point-to-point memory fence between processes (some of the new instructions in AArch64 have this feel to them, btw). And it is absolutely clear in MPI-3 that shared memory is associated with a particular process. See the rank argument in http://www.mpich.org/static/docs/v3.1/www3/MPI_Win_shared_query.html. This function takes process rank as an _input_, which means that there is a well-defined notion of memory associated with a particular process. Jeff On Sun, Jan 25, 2015 at 11:19 AM, William Gropp <wgropp(a)illinois.edu> wrote: > I think this is simply more evidence that the shared memory access should > entirely be through the valid language-specified mechanisms, including > interaction with other routines and threads that may access the same memory. > > I most definitely do not think that the load/store access to shared memory > is intended to be associated only with memory that was somehow “allocated” > by a particular process. I’m not even sure how I’d define that without > referring to a particular implementation. > > Bill > > On Jan 22, 2015, at 2:30 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > > #1 -- The effective target rank of load/store operations is currently not > defined on shared memory windows. That statement may seem like nonsense. > However, it is important for situations where load/store operations interact > with RMA operations that do have an explicit target rank (e.g. > lock/unlock). I believe this was an oversight, and we intended that the > load/store operations target the rank that allocated the memory. > > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0