- mpiwg-rma - mailman3.cels.anl.gov

Re: [mpiwg-rma] RMA Errata
by Rolf Rabenseifner 29 Jan '15

29 Jan '15

> local part of a shared memory window and > remote parts of the shared memory, For that there is no such "accepted definitions" in the literature. MPI-3.0 is an exception. In the MPI one-sided chapter, it is defined in detail what is presented Figures 11.2 and 11.3 on pages 439 and 440: local window accesses, RMA accesses and synchronizations. And the sentence page 410 lines 17-19 generalizes this for local and remote accesses to shared memory. For me, this is strange. The solution in ticket #456 does not has this differentiation between "local" and "remote". The proposed new wording in new Section 11.5.4A does not contain any "local" or "remote" wording. The solution in #456 is based on: Paul E. McKenney? (ed.). Is Parallel Programming Hard, And, If So, What Can You Do About It? First Edition, Linux Technology Center, IBM Beaverton, March 10, 2014. http://kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook-e1.pdf Best regards Rolf ----- Original Message ----- > From: "William Gropp" <wgropp(a)illinois.edu> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Thursday, January 29, 2015 10:38:59 AM > Subject: Re: [mpiwg-rma] RMA Errata > > What does this mean? Please provide a reference into the accepted > definitions in the shared memory literature. > > Bill > > On Jan 29, 2015, at 10:12 AM, Rolf Rabenseifner < rabenseifner(a)hlrs.de > > wrote: > > > > local part of a shared memory window and > remote parts of the shared memory, > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] RMA Errata
by William Gropp 29 Jan '15

29 Jan '15

What does this mean? Please provide a reference into the accepted definitions in the shared memory literature. Bill On Jan 29, 2015, at 10:12 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > local part of a shared memory window and > remote parts of the shared memory,

1 0

Re: [mpiwg-rma] RMA Errata
by Rolf Rabenseifner 29 Jan '15

29 Jan '15

As far as I see, the Forum went into the direction to keep the sentence that (MPI-3.0 page 410 lines 17-19) "A consistent view can be created in the unified memory model (see Section 11.4) by utilizing the window synchronization functions (see Section 11.5) or explicitly completing outstanding store accesses (e.g., by calling MPI_WIN_FLUSH)." For the PSCW synchronization, this definition seems for me to restrictive, because it looks like that wording about local and remote accesses would imply a differentiation between a local part of a shared memory window and remote parts of the shared memory, which is unnatural for a shared Memory. To overcome this unnatural definition of PSCW synchronization (and also for the other one-sided synchronization methods) I wrote the solution in ticket #456 with a generalized definition that - has identical semantics with the existing text (cited above) for the case that one makes such unnatural Differentiation between local and remote shared memory parts, - but additional describes the synchronization without this differentiation, i.e., by defining the semantics of the synchronization functions for accesses to the same shared memory location by two processes. As far as I see, this more general definition does not require any implementation changes in the MPI libraries because all what is needed for the cited sentence implies the semantics defined in ticket #456. By the way, using only C11 functionality means that the user has to program its own process to process synchronization and all needed memory fences. It may make a code faster but not really easier to program. There is a high risk to make bugs or to use more fences than necessary. The definitions in #456 by using general access patterns is easy to program because the synchronization and memory fences are combined inside of the one-sided synchronization routines. Rolf ----- Original Message ----- > From: "William Gropp" <wgropp(a)illinois.edu> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Thursday, January 29, 2015 9:01:25 AM > Subject: Re: [mpiwg-rma] RMA Errata > > I believe that this is not only a pragmatic approach, but the correct one. > You should be able to use the C11 atomic memory features to have a > portable shared memory code (once we have full C11 compilers). > > Bill > > On Jan 28, 2015, at 11:22 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > > Thanks. I understand that this isn't an easy issue. We are trying to > > use MPI-3 shared memory windows internally with PSCW-like > > synchronization and I am inclined to just give up and use x86 > > intrinsics to do what I know is sufficient for our processors and > > compilers, just to make it easy (and fast). > > > > Jeff > > > > On Wed, Jan 28, 2015 at 1:43 PM, William Gropp <wgropp(a)illinois.edu> wrote: > >> That’s a harder topic and I wanted to think more about it. Here are the > >> two issues with which I am most concerned: > >> > >> 1) We are defining the interaction of MPI with stuff that happens outside of > >> MPI, in the programming language. Yes, we always sort of did, but this is > >> in an area where many skilled people have made mistakes, and the likelyhood > >> of an error is much higher. The lack of precision in these discussions > >> reinforces my concern. > >> > >> 2) We are not considering the potential overhead that requiring other than a > >> small set of RMA synchronization routines to guarantee some shared memory > >> synchronization. The sense that I get is that the Forum is unconcerned > >> about this, which I think is a serious mistake, and one for which I have > >> yet to see a compelling use case. > >> > >> Bill > >> > >> On Jan 28, 2015, at 7:46 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > >> > >> Fine, but it would be helpful if you would respond to my comments > >> about allowing point-to-point _synchronization_ on shared memory using > >> PSCW, since those were the more germane ones anyways. > >> > >> > >> > >> _______________________________________________ > >> mpiwg-rma mailing list > >> mpiwg-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > > > > > -- > > Jeff Hammond > > jeff.science(a)gmail.com > > http://jeffhammond.github.io/ > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] RMA Errata
by William Gropp 29 Jan '15

29 Jan '15

I believe that this is not only a pragmatic approach, but the correct one. You should be able to use the C11 atomic memory features to have a portable shared memory code (once we have full C11 compilers). Bill On Jan 28, 2015, at 11:22 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > Thanks. I understand that this isn't an easy issue. We are trying to > use MPI-3 shared memory windows internally with PSCW-like > synchronization and I am inclined to just give up and use x86 > intrinsics to do what I know is sufficient for our processors and > compilers, just to make it easy (and fast). > > Jeff > > On Wed, Jan 28, 2015 at 1:43 PM, William Gropp <wgropp(a)illinois.edu> wrote: >> That’s a harder topic and I wanted to think more about it. Here are the two >> issues with which I am most concerned: >> >> 1) We are defining the interaction of MPI with stuff that happens outside of >> MPI, in the programming language. Yes, we always sort of did, but this is >> in an area where many skilled people have made mistakes, and the likelyhood >> of an error is much higher. The lack of precision in these discussions >> reinforces my concern. >> >> 2) We are not considering the potential overhead that requiring other than a >> small set of RMA synchronization routines to guarantee some shared memory >> synchronization. The sense that I get is that the Forum is unconcerned >> about this, which I think is a serious mistake, and one for which I have yet >> to see a compelling use case. >> >> Bill >> >> On Jan 28, 2015, at 7:46 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >> >> Fine, but it would be helpful if you would respond to my comments >> about allowing point-to-point _synchronization_ on shared memory using >> PSCW, since those were the more germane ones anyways. >> >> >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] RMA Errata
by Jeff Hammond 28 Jan '15

28 Jan '15

Thanks. I understand that this isn't an easy issue. We are trying to use MPI-3 shared memory windows internally with PSCW-like synchronization and I am inclined to just give up and use x86 intrinsics to do what I know is sufficient for our processors and compilers, just to make it easy (and fast). Jeff On Wed, Jan 28, 2015 at 1:43 PM, William Gropp <wgropp(a)illinois.edu> wrote: > That’s a harder topic and I wanted to think more about it. Here are the two > issues with which I am most concerned: > > 1) We are defining the interaction of MPI with stuff that happens outside of > MPI, in the programming language. Yes, we always sort of did, but this is > in an area where many skilled people have made mistakes, and the likelyhood > of an error is much higher. The lack of precision in these discussions > reinforces my concern. > > 2) We are not considering the potential overhead that requiring other than a > small set of RMA synchronization routines to guarantee some shared memory > synchronization. The sense that I get is that the Forum is unconcerned > about this, which I think is a serious mistake, and one for which I have yet > to see a compelling use case. > > Bill > > On Jan 28, 2015, at 7:46 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > Fine, but it would be helpful if you would respond to my comments > about allowing point-to-point _synchronization_ on shared memory using > PSCW, since those were the more germane ones anyways. > > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpiwg-rma] RMA Errata
by William Gropp 28 Jan '15

28 Jan '15

That’s a harder topic and I wanted to think more about it. Here are the two issues with which I am most concerned: 1) We are defining the interaction of MPI with stuff that happens outside of MPI, in the programming language. Yes, we always sort of did, but this is in an area where many skilled people have made mistakes, and the likelyhood of an error is much higher. The lack of precision in these discussions reinforces my concern. 2) We are not considering the potential overhead that requiring other than a small set of RMA synchronization routines to guarantee some shared memory synchronization. The sense that I get is that the Forum is unconcerned about this, which I think is a serious mistake, and one for which I have yet to see a compelling use case. Bill On Jan 28, 2015, at 7:46 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > Fine, but it would be helpful if you would respond to my comments > about allowing point-to-point _synchronization_ on shared memory using > PSCW, since those were the more germane ones anyways.

1 0

Re: [mpiwg-rma] RMA Errata
by Jeff Hammond 28 Jan '15

28 Jan '15

Fine, but it would be helpful if you would respond to my comments about allowing point-to-point _synchronization_ on shared memory using PSCW, since those were the more germane ones anyways. Thanks, Jeff On Tue, Jan 27, 2015 at 11:12 PM, William Gropp <wgropp(a)illinois.edu> wrote: > No, this routine only tells you what address a particular process (specified > by rank) uses as the base address of a particular segment of shared memory. > It doesn’t say anything about which process “owns” the shared memory or is > “associated” with it. > > The issue is really one of care and precision in terminology, as well as > consistency with the language and definitions used by others in describing > shared memory. MPI shared memory RMA will not be credible if we make up own > own vague terms. > > Bill > > On Jan 28, 2015, at 6:45 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > And it is absolutely clear in MPI-3 that shared memory is associated > with a particular process. See the rank argument in > http://www.mpich.org/static/docs/v3.1/www3/MPI_Win_shared_query.html. > This function takes process rank as an _input_, which means that there > is a well-defined notion of memory associated with a particular > process. > > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpiwg-rma] RMA Errata
by William Gropp 28 Jan '15

28 Jan '15

No, this routine only tells you what address a particular process (specified by rank) uses as the base address of a particular segment of shared memory. It doesn’t say anything about which process “owns” the shared memory or is “associated” with it. The issue is really one of care and precision in terminology, as well as consistency with the language and definitions used by others in describing shared memory. MPI shared memory RMA will not be credible if we make up own own vague terms. Bill On Jan 28, 2015, at 6:45 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > And it is absolutely clear in MPI-3 that shared memory is associated > with a particular process. See the rank argument in > http://www.mpich.org/static/docs/v3.1/www3/MPI_Win_shared_query.html. > This function takes process rank as an _input_, which means that there > is a well-defined notion of memory associated with a particular > process.

1 0

Re: [mpiwg-rma] RMA Errata
by Jeff Hammond 27 Jan '15

27 Jan '15

I don't see why it matters which process allocated the memory. These are synchronization operations we are talking about and the notion of synchronization on the memory in a shared memory window should be valid in all the synchronization modes we have specified. I think it's reasonable for Rolf to want PSCW to make sense for shared memory windows. Imagine there is hardware with a point-to-point memory fence between processes (some of the new instructions in AArch64 have this feel to them, btw). And it is absolutely clear in MPI-3 that shared memory is associated with a particular process. See the rank argument in http://www.mpich.org/static/docs/v3.1/www3/MPI_Win_shared_query.html. This function takes process rank as an _input_, which means that there is a well-defined notion of memory associated with a particular process. Jeff On Sun, Jan 25, 2015 at 11:19 AM, William Gropp <wgropp(a)illinois.edu> wrote: > I think this is simply more evidence that the shared memory access should > entirely be through the valid language-specified mechanisms, including > interaction with other routines and threads that may access the same memory. > > I most definitely do not think that the load/store access to shared memory > is intended to be associated only with memory that was somehow “allocated” > by a particular process. I’m not even sure how I’d define that without > referring to a particular implementation. > > Bill > > On Jan 22, 2015, at 2:30 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > > #1 -- The effective target rank of load/store operations is currently not > defined on shared memory windows. That statement may seem like nonsense. > However, it is important for situations where load/store operations interact > with RMA operations that do have an explicit target rank (e.g. > lock/unlock). I believe this was an oversight, and we intended that the > load/store operations target the rank that allocated the memory. > > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpiwg-rma] RMA Errata
by William Gropp 25 Jan '15

25 Jan '15

I think this is simply more evidence that the shared memory access should entirely be through the valid language-specified mechanisms, including interaction with other routines and threads that may access the same memory. I most definitely do not think that the load/store access to shared memory is intended to be associated only with memory that was somehow “allocated” by a particular process. I’m not even sure how I’d define that without referring to a particular implementation. Bill On Jan 22, 2015, at 2:30 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: >> #1 -- The effective target rank of load/store operations is currently not >> defined on shared memory windows. That statement may seem like nonsense. >> However, it is important for situations where load/store operations interact >> with RMA operations that do have an explicit target rank (e.g. >> lock/unlock). I believe this was an oversight, and we intended that the >> load/store operations target the rank that allocated the memory.

1 0