- mpiwg-rma - mailman3.cels.anl.gov

Memory model and source code portability
by Rolf Rabenseifner 30 Sep '14

30 Sep '14

The follwing discussion is (partially) independent from RMA. The example is about MPI_Irecv and MPI_Wait. > Comment (by gropp): > > I fully agree with Dave. As I've said before, MPI should only talk about > MPI. What the compiler might to with non-MPI is not in scope for us. > Yes, this makes it difficult to provide examples that are guaranteed to > work; that's part of the point of Boehm's article. But that doesn't > change the fact that the MPI Forum must not specify the behavior of C (or > Fortran) codes that happen to be accessing shared memory. > > -- > Ticket URL: > <https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/456#comment:16> For me it looks like that you give up with the important goal that the MPI standard should provide source code portability. I mean source code portability for MPI applications written in one of the supported language bindings. For me, the MPI standard has to specify all that is needed that a source code that is consistent with the MPI standard is portable from one MPI library with companion compiler to another MPI library with its companion compiler. With MPI-3.0 and the new Fortran interface, we tried to achieve this goal. Maybe that further investigation is needed. People always thought that there is no such problem with C. MPI-3.0 Section 17.1.20 "Comparison with C" tries to say this. As I learned in this discussion, such expectation about brave C is completely wrong, as seen in https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/456#comment:13 There is no source code portability for the MPI C Interface if C = C99 or older: Such a small code for(...) { MPI_Irecv(&buf,...,&rq) MPI_Wait(&rq,...) print(...) buf } may be compiled by C99 (not C11) into reg0=buf; for(...) { buf=reg0; MPI_Irecv(&buf,...,&rq) reg0=buf; buf=reg0; MPI_Wait(&rq,...) with internally: buf=received_value reg0=buf; print(...) reg0 } buf=reg0; And with an MPI library making progress between MPI_Irecv and MPI_Wait, the outcome may be reg0=buf; for(...) { buf=reg0; MPI_Irecv(&buf,...,&rq) reg0=buf; with internally by the MPI lib progress engine: buf=received_value buf=reg0; /* i.e. the received value is overwritten by the old value!!! */ MPI_Wait(&rq,...) reg0=buf; print(...) reg0 } buf=reg0; Or in other words, according to - MPI-3.0 Section 17.1.20, and - https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/456#comment:13 already MPI-1.1 for C required the C11/C11++ memory model, which does not allow such optimizations. This has direct influence on the wording we use in MPI-3.0 Section 2.6. Saying "ISO C is enough" is outdated, i.e., closing the eyes in front of the knowledge that exists. I would like that the MPI Forum gets this right, in C and Fortran. Rolf -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] MPI RMA status summary
by Jeff Hammond 30 Sep '14

30 Sep '14

If we require a memory barrier in Flush_local, then any implementation of SHMEM over MPI-3 is going to suck and we have failed in one of the goals of MPI-3 RMA. See https://github.com/jeffhammond/oshmpi/blob/master/src/shmem-internals.c#L517 for why. Jeff On Tue, Sep 30, 2014 at 7:35 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > There are different ways to look at it. > Remote memory access like remote load and store can > be interpreted as RMA by one reader and non-RMA by another one. > > Torsten Hoefler, James Dinan, Darius Buntinas, Pavan Balaji, Brian Barrett, > Ron Brightwell, William Gropp, Vivek Kale, Rajeev Thakur > in > "MPI + MPI: a new hybrid approach to parallel programming with MPI plus shared Memory" > at EuroMPI 2012 > obviously treated them as RMA, because otherwise their store+Fence+load would > have been wrong. > > The major reason for going with b) and not with a) is simple: > > If somenbody does not want to have a shared memory > synchronization semantics for an existing MPI_Win > synchronization routine (e.g. MPI_Win_fence), then > he/she Need not to use the Routine together with > shared memory windows. > If he/she wants to stay with MP_Put/Get, > then he/she needs not to use MPI_Win_allocate_shared. > It is allowed to use MPI_Win_allocate on a shered memory node > and MPI internally can do the same optimizations. > > Therefore shared memory synchronization semantics > for MPI_Win_fence, MPI_Win_post/start/complete/wait, > MPI_Win_lock/unlock, ... > is never a drawback. > > But it is a clear advantage, because process-to-process > synchronization is combined with local memory synchronization > which may be implemented faster than if seperated into > different routines. > > Rolf > > > ----- Original Message ----- >> From: "Jeff Hammond" <jeff.science(a)gmail.com> >> To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> >> Sent: Tuesday, September 30, 2014 4:09:02 PM >> Subject: Re: [mpiwg-rma] MPI RMA status summary >> >> Option A is what the standard says today outside of a sloppy offhand >> remark in parentheses. See my note please. >> >> Jeff >> >> Sent from my iPhone >> >> > On Sep 30, 2014, at 7:05 AM, Rolf Rabenseifner >> > <rabenseifner(a)hlrs.de> wrote: >> > >> > I strongly agree with your statement: >> >> These piecemeal changes are one of the sources of our problems. >> > >> > I only wanted to strongly say, that I would never vote for your a), >> > because it is not backward compatible to what is already used. >> > And with b) I've the problem that b1) is clear to me (see #456), >> > but the Win_flush semantics for load/store is unclear to me. >> > >> > Of course, a total solution is needed and not parts of it. >> > #456 is such a trial for a complete solution. >> > >> > Rolf >> > >> > ----- Original Message ----- >> >> From: "William Gropp" <wgropp(a)illinois.edu> >> >> To: "MPI WG Remote Memory Access working group" >> >> <mpiwg-rma(a)lists.mpi-forum.org> >> >> Sent: Tuesday, September 30, 2014 3:19:06 PM >> >> Subject: Re: [mpiwg-rma] MPI RMA status summary >> >> >> >> I disagree with this approach. The most important thing to do is >> >> to >> >> figure out the correct definitions and semantics. Once we agree >> >> on >> >> that, we can determine what can be handled as an errata and what >> >> will require an update to the chapter and an update to the MPI >> >> standard. These piecemeal changes are one of the sources of our >> >> problems. >> >> >> >> Bill >> >> >> >> On Sep 30, 2014, at 7:38 AM, Rolf Rabenseifner >> >> <rabenseifner(a)hlrs.de> >> >> wrote: >> >> >> >>>> Vote for 1 of the following: >> >>>> >> >>>> a) Only Win_sync provides memory barrier semantics to shared >> >>>> memory >> >>>> windows >> >>>> b) All RMA completion/sync routines (e.g., MPI_Win_lock, >> >>>> MPI_Win_fence, MPI_Win_flush) provide memory barrier semantics >> >>>> c) Some as yet undetermined blend of a and b, which might >> >>>> include >> >>>> additional asserts >> >>>> d) This topic needs further discussion >> >>> >> >>> Because we only have to clarify MPI-3.0 (this is an errata issue) >> >>> and >> >>> - obviously the MPI Forum and the readers expected that >> >>> MPI_Win_fence >> >>> (and therefore also the other MPI-2 synchronizations >> >>> MPI_Win_post/start/complete/wait and MPI_Win_lock/unlock) >> >>> works if MPI_Get/Put are sustituted by shared memory load/store >> >>> (see the many Forum members as authors of the EuroMPI paper) >> >>> - and the Forum decided that also MPI_Win_sync acts as if >> >>> a memory barrier is inside, >> >>> for me, >> >>> - a) cannot be chosen bacause an erratum cannot remove >> >>> a given functionality >> >>> - and b) is automatically given, see reasons above. Therefore >> >>> #456. >> >>> >> >>> The only open question for me is about the meaning of >> >>> MPI_Win_flush. >> >>> Therefore MPI_Win_flush is still missing in #456. >> >>> >> >>> Therefore for me, the Major choices seems to be >> >>> b1) MPI-2 synchronizations + MPI_Win_sync >> >>> b2) MPI-2 synchronizations + MPI_Win_sync + MPI_Win_flush >> >>> >> >>> For this vote, I clearly want to see a clear proposal >> >>> about the meaning of MPI_Win_flush together with >> >>> sared memory load/store, hopefully with the notation >> >>> used in #456. >> >>> >> >>> Best regards >> >>> Rolf >> >>> >> >>> ----- Original Message ----- >> >>>> From: "William Gropp" <wgropp(a)illinois.edu> >> >>>> To: "MPI WG Remote Memory Access working group" >> >>>> <mpiwg-rma(a)lists.mpi-forum.org> >> >>>> Sent: Monday, September 29, 2014 11:39:51 PM >> >>>> Subject: Re: [mpiwg-rma] MPI RMA status summary >> >>>> >> >>>> >> >>>> Thanks, Jeff. >> >>>> >> >>>> >> >>>> I agree that I don’t want load/store to be considered RMA >> >>>> operations. >> >>>> But the issue of the memory consistency on RMA synchronization >> >>>> and >> >>>> completion operations to a shared memory window is complex. In >> >>>> some >> >>>> ways, the most consistent with RMA in other situations is the >> >>>> case >> >>>> of MPI_Win_lock to your own process; the easiest extension for >> >>>> the >> >>>> user is to have reasonably strong memory barrier semantics at >> >>>> all >> >>>> sync/completion operations (thus including Fence). As you note, >> >>>> this has costs. At the other extreme, we could say that only >> >>>> Win_sync provides these memory barrier semantics. And we could >> >>>> pick >> >>>> a more complex blend (yes for some, no for others). >> >>>> >> >>>> >> >>>> One of the first questions is whether we want to only Win_sync, >> >>>> all >> >>>> completion/sync RMA routines, or some subset to provide memory >> >>>> barrier semantics for shared memory windows (this would include >> >>>> RMA >> >>>> windows that claimed to be shared memory, since there is a >> >>>> proposal >> >>>> to extend that property to other RMA windows). It would be good >> >>>> to >> >>>> make progress on this question, so I propose a straw vote of >> >>>> this >> >>>> group by email. Vote for 1 of the following: >> >>>> >> >>>> >> >>>> a) Only Win_sync provides memory barrier semantics to shared >> >>>> memory >> >>>> windows >> >>>> b) All RMA completion/sync routines (e.g., MPI_Win_lock, >> >>>> MPI_Win_fence, MPI_Win_flush) provide memory barrier semantics >> >>>> c) Some as yet undetermined blend of a and b, which might >> >>>> include >> >>>> additional asserts >> >>>> d) This topic needs further discussion >> >>>> >> >>>> >> >>>> Note that I’ve left off what “memory barrier semantics” means. >> >>>> That >> >>>> will need to be precisely defined for the standard, but I >> >>>> believe >> >>>> we >> >>>> know what we intend for this. We specifically are not defining >> >>>> what >> >>>> happens with non-MPI code. Also note that this is separate from >> >>>> whether the RMA sync routines appear to be blocking when applied >> >>>> to >> >>>> a shared memory window; we can do a separate straw vote on that. >> >>>> >> >>>> >> >>>> Bill >> >>>> >> >>>> >> >>>> >> >>>> On Sep 29, 2014, at 3:49 PM, Jeff Hammond < >> >>>> jeff.science(a)gmail.com >> >>>> wrote: >> >>>> >> >>>> >> >>>> >> >>>> On Mon, Sep 29, 2014 at 9:16 AM, Rolf Rabenseifner < >> >>>> rabenseifner(a)hlrs.de > wrote: >> >>>> >> >>>> >> >>>> Only about the issues on #456 (shared memory syncronization): >> >>>> >> >>>> >> >>>> >> >>>> For the ones requiring discussion, assign someone to organize a >> >>>> position and discussion. We can schedule telecons to go over >> >>>> those >> >>>> issues. The first item in the list is certainly in this class. >> >>>> >> >>>> Who can organize telecons on #456. >> >>>> Would it be possible to organize a RMA meeting at SC? >> >>>> >> >>>> I will be there Monday through part of Thursday but am usually >> >>>> triple-booked from 8 AM to midnight. >> >>>> >> >>>> >> >>>> >> >>>> The position expressed by the solution #456 is based on the idea >> >>>> that the MPI RMA synchronization routines should have the same >> >>>> outcome when RMA PUT and GET calls are substituted by stores and >> >>>> loads. >> >>>> >> >>>> The outcome for the flush routines is still not defined. >> >>>> >> >>>> It is interesting, because the standard is actually conflicting >> >>>> on >> >>>> whether Flush affects load-store. I find this incredibly >> >>>> frustrating. >> >>>> >> >>>> Page 450: >> >>>> >> >>>> "Locally completes at the origin all outstanding RMA operations >> >>>> initiated by the calling process to the target process specified >> >>>> by >> >>>> rank on the specified window. For example, after this routine >> >>>> completes, the user may reuse any buffers provided to put, get, >> >>>> or >> >>>> accumulate operations." >> >>>> >> >>>> I do not not think "RMA operations" includes load-store. >> >>>> >> >>>> Page 410: >> >>>> >> >>>> "The consistency of load/store accesses from/to the shared >> >>>> memory >> >>>> as >> >>>> observed by the user program depends on the architecture. A >> >>>> consistent >> >>>> view can be created in the unified memory model (see Section >> >>>> 11.4) >> >>>> by >> >>>> utilizing the window synchronization functions (see Section >> >>>> 11.5) >> >>>> or >> >>>> explicitly completing outstanding store accesses (e.g., by >> >>>> calling >> >>>> MPI_WIN_FLUSH)." >> >>>> >> >>>> Here it is unambiguously implied that MPI_WIN_FLUSH affects >> >>>> load-stores. >> >>>> >> >>>> My preference is to fix the statement on 410 since it is less >> >>>> canonical than the one on 450, and because I do not want to have >> >>>> a >> >>>> memory barrier in every call to WIN_FLUSH. >> >>>> >> >>>> Jeff >> >>>> >> >>>> >> >>>> >> >>>> I would prefere to have an organizer of the discussion inside of >> >>>> the RMA subgroup that proposed the changes for MPI-3.1 >> >>>> rather that I'm the organizer. >> >>>> I tried to bring all the input together and hope that #456 >> >>>> is now state that it is consistent itsself and with the >> >>>> expectations expressed by the group that published the >> >>>> paper at EuroMPI on first usage of this shared memory interface. >> >>>> >> >>>> The ticket is (together with the help of recent C11 >> >>>> standadization) >> >>>> on a good way to be also consistent with the compiler >> >>>> optimizations - >> >>>> in other words - the C standardization body has learnt from the >> >>>> pthreads problems. Fortran is still an open question to me, >> >>>> i.e., I do not know the status, see >> >>>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/456#comment:13 >> >>>> >> >>>> Best regards >> >>>> Rolf >> >>>> >> >>>> >> >>>> >> >>>> ----- Original Message ----- >> >>>> >> >>>> >> >>>> From: "William Gropp" <wgropp(a)illinois.edu> >> >>>> To: "MPI WG Remote Memory Access working group" >> >>>> <mpiwg-rma(a)lists.mpi-forum.org> >> >>>> Sent: Thursday, September 25, 2014 4:19:14 PM >> >>>> Subject: [mpiwg-rma] MPI RMA status summary >> >>>> >> >>>> I looked through all of the tickets and wrote a summary of the >> >>>> open >> >>>> issues, which I’ve attached. I propose the following: >> >>>> >> >>>> Determine which of these issues can be resolved by email. A >> >>>> significant number can probably be closed with no further >> >>>> action. >> >>>> >> >>>> For those requiring rework, determine if there is still interest >> >>>> in >> >>>> them, and if not, close them as well. >> >>>> >> >>>> For the ones requiring discussion, assign someone to organize a >> >>>> position and discussion. We can schedule telecons to go over >> >>>> those >> >>>> issues. The first item in the list is certainly in this class. >> >>>> >> >>>> Comments? >> >>>> >> >>>> Bill >> >>>> >> >>>> _______________________________________________ >> >>>> mpiwg-rma mailing list >> >>>> mpiwg-rma(a)lists.mpi-forum.org >> >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>>> >> >>>> -- >> >>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email >> >>>> rabenseifner(a)hlrs.de >> >>>> High Performance Computing Center (HLRS) . phone >> >>>> ++49(0)711/685-65530 >> >>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / >> >>>> 685-65832 >> >>>> Head of Dpmt Parallel Computing . . . >> >>>> www.hlrs.de/people/rabenseifner >> >>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room >> >>>> 1.307) >> >>>> _______________________________________________ >> >>>> mpiwg-rma mailing list >> >>>> mpiwg-rma(a)lists.mpi-forum.org >> >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>>> >> >>>> >> >>>> >> >>>> -- >> >>>> Jeff Hammond >> >>>> jeff.science(a)gmail.com >> >>>> http://jeffhammond.github.io/ >> >>>> _______________________________________________ >> >>>> mpiwg-rma mailing list >> >>>> mpiwg-rma(a)lists.mpi-forum.org >> >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>>> >> >>>> _______________________________________________ >> >>>> mpiwg-rma mailing list >> >>>> mpiwg-rma(a)lists.mpi-forum.org >> >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>> >> >>> -- >> >>> Dr. Rolf Rabenseifner . . . . . . . . . .. email >> >>> rabenseifner(a)hlrs.de >> >>> High Performance Computing Center (HLRS) . phone >> >>> ++49(0)711/685-65530 >> >>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / >> >>> 685-65832 >> >>> Head of Dpmt Parallel Computing . . . >> >>> www.hlrs.de/people/rabenseifner >> >>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room >> >>> 1.307) >> >>> _______________________________________________ >> >>> mpiwg-rma mailing list >> >>> mpiwg-rma(a)lists.mpi-forum.org >> >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> >> >> _______________________________________________ >> >> mpiwg-rma mailing list >> >> mpiwg-rma(a)lists.mpi-forum.org >> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> > >> > -- >> > Dr. Rolf Rabenseifner . . . . . . . . . .. email >> > rabenseifner(a)hlrs.de >> > High Performance Computing Center (HLRS) . phone >> > ++49(0)711/685-65530 >> > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / >> > 685-65832 >> > Head of Dpmt Parallel Computing . . . >> > www.hlrs.de/people/rabenseifner >> > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room >> > 1.307) >> > _______________________________________________ >> > mpiwg-rma mailing list >> > mpiwg-rma(a)lists.mpi-forum.org >> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpiwg-rma] MPI RMA status summary
by Rajeev Thakur 30 Sep '14

30 Sep '14

There are two separate issues that are being confused: memory barrier and interprocess synchronization. Jeff's note and Bill's a, b, c choices are for memory barrier. Rolf's ticket 456 is for interprocess synchronization, i.e., whether the interprocess synchronization semantics of fence/pscw for RMA operations also apply to shared-memory loads/stores. Rajeev On Sep 30, 2014, at 9:35 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > There are different ways to look at it. > Remote memory access like remote load and store can > be interpreted as RMA by one reader and non-RMA by another one. > > Torsten Hoefler, James Dinan, Darius Buntinas, Pavan Balaji, Brian Barrett, > Ron Brightwell, William Gropp, Vivek Kale, Rajeev Thakur > in > "MPI + MPI: a new hybrid approach to parallel programming with MPI plus shared Memory" > at EuroMPI 2012 > obviously treated them as RMA, because otherwise their store+Fence+load would > have been wrong. > > The major reason for going with b) and not with a) is simple: > > If somenbody does not want to have a shared memory > synchronization semantics for an existing MPI_Win > synchronization routine (e.g. MPI_Win_fence), then > he/she Need not to use the Routine together with > shared memory windows. > If he/she wants to stay with MP_Put/Get, > then he/she needs not to use MPI_Win_allocate_shared. > It is allowed to use MPI_Win_allocate on a shered memory node > and MPI internally can do the same optimizations. > > Therefore shared memory synchronization semantics > for MPI_Win_fence, MPI_Win_post/start/complete/wait, > MPI_Win_lock/unlock, ... > is never a drawback. > > But it is a clear advantage, because process-to-process > synchronization is combined with local memory synchronization > which may be implemented faster than if seperated into > different routines. > > Rolf > > > ----- Original Message ----- >> From: "Jeff Hammond" <jeff.science(a)gmail.com> >> To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> >> Sent: Tuesday, September 30, 2014 4:09:02 PM >> Subject: Re: [mpiwg-rma] MPI RMA status summary >> >> Option A is what the standard says today outside of a sloppy offhand >> remark in parentheses. See my note please. >> >> Jeff >> >> Sent from my iPhone >> >>> On Sep 30, 2014, at 7:05 AM, Rolf Rabenseifner >>> <rabenseifner(a)hlrs.de> wrote: >>> >>> I strongly agree with your statement: >>>> These piecemeal changes are one of the sources of our problems. >>> >>> I only wanted to strongly say, that I would never vote for your a), >>> because it is not backward compatible to what is already used. >>> And with b) I've the problem that b1) is clear to me (see #456), >>> but the Win_flush semantics for load/store is unclear to me. >>> >>> Of course, a total solution is needed and not parts of it. >>> #456 is such a trial for a complete solution. >>> >>> Rolf >>> >>> ----- Original Message ----- >>>> From: "William Gropp" <wgropp(a)illinois.edu> >>>> To: "MPI WG Remote Memory Access working group" >>>> <mpiwg-rma(a)lists.mpi-forum.org> >>>> Sent: Tuesday, September 30, 2014 3:19:06 PM >>>> Subject: Re: [mpiwg-rma] MPI RMA status summary >>>> >>>> I disagree with this approach. The most important thing to do is >>>> to >>>> figure out the correct definitions and semantics. Once we agree >>>> on >>>> that, we can determine what can be handled as an errata and what >>>> will require an update to the chapter and an update to the MPI >>>> standard. These piecemeal changes are one of the sources of our >>>> problems. >>>> >>>> Bill >>>> >>>> On Sep 30, 2014, at 7:38 AM, Rolf Rabenseifner >>>> <rabenseifner(a)hlrs.de> >>>> wrote: >>>> >>>>>> Vote for 1 of the following: >>>>>> >>>>>> a) Only Win_sync provides memory barrier semantics to shared >>>>>> memory >>>>>> windows >>>>>> b) All RMA completion/sync routines (e.g., MPI_Win_lock, >>>>>> MPI_Win_fence, MPI_Win_flush) provide memory barrier semantics >>>>>> c) Some as yet undetermined blend of a and b, which might >>>>>> include >>>>>> additional asserts >>>>>> d) This topic needs further discussion >>>>> >>>>> Because we only have to clarify MPI-3.0 (this is an errata issue) >>>>> and >>>>> - obviously the MPI Forum and the readers expected that >>>>> MPI_Win_fence >>>>> (and therefore also the other MPI-2 synchronizations >>>>> MPI_Win_post/start/complete/wait and MPI_Win_lock/unlock) >>>>> works if MPI_Get/Put are sustituted by shared memory load/store >>>>> (see the many Forum members as authors of the EuroMPI paper) >>>>> - and the Forum decided that also MPI_Win_sync acts as if >>>>> a memory barrier is inside, >>>>> for me, >>>>> - a) cannot be chosen bacause an erratum cannot remove >>>>> a given functionality >>>>> - and b) is automatically given, see reasons above. Therefore >>>>> #456. >>>>> >>>>> The only open question for me is about the meaning of >>>>> MPI_Win_flush. >>>>> Therefore MPI_Win_flush is still missing in #456. >>>>> >>>>> Therefore for me, the Major choices seems to be >>>>> b1) MPI-2 synchronizations + MPI_Win_sync >>>>> b2) MPI-2 synchronizations + MPI_Win_sync + MPI_Win_flush >>>>> >>>>> For this vote, I clearly want to see a clear proposal >>>>> about the meaning of MPI_Win_flush together with >>>>> sared memory load/store, hopefully with the notation >>>>> used in #456. >>>>> >>>>> Best regards >>>>> Rolf >>>>> >>>>> ----- Original Message ----- >>>>>> From: "William Gropp" <wgropp(a)illinois.edu> >>>>>> To: "MPI WG Remote Memory Access working group" >>>>>> <mpiwg-rma(a)lists.mpi-forum.org> >>>>>> Sent: Monday, September 29, 2014 11:39:51 PM >>>>>> Subject: Re: [mpiwg-rma] MPI RMA status summary >>>>>> >>>>>> >>>>>> Thanks, Jeff. >>>>>> >>>>>> >>>>>> I agree that I don’t want load/store to be considered RMA >>>>>> operations. >>>>>> But the issue of the memory consistency on RMA synchronization >>>>>> and >>>>>> completion operations to a shared memory window is complex. In >>>>>> some >>>>>> ways, the most consistent with RMA in other situations is the >>>>>> case >>>>>> of MPI_Win_lock to your own process; the easiest extension for >>>>>> the >>>>>> user is to have reasonably strong memory barrier semantics at >>>>>> all >>>>>> sync/completion operations (thus including Fence). As you note, >>>>>> this has costs. At the other extreme, we could say that only >>>>>> Win_sync provides these memory barrier semantics. And we could >>>>>> pick >>>>>> a more complex blend (yes for some, no for others). >>>>>> >>>>>> >>>>>> One of the first questions is whether we want to only Win_sync, >>>>>> all >>>>>> completion/sync RMA routines, or some subset to provide memory >>>>>> barrier semantics for shared memory windows (this would include >>>>>> RMA >>>>>> windows that claimed to be shared memory, since there is a >>>>>> proposal >>>>>> to extend that property to other RMA windows). It would be good >>>>>> to >>>>>> make progress on this question, so I propose a straw vote of >>>>>> this >>>>>> group by email. Vote for 1 of the following: >>>>>> >>>>>> >>>>>> a) Only Win_sync provides memory barrier semantics to shared >>>>>> memory >>>>>> windows >>>>>> b) All RMA completion/sync routines (e.g., MPI_Win_lock, >>>>>> MPI_Win_fence, MPI_Win_flush) provide memory barrier semantics >>>>>> c) Some as yet undetermined blend of a and b, which might >>>>>> include >>>>>> additional asserts >>>>>> d) This topic needs further discussion >>>>>> >>>>>> >>>>>> Note that I’ve left off what “memory barrier semantics” means. >>>>>> That >>>>>> will need to be precisely defined for the standard, but I >>>>>> believe >>>>>> we >>>>>> know what we intend for this. We specifically are not defining >>>>>> what >>>>>> happens with non-MPI code. Also note that this is separate from >>>>>> whether the RMA sync routines appear to be blocking when applied >>>>>> to >>>>>> a shared memory window; we can do a separate straw vote on that. >>>>>> >>>>>> >>>>>> Bill >>>>>> >>>>>> >>>>>> >>>>>> On Sep 29, 2014, at 3:49 PM, Jeff Hammond < >>>>>> jeff.science(a)gmail.com >>>>>> wrote: >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Sep 29, 2014 at 9:16 AM, Rolf Rabenseifner < >>>>>> rabenseifner(a)hlrs.de > wrote: >>>>>> >>>>>> >>>>>> Only about the issues on #456 (shared memory syncronization): >>>>>> >>>>>> >>>>>> >>>>>> For the ones requiring discussion, assign someone to organize a >>>>>> position and discussion. We can schedule telecons to go over >>>>>> those >>>>>> issues. The first item in the list is certainly in this class. >>>>>> >>>>>> Who can organize telecons on #456. >>>>>> Would it be possible to organize a RMA meeting at SC? >>>>>> >>>>>> I will be there Monday through part of Thursday but am usually >>>>>> triple-booked from 8 AM to midnight. >>>>>> >>>>>> >>>>>> >>>>>> The position expressed by the solution #456 is based on the idea >>>>>> that the MPI RMA synchronization routines should have the same >>>>>> outcome when RMA PUT and GET calls are substituted by stores and >>>>>> loads. >>>>>> >>>>>> The outcome for the flush routines is still not defined. >>>>>> >>>>>> It is interesting, because the standard is actually conflicting >>>>>> on >>>>>> whether Flush affects load-store. I find this incredibly >>>>>> frustrating. >>>>>> >>>>>> Page 450: >>>>>> >>>>>> "Locally completes at the origin all outstanding RMA operations >>>>>> initiated by the calling process to the target process specified >>>>>> by >>>>>> rank on the specified window. For example, after this routine >>>>>> completes, the user may reuse any buffers provided to put, get, >>>>>> or >>>>>> accumulate operations." >>>>>> >>>>>> I do not not think "RMA operations" includes load-store. >>>>>> >>>>>> Page 410: >>>>>> >>>>>> "The consistency of load/store accesses from/to the shared >>>>>> memory >>>>>> as >>>>>> observed by the user program depends on the architecture. A >>>>>> consistent >>>>>> view can be created in the unified memory model (see Section >>>>>> 11.4) >>>>>> by >>>>>> utilizing the window synchronization functions (see Section >>>>>> 11.5) >>>>>> or >>>>>> explicitly completing outstanding store accesses (e.g., by >>>>>> calling >>>>>> MPI_WIN_FLUSH)." >>>>>> >>>>>> Here it is unambiguously implied that MPI_WIN_FLUSH affects >>>>>> load-stores. >>>>>> >>>>>> My preference is to fix the statement on 410 since it is less >>>>>> canonical than the one on 450, and because I do not want to have >>>>>> a >>>>>> memory barrier in every call to WIN_FLUSH. >>>>>> >>>>>> Jeff >>>>>> >>>>>> >>>>>> >>>>>> I would prefere to have an organizer of the discussion inside of >>>>>> the RMA subgroup that proposed the changes for MPI-3.1 >>>>>> rather that I'm the organizer. >>>>>> I tried to bring all the input together and hope that #456 >>>>>> is now state that it is consistent itsself and with the >>>>>> expectations expressed by the group that published the >>>>>> paper at EuroMPI on first usage of this shared memory interface. >>>>>> >>>>>> The ticket is (together with the help of recent C11 >>>>>> standadization) >>>>>> on a good way to be also consistent with the compiler >>>>>> optimizations - >>>>>> in other words - the C standardization body has learnt from the >>>>>> pthreads problems. Fortran is still an open question to me, >>>>>> i.e., I do not know the status, see >>>>>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/456#comment:13 >>>>>> >>>>>> Best regards >>>>>> Rolf >>>>>> >>>>>> >>>>>> >>>>>> ----- Original Message ----- >>>>>> >>>>>> >>>>>> From: "William Gropp" <wgropp(a)illinois.edu> >>>>>> To: "MPI WG Remote Memory Access working group" >>>>>> <mpiwg-rma(a)lists.mpi-forum.org> >>>>>> Sent: Thursday, September 25, 2014 4:19:14 PM >>>>>> Subject: [mpiwg-rma] MPI RMA status summary >>>>>> >>>>>> I looked through all of the tickets and wrote a summary of the >>>>>> open >>>>>> issues, which I’ve attached. I propose the following: >>>>>> >>>>>> Determine which of these issues can be resolved by email. A >>>>>> significant number can probably be closed with no further >>>>>> action. >>>>>> >>>>>> For those requiring rework, determine if there is still interest >>>>>> in >>>>>> them, and if not, close them as well. >>>>>> >>>>>> For the ones requiring discussion, assign someone to organize a >>>>>> position and discussion. We can schedule telecons to go over >>>>>> those >>>>>> issues. The first item in the list is certainly in this class. >>>>>> >>>>>> Comments? >>>>>> >>>>>> Bill >>>>>> >>>>>> _______________________________________________ >>>>>> mpiwg-rma mailing list >>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>> >>>>>> -- >>>>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email >>>>>> rabenseifner(a)hlrs.de >>>>>> High Performance Computing Center (HLRS) . phone >>>>>> ++49(0)711/685-65530 >>>>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / >>>>>> 685-65832 >>>>>> Head of Dpmt Parallel Computing . . . >>>>>> www.hlrs.de/people/rabenseifner >>>>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room >>>>>> 1.307) >>>>>> _______________________________________________ >>>>>> mpiwg-rma mailing list >>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Jeff Hammond >>>>>> jeff.science(a)gmail.com >>>>>> http://jeffhammond.github.io/ >>>>>> _______________________________________________ >>>>>> mpiwg-rma mailing list >>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>> >>>>>> _______________________________________________ >>>>>> mpiwg-rma mailing list >>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>> >>>>> -- >>>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email >>>>> rabenseifner(a)hlrs.de >>>>> High Performance Computing Center (HLRS) . phone >>>>> ++49(0)711/685-65530 >>>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / >>>>> 685-65832 >>>>> Head of Dpmt Parallel Computing . . . >>>>> www.hlrs.de/people/rabenseifner >>>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room >>>>> 1.307) >>>>> _______________________________________________ >>>>> mpiwg-rma mailing list >>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>> >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>> >>> -- >>> Dr. Rolf Rabenseifner . . . . . . . . . .. email >>> rabenseifner(a)hlrs.de >>> High Performance Computing Center (HLRS) . phone >>> ++49(0)711/685-65530 >>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / >>> 685-65832 >>> Head of Dpmt Parallel Computing . . . >>> www.hlrs.de/people/rabenseifner >>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room >>> 1.307) >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] MPI RMA status summary
by William Gropp 30 Sep '14

30 Sep '14

The current straw vote question is only about memory synchronization. We are not discussing whether the RMA routines provide interprocess synchronization when applied to shared memory windows. That is a separate question that we will discuss later. Bill On Sep 30, 2014, at 9:35 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > There are different ways to look at it. > Remote memory access like remote load and store can > be interpreted as RMA by one reader and non-RMA by another one. > > Torsten Hoefler, James Dinan, Darius Buntinas, Pavan Balaji, Brian Barrett, > Ron Brightwell, William Gropp, Vivek Kale, Rajeev Thakur > in > "MPI + MPI: a new hybrid approach to parallel programming with MPI plus shared Memory" > at EuroMPI 2012 > obviously treated them as RMA, because otherwise their store+Fence+load would > have been wrong. > > The major reason for going with b) and not with a) is simple: > > If somenbody does not want to have a shared memory > synchronization semantics for an existing MPI_Win > synchronization routine (e.g. MPI_Win_fence), then > he/she Need not to use the Routine together with > shared memory windows. > If he/she wants to stay with MP_Put/Get, > then he/she needs not to use MPI_Win_allocate_shared. > It is allowed to use MPI_Win_allocate on a shered memory node > and MPI internally can do the same optimizations. > > Therefore shared memory synchronization semantics > for MPI_Win_fence, MPI_Win_post/start/complete/wait, > MPI_Win_lock/unlock, ... > is never a drawback. > > But it is a clear advantage, because process-to-process > synchronization is combined with local memory synchronization > which may be implemented faster than if seperated into > different routines. > > Rolf > > > ----- Original Message ----- >> From: "Jeff Hammond" <jeff.science(a)gmail.com> >> To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> >> Sent: Tuesday, September 30, 2014 4:09:02 PM >> Subject: Re: [mpiwg-rma] MPI RMA status summary >> >> Option A is what the standard says today outside of a sloppy offhand >> remark in parentheses. See my note please. >> >> Jeff >> >> Sent from my iPhone >> >>> On Sep 30, 2014, at 7:05 AM, Rolf Rabenseifner >>> <rabenseifner(a)hlrs.de> wrote: >>> >>> I strongly agree with your statement: >>>> These piecemeal changes are one of the sources of our problems. >>> >>> I only wanted to strongly say, that I would never vote for your a), >>> because it is not backward compatible to what is already used. >>> And with b) I've the problem that b1) is clear to me (see #456), >>> but the Win_flush semantics for load/store is unclear to me. >>> >>> Of course, a total solution is needed and not parts of it. >>> #456 is such a trial for a complete solution. >>> >>> Rolf >>> >>> ----- Original Message ----- >>>> From: "William Gropp" <wgropp(a)illinois.edu> >>>> To: "MPI WG Remote Memory Access working group" >>>> <mpiwg-rma(a)lists.mpi-forum.org> >>>> Sent: Tuesday, September 30, 2014 3:19:06 PM >>>> Subject: Re: [mpiwg-rma] MPI RMA status summary >>>> >>>> I disagree with this approach. The most important thing to do is >>>> to >>>> figure out the correct definitions and semantics. Once we agree >>>> on >>>> that, we can determine what can be handled as an errata and what >>>> will require an update to the chapter and an update to the MPI >>>> standard. These piecemeal changes are one of the sources of our >>>> problems. >>>> >>>> Bill >>>> >>>> On Sep 30, 2014, at 7:38 AM, Rolf Rabenseifner >>>> <rabenseifner(a)hlrs.de> >>>> wrote: >>>> >>>>>> Vote for 1 of the following: >>>>>> >>>>>> a) Only Win_sync provides memory barrier semantics to shared >>>>>> memory >>>>>> windows >>>>>> b) All RMA completion/sync routines (e.g., MPI_Win_lock, >>>>>> MPI_Win_fence, MPI_Win_flush) provide memory barrier semantics >>>>>> c) Some as yet undetermined blend of a and b, which might >>>>>> include >>>>>> additional asserts >>>>>> d) This topic needs further discussion >>>>> >>>>> Because we only have to clarify MPI-3.0 (this is an errata issue) >>>>> and >>>>> - obviously the MPI Forum and the readers expected that >>>>> MPI_Win_fence >>>>> (and therefore also the other MPI-2 synchronizations >>>>> MPI_Win_post/start/complete/wait and MPI_Win_lock/unlock) >>>>> works if MPI_Get/Put are sustituted by shared memory load/store >>>>> (see the many Forum members as authors of the EuroMPI paper) >>>>> - and the Forum decided that also MPI_Win_sync acts as if >>>>> a memory barrier is inside, >>>>> for me, >>>>> - a) cannot be chosen bacause an erratum cannot remove >>>>> a given functionality >>>>> - and b) is automatically given, see reasons above. Therefore >>>>> #456. >>>>> >>>>> The only open question for me is about the meaning of >>>>> MPI_Win_flush. >>>>> Therefore MPI_Win_flush is still missing in #456. >>>>> >>>>> Therefore for me, the Major choices seems to be >>>>> b1) MPI-2 synchronizations + MPI_Win_sync >>>>> b2) MPI-2 synchronizations + MPI_Win_sync + MPI_Win_flush >>>>> >>>>> For this vote, I clearly want to see a clear proposal >>>>> about the meaning of MPI_Win_flush together with >>>>> sared memory load/store, hopefully with the notation >>>>> used in #456. >>>>> >>>>> Best regards >>>>> Rolf >>>>> >>>>> ----- Original Message ----- >>>>>> From: "William Gropp" <wgropp(a)illinois.edu> >>>>>> To: "MPI WG Remote Memory Access working group" >>>>>> <mpiwg-rma(a)lists.mpi-forum.org> >>>>>> Sent: Monday, September 29, 2014 11:39:51 PM >>>>>> Subject: Re: [mpiwg-rma] MPI RMA status summary >>>>>> >>>>>> >>>>>> Thanks, Jeff. >>>>>> >>>>>> >>>>>> I agree that I don’t want load/store to be considered RMA >>>>>> operations. >>>>>> But the issue of the memory consistency on RMA synchronization >>>>>> and >>>>>> completion operations to a shared memory window is complex. In >>>>>> some >>>>>> ways, the most consistent with RMA in other situations is the >>>>>> case >>>>>> of MPI_Win_lock to your own process; the easiest extension for >>>>>> the >>>>>> user is to have reasonably strong memory barrier semantics at >>>>>> all >>>>>> sync/completion operations (thus including Fence). As you note, >>>>>> this has costs. At the other extreme, we could say that only >>>>>> Win_sync provides these memory barrier semantics. And we could >>>>>> pick >>>>>> a more complex blend (yes for some, no for others). >>>>>> >>>>>> >>>>>> One of the first questions is whether we want to only Win_sync, >>>>>> all >>>>>> completion/sync RMA routines, or some subset to provide memory >>>>>> barrier semantics for shared memory windows (this would include >>>>>> RMA >>>>>> windows that claimed to be shared memory, since there is a >>>>>> proposal >>>>>> to extend that property to other RMA windows). It would be good >>>>>> to >>>>>> make progress on this question, so I propose a straw vote of >>>>>> this >>>>>> group by email. Vote for 1 of the following: >>>>>> >>>>>> >>>>>> a) Only Win_sync provides memory barrier semantics to shared >>>>>> memory >>>>>> windows >>>>>> b) All RMA completion/sync routines (e.g., MPI_Win_lock, >>>>>> MPI_Win_fence, MPI_Win_flush) provide memory barrier semantics >>>>>> c) Some as yet undetermined blend of a and b, which might >>>>>> include >>>>>> additional asserts >>>>>> d) This topic needs further discussion >>>>>> >>>>>> >>>>>> Note that I’ve left off what “memory barrier semantics” means. >>>>>> That >>>>>> will need to be precisely defined for the standard, but I >>>>>> believe >>>>>> we >>>>>> know what we intend for this. We specifically are not defining >>>>>> what >>>>>> happens with non-MPI code. Also note that this is separate from >>>>>> whether the RMA sync routines appear to be blocking when applied >>>>>> to >>>>>> a shared memory window; we can do a separate straw vote on that. >>>>>> >>>>>> >>>>>> Bill >>>>>> >>>>>> >>>>>> >>>>>> On Sep 29, 2014, at 3:49 PM, Jeff Hammond < >>>>>> jeff.science(a)gmail.com >>>>>> wrote: >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Sep 29, 2014 at 9:16 AM, Rolf Rabenseifner < >>>>>> rabenseifner(a)hlrs.de > wrote: >>>>>> >>>>>> >>>>>> Only about the issues on #456 (shared memory syncronization): >>>>>> >>>>>> >>>>>> >>>>>> For the ones requiring discussion, assign someone to organize a >>>>>> position and discussion. We can schedule telecons to go over >>>>>> those >>>>>> issues. The first item in the list is certainly in this class. >>>>>> >>>>>> Who can organize telecons on #456. >>>>>> Would it be possible to organize a RMA meeting at SC? >>>>>> >>>>>> I will be there Monday through part of Thursday but am usually >>>>>> triple-booked from 8 AM to midnight. >>>>>> >>>>>> >>>>>> >>>>>> The position expressed by the solution #456 is based on the idea >>>>>> that the MPI RMA synchronization routines should have the same >>>>>> outcome when RMA PUT and GET calls are substituted by stores and >>>>>> loads. >>>>>> >>>>>> The outcome for the flush routines is still not defined. >>>>>> >>>>>> It is interesting, because the standard is actually conflicting >>>>>> on >>>>>> whether Flush affects load-store. I find this incredibly >>>>>> frustrating. >>>>>> >>>>>> Page 450: >>>>>> >>>>>> "Locally completes at the origin all outstanding RMA operations >>>>>> initiated by the calling process to the target process specified >>>>>> by >>>>>> rank on the specified window. For example, after this routine >>>>>> completes, the user may reuse any buffers provided to put, get, >>>>>> or >>>>>> accumulate operations." >>>>>> >>>>>> I do not not think "RMA operations" includes load-store. >>>>>> >>>>>> Page 410: >>>>>> >>>>>> "The consistency of load/store accesses from/to the shared >>>>>> memory >>>>>> as >>>>>> observed by the user program depends on the architecture. A >>>>>> consistent >>>>>> view can be created in the unified memory model (see Section >>>>>> 11.4) >>>>>> by >>>>>> utilizing the window synchronization functions (see Section >>>>>> 11.5) >>>>>> or >>>>>> explicitly completing outstanding store accesses (e.g., by >>>>>> calling >>>>>> MPI_WIN_FLUSH)." >>>>>> >>>>>> Here it is unambiguously implied that MPI_WIN_FLUSH affects >>>>>> load-stores. >>>>>> >>>>>> My preference is to fix the statement on 410 since it is less >>>>>> canonical than the one on 450, and because I do not want to have >>>>>> a >>>>>> memory barrier in every call to WIN_FLUSH. >>>>>> >>>>>> Jeff >>>>>> >>>>>> >>>>>> >>>>>> I would prefere to have an organizer of the discussion inside of >>>>>> the RMA subgroup that proposed the changes for MPI-3.1 >>>>>> rather that I'm the organizer. >>>>>> I tried to bring all the input together and hope that #456 >>>>>> is now state that it is consistent itsself and with the >>>>>> expectations expressed by the group that published the >>>>>> paper at EuroMPI on first usage of this shared memory interface. >>>>>> >>>>>> The ticket is (together with the help of recent C11 >>>>>> standadization) >>>>>> on a good way to be also consistent with the compiler >>>>>> optimizations - >>>>>> in other words - the C standardization body has learnt from the >>>>>> pthreads problems. Fortran is still an open question to me, >>>>>> i.e., I do not know the status, see >>>>>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/456#comment:13 >>>>>> >>>>>> Best regards >>>>>> Rolf >>>>>> >>>>>> >>>>>> >>>>>> ----- Original Message ----- >>>>>> >>>>>> >>>>>> From: "William Gropp" <wgropp(a)illinois.edu> >>>>>> To: "MPI WG Remote Memory Access working group" >>>>>> <mpiwg-rma(a)lists.mpi-forum.org> >>>>>> Sent: Thursday, September 25, 2014 4:19:14 PM >>>>>> Subject: [mpiwg-rma] MPI RMA status summary >>>>>> >>>>>> I looked through all of the tickets and wrote a summary of the >>>>>> open >>>>>> issues, which I’ve attached. I propose the following: >>>>>> >>>>>> Determine which of these issues can be resolved by email. A >>>>>> significant number can probably be closed with no further >>>>>> action. >>>>>> >>>>>> For those requiring rework, determine if there is still interest >>>>>> in >>>>>> them, and if not, close them as well. >>>>>> >>>>>> For the ones requiring discussion, assign someone to organize a >>>>>> position and discussion. We can schedule telecons to go over >>>>>> those >>>>>> issues. The first item in the list is certainly in this class. >>>>>> >>>>>> Comments? >>>>>> >>>>>> Bill >>>>>> >>>>>> _______________________________________________ >>>>>> mpiwg-rma mailing list >>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>> >>>>>> -- >>>>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email >>>>>> rabenseifner(a)hlrs.de >>>>>> High Performance Computing Center (HLRS) . phone >>>>>> ++49(0)711/685-65530 >>>>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / >>>>>> 685-65832 >>>>>> Head of Dpmt Parallel Computing . . . >>>>>> www.hlrs.de/people/rabenseifner >>>>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room >>>>>> 1.307) >>>>>> _______________________________________________ >>>>>> mpiwg-rma mailing list >>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Jeff Hammond >>>>>> jeff.science(a)gmail.com >>>>>> http://jeffhammond.github.io/ >>>>>> _______________________________________________ >>>>>> mpiwg-rma mailing list >>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>> >>>>>> _______________________________________________ >>>>>> mpiwg-rma mailing list >>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>> >>>>> -- >>>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email >>>>> rabenseifner(a)hlrs.de >>>>> High Performance Computing Center (HLRS) . phone >>>>> ++49(0)711/685-65530 >>>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / >>>>> 685-65832 >>>>> Head of Dpmt Parallel Computing . . . >>>>> www.hlrs.de/people/rabenseifner >>>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room >>>>> 1.307) >>>>> _______________________________________________ >>>>> mpiwg-rma mailing list >>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>> >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>> >>> -- >>> Dr. Rolf Rabenseifner . . . . . . . . . .. email >>> rabenseifner(a)hlrs.de >>> High Performance Computing Center (HLRS) . phone >>> ++49(0)711/685-65530 >>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / >>> 685-65832 >>> Head of Dpmt Parallel Computing . . . >>> www.hlrs.de/people/rabenseifner >>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room >>> 1.307) >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] MPI RMA status summary
by Rolf Rabenseifner 30 Sep '14

30 Sep '14

There are different ways to look at it. Remote memory access like remote load and store can be interpreted as RMA by one reader and non-RMA by another one. Torsten Hoefler, James Dinan, Darius Buntinas, Pavan Balaji, Brian Barrett, Ron Brightwell, William Gropp, Vivek Kale, Rajeev Thakur in "MPI + MPI: a new hybrid approach to parallel programming with MPI plus shared Memory" at EuroMPI 2012 obviously treated them as RMA, because otherwise their store+Fence+load would have been wrong. The major reason for going with b) and not with a) is simple: If somenbody does not want to have a shared memory synchronization semantics for an existing MPI_Win synchronization routine (e.g. MPI_Win_fence), then he/she Need not to use the Routine together with shared memory windows. If he/she wants to stay with MP_Put/Get, then he/she needs not to use MPI_Win_allocate_shared. It is allowed to use MPI_Win_allocate on a shered memory node and MPI internally can do the same optimizations. Therefore shared memory synchronization semantics for MPI_Win_fence, MPI_Win_post/start/complete/wait, MPI_Win_lock/unlock, ... is never a drawback. But it is a clear advantage, because process-to-process synchronization is combined with local memory synchronization which may be implemented faster than if seperated into different routines. Rolf ----- Original Message ----- > From: "Jeff Hammond" <jeff.science(a)gmail.com> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Tuesday, September 30, 2014 4:09:02 PM > Subject: Re: [mpiwg-rma] MPI RMA status summary > > Option A is what the standard says today outside of a sloppy offhand > remark in parentheses. See my note please. > > Jeff > > Sent from my iPhone > > > On Sep 30, 2014, at 7:05 AM, Rolf Rabenseifner > > <rabenseifner(a)hlrs.de> wrote: > > > > I strongly agree with your statement: > >> These piecemeal changes are one of the sources of our problems. > > > > I only wanted to strongly say, that I would never vote for your a), > > because it is not backward compatible to what is already used. > > And with b) I've the problem that b1) is clear to me (see #456), > > but the Win_flush semantics for load/store is unclear to me. > > > > Of course, a total solution is needed and not parts of it. > > #456 is such a trial for a complete solution. > > > > Rolf > > > > ----- Original Message ----- > >> From: "William Gropp" <wgropp(a)illinois.edu> > >> To: "MPI WG Remote Memory Access working group" > >> <mpiwg-rma(a)lists.mpi-forum.org> > >> Sent: Tuesday, September 30, 2014 3:19:06 PM > >> Subject: Re: [mpiwg-rma] MPI RMA status summary > >> > >> I disagree with this approach. The most important thing to do is > >> to > >> figure out the correct definitions and semantics. Once we agree > >> on > >> that, we can determine what can be handled as an errata and what > >> will require an update to the chapter and an update to the MPI > >> standard. These piecemeal changes are one of the sources of our > >> problems. > >> > >> Bill > >> > >> On Sep 30, 2014, at 7:38 AM, Rolf Rabenseifner > >> <rabenseifner(a)hlrs.de> > >> wrote: > >> > >>>> Vote for 1 of the following: > >>>> > >>>> a) Only Win_sync provides memory barrier semantics to shared > >>>> memory > >>>> windows > >>>> b) All RMA completion/sync routines (e.g., MPI_Win_lock, > >>>> MPI_Win_fence, MPI_Win_flush) provide memory barrier semantics > >>>> c) Some as yet undetermined blend of a and b, which might > >>>> include > >>>> additional asserts > >>>> d) This topic needs further discussion > >>> > >>> Because we only have to clarify MPI-3.0 (this is an errata issue) > >>> and > >>> - obviously the MPI Forum and the readers expected that > >>> MPI_Win_fence > >>> (and therefore also the other MPI-2 synchronizations > >>> MPI_Win_post/start/complete/wait and MPI_Win_lock/unlock) > >>> works if MPI_Get/Put are sustituted by shared memory load/store > >>> (see the many Forum members as authors of the EuroMPI paper) > >>> - and the Forum decided that also MPI_Win_sync acts as if > >>> a memory barrier is inside, > >>> for me, > >>> - a) cannot be chosen bacause an erratum cannot remove > >>> a given functionality > >>> - and b) is automatically given, see reasons above. Therefore > >>> #456. > >>> > >>> The only open question for me is about the meaning of > >>> MPI_Win_flush. > >>> Therefore MPI_Win_flush is still missing in #456. > >>> > >>> Therefore for me, the Major choices seems to be > >>> b1) MPI-2 synchronizations + MPI_Win_sync > >>> b2) MPI-2 synchronizations + MPI_Win_sync + MPI_Win_flush > >>> > >>> For this vote, I clearly want to see a clear proposal > >>> about the meaning of MPI_Win_flush together with > >>> sared memory load/store, hopefully with the notation > >>> used in #456. > >>> > >>> Best regards > >>> Rolf > >>> > >>> ----- Original Message ----- > >>>> From: "William Gropp" <wgropp(a)illinois.edu> > >>>> To: "MPI WG Remote Memory Access working group" > >>>> <mpiwg-rma(a)lists.mpi-forum.org> > >>>> Sent: Monday, September 29, 2014 11:39:51 PM > >>>> Subject: Re: [mpiwg-rma] MPI RMA status summary > >>>> > >>>> > >>>> Thanks, Jeff. > >>>> > >>>> > >>>> I agree that I don’t want load/store to be considered RMA > >>>> operations. > >>>> But the issue of the memory consistency on RMA synchronization > >>>> and > >>>> completion operations to a shared memory window is complex. In > >>>> some > >>>> ways, the most consistent with RMA in other situations is the > >>>> case > >>>> of MPI_Win_lock to your own process; the easiest extension for > >>>> the > >>>> user is to have reasonably strong memory barrier semantics at > >>>> all > >>>> sync/completion operations (thus including Fence). As you note, > >>>> this has costs. At the other extreme, we could say that only > >>>> Win_sync provides these memory barrier semantics. And we could > >>>> pick > >>>> a more complex blend (yes for some, no for others). > >>>> > >>>> > >>>> One of the first questions is whether we want to only Win_sync, > >>>> all > >>>> completion/sync RMA routines, or some subset to provide memory > >>>> barrier semantics for shared memory windows (this would include > >>>> RMA > >>>> windows that claimed to be shared memory, since there is a > >>>> proposal > >>>> to extend that property to other RMA windows). It would be good > >>>> to > >>>> make progress on this question, so I propose a straw vote of > >>>> this > >>>> group by email. Vote for 1 of the following: > >>>> > >>>> > >>>> a) Only Win_sync provides memory barrier semantics to shared > >>>> memory > >>>> windows > >>>> b) All RMA completion/sync routines (e.g., MPI_Win_lock, > >>>> MPI_Win_fence, MPI_Win_flush) provide memory barrier semantics > >>>> c) Some as yet undetermined blend of a and b, which might > >>>> include > >>>> additional asserts > >>>> d) This topic needs further discussion > >>>> > >>>> > >>>> Note that I’ve left off what “memory barrier semantics” means. > >>>> That > >>>> will need to be precisely defined for the standard, but I > >>>> believe > >>>> we > >>>> know what we intend for this. We specifically are not defining > >>>> what > >>>> happens with non-MPI code. Also note that this is separate from > >>>> whether the RMA sync routines appear to be blocking when applied > >>>> to > >>>> a shared memory window; we can do a separate straw vote on that. > >>>> > >>>> > >>>> Bill > >>>> > >>>> > >>>> > >>>> On Sep 29, 2014, at 3:49 PM, Jeff Hammond < > >>>> jeff.science(a)gmail.com > >>>> wrote: > >>>> > >>>> > >>>> > >>>> On Mon, Sep 29, 2014 at 9:16 AM, Rolf Rabenseifner < > >>>> rabenseifner(a)hlrs.de > wrote: > >>>> > >>>> > >>>> Only about the issues on #456 (shared memory syncronization): > >>>> > >>>> > >>>> > >>>> For the ones requiring discussion, assign someone to organize a > >>>> position and discussion. We can schedule telecons to go over > >>>> those > >>>> issues. The first item in the list is certainly in this class. > >>>> > >>>> Who can organize telecons on #456. > >>>> Would it be possible to organize a RMA meeting at SC? > >>>> > >>>> I will be there Monday through part of Thursday but am usually > >>>> triple-booked from 8 AM to midnight. > >>>> > >>>> > >>>> > >>>> The position expressed by the solution #456 is based on the idea > >>>> that the MPI RMA synchronization routines should have the same > >>>> outcome when RMA PUT and GET calls are substituted by stores and > >>>> loads. > >>>> > >>>> The outcome for the flush routines is still not defined. > >>>> > >>>> It is interesting, because the standard is actually conflicting > >>>> on > >>>> whether Flush affects load-store. I find this incredibly > >>>> frustrating. > >>>> > >>>> Page 450: > >>>> > >>>> "Locally completes at the origin all outstanding RMA operations > >>>> initiated by the calling process to the target process specified > >>>> by > >>>> rank on the specified window. For example, after this routine > >>>> completes, the user may reuse any buffers provided to put, get, > >>>> or > >>>> accumulate operations." > >>>> > >>>> I do not not think "RMA operations" includes load-store. > >>>> > >>>> Page 410: > >>>> > >>>> "The consistency of load/store accesses from/to the shared > >>>> memory > >>>> as > >>>> observed by the user program depends on the architecture. A > >>>> consistent > >>>> view can be created in the unified memory model (see Section > >>>> 11.4) > >>>> by > >>>> utilizing the window synchronization functions (see Section > >>>> 11.5) > >>>> or > >>>> explicitly completing outstanding store accesses (e.g., by > >>>> calling > >>>> MPI_WIN_FLUSH)." > >>>> > >>>> Here it is unambiguously implied that MPI_WIN_FLUSH affects > >>>> load-stores. > >>>> > >>>> My preference is to fix the statement on 410 since it is less > >>>> canonical than the one on 450, and because I do not want to have > >>>> a > >>>> memory barrier in every call to WIN_FLUSH. > >>>> > >>>> Jeff > >>>> > >>>> > >>>> > >>>> I would prefere to have an organizer of the discussion inside of > >>>> the RMA subgroup that proposed the changes for MPI-3.1 > >>>> rather that I'm the organizer. > >>>> I tried to bring all the input together and hope that #456 > >>>> is now state that it is consistent itsself and with the > >>>> expectations expressed by the group that published the > >>>> paper at EuroMPI on first usage of this shared memory interface. > >>>> > >>>> The ticket is (together with the help of recent C11 > >>>> standadization) > >>>> on a good way to be also consistent with the compiler > >>>> optimizations - > >>>> in other words - the C standardization body has learnt from the > >>>> pthreads problems. Fortran is still an open question to me, > >>>> i.e., I do not know the status, see > >>>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/456#comment:13 > >>>> > >>>> Best regards > >>>> Rolf > >>>> > >>>> > >>>> > >>>> ----- Original Message ----- > >>>> > >>>> > >>>> From: "William Gropp" <wgropp(a)illinois.edu> > >>>> To: "MPI WG Remote Memory Access working group" > >>>> <mpiwg-rma(a)lists.mpi-forum.org> > >>>> Sent: Thursday, September 25, 2014 4:19:14 PM > >>>> Subject: [mpiwg-rma] MPI RMA status summary > >>>> > >>>> I looked through all of the tickets and wrote a summary of the > >>>> open > >>>> issues, which I’ve attached. I propose the following: > >>>> > >>>> Determine which of these issues can be resolved by email. A > >>>> significant number can probably be closed with no further > >>>> action. > >>>> > >>>> For those requiring rework, determine if there is still interest > >>>> in > >>>> them, and if not, close them as well. > >>>> > >>>> For the ones requiring discussion, assign someone to organize a > >>>> position and discussion. We can schedule telecons to go over > >>>> those > >>>> issues. The first item in the list is certainly in this class. > >>>> > >>>> Comments? > >>>> > >>>> Bill > >>>> > >>>> _______________________________________________ > >>>> mpiwg-rma mailing list > >>>> mpiwg-rma(a)lists.mpi-forum.org > >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>>> > >>>> -- > >>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email > >>>> rabenseifner(a)hlrs.de > >>>> High Performance Computing Center (HLRS) . phone > >>>> ++49(0)711/685-65530 > >>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > >>>> 685-65832 > >>>> Head of Dpmt Parallel Computing . . . > >>>> www.hlrs.de/people/rabenseifner > >>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room > >>>> 1.307) > >>>> _______________________________________________ > >>>> mpiwg-rma mailing list > >>>> mpiwg-rma(a)lists.mpi-forum.org > >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>>> > >>>> > >>>> > >>>> -- > >>>> Jeff Hammond > >>>> jeff.science(a)gmail.com > >>>> http://jeffhammond.github.io/ > >>>> _______________________________________________ > >>>> mpiwg-rma mailing list > >>>> mpiwg-rma(a)lists.mpi-forum.org > >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>>> > >>>> _______________________________________________ > >>>> mpiwg-rma mailing list > >>>> mpiwg-rma(a)lists.mpi-forum.org > >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>> > >>> -- > >>> Dr. Rolf Rabenseifner . . . . . . . . . .. email > >>> rabenseifner(a)hlrs.de > >>> High Performance Computing Center (HLRS) . phone > >>> ++49(0)711/685-65530 > >>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > >>> 685-65832 > >>> Head of Dpmt Parallel Computing . . . > >>> www.hlrs.de/people/rabenseifner > >>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room > >>> 1.307) > >>> _______________________________________________ > >>> mpiwg-rma mailing list > >>> mpiwg-rma(a)lists.mpi-forum.org > >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >> > >> _______________________________________________ > >> mpiwg-rma mailing list > >> mpiwg-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > -- > > Dr. Rolf Rabenseifner . . . . . . . . . .. email > > rabenseifner(a)hlrs.de > > High Performance Computing Center (HLRS) . phone > > ++49(0)711/685-65530 > > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > > 685-65832 > > Head of Dpmt Parallel Computing . . . > > www.hlrs.de/people/rabenseifner > > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room > > 1.307) > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] MPI RMA status summary
by Jeff Hammond 30 Sep '14

30 Sep '14

Option A is what the standard says today outside of a sloppy offhand remark in parentheses. See my note please. Jeff Sent from my iPhone > On Sep 30, 2014, at 7:05 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > > I strongly agree with your statement: >> These piecemeal changes are one of the sources of our problems. > > I only wanted to strongly say, that I would never vote for your a), > because it is not backward compatible to what is already used. > And with b) I've the problem that b1) is clear to me (see #456), > but the Win_flush semantics for load/store is unclear to me. > > Of course, a total solution is needed and not parts of it. > #456 is such a trial for a complete solution. > > Rolf > > ----- Original Message ----- >> From: "William Gropp" <wgropp(a)illinois.edu> >> To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> >> Sent: Tuesday, September 30, 2014 3:19:06 PM >> Subject: Re: [mpiwg-rma] MPI RMA status summary >> >> I disagree with this approach. The most important thing to do is to >> figure out the correct definitions and semantics. Once we agree on >> that, we can determine what can be handled as an errata and what >> will require an update to the chapter and an update to the MPI >> standard. These piecemeal changes are one of the sources of our >> problems. >> >> Bill >> >> On Sep 30, 2014, at 7:38 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> >> wrote: >> >>>> Vote for 1 of the following: >>>> >>>> a) Only Win_sync provides memory barrier semantics to shared >>>> memory >>>> windows >>>> b) All RMA completion/sync routines (e.g., MPI_Win_lock, >>>> MPI_Win_fence, MPI_Win_flush) provide memory barrier semantics >>>> c) Some as yet undetermined blend of a and b, which might include >>>> additional asserts >>>> d) This topic needs further discussion >>> >>> Because we only have to clarify MPI-3.0 (this is an errata issue) >>> and >>> - obviously the MPI Forum and the readers expected that >>> MPI_Win_fence >>> (and therefore also the other MPI-2 synchronizations >>> MPI_Win_post/start/complete/wait and MPI_Win_lock/unlock) >>> works if MPI_Get/Put are sustituted by shared memory load/store >>> (see the many Forum members as authors of the EuroMPI paper) >>> - and the Forum decided that also MPI_Win_sync acts as if >>> a memory barrier is inside, >>> for me, >>> - a) cannot be chosen bacause an erratum cannot remove >>> a given functionality >>> - and b) is automatically given, see reasons above. Therefore #456. >>> >>> The only open question for me is about the meaning of >>> MPI_Win_flush. >>> Therefore MPI_Win_flush is still missing in #456. >>> >>> Therefore for me, the Major choices seems to be >>> b1) MPI-2 synchronizations + MPI_Win_sync >>> b2) MPI-2 synchronizations + MPI_Win_sync + MPI_Win_flush >>> >>> For this vote, I clearly want to see a clear proposal >>> about the meaning of MPI_Win_flush together with >>> sared memory load/store, hopefully with the notation >>> used in #456. >>> >>> Best regards >>> Rolf >>> >>> ----- Original Message ----- >>>> From: "William Gropp" <wgropp(a)illinois.edu> >>>> To: "MPI WG Remote Memory Access working group" >>>> <mpiwg-rma(a)lists.mpi-forum.org> >>>> Sent: Monday, September 29, 2014 11:39:51 PM >>>> Subject: Re: [mpiwg-rma] MPI RMA status summary >>>> >>>> >>>> Thanks, Jeff. >>>> >>>> >>>> I agree that I don’t want load/store to be considered RMA >>>> operations. >>>> But the issue of the memory consistency on RMA synchronization >>>> and >>>> completion operations to a shared memory window is complex. In >>>> some >>>> ways, the most consistent with RMA in other situations is the case >>>> of MPI_Win_lock to your own process; the easiest extension for the >>>> user is to have reasonably strong memory barrier semantics at all >>>> sync/completion operations (thus including Fence). As you note, >>>> this has costs. At the other extreme, we could say that only >>>> Win_sync provides these memory barrier semantics. And we could >>>> pick >>>> a more complex blend (yes for some, no for others). >>>> >>>> >>>> One of the first questions is whether we want to only Win_sync, >>>> all >>>> completion/sync RMA routines, or some subset to provide memory >>>> barrier semantics for shared memory windows (this would include >>>> RMA >>>> windows that claimed to be shared memory, since there is a >>>> proposal >>>> to extend that property to other RMA windows). It would be good >>>> to >>>> make progress on this question, so I propose a straw vote of this >>>> group by email. Vote for 1 of the following: >>>> >>>> >>>> a) Only Win_sync provides memory barrier semantics to shared >>>> memory >>>> windows >>>> b) All RMA completion/sync routines (e.g., MPI_Win_lock, >>>> MPI_Win_fence, MPI_Win_flush) provide memory barrier semantics >>>> c) Some as yet undetermined blend of a and b, which might include >>>> additional asserts >>>> d) This topic needs further discussion >>>> >>>> >>>> Note that I’ve left off what “memory barrier semantics” means. >>>> That >>>> will need to be precisely defined for the standard, but I believe >>>> we >>>> know what we intend for this. We specifically are not defining >>>> what >>>> happens with non-MPI code. Also note that this is separate from >>>> whether the RMA sync routines appear to be blocking when applied >>>> to >>>> a shared memory window; we can do a separate straw vote on that. >>>> >>>> >>>> Bill >>>> >>>> >>>> >>>> On Sep 29, 2014, at 3:49 PM, Jeff Hammond < jeff.science(a)gmail.com >>>> wrote: >>>> >>>> >>>> >>>> On Mon, Sep 29, 2014 at 9:16 AM, Rolf Rabenseifner < >>>> rabenseifner(a)hlrs.de > wrote: >>>> >>>> >>>> Only about the issues on #456 (shared memory syncronization): >>>> >>>> >>>> >>>> For the ones requiring discussion, assign someone to organize a >>>> position and discussion. We can schedule telecons to go over >>>> those >>>> issues. The first item in the list is certainly in this class. >>>> >>>> Who can organize telecons on #456. >>>> Would it be possible to organize a RMA meeting at SC? >>>> >>>> I will be there Monday through part of Thursday but am usually >>>> triple-booked from 8 AM to midnight. >>>> >>>> >>>> >>>> The position expressed by the solution #456 is based on the idea >>>> that the MPI RMA synchronization routines should have the same >>>> outcome when RMA PUT and GET calls are substituted by stores and >>>> loads. >>>> >>>> The outcome for the flush routines is still not defined. >>>> >>>> It is interesting, because the standard is actually conflicting on >>>> whether Flush affects load-store. I find this incredibly >>>> frustrating. >>>> >>>> Page 450: >>>> >>>> "Locally completes at the origin all outstanding RMA operations >>>> initiated by the calling process to the target process specified >>>> by >>>> rank on the specified window. For example, after this routine >>>> completes, the user may reuse any buffers provided to put, get, or >>>> accumulate operations." >>>> >>>> I do not not think "RMA operations" includes load-store. >>>> >>>> Page 410: >>>> >>>> "The consistency of load/store accesses from/to the shared memory >>>> as >>>> observed by the user program depends on the architecture. A >>>> consistent >>>> view can be created in the unified memory model (see Section 11.4) >>>> by >>>> utilizing the window synchronization functions (see Section 11.5) >>>> or >>>> explicitly completing outstanding store accesses (e.g., by calling >>>> MPI_WIN_FLUSH)." >>>> >>>> Here it is unambiguously implied that MPI_WIN_FLUSH affects >>>> load-stores. >>>> >>>> My preference is to fix the statement on 410 since it is less >>>> canonical than the one on 450, and because I do not want to have a >>>> memory barrier in every call to WIN_FLUSH. >>>> >>>> Jeff >>>> >>>> >>>> >>>> I would prefere to have an organizer of the discussion inside of >>>> the RMA subgroup that proposed the changes for MPI-3.1 >>>> rather that I'm the organizer. >>>> I tried to bring all the input together and hope that #456 >>>> is now state that it is consistent itsself and with the >>>> expectations expressed by the group that published the >>>> paper at EuroMPI on first usage of this shared memory interface. >>>> >>>> The ticket is (together with the help of recent C11 >>>> standadization) >>>> on a good way to be also consistent with the compiler >>>> optimizations - >>>> in other words - the C standardization body has learnt from the >>>> pthreads problems. Fortran is still an open question to me, >>>> i.e., I do not know the status, see >>>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/456#comment:13 >>>> >>>> Best regards >>>> Rolf >>>> >>>> >>>> >>>> ----- Original Message ----- >>>> >>>> >>>> From: "William Gropp" <wgropp(a)illinois.edu> >>>> To: "MPI WG Remote Memory Access working group" >>>> <mpiwg-rma(a)lists.mpi-forum.org> >>>> Sent: Thursday, September 25, 2014 4:19:14 PM >>>> Subject: [mpiwg-rma] MPI RMA status summary >>>> >>>> I looked through all of the tickets and wrote a summary of the >>>> open >>>> issues, which I’ve attached. I propose the following: >>>> >>>> Determine which of these issues can be resolved by email. A >>>> significant number can probably be closed with no further action. >>>> >>>> For those requiring rework, determine if there is still interest >>>> in >>>> them, and if not, close them as well. >>>> >>>> For the ones requiring discussion, assign someone to organize a >>>> position and discussion. We can schedule telecons to go over >>>> those >>>> issues. The first item in the list is certainly in this class. >>>> >>>> Comments? >>>> >>>> Bill >>>> >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>> >>>> -- >>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email >>>> rabenseifner(a)hlrs.de >>>> High Performance Computing Center (HLRS) . phone >>>> ++49(0)711/685-65530 >>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / >>>> 685-65832 >>>> Head of Dpmt Parallel Computing . . . >>>> www.hlrs.de/people/rabenseifner >>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room >>>> 1.307) >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>> >>>> >>>> >>>> -- >>>> Jeff Hammond >>>> jeff.science(a)gmail.com >>>> http://jeffhammond.github.io/ >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>> >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>> >>> -- >>> Dr. Rolf Rabenseifner . . . . . . . . . .. email >>> rabenseifner(a)hlrs.de >>> High Performance Computing Center (HLRS) . phone >>> ++49(0)711/685-65530 >>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / >>> 685-65832 >>> Head of Dpmt Parallel Computing . . . >>> www.hlrs.de/people/rabenseifner >>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room >>> 1.307) >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] MPI RMA status summary
by Rolf Rabenseifner 30 Sep '14

30 Sep '14

I strongly agree with your statement: > These piecemeal changes are one of the sources of our problems. I only wanted to strongly say, that I would never vote for your a), because it is not backward compatible to what is already used. And with b) I've the problem that b1) is clear to me (see #456), but the Win_flush semantics for load/store is unclear to me. Of course, a total solution is needed and not parts of it. #456 is such a trial for a complete solution. Rolf ----- Original Message ----- > From: "William Gropp" <wgropp(a)illinois.edu> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Tuesday, September 30, 2014 3:19:06 PM > Subject: Re: [mpiwg-rma] MPI RMA status summary > > I disagree with this approach. The most important thing to do is to > figure out the correct definitions and semantics. Once we agree on > that, we can determine what can be handled as an errata and what > will require an update to the chapter and an update to the MPI > standard. These piecemeal changes are one of the sources of our > problems. > > Bill > > On Sep 30, 2014, at 7:38 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> > wrote: > > >> Vote for 1 of the following: > >> > >> a) Only Win_sync provides memory barrier semantics to shared > >> memory > >> windows > >> b) All RMA completion/sync routines (e.g., MPI_Win_lock, > >> MPI_Win_fence, MPI_Win_flush) provide memory barrier semantics > >> c) Some as yet undetermined blend of a and b, which might include > >> additional asserts > >> d) This topic needs further discussion > > > > Because we only have to clarify MPI-3.0 (this is an errata issue) > > and > > - obviously the MPI Forum and the readers expected that > > MPI_Win_fence > > (and therefore also the other MPI-2 synchronizations > > MPI_Win_post/start/complete/wait and MPI_Win_lock/unlock) > > works if MPI_Get/Put are sustituted by shared memory load/store > > (see the many Forum members as authors of the EuroMPI paper) > > - and the Forum decided that also MPI_Win_sync acts as if > > a memory barrier is inside, > > for me, > > - a) cannot be chosen bacause an erratum cannot remove > > a given functionality > > - and b) is automatically given, see reasons above. Therefore #456. > > > > The only open question for me is about the meaning of > > MPI_Win_flush. > > Therefore MPI_Win_flush is still missing in #456. > > > > Therefore for me, the Major choices seems to be > > b1) MPI-2 synchronizations + MPI_Win_sync > > b2) MPI-2 synchronizations + MPI_Win_sync + MPI_Win_flush > > > > For this vote, I clearly want to see a clear proposal > > about the meaning of MPI_Win_flush together with > > sared memory load/store, hopefully with the notation > > used in #456. > > > > Best regards > > Rolf > > > > ----- Original Message ----- > >> From: "William Gropp" <wgropp(a)illinois.edu> > >> To: "MPI WG Remote Memory Access working group" > >> <mpiwg-rma(a)lists.mpi-forum.org> > >> Sent: Monday, September 29, 2014 11:39:51 PM > >> Subject: Re: [mpiwg-rma] MPI RMA status summary > >> > >> > >> Thanks, Jeff. > >> > >> > >> I agree that I don’t want load/store to be considered RMA > >> operations. > >> But the issue of the memory consistency on RMA synchronization > >> and > >> completion operations to a shared memory window is complex. In > >> some > >> ways, the most consistent with RMA in other situations is the case > >> of MPI_Win_lock to your own process; the easiest extension for the > >> user is to have reasonably strong memory barrier semantics at all > >> sync/completion operations (thus including Fence). As you note, > >> this has costs. At the other extreme, we could say that only > >> Win_sync provides these memory barrier semantics. And we could > >> pick > >> a more complex blend (yes for some, no for others). > >> > >> > >> One of the first questions is whether we want to only Win_sync, > >> all > >> completion/sync RMA routines, or some subset to provide memory > >> barrier semantics for shared memory windows (this would include > >> RMA > >> windows that claimed to be shared memory, since there is a > >> proposal > >> to extend that property to other RMA windows). It would be good > >> to > >> make progress on this question, so I propose a straw vote of this > >> group by email. Vote for 1 of the following: > >> > >> > >> a) Only Win_sync provides memory barrier semantics to shared > >> memory > >> windows > >> b) All RMA completion/sync routines (e.g., MPI_Win_lock, > >> MPI_Win_fence, MPI_Win_flush) provide memory barrier semantics > >> c) Some as yet undetermined blend of a and b, which might include > >> additional asserts > >> d) This topic needs further discussion > >> > >> > >> Note that I’ve left off what “memory barrier semantics” means. > >> That > >> will need to be precisely defined for the standard, but I believe > >> we > >> know what we intend for this. We specifically are not defining > >> what > >> happens with non-MPI code. Also note that this is separate from > >> whether the RMA sync routines appear to be blocking when applied > >> to > >> a shared memory window; we can do a separate straw vote on that. > >> > >> > >> Bill > >> > >> > >> > >> On Sep 29, 2014, at 3:49 PM, Jeff Hammond < jeff.science(a)gmail.com > >> > > >> wrote: > >> > >> > >> > >> On Mon, Sep 29, 2014 at 9:16 AM, Rolf Rabenseifner < > >> rabenseifner(a)hlrs.de > wrote: > >> > >> > >> Only about the issues on #456 (shared memory syncronization): > >> > >> > >> > >> For the ones requiring discussion, assign someone to organize a > >> position and discussion. We can schedule telecons to go over > >> those > >> issues. The first item in the list is certainly in this class. > >> > >> Who can organize telecons on #456. > >> Would it be possible to organize a RMA meeting at SC? > >> > >> I will be there Monday through part of Thursday but am usually > >> triple-booked from 8 AM to midnight. > >> > >> > >> > >> The position expressed by the solution #456 is based on the idea > >> that the MPI RMA synchronization routines should have the same > >> outcome when RMA PUT and GET calls are substituted by stores and > >> loads. > >> > >> The outcome for the flush routines is still not defined. > >> > >> It is interesting, because the standard is actually conflicting on > >> whether Flush affects load-store. I find this incredibly > >> frustrating. > >> > >> Page 450: > >> > >> "Locally completes at the origin all outstanding RMA operations > >> initiated by the calling process to the target process specified > >> by > >> rank on the specified window. For example, after this routine > >> completes, the user may reuse any buffers provided to put, get, or > >> accumulate operations." > >> > >> I do not not think "RMA operations" includes load-store. > >> > >> Page 410: > >> > >> "The consistency of load/store accesses from/to the shared memory > >> as > >> observed by the user program depends on the architecture. A > >> consistent > >> view can be created in the unified memory model (see Section 11.4) > >> by > >> utilizing the window synchronization functions (see Section 11.5) > >> or > >> explicitly completing outstanding store accesses (e.g., by calling > >> MPI_WIN_FLUSH)." > >> > >> Here it is unambiguously implied that MPI_WIN_FLUSH affects > >> load-stores. > >> > >> My preference is to fix the statement on 410 since it is less > >> canonical than the one on 450, and because I do not want to have a > >> memory barrier in every call to WIN_FLUSH. > >> > >> Jeff > >> > >> > >> > >> I would prefere to have an organizer of the discussion inside of > >> the RMA subgroup that proposed the changes for MPI-3.1 > >> rather that I'm the organizer. > >> I tried to bring all the input together and hope that #456 > >> is now state that it is consistent itsself and with the > >> expectations expressed by the group that published the > >> paper at EuroMPI on first usage of this shared memory interface. > >> > >> The ticket is (together with the help of recent C11 > >> standadization) > >> on a good way to be also consistent with the compiler > >> optimizations - > >> in other words - the C standardization body has learnt from the > >> pthreads problems. Fortran is still an open question to me, > >> i.e., I do not know the status, see > >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/456#comment:13 > >> > >> Best regards > >> Rolf > >> > >> > >> > >> ----- Original Message ----- > >> > >> > >> From: "William Gropp" <wgropp(a)illinois.edu> > >> To: "MPI WG Remote Memory Access working group" > >> <mpiwg-rma(a)lists.mpi-forum.org> > >> Sent: Thursday, September 25, 2014 4:19:14 PM > >> Subject: [mpiwg-rma] MPI RMA status summary > >> > >> I looked through all of the tickets and wrote a summary of the > >> open > >> issues, which I’ve attached. I propose the following: > >> > >> Determine which of these issues can be resolved by email. A > >> significant number can probably be closed with no further action. > >> > >> For those requiring rework, determine if there is still interest > >> in > >> them, and if not, close them as well. > >> > >> For the ones requiring discussion, assign someone to organize a > >> position and discussion. We can schedule telecons to go over > >> those > >> issues. The first item in the list is certainly in this class. > >> > >> Comments? > >> > >> Bill > >> > >> _______________________________________________ > >> mpiwg-rma mailing list > >> mpiwg-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >> > >> -- > >> Dr. Rolf Rabenseifner . . . . . . . . . .. email > >> rabenseifner(a)hlrs.de > >> High Performance Computing Center (HLRS) . phone > >> ++49(0)711/685-65530 > >> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > >> 685-65832 > >> Head of Dpmt Parallel Computing . . . > >> www.hlrs.de/people/rabenseifner > >> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room > >> 1.307) > >> _______________________________________________ > >> mpiwg-rma mailing list > >> mpiwg-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >> > >> > >> > >> -- > >> Jeff Hammond > >> jeff.science(a)gmail.com > >> http://jeffhammond.github.io/ > >> _______________________________________________ > >> mpiwg-rma mailing list > >> mpiwg-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >> > >> _______________________________________________ > >> mpiwg-rma mailing list > >> mpiwg-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > -- > > Dr. Rolf Rabenseifner . . . . . . . . . .. email > > rabenseifner(a)hlrs.de > > High Performance Computing Center (HLRS) . phone > > ++49(0)711/685-65530 > > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > > 685-65832 > > Head of Dpmt Parallel Computing . . . > > www.hlrs.de/people/rabenseifner > > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room > > 1.307) > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] FENCE local completion requirements
by Balaji, Pavan 30 Sep '14

30 Sep '14

On Sep 30, 2014, at 7:59 AM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: >> Now to my next point — barrier synchronization only means that other processes have entered WIN_FENCE, not returned from WIN_FENCE. This means that they might not have completed all operations at the target. >> >> If we change my initial example to the following: >> >> MPI_WIN_FENCE(win1) >> MPI_PUT(win1) >> MPI_WIN_FENCE(win1) >> >> MPI_WIN_LOCK(win2) >> MPI_GET(win2) >> MPI_WIN_UNLOCK(win2) >> >> .. the GET might not see the data written by PUT, since the WIN_FENCE only guarantees local completion and barrier semantics of the second FENCE only ensures that the target has “called” FENCE, not “returned" from FENCE. > > Yes, that would need explicit synchronization before the lock, and it is covered in the standard: pg 448, ln 11-20. Right. And so would the following single-window code: MPI_WIN_FENCE(win) MPI_PUT(win) MPI_WIN_FENCE(win) MPI_WIN_LOCK(win) MPI_GET(win) MPI_WIN_UNLOCK(win) especially if I gave a MODE_NOCHECK argument to WIN_LOCK. It seems unnecessary to have two synchronizations, one in the implementation and one in the user code. That’s issue #1. In my proposal only one synchronization is needed. Issue #2 is that the standard says "RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process.” That’s a bogus statement as it doesn’t say anything about the state of the target window. The target window is in a defined state only after WIN_FENCE returns, not after it has been called (e.g., if the MPI implementation queues up all the RMA operations and issues them during WIN_FENCE). I believe the intention of the statement was to say that “RMA operations on win started by a process after the fence call returns will access the target window only after all operations to that target in the previous epoch have completed.” Issue #3 is whether FENCE needs remote completion. Note that your statement in one of the previous emails that the previous text ensures that you don’t need to send an ACK back from the target to the origin might not be valid on most systems. For example, this doesn’t work for any system that has hardware RDMA since the target doesn’t know of the operation. It also does not work for systems that use active messages unless the MPI implementation wants to queue up all of the potentially large number of RMA operations till the next FENCE, since it doesn’t know if the target process has called/completed FENCE yet. My point is that the “optimization” of not requiring remote completion might not be very useful in practice, though from the standards perspective I’d be OK with allowing this model. My proposal is to clean this up with the following two changes -- 1. Update the text in the standard to say: "A call to MPI_WIN_FENCE that ends an epoch entais a barrier synchronization that ensures that all operations issued in the epoch have completed both locally and at the target. A call to MPI_WIN_FENCE that is known not to end an epoch (in particular, a call with assert equal to MPI_MODE_NOPRECEDE) does not necessarily act as a barrier.” 2. Add a new assert that brings back the local-completion-only semantics similar to MPI-3. — Pavan -- Pavan Balaji ✉️ http://www.mcs.anl.gov/~balaji

1 0

Re: [mpiwg-rma] MPI RMA status summary
by William Gropp 30 Sep '14

30 Sep '14

I disagree with this approach. The most important thing to do is to figure out the correct definitions and semantics. Once we agree on that, we can determine what can be handled as an errata and what will require an update to the chapter and an update to the MPI standard. These piecemeal changes are one of the sources of our problems. Bill On Sep 30, 2014, at 7:38 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: >> Vote for 1 of the following: >> >> a) Only Win_sync provides memory barrier semantics to shared memory >> windows >> b) All RMA completion/sync routines (e.g., MPI_Win_lock, >> MPI_Win_fence, MPI_Win_flush) provide memory barrier semantics >> c) Some as yet undetermined blend of a and b, which might include >> additional asserts >> d) This topic needs further discussion > > Because we only have to clarify MPI-3.0 (this is an errata issue) and > - obviously the MPI Forum and the readers expected that MPI_Win_fence > (and therefore also the other MPI-2 synchronizations > MPI_Win_post/start/complete/wait and MPI_Win_lock/unlock) > works if MPI_Get/Put are sustituted by shared memory load/store > (see the many Forum members as authors of the EuroMPI paper) > - and the Forum decided that also MPI_Win_sync acts as if > a memory barrier is inside, > for me, > - a) cannot be chosen bacause an erratum cannot remove > a given functionality > - and b) is automatically given, see reasons above. Therefore #456. > > The only open question for me is about the meaning of MPI_Win_flush. > Therefore MPI_Win_flush is still missing in #456. > > Therefore for me, the Major choices seems to be > b1) MPI-2 synchronizations + MPI_Win_sync > b2) MPI-2 synchronizations + MPI_Win_sync + MPI_Win_flush > > For this vote, I clearly want to see a clear proposal > about the meaning of MPI_Win_flush together with > sared memory load/store, hopefully with the notation > used in #456. > > Best regards > Rolf > > ----- Original Message ----- >> From: "William Gropp" <wgropp(a)illinois.edu> >> To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> >> Sent: Monday, September 29, 2014 11:39:51 PM >> Subject: Re: [mpiwg-rma] MPI RMA status summary >> >> >> Thanks, Jeff. >> >> >> I agree that I don’t want load/store to be considered RMA operations. >> But the issue of the memory consistency on RMA synchronization and >> completion operations to a shared memory window is complex. In some >> ways, the most consistent with RMA in other situations is the case >> of MPI_Win_lock to your own process; the easiest extension for the >> user is to have reasonably strong memory barrier semantics at all >> sync/completion operations (thus including Fence). As you note, >> this has costs. At the other extreme, we could say that only >> Win_sync provides these memory barrier semantics. And we could pick >> a more complex blend (yes for some, no for others). >> >> >> One of the first questions is whether we want to only Win_sync, all >> completion/sync RMA routines, or some subset to provide memory >> barrier semantics for shared memory windows (this would include RMA >> windows that claimed to be shared memory, since there is a proposal >> to extend that property to other RMA windows). It would be good to >> make progress on this question, so I propose a straw vote of this >> group by email. Vote for 1 of the following: >> >> >> a) Only Win_sync provides memory barrier semantics to shared memory >> windows >> b) All RMA completion/sync routines (e.g., MPI_Win_lock, >> MPI_Win_fence, MPI_Win_flush) provide memory barrier semantics >> c) Some as yet undetermined blend of a and b, which might include >> additional asserts >> d) This topic needs further discussion >> >> >> Note that I’ve left off what “memory barrier semantics” means. That >> will need to be precisely defined for the standard, but I believe we >> know what we intend for this. We specifically are not defining what >> happens with non-MPI code. Also note that this is separate from >> whether the RMA sync routines appear to be blocking when applied to >> a shared memory window; we can do a separate straw vote on that. >> >> >> Bill >> >> >> >> On Sep 29, 2014, at 3:49 PM, Jeff Hammond < jeff.science(a)gmail.com > >> wrote: >> >> >> >> On Mon, Sep 29, 2014 at 9:16 AM, Rolf Rabenseifner < >> rabenseifner(a)hlrs.de > wrote: >> >> >> Only about the issues on #456 (shared memory syncronization): >> >> >> >> For the ones requiring discussion, assign someone to organize a >> position and discussion. We can schedule telecons to go over those >> issues. The first item in the list is certainly in this class. >> >> Who can organize telecons on #456. >> Would it be possible to organize a RMA meeting at SC? >> >> I will be there Monday through part of Thursday but am usually >> triple-booked from 8 AM to midnight. >> >> >> >> The position expressed by the solution #456 is based on the idea >> that the MPI RMA synchronization routines should have the same >> outcome when RMA PUT and GET calls are substituted by stores and >> loads. >> >> The outcome for the flush routines is still not defined. >> >> It is interesting, because the standard is actually conflicting on >> whether Flush affects load-store. I find this incredibly >> frustrating. >> >> Page 450: >> >> "Locally completes at the origin all outstanding RMA operations >> initiated by the calling process to the target process specified by >> rank on the specified window. For example, after this routine >> completes, the user may reuse any buffers provided to put, get, or >> accumulate operations." >> >> I do not not think "RMA operations" includes load-store. >> >> Page 410: >> >> "The consistency of load/store accesses from/to the shared memory as >> observed by the user program depends on the architecture. A >> consistent >> view can be created in the unified memory model (see Section 11.4) by >> utilizing the window synchronization functions (see Section 11.5) or >> explicitly completing outstanding store accesses (e.g., by calling >> MPI_WIN_FLUSH)." >> >> Here it is unambiguously implied that MPI_WIN_FLUSH affects >> load-stores. >> >> My preference is to fix the statement on 410 since it is less >> canonical than the one on 450, and because I do not want to have a >> memory barrier in every call to WIN_FLUSH. >> >> Jeff >> >> >> >> I would prefere to have an organizer of the discussion inside of >> the RMA subgroup that proposed the changes for MPI-3.1 >> rather that I'm the organizer. >> I tried to bring all the input together and hope that #456 >> is now state that it is consistent itsself and with the >> expectations expressed by the group that published the >> paper at EuroMPI on first usage of this shared memory interface. >> >> The ticket is (together with the help of recent C11 standadization) >> on a good way to be also consistent with the compiler optimizations - >> in other words - the C standardization body has learnt from the >> pthreads problems. Fortran is still an open question to me, >> i.e., I do not know the status, see >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/456#comment:13 >> >> Best regards >> Rolf >> >> >> >> ----- Original Message ----- >> >> >> From: "William Gropp" <wgropp(a)illinois.edu> >> To: "MPI WG Remote Memory Access working group" >> <mpiwg-rma(a)lists.mpi-forum.org> >> Sent: Thursday, September 25, 2014 4:19:14 PM >> Subject: [mpiwg-rma] MPI RMA status summary >> >> I looked through all of the tickets and wrote a summary of the open >> issues, which I’ve attached. I propose the following: >> >> Determine which of these issues can be resolved by email. A >> significant number can probably be closed with no further action. >> >> For those requiring rework, determine if there is still interest in >> them, and if not, close them as well. >> >> For the ones requiring discussion, assign someone to organize a >> position and discussion. We can schedule telecons to go over those >> issues. The first item in the list is certainly in this class. >> >> Comments? >> >> Bill >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> -- >> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de >> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 >> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 >> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner >> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> http://jeffhammond.github.io/ >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] FENCE local completion requirements
by Rajeev Thakur 30 Sep '14

30 Sep '14

> Now to my next point — barrier synchronization only means that other processes have entered WIN_FENCE, not returned from WIN_FENCE. This means that they might not have completed all operations at the target. > > If we change my initial example to the following: > > MPI_WIN_FENCE(win1) > MPI_PUT(win1) > MPI_WIN_FENCE(win1) > > MPI_WIN_LOCK(win2) > MPI_GET(win2) > MPI_WIN_UNLOCK(win2) > > .. the GET might not see the data written by PUT, since the WIN_FENCE only guarantees local completion and barrier semantics of the second FENCE only ensures that the target has “called” FENCE, not “returned" from FENCE. Yes, that would need explicit synchronization before the lock, and it is covered in the standard: pg 448, ln 11-20. Rajeev On Sep 29, 2014, at 11:09 PM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: > > On Sep 29, 2014, at 11:03 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > >>> Let me rephrase — >>> >>> Is the sentence in the standard to be read as: if there are preceding PUT/GET operations, it *will* be a barrier? >>> >>> I don’t think the sentence says that. I agree that an MPI implementation has to do that, but the standard is not explicit about it. >>> >>> — Pavan >> >> Yes, that's what the standard says. And it says that because the implementation has no other choice if it has to maintain fence semantics. >> >> The standard says >> >> "A fence call usually entails a barrier synchronization: a process completes a call to MPI_WIN_FENCE only after all other processes in the group entered their matching call." >> >> Then it qualifies the "usually" by giving the case where it doesn't act as a barrier: >> >> "However, a call to MPI_WIN_FENCE that is known not to end any epoch (in particular, a call with assert equal to MPI_MODE_NOPRECEDE) does not necessarily act as a barrier." >> >> Which means in all other cases, it acts as a barrier. > > The right way to phrase that would have been: > > A call to MPI_WIN_FENCE that ends an epoch entails a barrier synchronization: a process completes a call to MPI_WIN_FENCE only after all other processes in the group entered their matching call. A call to MPI_WIN_FENCE that is known not to end any epoch (in particular, a call with assert equal to MPI_MODE_NOPRECEDE) does not necessarily act as a barrier. > > If this was the intention, it should be fixed in the standard. > > Now to my next point — barrier synchronization only means that other processes have entered WIN_FENCE, not returned from WIN_FENCE. This means that they might not have completed all operations at the target. > > If we change my initial example to the following: > > MPI_WIN_FENCE(win1) > MPI_PUT(win1) > MPI_WIN_FENCE(win1) > > MPI_WIN_LOCK(win2) > MPI_GET(win2) > MPI_WIN_UNLOCK(win2) > > .. the GET might not see the data written by PUT, since the WIN_FENCE only guarantees local completion and barrier semantics of the second FENCE only ensures that the target has “called” FENCE, not “returned" from FENCE. > > — Pavan > > -- > Pavan Balaji ✉️ > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0