September 2014 - mpiwg-rma - mailman3.cels.anl.gov

Re: [mpiwg-rma] FENCE local completion requirements
by Rajeev Thakur 30 Sep '14

30 Sep '14

> Now to my next point — barrier synchronization only means that other processes have entered WIN_FENCE, not returned from WIN_FENCE. This means that they might not have completed all operations at the target. > > If we change my initial example to the following: > > MPI_WIN_FENCE(win1) > MPI_PUT(win1) > MPI_WIN_FENCE(win1) > > MPI_WIN_LOCK(win2) > MPI_GET(win2) > MPI_WIN_UNLOCK(win2) > > .. the GET might not see the data written by PUT, since the WIN_FENCE only guarantees local completion and barrier semantics of the second FENCE only ensures that the target has “called” FENCE, not “returned" from FENCE. Yes, that would need explicit synchronization before the lock, and it is covered in the standard: pg 448, ln 11-20. Rajeev On Sep 29, 2014, at 11:09 PM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: > > On Sep 29, 2014, at 11:03 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > >>> Let me rephrase — >>> >>> Is the sentence in the standard to be read as: if there are preceding PUT/GET operations, it *will* be a barrier? >>> >>> I don’t think the sentence says that. I agree that an MPI implementation has to do that, but the standard is not explicit about it. >>> >>> — Pavan >> >> Yes, that's what the standard says. And it says that because the implementation has no other choice if it has to maintain fence semantics. >> >> The standard says >> >> "A fence call usually entails a barrier synchronization: a process completes a call to MPI_WIN_FENCE only after all other processes in the group entered their matching call." >> >> Then it qualifies the "usually" by giving the case where it doesn't act as a barrier: >> >> "However, a call to MPI_WIN_FENCE that is known not to end any epoch (in particular, a call with assert equal to MPI_MODE_NOPRECEDE) does not necessarily act as a barrier." >> >> Which means in all other cases, it acts as a barrier. > > The right way to phrase that would have been: > > A call to MPI_WIN_FENCE that ends an epoch entails a barrier synchronization: a process completes a call to MPI_WIN_FENCE only after all other processes in the group entered their matching call. A call to MPI_WIN_FENCE that is known not to end any epoch (in particular, a call with assert equal to MPI_MODE_NOPRECEDE) does not necessarily act as a barrier. > > If this was the intention, it should be fixed in the standard. > > Now to my next point — barrier synchronization only means that other processes have entered WIN_FENCE, not returned from WIN_FENCE. This means that they might not have completed all operations at the target. > > If we change my initial example to the following: > > MPI_WIN_FENCE(win1) > MPI_PUT(win1) > MPI_WIN_FENCE(win1) > > MPI_WIN_LOCK(win2) > MPI_GET(win2) > MPI_WIN_UNLOCK(win2) > > .. the GET might not see the data written by PUT, since the WIN_FENCE only guarantees local completion and barrier semantics of the second FENCE only ensures that the target has “called” FENCE, not “returned" from FENCE. > > — Pavan > > -- > Pavan Balaji ✉️ > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] MPI RMA status summary
by Rolf Rabenseifner 30 Sep '14

30 Sep '14

> Vote for 1 of the following: > > a) Only Win_sync provides memory barrier semantics to shared memory > windows > b) All RMA completion/sync routines (e.g., MPI_Win_lock, > MPI_Win_fence, MPI_Win_flush) provide memory barrier semantics > c) Some as yet undetermined blend of a and b, which might include > additional asserts > d) This topic needs further discussion Because we only have to clarify MPI-3.0 (this is an errata issue) and - obviously the MPI Forum and the readers expected that MPI_Win_fence (and therefore also the other MPI-2 synchronizations MPI_Win_post/start/complete/wait and MPI_Win_lock/unlock) works if MPI_Get/Put are sustituted by shared memory load/store (see the many Forum members as authors of the EuroMPI paper) - and the Forum decided that also MPI_Win_sync acts as if a memory barrier is inside, for me, - a) cannot be chosen bacause an erratum cannot remove a given functionality - and b) is automatically given, see reasons above. Therefore #456. The only open question for me is about the meaning of MPI_Win_flush. Therefore MPI_Win_flush is still missing in #456. Therefore for me, the Major choices seems to be b1) MPI-2 synchronizations + MPI_Win_sync b2) MPI-2 synchronizations + MPI_Win_sync + MPI_Win_flush For this vote, I clearly want to see a clear proposal about the meaning of MPI_Win_flush together with sared memory load/store, hopefully with the notation used in #456. Best regards Rolf ----- Original Message ----- > From: "William Gropp" <wgropp(a)illinois.edu> > To: "MPI WG Remote Memory Access working group" <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Monday, September 29, 2014 11:39:51 PM > Subject: Re: [mpiwg-rma] MPI RMA status summary > > > Thanks, Jeff. > > > I agree that I don’t want load/store to be considered RMA operations. > But the issue of the memory consistency on RMA synchronization and > completion operations to a shared memory window is complex. In some > ways, the most consistent with RMA in other situations is the case > of MPI_Win_lock to your own process; the easiest extension for the > user is to have reasonably strong memory barrier semantics at all > sync/completion operations (thus including Fence). As you note, > this has costs. At the other extreme, we could say that only > Win_sync provides these memory barrier semantics. And we could pick > a more complex blend (yes for some, no for others). > > > One of the first questions is whether we want to only Win_sync, all > completion/sync RMA routines, or some subset to provide memory > barrier semantics for shared memory windows (this would include RMA > windows that claimed to be shared memory, since there is a proposal > to extend that property to other RMA windows). It would be good to > make progress on this question, so I propose a straw vote of this > group by email. Vote for 1 of the following: > > > a) Only Win_sync provides memory barrier semantics to shared memory > windows > b) All RMA completion/sync routines (e.g., MPI_Win_lock, > MPI_Win_fence, MPI_Win_flush) provide memory barrier semantics > c) Some as yet undetermined blend of a and b, which might include > additional asserts > d) This topic needs further discussion > > > Note that I’ve left off what “memory barrier semantics” means. That > will need to be precisely defined for the standard, but I believe we > know what we intend for this. We specifically are not defining what > happens with non-MPI code. Also note that this is separate from > whether the RMA sync routines appear to be blocking when applied to > a shared memory window; we can do a separate straw vote on that. > > > Bill > > > > On Sep 29, 2014, at 3:49 PM, Jeff Hammond < jeff.science(a)gmail.com > > wrote: > > > > On Mon, Sep 29, 2014 at 9:16 AM, Rolf Rabenseifner < > rabenseifner(a)hlrs.de > wrote: > > > Only about the issues on #456 (shared memory syncronization): > > > > For the ones requiring discussion, assign someone to organize a > position and discussion. We can schedule telecons to go over those > issues. The first item in the list is certainly in this class. > > Who can organize telecons on #456. > Would it be possible to organize a RMA meeting at SC? > > I will be there Monday through part of Thursday but am usually > triple-booked from 8 AM to midnight. > > > > The position expressed by the solution #456 is based on the idea > that the MPI RMA synchronization routines should have the same > outcome when RMA PUT and GET calls are substituted by stores and > loads. > > The outcome for the flush routines is still not defined. > > It is interesting, because the standard is actually conflicting on > whether Flush affects load-store. I find this incredibly > frustrating. > > Page 450: > > "Locally completes at the origin all outstanding RMA operations > initiated by the calling process to the target process specified by > rank on the specified window. For example, after this routine > completes, the user may reuse any buffers provided to put, get, or > accumulate operations." > > I do not not think "RMA operations" includes load-store. > > Page 410: > > "The consistency of load/store accesses from/to the shared memory as > observed by the user program depends on the architecture. A > consistent > view can be created in the unified memory model (see Section 11.4) by > utilizing the window synchronization functions (see Section 11.5) or > explicitly completing outstanding store accesses (e.g., by calling > MPI_WIN_FLUSH)." > > Here it is unambiguously implied that MPI_WIN_FLUSH affects > load-stores. > > My preference is to fix the statement on 410 since it is less > canonical than the one on 450, and because I do not want to have a > memory barrier in every call to WIN_FLUSH. > > Jeff > > > > I would prefere to have an organizer of the discussion inside of > the RMA subgroup that proposed the changes for MPI-3.1 > rather that I'm the organizer. > I tried to bring all the input together and hope that #456 > is now state that it is consistent itsself and with the > expectations expressed by the group that published the > paper at EuroMPI on first usage of this shared memory interface. > > The ticket is (together with the help of recent C11 standadization) > on a good way to be also consistent with the compiler optimizations - > in other words - the C standardization body has learnt from the > pthreads problems. Fortran is still an open question to me, > i.e., I do not know the status, see > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/456#comment:13 > > Best regards > Rolf > > > > ----- Original Message ----- > > > From: "William Gropp" <wgropp(a)illinois.edu> > To: "MPI WG Remote Memory Access working group" > <mpiwg-rma(a)lists.mpi-forum.org> > Sent: Thursday, September 25, 2014 4:19:14 PM > Subject: [mpiwg-rma] MPI RMA status summary > > I looked through all of the tickets and wrote a summary of the open > issues, which I’ve attached. I propose the following: > > Determine which of these issues can be resolved by email. A > significant number can probably be closed with no further action. > > For those requiring rework, determine if there is still interest in > them, and if not, close them as well. > > For the ones requiring discussion, assign someone to organize a > position and discussion. We can schedule telecons to go over those > issues. The first item in the list is certainly in this class. > > Comments? > > Bill > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-rma] FENCE local completion requirements
by Balaji, Pavan 29 Sep '14

29 Sep '14

On Sep 29, 2014, at 11:03 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: >> Let me rephrase — >> >> Is the sentence in the standard to be read as: if there are preceding PUT/GET operations, it *will* be a barrier? >> >> I don’t think the sentence says that. I agree that an MPI implementation has to do that, but the standard is not explicit about it. >> >> — Pavan > > Yes, that's what the standard says. And it says that because the implementation has no other choice if it has to maintain fence semantics. > > The standard says > > "A fence call usually entails a barrier synchronization: a process completes a call to MPI_WIN_FENCE only after all other processes in the group entered their matching call." > > Then it qualifies the "usually" by giving the case where it doesn't act as a barrier: > > "However, a call to MPI_WIN_FENCE that is known not to end any epoch (in particular, a call with assert equal to MPI_MODE_NOPRECEDE) does not necessarily act as a barrier." > > Which means in all other cases, it acts as a barrier. The right way to phrase that would have been: A call to MPI_WIN_FENCE that ends an epoch entails a barrier synchronization: a process completes a call to MPI_WIN_FENCE only after all other processes in the group entered their matching call. A call to MPI_WIN_FENCE that is known not to end any epoch (in particular, a call with assert equal to MPI_MODE_NOPRECEDE) does not necessarily act as a barrier. If this was the intention, it should be fixed in the standard. Now to my next point — barrier synchronization only means that other processes have entered WIN_FENCE, not returned from WIN_FENCE. This means that they might not have completed all operations at the target. If we change my initial example to the following: MPI_WIN_FENCE(win1) MPI_PUT(win1) MPI_WIN_FENCE(win1) MPI_WIN_LOCK(win2) MPI_GET(win2) MPI_WIN_UNLOCK(win2) .. the GET might not see the data written by PUT, since the WIN_FENCE only guarantees local completion and barrier semantics of the second FENCE only ensures that the target has “called” FENCE, not “returned" from FENCE. — Pavan -- Pavan Balaji ✉️ http://www.mcs.anl.gov/~balaji

1 0

Re: [mpiwg-rma] FENCE local completion requirements
by Rajeev Thakur 29 Sep '14

29 Sep '14

> Let me rephrase — > > Is the sentence in the standard to be read as: if there are preceding PUT/GET operations, it *will* be a barrier? > > I don’t think the sentence says that. I agree that an MPI implementation has to do that, but the standard is not explicit about it. > > — Pavan Yes, that's what the standard says. And it says that because the implementation has no other choice if it has to maintain fence semantics. The standard says "A fence call usually entails a barrier synchronization: a process completes a call to MPI_WIN_FENCE only after all other processes in the group entered their matching call." Then it qualifies the "usually" by giving the case where it doesn't act as a barrier: "However, a call to MPI_WIN_FENCE that is known not to end any epoch (in particular, a call with assert equal to MPI_MODE_NOPRECEDE) does not necessarily act as a barrier." Which means in all other cases, it acts as a barrier. What are the other cases? Those that are not covered by the above sentence, i.e., those where the fence ends an access or exposure epoch, i.e., where there are RMA operations. Rajeev > > On Sep 29, 2014, at 10:37 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > >>>>> All RMA operations on win originating at a given process and started before the fence call will complete at that process before the fence call returns. They will be completed at their target before the fence call returns at the target. >>>> >>>> The intent of this is to avoid an ack at the end of the fence from target to origin. However, since a fence that completes RMA operations is effectively a barrier, when the fence returns on the origin, the origin can be sure that the target has at least called its fence (but may not have exited the fence). >>>> >>>> pg 441, last para: "A fence call usually entails a barrier synchronization: a process completes a call to MPI_WIN_FENCE only after all other processes in the group entered their matching call. However, a call to MPI_WIN_FENCE that is known not to end any epoch (in particular, a call with assert equal to MPI_MODE_NOPRECEDE) does not necessarily act as a barrier.” >>> >>> Is the above sentence to be read as: if you don’t have MPI_MODE_NOPRECEDE, it *will* be a barrier? That’s not how I interpret it. If it does not *guarantee* a barrier, then we still need to do an MPI_BARRIER in the application in the example I gave. >> >> The above sentence reflects what an implementation would need to do in order to support fence semantics, namely, if there are RMA operations preceding a fence, then a process cannot get out of the fence unless all RMA operations targeted to it are completed. And it doesn't know how many RMA operations are targeted to it, and from whom, until every other process has reached its fence. Therefore, every process has to wait for every other process to reach its fence -- effectively a barrier. >> >> The cases where this situation does not occur are where there are guaranteed to be no preceding RMA operations. One such case is the very first fence in the program. Another case is where the user says don't worry there were no RMA operations, by passing an assert. In these cases the implementation of the fence may not act like a barrier. >> >> Whatever an implementation does, it has to support the fence semantics, which are: >> >> 1. The memory of a target process cannot be accessed until the target has called fence. >> 2. A process cannot exit from a fence until all RMA operations targeted to it from other processes have completed, and all RMA operations from it to others have locally completed. >> >> Rajeev >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > -- > Pavan Balaji ✉️ > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] FENCE local completion requirements
by Balaji, Pavan 29 Sep '14

29 Sep '14

Let me rephrase — Is the sentence in the standard to be read as: if there are preceding PUT/GET operations, it *will* be a barrier? I don’t think the sentence says that. I agree that an MPI implementation has to do that, but the standard is not explicit about it. — Pavan On Sep 29, 2014, at 10:37 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: >>>> All RMA operations on win originating at a given process and started before the fence call will complete at that process before the fence call returns. They will be completed at their target before the fence call returns at the target. >>> >>> The intent of this is to avoid an ack at the end of the fence from target to origin. However, since a fence that completes RMA operations is effectively a barrier, when the fence returns on the origin, the origin can be sure that the target has at least called its fence (but may not have exited the fence). >>> >>> pg 441, last para: "A fence call usually entails a barrier synchronization: a process completes a call to MPI_WIN_FENCE only after all other processes in the group entered their matching call. However, a call to MPI_WIN_FENCE that is known not to end any epoch (in particular, a call with assert equal to MPI_MODE_NOPRECEDE) does not necessarily act as a barrier.” >> >> Is the above sentence to be read as: if you don’t have MPI_MODE_NOPRECEDE, it *will* be a barrier? That’s not how I interpret it. If it does not *guarantee* a barrier, then we still need to do an MPI_BARRIER in the application in the example I gave. > > The above sentence reflects what an implementation would need to do in order to support fence semantics, namely, if there are RMA operations preceding a fence, then a process cannot get out of the fence unless all RMA operations targeted to it are completed. And it doesn't know how many RMA operations are targeted to it, and from whom, until every other process has reached its fence. Therefore, every process has to wait for every other process to reach its fence -- effectively a barrier. > > The cases where this situation does not occur are where there are guaranteed to be no preceding RMA operations. One such case is the very first fence in the program. Another case is where the user says don't worry there were no RMA operations, by passing an assert. In these cases the implementation of the fence may not act like a barrier. > > Whatever an implementation does, it has to support the fence semantics, which are: > > 1. The memory of a target process cannot be accessed until the target has called fence. > 2. A process cannot exit from a fence until all RMA operations targeted to it from other processes have completed, and all RMA operations from it to others have locally completed. > > Rajeev > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Pavan Balaji ✉️ http://www.mcs.anl.gov/~balaji

1 0

Re: [mpiwg-rma] FENCE local completion requirements
by Rajeev Thakur 29 Sep '14

29 Sep '14

>>> All RMA operations on win originating at a given process and started before the fence call will complete at that process before the fence call returns. They will be completed at their target before the fence call returns at the target. >> >> The intent of this is to avoid an ack at the end of the fence from target to origin. However, since a fence that completes RMA operations is effectively a barrier, when the fence returns on the origin, the origin can be sure that the target has at least called its fence (but may not have exited the fence). >> >> pg 441, last para: "A fence call usually entails a barrier synchronization: a process completes a call to MPI_WIN_FENCE only after all other processes in the group entered their matching call. However, a call to MPI_WIN_FENCE that is known not to end any epoch (in particular, a call with assert equal to MPI_MODE_NOPRECEDE) does not necessarily act as a barrier.” > > Is the above sentence to be read as: if you don’t have MPI_MODE_NOPRECEDE, it *will* be a barrier? That’s not how I interpret it. If it does not *guarantee* a barrier, then we still need to do an MPI_BARRIER in the application in the example I gave. The above sentence reflects what an implementation would need to do in order to support fence semantics, namely, if there are RMA operations preceding a fence, then a process cannot get out of the fence unless all RMA operations targeted to it are completed. And it doesn't know how many RMA operations are targeted to it, and from whom, until every other process has reached its fence. Therefore, every process has to wait for every other process to reach its fence -- effectively a barrier. The cases where this situation does not occur are where there are guaranteed to be no preceding RMA operations. One such case is the very first fence in the program. Another case is where the user says don't worry there were no RMA operations, by passing an assert. In these cases the implementation of the fence may not act like a barrier. Whatever an implementation does, it has to support the fence semantics, which are: 1. The memory of a target process cannot be accessed until the target has called fence. 2. A process cannot exit from a fence until all RMA operations targeted to it from other processes have completed, and all RMA operations from it to others have locally completed. Rajeev

1 0

Re: [mpiwg-rma] FENCE local completion requirements
by Balaji, Pavan 29 Sep '14

29 Sep '14

On Sep 29, 2014, at 6:30 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: >> All RMA operations on win originating at a given process and started before the fence call will complete at that process before the fence call returns. They will be completed at their target before the fence call returns at the target. > > The intent of this is to avoid an ack at the end of the fence from target to origin. However, since a fence that completes RMA operations is effectively a barrier, when the fence returns on the origin, the origin can be sure that the target has at least called its fence (but may not have exited the fence). > > pg 441, last para: "A fence call usually entails a barrier synchronization: a process completes a call to MPI_WIN_FENCE only after all other processes in the group entered their matching call. However, a call to MPI_WIN_FENCE that is known not to end any epoch (in particular, a call with assert equal to MPI_MODE_NOPRECEDE) does not necessarily act as a barrier.” Is the above sentence to be read as: if you don’t have MPI_MODE_NOPRECEDE, it *will* be a barrier? That’s not how I interpret it. If it does not *guarantee* a barrier, then we still need to do an MPI_BARRIER in the application in the example I gave. — Pavan -- Pavan Balaji ✉️ http://www.mcs.anl.gov/~balaji

1 0

Re: [mpiwg-rma] FENCE local completion requirements
by Rajeev Thakur 29 Sep '14

29 Sep '14

> In one of the projects here, we are using multiple overlapping windows (i.e., we have more than one window handle for the same set of buffers). In this case, the following program would be incorrect: > > MPI_WIN_FENCE(win1) > MPI_PUT(<something>, remote_offset, win1) > MPI_WIN_FENCE(win1) > > MPI_WIN_FENCE(win2) > MPI_GET(<something>, remote_offset, win2) > MPI_WIN_FENCE(win2) > > Since there is no remote completion on win1, the GET might not get the same data as what was put on win1. I’ll need to add extra synchronization here: No need for extra synchronization. MPI_Get cannot access the target memory until the target has called Win_fence(win2), which means the target has already called Win_fence(win1). Rajeev On Sep 29, 2014, at 5:15 PM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: > Hello, > > The MPI-3 RMA chapter says the following (pg. 440, lines 45-48 and pg. 441, lines 34-35): > > “All RMA operations on win originating at a given process and started before the fence call will complete at that process before the fence call returns. They will be completed at their target before the fence call returns at the target. RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process.” > > This seems to imply that: > > 1. The FENCE call only implies local completion for a given process. > > 2. Operations issued after the local FENCE need to figure out whether the target has completed the FENCE operation. > > Does anyone remember why the standard was written this way? If the MPI implementation uses hardware RDMA, it has to enforce remote completion anyway (in order to tell the target process that data issued to it has completed). Even if the MPI implementation uses strictly active messages, we either need to enforce remote completion or RMA operations after the FENCE have to check in their AM handlers if the target has completed the previous FENCE (or queue up potentially many RMA operations if this has not happened yet). > > The only case I can think of whether this weaker semantics would benefit is if the FENCE operation locally queues up the issued RMA operations and implements them as an Alltoall-style collective internally. I guess this is valid implementation choice, though it’s unclear which implementations do this. > > Now, the question is what are we losing by this weaker semantics — > > In one of the projects here, we are using multiple overlapping windows (i.e., we have more than one window handle for the same set of buffers). In this case, the following program would be incorrect: > > MPI_WIN_FENCE(win1) > MPI_PUT(<something>, remote_offset, win1) > MPI_WIN_FENCE(win1) > > MPI_WIN_FENCE(win2) > MPI_GET(<something>, remote_offset, win2) > MPI_WIN_FENCE(win2) > > Since there is no remote completion on win1, the GET might not get the same data as what was put on win1. I’ll need to add extra synchronization here: > > MPI_WIN_FENCE(win1) > MPI_PUT(<something>, remote_offset, win1) > MPI_WIN_FENCE(win1) > > MPI_BARRIER() > > MPI_WIN_FENCE(win2) > MPI_GET(<something>, remote_offset, win2) > MPI_WIN_FENCE(win2) > > This seems unnecessary for most implementations since they already do a synchronization. > > In short: (1) the standard gives weak semantics hoping the implementation can take advantage of it, (2) most implementations are not taking advantage of it due to the use of hardware RDMA or otherwise, and (3) it’s creating annoying synchronization requirements for users. > > Is there some way we can improve this situation? I’m not advocating making the synchronization requirements stronger in the standard. But if there’s a way for the user to either request for a strong synchronization or if there’s a way for the implementation to expose this to the user, it might be more beneficial. > > One option is make the default the stronger semantics and provide an assert for the weaker semantics. > > Thoughts? > > — Pavan > > -- > Pavan Balaji ✉️ > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] FENCE local completion requirements
by Rajeev Thakur 29 Sep '14

29 Sep '14

> RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process.” The intent of this sentence is that unless the target has called fence, it has not exposed its memory for remote access. So, any remote operation must not access the target's memory until it has called fence. > All RMA operations on win originating at a given process and started before the fence call will complete at that process before the fence call returns. They will be completed at their target before the fence call returns at the target. The intent of this is to avoid an ack at the end of the fence from target to origin. However, since a fence that completes RMA operations is effectively a barrier, when the fence returns on the origin, the origin can be sure that the target has at least called its fence (but may not have exited the fence). pg 441, last para: "A fence call usually entails a barrier synchronization: a process completes a call to MPI_WIN_FENCE only after all other processes in the group entered their matching call. However, a call to MPI_WIN_FENCE that is known not to end any epoch (in particular, a call with assert equal to MPI_MODE_NOPRECEDE) does not necessarily act as a barrier." Rajeev On Sep 29, 2014, at 5:15 PM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: > Hello, > > The MPI-3 RMA chapter says the following (pg. 440, lines 45-48 and pg. 441, lines 34-35): > > “All RMA operations on win originating at a given process and started before the fence call will complete at that process before the fence call returns. They will be completed at their target before the fence call returns at the target. RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process.” > > This seems to imply that: > > 1. The FENCE call only implies local completion for a given process. > > 2. Operations issued after the local FENCE need to figure out whether the target has completed the FENCE operation. > > Does anyone remember why the standard was written this way? If the MPI implementation uses hardware RDMA, it has to enforce remote completion anyway (in order to tell the target process that data issued to it has completed). Even if the MPI implementation uses strictly active messages, we either need to enforce remote completion or RMA operations after the FENCE have to check in their AM handlers if the target has completed the previous FENCE (or queue up potentially many RMA operations if this has not happened yet). > > The only case I can think of whether this weaker semantics would benefit is if the FENCE operation locally queues up the issued RMA operations and implements them as an Alltoall-style collective internally. I guess this is valid implementation choice, though it’s unclear which implementations do this. > > Now, the question is what are we losing by this weaker semantics — > > In one of the projects here, we are using multiple overlapping windows (i.e., we have more than one window handle for the same set of buffers). In this case, the following program would be incorrect: > > MPI_WIN_FENCE(win1) > MPI_PUT(<something>, remote_offset, win1) > MPI_WIN_FENCE(win1) > > MPI_WIN_FENCE(win2) > MPI_GET(<something>, remote_offset, win2) > MPI_WIN_FENCE(win2) > > Since there is no remote completion on win1, the GET might not get the same data as what was put on win1. I’ll need to add extra synchronization here: > > MPI_WIN_FENCE(win1) > MPI_PUT(<something>, remote_offset, win1) > MPI_WIN_FENCE(win1) > > MPI_BARRIER() > > MPI_WIN_FENCE(win2) > MPI_GET(<something>, remote_offset, win2) > MPI_WIN_FENCE(win2) > > This seems unnecessary for most implementations since they already do a synchronization. > > In short: (1) the standard gives weak semantics hoping the implementation can take advantage of it, (2) most implementations are not taking advantage of it due to the use of hardware RDMA or otherwise, and (3) it’s creating annoying synchronization requirements for users. > > Is there some way we can improve this situation? I’m not advocating making the synchronization requirements stronger in the standard. But if there’s a way for the user to either request for a strong synchronization or if there’s a way for the implementation to expose this to the user, it might be more beneficial. > > One option is make the default the stronger semantics and provide an assert for the weaker semantics. > > Thoughts? > > — Pavan > > -- > Pavan Balaji ✉️ > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] FENCE local completion requirements
by Jeff Hammond 29 Sep '14

29 Sep '14

I have always felt that WIN_FENCE was horrible and unnecessarily hard to use except in the trivial cases. Can we just deprecate it and tell people to use LOCK_ALL and UNLOCK_ALL instead of WIN_FENCE? If you insist on a serious answer, we should not strengthen this model. We should add an advice to users that tells them to add the BARRIER you note. Jeff On Mon, Sep 29, 2014 at 3:15 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > Hello, > > The MPI-3 RMA chapter says the following (pg. 440, lines 45-48 and pg. 441, lines 34-35): > > “All RMA operations on win originating at a given process and started before the fence call will complete at that process before the fence call returns. They will be completed at their target before the fence call returns at the target. RMA operations on win started by a process after the fence call returns will access their target window only after MPI_WIN_FENCE has been called by the target process.” > > This seems to imply that: > > 1. The FENCE call only implies local completion for a given process. > > 2. Operations issued after the local FENCE need to figure out whether the target has completed the FENCE operation. > > Does anyone remember why the standard was written this way? If the MPI implementation uses hardware RDMA, it has to enforce remote completion anyway (in order to tell the target process that data issued to it has completed). Even if the MPI implementation uses strictly active messages, we either need to enforce remote completion or RMA operations after the FENCE have to check in their AM handlers if the target has completed the previous FENCE (or queue up potentially many RMA operations if this has not happened yet). > > The only case I can think of whether this weaker semantics would benefit is if the FENCE operation locally queues up the issued RMA operations and implements them as an Alltoall-style collective internally. I guess this is valid implementation choice, though it’s unclear which implementations do this. > > Now, the question is what are we losing by this weaker semantics — > > In one of the projects here, we are using multiple overlapping windows (i.e., we have more than one window handle for the same set of buffers). In this case, the following program would be incorrect: > > MPI_WIN_FENCE(win1) > MPI_PUT(<something>, remote_offset, win1) > MPI_WIN_FENCE(win1) > > MPI_WIN_FENCE(win2) > MPI_GET(<something>, remote_offset, win2) > MPI_WIN_FENCE(win2) > > Since there is no remote completion on win1, the GET might not get the same data as what was put on win1. I’ll need to add extra synchronization here: > > MPI_WIN_FENCE(win1) > MPI_PUT(<something>, remote_offset, win1) > MPI_WIN_FENCE(win1) > > MPI_BARRIER() > > MPI_WIN_FENCE(win2) > MPI_GET(<something>, remote_offset, win2) > MPI_WIN_FENCE(win2) > > This seems unnecessary for most implementations since they already do a synchronization. > > In short: (1) the standard gives weak semantics hoping the implementation can take advantage of it, (2) most implementations are not taking advantage of it due to the use of hardware RDMA or otherwise, and (3) it’s creating annoying synchronization requirements for users. > > Is there some way we can improve this situation? I’m not advocating making the synchronization requirements stronger in the standard. But if there’s a way for the user to either request for a strong synchronization or if there’s a way for the implementation to expose this to the user, it might be more beneficial. > > One option is make the default the stronger semantics and provide an assert for the weaker semantics. > > Thoughts? > > — Pavan > > -- > Pavan Balaji ✉️ > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0