July 2013 - mpiwg-rma - mailman3.cels.anl.gov

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Jed Brown 29 Jul '13

29 Jul '13

Pavan Balaji <balaji(a)mcs.anl.gov> writes: > FLUSH will guarantee that the data is present in the remote public > memory. And UNIFIED states that public memory == private memory. So, > the user can read it directly without any additional MPI calls. > > That's the broken part. Modulo a memory barrier, depending on the architecture. Should the statement be strengthened so that the user must _either_ call MPI_Win_sync (with implementations expected to provide a fast path for UNIFIED) or perform any architecture-specific synchronization, which of course may piggyback on other synchronization, thus being lighter weight. Alternatively, add a new MPI function that does not take an MPI_Win argument and just performs a suitable memory fence. This would make it explicit that UNIFIED does not need per-window synchronization. Should MPI be in the business of providing basic memory fences, or is the user expected to get it from their language (C11/C++11), their own inline assembly, or third-party libraries like ConcurrencyKit? As it is now, the user has to put in a fair bit of effort to write portable code with UNIFIED.

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Pavan Balaji 29 Jul '13

29 Jul '13

On 07/29/2013 02:27 PM, Sur, Sayantan wrote: >> Yes, doing a remote memory barrier using an active message is an option. >> But I think the point is that any of these approaches adds more unnecessary >> overhead. > > I'm not clear as to for which option we can avoid overhead of the barrier. > > UBER_SUPER_UNIFIED: MPI_Win_flush is truly one sided and utilizes network remote completion. > > KIND_OF_UNIFIED: You need to have the memory barrier somewhere, so you will move it to MPI_Win_sync. A well-optimized RMA app will avoid unnecessary flushes and only do so when it intends updates to be visible to the remote side. I think in the end, the cost of the barrier will end up being equal, since a well optimized app will synchronize (local or remote) as infrequently as possible. I'm not sure I follow your point here. For non-x86 to provide the currently specified UNIFIED model, it has to do active-message based remote memory barriers for every flush and unlock (even if the target process is not planning to read the data). With KIND_OF_UNIFIED, I'm saying that the user will need to WIN_SYNC, thus avoiding this overhead in flush. MPI will do the memory barrier in WIN_SYNC, when the user wants it. > I have a suspicion that KIND_OF_UNIFIED is going to end up being no > different than SEPARATE. Although I haven't thought this angle > through. It needs further clarification as to how KIND_OF_UNIFIED > differs from UBER_SUPER_UNIFIED. That was my first thought as well, but note that KIND_OF_UNIFIED is much stronger (and almost identical to UNIFIED), when we are talking about nonoverlapping memory. > " In the RMA unified model, public and private copies are identical and updates via put > or accumulate calls are eventually observed by load operations without additional RMA > calls." > > I take "eventually" to mean locks/flushes. But it seems that not everyone shares that interpretation? FLUSH will guarantee that the data is present in the remote public memory. And UNIFIED states that public memory == private memory. So, the user can read it directly without any additional MPI calls. That's the broken part. -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Pavan Balaji 29 Jul '13

29 Jul '13

On 07/29/2013 03:05 PM, Sur, Sayantan wrote: > I agree with the sentiment, also with the implementation issue of > UNIFIED with multi-rail. I'm not certain that a model exists in > between UNIFIED and SEPARATE. Either you are able to observe changes > in memory without further MPI calls, or not. Pavan do you have a > specific model in mind? Yes. I'd like a model where the user has to explicitly do a WIN_SYNC at the target. I agree that UNIFIED's current definition is stronger than this. Hence the request for either a new memory model or to weaken UNIFIED's definition. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Pavan Balaji 29 Jul '13

29 Jul '13

On 07/29/2013 01:58 PM, Jeff Hammond wrote: > I vehemently object to the definition of a new memory model that only > works for x86 (actually a subset thereof since Intel SCC is x86 but > not even cache-coherent), which is older than I am. If we're going to > make CPU-specific memory models for RMA, let's be fair and thorough > and define ones that work for PPC, DEC Alpha, and any other CPU memory > models that exist now or may in the future. > > WIN_SYNC is the right way to do this and requires no changes to the > standard and affects a very small number of users. How many people > are writing MPI-3 RMA code that aren't on this list? My position is that as long as there is a memory model that is usable by non-x86 architectures, that's sufficient. If we say that, that's UNIFIED, I'm happy with that. If we say that, that's not UNIFIED, but here's another memory model that does that, that'll work too. However, note that the point of these memory models is to allow more capable hardware to give more guarantees to the user. So, if some hardware is capable of doing more, I don't see any harm in exposing such a model to users. > Note also that I don't even think UBER_SUPER_UNIFIED is practical. > How does x86's memory model solve the problem of a multi-rail NIC > where the Send-Recv happens on rail 0 and the Put+Flush happens on the > other? Are MPI implementers going to be required to serialize all > communication through the NIC in order to support this model? I don't think this is a problem for multi-rail NICs. The flush already has to guarantee that the data is visible on the remote public memory, so it'll need to do the required synchronization. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Jed Brown 29 Jul '13

29 Jul '13

"Sur, Sayantan" <sayantan.sur(a)intel.com> writes: > What this discussion brings to light is a lack of common understanding of the following comment on Pg 436, line 37-39: > > " In the RMA unified model, public and private copies are identical and updates via put > or accumulate calls are eventually observed by load operations without additional RMA > calls." > > I take "eventually" to mean locks/flushes. But it seems that not everyone shares that interpretation? I think something along the lines of the subsequent advice section should have been in the main paragraph. Advice to users. If accesses in the RMA unified model are not synchronized (with locks or flushes, see Section 11.5.3), load and store operations might observe changes to the memory while they are in progress. The order in which data is written is not specified unless further synchronization is used. This might lead to inconsistent views on memory and programs that assume that a transfer is complete by only checking parts of the message are erroneous. (End of advice to users.) Specifically, architecture-specific coaxing (memory fences) might be required to see the result (ever; no guarantee that it happens "eventually" on its own), especially if one would like to preserve some ordering.

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Sur, Sayantan 29 Jul '13

29 Jul '13

> I vehemently object to the definition of a new memory model that only > works for x86 (actually a subset thereof since Intel SCC is x86 but not even > cache-coherent), which is older than I am. If we're going to make CPU- > specific memory models for RMA, let's be fair and thorough and define ones > that work for PPC, DEC Alpha, and any other CPU memory models that exist > now or may in the future. > > WIN_SYNC is the right way to do this and requires no changes to the > standard and affects a very small number of users. How many people are > writing MPI-3 RMA code that aren't on this list? > > Note also that I don't even think UBER_SUPER_UNIFIED is practical. > How does x86's memory model solve the problem of a multi-rail NIC where > the Send-Recv happens on rail 0 and the Put+Flush happens on the other? > Are MPI implementers going to be required to serialize all communication > through the NIC in order to support this model? > I agree with the sentiment, also with the implementation issue of UNIFIED with multi-rail. I'm not certain that a model exists in between UNIFIED and SEPARATE. Either you are able to observe changes in memory without further MPI calls, or not. Pavan do you have a specific model in mind? I do not agree that mandating MPI_Win_sync requires no changes to the standard. UNIFIED is clearly defined in the current spec as the model in which public and private copies of the window are identical. MPI_Win_sync is also clearly defined as synchronizing the private and public copies of the window. Therefore, we cannot simply just require UNIFIED to call MPI_Win_sync as that would be meaningless. Sayantan. > Jeff > > On Mon, Jul 29, 2013 at 1:41 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > Sayantan, > > > > Yes, doing a remote memory barrier using an active message is an option. > > But I think the point is that any of these approaches adds more > > unnecessary overhead. > > > > Separating into three memory models provides better flexibility to > > implementations (and to applications, once we allow them to ask for > > the "required" value through an info argument). > > > > -- Pavan > > > > > > On 07/29/2013 12:00 PM, Sur, Sayantan wrote: > >> > >> Hi Pavan, > >> > >>> Specifically, the concern was that some members of the WG believed > >>> that in the UNIFIED model, data is usable by the remote process > >>> after a PUT without an additional WIN_SYNC, while some members > >>> believed that it is not. > >>> Here's > >>> the example in question: > >>> > >>> P0: > >>> Win_lock_all > >>> Put(a, P1) > >>> Flush > >>> MPI_Send(P1) > >>> > >>> P1: > >>> Win_lock_all > >>> MPI_Recv(P0) > >>> read a > >>> > >>> The question was whether the above program was valid without a > >>> WIN_SYNC on P1 between the Recv(P0) and "read a". If we want this > >>> to be valid in the UNIFIED model, only x86-like architectures can > >>> provide UNIFIED efficiently. Other architectures, such as PPC or > >>> ARM, that require an additional read barrier on P1 will not be able > >>> to provide UNIFIED even if they are cache-coherent, unless they add > >>> a memory barrier in every other MPI call (e.g., MPI_Recv in this > >>> case). > >>> > >> > >> Adding a memory barrier in MPI_Recv is one of the implementation > >> options, and probably not the best one. For relaxed memory > >> architectures, one may want to shift the burden onto Flush to do a > >> memory barrier after the data has been written (through an active > message for example). > >> > >> The description of Flush in the spec is: "MPI_WIN_FLUSH completes all > >> outstanding RMA operations initiated by the calling process to the > >> target rank on the specified window. The operations are completed > >> both at the origin and at the target." > >> > >> The way I read it, no further action should be required to view > >> contents of the memory attached to the window after MPI_Win_flush. > >> Therefore, an implementation of MPI_Win_flush needs to do whatever is > >> required by the underlying platform and the model the MPI is supposed > to provide. > >> > >>> One possible solution we discussed was to clarify that this is not > >>> allowed in UNIFIED, but provide a third memory model called > >>> UBER_SUPER_UNIFIED, that will allow this. (or say that it is > >>> allowed in UNIFIED and provide a third model called KIND_OF_UNIFIED, > >>> which is in between UNIFIED and SEPARATE). > >>> > >>> Other solutions are welcome. > >>> > >>> Irrespective of when we make the change of possibly adding an > >>> additional memory model (MPI-3.1, MPI-4, whatever), we should > >>> clarify the standard on what is allowed in MPI-3 and what is not, as > >>> an errata item. Without that, it's confusing for implementors. > >>> > >>> -- Pavan > >>> > >>> -- > >>> Pavan Balaji > >>> http://www.mcs.anl.gov/~balaji > >>> _______________________________________________ > >>> mpi3-rma mailing list > >>> mpi3-rma(a)lists.mpi-forum.org > >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > >> > >> > >> _______________________________________________ > >> mpi3-rma mailing list > >> mpi3-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > >> > > > > -- > > Pavan Balaji > > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > > mpi3-rma mailing list > > mpi3-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Sur, Sayantan 29 Jul '13

29 Jul '13

> Yes, doing a remote memory barrier using an active message is an option. > But I think the point is that any of these approaches adds more unnecessary > overhead. > I'm not clear as to for which option we can avoid overhead of the barrier. UBER_SUPER_UNIFIED: MPI_Win_flush is truly one sided and utilizes network remote completion. KIND_OF_UNIFIED: You need to have the memory barrier somewhere, so you will move it to MPI_Win_sync. A well-optimized RMA app will avoid unnecessary flushes and only do so when it intends updates to be visible to the remote side. I think in the end, the cost of the barrier will end up being equal, since a well optimized app will synchronize (local or remote) as infrequently as possible. > Separating into three memory models provides better flexibility to > implementations (and to applications, once we allow them to ask for the > "required" value through an info argument). > I have a suspicion that KIND_OF_UNIFIED is going to end up being no different than SEPARATE. Although I haven't thought this angle through. It needs further clarification as to how KIND_OF_UNIFIED differs from UBER_SUPER_UNIFIED. What this discussion brings to light is a lack of common understanding of the following comment on Pg 436, line 37-39: " In the RMA unified model, public and private copies are identical and updates via put or accumulate calls are eventually observed by load operations without additional RMA calls." I take "eventually" to mean locks/flushes. But it seems that not everyone shares that interpretation? Thanks, Sayantan. > -- Pavan > > On 07/29/2013 12:00 PM, Sur, Sayantan wrote: > > Hi Pavan, > > > >> Specifically, the concern was that some members of the WG believed > >> that in the UNIFIED model, data is usable by the remote process after > >> a PUT without an additional WIN_SYNC, while some members believed > >> that it is not. Here's the example in question: > >> > >> P0: > >> Win_lock_all > >> Put(a, P1) > >> Flush > >> MPI_Send(P1) > >> > >> P1: > >> Win_lock_all > >> MPI_Recv(P0) > >> read a > >> > >> The question was whether the above program was valid without a > >> WIN_SYNC on P1 between the Recv(P0) and "read a". If we want this to > >> be valid in the UNIFIED model, only x86-like architectures can > >> provide UNIFIED efficiently. Other architectures, such as PPC or > >> ARM, that require an additional read barrier on P1 will not be able > >> to provide UNIFIED even if they are cache-coherent, unless they add a > >> memory barrier in every other MPI call (e.g., MPI_Recv in this case). > >> > > > > Adding a memory barrier in MPI_Recv is one of the implementation > options, and probably not the best one. For relaxed memory architectures, > one may want to shift the burden onto Flush to do a memory barrier after > the data has been written (through an active message for example). > > > > The description of Flush in the spec is: "MPI_WIN_FLUSH completes all > outstanding RMA operations initiated by the calling process to the target rank > on the specified window. The operations are completed both at the origin > and at the target." > > > > The way I read it, no further action should be required to view contents of > the memory attached to the window after MPI_Win_flush. Therefore, an > implementation of MPI_Win_flush needs to do whatever is required by the > underlying platform and the model the MPI is supposed to provide. > > > >> One possible solution we discussed was to clarify that this is not > >> allowed in UNIFIED, but provide a third memory model called > >> UBER_SUPER_UNIFIED, that will allow this. (or say that it is allowed > >> in UNIFIED and provide a third model called KIND_OF_UNIFIED, which is > >> in between UNIFIED and SEPARATE). > >> > >> Other solutions are welcome. > >> > >> Irrespective of when we make the change of possibly adding an > >> additional memory model (MPI-3.1, MPI-4, whatever), we should clarify > >> the standard on what is allowed in MPI-3 and what is not, as an > >> errata item. Without that, it's confusing for implementors. > >> > >> -- Pavan > >> > >> -- > >> Pavan Balaji > >> http://www.mcs.anl.gov/~balaji > >> _______________________________________________ > >> mpi3-rma mailing list > >> mpi3-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > > _______________________________________________ > > mpi3-rma mailing list > > mpi3-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Jeff Hammond 29 Jul '13

29 Jul '13

I vehemently object to the definition of a new memory model that only works for x86 (actually a subset thereof since Intel SCC is x86 but not even cache-coherent), which is older than I am. If we're going to make CPU-specific memory models for RMA, let's be fair and thorough and define ones that work for PPC, DEC Alpha, and any other CPU memory models that exist now or may in the future. WIN_SYNC is the right way to do this and requires no changes to the standard and affects a very small number of users. How many people are writing MPI-3 RMA code that aren't on this list? Note also that I don't even think UBER_SUPER_UNIFIED is practical. How does x86's memory model solve the problem of a multi-rail NIC where the Send-Recv happens on rail 0 and the Put+Flush happens on the other? Are MPI implementers going to be required to serialize all communication through the NIC in order to support this model? Jeff On Mon, Jul 29, 2013 at 1:41 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > Sayantan, > > Yes, doing a remote memory barrier using an active message is an option. > But I think the point is that any of these approaches adds more unnecessary > overhead. > > Separating into three memory models provides better flexibility to > implementations (and to applications, once we allow them to ask for the > "required" value through an info argument). > > -- Pavan > > > On 07/29/2013 12:00 PM, Sur, Sayantan wrote: >> >> Hi Pavan, >> >>> Specifically, the concern was that some members of the WG believed that >>> in >>> the UNIFIED model, data is usable by the remote process after a PUT >>> without >>> an additional WIN_SYNC, while some members believed that it is not. >>> Here's >>> the example in question: >>> >>> P0: >>> Win_lock_all >>> Put(a, P1) >>> Flush >>> MPI_Send(P1) >>> >>> P1: >>> Win_lock_all >>> MPI_Recv(P0) >>> read a >>> >>> The question was whether the above program was valid without a >>> WIN_SYNC on P1 between the Recv(P0) and "read a". If we want this to be >>> valid in the UNIFIED model, only x86-like architectures can provide >>> UNIFIED >>> efficiently. Other architectures, such as PPC or ARM, that require an >>> additional read barrier on P1 will not be able to provide UNIFIED even if >>> they >>> are cache-coherent, unless they add a memory barrier in every other MPI >>> call >>> (e.g., MPI_Recv in this case). >>> >> >> Adding a memory barrier in MPI_Recv is one of the implementation options, >> and probably not the best one. For relaxed memory architectures, one may >> want to shift the burden onto Flush to do a memory barrier after the data >> has been written (through an active message for example). >> >> The description of Flush in the spec is: "MPI_WIN_FLUSH completes all >> outstanding RMA operations initiated by the calling process to the target >> rank on the specified window. The operations are completed both at the >> origin and at the target." >> >> The way I read it, no further action should be required to view contents >> of the memory attached to the window after MPI_Win_flush. Therefore, an >> implementation of MPI_Win_flush needs to do whatever is required by the >> underlying platform and the model the MPI is supposed to provide. >> >>> One possible solution we discussed was to clarify that this is not >>> allowed in >>> UNIFIED, but provide a third memory model called UBER_SUPER_UNIFIED, >>> that will allow this. (or say that it is allowed in UNIFIED and provide >>> a third >>> model called KIND_OF_UNIFIED, which is in between UNIFIED and >>> SEPARATE). >>> >>> Other solutions are welcome. >>> >>> Irrespective of when we make the change of possibly adding an additional >>> memory model (MPI-3.1, MPI-4, whatever), we should clarify the standard >>> on what is allowed in MPI-3 and what is not, as an errata item. Without >>> that, >>> it's confusing for implementors. >>> >>> -- Pavan >>> >>> -- >>> Pavan Balaji >>> http://www.mcs.anl.gov/~balaji >>> _______________________________________________ >>> mpi3-rma mailing list >>> mpi3-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >> >> >> _______________________________________________ >> mpi3-rma mailing list >> mpi3-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma >> > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Pavan Balaji 29 Jul '13

29 Jul '13

Sayantan, Yes, doing a remote memory barrier using an active message is an option. But I think the point is that any of these approaches adds more unnecessary overhead. Separating into three memory models provides better flexibility to implementations (and to applications, once we allow them to ask for the "required" value through an info argument). -- Pavan On 07/29/2013 12:00 PM, Sur, Sayantan wrote: > Hi Pavan, > >> Specifically, the concern was that some members of the WG believed that in >> the UNIFIED model, data is usable by the remote process after a PUT without >> an additional WIN_SYNC, while some members believed that it is not. Here's >> the example in question: >> >> P0: >> Win_lock_all >> Put(a, P1) >> Flush >> MPI_Send(P1) >> >> P1: >> Win_lock_all >> MPI_Recv(P0) >> read a >> >> The question was whether the above program was valid without a >> WIN_SYNC on P1 between the Recv(P0) and "read a". If we want this to be >> valid in the UNIFIED model, only x86-like architectures can provide UNIFIED >> efficiently. Other architectures, such as PPC or ARM, that require an >> additional read barrier on P1 will not be able to provide UNIFIED even if they >> are cache-coherent, unless they add a memory barrier in every other MPI call >> (e.g., MPI_Recv in this case). >> > > Adding a memory barrier in MPI_Recv is one of the implementation options, and probably not the best one. For relaxed memory architectures, one may want to shift the burden onto Flush to do a memory barrier after the data has been written (through an active message for example). > > The description of Flush in the spec is: "MPI_WIN_FLUSH completes all outstanding RMA operations initiated by the calling process to the target rank on the specified window. The operations are completed both at the origin and at the target." > > The way I read it, no further action should be required to view contents of the memory attached to the window after MPI_Win_flush. Therefore, an implementation of MPI_Win_flush needs to do whatever is required by the underlying platform and the model the MPI is supposed to provide. > >> One possible solution we discussed was to clarify that this is not allowed in >> UNIFIED, but provide a third memory model called UBER_SUPER_UNIFIED, >> that will allow this. (or say that it is allowed in UNIFIED and provide a third >> model called KIND_OF_UNIFIED, which is in between UNIFIED and >> SEPARATE). >> >> Other solutions are welcome. >> >> Irrespective of when we make the change of possibly adding an additional >> memory model (MPI-3.1, MPI-4, whatever), we should clarify the standard >> on what is allowed in MPI-3 and what is not, as an errata item. Without that, >> it's confusing for implementors. >> >> -- Pavan >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji >> _______________________________________________ >> mpi3-rma mailing list >> mpi3-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Jeff Hammond 29 Jul '13

29 Jul '13

How does Flush invoke a memory barrier on the _remote_ side? Look at the code again. In the absence of a redefinition of the unified model (which Pavan discussed), there are really only two logical choices: (1) Recv calls an expensive memory barrier every time or (2) P1 invokes a memory barrier via MPI_Win_sync. I strongly favor (2). As much as I love RMA, it is totally inappropriate for RMA to burden other features of MPI with synchronization overhead, particularly when we already have a feature (WIN_SYNC) that is explicitly designed for this case. Jeff On Mon, Jul 29, 2013 at 12:00 PM, Sur, Sayantan <sayantan.sur(a)intel.com> wrote: > Hi Pavan, > >> Specifically, the concern was that some members of the WG believed that in >> the UNIFIED model, data is usable by the remote process after a PUT without >> an additional WIN_SYNC, while some members believed that it is not. Here's >> the example in question: >> >> P0: >> Win_lock_all >> Put(a, P1) >> Flush >> MPI_Send(P1) >> >> P1: >> Win_lock_all >> MPI_Recv(P0) >> read a >> >> The question was whether the above program was valid without a >> WIN_SYNC on P1 between the Recv(P0) and "read a". If we want this to be >> valid in the UNIFIED model, only x86-like architectures can provide UNIFIED >> efficiently. Other architectures, such as PPC or ARM, that require an >> additional read barrier on P1 will not be able to provide UNIFIED even if they >> are cache-coherent, unless they add a memory barrier in every other MPI call >> (e.g., MPI_Recv in this case). >> > > Adding a memory barrier in MPI_Recv is one of the implementation options, and probably not the best one. For relaxed memory architectures, one may want to shift the burden onto Flush to do a memory barrier after the data has been written (through an active message for example). > > The description of Flush in the spec is: "MPI_WIN_FLUSH completes all outstanding RMA operations initiated by the calling process to the target rank on the specified window. The operations are completed both at the origin and at the target." > > The way I read it, no further action should be required to view contents of the memory attached to the window after MPI_Win_flush. Therefore, an implementation of MPI_Win_flush needs to do whatever is required by the underlying platform and the model the MPI is supposed to provide. > >> One possible solution we discussed was to clarify that this is not allowed in >> UNIFIED, but provide a third memory model called UBER_SUPER_UNIFIED, >> that will allow this. (or say that it is allowed in UNIFIED and provide a third >> model called KIND_OF_UNIFIED, which is in between UNIFIED and >> SEPARATE). >> >> Other solutions are welcome. >> >> Irrespective of when we make the change of possibly adding an additional >> memory model (MPI-3.1, MPI-4, whatever), we should clarify the standard >> on what is allowed in MPI-3 and what is not, as an errata item. Without that, >> it's confusing for implementors. >> >> -- Pavan >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji >> _______________________________________________ >> mpi3-rma mailing list >> mpi3-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma -- Jeff Hammond jeff.science(a)gmail.com

1 0