July 2013 - mpiwg-rma - mailman3.cels.anl.gov

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Pavan Balaji 31 Jul '13

31 Jul '13

On 07/31/2013 12:58 PM, Sur, Sayantan wrote: >> On 07/31/2013 12:00 PM, Jim Dinan wrote: >>> I would bet that past Jim suggested striking the >>> polling/eventually visibile clause and relying on window >>> synchronization to see updates. :) >> >> Yup, so did past, present, and future Pavan. IMO, that's a useless >> guarantee. >> >>> The downside to this is that libraries like SHMEM that rely on >>> passive progress and polling, would not be implementable on top >>> of Unified. >> >> It's pretty useless even for SHMEM, since the user doesn't know >> when the data is valid. You could poll on a byte for it to turn to >> one, but at that point you only know about that one byte and >> nothing else. >> > > Past Sayantan had missed this discussion, but present Sayantan does > agree that "eventually" as defined is useless. But he is also > confused by the guarantee given by MPI_Win_flush, that when the call > returns, all previously issued RMA ops are complete locally and > remotely + UNIFIED (public win == private win). Unfortunately, that part slipped the checks of the folks who believed you need a WIN_SYNC. So the standard is inconsistent. We have now made a full circle and gotten back to where this email thread started :-). -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Jeff Hammond 31 Jul '13

31 Jul '13

Maybe the RMA WG should write a paper for http://www.csm.ornl.gov/workshops/openshmem2013/ that explains why passive polling on bytes is an indicator of a bad programming model and how they should fix their semantics :-) Jeff On Wed, Jul 31, 2013 at 12:58 PM, Sur, Sayantan <sayantan.sur(a)intel.com> wrote: >> On 07/31/2013 12:00 PM, Jim Dinan wrote: >> > I would bet that past Jim suggested striking the polling/eventually >> > visibile clause and relying on window synchronization to see updates. >> > :) >> >> Yup, so did past, present, and future Pavan. IMO, that's a useless guarantee. >> >> > The downside to this is that libraries like SHMEM that rely on passive >> > progress and polling, would not be implementable on top of Unified. >> >> It's pretty useless even for SHMEM, since the user doesn't know when the >> data is valid. You could poll on a byte for it to turn to one, but at that point >> you only know about that one byte and nothing else. >> > > Past Sayantan had missed this discussion, but present Sayantan does agree that "eventually" as defined is useless. But he is also confused by the guarantee given by MPI_Win_flush, that when the call returns, all previously issued RMA ops are complete locally and remotely + UNIFIED (public win == private win). > > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Sur, Sayantan 31 Jul '13

31 Jul '13

> On 07/31/2013 12:00 PM, Jim Dinan wrote: > > I would bet that past Jim suggested striking the polling/eventually > > visibile clause and relying on window synchronization to see updates. > > :) > > Yup, so did past, present, and future Pavan. IMO, that's a useless guarantee. > > > The downside to this is that libraries like SHMEM that rely on passive > > progress and polling, would not be implementable on top of Unified. > > It's pretty useless even for SHMEM, since the user doesn't know when the > data is valid. You could poll on a byte for it to turn to one, but at that point > you only know about that one byte and nothing else. > Past Sayantan had missed this discussion, but present Sayantan does agree that "eventually" as defined is useless. But he is also confused by the guarantee given by MPI_Win_flush, that when the call returns, all previously issued RMA ops are complete locally and remotely + UNIFIED (public win == private win).

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Pavan Balaji 31 Jul '13

31 Jul '13

On 07/31/2013 12:00 PM, Jim Dinan wrote: > I would bet that past Jim suggested striking the polling/eventually > visibile clause and relying on window synchronization to see updates. > :) Yup, so did past, present, and future Pavan. IMO, that's a useless guarantee. > The downside to this is that libraries like SHMEM that rely on > passive progress and polling, would not be implementable on top of Unified. It's pretty useless even for SHMEM, since the user doesn't know when the data is valid. You could poll on a byte for it to turn to one, but at that point you only know about that one byte and nothing else. -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Jed Brown 31 Jul '13

31 Jul '13

Jim Dinan <james.dinan(a)gmail.com> writes: > Another question that was raised outside of this email discussion is > whether we can rely on the architecture (in the absence of MPI calls) to > make the results of operations visible in the order in which the origin > process expected (e.g. through ordered accumulates, flushes, etc). I'm not sure exactly what the question was, but you certainly don't get (portable) read memory ordering without memory barriers on the read side. Recall that explicit data or control dependencies are not enough (Alpha and ARMv7, respectively).

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Jim Dinan 31 Jul '13

31 Jul '13

I would bet that past Jim suggested striking the polling/eventually visibile clause and relying on window synchronization to see updates. :) The downside to this is that libraries like SHMEM that rely on passive progress and polling, would not be implementable on top of Unified. Another question that was raised outside of this email discussion is whether we can rely on the architecture (in the absence of MPI calls) to make the results of operations visible in the order in which the origin process expected (e.g. through ordered accumulates, flushes, etc). ~Jim. On Tue, Jul 30, 2013 at 6:45 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > On 07/30/2013 05:33 PM, Sur, Sayantan wrote: > >> On 07/30/2013 10:28 AM, Jim Dinan wrote: >>> >>>> I believe this is adequately specified on pg 436, line 37. P1 will >>>> *eventually* see the new value for "a" without any additional >>>> synchronization operations, but neither the flush by P0 nor the Recv >>>> by >>>> P1 guarantee that P1 will see the new value immediately. >>>> >>> >>> This is what I said is the disagreement in the WG. I can pull up the >>> old email >>> chain if needed, but I think others can too. One side was arguing that >>> there's >>> no such guarantee and you need to do a WIN_SYNC to see the value. The >>> other side was arguing that the WIN_SYNC should not be needed; FLUSH + >>> SEND on the origin should be enough. >>> >>> >> Here's the old thread: http://lists.mpi-forum.org/** >> mpi3-rma/2011/03/0533.php<http://lists.mpi-forum.org/mpi3-rma/2011/03/0533.php> >> >> Looks like the idea to call MPI_Win_sync for Unified got votes from both >> Bill and Torsten. Were there others who were in this camp? >> > > IIRC, Torsten was against it. Jim and I were arguing for it (i.e., having > to call WIN_SYNC in UNIFIED). > > -- Pavan > > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > ______________________________**_________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/**mailman/listinfo.cgi/mpi3-rma<http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma> >

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Pavan Balaji 30 Jul '13

30 Jul '13

This is needed for UNIFIED. If the hardware doesn't support this, it can still support SEPARATE. On 07/30/2013 06:01 PM, Jeff Hammond wrote: > I would really not like to use the very narrow scope of today's > architectures to influence fundamental decisions like the RMA memory > model. Things are going to change... > > Jeff > > Sent from my iPhone > > On Jul 30, 2013, at 5:51 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > >> >> While this might be true in the general case, I don't think this is true on any real network today. Before the DMA is permitted, the cache is flushed/invalidated. >> >> -- Pavan >> >> On 07/30/2013 05:40 PM, Jed Brown wrote: >>> Pavan Balaji <balaji(a)mcs.anl.gov> writes: >>>> This is what I said is the disagreement in the WG. I can pull up the >>>> old email chain if needed, but I think others can too. One side was >>>> arguing that there's no such guarantee and you need to do a WIN_SYNC to >>>> see the value. The other side was arguing that the WIN_SYNC should not >>>> be needed; FLUSH + SEND on the origin should be enough. >>> >>> Hmm, linux/Documentation/memory-barriers.txt says: >>> >>> CACHE COHERENCY VS DMA >>> ---------------------- >>> >>> Not all systems maintain cache coherency with respect to devices doing >>> DMA. In such cases, a device attempting DMA may obtain stale data >>> from RAM because dirty cache lines may be resident in the caches of >>> various CPUs, and may not have been written back to RAM yet. To deal >>> with this, the appropriate part of the kernel must flush the >>> overlapping bits of cache on each CPU (and maybe invalidate them as >>> well). >>> >>> In addition, the data DMA'd to RAM by a device may be overwritten by >>> dirty cache lines being written back to RAM from a CPU's cache after >>> the device has installed its own data, or cache lines present in the >>> CPU's cache may simply obscure the fact that RAM has been updated, >>> until at such time as the cacheline is discarded from the CPU's cache >>> and reloaded. To deal with this, the appropriate part of the kernel >>> must invalidate the overlapping bits of the cache on each CPU. >>> >>> >>> I've taken this to mean that you can't guarantee that a DMA write will >>> "eventually" be visible to the CPU (because the cache line could hang >>> around arbitrarily long). Are implementations doing something here to >>> ensure that cache on the target is invalidated after RDMA operations? >>> >>> I think (though not with confidence) that CPU cache invalidation will >>> eventually (with a practical upper bound) propagate to other CPUs. >>> Comparing to Alpha with split cache lines, one of the buses could be >>> busy and thus not update despite proper memory ordering on the write >>> end. The implication is that each cache has a (fair?) queue and it >>> cannot be arbitrarily long, though I've never seen a statement providing >>> an upper bound on how long the cache bank could be busy. >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji >> _______________________________________________ >> mpi3-rma mailing list >> mpi3-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Jeff Hammond 30 Jul '13

30 Jul '13

I would really not like to use the very narrow scope of today's architectures to influence fundamental decisions like the RMA memory model. Things are going to change... Jeff Sent from my iPhone On Jul 30, 2013, at 5:51 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > While this might be true in the general case, I don't think this is true on any real network today. Before the DMA is permitted, the cache is flushed/invalidated. > > -- Pavan > > On 07/30/2013 05:40 PM, Jed Brown wrote: >> Pavan Balaji <balaji(a)mcs.anl.gov> writes: >>> This is what I said is the disagreement in the WG. I can pull up the >>> old email chain if needed, but I think others can too. One side was >>> arguing that there's no such guarantee and you need to do a WIN_SYNC to >>> see the value. The other side was arguing that the WIN_SYNC should not >>> be needed; FLUSH + SEND on the origin should be enough. >> >> Hmm, linux/Documentation/memory-barriers.txt says: >> >> CACHE COHERENCY VS DMA >> ---------------------- >> >> Not all systems maintain cache coherency with respect to devices doing >> DMA. In such cases, a device attempting DMA may obtain stale data >> from RAM because dirty cache lines may be resident in the caches of >> various CPUs, and may not have been written back to RAM yet. To deal >> with this, the appropriate part of the kernel must flush the >> overlapping bits of cache on each CPU (and maybe invalidate them as >> well). >> >> In addition, the data DMA'd to RAM by a device may be overwritten by >> dirty cache lines being written back to RAM from a CPU's cache after >> the device has installed its own data, or cache lines present in the >> CPU's cache may simply obscure the fact that RAM has been updated, >> until at such time as the cacheline is discarded from the CPU's cache >> and reloaded. To deal with this, the appropriate part of the kernel >> must invalidate the overlapping bits of the cache on each CPU. >> >> >> I've taken this to mean that you can't guarantee that a DMA write will >> "eventually" be visible to the CPU (because the cache line could hang >> around arbitrarily long). Are implementations doing something here to >> ensure that cache on the target is invalidated after RDMA operations? >> >> I think (though not with confidence) that CPU cache invalidation will >> eventually (with a practical upper bound) propagate to other CPUs. >> Comparing to Alpha with split cache lines, one of the buses could be >> busy and thus not update despite proper memory ordering on the write >> end. The implication is that each cache has a (fair?) queue and it >> cannot be arbitrarily long, though I've never seen a statement providing >> an upper bound on how long the cache bank could be busy. > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Pavan Balaji 30 Jul '13

30 Jul '13

While this might be true in the general case, I don't think this is true on any real network today. Before the DMA is permitted, the cache is flushed/invalidated. -- Pavan On 07/30/2013 05:40 PM, Jed Brown wrote: > Pavan Balaji <balaji(a)mcs.anl.gov> writes: >> This is what I said is the disagreement in the WG. I can pull up the >> old email chain if needed, but I think others can too. One side was >> arguing that there's no such guarantee and you need to do a WIN_SYNC to >> see the value. The other side was arguing that the WIN_SYNC should not >> be needed; FLUSH + SEND on the origin should be enough. > > Hmm, linux/Documentation/memory-barriers.txt says: > > CACHE COHERENCY VS DMA > ---------------------- > > Not all systems maintain cache coherency with respect to devices doing > DMA. In such cases, a device attempting DMA may obtain stale data > from RAM because dirty cache lines may be resident in the caches of > various CPUs, and may not have been written back to RAM yet. To deal > with this, the appropriate part of the kernel must flush the > overlapping bits of cache on each CPU (and maybe invalidate them as > well). > > In addition, the data DMA'd to RAM by a device may be overwritten by > dirty cache lines being written back to RAM from a CPU's cache after > the device has installed its own data, or cache lines present in the > CPU's cache may simply obscure the fact that RAM has been updated, > until at such time as the cacheline is discarded from the CPU's cache > and reloaded. To deal with this, the appropriate part of the kernel > must invalidate the overlapping bits of the cache on each CPU. > > > I've taken this to mean that you can't guarantee that a DMA write will > "eventually" be visible to the CPU (because the cache line could hang > around arbitrarily long). Are implementations doing something here to > ensure that cache on the target is invalidated after RDMA operations? > > I think (though not with confidence) that CPU cache invalidation will > eventually (with a practical upper bound) propagate to other CPUs. > Comparing to Alpha with split cache lines, one of the buses could be > busy and thus not update despite proper memory ordering on the write > end. The implication is that each cache has a (fair?) queue and it > cannot be arbitrarily long, though I've never seen a statement providing > an upper bound on how long the cache bank could be busy. > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] MPI-3 UNIFIED model clarification
by Pavan Balaji 30 Jul '13

30 Jul '13

On 07/30/2013 05:33 PM, Sur, Sayantan wrote: >> On 07/30/2013 10:28 AM, Jim Dinan wrote: >>> I believe this is adequately specified on pg 436, line 37. P1 will >>> *eventually* see the new value for "a" without any additional >>> synchronization operations, but neither the flush by P0 nor the Recv >>> by >>> P1 guarantee that P1 will see the new value immediately. >> >> This is what I said is the disagreement in the WG. I can pull up the old email >> chain if needed, but I think others can too. One side was arguing that there's >> no such guarantee and you need to do a WIN_SYNC to see the value. The >> other side was arguing that the WIN_SYNC should not be needed; FLUSH + >> SEND on the origin should be enough. >> > > Here's the old thread: http://lists.mpi-forum.org/mpi3-rma/2011/03/0533.php > > Looks like the idea to call MPI_Win_sync for Unified got votes from both Bill and Torsten. Were there others who were in this camp? IIRC, Torsten was against it. Jim and I were arguing for it (i.e., having to call WIN_SYNC in UNIFIED). -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0