February 2015 - mpiwg-rma - mailman3.cels.anl.gov

Re: [mpiwg-rma] Ticket #458
by Jim Dinan 06 Feb '15

06 Feb '15

I believe this ticket exists because an implementer asked if any outcome was allowed, including the machine catching on fire. It seems useful to clarify what we mean by outcome, since we use this term throughout the text. ~Jim. On Fri, Feb 6, 2015 at 12:04 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > I think the chapter committee should handle this. I think the current > text is obvious enough that no one is going to screw it up, but that > the clarification has value and thus should be implemented. > > Best, > > Jeff > > On Fri, Feb 6, 2015 at 8:43 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: > > Hi All, > > > > The following ticket is marked as MPI 3.0 errata, but appears not to have > > gone forward for a vote: > > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/458 > > > > Did we either abandon this or decide that it should be handled by the > > chapter committee? It seems likely that it was the latter and I forgot > to > > update the ticket. Just want to make sure it doesn't get lost in the > > shuffle. > > > > Thanks, > > ~Jim. > > > > ---8<--- > > > > Here is the proposed change. > > > > Existing Text (Pg. 418, line 44): > > > > The outcome of concurrent conflicting accesses to the same memory > locations > > is undefined... > > > > Proposed Change > > > > The resulting data values, or outcome, of concurrent conflicting > accesses to > > the same memory locations is undefined... > > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >

1 0

Re: [mpiwg-rma] Ticket #458
by Jeff Hammond 06 Feb '15

06 Feb '15

I think the chapter committee should handle this. I think the current text is obvious enough that no one is going to screw it up, but that the clarification has value and thus should be implemented. Best, Jeff On Fri, Feb 6, 2015 at 8:43 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: > Hi All, > > The following ticket is marked as MPI 3.0 errata, but appears not to have > gone forward for a vote: > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/458 > > Did we either abandon this or decide that it should be handled by the > chapter committee? It seems likely that it was the latter and I forgot to > update the ticket. Just want to make sure it doesn't get lost in the > shuffle. > > Thanks, > ~Jim. > > ---8<--- > > Here is the proposed change. > > Existing Text (Pg. 418, line 44): > > The outcome of concurrent conflicting accesses to the same memory locations > is undefined... > > Proposed Change > > The resulting data values, or outcome, of concurrent conflicting accesses to > the same memory locations is undefined... > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Ticket #458
by Jim Dinan 06 Feb '15

06 Feb '15

Hi All, The following ticket is marked as MPI 3.0 errata, but appears not to have gone forward for a vote: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/458 Did we either abandon this or decide that it should be handled by the chapter committee? It seems likely that it was the latter and I forgot to update the ticket. Just want to make sure it doesn't get lost in the shuffle. Thanks, ~Jim. ---8<--- Here is the proposed change. Existing Text (Pg. 418, line 44): The outcome of concurrent conflicting accesses to the same memory locations is undefined... Proposed Change The resulting *data values, or outcome,* of concurrent conflicting accesses to the same memory locations is undefined...

1 0

Re: [mpiwg-rma] MPI_Win_all_lock_all
by Balaji, Pavan 04 Feb '15

04 Feb '15

It doesn't in the common case. The only case locks are pre-acquired is when it runs into resource exhaustion and doesn't have memory to store target-specific lock information. Mode nocheck is optimized irrespective, so this is not a concern. -- Pavan Sent from my iPhone On Feb 4, 2015, at 8:50 AM, Jim Dinan <james.dinan(a)gmail.com<mailto:[email protected]>> wrote: Does MPICH actually go out and acquire all of the shared locks immediately? This used to be deferred and it only acquired the locks on processes that you actually communicate with. Beyond that, MPICH should also piggyback the lock requests on the first RMA operation to a given target, when you give MPI_MODE_NOCHECK. So, if you use the interface in the way intended, there should not be any extra communication from a lock_all. ~Jim. On Tue, Feb 3, 2015 at 1:09 AM, Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> wrote: I know we discussed a collective lock_all function at some point. Why didn't we do it? In every single MPI-3 RMA program I have every written, I do this: ... MPI_Win_allocate(..,&win); MPI_Win_lock_all(win); ... When everyone calls MPI_Win_lock_all, the traffic pattern is consistent with MPI_Alltoall. On the other hand, a collective invocation would allow for setting the LOCK_ALL state locally and merely achieving consensus on this effect, which means MPI_Allreduce traffic. In terms of the performance difference one will see with these implementations, here is some data for Blue Gene/Q on 48 racks (49152 nodes with 16 ppn): MPI_Allreduce 1 integers in 0.000018 seconds MPI_Alltoall 786432 integers in 1.52 seconds <- the count here means one integer per process For those not keeping score at home, that's a performance difference of approximately 36314x. And this is quite generous to MPI_Win_lock_all, because not only is it not collective and therefore MPI cannot possibly coalesce or otherwise optimize the packet storm, but instead of a simple data copy, an MPICH:Ch3-derived implementation is going to slam the target with nproc active message requests, leading to an enormously pile up at scale. Here is more data (approximate, due to lack of repetition in measurement, although the times do not include connection overhead, etc.): Cray XC30 np=8192 MPI_Allreduce 1 integers in 0.000033 seconds MPI_Alltoall 8192 integers in 0.0051 seconds np=8192 (1024x8) MPI_Allreduce 2 integers in 0.000015 seconds MPI_Alltoall 8192 integers in 0.0034 seconds In the limit of small np, the differences are less noticeable, as one might expect. Unless there's a compelling reason not to fix this, I'll propose either MPI_Win_all_lock_all. I have no use case for locking a subset of the window group collectively, so I do not intend to propose this. Thanks, Jeff -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> http://jeffhammond.github.io/ _______________________________________________ mpiwg-rma mailing list mpiwg-rma(a)lists.mpi-forum.org<mailto:[email protected]> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma _______________________________________________ mpiwg-rma mailing list mpiwg-rma(a)lists.mpi-forum.org<mailto:[email protected]> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] MPI_Win_all_lock_all
by William Gropp 04 Feb '15

04 Feb '15

It shouldn’t, for exactly this reason. The place to fix this behavior is in the implementations, not in the standard. Bill On Feb 4, 2015, at 8:50 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: > Does MPICH actually go out and acquire all of the shared locks immediately? This used to be deferred and it only acquired the locks on processes that you actually communicate with. Beyond that, MPICH should also piggyback the lock requests on the first RMA operation to a given target, when you give MPI_MODE_NOCHECK. So, if you use the interface in the way intended, there should not be any extra communication from a lock_all. > > ~Jim. > > > > On Tue, Feb 3, 2015 at 1:09 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > I know we discussed a collective lock_all function at some point. Why > didn't we do it? > > In every single MPI-3 RMA program I have every written, I do this: > > ... > MPI_Win_allocate(..,&win); > MPI_Win_lock_all(win); > ... > > When everyone calls MPI_Win_lock_all, the traffic pattern is > consistent with MPI_Alltoall. On the other hand, a collective > invocation would allow for setting the LOCK_ALL state locally and > merely achieving consensus on this effect, which means MPI_Allreduce > traffic. > > In terms of the performance difference one will see with these > implementations, here is some data for Blue Gene/Q on 48 racks (49152 > nodes with 16 ppn): > > MPI_Allreduce 1 integers in 0.000018 seconds > MPI_Alltoall 786432 integers in 1.52 seconds <- the count here means > one integer per process > > For those not keeping score at home, that's a performance difference > of approximately 36314x. And this is quite generous to > MPI_Win_lock_all, because not only is it not collective and therefore > MPI cannot possibly coalesce or otherwise optimize the packet storm, > but instead of a simple data copy, an MPICH:Ch3-derived implementation > is going to slam the target with nproc active message requests, > leading to an enormously pile up at scale. > > Here is more data (approximate, due to lack of repetition in > measurement, although the times do not include connection overhead, > etc.): > > Cray XC30 > np=8192 > MPI_Allreduce 1 integers in 0.000033 seconds > MPI_Alltoall 8192 integers in 0.0051 seconds > > np=8192 (1024x8) > MPI_Allreduce 2 integers in 0.000015 seconds > MPI_Alltoall 8192 integers in 0.0034 seconds > > In the limit of small np, the differences are less noticeable, as one > might expect. > > Unless there's a compelling reason not to fix this, I'll propose > either MPI_Win_all_lock_all. I have no use case for locking a subset > of the window group collectively, so I do not intend to propose this. > > Thanks, > > Jeff > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] MPI_Win_all_lock_all
by Jim Dinan 04 Feb '15

04 Feb '15

Does MPICH actually go out and acquire all of the shared locks immediately? This used to be deferred and it only acquired the locks on processes that you actually communicate with. Beyond that, MPICH should also piggyback the lock requests on the first RMA operation to a given target, when you give MPI_MODE_NOCHECK. So, if you use the interface in the way intended, there should not be any extra communication from a lock_all. ~Jim. On Tue, Feb 3, 2015 at 1:09 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > I know we discussed a collective lock_all function at some point. Why > didn't we do it? > > In every single MPI-3 RMA program I have every written, I do this: > > ... > MPI_Win_allocate(..,&win); > MPI_Win_lock_all(win); > ... > > When everyone calls MPI_Win_lock_all, the traffic pattern is > consistent with MPI_Alltoall. On the other hand, a collective > invocation would allow for setting the LOCK_ALL state locally and > merely achieving consensus on this effect, which means MPI_Allreduce > traffic. > > In terms of the performance difference one will see with these > implementations, here is some data for Blue Gene/Q on 48 racks (49152 > nodes with 16 ppn): > > MPI_Allreduce 1 integers in 0.000018 seconds > MPI_Alltoall 786432 integers in 1.52 seconds <- the count here means > one integer per process > > For those not keeping score at home, that's a performance difference > of approximately 36314x. And this is quite generous to > MPI_Win_lock_all, because not only is it not collective and therefore > MPI cannot possibly coalesce or otherwise optimize the packet storm, > but instead of a simple data copy, an MPICH:Ch3-derived implementation > is going to slam the target with nproc active message requests, > leading to an enormously pile up at scale. > > Here is more data (approximate, due to lack of repetition in > measurement, although the times do not include connection overhead, > etc.): > > Cray XC30 > np=8192 > MPI_Allreduce 1 integers in 0.000033 seconds > MPI_Alltoall 8192 integers in 0.0051 seconds > > np=8192 (1024x8) > MPI_Allreduce 2 integers in 0.000015 seconds > MPI_Alltoall 8192 integers in 0.0034 seconds > > In the limit of small np, the differences are less noticeable, as one > might expect. > > Unless there's a compelling reason not to fix this, I'll propose > either MPI_Win_all_lock_all. I have no use case for locking a subset > of the window group collectively, so I do not intend to propose this. > > Thanks, > > Jeff > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >

1 0

Re: [mpiwg-rma] MPI_Win_all_lock_all
by Underwood, Keith D 03 Feb '15

03 Feb '15

The expectation was that people who really wanted to do that would set MPI_MODE_NOCHECK and SHARED for the lock and handle the locking themselves in between calls to MPI_WIN_FLUSH and MPI_WIN_SYNC. This turns MPI_WIN_LOCK_ALL() into kind of a syntactically required, start of program, no-op. As such, further optimization of it was not performed. > -----Original Message----- > From: mpiwg-rma [mailto:[email protected]] On > Behalf Of Jeff Hammond > Sent: Tuesday, February 03, 2015 1:09 AM > To: MPI WG Remote Memory Access working group > Subject: [mpiwg-rma] MPI_Win_all_lock_all > > I know we discussed a collective lock_all function at some point. Why didn't > we do it? > > In every single MPI-3 RMA program I have every written, I do this: > > ... > MPI_Win_allocate(..,&win); > MPI_Win_lock_all(win); > ... > > When everyone calls MPI_Win_lock_all, the traffic pattern is consistent with > MPI_Alltoall. On the other hand, a collective invocation would allow for > setting the LOCK_ALL state locally and merely achieving consensus on this > effect, which means MPI_Allreduce traffic. > > In terms of the performance difference one will see with these > implementations, here is some data for Blue Gene/Q on 48 racks (49152 > nodes with 16 ppn): > > MPI_Allreduce 1 integers in 0.000018 seconds MPI_Alltoall 786432 integers > in 1.52 seconds <- the count here means one integer per process > > For those not keeping score at home, that's a performance difference of > approximately 36314x. And this is quite generous to MPI_Win_lock_all, > because not only is it not collective and therefore MPI cannot possibly > coalesce or otherwise optimize the packet storm, but instead of a simple > data copy, an MPICH:Ch3-derived implementation is going to slam the target > with nproc active message requests, leading to an enormously pile up at > scale. > > Here is more data (approximate, due to lack of repetition in measurement, > although the times do not include connection overhead, > etc.): > > Cray XC30 > np=8192 > MPI_Allreduce 1 integers in 0.000033 seconds MPI_Alltoall 8192 integers in > 0.0051 seconds > > np=8192 (1024x8) > MPI_Allreduce 2 integers in 0.000015 seconds MPI_Alltoall 8192 integers in > 0.0034 seconds > > In the limit of small np, the differences are less noticeable, as one might > expect. > > Unless there's a compelling reason not to fix this, I'll propose either > MPI_Win_all_lock_all. I have no use case for locking a subset of the window > group collectively, so I do not intend to propose this. > > Thanks, > > Jeff > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] MPI_Win_all_lock_all
by Jeff Hammond 03 Feb '15

03 Feb '15

MPI_Win_fence is completely useless in this context since I cannot call MPI_Win_flush, etc. within a FENCE epoch. So you are saying that MPI_MODE_NOCHECK turns MPI_Win_lock_all into a local operation? I'm fine with that if people agree it's a valid optimization. Jeff On Mon, Feb 2, 2015 at 10:16 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > The argument was that we don't want to add a collective function in passive target RMA, and should instead use Fence. Specifically, since all-lock-all forces shared lock (it doesn't make sense for exclusive lock), one would give mode-no-check, and the MPI implementation can optimize the lock functionality to a no-op anyway. > > However, if there was a proposal for collective flush or unlock, I'd be supportive of that. The reason being that I can decide dynamically if I want to do a single flush-all or a collective flush-all based on my program semantics, instead of having to decide that at epoch opening time. > > -- Pavan > > Sent from my iPhone > >> On Feb 3, 2015, at 11:39 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >> >> I know we discussed a collective lock_all function at some point. Why >> didn't we do it? >> >> In every single MPI-3 RMA program I have every written, I do this: >> >> ... >> MPI_Win_allocate(..,&win); >> MPI_Win_lock_all(win); >> ... >> >> When everyone calls MPI_Win_lock_all, the traffic pattern is >> consistent with MPI_Alltoall. On the other hand, a collective >> invocation would allow for setting the LOCK_ALL state locally and >> merely achieving consensus on this effect, which means MPI_Allreduce >> traffic. >> >> In terms of the performance difference one will see with these >> implementations, here is some data for Blue Gene/Q on 48 racks (49152 >> nodes with 16 ppn): >> >> MPI_Allreduce 1 integers in 0.000018 seconds >> MPI_Alltoall 786432 integers in 1.52 seconds <- the count here means >> one integer per process >> >> For those not keeping score at home, that's a performance difference >> of approximately 36314x. And this is quite generous to >> MPI_Win_lock_all, because not only is it not collective and therefore >> MPI cannot possibly coalesce or otherwise optimize the packet storm, >> but instead of a simple data copy, an MPICH:Ch3-derived implementation >> is going to slam the target with nproc active message requests, >> leading to an enormously pile up at scale. >> >> Here is more data (approximate, due to lack of repetition in >> measurement, although the times do not include connection overhead, >> etc.): >> >> Cray XC30 >> np=8192 >> MPI_Allreduce 1 integers in 0.000033 seconds >> MPI_Alltoall 8192 integers in 0.0051 seconds >> >> np=8192 (1024x8) >> MPI_Allreduce 2 integers in 0.000015 seconds >> MPI_Alltoall 8192 integers in 0.0034 seconds >> >> In the limit of small np, the differences are less noticeable, as one >> might expect. >> >> Unless there's a compelling reason not to fix this, I'll propose >> either MPI_Win_all_lock_all. I have no use case for locking a subset >> of the window group collectively, so I do not intend to propose this. >> >> Thanks, >> >> Jeff >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> http://jeffhammond.github.io/ >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpiwg-rma] MPI_Win_all_lock_all
by Balaji, Pavan 03 Feb '15

03 Feb '15

The argument was that we don't want to add a collective function in passive target RMA, and should instead use Fence. Specifically, since all-lock-all forces shared lock (it doesn't make sense for exclusive lock), one would give mode-no-check, and the MPI implementation can optimize the lock functionality to a no-op anyway. However, if there was a proposal for collective flush or unlock, I'd be supportive of that. The reason being that I can decide dynamically if I want to do a single flush-all or a collective flush-all based on my program semantics, instead of having to decide that at epoch opening time. -- Pavan Sent from my iPhone > On Feb 3, 2015, at 11:39 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > I know we discussed a collective lock_all function at some point. Why > didn't we do it? > > In every single MPI-3 RMA program I have every written, I do this: > > ... > MPI_Win_allocate(..,&win); > MPI_Win_lock_all(win); > ... > > When everyone calls MPI_Win_lock_all, the traffic pattern is > consistent with MPI_Alltoall. On the other hand, a collective > invocation would allow for setting the LOCK_ALL state locally and > merely achieving consensus on this effect, which means MPI_Allreduce > traffic. > > In terms of the performance difference one will see with these > implementations, here is some data for Blue Gene/Q on 48 racks (49152 > nodes with 16 ppn): > > MPI_Allreduce 1 integers in 0.000018 seconds > MPI_Alltoall 786432 integers in 1.52 seconds <- the count here means > one integer per process > > For those not keeping score at home, that's a performance difference > of approximately 36314x. And this is quite generous to > MPI_Win_lock_all, because not only is it not collective and therefore > MPI cannot possibly coalesce or otherwise optimize the packet storm, > but instead of a simple data copy, an MPICH:Ch3-derived implementation > is going to slam the target with nproc active message requests, > leading to an enormously pile up at scale. > > Here is more data (approximate, due to lack of repetition in > measurement, although the times do not include connection overhead, > etc.): > > Cray XC30 > np=8192 > MPI_Allreduce 1 integers in 0.000033 seconds > MPI_Alltoall 8192 integers in 0.0051 seconds > > np=8192 (1024x8) > MPI_Allreduce 2 integers in 0.000015 seconds > MPI_Alltoall 8192 integers in 0.0034 seconds > > In the limit of small np, the differences are less noticeable, as one > might expect. > > Unless there's a compelling reason not to fix this, I'll propose > either MPI_Win_all_lock_all. I have no use case for locking a subset > of the window group collectively, so I do not intend to propose this. > > Thanks, > > Jeff > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

MPI_Win_all_lock_all
by Jeff Hammond 03 Feb '15

03 Feb '15

I know we discussed a collective lock_all function at some point. Why didn't we do it? In every single MPI-3 RMA program I have every written, I do this: ... MPI_Win_allocate(..,&win); MPI_Win_lock_all(win); ... When everyone calls MPI_Win_lock_all, the traffic pattern is consistent with MPI_Alltoall. On the other hand, a collective invocation would allow for setting the LOCK_ALL state locally and merely achieving consensus on this effect, which means MPI_Allreduce traffic. In terms of the performance difference one will see with these implementations, here is some data for Blue Gene/Q on 48 racks (49152 nodes with 16 ppn): MPI_Allreduce 1 integers in 0.000018 seconds MPI_Alltoall 786432 integers in 1.52 seconds <- the count here means one integer per process For those not keeping score at home, that's a performance difference of approximately 36314x. And this is quite generous to MPI_Win_lock_all, because not only is it not collective and therefore MPI cannot possibly coalesce or otherwise optimize the packet storm, but instead of a simple data copy, an MPICH:Ch3-derived implementation is going to slam the target with nproc active message requests, leading to an enormously pile up at scale. Here is more data (approximate, due to lack of repetition in measurement, although the times do not include connection overhead, etc.): Cray XC30 np=8192 MPI_Allreduce 1 integers in 0.000033 seconds MPI_Alltoall 8192 integers in 0.0051 seconds np=8192 (1024x8) MPI_Allreduce 2 integers in 0.000015 seconds MPI_Alltoall 8192 integers in 0.0034 seconds In the limit of small np, the differences are less noticeable, as one might expect. Unless there's a compelling reason not to fix this, I'll propose either MPI_Win_all_lock_all. I have no use case for locking a subset of the window group collectively, so I do not intend to propose this. Thanks, Jeff -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0