So, perhaps enumerating the relevant GA constructs and their semantics would be informative here...
Keith
> -----Original Message-----
> From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma-
> bounces(a)lists.mpi-forum.org] On Behalf Of Jeff Hammond
> Sent: Wednesday, May 19, 2010 7:23 PM
> To: MPI 3.0 Remote Memory Access working group
> Cc: MPI 3.0 Remote Memory Access working group
> Subject: Re: [Mpi3-rma] RMA proposal 1 update
>
> Can I mix that call with other sync mechanisms?
>
> So I implement GA by calling fence inside of GA_Create to expose the
> window and use fence+barrier for GA_Sync, but can I mix in lock and
> unlock as well as the forthcoming p2p flush (as I can do in GA/ARMCI
> now)?
>
> The standard presents three synchronization schemes. It does not
> suggest one can intermix them at will.
>
> Jeff
>
> Sent from my iPhone
>
> On May 19, 2010, at 2:58 PM, "Underwood, Keith D"
> <keith.d.underwood(a)intel.com
> > wrote:
>
> > Jeff,
> >
> > Another question for you: If you are going to call
> > MPI_Win_all_flush_all, why not just use active target and call
> > MPI_Win_fence?
> >
> > Keith
> >
> >> -----Original Message-----
> >> From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma-
> >> bounces(a)lists.mpi-forum.org] On Behalf Of Jeff Hammond
> >> Sent: Sunday, May 16, 2010 7:27 PM
> >> To: MPI 3.0 Remote Memory Access working group
> >> Subject: Re: [Mpi3-rma] RMA proposal 1 update
> >>
> >> Tortsten,
> >>
> >> There seemed to be decent agreement on adding MPI_Win_all_flush_all
> >> (equivalent to MPI_Win_flush_all called from every rank in the
> >> communicator associated with the window) since this function can be
> >> implemented far more efficiently as a collective than the equivalent
> >> point-wise function calls.
> >>
> >> Is there a problem with adding this to your proposal?
> >>
> >> Jeff
> >>
> >> On Sun, May 16, 2010 at 12:48 AM, Torsten Hoefler
> <htor(a)illinois.edu>
> >> wrote:
> >>> Hello all,
> >>>
> >>> After the discussions at the last Forum I updated the group's first
> >>> proposal.
> >>>
> >>> The proposal (one-side-2.pdf) is attached to the wiki page
> >>> https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/RmaWikiPage
> >>>
> >>> The changes with regards to the last version are:
> >>>
> >>> 1) added MPI_NOOP to MPI_Get_accumulate and MPI_Accumulate_get
> >>>
> >>> 2) (re)added MPI_Win_flush and MPI_Win_flush_all to passive target
> >> mode
> >>>
> >>> Some remarks:
> >>>
> >>> 1) We didn't straw-vote on MPI_Accumulate_get, so this function
> >>> might
> >>> go. The removal would be very clean.
> >>>
> >>> 2) Should we allow MPI_NOOP in MPI_Accumulate (this does not make
> >> sense
> >>> and is incorrect in my current proposal)
> >>>
> >>> 3) Should we allow MPI_REPLACE in
> >> MPI_Get_accumulate/MPI_Accumulate_get?
> >>> (this would make sense and is allowed in the current proposal but
> >> we
> >>> didn't talk about it in the group)
> >>>
> >>>
> >>> All the Best,
> >>> Torsten
> >>>
> >>> --
> >>> bash$ :(){ :|:&};: --------------------- http://www.unixer.de/
> >>> -----
> >>> Torsten Hoefler | Research Associate
> >>> Blue Waters Directorate | University of Illinois
> >>> 1205 W Clark Street | Urbana, IL, 61801
> >>> NCSA Building | +01 (217) 244-7736
> >>> _______________________________________________
> >>> mpi3-rma mailing list
> >>> mpi3-rma(a)lists.mpi-forum.org
> >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma
> >>>
> >>
> >>
> >>
> >> --
> >> Jeff Hammond
> >> Argonne Leadership Computing Facility
> >> jhammond(a)mcs.anl.gov / (630) 252-5381
> >> http://www.linkedin.com/in/jeffhammond
> >>
> >> _______________________________________________
> >> mpi3-rma mailing list
> >> mpi3-rma(a)lists.mpi-forum.org
> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma
> >
> > _______________________________________________
> > mpi3-rma mailing list
> > mpi3-rma(a)lists.mpi-forum.org
> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma
> _______________________________________________
> mpi3-rma mailing list
> mpi3-rma(a)lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma
Can I mix that call with other sync mechanisms?
So I implement GA by calling fence inside of GA_Create to expose the
window and use fence+barrier for GA_Sync, but can I mix in lock and
unlock as well as the forthcoming p2p flush (as I can do in GA/ARMCI
now)?
The standard presents three synchronization schemes. It does not
suggest one can intermix them at will.
Jeff
Sent from my iPhone
On May 19, 2010, at 2:58 PM, "Underwood, Keith D" <keith.d.underwood(a)intel.com
> wrote:
> Jeff,
>
> Another question for you: If you are going to call
> MPI_Win_all_flush_all, why not just use active target and call
> MPI_Win_fence?
>
> Keith
>
>> -----Original Message-----
>> From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma-
>> bounces(a)lists.mpi-forum.org] On Behalf Of Jeff Hammond
>> Sent: Sunday, May 16, 2010 7:27 PM
>> To: MPI 3.0 Remote Memory Access working group
>> Subject: Re: [Mpi3-rma] RMA proposal 1 update
>>
>> Tortsten,
>>
>> There seemed to be decent agreement on adding MPI_Win_all_flush_all
>> (equivalent to MPI_Win_flush_all called from every rank in the
>> communicator associated with the window) since this function can be
>> implemented far more efficiently as a collective than the equivalent
>> point-wise function calls.
>>
>> Is there a problem with adding this to your proposal?
>>
>> Jeff
>>
>> On Sun, May 16, 2010 at 12:48 AM, Torsten Hoefler <htor(a)illinois.edu>
>> wrote:
>>> Hello all,
>>>
>>> After the discussions at the last Forum I updated the group's first
>>> proposal.
>>>
>>> The proposal (one-side-2.pdf) is attached to the wiki page
>>> https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/RmaWikiPage
>>>
>>> The changes with regards to the last version are:
>>>
>>> 1) added MPI_NOOP to MPI_Get_accumulate and MPI_Accumulate_get
>>>
>>> 2) (re)added MPI_Win_flush and MPI_Win_flush_all to passive target
>> mode
>>>
>>> Some remarks:
>>>
>>> 1) We didn't straw-vote on MPI_Accumulate_get, so this function
>>> might
>>> go. The removal would be very clean.
>>>
>>> 2) Should we allow MPI_NOOP in MPI_Accumulate (this does not make
>> sense
>>> and is incorrect in my current proposal)
>>>
>>> 3) Should we allow MPI_REPLACE in
>> MPI_Get_accumulate/MPI_Accumulate_get?
>>> (this would make sense and is allowed in the current proposal but
>> we
>>> didn't talk about it in the group)
>>>
>>>
>>> All the Best,
>>> Torsten
>>>
>>> --
>>> bash$ :(){ :|:&};: --------------------- http://www.unixer.de/
>>> -----
>>> Torsten Hoefler | Research Associate
>>> Blue Waters Directorate | University of Illinois
>>> 1205 W Clark Street | Urbana, IL, 61801
>>> NCSA Building | +01 (217) 244-7736
>>> _______________________________________________
>>> mpi3-rma mailing list
>>> mpi3-rma(a)lists.mpi-forum.org
>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma
>>>
>>
>>
>>
>> --
>> Jeff Hammond
>> Argonne Leadership Computing Facility
>> jhammond(a)mcs.anl.gov / (630) 252-5381
>> http://www.linkedin.com/in/jeffhammond
>>
>> _______________________________________________
>> mpi3-rma mailing list
>> mpi3-rma(a)lists.mpi-forum.org
>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma
>
> _______________________________________________
> mpi3-rma mailing list
> mpi3-rma(a)lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma
Jeff,
Another question for you: If you are going to call MPI_Win_all_flush_all, why not just use active target and call MPI_Win_fence?
Keith
> -----Original Message-----
> From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma-
> bounces(a)lists.mpi-forum.org] On Behalf Of Jeff Hammond
> Sent: Sunday, May 16, 2010 7:27 PM
> To: MPI 3.0 Remote Memory Access working group
> Subject: Re: [Mpi3-rma] RMA proposal 1 update
>
> Tortsten,
>
> There seemed to be decent agreement on adding MPI_Win_all_flush_all
> (equivalent to MPI_Win_flush_all called from every rank in the
> communicator associated with the window) since this function can be
> implemented far more efficiently as a collective than the equivalent
> point-wise function calls.
>
> Is there a problem with adding this to your proposal?
>
> Jeff
>
> On Sun, May 16, 2010 at 12:48 AM, Torsten Hoefler <htor(a)illinois.edu>
> wrote:
> > Hello all,
> >
> > After the discussions at the last Forum I updated the group's first
> > proposal.
> >
> > The proposal (one-side-2.pdf) is attached to the wiki page
> > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/RmaWikiPage
> >
> > The changes with regards to the last version are:
> >
> > 1) added MPI_NOOP to MPI_Get_accumulate and MPI_Accumulate_get
> >
> > 2) (re)added MPI_Win_flush and MPI_Win_flush_all to passive target
> mode
> >
> > Some remarks:
> >
> > 1) We didn't straw-vote on MPI_Accumulate_get, so this function might
> > � go. The removal would be very clean.
> >
> > 2) Should we allow MPI_NOOP in MPI_Accumulate (this does not make
> sense
> > � and is incorrect in my current proposal)
> >
> > 3) Should we allow MPI_REPLACE in
> MPI_Get_accumulate/MPI_Accumulate_get?
> > � (this would make sense and is allowed in the current proposal but
> we
> > � didn't talk about it in the group)
> >
> >
> > All the Best,
> > �Torsten
> >
> > --
> > �bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ -----
> > Torsten Hoefler � � � � | Research Associate
> > Blue Waters Directorate | University of Illinois
> > 1205 W Clark Street � � | Urbana, IL, 61801
> > NCSA Building � � � � � | +01 (217) 244-7736
> > _______________________________________________
> > mpi3-rma mailing list
> > mpi3-rma(a)lists.mpi-forum.org
> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma
> >
>
>
>
> --
> Jeff Hammond
> Argonne Leadership Computing Facility
> jhammond(a)mcs.anl.gov / (630) 252-5381
> http://www.linkedin.com/in/jeffhammond
>
> _______________________________________________
> mpi3-rma mailing list
> mpi3-rma(a)lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma
What are you talking about? This thread related to whether there should be an allflushall() call added in addition to a flushall(). I argued that it is not obvious that there is a real use case where allflushall() is substantially better than simply flushall()+barrier(). IBM guys said that the allflushall() could invoke an allreduce() of a relatively large vector to get better performance. I challenged that notion and asked for a back of the envelope calculation. You seemed to be replying to that thread... I'm not sure what comparison related to allflushall() you are referring to or how you would map your example below into an allflushall() call. If that is not the thread you meant to refer to, please clarify.
Keith
From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Vinod tipparaju
Sent: Wednesday, May 19, 2010 1:50 PM
To: MPI 3.0 Remote Memory Access working group
Subject: Re: [Mpi3-rma] RMA proposal 1 update
Are you referring to this email?
>you allreduce(MPI_SUM, 1 million element integer vector == 4MB)? Then you know how many things you should have received and when you have received all of those you can enter a barrier?
>> If each origin (in a fence epoch) keeps track of the count(s) of RMA
>> operations to each of its targets, then an allreduce of those arrays
>> will
>> tell each target how many operations were done to itself and can be
>> used to
>> determine completion.
please help undestand this better, where does allreduce (in MPI terms) came into the picture here? There is no big vector. I want to do get_1, add_1, get_2, add_2, get_3, add_3 when there is not support for collective reduce in the network. Can this be worse than complete_1, complete_2, complete_3
Vinod.
________________________________
From: keith.d.underwood(a)intel.com
To: mpi3-rma(a)lists.mpi-forum.org
Date: Wed, 19 May 2010 13:38:43 -0600
Subject: Re: [Mpi3-rma] RMA proposal 1 update
Yes, it can be worse. See later emails in the thread...
Keith
From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Vinod tipparaju
Sent: Wednesday, May 19, 2010 1:34 PM
To: MPI 3.0 Remote Memory Access working group
Subject: Re: [Mpi3-rma] RMA proposal 1 update
I am very much for supporting collective remote completion. Many optimizations are possible here.
>The question then turns to the "other networks". If you can't figure out remote completion, then the collective is going to be pretty heavy, right?
May be this will help make the point. When there is a network that does support a collective remote completion semantic (say, using its collective network reduce operation), would this functionality not help get better performance on that network? Would this functionality, for n collective completions, ever be worse than completion_1+completion_2+completion_3+...completion_n? If not, why not have it?
> From: keith.d.underwood(a)intel.com
> To: mpi3-rma(a)lists.mpi-forum.org
> Date: Sun, 16 May 2010 20:32:36 -0600
> Subject: Re: [Mpi3-rma] RMA proposal 1 update
>
> Before doing that, can someone sketch out the platform/API and the implementation that makes that more efficient? There is no gain for Portals (3 or 4). There is no gain for anything that supports Cray SHMEM reasonably well (shmem_quiet() is approximately the same semantics as MPI_flush_all). Hrm, you can probably say the same thing about anything that supports UPC well - a strict access is basically a MPI_flush_all(); MPI_Put(); MPI_flush_all();... Also, I thought somebody said that IB gave you a notification of remote completion...
>
> The question then turns to the "other networks". If you can't figure out remote completion, then the collective is going to be pretty heavy, right?
>
> Keith
>
> > -----Original Message-----
> > From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma-
> > bounces(a)lists.mpi-forum.org] On Behalf Of Jeff Hammond
> > Sent: Sunday, May 16, 2010 7:27 PM
> > To: MPI 3.0 Remote Memory Access working group
> > Subject: Re: [Mpi3-rma] RMA proposal 1 update
> >
> > Tortsten,
> >
> > There seemed to be decent agreement on adding MPI_Win_all_flush_all
> > (equivalent to MPI_Win_flush_all called from every rank in the
> > communicator associated with the window) since this function can be
> > implemented far more efficiently as a collective than the equivalent
> > point-wise function calls.
> >
> > Is there a problem with adding this to your proposal?
> >
> > Jeff
> >
> > On Sun, May 16, 2010 at 12:48 AM, Torsten Hoefler <htor(a)illinois.edu>
> > wrote:
> > > Hello all,
> > >
> > > After the discussions at the last Forum I updated the group's first
> > > proposal.
> > >
> > > The proposal (one-side-2.pdf) is attached to the wiki page
> > > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/RmaWikiPage
> > >
> > > The changes with regards to the last version are:
> > >
> > > 1) added MPI_NOOP to MPI_Get_accumulate and MPI_Accumulate_get
> > >
> > > 2) (re)added MPI_Win_flush and MPI_Win_flush_all to passive target
> > mode
> > >
> > > Some remarks:
> > >
> > > 1) We didn't straw-vote on MPI_Accumulate_get, so this function might
> > > go. The removal would be very clean.
> > >
> > > 2) Should we allow MPI_NOOP in MPI_Accumulate (this does not make
> > sense
> > > and is incorrect in my current proposal)
> > >
> > > 3) Should we allow MPI_REPLACE in
> > MPI_Get_accumulate/MPI_Accumulate_get?
> > > (this would make sense and is allowed in the current proposal but
> > we
> > > didn't talk about it in the group)
> > >
> > >
> > > All the Best,
> > > Torsten
> > >
> > > --
> > > bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ -----
> > > Torsten Hoefler | Research Associate
> > > Blue Waters Directorate | University of Illinois
> > > 1205 W Clark Street | Urbana, IL, 61801
> > > NCSA Building | +01 (217) 244-7736
> > > _______________________________________________
> > > mpi3-rma mailing list
> > > mpi3-rma(a)lists.mpi-forum.org
> > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma
> > >
> >
> >
> >
> > --
> > Jeff Hammond
> > Argonne Leadership Computing Facility
> > jhammond(a)mcs.anl.gov / (630) 252-5381
> > http://www.linkedin.com/in/jeffhammond
> >
> > _______________________________________________
> > mpi3-rma mailing list
> > mpi3-rma(a)lists.mpi-forum.org
> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma
>
> _______________________________________________
> mpi3-rma mailing list
> mpi3-rma(a)lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma
Are you referring to this email?
>you allreduce(MPI_SUM, 1 million element integer vector == 4MB)? Then you know how many things you should have received and when you have received all of those you can enter a barrier?
>> If each origin (in a fence epoch) keeps track of the count(s) of RMA>> operations to each of its targets, then an allreduce of those arrays>> will>> tell each target how many operations were done to itself and can be>> used to>> determine completion.
please help undestand this better, where does allreduce (in MPI terms) came into the picture here? There is no big vector. I want to do get_1, add_1, get_2, add_2, get_3, add_3 when there is not support for collective reduce in the network. Can this be worse than complete_1, complete_2, complete_3
Vinod.
From: keith.d.underwood(a)intel.com
To: mpi3-rma(a)lists.mpi-forum.org
Date: Wed, 19 May 2010 13:38:43 -0600
Subject: Re: [Mpi3-rma] RMA proposal 1 update
Yes, it can be worse. See later emails in the thread…
Keith
From:
mpi3-rma-bounces(a)lists.mpi-forum.org
[mailto:[email protected]] On Behalf Of Vinod
tipparaju
Sent: Wednesday, May 19, 2010 1:34 PM
To: MPI 3.0 Remote Memory Access working group
Subject: Re: [Mpi3-rma] RMA proposal 1 update
I
am very much for supporting collective remote completion. Many optimizations
are possible here.
>The question then turns to the
"other networks". If you can't figure out remote completion, then the
collective is going to be pretty heavy, right?
May
be this will help make the point. When there is a network that does support a
collective remote completion semantic (say, using its collective network reduce
operation), would this functionality not help get better performance on that
network? Would this functionality, for n collective completions, ever be
worse than completion_1+completion_2+completion_3+...completion_n? If not, why
not have it?
>
From: keith.d.underwood(a)intel.com
> To: mpi3-rma(a)lists.mpi-forum.org
> Date: Sun, 16 May 2010 20:32:36 -0600
> Subject: Re: [Mpi3-rma] RMA proposal 1 update
>
> Before doing that, can someone sketch out the platform/API and the
implementation that makes that more efficient? There is no gain for Portals (3
or 4). There is no gain for anything that supports Cray SHMEM reasonably well
(shmem_quiet() is approximately the same semantics as MPI_flush_all). Hrm, you
can probably say the same thing about anything that supports UPC well - a
strict access is basically a MPI_flush_all(); MPI_Put(); MPI_flush_all();...
Also, I thought somebody said that IB gave you a notification of remote
completion...
>
> The question then turns to the "other networks". If you can't
figure out remote completion, then the collective is going to be pretty heavy,
right?
>
> Keith
>
> > -----Original Message-----
> > From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma-
> > bounces(a)lists.mpi-forum.org] On Behalf Of Jeff Hammond
> > Sent: Sunday, May 16, 2010 7:27 PM
> > To: MPI 3.0 Remote Memory Access working group
> > Subject: Re: [Mpi3-rma] RMA proposal 1 update
> >
> > Tortsten,
> >
> > There seemed to be decent agreement on adding MPI_Win_all_flush_all
> > (equivalent to MPI_Win_flush_all called from every rank in the
> > communicator associated with the window) since this function can be
> > implemented far more efficiently as a collective than the equivalent
> > point-wise function calls.
> >
> > Is there a problem with adding this to your proposal?
> >
> > Jeff
> >
> > On Sun, May 16, 2010 at 12:48 AM, Torsten Hoefler
<htor(a)illinois.edu>
> > wrote:
> > > Hello all,
> > >
> > > After the discussions at the last Forum I updated the group's
first
> > > proposal.
> > >
> > > The proposal (one-side-2.pdf) is attached to the wiki page
> > > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/RmaWikiPage
> > >
> > > The changes with regards to the last version are:
> > >
> > > 1) added MPI_NOOP to MPI_Get_accumulate and MPI_Accumulate_get
> > >
> > > 2) (re)added MPI_Win_flush and MPI_Win_flush_all to passive
target
> > mode
> > >
> > > Some remarks:
> > >
> > > 1) We didn't straw-vote on MPI_Accumulate_get, so this function
might
> > > go. The removal would be very clean.
> > >
> > > 2) Should we allow MPI_NOOP in MPI_Accumulate (this does not
make
> > sense
> > > and is incorrect in my current proposal)
> > >
> > > 3) Should we allow MPI_REPLACE in
> > MPI_Get_accumulate/MPI_Accumulate_get?
> > > (this would make sense and is allowed in the current
proposal but
> > we
> > > didn't talk about it in the group)
> > >
> > >
> > > All the Best,
> > > Torsten
> > >
> > > --
> > > bash$ :(){ :|:&};: ---------------------
http://www.unixer.de/ -----
> > > Torsten Hoefler | Research Associate
> > > Blue Waters Directorate | University of Illinois
> > > 1205 W Clark Street | Urbana, IL, 61801
> > > NCSA Building | +01 (217)
244-7736
> > > _______________________________________________
> > > mpi3-rma mailing list
> > > mpi3-rma(a)lists.mpi-forum.org
> > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma
> > >
> >
> >
> >
> > --
> > Jeff Hammond
> > Argonne Leadership Computing Facility
> > jhammond(a)mcs.anl.gov / (630) 252-5381
> > http://www.linkedin.com/in/jeffhammond
> >
> > _______________________________________________
> > mpi3-rma mailing list
> > mpi3-rma(a)lists.mpi-forum.org
> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma
>
> _______________________________________________
> mpi3-rma mailing list
> mpi3-rma(a)lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma
Yes, it can be worse. See later emails in the thread...
Keith
From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Vinod tipparaju
Sent: Wednesday, May 19, 2010 1:34 PM
To: MPI 3.0 Remote Memory Access working group
Subject: Re: [Mpi3-rma] RMA proposal 1 update
I am very much for supporting collective remote completion. Many optimizations are possible here.
>The question then turns to the "other networks". If you can't figure out remote completion, then the collective is going to be pretty heavy, right?
May be this will help make the point. When there is a network that does support a collective remote completion semantic (say, using its collective network reduce operation), would this functionality not help get better performance on that network? Would this functionality, for n collective completions, ever be worse than completion_1+completion_2+completion_3+...completion_n? If not, why not have it?
> From: keith.d.underwood(a)intel.com
> To: mpi3-rma(a)lists.mpi-forum.org
> Date: Sun, 16 May 2010 20:32:36 -0600
> Subject: Re: [Mpi3-rma] RMA proposal 1 update
>
> Before doing that, can someone sketch out the platform/API and the implementation that makes that more efficient? There is no gain for Portals (3 or 4). There is no gain for anything that supports Cray SHMEM reasonably well (shmem_quiet() is approximately the same semantics as MPI_flush_all). Hrm, you can probably say the same thing about anything that supports UPC well - a strict access is basically a MPI_flush_all(); MPI_Put(); MPI_flush_all();... Also, I thought somebody said that IB gave you a notification of remote completion...
>
> The question then turns to the "other networks". If you can't figure out remote completion, then the collective is going to be pretty heavy, right?
>
> Keith
>
> > -----Original Message-----
> > From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma-
> > bounces(a)lists.mpi-forum.org] On Behalf Of Jeff Hammond
> > Sent: Sunday, May 16, 2010 7:27 PM
> > To: MPI 3.0 Remote Memory Access working group
> > Subject: Re: [Mpi3-rma] RMA proposal 1 update
> >
> > Tortsten,
> >
> > There seemed to be decent agreement on adding MPI_Win_all_flush_all
> > (equivalent to MPI_Win_flush_all called from every rank in the
> > communicator associated with the window) since this function can be
> > implemented far more efficiently as a collective than the equivalent
> > point-wise function calls.
> >
> > Is there a problem with adding this to your proposal?
> >
> > Jeff
> >
> > On Sun, May 16, 2010 at 12:48 AM, Torsten Hoefler <htor(a)illinois.edu>
> > wrote:
> > > Hello all,
> > >
> > > After the discussions at the last Forum I updated the group's first
> > > proposal.
> > >
> > > The proposal (one-side-2.pdf) is attached to the wiki page
> > > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/RmaWikiPage
> > >
> > > The changes with regards to the last version are:
> > >
> > > 1) added MPI_NOOP to MPI_Get_accumulate and MPI_Accumulate_get
> > >
> > > 2) (re)added MPI_Win_flush and MPI_Win_flush_all to passive target
> > mode
> > >
> > > Some remarks:
> > >
> > > 1) We didn't straw-vote on MPI_Accumulate_get, so this function might
> > > go. The removal would be very clean.
> > >
> > > 2) Should we allow MPI_NOOP in MPI_Accumulate (this does not make
> > sense
> > > and is incorrect in my current proposal)
> > >
> > > 3) Should we allow MPI_REPLACE in
> > MPI_Get_accumulate/MPI_Accumulate_get?
> > > (this would make sense and is allowed in the current proposal but
> > we
> > > didn't talk about it in the group)
> > >
> > >
> > > All the Best,
> > > Torsten
> > >
> > > --
> > > bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ -----
> > > Torsten Hoefler | Research Associate
> > > Blue Waters Directorate | University of Illinois
> > > 1205 W Clark Street | Urbana, IL, 61801
> > > NCSA Building | +01 (217) 244-7736
> > > _______________________________________________
> > > mpi3-rma mailing list
> > > mpi3-rma(a)lists.mpi-forum.org
> > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma
> > >
> >
> >
> >
> > --
> > Jeff Hammond
> > Argonne Leadership Computing Facility
> > jhammond(a)mcs.anl.gov / (630) 252-5381
> > http://www.linkedin.com/in/jeffhammond
> >
> > _______________________________________________
> > mpi3-rma mailing list
> > mpi3-rma(a)lists.mpi-forum.org
> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma
>
> _______________________________________________
> mpi3-rma mailing list
> mpi3-rma(a)lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma
I am very much for supporting collective remote completion. Many optimizations are possible here.
>The question then turns to the "other networks". If you can't figure out remote completion, then the collective is going to be pretty heavy, right?
May be this will help make the point. When there is a network that does support a collective remote completion semantic (say, using its collective network reduce operation), would this functionality not help get better performance on that network? Would this functionality, for n collective completions, ever be worse than completion_1+completion_2+completion_3+...completion_n? If not, why not have it?
> From: keith.d.underwood(a)intel.com
> To: mpi3-rma(a)lists.mpi-forum.org
> Date: Sun, 16 May 2010 20:32:36 -0600
> Subject: Re: [Mpi3-rma] RMA proposal 1 update
>
> Before doing that, can someone sketch out the platform/API and the implementation that makes that more efficient? There is no gain for Portals (3 or 4). There is no gain for anything that supports Cray SHMEM reasonably well (shmem_quiet() is approximately the same semantics as MPI_flush_all). Hrm, you can probably say the same thing about anything that supports UPC well - a strict access is basically a MPI_flush_all(); MPI_Put(); MPI_flush_all();... Also, I thought somebody said that IB gave you a notification of remote completion...
>
> The question then turns to the "other networks". If you can't figure out remote completion, then the collective is going to be pretty heavy, right?
>
> Keith
>
> > -----Original Message-----
> > From: mpi3-rma-bounces(a)lists.mpi-forum.org [mailto:mpi3-rma-
> > bounces(a)lists.mpi-forum.org] On Behalf Of Jeff Hammond
> > Sent: Sunday, May 16, 2010 7:27 PM
> > To: MPI 3.0 Remote Memory Access working group
> > Subject: Re: [Mpi3-rma] RMA proposal 1 update
> >
> > Tortsten,
> >
> > There seemed to be decent agreement on adding MPI_Win_all_flush_all
> > (equivalent to MPI_Win_flush_all called from every rank in the
> > communicator associated with the window) since this function can be
> > implemented far more efficiently as a collective than the equivalent
> > point-wise function calls.
> >
> > Is there a problem with adding this to your proposal?
> >
> > Jeff
> >
> > On Sun, May 16, 2010 at 12:48 AM, Torsten Hoefler <htor(a)illinois.edu>
> > wrote:
> > > Hello all,
> > >
> > > After the discussions at the last Forum I updated the group's first
> > > proposal.
> > >
> > > The proposal (one-side-2.pdf) is attached to the wiki page
> > > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/RmaWikiPage
> > >
> > > The changes with regards to the last version are:
> > >
> > > 1) added MPI_NOOP to MPI_Get_accumulate and MPI_Accumulate_get
> > >
> > > 2) (re)added MPI_Win_flush and MPI_Win_flush_all to passive target
> > mode
> > >
> > > Some remarks:
> > >
> > > 1) We didn't straw-vote on MPI_Accumulate_get, so this function might
> > > go. The removal would be very clean.
> > >
> > > 2) Should we allow MPI_NOOP in MPI_Accumulate (this does not make
> > sense
> > > and is incorrect in my current proposal)
> > >
> > > 3) Should we allow MPI_REPLACE in
> > MPI_Get_accumulate/MPI_Accumulate_get?
> > > (this would make sense and is allowed in the current proposal but
> > we
> > > didn't talk about it in the group)
> > >
> > >
> > > All the Best,
> > > Torsten
> > >
> > > --
> > > bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ -----
> > > Torsten Hoefler | Research Associate
> > > Blue Waters Directorate | University of Illinois
> > > 1205 W Clark Street | Urbana, IL, 61801
> > > NCSA Building | +01 (217) 244-7736
> > > _______________________________________________
> > > mpi3-rma mailing list
> > > mpi3-rma(a)lists.mpi-forum.org
> > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma
> > >
> >
> >
> >
> > --
> > Jeff Hammond
> > Argonne Leadership Computing Facility
> > jhammond(a)mcs.anl.gov / (630) 252-5381
> > http://www.linkedin.com/in/jeffhammond
> >
> > _______________________________________________
> > mpi3-rma mailing list
> > mpi3-rma(a)lists.mpi-forum.org
> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma
>
> _______________________________________________
> mpi3-rma mailing list
> mpi3-rma(a)lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma
The only way it can be done today is by using a remote software agent,
not through hardware -- which has been my point all along. That is, from
though IB claims that it does remote completion for every request in
hardware, as far as the MPI semantics go, remote completion and local
completion cannot have the same cost.
-- Pavan
On 05/18/2010 02:44 PM, Underwood, Keith D wrote:
> The same way it does it for unlock today...
>
>> Ok, in that case, how will a network that only gives remote completion
>> till the adapter ensure ordering between the "foo" and "bar" variables
>> if they go over different adapters?
>>
>> Btw, there are a number of production systems that use multi-rail IB.
>>
>> -- Pavan
>>
>> On 05/18/2010 02:07 PM, Underwood, Keith D wrote:
>>> Yes, you should get 100. MPI_Flush() does remote completion, just
>> like MPI_Win_unlock(). How you do that on some hacked together dual
>> rail solution is up to the implementation ;-)
>>> Keith
>>>
>>>> -----Original Message-----
>>>> From: Pavan Balaji [mailto:[email protected]]
>>>> Sent: Tuesday, May 18, 2010 1:05 PM
>>>> To: Underwood, Keith D
>>>> Cc: MPI 3.0 Remote Memory Access working group
>>>> Subject: Re: [Mpi3-rma] RMA proposal 1 update
>>>>
>>>>
>>>> On 05/18/2010 01:52 PM, Underwood, Keith D wrote:
>>>>> Now you are just trying to be difficult... First, your scenario is
>>>> not legal. You have to call a local MPI_Lock()/MPI_Unlock() before
>>>> that data is visible in the private window to allow loads and
>> stores.
>>>> Even accessing that item that was Put over NIC1 is undefined until
>> the
>>>> source has done a completion operation.
>>>>
>>>> Sorry, I don't mean to. Relying on network ordering till memory just
>>>> seems hacky. So, I'm trying to see if there are cases where the
>> network
>>>> doesn't have full control on when things are written to memory.
>>>>
>>>>> Even then, I think you are discussing an ordering problem that
>> exists
>>>> in the base standard: completing an MPI_Unlock() implies remote
>>>> completing. Real remote completion. Until MPI_Unlock() completes,
>>>> there is no guarantee of ordering between anything. MPI_flush()
>> does
>>>> not add to this issue.
>>>>
>>>> Hmm.. Maybe I don't understand MPI_Flush very well then. Here's the
>>>> example case I was thinking of:
>>>>
>>>> MPI_Win_lock(target = 1, SHARED);
>>>> if (rank == 1) {
>>>> MPI_Put(win, target = 1, foo = 100, ...);
>>>> MPI_Flush(win, target = 1, ...);
>>>> MPI_Get_accumulate(win, target = 1, &bar, ...);
>>>> }
>>>> else if (rank == 0) {
>>>> do {
>>>> MPI_Get_accumulate(win, target = 1, &bar, ...);
>>>> } while (bar != 1); /* Get the mutex */
>>>> MPI_Get(win, target = 1, &foo, ...);
>>>> }
>>>> MPI_Win_unlock(target = 1);
>>>>
>>>> So, the question is, is process 1 guaranteed to get foo = 100 in
>> this
>>>> case? Note that there are no direct load/stores here, so everything
>> can
>>>> happen in shared lock mode.
>>>>
>>>> -- Pavan
>>>>
>>>> --
>>>> Pavan Balaji
>>>> http://www.mcs.anl.gov/~balaji
>> --
>> Pavan Balaji
>> http://www.mcs.anl.gov/~balaji
--
Pavan Balaji
http://www.mcs.anl.gov/~balaji
The same way it does it for unlock today...
>
> Ok, in that case, how will a network that only gives remote completion
> till the adapter ensure ordering between the "foo" and "bar" variables
> if they go over different adapters?
>
> Btw, there are a number of production systems that use multi-rail IB.
>
> -- Pavan
>
> On 05/18/2010 02:07 PM, Underwood, Keith D wrote:
> > Yes, you should get 100. MPI_Flush() does remote completion, just
> like MPI_Win_unlock(). How you do that on some hacked together dual
> rail solution is up to the implementation ;-)
> >
> > Keith
> >
> >> -----Original Message-----
> >> From: Pavan Balaji [mailto:[email protected]]
> >> Sent: Tuesday, May 18, 2010 1:05 PM
> >> To: Underwood, Keith D
> >> Cc: MPI 3.0 Remote Memory Access working group
> >> Subject: Re: [Mpi3-rma] RMA proposal 1 update
> >>
> >>
> >> On 05/18/2010 01:52 PM, Underwood, Keith D wrote:
> >>> Now you are just trying to be difficult... First, your scenario is
> >> not legal. You have to call a local MPI_Lock()/MPI_Unlock() before
> >> that data is visible in the private window to allow loads and
> stores.
> >> Even accessing that item that was Put over NIC1 is undefined until
> the
> >> source has done a completion operation.
> >>
> >> Sorry, I don't mean to. Relying on network ordering till memory just
> >> seems hacky. So, I'm trying to see if there are cases where the
> network
> >> doesn't have full control on when things are written to memory.
> >>
> >>> Even then, I think you are discussing an ordering problem that
> exists
> >> in the base standard: completing an MPI_Unlock() implies remote
> >> completing. Real remote completion. Until MPI_Unlock() completes,
> >> there is no guarantee of ordering between anything. MPI_flush()
> does
> >> not add to this issue.
> >>
> >> Hmm.. Maybe I don't understand MPI_Flush very well then. Here's the
> >> example case I was thinking of:
> >>
> >> MPI_Win_lock(target = 1, SHARED);
> >> if (rank == 1) {
> >> MPI_Put(win, target = 1, foo = 100, ...);
> >> MPI_Flush(win, target = 1, ...);
> >> MPI_Get_accumulate(win, target = 1, &bar, ...);
> >> }
> >> else if (rank == 0) {
> >> do {
> >> MPI_Get_accumulate(win, target = 1, &bar, ...);
> >> } while (bar != 1); /* Get the mutex */
> >> MPI_Get(win, target = 1, &foo, ...);
> >> }
> >> MPI_Win_unlock(target = 1);
> >>
> >> So, the question is, is process 1 guaranteed to get foo = 100 in
> this
> >> case? Note that there are no direct load/stores here, so everything
> can
> >> happen in shared lock mode.
> >>
> >> -- Pavan
> >>
> >> --
> >> Pavan Balaji
> >> http://www.mcs.anl.gov/~balaji
>
> --
> Pavan Balaji
> http://www.mcs.anl.gov/~balaji
Ok, in that case, how will a network that only gives remote completion
till the adapter ensure ordering between the "foo" and "bar" variables
if they go over different adapters?
Btw, there are a number of production systems that use multi-rail IB.
-- Pavan
On 05/18/2010 02:07 PM, Underwood, Keith D wrote:
> Yes, you should get 100. MPI_Flush() does remote completion, just like MPI_Win_unlock(). How you do that on some hacked together dual rail solution is up to the implementation ;-)
>
> Keith
>
>> -----Original Message-----
>> From: Pavan Balaji [mailto:[email protected]]
>> Sent: Tuesday, May 18, 2010 1:05 PM
>> To: Underwood, Keith D
>> Cc: MPI 3.0 Remote Memory Access working group
>> Subject: Re: [Mpi3-rma] RMA proposal 1 update
>>
>>
>> On 05/18/2010 01:52 PM, Underwood, Keith D wrote:
>>> Now you are just trying to be difficult... First, your scenario is
>> not legal. You have to call a local MPI_Lock()/MPI_Unlock() before
>> that data is visible in the private window to allow loads and stores.
>> Even accessing that item that was Put over NIC1 is undefined until the
>> source has done a completion operation.
>>
>> Sorry, I don't mean to. Relying on network ordering till memory just
>> seems hacky. So, I'm trying to see if there are cases where the network
>> doesn't have full control on when things are written to memory.
>>
>>> Even then, I think you are discussing an ordering problem that exists
>> in the base standard: completing an MPI_Unlock() implies remote
>> completing. Real remote completion. Until MPI_Unlock() completes,
>> there is no guarantee of ordering between anything. MPI_flush() does
>> not add to this issue.
>>
>> Hmm.. Maybe I don't understand MPI_Flush very well then. Here's the
>> example case I was thinking of:
>>
>> MPI_Win_lock(target = 1, SHARED);
>> if (rank == 1) {
>> MPI_Put(win, target = 1, foo = 100, ...);
>> MPI_Flush(win, target = 1, ...);
>> MPI_Get_accumulate(win, target = 1, &bar, ...);
>> }
>> else if (rank == 0) {
>> do {
>> MPI_Get_accumulate(win, target = 1, &bar, ...);
>> } while (bar != 1); /* Get the mutex */
>> MPI_Get(win, target = 1, &foo, ...);
>> }
>> MPI_Win_unlock(target = 1);
>>
>> So, the question is, is process 1 guaranteed to get foo = 100 in this
>> case? Note that there are no direct load/stores here, so everything can
>> happen in shared lock mode.
>>
>> -- Pavan
>>
>> --
>> Pavan Balaji
>> http://www.mcs.anl.gov/~balaji
--
Pavan Balaji
http://www.mcs.anl.gov/~balaji
