March 2014 - mpiwg-rma - mailman3.cels.anl.gov

Re: [mpiwg-rma] same_op_no_op
by Balaji, Pavan 13 Mar '14

13 Mar '14

I’m not sure what “it is indirect to the user since the user needs to infer by themselves” means. As Dave suggested, the user is making an implicit promise that same_op_no_op is used, by default. With this promise set (either explicitly or implicitly), it is an errorneous program to have accumulate with different ops on the same location. In MPI-2, it was not erroneous to do so, though the atomicity was not guaranteed so the data might be corrupted. — Pavan On Mar 13, 2014, at 1:07 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > Indeed Xin is capturing things well. > > My point is that the user promised nothing when not setting this info key and thus it cannot be erroneous to not abide by it. > > I've posted my question about whether or not info text is normative on the forum list... > > Jeff > > Sent from my iPhone > >> On Mar 13, 2014, at 1:00 PM, "Zhao, Xin" <xinzhao3(a)illinois.edu> wrote: >> >> For my understanding, the difference between Pavan and Jeff is that, Pavan thinks the default behavior of an info value is actually a standard behavior ("same_op_no_op"), and user should follow that, whereas Jeff thinks it is indirect to the user since user needs to infer by themselves. It should be clearly defined in MPI-3 standard. Sorry if I understand wrongly. >> >> If I understand correctly, I agree with Jeff. >> >> Xin >> ________________________________________ >> From: mpiwg-rma [mpiwg-rma-bounces(a)lists.mpi-forum.org] on behalf of Balaji, Pavan [balaji(a)anl.gov] >> Sent: Thursday, March 13, 2014 12:52 PM >> To: MPI WG Remote Memory Access working group >> Subject: Re: [mpiwg-rma] same_op_no_op >> >>> On Mar 13, 2014, at 12:37 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>> How does MPI-3 indicate even indirectly that they are not allowed? I >>> do not believe the standard meets the minimum burden of telling the >>> user this and therefore it cannot be inferred. >> >> I’m still referring to the same sentence we have been discussing: >> >> "If set to same_op_no_op, then the >> implementation will assume that all concurrent accumulate calls to the >> same target address will use the same operation or MPI_NO_OP.” >> >> This means that the user promised that (s)he will only issue the same op or no_op. If the user breaks this promise, that’s an erroneous program and the MPI implementation is allowed to set the machine room on fire. >> >> This is the same model we have been using for other info arguments, e.g., >> >> "no_locks — if set to true, then the implementation may assume that passive target syn- chronization (i.e., MPI_WIN_LOCK, MPI_LOCK_ALL) will not be used on the given window." >> >> If the user sets no_locks and uses MPI_WIN_LOCK, it’s an error. >> >> — Pavan >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] same_op_no_op
by Jeff Hammond 13 Mar '14

13 Mar '14

Is "" a valid info key-value that unsets the default and allows the general case? Jeff Sent from my iPhone > On Mar 13, 2014, at 1:17 PM, "Dave Goodell (dgoodell)" <dgoodell(a)cisco.com> wrote: > > On Mar 13, 2014, at 1:07 PM, Jeff Hammond <jeff.science(a)gmail.com> > wrote: > >> Indeed Xin is capturing things well. >> >> My point is that the user promised nothing when not setting this info key and thus it cannot be erroneous to not abide by it. >> >> I've posted my question about whether or not info text is normative on the forum list... > > The user implicitly promised "same_op_no_op", which is why the "default" text is present. If you want different behavior in your app, just set "accumulate_ops" to "" or "ignoreme". What's your objection to that simple step? > > The intent of the standard (and the WG) seems pretty clear here. You can argue about whether things are truly normative or not, but that's just likely to result in strengthened text in 4.0 that conveys the same intent... > > -Dave > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] same_op_no_op
by Dave Goodell (dgoodell) 13 Mar '14

13 Mar '14

On Mar 13, 2014, at 1:07 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > Indeed Xin is capturing things well. > > My point is that the user promised nothing when not setting this info key and thus it cannot be erroneous to not abide by it. > > I've posted my question about whether or not info text is normative on the forum list... The user implicitly promised "same_op_no_op", which is why the "default" text is present. If you want different behavior in your app, just set "accumulate_ops" to "" or "ignoreme". What's your objection to that simple step? The intent of the standard (and the WG) seems pretty clear here. You can argue about whether things are truly normative or not, but that's just likely to result in strengthened text in 4.0 that conveys the same intent... -Dave

1 0

Re: [mpiwg-rma] same_op_no_op
by Jeff Hammond 13 Mar '14

13 Mar '14

Indeed Xin is capturing things well. My point is that the user promised nothing when not setting this info key and thus it cannot be erroneous to not abide by it. I've posted my question about whether or not info text is normative on the forum list... Jeff Sent from my iPhone > On Mar 13, 2014, at 1:00 PM, "Zhao, Xin" <xinzhao3(a)illinois.edu> wrote: > > For my understanding, the difference between Pavan and Jeff is that, Pavan thinks the default behavior of an info value is actually a standard behavior ("same_op_no_op"), and user should follow that, whereas Jeff thinks it is indirect to the user since user needs to infer by themselves. It should be clearly defined in MPI-3 standard. Sorry if I understand wrongly. > > If I understand correctly, I agree with Jeff. > > Xin > ________________________________________ > From: mpiwg-rma [mpiwg-rma-bounces(a)lists.mpi-forum.org] on behalf of Balaji, Pavan [balaji(a)anl.gov] > Sent: Thursday, March 13, 2014 12:52 PM > To: MPI WG Remote Memory Access working group > Subject: Re: [mpiwg-rma] same_op_no_op > >> On Mar 13, 2014, at 12:37 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >> How does MPI-3 indicate even indirectly that they are not allowed? I >> do not believe the standard meets the minimum burden of telling the >> user this and therefore it cannot be inferred. > > I’m still referring to the same sentence we have been discussing: > > "If set to same_op_no_op, then the > implementation will assume that all concurrent accumulate calls to the > same target address will use the same operation or MPI_NO_OP.” > > This means that the user promised that (s)he will only issue the same op or no_op. If the user breaks this promise, that’s an erroneous program and the MPI implementation is allowed to set the machine room on fire. > > This is the same model we have been using for other info arguments, e.g., > > "no_locks — if set to true, then the implementation may assume that passive target syn- chronization (i.e., MPI_WIN_LOCK, MPI_LOCK_ALL) will not be used on the given window." > > If the user sets no_locks and uses MPI_WIN_LOCK, it’s an error. > > — Pavan > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] same_op_no_op
by Zhao, Xin 13 Mar '14

13 Mar '14

For my understanding, the difference between Pavan and Jeff is that, Pavan thinks the default behavior of an info value is actually a standard behavior ("same_op_no_op"), and user should follow that, whereas Jeff thinks it is indirect to the user since user needs to infer by themselves. It should be clearly defined in MPI-3 standard. Sorry if I understand wrongly. If I understand correctly, I agree with Jeff. Xin ________________________________________ From: mpiwg-rma [mpiwg-rma-bounces(a)lists.mpi-forum.org] on behalf of Balaji, Pavan [balaji(a)anl.gov] Sent: Thursday, March 13, 2014 12:52 PM To: MPI WG Remote Memory Access working group Subject: Re: [mpiwg-rma] same_op_no_op On Mar 13, 2014, at 12:37 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > How does MPI-3 indicate even indirectly that they are not allowed? I > do not believe the standard meets the minimum burden of telling the > user this and therefore it cannot be inferred. I’m still referring to the same sentence we have been discussing: "If set to same_op_no_op, then the implementation will assume that all concurrent accumulate calls to the same target address will use the same operation or MPI_NO_OP.” This means that the user promised that (s)he will only issue the same op or no_op. If the user breaks this promise, that’s an erroneous program and the MPI implementation is allowed to set the machine room on fire. This is the same model we have been using for other info arguments, e.g., "no_locks — if set to true, then the implementation may assume that passive target syn- chronization (i.e., MPI_WIN_LOCK, MPI_LOCK_ALL) will not be used on the given window." If the user sets no_locks and uses MPI_WIN_LOCK, it’s an error. — Pavan _______________________________________________ mpiwg-rma mailing list mpiwg-rma(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] same_op_no_op
by Balaji, Pavan 13 Mar '14

13 Mar '14

On Mar 13, 2014, at 12:37 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > How does MPI-3 indicate even indirectly that they are not allowed? I > do not believe the standard meets the minimum burden of telling the > user this and therefore it cannot be inferred. I’m still referring to the same sentence we have been discussing: "If set to same_op_no_op, then the implementation will assume that all concurrent accumulate calls to the same target address will use the same operation or MPI_NO_OP.” This means that the user promised that (s)he will only issue the same op or no_op. If the user breaks this promise, that’s an erroneous program and the MPI implementation is allowed to set the machine room on fire. This is the same model we have been using for other info arguments, e.g., "no_locks — if set to true, then the implementation may assume that passive target syn- chronization (i.e., MPI_WIN_LOCK, MPI_LOCK_ALL) will not be used on the given window." If the user sets no_locks and uses MPI_WIN_LOCK, it’s an error. — Pavan

1 0

Re: [mpiwg-rma] same_op_no_op
by Jeff Hammond 13 Mar '14

13 Mar '14

How does MPI-3 indicate even indirectly that they are not allowed? I do not believe the standard meets the minimum burden of telling the user this and therefore it cannot be inferred. Jeff On Thu, Mar 13, 2014 at 12:21 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > Well, I think they are not allowed. If the implementation can assume that they are the same, if a single process does two accumulate operations, the implementation is allowed to add an assert saying first_op == second_op. That’ll blow up the program if they are not the same. > > — Pavan > > On Mar 13, 2014, at 12:16 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > >> I don't think MPI-3 says they are not allowed, just that the >> implementation "will assume" that >1 operations (except for NO_OP) are >> never used concurrently. Since the effect of the implementation's >> assumptions cannot be known, I would like to assume that concurrent >> different operations are not necessarily atomic i.e. implementation >> defined, and most certainly not erroneous. >> >> As to screwing this up, yes, I agree on multiple levels :-) >> >> Jeff >> >> On Thu, Mar 13, 2014 at 12:02 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: >>> >>> MPI-2.2 says that accumulate with different ops are not atomic. >>> >>> MPI-3 says that accumulate with different ops are not allowed (since same_op_no_op is default). >>> >>> I think we screwed that up? >>> >>> — Pavan >>> >>> On Mar 13, 2014, at 11:48 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>> >>>> It is extremely difficult to see that this is what the MPI-3 standard says. >>>> >>>> First we have this: >>>> >>>> "The outcome of concurrent accumulate operations to the same location >>>> with the same predefined datatype is as if the accumulates were done >>>> at that location in some serial order. Additional restrictions on the >>>> operation apply; see the info key accumulate_ops in Section 11.2.1. >>>> Concurrent accumulate operations with different origin and target >>>> pairs are not ordered. Thus, there is no guarantee that the entire >>>> call to an accumulate operation is executed atomically. The effect of >>>> this lack of atomicity is limited: The previous correctness conditions >>>> imply that a location updated by a call to an accumulate operation >>>> cannot be accessed by a load or an RMA call other than accumulate >>>> until the accumulate operation has completed (at the target). >>>> Different interleavings can lead to different results only to the >>>> extent that computer arithmetics are not truly associative or >>>> commutative. The outcome of accumulate operations with overlapping >>>> types of different sizes or target displacements is undefined." >>>> [11.7.1 Atomicity] >>>> >>>> Then we have this: >>>> >>>> "accumulate_ops — if set to same_op, the implementation will assume >>>> that all concurrent accumulate calls to the same target address will >>>> use the same operation. If set to same_op_no_op, then the >>>> implementation will assume that all concurrent accumulate calls to the >>>> same target address will use the same operation or MPI_NO_OP. This can >>>> eliminate the need to protect access for certain operation types where >>>> the hardware can guarantee atomicity. The default is same_op_no_op." >>>> [11.2.1 Window Creation] >>>> >>>> I was not aware that the definition of info keys was normative, given >>>> that implementations are free to ignore them. Even if info key text >>>> is normative, one has to infer from the fact that same_op_no_op is the >>>> default info behavior - and thus RMA semantic - that accumulate >>>> atomicity is restricted to the case where one uses the same op or noop >>>> but not replace. >>>> >>>> The MPI-2.2 spec is unambiguous because it explicitly requires the >>>> same operation in 11.7.1 Atomicity. This text was removed in MPI-3.0 >>>> in favor of the info key text. >>>> >>>> Best, >>>> >>>> Jeff >>>> >>>> On Tue, Mar 11, 2014 at 12:04 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: >>>>> >>>>> MPI-2 defines atomicity only for the same operation, not any operation for MPI_ACCUMULATE. >>>>> >>>>> — Pavan >>>>> >>>>> On Mar 10, 2014, at 11:22 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>>>> >>>>>> So MPI-2 denied compatibility between replace and not-replace? >>>>>> >>>>>> Jeff >>>>>> >>>>>> Sent from my iPhone >>>>>> >>>>>>> On Mar 11, 2014, at 12:06 AM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: >>>>>>> >>>>>>> >>>>>>> It doesn’t break backward compatibility. The info argument is still useful when you don’t want to use replace. I don’t see anything wrong with it. >>>>>>> >>>>>>>> On Mar 10, 2014, at 11:01 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>>>>>>> >>>>>>>> Does this or does this not break BW compatibility w.r.t. MPI-2.2 and >>>>>>>> did we do it intentionally? Unless we did so intentionally and >>>>>>>> explicitly, I will argue that the WG screwed up and the info key+val >>>>>>>> is invalid. >>>>>>>> >>>>>>>> Jeff >>>>>>>> >>>>>>>>> On Mon, Mar 10, 2014 at 11:03 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: >>>>>>>>> >>>>>>>>> If a hardware can implement MPI_SUM, it should be able to implement MPI_SUM with 0 as well. >>>>>>>>> >>>>>>>>> But that’s not a generic solution. >>>>>>>>> >>>>>>>>> Jeff: at some point you were planning to bring in a ticket which does more combinations of operations than just same_op and no_op. Maybe it’s worthwhile bringing that up again? >>>>>>>>> >>>>>>>>> — Pavan >>>>>>>>> >>>>>>>>>> On Mar 10, 2014, at 9:26 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: >>>>>>>>>> >>>>>>>>>> Maybe there's a loophole that I'm forgetting? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Mar 10, 2014 at 9:43 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>>>>>>>>> How the hell can I do GA or SHMEM then? Roll my own mutexes and commit perf-suicide? >>>>>>>>>> >>>>>>>>>> Jeff >>>>>>>>>> >>>>>>>>>> Sent from my iPhone >>>>>>>>>> >>>>>>>>>>> On Mar 10, 2014, at 8:32 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>> You can't use replace and sum concurrently at a given target address. >>>>>>>>>>> >>>>>>>>>>> ~Jim. >>>>>>>>>>> >>>>>>>>>>> On Mon, Mar 10, 2014 at 4:30 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>>>>>>>>>> Given the following, how do I use MPI_NO_OP, MPI_REPLACE and MPI_SUM >>>>>>>>>>> in accumulate/atomic operations in a standard-compliant way? >>>>>>>>>>> >>>>>>>>>>> accumulate_ops — if set to same_op, the implementation will assume >>>>>>>>>>> that all concurrent accumulate calls to the same target address will >>>>>>>>>>> use the same operation. If set to same_op_no_op, then the >>>>>>>>>>> implementation will assume that all concurrent accumulate calls to the >>>>>>>>>>> same target address will use the same operation or MPI_NO_OP. This can >>>>>>>>>>> eliminate the need to protect access for certain operation types where >>>>>>>>>>> the hardware can guarantee atomicity. The default is same_op_no_op. >>>>>>>>>>> >>>>>>>>>>> We discuss this before and the resolution was not satisfying to me. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Jeff >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Jeff Hammond >>>>>>>>>>> jeff.science(a)gmail.com >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> mpiwg-rma mailing list >>>>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> mpiwg-rma mailing list >>>>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> mpiwg-rma mailing list >>>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> mpiwg-rma mailing list >>>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> mpiwg-rma mailing list >>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Jeff Hammond >>>>>>>> jeff.science(a)gmail.com >>>>>>>> _______________________________________________ >>>>>>>> mpiwg-rma mailing list >>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>> >>>>>>> _______________________________________________ >>>>>>> mpiwg-rma mailing list >>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>> _______________________________________________ >>>>>> mpiwg-rma mailing list >>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>> >>>>> _______________________________________________ >>>>> mpiwg-rma mailing list >>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>> >>>> >>>> >>>> -- >>>> Jeff Hammond >>>> jeff.science(a)gmail.com >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>> >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] same_op_no_op
by Balaji, Pavan 13 Mar '14

13 Mar '14

Well, I think they are not allowed. If the implementation can assume that they are the same, if a single process does two accumulate operations, the implementation is allowed to add an assert saying first_op == second_op. That’ll blow up the program if they are not the same. — Pavan On Mar 13, 2014, at 12:16 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > I don't think MPI-3 says they are not allowed, just that the > implementation "will assume" that >1 operations (except for NO_OP) are > never used concurrently. Since the effect of the implementation's > assumptions cannot be known, I would like to assume that concurrent > different operations are not necessarily atomic i.e. implementation > defined, and most certainly not erroneous. > > As to screwing this up, yes, I agree on multiple levels :-) > > Jeff > > On Thu, Mar 13, 2014 at 12:02 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: >> >> MPI-2.2 says that accumulate with different ops are not atomic. >> >> MPI-3 says that accumulate with different ops are not allowed (since same_op_no_op is default). >> >> I think we screwed that up? >> >> — Pavan >> >> On Mar 13, 2014, at 11:48 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >> >>> It is extremely difficult to see that this is what the MPI-3 standard says. >>> >>> First we have this: >>> >>> "The outcome of concurrent accumulate operations to the same location >>> with the same predefined datatype is as if the accumulates were done >>> at that location in some serial order. Additional restrictions on the >>> operation apply; see the info key accumulate_ops in Section 11.2.1. >>> Concurrent accumulate operations with different origin and target >>> pairs are not ordered. Thus, there is no guarantee that the entire >>> call to an accumulate operation is executed atomically. The effect of >>> this lack of atomicity is limited: The previous correctness conditions >>> imply that a location updated by a call to an accumulate operation >>> cannot be accessed by a load or an RMA call other than accumulate >>> until the accumulate operation has completed (at the target). >>> Different interleavings can lead to different results only to the >>> extent that computer arithmetics are not truly associative or >>> commutative. The outcome of accumulate operations with overlapping >>> types of different sizes or target displacements is undefined." >>> [11.7.1 Atomicity] >>> >>> Then we have this: >>> >>> "accumulate_ops — if set to same_op, the implementation will assume >>> that all concurrent accumulate calls to the same target address will >>> use the same operation. If set to same_op_no_op, then the >>> implementation will assume that all concurrent accumulate calls to the >>> same target address will use the same operation or MPI_NO_OP. This can >>> eliminate the need to protect access for certain operation types where >>> the hardware can guarantee atomicity. The default is same_op_no_op." >>> [11.2.1 Window Creation] >>> >>> I was not aware that the definition of info keys was normative, given >>> that implementations are free to ignore them. Even if info key text >>> is normative, one has to infer from the fact that same_op_no_op is the >>> default info behavior - and thus RMA semantic - that accumulate >>> atomicity is restricted to the case where one uses the same op or noop >>> but not replace. >>> >>> The MPI-2.2 spec is unambiguous because it explicitly requires the >>> same operation in 11.7.1 Atomicity. This text was removed in MPI-3.0 >>> in favor of the info key text. >>> >>> Best, >>> >>> Jeff >>> >>> On Tue, Mar 11, 2014 at 12:04 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: >>>> >>>> MPI-2 defines atomicity only for the same operation, not any operation for MPI_ACCUMULATE. >>>> >>>> — Pavan >>>> >>>> On Mar 10, 2014, at 11:22 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>>> >>>>> So MPI-2 denied compatibility between replace and not-replace? >>>>> >>>>> Jeff >>>>> >>>>> Sent from my iPhone >>>>> >>>>>> On Mar 11, 2014, at 12:06 AM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: >>>>>> >>>>>> >>>>>> It doesn’t break backward compatibility. The info argument is still useful when you don’t want to use replace. I don’t see anything wrong with it. >>>>>> >>>>>>> On Mar 10, 2014, at 11:01 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>>>>>> >>>>>>> Does this or does this not break BW compatibility w.r.t. MPI-2.2 and >>>>>>> did we do it intentionally? Unless we did so intentionally and >>>>>>> explicitly, I will argue that the WG screwed up and the info key+val >>>>>>> is invalid. >>>>>>> >>>>>>> Jeff >>>>>>> >>>>>>>> On Mon, Mar 10, 2014 at 11:03 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: >>>>>>>> >>>>>>>> If a hardware can implement MPI_SUM, it should be able to implement MPI_SUM with 0 as well. >>>>>>>> >>>>>>>> But that’s not a generic solution. >>>>>>>> >>>>>>>> Jeff: at some point you were planning to bring in a ticket which does more combinations of operations than just same_op and no_op. Maybe it’s worthwhile bringing that up again? >>>>>>>> >>>>>>>> — Pavan >>>>>>>> >>>>>>>>> On Mar 10, 2014, at 9:26 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: >>>>>>>>> >>>>>>>>> Maybe there's a loophole that I'm forgetting? >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Mar 10, 2014 at 9:43 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>>>>>>>> How the hell can I do GA or SHMEM then? Roll my own mutexes and commit perf-suicide? >>>>>>>>> >>>>>>>>> Jeff >>>>>>>>> >>>>>>>>> Sent from my iPhone >>>>>>>>> >>>>>>>>>> On Mar 10, 2014, at 8:32 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: >>>>>>>>>> >>>>>>>>>> You can't use replace and sum concurrently at a given target address. >>>>>>>>>> >>>>>>>>>> ~Jim. >>>>>>>>>> >>>>>>>>>> On Mon, Mar 10, 2014 at 4:30 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>>>>>>>>> Given the following, how do I use MPI_NO_OP, MPI_REPLACE and MPI_SUM >>>>>>>>>> in accumulate/atomic operations in a standard-compliant way? >>>>>>>>>> >>>>>>>>>> accumulate_ops — if set to same_op, the implementation will assume >>>>>>>>>> that all concurrent accumulate calls to the same target address will >>>>>>>>>> use the same operation. If set to same_op_no_op, then the >>>>>>>>>> implementation will assume that all concurrent accumulate calls to the >>>>>>>>>> same target address will use the same operation or MPI_NO_OP. This can >>>>>>>>>> eliminate the need to protect access for certain operation types where >>>>>>>>>> the hardware can guarantee atomicity. The default is same_op_no_op. >>>>>>>>>> >>>>>>>>>> We discuss this before and the resolution was not satisfying to me. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Jeff >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Jeff Hammond >>>>>>>>>> jeff.science(a)gmail.com >>>>>>>>>> _______________________________________________ >>>>>>>>>> mpiwg-rma mailing list >>>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> mpiwg-rma mailing list >>>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> mpiwg-rma mailing list >>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> mpiwg-rma mailing list >>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> mpiwg-rma mailing list >>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Jeff Hammond >>>>>>> jeff.science(a)gmail.com >>>>>>> _______________________________________________ >>>>>>> mpiwg-rma mailing list >>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>> >>>>>> _______________________________________________ >>>>>> mpiwg-rma mailing list >>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>> _______________________________________________ >>>>> mpiwg-rma mailing list >>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>> >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>> >>> >>> >>> -- >>> Jeff Hammond >>> jeff.science(a)gmail.com >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] same_op_no_op
by Jeff Hammond 13 Mar '14

13 Mar '14

I don't think MPI-3 says they are not allowed, just that the implementation "will assume" that >1 operations (except for NO_OP) are never used concurrently. Since the effect of the implementation's assumptions cannot be known, I would like to assume that concurrent different operations are not necessarily atomic i.e. implementation defined, and most certainly not erroneous. As to screwing this up, yes, I agree on multiple levels :-) Jeff On Thu, Mar 13, 2014 at 12:02 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > MPI-2.2 says that accumulate with different ops are not atomic. > > MPI-3 says that accumulate with different ops are not allowed (since same_op_no_op is default). > > I think we screwed that up? > > — Pavan > > On Mar 13, 2014, at 11:48 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > >> It is extremely difficult to see that this is what the MPI-3 standard says. >> >> First we have this: >> >> "The outcome of concurrent accumulate operations to the same location >> with the same predefined datatype is as if the accumulates were done >> at that location in some serial order. Additional restrictions on the >> operation apply; see the info key accumulate_ops in Section 11.2.1. >> Concurrent accumulate operations with different origin and target >> pairs are not ordered. Thus, there is no guarantee that the entire >> call to an accumulate operation is executed atomically. The effect of >> this lack of atomicity is limited: The previous correctness conditions >> imply that a location updated by a call to an accumulate operation >> cannot be accessed by a load or an RMA call other than accumulate >> until the accumulate operation has completed (at the target). >> Different interleavings can lead to different results only to the >> extent that computer arithmetics are not truly associative or >> commutative. The outcome of accumulate operations with overlapping >> types of different sizes or target displacements is undefined." >> [11.7.1 Atomicity] >> >> Then we have this: >> >> "accumulate_ops — if set to same_op, the implementation will assume >> that all concurrent accumulate calls to the same target address will >> use the same operation. If set to same_op_no_op, then the >> implementation will assume that all concurrent accumulate calls to the >> same target address will use the same operation or MPI_NO_OP. This can >> eliminate the need to protect access for certain operation types where >> the hardware can guarantee atomicity. The default is same_op_no_op." >> [11.2.1 Window Creation] >> >> I was not aware that the definition of info keys was normative, given >> that implementations are free to ignore them. Even if info key text >> is normative, one has to infer from the fact that same_op_no_op is the >> default info behavior - and thus RMA semantic - that accumulate >> atomicity is restricted to the case where one uses the same op or noop >> but not replace. >> >> The MPI-2.2 spec is unambiguous because it explicitly requires the >> same operation in 11.7.1 Atomicity. This text was removed in MPI-3.0 >> in favor of the info key text. >> >> Best, >> >> Jeff >> >> On Tue, Mar 11, 2014 at 12:04 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: >>> >>> MPI-2 defines atomicity only for the same operation, not any operation for MPI_ACCUMULATE. >>> >>> — Pavan >>> >>> On Mar 10, 2014, at 11:22 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>> >>>> So MPI-2 denied compatibility between replace and not-replace? >>>> >>>> Jeff >>>> >>>> Sent from my iPhone >>>> >>>>> On Mar 11, 2014, at 12:06 AM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: >>>>> >>>>> >>>>> It doesn’t break backward compatibility. The info argument is still useful when you don’t want to use replace. I don’t see anything wrong with it. >>>>> >>>>>> On Mar 10, 2014, at 11:01 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>>>>> >>>>>> Does this or does this not break BW compatibility w.r.t. MPI-2.2 and >>>>>> did we do it intentionally? Unless we did so intentionally and >>>>>> explicitly, I will argue that the WG screwed up and the info key+val >>>>>> is invalid. >>>>>> >>>>>> Jeff >>>>>> >>>>>>> On Mon, Mar 10, 2014 at 11:03 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: >>>>>>> >>>>>>> If a hardware can implement MPI_SUM, it should be able to implement MPI_SUM with 0 as well. >>>>>>> >>>>>>> But that’s not a generic solution. >>>>>>> >>>>>>> Jeff: at some point you were planning to bring in a ticket which does more combinations of operations than just same_op and no_op. Maybe it’s worthwhile bringing that up again? >>>>>>> >>>>>>> — Pavan >>>>>>> >>>>>>>> On Mar 10, 2014, at 9:26 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: >>>>>>>> >>>>>>>> Maybe there's a loophole that I'm forgetting? >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Mar 10, 2014 at 9:43 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>>>>>>> How the hell can I do GA or SHMEM then? Roll my own mutexes and commit perf-suicide? >>>>>>>> >>>>>>>> Jeff >>>>>>>> >>>>>>>> Sent from my iPhone >>>>>>>> >>>>>>>>> On Mar 10, 2014, at 8:32 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: >>>>>>>>> >>>>>>>>> You can't use replace and sum concurrently at a given target address. >>>>>>>>> >>>>>>>>> ~Jim. >>>>>>>>> >>>>>>>>> On Mon, Mar 10, 2014 at 4:30 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>>>>>>>> Given the following, how do I use MPI_NO_OP, MPI_REPLACE and MPI_SUM >>>>>>>>> in accumulate/atomic operations in a standard-compliant way? >>>>>>>>> >>>>>>>>> accumulate_ops — if set to same_op, the implementation will assume >>>>>>>>> that all concurrent accumulate calls to the same target address will >>>>>>>>> use the same operation. If set to same_op_no_op, then the >>>>>>>>> implementation will assume that all concurrent accumulate calls to the >>>>>>>>> same target address will use the same operation or MPI_NO_OP. This can >>>>>>>>> eliminate the need to protect access for certain operation types where >>>>>>>>> the hardware can guarantee atomicity. The default is same_op_no_op. >>>>>>>>> >>>>>>>>> We discuss this before and the resolution was not satisfying to me. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Jeff >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Jeff Hammond >>>>>>>>> jeff.science(a)gmail.com >>>>>>>>> _______________________________________________ >>>>>>>>> mpiwg-rma mailing list >>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> mpiwg-rma mailing list >>>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> mpiwg-rma mailing list >>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> mpiwg-rma mailing list >>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>> >>>>>>> _______________________________________________ >>>>>>> mpiwg-rma mailing list >>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Jeff Hammond >>>>>> jeff.science(a)gmail.com >>>>>> _______________________________________________ >>>>>> mpiwg-rma mailing list >>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>> >>>>> _______________________________________________ >>>>> mpiwg-rma mailing list >>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>> >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] same_op_no_op
by Balaji, Pavan 13 Mar '14

13 Mar '14

MPI-2.2 says that accumulate with different ops are not atomic. MPI-3 says that accumulate with different ops are not allowed (since same_op_no_op is default). I think we screwed that up? — Pavan On Mar 13, 2014, at 11:48 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > It is extremely difficult to see that this is what the MPI-3 standard says. > > First we have this: > > "The outcome of concurrent accumulate operations to the same location > with the same predefined datatype is as if the accumulates were done > at that location in some serial order. Additional restrictions on the > operation apply; see the info key accumulate_ops in Section 11.2.1. > Concurrent accumulate operations with different origin and target > pairs are not ordered. Thus, there is no guarantee that the entire > call to an accumulate operation is executed atomically. The effect of > this lack of atomicity is limited: The previous correctness conditions > imply that a location updated by a call to an accumulate operation > cannot be accessed by a load or an RMA call other than accumulate > until the accumulate operation has completed (at the target). > Different interleavings can lead to different results only to the > extent that computer arithmetics are not truly associative or > commutative. The outcome of accumulate operations with overlapping > types of different sizes or target displacements is undefined." > [11.7.1 Atomicity] > > Then we have this: > > "accumulate_ops — if set to same_op, the implementation will assume > that all concurrent accumulate calls to the same target address will > use the same operation. If set to same_op_no_op, then the > implementation will assume that all concurrent accumulate calls to the > same target address will use the same operation or MPI_NO_OP. This can > eliminate the need to protect access for certain operation types where > the hardware can guarantee atomicity. The default is same_op_no_op." > [11.2.1 Window Creation] > > I was not aware that the definition of info keys was normative, given > that implementations are free to ignore them. Even if info key text > is normative, one has to infer from the fact that same_op_no_op is the > default info behavior - and thus RMA semantic - that accumulate > atomicity is restricted to the case where one uses the same op or noop > but not replace. > > The MPI-2.2 spec is unambiguous because it explicitly requires the > same operation in 11.7.1 Atomicity. This text was removed in MPI-3.0 > in favor of the info key text. > > Best, > > Jeff > > On Tue, Mar 11, 2014 at 12:04 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: >> >> MPI-2 defines atomicity only for the same operation, not any operation for MPI_ACCUMULATE. >> >> — Pavan >> >> On Mar 10, 2014, at 11:22 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >> >>> So MPI-2 denied compatibility between replace and not-replace? >>> >>> Jeff >>> >>> Sent from my iPhone >>> >>>> On Mar 11, 2014, at 12:06 AM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: >>>> >>>> >>>> It doesn’t break backward compatibility. The info argument is still useful when you don’t want to use replace. I don’t see anything wrong with it. >>>> >>>>> On Mar 10, 2014, at 11:01 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>>>> >>>>> Does this or does this not break BW compatibility w.r.t. MPI-2.2 and >>>>> did we do it intentionally? Unless we did so intentionally and >>>>> explicitly, I will argue that the WG screwed up and the info key+val >>>>> is invalid. >>>>> >>>>> Jeff >>>>> >>>>>> On Mon, Mar 10, 2014 at 11:03 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: >>>>>> >>>>>> If a hardware can implement MPI_SUM, it should be able to implement MPI_SUM with 0 as well. >>>>>> >>>>>> But that’s not a generic solution. >>>>>> >>>>>> Jeff: at some point you were planning to bring in a ticket which does more combinations of operations than just same_op and no_op. Maybe it’s worthwhile bringing that up again? >>>>>> >>>>>> — Pavan >>>>>> >>>>>>> On Mar 10, 2014, at 9:26 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: >>>>>>> >>>>>>> Maybe there's a loophole that I'm forgetting? >>>>>>> >>>>>>> >>>>>>> On Mon, Mar 10, 2014 at 9:43 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>>>>>> How the hell can I do GA or SHMEM then? Roll my own mutexes and commit perf-suicide? >>>>>>> >>>>>>> Jeff >>>>>>> >>>>>>> Sent from my iPhone >>>>>>> >>>>>>>> On Mar 10, 2014, at 8:32 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: >>>>>>>> >>>>>>>> You can't use replace and sum concurrently at a given target address. >>>>>>>> >>>>>>>> ~Jim. >>>>>>>> >>>>>>>> On Mon, Mar 10, 2014 at 4:30 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>>>>>>> Given the following, how do I use MPI_NO_OP, MPI_REPLACE and MPI_SUM >>>>>>>> in accumulate/atomic operations in a standard-compliant way? >>>>>>>> >>>>>>>> accumulate_ops — if set to same_op, the implementation will assume >>>>>>>> that all concurrent accumulate calls to the same target address will >>>>>>>> use the same operation. If set to same_op_no_op, then the >>>>>>>> implementation will assume that all concurrent accumulate calls to the >>>>>>>> same target address will use the same operation or MPI_NO_OP. This can >>>>>>>> eliminate the need to protect access for certain operation types where >>>>>>>> the hardware can guarantee atomicity. The default is same_op_no_op. >>>>>>>> >>>>>>>> We discuss this before and the resolution was not satisfying to me. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Jeff >>>>>>>> >>>>>>>> -- >>>>>>>> Jeff Hammond >>>>>>>> jeff.science(a)gmail.com >>>>>>>> _______________________________________________ >>>>>>>> mpiwg-rma mailing list >>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> mpiwg-rma mailing list >>>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>> >>>>>>> _______________________________________________ >>>>>>> mpiwg-rma mailing list >>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>>> >>>>>>> _______________________________________________ >>>>>>> mpiwg-rma mailing list >>>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>>> >>>>>> _______________________________________________ >>>>>> mpiwg-rma mailing list >>>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>>> >>>>> >>>>> >>>>> -- >>>>> Jeff Hammond >>>>> jeff.science(a)gmail.com >>>>> _______________________________________________ >>>>> mpiwg-rma mailing list >>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>> >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0