January 2014 - mpiwg-rma - mailman3.cels.anl.gov

Re: [mpiwg-rma] MPI_WIN_CREATE for intercommunicators
by Jeff Hammond 22 Jan '14

22 Jan '14

The other way to solve your issue is to use two windows. I'll write it up in detail later. In any case, I'm not totally opposed to your use case. I just don't know about the cost-benefit relative to many other features. Jeff On Wed, Jan 22, 2014 at 4:48 AM, Thomas Jahns <jahns(a)dkrz.de> wrote: > Hello Jeff, > > On 01/21/14 20:03, Jeff Hammond wrote: >> It makes my brain hurt just to think about this. > > I certainly intended no harm, however mild ;-). > >> First, one should recognize that RMA communication occurs using the >> window object, not the communicator as in p2p and collectives. The >> reason for a comm object in WIN_CREATE et al. is to enable MPI to >> communicate between participating processes in order to create the >> window. For example, many implementations will do an ALLGATHER inside >> of WIN_CREATE. That operation needs a comm object. > > So there is no problem here with using an intercommunicator? > >> The second purpose of the comm is to generate the group of the window >> in order to denote which ranks know about the window. It bothers me a >> negligible amount that the exact definition of the rank arguments for >> RMA is not defined explicitly, but one can reasonably assume that >> those are ranks on the group of the window. I suppose one could talk >> about local and remote groups for a window but again, brain hurt. > > I think what's defined in lines 20-25 of page 420 of MPI 3.0 already makes > perfect sense if comm were an intercommunicator since it's specified in terms of > what would happen for point-to-point communications. So no problem either. > >> I do not believe that the communicator has any purpose beyond this and >> thus do not know what the consequences of allowing intercomms would >> be. Perhaps we can use - only for the sake of argument since it is >> not required to have these semantics - that WIN_CREATE is like >> ALLGATHER and use the semantics of ALLGATHER on intercomms to define >> the semantics of win objects created using intercomms. > > For e.g. MPI_Win_post one needs a group, based on the communicator originally > passed to MPI_Win_create but that's already solved for intracommunicators and no > new problem. > >> In any case, I don't see any real value in this, honestly. Efficiency >> arguments assume implementation details not specified by the standard. >> Making a promise to not communicate with some ranks has no effect on >> some implementations. Of course we know ones where it does, but >> that's not germane. > > Still it's something I cannot express currently. > >> Honestly, I think any sane implementer will just MPI_COMM_TEST_INTER >> -> INTERCOMM_MERGE inside of of WIN_CREATE (et al.), do everything as >> before and then free the temp intracomm at the end. Then I would >> never, ever think about this issue again until someone paid me a huge >> sum of money to optimize for the intercomm case. Thus, the result of >> changing the standard will be almost nothing other than people will be >> able to avoid doing the following explicitly: > > For the above scenario the original communicator needs to be retained, i.e. to > make use of for a later MPI_Group_translate, but otherwise you are right unless > mpi_win_create were to be significantly less expensive with less potential paths. > >> int MPE_Win_create(void *base, MPI_Aint size, int disp_unit, MPI_Info >> info, MPI_Comm comm, MPI_Win *win) >> { >> int is_intercomm; >> MPI_Comm intracomm; >> MPI_Comm_test_inter(comm, &is_intercomm); >> if (is_intercomm) >> MPI_Intercomm_merge(comm, 0, &intracomm); >> else >> intracomm = comm; >> MPI_Win_create(base, size, disp_unit, info, intracomm, win); >> if (is_intercomm) >> MPI_Comm_free(&intracomm); >> return MPI_SUCCESS; >> } >> >> In short, I don't see any value in trying to define the perverse >> semantics of intercomm-based windows to avoid the ~12 lines of code >> above. > > In my case, where I have a client-server setup with point-to-point communication > happening via an intercommunicator, adding bulk communication via RMA the > current state of affairs means I have to use an additional communicator and > translate IDs. But sure, it can be done with something similar to the above, > it's just more code on the user side. > >> In the event we try to act on this proposal, I suppose we use the very >> limited text regarding send-recv on intercomms as our guide: "A target >> process is addressed by its rank in the remote group, both for sends >> and for receives." [pg. 258] > > That's what seemed intuitive to me too (see above). > > Regards, Thomas > -- > Thomas Jahns > DKRZ GmbH, Department: Application software > > Deutsches Klimarechenzentrum > Bundesstraße 45a > D-20146 Hamburg > > Phone: +49-40-460094-151 > Fax: +49-40-460094-270 > Email: Thomas Jahns <jahns(a)dkrz.de> > -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: MPI_WIN_CREATE for intercommunicators
by Thomas Jahns 22 Jan '14

22 Jan '14

Hello, On Jan 22, 2014, at 19:56 , Jeff Hammond wrote: > On Wed, Jan 22, 2014 at 12:16 PM, Barrett, Brian W > <bwbarre(a)sandia.gov> wrote: >> On 1/22/14 11:09 AM, "Jeff Hammond" <jeff.science(a)gmail.com> wrote: >> >>> On Wed, Jan 22, 2014 at 11:53 AM, Barrett, Brian W <bwbarre(a)sandia.gov >>> > >>> wrote: >>>> On 1/22/14 3:41 AM, "Thomas Jahns" <jahns(a)dkrz.de> wrote: >>>> >>>>> Hello Maik, >>>>> >>>>> On 01/22/14 10:14, maik peterson wrote: >>>>>> thomas, today you should start thinking about other possibilities >>>>>> to solve your problem. this mpi rma stuff is crappy. >>>>> >>>>> I guess that very much depends on your kind of machine. We've >>>>> found it >>>>> to >>>>> work >>>>> very nice with MVAPICH, not so great with PE and OpenMPI. perhaps I should qualify my very brief original statement before it becomes more flammable: MVAPICH gave the best out-of-the-box performance on that particular system (IBM xSeries with FDR IB) and while OpenMPI (on the same machine) did give the desired performance improvements with RMA, base-line performance was lower for some yet undiscovered reason, but system adminstration will look into it. I have no doubt this could have been different. For PE we asked IBM what to do differently but have no final answer so far. Regards, Thomas -- Thomas Jahns DKRZ GmbH, Department: Application software Deutsches Klimarechenzentrum Bundesstraße 45a D-20146 Hamburg Phone: +49-40-460094-151 Fax: +49-40-460094-270 Email: Thomas Jahns <jahns(a)dkrz.de>

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: MPI_WIN_CREATE for intercommunicators
by Jeff Hammond 22 Jan '14

22 Jan '14

On Wed, Jan 22, 2014 at 12:16 PM, Barrett, Brian W <bwbarre(a)sandia.gov> wrote: > On 1/22/14 11:09 AM, "Jeff Hammond" <jeff.science(a)gmail.com> wrote: > >>On Wed, Jan 22, 2014 at 11:53 AM, Barrett, Brian W <bwbarre(a)sandia.gov> >>wrote: >>> On 1/22/14 3:41 AM, "Thomas Jahns" <jahns(a)dkrz.de> wrote: >>> >>>>Hello Maik, >>>> >>>>On 01/22/14 10:14, maik peterson wrote: >>>>> thomas, today you should start thinking about other possibilities >>>>> to solve your problem. this mpi rma stuff is crappy. >>>> >>>>I guess that very much depends on your kind of machine. We've found it >>>>to >>>>work >>>>very nice with MVAPICH, not so great with PE and OpenMPI. >>> >>> Another way to put it; it really depends on how much work your MPI >>> implementor has put into one-sided for your hardware. MVAPICH has put >>>in >>> a good deal of effort to use IB-natvie RDMA semantics when implementing >>> MPI one-sided. Open MPI has not put that effort into IB (it has for >>> another network, but that's a different story), so the performance is >>>much >>> lower. Open MPI's largely driven by squeaky wheels, and that wheel's >>>been >>> pretty quiet :).. >> >>Are you saying that if I make a big deal about OpenMPI's RMA warts on >>your list that you all will make it MPI-3-feature-complete and perform >>really well? > > Chances of me personally making Open MPI perform really well over IB are > pretty minimal. But Mellanox might have a different view ;). My point > was that of all the things people complain about, RMA performance > generally isn't one of them. Function comes before performance. Once datatypes work, I'll worry about bandwidth and latency. My completely uninformed assumption is that Mellanox's customers are happy with MVAPICH2 for RMA and thus they don't have any incentive to duplicate OSU's effort. Jeff -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: MPI_WIN_CREATE for intercommunicators
by Barrett, Brian W 22 Jan '14

22 Jan '14

On 1/22/14 11:09 AM, "Jeff Hammond" <jeff.science(a)gmail.com> wrote: >On Wed, Jan 22, 2014 at 11:53 AM, Barrett, Brian W <bwbarre(a)sandia.gov> >wrote: >> On 1/22/14 3:41 AM, "Thomas Jahns" <jahns(a)dkrz.de> wrote: >> >>>Hello Maik, >>> >>>On 01/22/14 10:14, maik peterson wrote: >>>> thomas, today you should start thinking about other possibilities >>>> to solve your problem. this mpi rma stuff is crappy. >>> >>>I guess that very much depends on your kind of machine. We've found it >>>to >>>work >>>very nice with MVAPICH, not so great with PE and OpenMPI. >> >> Another way to put it; it really depends on how much work your MPI >> implementor has put into one-sided for your hardware. MVAPICH has put >>in >> a good deal of effort to use IB-natvie RDMA semantics when implementing >> MPI one-sided. Open MPI has not put that effort into IB (it has for >> another network, but that's a different story), so the performance is >>much >> lower. Open MPI's largely driven by squeaky wheels, and that wheel's >>been >> pretty quiet :).. > >Are you saying that if I make a big deal about OpenMPI's RMA warts on >your list that you all will make it MPI-3-feature-complete and perform >really well? Chances of me personally making Open MPI perform really well over IB are pretty minimal. But Mellanox might have a different view ;). My point was that of all the things people complain about, RMA performance generally isn't one of them. Brian -- Brian W. Barrett Scalable System Software Group Sandia National Laboratories

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: MPI_WIN_CREATE for intercommunicators
by Jeff Hammond 22 Jan '14

22 Jan '14

On Wed, Jan 22, 2014 at 11:53 AM, Barrett, Brian W <bwbarre(a)sandia.gov> wrote: > On 1/22/14 3:41 AM, "Thomas Jahns" <jahns(a)dkrz.de> wrote: > >>Hello Maik, >> >>On 01/22/14 10:14, maik peterson wrote: >>> thomas, today you should start thinking about other possibilities >>> to solve your problem. this mpi rma stuff is crappy. >> >>I guess that very much depends on your kind of machine. We've found it to >>work >>very nice with MVAPICH, not so great with PE and OpenMPI. > > Another way to put it; it really depends on how much work your MPI > implementor has put into one-sided for your hardware. MVAPICH has put in > a good deal of effort to use IB-natvie RDMA semantics when implementing > MPI one-sided. Open MPI has not put that effort into IB (it has for > another network, but that's a different story), so the performance is much > lower. Open MPI's largely driven by squeaky wheels, and that wheel's been > pretty quiet :).. Are you saying that if I make a big deal about OpenMPI's RMA warts on your list that you all will make it MPI-3-feature-complete and perform really well? Or maybe you are using squeaky as a synonym for remunerating :-) Jeff -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: MPI_WIN_CREATE for intercommunicators
by Barrett, Brian W 22 Jan '14

22 Jan '14

On 1/22/14 3:41 AM, "Thomas Jahns" <jahns(a)dkrz.de> wrote: >Hello Maik, > >On 01/22/14 10:14, maik peterson wrote: >> thomas, today you should start thinking about other possibilities >> to solve your problem. this mpi rma stuff is crappy. > >I guess that very much depends on your kind of machine. We've found it to >work >very nice with MVAPICH, not so great with PE and OpenMPI. Another way to put it; it really depends on how much work your MPI implementor has put into one-sided for your hardware. MVAPICH has put in a good deal of effort to use IB-natvie RDMA semantics when implementing MPI one-sided. Open MPI has not put that effort into IB (it has for another network, but that's a different story), so the performance is much lower. Open MPI's largely driven by squeaky wheels, and that wheel's been pretty quiet :).. Brian -- Brian W. Barrett Scalable System Software Group Sandia National Laboratories

1 0

Re: [mpiwg-rma] MPI_WIN_CREATE for intercommunicators
by Thomas Jahns 22 Jan '14

22 Jan '14

Hello Jeff, On 01/21/14 20:03, Jeff Hammond wrote: > It makes my brain hurt just to think about this. I certainly intended no harm, however mild ;-). > First, one should recognize that RMA communication occurs using the > window object, not the communicator as in p2p and collectives. The > reason for a comm object in WIN_CREATE et al. is to enable MPI to > communicate between participating processes in order to create the > window. For example, many implementations will do an ALLGATHER inside > of WIN_CREATE. That operation needs a comm object. So there is no problem here with using an intercommunicator? > The second purpose of the comm is to generate the group of the window > in order to denote which ranks know about the window. It bothers me a > negligible amount that the exact definition of the rank arguments for > RMA is not defined explicitly, but one can reasonably assume that > those are ranks on the group of the window. I suppose one could talk > about local and remote groups for a window but again, brain hurt. I think what's defined in lines 20-25 of page 420 of MPI 3.0 already makes perfect sense if comm were an intercommunicator since it's specified in terms of what would happen for point-to-point communications. So no problem either. > I do not believe that the communicator has any purpose beyond this and > thus do not know what the consequences of allowing intercomms would > be. Perhaps we can use - only for the sake of argument since it is > not required to have these semantics - that WIN_CREATE is like > ALLGATHER and use the semantics of ALLGATHER on intercomms to define > the semantics of win objects created using intercomms. For e.g. MPI_Win_post one needs a group, based on the communicator originally passed to MPI_Win_create but that's already solved for intracommunicators and no new problem. > In any case, I don't see any real value in this, honestly. Efficiency > arguments assume implementation details not specified by the standard. > Making a promise to not communicate with some ranks has no effect on > some implementations. Of course we know ones where it does, but > that's not germane. Still it's something I cannot express currently. > Honestly, I think any sane implementer will just MPI_COMM_TEST_INTER > -> INTERCOMM_MERGE inside of of WIN_CREATE (et al.), do everything as > before and then free the temp intracomm at the end. Then I would > never, ever think about this issue again until someone paid me a huge > sum of money to optimize for the intercomm case. Thus, the result of > changing the standard will be almost nothing other than people will be > able to avoid doing the following explicitly: For the above scenario the original communicator needs to be retained, i.e. to make use of for a later MPI_Group_translate, but otherwise you are right unless mpi_win_create were to be significantly less expensive with less potential paths. > int MPE_Win_create(void *base, MPI_Aint size, int disp_unit, MPI_Info > info, MPI_Comm comm, MPI_Win *win) > { > int is_intercomm; > MPI_Comm intracomm; > MPI_Comm_test_inter(comm, &is_intercomm); > if (is_intercomm) > MPI_Intercomm_merge(comm, 0, &intracomm); > else > intracomm = comm; > MPI_Win_create(base, size, disp_unit, info, intracomm, win); > if (is_intercomm) > MPI_Comm_free(&intracomm); > return MPI_SUCCESS; > } > > In short, I don't see any value in trying to define the perverse > semantics of intercomm-based windows to avoid the ~12 lines of code > above. In my case, where I have a client-server setup with point-to-point communication happening via an intercommunicator, adding bulk communication via RMA the current state of affairs means I have to use an additional communicator and translate IDs. But sure, it can be done with something similar to the above, it's just more code on the user side. > In the event we try to act on this proposal, I suppose we use the very > limited text regarding send-recv on intercomms as our guide: "A target > process is addressed by its rank in the remote group, both for sends > and for receives." [pg. 258] That's what seemed intuitive to me too (see above). Regards, Thomas -- Thomas Jahns DKRZ GmbH, Department: Application software Deutsches Klimarechenzentrum Bundesstraße 45a D-20146 Hamburg Phone: +49-40-460094-151 Fax: +49-40-460094-270 Email: Thomas Jahns <jahns(a)dkrz.de>

1 0

Re: [mpiwg-rma] MPI_WIN_CREATE for intercommunicators
by Thomas Jahns 22 Jan '14

22 Jan '14

Hello Maik, On 01/22/14 10:14, maik peterson wrote: > thomas, today you should start thinking about other possibilities > to solve your problem. this mpi rma stuff is crappy. I guess that very much depends on your kind of machine. We've found it to work very nice with MVAPICH, not so great with PE and OpenMPI. Regards, Thomas -- Thomas Jahns DKRZ GmbH, Department: Application software Deutsches Klimarechenzentrum Bundesstraße 45a D-20146 Hamburg Phone: +49-40-460094-151 Fax: +49-40-460094-270 Email: Thomas Jahns <jahns(a)dkrz.de>

1 0

Re: [mpiwg-rma] MPI_WIN_CREATE for intercommunicators
by maik peterson 22 Jan '14

22 Jan '14

thomas, today you should start thinking about other possibilities to solve your problem. this mpi rma stuff is crappy. 2014/1/21 Thomas Jahns <jahns(a)dkrz.de> > Hello, > > should this be the wrong mailing list[1], I apologize in advance and hope > for > directions to the correct place to pose my question. > > From the wording of the MPI 2.2 and 3.0 standards I can see (first > paragraph of 11.2.1 in MPI 2.2, procedurce specification of > MPI_WIN_CREATE in MPI 3.0) that MPI_WIN_CREATE is only allowed for > intracommunicators. > > I was wondering why this restriction is there, when > > 1. cases where an intercommunicator is more clear for point-to-point > communications, communications using rma would equally gain in > clarity with an intercommunicator (from my POV at least), > > 2. the required resources (because the number of potential > communication paths would be much lower) could be reduced because > the intercommunicator not only specifies two groups communicating > with each other but also not communicating (at least via the > intercommunicator) within each group. I.e. when it is already known > that the any two processes within one of the two groups of an > intercommunicator would not issue RMA calls for the other, > corresponding resources need not be reserved. Intra-communicator RMA > on the other hand leaves the possibility of any communication pair > open. > > Clearly with two groups of processes of sizes N and M the number of > potential pairs is N * M where a corresponding intercommunicator is > concerned but (N * M)**2 for an intracommunicator and > > 3. the semantics appropriate for an intercommunicator should be easy > to emulate with MPI_INTERCOMM_MERGE and MPI_GROUP_TRANSLATE_RANKS. > > Since allowing intercommunicators in MPI_WIN_CREATE would consequently > > 1. be aesthetically pleasing, > 2. offer increased potential for efficient execution, > 3. seem to have an easy proof-of-concept implementation strategy and > 4. forms a pattern arising naturally from the rest of the standard in my > eyes, > > I'm clearly missing some ambiguity or technical/definition difficulty > here, but can't figure out which, why else would the standard > explicitly restrict MPI_WIN_CREATE to intracommunicators if the above > were the whole story? > > Kind regards, > Thomas Jahns > > [1] I've done some searching and came up with either > <mpiwg-rma(a)lists.mpi-forum.org> or <mpi-comment(a)lists.mpi-forum.org> > -- > Thomas Jahns > DKRZ GmbH, Department: Application software > > Deutsches Klimarechenzentrum > Bundesstraße 45a > D-20146 Hamburg > > Phone: +49-40-460094-151 > Fax: +49-40-460094-270 > Email: Thomas Jahns <jahns(a)dkrz.de> > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >

1 0

Re: [mpiwg-rma] MPI_WIN_CREATE for intercommunicators
by Jeff Hammond 21 Jan '14

21 Jan '14

It makes my brain hurt just to think about this. First, one should recognize that RMA communication occurs using the window object, not the communicator as in p2p and collectives. The reason for a comm object in WIN_CREATE et al. is to enable MPI to communicate between participating processes in order to create the window. For example, many implementations will do an ALLGATHER inside of WIN_CREATE. That operation needs a comm object. The second purpose of the comm is to generate the group of the window in order to denote which ranks know about the window. It bothers me a negligible amount that the exact definition of the rank arguments for RMA is not defined explicitly, but one can reasonably assume that those are ranks on the group of the window. I suppose one could talk about local and remote groups for a window but again, brain hurt. I do not believe that the communicator has any purpose beyond this and thus do not know what the consequences of allowing intercomms would be. Perhaps we can use - only for the sake of argument since it is not required to have these semantics - that WIN_CREATE is like ALLGATHER and use the semantics of ALLGATHER on intercomms to define the semantics of win objects created using intercomms. In any case, I don't see any real value in this, honestly. Efficiency arguments assume implementation details not specified by the standard. Making a promise to not communicate with some ranks has no effect on some implementations. Of course we know ones where it does, but that's not germane. Honestly, I think any sane implementer will just MPI_COMM_TEST_INTER -> INTERCOMM_MERGE inside of of WIN_CREATE (et al.), do everything as before and then free the temp intracomm at the end. Then I would never, ever think about this issue again until someone paid me a huge sum of money to optimize for the intercomm case. Thus, the result of changing the standard will be almost nothing other than people will be able to avoid doing the following explicitly: int MPE_Win_create(void *base, MPI_Aint size, int disp_unit, MPI_Info info, MPI_Comm comm, MPI_Win *win) { int is_intercomm; MPI_Comm intracomm; MPI_Comm_test_inter(comm, &is_intercomm); if (is_intercomm) MPI_Intercomm_merge(comm, 0, &intracomm); else intracomm = comm; MPI_Win_create(base, size, disp_unit, info, intracomm, win); if (is_intercomm) MPI_Comm_free(&intracomm); return MPI_SUCCESS; } In short, I don't see any value in trying to define the perverse semantics of intercomm-based windows to avoid the ~12 lines of code above. In the event we try to act on this proposal, I suppose we use the very limited text regarding send-recv on intercomms as our guide: "A target process is addressed by its rank in the remote group, both for sends and for receives." [pg. 258] Best, Jeff On Tue, Jan 21, 2014 at 12:28 PM, Thomas Jahns <jahns(a)dkrz.de> wrote: > Hello, > > should this be the wrong mailing list[1], I apologize in advance and hope for > directions to the correct place to pose my question. > > From the wording of the MPI 2.2 and 3.0 standards I can see (first > paragraph of 11.2.1 in MPI 2.2, procedurce specification of > MPI_WIN_CREATE in MPI 3.0) that MPI_WIN_CREATE is only allowed for > intracommunicators. > > I was wondering why this restriction is there, when > > 1. cases where an intercommunicator is more clear for point-to-point > communications, communications using rma would equally gain in > clarity with an intercommunicator (from my POV at least), > > 2. the required resources (because the number of potential > communication paths would be much lower) could be reduced because > the intercommunicator not only specifies two groups communicating > with each other but also not communicating (at least via the > intercommunicator) within each group. I.e. when it is already known > that the any two processes within one of the two groups of an > intercommunicator would not issue RMA calls for the other, > corresponding resources need not be reserved. Intra-communicator RMA > on the other hand leaves the possibility of any communication pair > open. > > Clearly with two groups of processes of sizes N and M the number of > potential pairs is N * M where a corresponding intercommunicator is > concerned but (N * M)**2 for an intracommunicator and > > 3. the semantics appropriate for an intercommunicator should be easy > to emulate with MPI_INTERCOMM_MERGE and MPI_GROUP_TRANSLATE_RANKS. > > Since allowing intercommunicators in MPI_WIN_CREATE would consequently > > 1. be aesthetically pleasing, > 2. offer increased potential for efficient execution, > 3. seem to have an easy proof-of-concept implementation strategy and > 4. forms a pattern arising naturally from the rest of the standard in my eyes, > > I'm clearly missing some ambiguity or technical/definition difficulty > here, but can't figure out which, why else would the standard > explicitly restrict MPI_WIN_CREATE to intracommunicators if the above > were the whole story? > > Kind regards, > Thomas Jahns > > [1] I've done some searching and came up with either > <mpiwg-rma(a)lists.mpi-forum.org> or <mpi-comment(a)lists.mpi-forum.org> > -- > Thomas Jahns > DKRZ GmbH, Department: Application software > > Deutsches Klimarechenzentrum > Bundesstraße 45a > D-20146 Hamburg > > Phone: +49-40-460094-151 > Fax: +49-40-460094-270 > Email: Thomas Jahns <jahns(a)dkrz.de> > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com

1 0