October 2013 - mpiwg-rma - mailman3.cels.anl.gov

Re: [mpiwg-rma] [EXTERNAL] Re: same_op_no_op and SHMEM
by Jeff Hammond 24 Oct '13

24 Oct '13

Honestly, I think REPLACE+NO_OP is a useful option, I just think REPLACE+NO_OP+<SUM or ...> is _also_ a useful option. Why don't we just turn our frowns upside down and add this to the standard? Does anyone object to info = same_op_no_op_replace? I would appreciate any feedback on my type-homogeneity suggestion as well. Do people agree that is worth adding? I imagine that there is hardware that can do e.g. sum+long but not sum+double, especially if it has to be atomic. Jeff On Thu, Oct 24, 2013 at 1:46 PM, Barrett, Brian W <bwbarre(a)sandia.gov> wrote: > Or "I will only do gets and compare-and-swap", which is something a couple > of graph codes I've looked at do. But I agree, we probably should have > made it REPLACE, NO_OP, and 1 other op or not added it at all. Sigh :). > > Brian > > On 10/24/13 11:39 AM, "Jeff Hammond" <jeff.science(a)gmail.com> wrote: > >>I read same_op_no_op as "I will use only MPI_REPLACE and MPI_NO_OP" >>i.e. give me nothing more than atomic Put/Get, I do not want to >>actually accumulate anything. >> >>Jeff >> >>On Thu, Oct 24, 2013 at 12:34 PM, Underwood, Keith D >><keith.d.underwood(a)intel.com> wrote: >>> Yes, that's the motivation, but I'm not sure if anybody does atomics >>>without puts.... It seems to me like we should have included >>>MPI_REPLACE in that list >>> >>>> -----Original Message----- >>>> From: mpiwg-rma [mailto:[email protected]] On >>>> Behalf Of Pavan Balaji >>>> Sent: Thursday, October 24, 2013 1:33 PM >>>> To: MPI WG Remote Memory Access working group >>>> Subject: Re: [mpiwg-rma] same_op_no_op and SHMEM >>>> >>>> >>>> The motivation was that it's hard to maintain atomicity when different >>>> operations are done. For example, if the hardware only supports some >>>> atomic operations, but not all, this might result in some operations >>>> happening in hardware and some in software making atomicity hard. In >>>>such >>>> cases, the MPI implementation might need to fall back to software-only >>>> implementations. >>>> >>>> -- Pavan >>>> >>>> On Oct 24, 2013, at 12:23 PM, Jeff Hammond <jeff.science(a)gmail.com> >>>> wrote: >>>> >>>> > I recall that Brian and/or Keith wanted same_op_no_op because of >>>> > SHMEM. However, SHMEM requires the use of MPI_NO_OP (for atomic >>>> Get >>>> > via Get_accumulate), MPI_REPLACE (for atomic Put via Accumulate) and >>>> > MPI_SUM (for add, fadd, inc and finc). So what is the benefit of >>>> > same_op_no_op to SHMEM? Perhaps I remember completely wrong and >>>> the >>>> > motivation was something that does not use the latter atomics. Or >>>> > perhaps it is common for SHMEM codes to not use these and thus the >>>> > assumption is MPI_SUM can be ignored. >>>> > >>>> > Jeff >>>> > >>>> > -- >>>> > Jeff Hammond >>>> > jeff.science(a)gmail.com >>>> > _______________________________________________ >>>> > mpiwg-rma mailing list >>>> > mpiwg-rma(a)lists.mpi-forum.org >>>> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>> >>>> -- >>>> Pavan Balaji >>>> http://www.mcs.anl.gov/~balaji >>>> >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> >> >>-- >>Jeff Hammond >>jeff.science(a)gmail.com >>_______________________________________________ >>mpiwg-rma mailing list >>mpiwg-rma(a)lists.mpi-forum.org >>http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> > > > -- > Brian W. Barrett > Scalable System Software Group > Sandia National Laboratories > > > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: same_op_no_op and SHMEM
by Barrett, Brian W 24 Oct '13

24 Oct '13

Or "I will only do gets and compare-and-swap", which is something a couple of graph codes I've looked at do. But I agree, we probably should have made it REPLACE, NO_OP, and 1 other op or not added it at all. Sigh :). Brian On 10/24/13 11:39 AM, "Jeff Hammond" <jeff.science(a)gmail.com> wrote: >I read same_op_no_op as "I will use only MPI_REPLACE and MPI_NO_OP" >i.e. give me nothing more than atomic Put/Get, I do not want to >actually accumulate anything. > >Jeff > >On Thu, Oct 24, 2013 at 12:34 PM, Underwood, Keith D ><keith.d.underwood(a)intel.com> wrote: >> Yes, that's the motivation, but I'm not sure if anybody does atomics >>without puts.... It seems to me like we should have included >>MPI_REPLACE in that list >> >>> -----Original Message----- >>> From: mpiwg-rma [mailto:[email protected]] On >>> Behalf Of Pavan Balaji >>> Sent: Thursday, October 24, 2013 1:33 PM >>> To: MPI WG Remote Memory Access working group >>> Subject: Re: [mpiwg-rma] same_op_no_op and SHMEM >>> >>> >>> The motivation was that it's hard to maintain atomicity when different >>> operations are done. For example, if the hardware only supports some >>> atomic operations, but not all, this might result in some operations >>> happening in hardware and some in software making atomicity hard. In >>>such >>> cases, the MPI implementation might need to fall back to software-only >>> implementations. >>> >>> -- Pavan >>> >>> On Oct 24, 2013, at 12:23 PM, Jeff Hammond <jeff.science(a)gmail.com> >>> wrote: >>> >>> > I recall that Brian and/or Keith wanted same_op_no_op because of >>> > SHMEM. However, SHMEM requires the use of MPI_NO_OP (for atomic >>> Get >>> > via Get_accumulate), MPI_REPLACE (for atomic Put via Accumulate) and >>> > MPI_SUM (for add, fadd, inc and finc). So what is the benefit of >>> > same_op_no_op to SHMEM? Perhaps I remember completely wrong and >>> the >>> > motivation was something that does not use the latter atomics. Or >>> > perhaps it is common for SHMEM codes to not use these and thus the >>> > assumption is MPI_SUM can be ignored. >>> > >>> > Jeff >>> > >>> > -- >>> > Jeff Hammond >>> > jeff.science(a)gmail.com >>> > _______________________________________________ >>> > mpiwg-rma mailing list >>> > mpiwg-rma(a)lists.mpi-forum.org >>> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>> >>> -- >>> Pavan Balaji >>> http://www.mcs.anl.gov/~balaji >>> >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > >-- >Jeff Hammond >jeff.science(a)gmail.com >_______________________________________________ >mpiwg-rma mailing list >mpiwg-rma(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > -- Brian W. Barrett Scalable System Software Group Sandia National Laboratories

1 0

Re: [mpiwg-rma] same_op_no_op and SHMEM
by Jeff Hammond 24 Oct '13

24 Oct '13

I read same_op_no_op as "I will use only MPI_REPLACE and MPI_NO_OP" i.e. give me nothing more than atomic Put/Get, I do not want to actually accumulate anything. Jeff On Thu, Oct 24, 2013 at 12:34 PM, Underwood, Keith D <keith.d.underwood(a)intel.com> wrote: > Yes, that's the motivation, but I'm not sure if anybody does atomics without puts.... It seems to me like we should have included MPI_REPLACE in that list > >> -----Original Message----- >> From: mpiwg-rma [mailto:[email protected]] On >> Behalf Of Pavan Balaji >> Sent: Thursday, October 24, 2013 1:33 PM >> To: MPI WG Remote Memory Access working group >> Subject: Re: [mpiwg-rma] same_op_no_op and SHMEM >> >> >> The motivation was that it's hard to maintain atomicity when different >> operations are done. For example, if the hardware only supports some >> atomic operations, but not all, this might result in some operations >> happening in hardware and some in software making atomicity hard. In such >> cases, the MPI implementation might need to fall back to software-only >> implementations. >> >> -- Pavan >> >> On Oct 24, 2013, at 12:23 PM, Jeff Hammond <jeff.science(a)gmail.com> >> wrote: >> >> > I recall that Brian and/or Keith wanted same_op_no_op because of >> > SHMEM. However, SHMEM requires the use of MPI_NO_OP (for atomic >> Get >> > via Get_accumulate), MPI_REPLACE (for atomic Put via Accumulate) and >> > MPI_SUM (for add, fadd, inc and finc). So what is the benefit of >> > same_op_no_op to SHMEM? Perhaps I remember completely wrong and >> the >> > motivation was something that does not use the latter atomics. Or >> > perhaps it is common for SHMEM codes to not use these and thus the >> > assumption is MPI_SUM can be ignored. >> > >> > Jeff >> > >> > -- >> > Jeff Hammond >> > jeff.science(a)gmail.com >> > _______________________________________________ >> > mpiwg-rma mailing list >> > mpiwg-rma(a)lists.mpi-forum.org >> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] same_op_no_op and SHMEM
by Jeff Hammond 24 Oct '13

24 Oct '13

Indeed, I was thinking of asking for that to be added to the standard one way or another. However, I decided that it would be way more useful to have window info arguments related to same_type and no_floats since it is far more likely to have atomics in hardware for fixed-point. Jeff On Thu, Oct 24, 2013 at 12:33 PM, Underwood, Keith D <keith.d.underwood(a)intel.com> wrote: > It is also possible that one person thought "NO_OP and REPLACE are really the same thing" and the other person didn't, but you are talking about conversations that have now been migrated off to tape ;-) > >> -----Original Message----- >> From: mpiwg-rma [mailto:[email protected]] On >> Behalf Of Jeff Hammond >> Sent: Thursday, October 24, 2013 1:24 PM >> To: MPI Forum >> Subject: [mpiwg-rma] same_op_no_op and SHMEM >> >> I recall that Brian and/or Keith wanted same_op_no_op because of SHMEM. >> However, SHMEM requires the use of MPI_NO_OP (for atomic Get via >> Get_accumulate), MPI_REPLACE (for atomic Put via Accumulate) and >> MPI_SUM (for add, fadd, inc and finc). So what is the benefit of >> same_op_no_op to SHMEM? Perhaps I remember completely wrong and >> the motivation was something that does not use the latter atomics. Or >> perhaps it is common for SHMEM codes to not use these and thus the >> assumption is MPI_SUM can be ignored. >> >> Jeff >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] same_op_no_op and SHMEM
by Underwood, Keith D 24 Oct '13

24 Oct '13

Yes, that's the motivation, but I'm not sure if anybody does atomics without puts.... It seems to me like we should have included MPI_REPLACE in that list > -----Original Message----- > From: mpiwg-rma [mailto:[email protected]] On > Behalf Of Pavan Balaji > Sent: Thursday, October 24, 2013 1:33 PM > To: MPI WG Remote Memory Access working group > Subject: Re: [mpiwg-rma] same_op_no_op and SHMEM > > > The motivation was that it's hard to maintain atomicity when different > operations are done. For example, if the hardware only supports some > atomic operations, but not all, this might result in some operations > happening in hardware and some in software making atomicity hard. In such > cases, the MPI implementation might need to fall back to software-only > implementations. > > -- Pavan > > On Oct 24, 2013, at 12:23 PM, Jeff Hammond <jeff.science(a)gmail.com> > wrote: > > > I recall that Brian and/or Keith wanted same_op_no_op because of > > SHMEM. However, SHMEM requires the use of MPI_NO_OP (for atomic > Get > > via Get_accumulate), MPI_REPLACE (for atomic Put via Accumulate) and > > MPI_SUM (for add, fadd, inc and finc). So what is the benefit of > > same_op_no_op to SHMEM? Perhaps I remember completely wrong and > the > > motivation was something that does not use the latter atomics. Or > > perhaps it is common for SHMEM codes to not use these and thus the > > assumption is MPI_SUM can be ignored. > > > > Jeff > > > > -- > > Jeff Hammond > > jeff.science(a)gmail.com > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] same_op_no_op and SHMEM
by Underwood, Keith D 24 Oct '13

24 Oct '13

It is also possible that one person thought "NO_OP and REPLACE are really the same thing" and the other person didn't, but you are talking about conversations that have now been migrated off to tape ;-) > -----Original Message----- > From: mpiwg-rma [mailto:[email protected]] On > Behalf Of Jeff Hammond > Sent: Thursday, October 24, 2013 1:24 PM > To: MPI Forum > Subject: [mpiwg-rma] same_op_no_op and SHMEM > > I recall that Brian and/or Keith wanted same_op_no_op because of SHMEM. > However, SHMEM requires the use of MPI_NO_OP (for atomic Get via > Get_accumulate), MPI_REPLACE (for atomic Put via Accumulate) and > MPI_SUM (for add, fadd, inc and finc). So what is the benefit of > same_op_no_op to SHMEM? Perhaps I remember completely wrong and > the motivation was something that does not use the latter atomics. Or > perhaps it is common for SHMEM codes to not use these and thus the > assumption is MPI_SUM can be ignored. > > Jeff > > -- > Jeff Hammond > jeff.science(a)gmail.com > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] same_op_no_op and SHMEM
by Pavan Balaji 24 Oct '13

24 Oct '13

The motivation was that it’s hard to maintain atomicity when different operations are done. For example, if the hardware only supports some atomic operations, but not all, this might result in some operations happening in hardware and some in software making atomicity hard. In such cases, the MPI implementation might need to fall back to software-only implementations. -— Pavan On Oct 24, 2013, at 12:23 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > I recall that Brian and/or Keith wanted same_op_no_op because of > SHMEM. However, SHMEM requires the use of MPI_NO_OP (for atomic Get > via Get_accumulate), MPI_REPLACE (for atomic Put via Accumulate) and > MPI_SUM (for add, fadd, inc and finc). So what is the benefit of > same_op_no_op to SHMEM? Perhaps I remember completely wrong and the > motivation was something that does not use the latter atomics. Or > perhaps it is common for SHMEM codes to not use these and thus the > assumption is MPI_SUM can be ignored. > > Jeff > > -- > Jeff Hammond > jeff.science(a)gmail.com > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

same_op_no_op and SHMEM
by Jeff Hammond 24 Oct '13

24 Oct '13

I recall that Brian and/or Keith wanted same_op_no_op because of SHMEM. However, SHMEM requires the use of MPI_NO_OP (for atomic Get via Get_accumulate), MPI_REPLACE (for atomic Put via Accumulate) and MPI_SUM (for add, fadd, inc and finc). So what is the benefit of same_op_no_op to SHMEM? Perhaps I remember completely wrong and the motivation was something that does not use the latter atomics. Or perhaps it is common for SHMEM codes to not use these and thus the assumption is MPI_SUM can be ignored. Jeff -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] shared-like access within a node with non-shared windows
by Jeff Hammond 22 Oct '13

22 Oct '13

https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/397 On Tue, Oct 22, 2013 at 12:38 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > Yes, please go ahead. It seems like we should schedule a WG meeting some > time soon, to organize ourselves for December. > > ~Jim. > > > On Tue, Oct 22, 2013 at 11:22 AM, Jeff Hammond <jeff.science(a)gmail.com> > wrote: >> >> Are we ready to make a ticket for this? Seems like we have mostly >> converged. >> >> Jeff >> >> On Fri, Oct 18, 2013 at 6:07 PM, Jeff Hammond <jeff.science(a)gmail.com> >> wrote: >> > I'm okay with that but it has to be clear that this is only the size >> > of the memory accessible by load-store and not the actual size of the >> > window at that rank. >> > >> > Jeff >> > >> > On Fri, Oct 18, 2013 at 5:40 PM, Jim Dinan <james.dinan(a)gmail.com> >> > wrote: >> >> You can return a size of 0. >> >> >> >> Jim. >> >> >> >> On Oct 18, 2013 5:48 PM, "Jeff Hammond" <jeff.science(a)gmail.com> wrote: >> >>> >> >>> Win shared query always returns a valid base for ranks in MPI comm >> >>> self, does it not? >> >>> >> >>> I don't like the error code approach. Can't we make a magic value >> >>> mpi_fortran_sucks_null? >> >>> >> >>> Sent from my iPhone >> >>> >> >>> On Oct 18, 2013, at 4:40 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >> >>> >> >>> > >> >>> > Ok, can I rephrase what you want as follows -- >> >>> > >> >>> > MPI_WIN_ALLOCATE(info = gimme_shared_memory) >> >>> > >> >>> > This will return a window, where you *might* be able to do direct >> >>> > load/store to some of the remote process address spaces (let's call >> >>> > this >> >>> > "direct access memory"). >> >>> > >> >>> > MPI_WIN_SHARED_QUERY will tell the user, through an appropriate >> >>> > error >> >>> > code, as to whether a given remote process gives you direct access >> >>> > memory. >> >>> > >> >>> > An MPI implementation is allowed to ignore the info argument, in >> >>> > which >> >>> > case, it will give an error for MPI_WIN_SHARED_QUERY for all target >> >>> > processes. >> >>> > >> >>> > Does that sound right? >> >>> > >> >>> > I guess the benefit of this compared to MPI_WIN_ALLOCATE_SHARED + >> >>> > MPI_WIN_CREATE is that the MPI implementation can better allocate >> >>> > memory. >> >>> > For example, it might create symmetric address space across nodes >> >>> > and use >> >>> > shared memory within each node. This is particularly useful when >> >>> > the >> >>> > allocation sizes on all processes is the same. >> >>> > >> >>> > -- Pavan >> >>> > >> >>> > On Oct 18, 2013, at 4:31 PM, Jeff Hammond wrote: >> >>> > >> >>> >> I believe that my assumptions about address translation are >> >>> >> appropriately conservative given the wide portability goals of MPI. >> >>> >> Assuming >> >>> >> Cray Gemini or similar isn't reasonable. >> >>> >> >> >>> >> Exclusive lock isn't my primary interest but I'll pay the >> >>> >> associated >> >>> >> costs as necessary. >> >>> >> >> >>> >> As discussed with Brian, this is a hint via info. Implementations >> >>> >> can >> >>> >> skip it if they want. I merely want us to standardize the expanded >> >>> >> use of >> >>> >> shared_query that allows this to work. >> >>> >> >> >>> >> Jeff >> >>> >> >> >>> >> Sent from my iPhone >> >>> >> >> >>> >> On Oct 18, 2013, at 4:25 PM, Jim Dinan <james.dinan(a)gmail.com> >> >>> >> wrote: >> >>> >> >> >>> >>> This is only correct if you assume that the remote NIC can't >> >>> >>> translate >> >>> >>> a displacement. >> >>> >>> >> >>> >>> Allowing all processes on the node direct access to the window >> >>> >>> buffer >> >>> >>> will require us to perform memory barriers in window >> >>> >>> synchronizations, and >> >>> >>> it will cause all lock operations that target the same node to >> >>> >>> block. I >> >>> >>> understand the value in providing this usage model, but what will >> >>> >>> be the >> >>> >>> performance cost? >> >>> >>> >> >>> >>> ~Jim. >> >>> >>> >> >>> >>> >> >>> >>> On Fri, Oct 18, 2013 at 4:50 PM, Jeff Hammond >> >>> >>> <jeff.science(a)gmail.com> >> >>> >>> wrote: >> >>> >>> It is impossible to so O(1) state w create unless your force the >> >>> >>> remote side to do all the translation and thus it precludes RDMA. >> >>> >>> If you >> >>> >>> want RDMA impl, create require O(P) state. Allocate does not >> >>> >>> require it >> >>> >>> >> >>> >>> I believe all of this was thoroughly discussed when we proposed >> >>> >>> allocate. >> >>> >>> >> >>> >>> Sent from my iPhone >> >>> >>> >> >>> >>> On Oct 18, 2013, at 3:16 PM, Jim Dinan <james.dinan(a)gmail.com> >> >>> >>> wrote: >> >>> >>> >> >>> >>>> Why is MPI_Win_create not scalable? There are certainly >> >>> >>>> implementations and use cases (e.g. not using different >> >>> >>>> disp_units) that can >> >>> >>>> avoid O(P) metadata per process. It's more likely that >> >>> >>>> MPI_Win_allocate can >> >>> >>>> avoid these in more cases, but it's not guaranteed. It seems >> >>> >>>> like an >> >>> >>>> implementation could leverage the Win_allocate_share/Win_create >> >>> >>>> combo to >> >>> >>>> achieve the same scaling result as MPI_Win_allocate. >> >>> >>>> >> >>> >>>> If we allow other window types to return pointers through >> >>> >>>> win_shared_query, then we will have to perform memory barriers in >> >>> >>>> all of the >> >>> >>>> RMA synchronization routines all the time. >> >>> >>>> >> >>> >>>> ~Jim. >> >>> >>>> >> >>> >>>> >> >>> >>>> On Fri, Oct 18, 2013 at 3:39 PM, Jeff Hammond >> >>> >>>> <jeff.science(a)gmail.com> wrote: >> >>> >>>> Yes, as I said, that's all I can do right now. But >> >>> >>>> MPI_WIN_CREATE is >> >>> >>>> not scalable. And it requires two windows instead of one. >> >>> >>>> >> >>> >>>> Brian, Pavan and Xin all seem to agree that this is >> >>> >>>> straightforward >> >>> >>>> to >> >>> >>>> implement as an optional feature. We just need to figure out how >> >>> >>>> to >> >>> >>>> extend the use of MPI_WIN_SHARED_QUERY to enable it. >> >>> >>>> >> >>> >>>> Jeff >> >>> >>>> >> >>> >>>> On Fri, Oct 18, 2013 at 2:35 PM, Jim Dinan >> >>> >>>> <james.dinan(a)gmail.com> >> >>> >>>> wrote: >> >>> >>>>> Jeff, >> >>> >>>>> >> >>> >>>>> Sorry, I haven't read the whole thread closely, so please ignore >> >>> >>>>> me >> >>> >>>>> if this >> >>> >>>>> is nonsense. Can you get what you want by doing >> >>> >>>>> MPI_Win_allocate_shared() >> >>> >>>>> to create an intranode window, and then pass the buffer >> >>> >>>>> allocated by >> >>> >>>>> MPI_Win_allocate_shared to MPI_Win_create() to create an >> >>> >>>>> internode >> >>> >>>>> window? >> >>> >>>>> >> >>> >>>>> ~Jim. >> >>> >>>>> >> >>> >>>>> >> >>> >>>>> On Sat, Oct 12, 2013 at 3:49 PM, Jeff Hammond >> >>> >>>>> <jeff.science(a)gmail.com> >> >>> >>>>> wrote: >> >>> >>>>>> >> >>> >>>>>> Pavan told me that (in MPICH) MPI_Win_allocate is way better >> >>> >>>>>> than >> >>> >>>>>> MPI_Win_create because the former allocates the shared memory >> >>> >>>>>> business. It was implied that the latter requires more work >> >>> >>>>>> within >> >>> >>>>>> the node. (I thought mmap could do the same magic on existing >> >>> >>>>>> allocations, but that's not really the point here.) >> >>> >>>>>> >> >>> >>>>>> But within a node, what's even better than a window allocated >> >>> >>>>>> with >> >>> >>>>>> MPI_Win_allocate is a window allowed with >> >>> >>>>>> MPI_Win_allocate_shared, >> >>> >>>>>> since the latter permits load-store. Then I wondered if it >> >>> >>>>>> would >> >>> >>>>>> be >> >>> >>>>>> possible to have both (1) direct load-store access within a >> >>> >>>>>> node >> >>> >>>>>> and >> >>> >>>>>> (2) scalable metadata for windows spanning many nodes. >> >>> >>>>>> >> >>> >>>>>> I can get (1) but not (2) by using MPI_Win_allocate_shared and >> >>> >>>>>> then >> >>> >>>>>> dropping a second window for the internode part on top of these >> >>> >>>>>> using >> >>> >>>>>> MPI_Win_create. Of course, I can get (2) but not (1) using >> >>> >>>>>> MPI_Win_allocate. >> >>> >>>>>> >> >>> >>>>>> I propose that it be possible to get (1) and (2) by allowing >> >>> >>>>>> MPI_Win_shared_query to return pointers to shared memory within >> >>> >>>>>> a >> >>> >>>>>> node >> >>> >>>>>> even if the window has >> >>> >>>>>> MPI_WIN_CREATE_FLAVOR=MPI_WIN_FLAVOR_ALLOCATE. >> >>> >>>>>> When the input argument rank to MPI_Win_shared_query >> >>> >>>>>> corresponds to >> >>> >>>>>> memory that is not accessible by load-store, the out arguments >> >>> >>>>>> size >> >>> >>>>>> and baseptr are 0 and NULL, respectively. >> >>> >>>>>> >> >>> >>>>>> The non-scalable use of this feature would be to loop over all >> >>> >>>>>> ranks >> >>> >>>>>> in the group associated with the window and test for >> >>> >>>>>> baseptr!=NULL, >> >>> >>>>>> while the scalable use would presumably utilize >> >>> >>>>>> MPI_Comm_split_type, >> >>> >>>>>> MPI_Comm_group and MPI_Group_translate_ranks to determine the >> >>> >>>>>> list >> >>> >>>>>> of >> >>> >>>>>> ranks corresponding to the node, hence the ones that might >> >>> >>>>>> permit >> >>> >>>>>> direct access. >> >>> >>>>>> >> >>> >>>>>> Comments are appreciate. >> >>> >>>>>> >> >>> >>>>>> Jeff >> >>> >>>>>> >> >>> >>>>>> -- >> >>> >>>>>> Jeff Hammond >> >>> >>>>>> jeff.science(a)gmail.com >> >>> >>>>>> _______________________________________________ >> >>> >>>>>> mpiwg-rma mailing list >> >>> >>>>>> mpiwg-rma(a)lists.mpi-forum.org >> >>> >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>> >>>>> >> >>> >>>>> >> >>> >>>>> >> >>> >>>>> _______________________________________________ >> >>> >>>>> mpiwg-rma mailing list >> >>> >>>>> mpiwg-rma(a)lists.mpi-forum.org >> >>> >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>> >>>> >> >>> >>>> >> >>> >>>> >> >>> >>>> -- >> >>> >>>> Jeff Hammond >> >>> >>>> jeff.science(a)gmail.com >> >>> >>>> _______________________________________________ >> >>> >>>> mpiwg-rma mailing list >> >>> >>>> mpiwg-rma(a)lists.mpi-forum.org >> >>> >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>> >>>> >> >>> >>>> _______________________________________________ >> >>> >>>> mpiwg-rma mailing list >> >>> >>>> mpiwg-rma(a)lists.mpi-forum.org >> >>> >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>> >>> >> >>> >>> _______________________________________________ >> >>> >>> mpiwg-rma mailing list >> >>> >>> mpiwg-rma(a)lists.mpi-forum.org >> >>> >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>> >>> >> >>> >>> _______________________________________________ >> >>> >>> mpiwg-rma mailing list >> >>> >>> mpiwg-rma(a)lists.mpi-forum.org >> >>> >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>> >> _______________________________________________ >> >>> >> mpiwg-rma mailing list >> >>> >> mpiwg-rma(a)lists.mpi-forum.org >> >>> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>> > >> >>> > -- >> >>> > Pavan Balaji >> >>> > http://www.mcs.anl.gov/~balaji >> >>> > >> >>> > _______________________________________________ >> >>> > mpiwg-rma mailing list >> >>> > mpiwg-rma(a)lists.mpi-forum.org >> >>> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >>> _______________________________________________ >> >>> mpiwg-rma mailing list >> >>> mpiwg-rma(a)lists.mpi-forum.org >> >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> >> >> >> >> _______________________________________________ >> >> mpiwg-rma mailing list >> >> mpiwg-rma(a)lists.mpi-forum.org >> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> > >> > >> > >> > -- >> > Jeff Hammond >> > jeff.science(a)gmail.com >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] shared-like access within a node with non-shared windows
by Jim Dinan 22 Oct '13

22 Oct '13

Yes, please go ahead. It seems like we should schedule a WG meeting some time soon, to organize ourselves for December. ~Jim. On Tue, Oct 22, 2013 at 11:22 AM, Jeff Hammond <jeff.science(a)gmail.com>wrote: > Are we ready to make a ticket for this? Seems like we have mostly > converged. > > Jeff > > On Fri, Oct 18, 2013 at 6:07 PM, Jeff Hammond <jeff.science(a)gmail.com> > wrote: > > I'm okay with that but it has to be clear that this is only the size > > of the memory accessible by load-store and not the actual size of the > > window at that rank. > > > > Jeff > > > > On Fri, Oct 18, 2013 at 5:40 PM, Jim Dinan <james.dinan(a)gmail.com> > wrote: > >> You can return a size of 0. > >> > >> Jim. > >> > >> On Oct 18, 2013 5:48 PM, "Jeff Hammond" <jeff.science(a)gmail.com> wrote: > >>> > >>> Win shared query always returns a valid base for ranks in MPI comm > >>> self, does it not? > >>> > >>> I don't like the error code approach. Can't we make a magic value > >>> mpi_fortran_sucks_null? > >>> > >>> Sent from my iPhone > >>> > >>> On Oct 18, 2013, at 4:40 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > >>> > >>> > > >>> > Ok, can I rephrase what you want as follows -- > >>> > > >>> > MPI_WIN_ALLOCATE(info = gimme_shared_memory) > >>> > > >>> > This will return a window, where you *might* be able to do direct > >>> > load/store to some of the remote process address spaces (let's call > this > >>> > "direct access memory"). > >>> > > >>> > MPI_WIN_SHARED_QUERY will tell the user, through an appropriate error > >>> > code, as to whether a given remote process gives you direct access > memory. > >>> > > >>> > An MPI implementation is allowed to ignore the info argument, in > which > >>> > case, it will give an error for MPI_WIN_SHARED_QUERY for all target > >>> > processes. > >>> > > >>> > Does that sound right? > >>> > > >>> > I guess the benefit of this compared to MPI_WIN_ALLOCATE_SHARED + > >>> > MPI_WIN_CREATE is that the MPI implementation can better allocate > memory. > >>> > For example, it might create symmetric address space across nodes > and use > >>> > shared memory within each node. This is particularly useful when the > >>> > allocation sizes on all processes is the same. > >>> > > >>> > -- Pavan > >>> > > >>> > On Oct 18, 2013, at 4:31 PM, Jeff Hammond wrote: > >>> > > >>> >> I believe that my assumptions about address translation are > >>> >> appropriately conservative given the wide portability goals of MPI. > Assuming > >>> >> Cray Gemini or similar isn't reasonable. > >>> >> > >>> >> Exclusive lock isn't my primary interest but I'll pay the associated > >>> >> costs as necessary. > >>> >> > >>> >> As discussed with Brian, this is a hint via info. Implementations > can > >>> >> skip it if they want. I merely want us to standardize the expanded > use of > >>> >> shared_query that allows this to work. > >>> >> > >>> >> Jeff > >>> >> > >>> >> Sent from my iPhone > >>> >> > >>> >> On Oct 18, 2013, at 4:25 PM, Jim Dinan <james.dinan(a)gmail.com> > wrote: > >>> >> > >>> >>> This is only correct if you assume that the remote NIC can't > translate > >>> >>> a displacement. > >>> >>> > >>> >>> Allowing all processes on the node direct access to the window > buffer > >>> >>> will require us to perform memory barriers in window > synchronizations, and > >>> >>> it will cause all lock operations that target the same node to > block. I > >>> >>> understand the value in providing this usage model, but what will > be the > >>> >>> performance cost? > >>> >>> > >>> >>> ~Jim. > >>> >>> > >>> >>> > >>> >>> On Fri, Oct 18, 2013 at 4:50 PM, Jeff Hammond < > jeff.science(a)gmail.com> > >>> >>> wrote: > >>> >>> It is impossible to so O(1) state w create unless your force the > >>> >>> remote side to do all the translation and thus it precludes RDMA. > If you > >>> >>> want RDMA impl, create require O(P) state. Allocate does not > require it > >>> >>> > >>> >>> I believe all of this was thoroughly discussed when we proposed > >>> >>> allocate. > >>> >>> > >>> >>> Sent from my iPhone > >>> >>> > >>> >>> On Oct 18, 2013, at 3:16 PM, Jim Dinan <james.dinan(a)gmail.com> > wrote: > >>> >>> > >>> >>>> Why is MPI_Win_create not scalable? There are certainly > >>> >>>> implementations and use cases (e.g. not using different > disp_units) that can > >>> >>>> avoid O(P) metadata per process. It's more likely that > MPI_Win_allocate can > >>> >>>> avoid these in more cases, but it's not guaranteed. It seems > like an > >>> >>>> implementation could leverage the Win_allocate_share/Win_create > combo to > >>> >>>> achieve the same scaling result as MPI_Win_allocate. > >>> >>>> > >>> >>>> If we allow other window types to return pointers through > >>> >>>> win_shared_query, then we will have to perform memory barriers in > all of the > >>> >>>> RMA synchronization routines all the time. > >>> >>>> > >>> >>>> ~Jim. > >>> >>>> > >>> >>>> > >>> >>>> On Fri, Oct 18, 2013 at 3:39 PM, Jeff Hammond > >>> >>>> <jeff.science(a)gmail.com> wrote: > >>> >>>> Yes, as I said, that's all I can do right now. But > MPI_WIN_CREATE is > >>> >>>> not scalable. And it requires two windows instead of one. > >>> >>>> > >>> >>>> Brian, Pavan and Xin all seem to agree that this is > straightforward > >>> >>>> to > >>> >>>> implement as an optional feature. We just need to figure out how > to > >>> >>>> extend the use of MPI_WIN_SHARED_QUERY to enable it. > >>> >>>> > >>> >>>> Jeff > >>> >>>> > >>> >>>> On Fri, Oct 18, 2013 at 2:35 PM, Jim Dinan <james.dinan(a)gmail.com > > > >>> >>>> wrote: > >>> >>>>> Jeff, > >>> >>>>> > >>> >>>>> Sorry, I haven't read the whole thread closely, so please ignore > me > >>> >>>>> if this > >>> >>>>> is nonsense. Can you get what you want by doing > >>> >>>>> MPI_Win_allocate_shared() > >>> >>>>> to create an intranode window, and then pass the buffer > allocated by > >>> >>>>> MPI_Win_allocate_shared to MPI_Win_create() to create an > internode > >>> >>>>> window? > >>> >>>>> > >>> >>>>> ~Jim. > >>> >>>>> > >>> >>>>> > >>> >>>>> On Sat, Oct 12, 2013 at 3:49 PM, Jeff Hammond > >>> >>>>> <jeff.science(a)gmail.com> > >>> >>>>> wrote: > >>> >>>>>> > >>> >>>>>> Pavan told me that (in MPICH) MPI_Win_allocate is way better > than > >>> >>>>>> MPI_Win_create because the former allocates the shared memory > >>> >>>>>> business. It was implied that the latter requires more work > within > >>> >>>>>> the node. (I thought mmap could do the same magic on existing > >>> >>>>>> allocations, but that's not really the point here.) > >>> >>>>>> > >>> >>>>>> But within a node, what's even better than a window allocated > with > >>> >>>>>> MPI_Win_allocate is a window allowed with > MPI_Win_allocate_shared, > >>> >>>>>> since the latter permits load-store. Then I wondered if it > would > >>> >>>>>> be > >>> >>>>>> possible to have both (1) direct load-store access within a node > >>> >>>>>> and > >>> >>>>>> (2) scalable metadata for windows spanning many nodes. > >>> >>>>>> > >>> >>>>>> I can get (1) but not (2) by using MPI_Win_allocate_shared and > then > >>> >>>>>> dropping a second window for the internode part on top of these > >>> >>>>>> using > >>> >>>>>> MPI_Win_create. Of course, I can get (2) but not (1) using > >>> >>>>>> MPI_Win_allocate. > >>> >>>>>> > >>> >>>>>> I propose that it be possible to get (1) and (2) by allowing > >>> >>>>>> MPI_Win_shared_query to return pointers to shared memory within > a > >>> >>>>>> node > >>> >>>>>> even if the window has > >>> >>>>>> MPI_WIN_CREATE_FLAVOR=MPI_WIN_FLAVOR_ALLOCATE. > >>> >>>>>> When the input argument rank to MPI_Win_shared_query > corresponds to > >>> >>>>>> memory that is not accessible by load-store, the out arguments > size > >>> >>>>>> and baseptr are 0 and NULL, respectively. > >>> >>>>>> > >>> >>>>>> The non-scalable use of this feature would be to loop over all > >>> >>>>>> ranks > >>> >>>>>> in the group associated with the window and test for > baseptr!=NULL, > >>> >>>>>> while the scalable use would presumably utilize > >>> >>>>>> MPI_Comm_split_type, > >>> >>>>>> MPI_Comm_group and MPI_Group_translate_ranks to determine the > list > >>> >>>>>> of > >>> >>>>>> ranks corresponding to the node, hence the ones that might > permit > >>> >>>>>> direct access. > >>> >>>>>> > >>> >>>>>> Comments are appreciate. > >>> >>>>>> > >>> >>>>>> Jeff > >>> >>>>>> > >>> >>>>>> -- > >>> >>>>>> Jeff Hammond > >>> >>>>>> jeff.science(a)gmail.com > >>> >>>>>> _______________________________________________ > >>> >>>>>> mpiwg-rma mailing list > >>> >>>>>> mpiwg-rma(a)lists.mpi-forum.org > >>> >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>> >>>>> > >>> >>>>> > >>> >>>>> > >>> >>>>> _______________________________________________ > >>> >>>>> mpiwg-rma mailing list > >>> >>>>> mpiwg-rma(a)lists.mpi-forum.org > >>> >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>> >>>> > >>> >>>> > >>> >>>> > >>> >>>> -- > >>> >>>> Jeff Hammond > >>> >>>> jeff.science(a)gmail.com > >>> >>>> _______________________________________________ > >>> >>>> mpiwg-rma mailing list > >>> >>>> mpiwg-rma(a)lists.mpi-forum.org > >>> >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>> >>>> > >>> >>>> _______________________________________________ > >>> >>>> mpiwg-rma mailing list > >>> >>>> mpiwg-rma(a)lists.mpi-forum.org > >>> >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>> >>> > >>> >>> _______________________________________________ > >>> >>> mpiwg-rma mailing list > >>> >>> mpiwg-rma(a)lists.mpi-forum.org > >>> >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>> >>> > >>> >>> _______________________________________________ > >>> >>> mpiwg-rma mailing list > >>> >>> mpiwg-rma(a)lists.mpi-forum.org > >>> >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>> >> _______________________________________________ > >>> >> mpiwg-rma mailing list > >>> >> mpiwg-rma(a)lists.mpi-forum.org > >>> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>> > > >>> > -- > >>> > Pavan Balaji > >>> > http://www.mcs.anl.gov/~balaji > >>> > > >>> > _______________________________________________ > >>> > mpiwg-rma mailing list > >>> > mpiwg-rma(a)lists.mpi-forum.org > >>> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >>> _______________________________________________ > >>> mpiwg-rma mailing list > >>> mpiwg-rma(a)lists.mpi-forum.org > >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >> > >> > >> _______________________________________________ > >> mpiwg-rma mailing list > >> mpiwg-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > > > > > -- > > Jeff Hammond > > jeff.science(a)gmail.com > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >

1 0