November 2013 - mpiwg-rma - mailman3.cels.anl.gov

Re: [mpiwg-rma] schedule telecon
by Pavan Balaji 16 Nov '13

16 Nov '13

Sorry to keep adding more items, but one more item that needs discussion is the statement in the MPI-3 standard that could potentially disallow loads and PUTs to nonoverlapping locations in SEPARATE windows. Specifically, see #3 on page 455. “A put or accumulate must not access a target window once a load/store update or a put or accumulate update to another (overlapping) target window has started on a location in the target window, until the update becomes visible in the public copy of the window." The first line should either say “load/store access” or “store updates”. “load/store updates” doesn’t make sense. Now which of the two it is, is unclear. Is load + PUTs valid? This will be fine for hardware managed caches (even noncoherent ones), but not for software managed caches. One way to decouple this issue is to add more memory models — one that allows this and another that disallows this. IMO, there should be 4 different memory models that cover all cases: 1. MPI_WIN_SUCKY_MEMORY — for software managed cache type of systems. Even load + PUTs to nonoverlapping memory regions is not valid. 2. MPI_WIN_SEPARATE — current SEPARATE memory model, but we allow load + PUTs. This will work as long as the cache is hardware managed. 3. MPI_WIN_UNIFIED — this is what we decided on for our UNIFIED model. WIN_SYNC is required for overlapping memory regions. Windows are not always and immediately unified. They’ll eventually become unified without MPI calls. 4. MPI_WIN_SUPERUNIFIED — no WIN_SYNC required. Application can assume that the windows are truly unified. This works on x86 type of architectures. Of course, all this will be heavily debated. I’m just queuing up the discussion item. :-) — Pavan On Nov 4, 2013, at 1:55 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > 369 seems trivial. Can we just enumerate the changes on the ticket > and vote yes/no on moving forward with a reading in December on the > telecon? > > Jeff > > On Mon, Nov 4, 2013 at 12:51 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: >> We also need to make sure that #349 is ready for a reading in December: >> >> MPI_Aint addressing arithmetic: >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/349 >> >> I think the main todo here is to update the dynamic window examples. I also >> have another open ticket that we should move forward: >> >> RMA needs same_disp info key: >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/369 >> >> ~Jim. >> >> >> On Sat, Nov 2, 2013 at 8:40 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>> >>> Yeah, I'll setup a doodle but I wanted to confirm the scope. Since >>> Pavan greatly expanded the possible scope, I will propose a doodle >>> that is oriented at discovering more than one time-slot since I >>> suspect we will need that. >>> >>> Jeff >>> >>> On Sat, Nov 2, 2013 at 5:04 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: >>>> OK, when?. Do you want to set up a Doodle poll or something? >>>> >>>> Rajeev >>>> >>>> On Nov 2, 2013, at 4:12 PM, Jeff Hammond wrote: >>>> >>>>> I would like to schedule a telecon for the RMA WG to discuss the >>>>> following three tickets. We've discussed them all over email but I >>>>> want to get more conclusive feedback before the Chicago meeting. >>>>> >>>>> - extend the use of MPI_WIN_SHARED_QUERY to all windows >>>>> (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/397) >>>>> - request-based remote completion for RMA >>>>> (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/398) >>>>> - generalize same_op_no_op and allow user to specify all ops to be >>>>> used (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/399) >>>>> >>>>> Thanks, >>>>> >>>>> Jeff >>>>> >>>>> -- >>>>> Jeff Hammond >>>>> jeff.science(a)gmail.com >>>>> _______________________________________________ >>>>> mpiwg-rma mailing list >>>>> mpiwg-rma(a)lists.mpi-forum.org >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>>> >>>> _______________________________________________ >>>> mpiwg-rma mailing list >>>> mpiwg-rma(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >>> >>> >>> >>> -- >>> Jeff Hammond >>> jeff.science(a)gmail.com >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

nextgen rma on aries/ib
by maik peterson 15 Nov '13

15 Nov '13

at the sc13 we present our new driver and sw stack for x86 systems. it is by far the fastest communication stack for rma operations on x86 today with up to 25gb/s bw and around 820ns latency. most mpi based appl. can be speedup by an order of magnitude (if communication intensive). come by to see demos at intel, cray and mellanox booth. mp

1 0

Re: [mpiwg-rma] schedule telecon
by Jeff Hammond 04 Nov '13

04 Nov '13

369 seems trivial. Can we just enumerate the changes on the ticket and vote yes/no on moving forward with a reading in December on the telecon? Jeff On Mon, Nov 4, 2013 at 12:51 PM, Jim Dinan <james.dinan(a)gmail.com> wrote: > We also need to make sure that #349 is ready for a reading in December: > > MPI_Aint addressing arithmetic: > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/349 > > I think the main todo here is to update the dynamic window examples. I also > have another open ticket that we should move forward: > > RMA needs same_disp info key: > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/369 > > ~Jim. > > > On Sat, Nov 2, 2013 at 8:40 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >> >> Yeah, I'll setup a doodle but I wanted to confirm the scope. Since >> Pavan greatly expanded the possible scope, I will propose a doodle >> that is oriented at discovering more than one time-slot since I >> suspect we will need that. >> >> Jeff >> >> On Sat, Nov 2, 2013 at 5:04 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: >> > OK, when?. Do you want to set up a Doodle poll or something? >> > >> > Rajeev >> > >> > On Nov 2, 2013, at 4:12 PM, Jeff Hammond wrote: >> > >> >> I would like to schedule a telecon for the RMA WG to discuss the >> >> following three tickets. We've discussed them all over email but I >> >> want to get more conclusive feedback before the Chicago meeting. >> >> >> >> - extend the use of MPI_WIN_SHARED_QUERY to all windows >> >> (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/397) >> >> - request-based remote completion for RMA >> >> (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/398) >> >> - generalize same_op_no_op and allow user to specify all ops to be >> >> used (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/399) >> >> >> >> Thanks, >> >> >> >> Jeff >> >> >> >> -- >> >> Jeff Hammond >> >> jeff.science(a)gmail.com >> >> _______________________________________________ >> >> mpiwg-rma mailing list >> >> mpiwg-rma(a)lists.mpi-forum.org >> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> > >> > _______________________________________________ >> > mpiwg-rma mailing list >> > mpiwg-rma(a)lists.mpi-forum.org >> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] schedule telecon
by Jim Dinan 04 Nov '13

04 Nov '13

We also need to make sure that #349 is ready for a reading in December: MPI_Aint addressing arithmetic: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/349 I think the main todo here is to update the dynamic window examples. I also have another open ticket that we should move forward: RMA needs same_disp info key: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/369 ~Jim. On Sat, Nov 2, 2013 at 8:40 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > Yeah, I'll setup a doodle but I wanted to confirm the scope. Since > Pavan greatly expanded the possible scope, I will propose a doodle > that is oriented at discovering more than one time-slot since I > suspect we will need that. > > Jeff > > On Sat, Nov 2, 2013 at 5:04 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > > OK, when?. Do you want to set up a Doodle poll or something? > > > > Rajeev > > > > On Nov 2, 2013, at 4:12 PM, Jeff Hammond wrote: > > > >> I would like to schedule a telecon for the RMA WG to discuss the > >> following three tickets. We've discussed them all over email but I > >> want to get more conclusive feedback before the Chicago meeting. > >> > >> - extend the use of MPI_WIN_SHARED_QUERY to all windows > >> (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/397) > >> - request-based remote completion for RMA > >> (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/398) > >> - generalize same_op_no_op and allow user to specify all ops to be > >> used (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/399) > >> > >> Thanks, > >> > >> Jeff > >> > >> -- > >> Jeff Hammond > >> jeff.science(a)gmail.com > >> _______________________________________________ > >> mpiwg-rma mailing list > >> mpiwg-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >

1 0

Re: [mpiwg-rma] schedule telecon
by Jeff Hammond 02 Nov '13

02 Nov '13

Yeah, I'll setup a doodle but I wanted to confirm the scope. Since Pavan greatly expanded the possible scope, I will propose a doodle that is oriented at discovering more than one time-slot since I suspect we will need that. Jeff On Sat, Nov 2, 2013 at 5:04 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > OK, when?. Do you want to set up a Doodle poll or something? > > Rajeev > > On Nov 2, 2013, at 4:12 PM, Jeff Hammond wrote: > >> I would like to schedule a telecon for the RMA WG to discuss the >> following three tickets. We've discussed them all over email but I >> want to get more conclusive feedback before the Chicago meeting. >> >> - extend the use of MPI_WIN_SHARED_QUERY to all windows >> (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/397) >> - request-based remote completion for RMA >> (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/398) >> - generalize same_op_no_op and allow user to specify all ops to be >> used (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/399) >> >> Thanks, >> >> Jeff >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] schedule telecon
by Jeff Hammond 02 Nov '13

02 Nov '13

So basically, we need a 3-4 hour telecon? :-) Jeff Sent from my iPhone On Nov 2, 2013, at 4:35 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > Some additional things to discuss: > > 1. non-interleaved Ops and attach/detach info argument —- if the application might interleave attach/detach with PUT/GET operations, the MPI implementation cannot cache things like the memory registration information for the target buffer. This info hint can help in such cases. > > 2. WIN_SYNC semantics in unified windows. We had some discussion on this over email, but this needs to be finalized. This is needed for ordering. I think the wording we agreed on was to say that the public/private windows are disjoint, but will eventually be consistent in UNIFIED. So for overlapping memory regions, UNIFIED and SEPARATE are not too different for portable programs. UNIFIED has richer capabilities for nonoverlapping memory regions. > > 3. I want to initiate some discussion on threads+RMA. We might need some discussion on this, as this would mean that each thread has its own private window. That is, load/store from one might not be visible in another without a WIN_SYNC. > > 4. I want to initiate some discussion on exposing memory barrier semantics in MPI. We are kind of half-exposing it with WIN_SYNC in UNIFIED. If we do this, I’d recommend that we expose WIN_SYNC_LOAD and WIN_SYNC_STORE as well for completeness. I expect this to be highly debated, of course; I just want to get this item on the agenda. > > —- Pavan > > On Nov 2, 2013, at 4:12 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > >> I would like to schedule a telecon for the RMA WG to discuss the >> following three tickets. We've discussed them all over email but I >> want to get more conclusive feedback before the Chicago meeting. >> >> - extend the use of MPI_WIN_SHARED_QUERY to all windows >> (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/397) >> - request-based remote completion for RMA >> (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/398) >> - generalize same_op_no_op and allow user to specify all ops to be >> used (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/399) >> >> Thanks, >> >> Jeff >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] schedule telecon
by Rajeev Thakur 02 Nov '13

02 Nov '13

OK, when?. Do you want to set up a Doodle poll or something? Rajeev On Nov 2, 2013, at 4:12 PM, Jeff Hammond wrote: > I would like to schedule a telecon for the RMA WG to discuss the > following three tickets. We've discussed them all over email but I > want to get more conclusive feedback before the Chicago meeting. > > - extend the use of MPI_WIN_SHARED_QUERY to all windows > (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/397) > - request-based remote completion for RMA > (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/398) > - generalize same_op_no_op and allow user to specify all ops to be > used (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/399) > > Thanks, > > Jeff > > -- > Jeff Hammond > jeff.science(a)gmail.com > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] schedule telecon
by Pavan Balaji 02 Nov '13

02 Nov '13

Some additional things to discuss: 1. non-interleaved Ops and attach/detach info argument —- if the application might interleave attach/detach with PUT/GET operations, the MPI implementation cannot cache things like the memory registration information for the target buffer. This info hint can help in such cases. 2. WIN_SYNC semantics in unified windows. We had some discussion on this over email, but this needs to be finalized. This is needed for ordering. I think the wording we agreed on was to say that the public/private windows are disjoint, but will eventually be consistent in UNIFIED. So for overlapping memory regions, UNIFIED and SEPARATE are not too different for portable programs. UNIFIED has richer capabilities for nonoverlapping memory regions. 3. I want to initiate some discussion on threads+RMA. We might need some discussion on this, as this would mean that each thread has its own private window. That is, load/store from one might not be visible in another without a WIN_SYNC. 4. I want to initiate some discussion on exposing memory barrier semantics in MPI. We are kind of half-exposing it with WIN_SYNC in UNIFIED. If we do this, I’d recommend that we expose WIN_SYNC_LOAD and WIN_SYNC_STORE as well for completeness. I expect this to be highly debated, of course; I just want to get this item on the agenda. —- Pavan On Nov 2, 2013, at 4:12 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > I would like to schedule a telecon for the RMA WG to discuss the > following three tickets. We've discussed them all over email but I > want to get more conclusive feedback before the Chicago meeting. > > - extend the use of MPI_WIN_SHARED_QUERY to all windows > (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/397) > - request-based remote completion for RMA > (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/398) > - generalize same_op_no_op and allow user to specify all ops to be > used (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/399) > > Thanks, > > Jeff > > -- > Jeff Hammond > jeff.science(a)gmail.com > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

schedule telecon
by Jeff Hammond 02 Nov '13

02 Nov '13

I would like to schedule a telecon for the RMA WG to discuss the following three tickets. We've discussed them all over email but I want to get more conclusive feedback before the Chicago meeting. - extend the use of MPI_WIN_SHARED_QUERY to all windows (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/397) - request-based remote completion for RMA (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/398) - generalize same_op_no_op and allow user to specify all ops to be used (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/399) Thanks, Jeff -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-rma] [EXTERNAL] Re: same_op_no_op and SHMEM
by Jeff Hammond 02 Nov '13

02 Nov '13

I created https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/399 to track this. Jeff On Thu, Oct 24, 2013 at 2:45 PM, Barrett, Brian W <bwbarre(a)sandia.gov> wrote: > On 10/24/13 1:24 PM, "Jeff Hammond" <jeff.science(a)gmail.com> wrote: > >>> I would have no objection to adding yet another info key. I think if we >>> keep at this for another year, we can make sure we have the longest >>> pre-defined info key in the spec. >> >>Or we can focus on the values. Instead of "same_op_no_op_replace = >>true" we can have "op_list = no_op,replace,sum" and let the >>implementation instantiate same_op_no_op_replace on its own. > > The advantage here would be that if the bad thing is FP divide, I can list > no_op,sum,replace,max,min and still have goodness. The disadvantage would > be all the test to list the options. So maybe long true/false keys have > some advantage. > >>> I admit to having very little medium term memory; which is the >>> type-homogeneity suggestion? >> >>Medium-term meaning in the last 2 hours? :-) > > Fine, no memory at all. > >>It addresses the lack of type-specification in MPI window creating. >>If I can say at window allocation time that I only want to use >>MPI_INT, then maybe the implementation can avoid a software >>implementation that might be required if I was to use MPI_FLOAT, for >>example. In my experience, NICs are more likely to support >>fixed-point arithmetic in hardware than floating-point. > > As an info key? I think that would be helpful for exactly the reason you > said. If I know the user isn't going to use long double complex data > types, that makes life a bit easier :). > > Brian > > -- > Brian W. Barrett > Scalable System Software Group > Sandia National Laboratories > > > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com

1 0