October 2014 - mpiwg-rma - mailman3.cels.anl.gov

Re: [mpiwg-rma] Proposed Info Key
by Balaji, Pavan 14 Oct '14

14 Oct '14

Sorry, I was clearly smoking something, so both my previous statements were incorrect: 1. Current mpich/master does check this unconditionally, but we are trying to make it conditional on an info key: alloc_shared. The patch will be in soon. 2. Converting it to an AVL tree might not be as easy as I thought initially. We do a bunch of Group_translate_ranks and Allgathers. I’ll need to think more carefully to see if it’ll be possible to do this in an inexpensive way to avoid the info key altogether. — Pavan On Oct 14, 2014, at 11:06 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > If we can optimize the search with an AVL tree, this info might be unnecessary. It’d be cheap enough that we’d do it everytime anyway. > > — Pavan > > On Oct 14, 2014, at 10:58 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > >> Thanks, Pavan, for sharing the details. I forgot about the MPICH info >> key used here. >> >> Now that I understand the issue, I think we should pursue both >> tickets, especially since my ticket is definitely 4.0 material. >> Standardizing the info key that makes the MPICH solution portable in >> the 3.1 timeframe is a good idea. >> >> And I should note that this is not only an academic problem - there >> are a number of projects making use of the (ppn+1)-windows trick >> already and will benefit from the info key Mikhail proposed. >> >> Jeff >> >> On Tue, Oct 14, 2014 at 8:54 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: >>> >>> On Oct 14, 2014, at 10:43 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: >>>> IIUC, MPICH has started detecting this automatically without the info key, so this ticket may be moot. Hopefully an MPICH developer familiar with this optimization can comment. >>> >>> Yes, but we rely on an internal info key as well in order to avoid the search time. Though I think we can optimize this search with an AVL tree or something. >>> >>> Note that Jeff’s ticket gives more capability that this — it allows the user to do load/store accesses. Your info key only allows the MPI implementation to internally optimize communication. >>> >>> — Pavan >>> >>> -- >>> Pavan Balaji ✉️ >>> http://www.mcs.anl.gov/~balaji >>> _______________________________________________ >>> mpiwg-rma mailing list >>> mpiwg-rma(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> http://jeffhammond.github.io/ >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > -- > Pavan Balaji ✉️ > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Pavan Balaji ✉️ http://www.mcs.anl.gov/~balaji

1 0

Re: [mpiwg-rma] Proposed Info Key
by Balaji, Pavan 14 Oct '14

14 Oct '14

If we can optimize the search with an AVL tree, this info might be unnecessary. It’d be cheap enough that we’d do it everytime anyway. — Pavan On Oct 14, 2014, at 10:58 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > Thanks, Pavan, for sharing the details. I forgot about the MPICH info > key used here. > > Now that I understand the issue, I think we should pursue both > tickets, especially since my ticket is definitely 4.0 material. > Standardizing the info key that makes the MPICH solution portable in > the 3.1 timeframe is a good idea. > > And I should note that this is not only an academic problem - there > are a number of projects making use of the (ppn+1)-windows trick > already and will benefit from the info key Mikhail proposed. > > Jeff > > On Tue, Oct 14, 2014 at 8:54 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: >> >> On Oct 14, 2014, at 10:43 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: >>> IIUC, MPICH has started detecting this automatically without the info key, so this ticket may be moot. Hopefully an MPICH developer familiar with this optimization can comment. >> >> Yes, but we rely on an internal info key as well in order to avoid the search time. Though I think we can optimize this search with an AVL tree or something. >> >> Note that Jeff’s ticket gives more capability that this — it allows the user to do load/store accesses. Your info key only allows the MPI implementation to internally optimize communication. >> >> — Pavan >> >> -- >> Pavan Balaji ✉️ >> http://www.mcs.anl.gov/~balaji >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Pavan Balaji ✉️ http://www.mcs.anl.gov/~balaji

1 0

Re: [mpiwg-rma] Proposed Info Key
by Jeff Hammond 14 Oct '14

14 Oct '14

Thanks, Pavan, for sharing the details. I forgot about the MPICH info key used here. Now that I understand the issue, I think we should pursue both tickets, especially since my ticket is definitely 4.0 material. Standardizing the info key that makes the MPICH solution portable in the 3.1 timeframe is a good idea. And I should note that this is not only an academic problem - there are a number of projects making use of the (ppn+1)-windows trick already and will benefit from the info key Mikhail proposed. Jeff On Tue, Oct 14, 2014 at 8:54 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > On Oct 14, 2014, at 10:43 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: >> IIUC, MPICH has started detecting this automatically without the info key, so this ticket may be moot. Hopefully an MPICH developer familiar with this optimization can comment. > > Yes, but we rely on an internal info key as well in order to avoid the search time. Though I think we can optimize this search with an AVL tree or something. > > Note that Jeff’s ticket gives more capability that this — it allows the user to do load/store accesses. Your info key only allows the MPI implementation to internally optimize communication. > > — Pavan > > -- > Pavan Balaji ✉️ > http://www.mcs.anl.gov/~balaji > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpiwg-rma] Proposed Info Key
by Balaji, Pavan 14 Oct '14

14 Oct '14

On Oct 14, 2014, at 10:43 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: > IIUC, MPICH has started detecting this automatically without the info key, so this ticket may be moot. Hopefully an MPICH developer familiar with this optimization can comment. Yes, but we rely on an internal info key as well in order to avoid the search time. Though I think we can optimize this search with an AVL tree or something. Note that Jeff’s ticket gives more capability that this — it allows the user to do load/store accesses. Your info key only allows the MPI implementation to internally optimize communication. — Pavan -- Pavan Balaji ✉️ http://www.mcs.anl.gov/~balaji

1 0

Re: [mpiwg-rma] Proposed Info Key
by Jim Dinan 14 Oct '14

14 Oct '14

I do like #397. The info key is something that could go in 3.1, and helps an existing usage pattern (e.g. in OSHMPI or ARMCI-MPI). Ticket #397 looks like something that we will have to target at MPI 4. IIUC, MPICH has started detecting this automatically without the info key, so this ticket may be moot. Hopefully an MPICH developer familiar with this optimization can comment. ~Jim. On Tue, Oct 14, 2014 at 10:58 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > I contend that https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/397 > is the right way to solve this problem. Using overlapping windows > here is unscalable because if you have one win_create for every > process in the node on top of one win_allocate_shared per node, you > end up with ppn+1 windows to hack the effect of one win_allocate if my > ticket is passed. This is pretty significant overhead if one wants to > run OpenSHMEM on an Intel Xeon Phi... > > And note that the time to setup all these windows properly is > nontrivial. It has been measured in the context of another project. > I can't share the details since it's not published and I'm not in a > position to decide to share it at this point. > > Jeff > > On Tue, Oct 14, 2014 at 7:42 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: > > This is an info key that e.g. OSHMPI could use to tell MPI that the > buffer > > being passed to MPI_Win_create was allocated via MPI_Win_allocate_shared. > > If this is already covered by another ticket, let me know the existing > trac > > number and I will close this one. If we want to pursue this as a 3.0 > > erratum, we could utilize this new ticket. > > > > ~Jim. > > > > On Tue, Oct 14, 2014 at 10:38 AM, Jeff Hammond <jeff.science(a)gmail.com> > > wrote: > >> > >> I don't fully understand. Is this designed to make it easier to use the > >> overlapping windows trick because we can get at shm inside of other > window > >> types? I have a ticket that solves that issue holistically already... > >> > >> Sent from my iPhone > >> > >> On Oct 14, 2014, at 7:17 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: > >> > >> Hi All, > >> > >> Please see the ticket: > >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/460 > >> > >> For a new info key that was proposed by Mikhail Brinsky of Intel. > >> > >> Thanks, > >> ~Jim. > >> > >> _______________________________________________ > >> mpiwg-rma mailing list > >> mpiwg-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > >> > >> > >> _______________________________________________ > >> mpiwg-rma mailing list > >> mpiwg-rma(a)lists.mpi-forum.org > >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > > > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >

1 0

Re: [mpiwg-rma] Proposed Info Key
by Jeff Hammond 14 Oct '14

14 Oct '14

I contend that https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/397 is the right way to solve this problem. Using overlapping windows here is unscalable because if you have one win_create for every process in the node on top of one win_allocate_shared per node, you end up with ppn+1 windows to hack the effect of one win_allocate if my ticket is passed. This is pretty significant overhead if one wants to run OpenSHMEM on an Intel Xeon Phi... And note that the time to setup all these windows properly is nontrivial. It has been measured in the context of another project. I can't share the details since it's not published and I'm not in a position to decide to share it at this point. Jeff On Tue, Oct 14, 2014 at 7:42 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: > This is an info key that e.g. OSHMPI could use to tell MPI that the buffer > being passed to MPI_Win_create was allocated via MPI_Win_allocate_shared. > If this is already covered by another ticket, let me know the existing trac > number and I will close this one. If we want to pursue this as a 3.0 > erratum, we could utilize this new ticket. > > ~Jim. > > On Tue, Oct 14, 2014 at 10:38 AM, Jeff Hammond <jeff.science(a)gmail.com> > wrote: >> >> I don't fully understand. Is this designed to make it easier to use the >> overlapping windows trick because we can get at shm inside of other window >> types? I have a ticket that solves that issue holistically already... >> >> Sent from my iPhone >> >> On Oct 14, 2014, at 7:17 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: >> >> Hi All, >> >> Please see the ticket: >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/460 >> >> For a new info key that was proposed by Mikhail Brinsky of Intel. >> >> Thanks, >> ~Jim. >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >> >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpiwg-rma] Proposed Info Key
by Jim Dinan 14 Oct '14

14 Oct '14

This is an info key that e.g. OSHMPI could use to tell MPI that the buffer being passed to MPI_Win_create was allocated via MPI_Win_allocate_shared. If this is already covered by another ticket, let me know the existing trac number and I will close this one. If we want to pursue this as a 3.0 erratum, we could utilize this new ticket. ~Jim. On Tue, Oct 14, 2014 at 10:38 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > I don't fully understand. Is this designed to make it easier to use the > overlapping windows trick because we can get at shm inside of other window > types? I have a ticket that solves that issue holistically already... > > Sent from my iPhone > > On Oct 14, 2014, at 7:17 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: > > Hi All, > > Please see the ticket: > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/460 > > For a new info key that was proposed by Mikhail Brinsky of Intel. > > Thanks, > ~Jim. > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma >

1 0

Re: [mpiwg-rma] Proposed Info Key
by Jeff Hammond 14 Oct '14

14 Oct '14

I don't fully understand. Is this designed to make it easier to use the overlapping windows trick because we can get at shm inside of other window types? I have a ticket that solves that issue holistically already... Sent from my iPhone > On Oct 14, 2014, at 7:17 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: > > Hi All, > > Please see the ticket: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/460 > > For a new info key that was proposed by Mikhail Brinsky of Intel. > > Thanks, > ~Jim. > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Proposed Info Key
by Jim Dinan 14 Oct '14

14 Oct '14

Hi All, Please see the ticket: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/460 For a new info key that was proposed by Mikhail Brinsky of Intel. Thanks, ~Jim.

1 0

Re: [mpiwg-rma] sync on vector of windows?
by Jeff Hammond 05 Oct '14

05 Oct '14

Lock/unlock are problematic. We'd need trylock semantics to avoid deadlock. And the end result is something harder to use. And I don't think these ops fall on the critical path. PGAS-style usage is going to lockall once per window and unlockall just prior to freeing it. Jeff Sent from my iPhone > On Oct 5, 2014, at 7:09 PM, Min Si <msi(a)il.is.s.u-tokyo.ac.jp> wrote: > > Looks good for me. How about lock{all} / unlock{all} ? An optimization could be combining internal lock/unlock messages especially in unlock{all} . > > Min > > > (14-10-5 20:57), Jeff Hammond wrote: >> I created https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/459 in >> response to my long-standing issue with this in ARMCI-MPI and the >> additional issues encountered by Min (copied) in another project. >> >> Please let me know what you think about this. >> >> Thanks, >> >> Jeff >

1 0