- mpiwg-rma - mailman3.cels.anl.gov

Re: [mpiwg-rma] Atomicity and Pair Types
by Balaji, Pavan 12 Sep '18

12 Sep '18

Yes, correct. Although, Nathan had a ticket at some point where he wanted to allow MPI to overwrite the padding bytes in such cases. -- Pavan > On Sep 12, 2018, at 3:49 PM, Jim Dinan via mpiwg-rma <mpiwg-rma(a)lists.mpi-forum.org> wrote: > > Hi All, > > Sanity check --- my reading of the spec. indicates that accumulate operations performed on pair types guarantee atomicity at the level of the whole pair type. Here are the relevant passages: > > Section 11.7.1: The outcome of concurrent accumulate operations to the same location with the same predefined datatype is as if the accumulates were done at that location in some serial order. > > Section 5.9.4: In order to use MPI_MINLOC and MPI_MAXLOC in a reduce operation, one must provide a datatype argument that represents a pair (value and index). MPI provides nine such predefined datatypes. > > Is that right? > > Thanks, > ~Jim. > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma

1 0

Atomicity and Pair Types
by Jim Dinan 12 Sep '18

12 Sep '18

Hi All, Sanity check --- my reading of the spec. indicates that accumulate operations performed on pair types guarantee atomicity at the level of the whole pair type. Here are the relevant passages: Section 11.7.1: The outcome of concurrent accumulate operations to the same location with the same *predefined* datatype is as if the accumulates were done at that location in some serial order. Section 5.9.4: In order to use MPI_MINLOC and MPI_MAXLOC in a reduce operation, one must provide a datatype argument that represents a pair (value and index). MPI provides nine such *predefined* datatypes. Is that right? Thanks, ~Jim.

1 0

Re: [mpiwg-rma] RMA WG Telecon
by Sameh S Sharkawi 12 Sep '18

12 Sep '18

1 0

Re: [mpiwg-rma] RMA WG Telecon
by Balaji, Pavan 12 Sep '18

12 Sep '18

There was interest in discussing GPU-related topics from NVIDIA, in the Hybrid WG, since it's not directly (or at least exclusively) related to RMA. I'll try to get those WG calls started after this F2F meeting, as well. -- Pavan > On Sep 12, 2018, at 12:58 PM, khaled hamidouche via mpiwg-rma <mpiwg-rma(a)lists.mpi-forum.org> wrote: > > Yes, GPU Direct Async (GDS), breaks the semantics of MPI as it offloads the task to the GPU and follows the stream semantics rather than the request semantics. All MPI Calls are non-blocking from the CPU perspective. > > Thanks > > On Wed, Sep 12, 2018 at 11:53 AM Sameh S Sharkawi via mpiwg-rma <mpiwg-rma(a)lists.mpi-forum.org> wrote: > Touching on the GPU issue is the GPU Direct Async feature. Nothing in the MPI standard supports the Async concept on an accelerator. Any wait/flush or equivalent call would always assume this is done on the CPU which defeats the purpose of Async on GPU. Implementing the feature on any of the current MPI APIs would either introduce limitation to the feature that negates its benefit or would break the standard. > > I wonder if we could start a separate email chain to discuss/brainstorm possible ideas ?? > > Thanks > Sameh > > > -----"mpiwg-rma" <mpiwg-rma-bounces(a)lists.mpi-forum.org> wrote: ----- > To: mark.hoemmen(a)gmail.com, MPI WG Remote Memory Access working group <mpiwg-rma(a)lists.mpi-forum.org> > From: Jeff Hammond via mpiwg-rma > Sent by: "mpiwg-rma" > Date: 09/12/2018 12:38AM > Cc: Jeff Hammond <jeff.science(a)gmail.com>, Torsten Hoefler <htor(a)inf.ethz.ch>, James Dinan <james.dinan(a)intel.com>, Jeff Hammond <jeff.r.hammond(a)intel.com>, Nathan Hjelm <hjelmn(a)lanl.gov> > Subject: Re: [mpiwg-rma] RMA WG Telecon > > Noncoherent shared memory was discussed in MPI-3 days but we have no hardware to motivate it besides Intel SCC and that was not widely available. > > Now that things like NVIDIA UM exist, there is a nontrivial hardware platform that think about if we try to define such features. However, I’m not aware of NVIDIA publishing a formal memory model for CUDA or any generation of hardware. Furthermore, one cannot - AFAIK - run an real MPI process on a GPU, so one would only be able to reason about noncoherent window memory living on GPU(s) but accessed only by CPUs. That’s much less interesting. > > Anyways, Jeff who works for Intel defers to literally anybody else who lays claim to superior understanding of NVIDIA stuff. > > Another angle would be to try to align MPI shared memory with OpenCL 2.2 SVM variants. I know exactly who to talk to about that topic. > > Jeff > > Sent from my iPhone > > On Sep 11, 2018, at 10:26 PM, Mark Hoemmen via mpiwg-rma <mpiwg-rma(a)lists.mpi-forum.org> wrote: > >> I’m sorry to have missed this — I’m very much interested in C++ binding and interoperability issues. I know Jeff at least is involved in our weekly C++ heterogeneity phone meetings — we’re trying to move the Standard to support multiple memory spaces in a sane way, and I think it helps to have RDMA represented there as well. We’ve also thought a bit about wrapping MPI RDMA in Kokkos, and about making the mdspan proposal flexible enough that we could perhaps support RDMA with it as well. The most interesting bit might be actually introducing semantics for noncoherent shared memory into the Standard — I think we want to go there eventually but it is really hard, and I think y’all are well equipped to help. >> >> Best, >> Mark Hoemmen >> >> On Mon, Sep 10, 2018 at 1:21 PM Balaji, Pavan via mpiwg-rma <mpiwg-rma(a)lists.mpi-forum.org> wrote: >> All, >> >> We are restarting the RMA WG telecons. A subset of the following proposals will be discussed during this call. >> >> - Interoperability of MPI shared memory with C11, C++11 language semantics. Lead: Bill Gropp >> - MPI Generalized atomics. Lead: Pavan Balaji >> - Neighborhood communication in RMA. Lead: Nathan Hjelm >> - Nonblocking RMA synchronization. Lead: Pavan Balaji >> - RMA Notify. Leads: Jim and Torsten >> - MPI_IN_PLACE semantics for collectives on shared memory. Lead: Pavan Balaji >> - Relax constraints on MPI_WIN_SHARED_QUERY. Lead: Jeff Hammond >> - <Unnamed proposal>. Lead: Nathan Hjelm_______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma > > > -- > K.H > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma

1 0

Re: [mpiwg-rma] RMA WG Telecon
by khaled hamidouche 12 Sep '18

12 Sep '18

Yes, GPU Direct Async (GDS), breaks the semantics of MPI as it offloads the task to the GPU and follows the stream semantics rather than the request semantics. All MPI Calls are non-blocking from the CPU perspective. Thanks On Wed, Sep 12, 2018 at 11:53 AM Sameh S Sharkawi via mpiwg-rma < mpiwg-rma(a)lists.mpi-forum.org> wrote: > Touching on the GPU issue is the GPU Direct Async feature. Nothing in the > MPI standard supports the Async concept on an accelerator. Any wait/flush > or equivalent call would always assume this is done on the CPU which > defeats the purpose of Async on GPU. Implementing the feature on any of the > current MPI APIs would either introduce limitation to the feature that > negates its benefit or would break the standard. > > I wonder if we could start a separate email chain to discuss/brainstorm > possible ideas ?? > > Thanks > Sameh > > > -----"mpiwg-rma" <mpiwg-rma-bounces(a)lists.mpi-forum.org> wrote: ----- > To: mark.hoemmen(a)gmail.com, MPI WG Remote Memory Access working group < > mpiwg-rma(a)lists.mpi-forum.org> > From: Jeff Hammond via mpiwg-rma > Sent by: "mpiwg-rma" > Date: 09/12/2018 12:38AM > Cc: Jeff Hammond <jeff.science(a)gmail.com>, Torsten Hoefler < > htor(a)inf.ethz.ch>, James Dinan <james.dinan(a)intel.com>, Jeff Hammond < > jeff.r.hammond(a)intel.com>, Nathan Hjelm <hjelmn(a)lanl.gov> > Subject: Re: [mpiwg-rma] RMA WG Telecon > > Noncoherent shared memory was discussed in MPI-3 days but we have no > hardware to motivate it besides Intel SCC and that was not widely > available. > > Now that things like NVIDIA UM exist, there is a nontrivial hardware > platform that think about if we try to define such features. However, I’m > not aware of NVIDIA publishing a formal memory model for CUDA or any > generation of hardware. Furthermore, one cannot - AFAIK - run an real MPI > process on a GPU, so one would only be able to reason about noncoherent > window memory living on GPU(s) but accessed only by CPUs. That’s much less > interesting. > > Anyways, Jeff who works for Intel defers to literally anybody else who > lays claim to superior understanding of NVIDIA stuff. > > Another angle would be to try to align MPI shared memory with OpenCL 2.2 > SVM variants. I know exactly who to talk to about that topic. > > Jeff > > Sent from my iPhone > > On Sep 11, 2018, at 10:26 PM, Mark Hoemmen via mpiwg-rma < > mpiwg-rma(a)lists.mpi-forum.org> wrote: > > I’m sorry to have missed this — I’m very much interested in C++ binding > and interoperability issues. I know Jeff at least is involved in our > weekly C++ heterogeneity phone meetings — we’re trying to move the Standard > to support multiple memory spaces in a sane way, and I think it helps to > have RDMA represented there as well. We’ve also thought a bit about > wrapping MPI RDMA in Kokkos, and about making the mdspan proposal flexible > enough that we could perhaps support RDMA with it as well. The most > interesting bit might be actually introducing semantics for noncoherent > shared memory into the Standard — I think we want to go there eventually > but it is really hard, and I think y’all are well equipped to help. > > Best, > Mark Hoemmen > > On Mon, Sep 10, 2018 at 1:21 PM Balaji, Pavan via mpiwg-rma < > mpiwg-rma(a)lists.mpi-forum.org> wrote: > >> All, >> >> We are restarting the RMA WG telecons. A subset of the following >> proposals will be discussed during this call. >> >> - Interoperability of MPI shared memory with C11, C++11 language >> semantics. Lead: Bill Gropp >> - MPI Generalized atomics. Lead: Pavan Balaji >> - Neighborhood communication in RMA. Lead: Nathan Hjelm >> - Nonblocking RMA synchronization. Lead: Pavan Balaji >> - RMA Notify. Leads: Jim and Torsten >> - MPI_IN_PLACE semantics for collectives on shared memory. Lead: Pavan >> Balaji >> - Relax constraints on MPI_WIN_SHARED_QUERY. Lead: Jeff Hammond >> - <Unnamed proposal>. Lead: Nathan >> Hjelm_______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma >> > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma > -- K.H

1 0

Re: [mpiwg-rma] RMA WG Telecon
by Sameh S Sharkawi 12 Sep '18

12 Sep '18

1 0

Re: [mpiwg-rma] RMA WG Telecon
by Balaji, Pavan 12 Sep '18

12 Sep '18

> On Sep 12, 2018, at 8:39 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > In that case, we could specify that window construction scrubs the info keys that require window memory that wasn't possible to allocate during construction. I was referring to the origin buffers? -- Pavan

1 0

Re: [mpiwg-rma] RMA WG Telecon
by Jeff Hammond 12 Sep '18

12 Sep '18

In that case, we could specify that window construction scrubs the info keys that require window memory that wasn't possible to allocate during construction. Jeff On Wed, Sep 12, 2018 at 6:26 AM, Balaji, Pavan via mpiwg-rma < mpiwg-rma(a)lists.mpi-forum.org> wrote: > Joseph, Jim, all, > > I had brought up something similar to this during MPI-2.2. The biggest > pushback on that proposal was that the MPI implementation might choose to > decide what to support in hardware based on dynamic circumstances. The > example given (by JeffS at that time) was memory registration. Something > along these lines: I can do this in hardware as long as I can register the > user memory, but when such memory registration fails I'll need to fallback > to active messages. You are welcome to open the discussion again, but my > guess is that the same pushback will still be there. > > -- Pavan > > > On Sep 12, 2018, at 7:21 AM, Jim Dinan via mpiwg-rma < > mpiwg-rma(a)lists.mpi-forum.org> wrote: > > > > In addition to op, hardware acceleration can also be tied to datatype. > For example, MPI_SUM may be accelerated for integer types, but not floating > point types. I think it's possible to support such a query using the > recent tools interfaces. I created a ticket, so we remember to discuss: > https://github.com/mpiwg-rma/rma-issues/issues/6 > > > > ~Jim. > > > > On Wed, Sep 12, 2018 at 4:26 AM Joseph Schuchart via mpiwg-rma < > mpiwg-rma(a)lists.mpi-forum.org> wrote: > > Pavan, > > > > I listened in for the MPI-RMA WG telco yesterday and found some > > interesting points in it. In particular, I am interested in the > > discussion on atomics and same_op. I support the notion that MPI should > > choose a conservative default to make sure users do not run into > > surprising UB because the implementation expects or supports only > > same_op, which the user may not be aware of. > > > > As a developer of a framework built on top of MPI RMA I would also be > > interested in getting information from the MPI implementation on which > > atomic operations are actually supported in hardware on the current > > system, which would allow me to pick different implementations of a > > specific feature to fully exploit the available hardware capabilities > > (similar to C++ `std::atomic_is_lock_free`). Are there any plans to > > provide such an interface? This could be used in combination with an > > info key (say `native_op`) that promises that the user will only use a > > mix of operations that are supported in hardware, which would then avoid > > the required fall-back to active messages discussed yesterday. > > > > Last but not least, I am also interested in the shared memory > > optimization for collectives (iirc, MPI_DISCARD?). I couldn't find an > > issue on Github on this. Is there any publicly available information you > > can share? > > > > Any input would be much appreciated. > > > > Many thanks in advance, > > Joseph > > > > On 09/10/2018 09:21 PM, Balaji, Pavan via mpiwg-rma wrote: > > > > > > > > > _______________________________________________ > > > mpiwg-rma mailing list > > > mpiwg-rma(a)lists.mpi-forum.org > > > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma > > > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpiwg-rma] RMA WG Telecon
by Balaji, Pavan 12 Sep '18

12 Sep '18

Joseph, Jim, all, I had brought up something similar to this during MPI-2.2. The biggest pushback on that proposal was that the MPI implementation might choose to decide what to support in hardware based on dynamic circumstances. The example given (by JeffS at that time) was memory registration. Something along these lines: I can do this in hardware as long as I can register the user memory, but when such memory registration fails I'll need to fallback to active messages. You are welcome to open the discussion again, but my guess is that the same pushback will still be there. -- Pavan > On Sep 12, 2018, at 7:21 AM, Jim Dinan via mpiwg-rma <mpiwg-rma(a)lists.mpi-forum.org> wrote: > > In addition to op, hardware acceleration can also be tied to datatype. For example, MPI_SUM may be accelerated for integer types, but not floating point types. I think it's possible to support such a query using the recent tools interfaces. I created a ticket, so we remember to discuss: https://github.com/mpiwg-rma/rma-issues/issues/6 > > ~Jim. > > On Wed, Sep 12, 2018 at 4:26 AM Joseph Schuchart via mpiwg-rma <mpiwg-rma(a)lists.mpi-forum.org> wrote: > Pavan, > > I listened in for the MPI-RMA WG telco yesterday and found some > interesting points in it. In particular, I am interested in the > discussion on atomics and same_op. I support the notion that MPI should > choose a conservative default to make sure users do not run into > surprising UB because the implementation expects or supports only > same_op, which the user may not be aware of. > > As a developer of a framework built on top of MPI RMA I would also be > interested in getting information from the MPI implementation on which > atomic operations are actually supported in hardware on the current > system, which would allow me to pick different implementations of a > specific feature to fully exploit the available hardware capabilities > (similar to C++ `std::atomic_is_lock_free`). Are there any plans to > provide such an interface? This could be used in combination with an > info key (say `native_op`) that promises that the user will only use a > mix of operations that are supported in hardware, which would then avoid > the required fall-back to active messages discussed yesterday. > > Last but not least, I am also interested in the shared memory > optimization for collectives (iirc, MPI_DISCARD?). I couldn't find an > issue on Github on this. Is there any publicly available information you > can share? > > Any input would be much appreciated. > > Many thanks in advance, > Joseph > > On 09/10/2018 09:21 PM, Balaji, Pavan via mpiwg-rma wrote: > > > > > > _______________________________________________ > > mpiwg-rma mailing list > > mpiwg-rma(a)lists.mpi-forum.org > > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma > > > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma

1 0

Re: [mpiwg-rma] RMA WG Telecon
by Jeff Hammond 12 Sep '18

12 Sep '18

On the topic of accumulate info keys, https://github.com/mpi-forum/mpi-issues/issues/36 is the necessary first step to make the default sufficiently general for RMA to support all usage models. https://github.com/mpi-forum/mpi-issues/issues/46 was my idea for the next step. It is not very specific, but the intent was to allow the user to describe their use of RMA in detail. This could go all the way to a database of (op,basic_datatype,complete_datatype,max_count). I suspect a more pragmatic approach is to define additional keys we think implementations will actually use, such as: * no_noncontiguous_datatypes - exclude e.g. MPI_Type_vector with stride!=1 and anything else that might lead to pack/unpack. * max_count (or should it be max_target_count) - specify upper bound on message size. Jeff On Wed, Sep 12, 2018 at 1:26 AM, Joseph Schuchart via mpiwg-rma < mpiwg-rma(a)lists.mpi-forum.org> wrote: > Pavan, > > I listened in for the MPI-RMA WG telco yesterday and found some > interesting points in it. In particular, I am interested in the discussion > on atomics and same_op. I support the notion that MPI should choose a > conservative default to make sure users do not run into surprising UB > because the implementation expects or supports only same_op, which the user > may not be aware of. > > As a developer of a framework built on top of MPI RMA I would also be > interested in getting information from the MPI implementation on which > atomic operations are actually supported in hardware on the current system, > which would allow me to pick different implementations of a specific > feature to fully exploit the available hardware capabilities (similar to > C++ `std::atomic_is_lock_free`). Are there any plans to provide such an > interface? This could be used in combination with an info key (say > `native_op`) that promises that the user will only use a mix of operations > that are supported in hardware, which would then avoid the required > fall-back to active messages discussed yesterday. > > Last but not least, I am also interested in the shared memory optimization > for collectives (iirc, MPI_DISCARD?). I couldn't find an issue on Github on > this. Is there any publicly available information you can share? > > Any input would be much appreciated. > > Many thanks in advance, > Joseph > > > On 09/10/2018 09:21 PM, Balaji, Pavan via mpiwg-rma wrote: > >> >> >> _______________________________________________ >> mpiwg-rma mailing list >> mpiwg-rma(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma >> >> _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-rma > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0