December 2014 - mpiwg-rma - mailman3.cels.anl.gov

#401: MPI_IN_PLACE in Fetching RMA Atomics
by Jeff Hammond 10 Dec '14

10 Dec '14

I have added the proposed text changes. WG members should review at their earliest convenience. Thanks, Jeff ---------- Forwarded message ---------- From: MPI Forum <mpi-forum(a)lists.mpi-forum.org> Date: Wed, Dec 10, 2014 at 10:58 AM Subject: Re: [MPI Forum] #401: MPI_IN_PLACE in Fetching RMA Atomics To: #401: MPI_IN_PLACE in Fetching RMA Atomics -------------------------------------+------------------------------------- Reporter: jdinan | Owner: jdinan Type: Enhancements to standard | Status: assigned Priority: Not ready / author | Milestone: 2014/12/08 rework | California, USA Version: MPI 3.1 | Resolution: Keywords: RMA | Implementation status: Unnecessary -------------------------------------+------------------------------------- Comment (by gropp): The working group would like to see a detailed proposal for each routine. The use of MPI_IN_PLACE should match the style in similar MPI routines. The buffering implications of each option needs to be considered and discussed on a routine-by-routine basis. -- Ticket URL: <https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/401#comment:10> MPI Forum <https://svn.mpi-forum.org/> MPI Forum -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpiwg-rma] Questions about multiple overlapped windows
by Tatsuya Abe 09 Dec '14

09 Dec '14

>I would not worry about overlapping windows. Anyone using it is on >their own anyways. >There is none. It is merely something one can infer from the text of >the standard. >This paper might interest you then: http://htor.inf.ethz.ch/publications/index.php?pub=201 . It presents a formal model for MPI-3 RMA. I'll read the text again, and try the paper. I'll send an e-mail to the ML if I have any question. Thank you. Tatsuya Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: >> I am developing a program verification tool with memory consistency >> models. As a first step, I try to formalize a part of MPI memory >> consistency model, and enable it to handle one window. > >This paper might interest you then: http://htor.inf.ethz.ch/publications/index.php?pub=201 . It presents a formal model for MPI-3 RMA. > >Rajeev > > >> ``However, concurrent communications to distinct, overlapping windows >> may lead to undefined results.''[pg 406] >> >> Thank you for the detailed explanation and pointers. >> >> I am developing a program verification tool with memory consistency >> models. As a first step, I try to formalize a part of MPI memory >> consistency model, and enable it to handle one window. > >I would not worry about overlapping windows. Anyone using it is on >their own anyways. > >>>>>Another reason is to allow ordered and unordered semantics. The >>>>>option to control this happens during window construction and cannot >>>>>be changed. If one wants one semantic (unordered) for performance and >>>>>another (ordered) for correctness, then one needs two windows, one for >>>>>each. >> >> It looks so interesting. I am wondering if you would give a >> pointer/reference to usage of multiple windows for ordered and >> unordered semantics. Thanks. > >There is none. It is merely something one can infer from the text of >the standard. > >Jeff > >On Dec 9, 2014, at 3:59 AM, Tatsuya Abe <abetatsuya(a)gmail.com> wrote: > >> Hi Jeff and Rajeev, >> >>> My ticket would allow one to use MPI_WIN_ALLOCATE to get both >>> intranode shared memory and internode RMA access. >> >>> See the text on lines 38-43 on pg 454 if it helps. >>> >>> See also ln 19-21 on pg 406. >> ``However, concurrent communications to distinct, overlapping windows >> may lead to undefined results.''[pg 406] >> >> Thank you for the detailed explanation and pointers. >> >> I am developing a program verification tool with memory consistency >> models. As a first step, I try to formalize a part of MPI memory >> consistency model, and enable it to handle one window. >> >>>>> Another reason is to allow ordered and unordered semantics. The >>>>> option to control this happens during window construction and cannot >>>>> be changed. If one wants one semantic (unordered) for performance and >>>>> another (ordered) for correctness, then one needs two windows, one for >>>>> each. >> >> It looks so interesting. I am wondering if you would give a >> pointer/reference to usage of multiple windows for ordered and >> unordered semantics. Thanks. >> >> Best, >> Tatsuya >> >> >> Jeff Hammond <jeff.science(a)gmail.com> wrote: >>> On Mon, Dec 8, 2014 at 12:00 AM, Tatsuya Abe <abetatsuya(a)gmail.com> wrote: >>>> Hi Jeff, >>>> >>>> Thank you so much for the reply. >>>> >>>>> A good example is using WIN_ALLOCATE_SHARED to create a shared memory >>>>> window, then passing the resulting buffer into WIN_CREATE. This >>>>> enables a window to support both intranode+interprocess load-store and >>>>> internode RMA. >>>> >>>> Doesn't MPI_Win_allocate_shared support internode RMA, i.e., when I >>>> create a window by using MPI_Win_allocate_shared, cannot I call >>>> MPI_Put/MPI_Get for the window? I haven't found any description about >>>> it in MPI 3.0 yet. >>> >>> WIN_ALLOCATE_SHARED is only valid on an MPI communicator corresponding >>> to a shared memory domain, so no, it does not enable internode RMA >>> unless you can do load-store between nodes. >>> >>> And yes, you can still use RMA function calls to access data in a >>> shared memory window. This is essential, for example, if you want to >>> use atomics, since some languages (e.g. C99) do not have a portable >>> abstraction for these. >>> >>>> I am a little confused. Could you let me ask you more details? >>>> >>>> 1. Do you mean that another window must be created by MPI_Win_create, >>>> in order to use MPI_Put/MPI_Get for a window created by >>>> MPI_Win_allocate_shared? Is this why multiple windows are allowed >>>> to be overlapped? >>> >>> Yes, but only for internode access, as noted above. >>> >>>> 2. Do you mean that the multiple overlapped windows (from >>>> MPI_Win_allocate_shared and MPI_Win_create) become unnecessary in >>>> this example if your ticket #397 passes? >>> >>> My ticket would allow one to use MPI_WIN_ALLOCATE to get both >>> intranode shared memory and internode RMA access. >>> >>> Jeff >>> >>>> Tatsuya >>>> >>>> >>>> >>>> Jeff Hammond <jeff.science(a)gmail.com> wrote: >>>>> On Sun, Dec 7, 2014 at 4:57 PM, Tatsuya Abe <abetatsuya(a)gmail.com> wrote: >>>>>> Dear all, >>>>>> >>>>>> I am interested in memory consistency models, and studying program >>>>>> verification with memory consistency models. I started to read MPI >>>>>> 3.0 document (and Dr. Torsten Hoefler's tutorial slides). >>>>>> http://www.eurompi2014.org/tutorials/hoefler-advanced-mpi-eurompi14.pdf >>>>>> >>>>>> I am wondering if you could give replies to my questions. >>>>>> I have two questions about multiple overlapped windows: >>>>>> >>>>>> 1. (Assume that I create two overlapped windows.) What happens if I >>>>>> do MPI_Put through one window and MPI_Get through the other window? >>>>> >>>>> It depends entirely on the synchronization epochs you use and whether >>>>> the memory model provided is UNIFIED or SEPARATE. >>>>> >>>>>> 2. I am wondering what are multiple overlapped windows for. >>>>> >>>>> To work around obnoxious shortcomings of MPI RMA. >>>>> >>>>>> 1. I can find no description about the first question in MPI 3.0 >>>>>> document. >>>>>> >>>>>> I found Torsten's slides (credit by RMA Working Group, MPI Forum), >>>>>> which explicitly showed relations between operations (Load, Store, >>>>>> Get, Put, and Acc) on pp.82--84. But, it seems an explanation >>>>>> when exactly one window is created, >>>>>> >>>>>> For example, page 83 of the slide seems to claim that Put and >>>>>> Get are allowed when they are not NOVL (non-overlapped) >>>>>> through one window in separate semantics. >>>>>> >>>>>> One of my questions is ``are Put and Get allowed when they are >>>>>> not NOVL (non-overlapped) through *two* *overlapped* windows >>>>>> in separate semantics?'' I would like to get such table about >>>>>> multiple windows. >>>>> >>>>> Within the unified model, one can reason about overlapping windows by >>>>> recognizing that the private windows of the overlapping windows are >>>>> all the same and thus when one synchronizes the public and private >>>>> windows of each window with WIN_SYNC, then all of the public windows >>>>> are the same as well. >>>>> >>>>>> 2. I can find no example about multiple overlapped windows except >>>>>> Figure 11.1 in MPI 3.0 document. Also, I suspect that Put and Get >>>>>> are allowed when they are not NOVL (non-overlapped) through two >>>>>> overlapped windows in separate semantics. >>>>>> >>>>>> I cannot come up with any useful example code. Why are multiple >>>>>> *and* overlapped windows allowed to be active? >>>>> >>>>> A good example is using WIN_ALLOCATE_SHARED to create a shared memory >>>>> window, then passing the resulting buffer into WIN_CREATE. This >>>>> enables a window to support both intranode+interprocess load-store and >>>>> internode RMA. It is useful if one wants to reproduce the behavior of >>>>> e.g. ARMCI in implementing Global Arrays. >>>>> >>>>> The aforementioned motivation will disappear if my ticket >>>>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/397 is passed and >>>>> part of MPI-4. >>>>> >>>>> Another reason is to allow ordered and unordered semantics. The >>>>> option to control this happens during window construction and cannot >>>>> be changed. If one wants one semantic (unordered) for performance and >>>>> another (ordered) for correctness, then one needs two windows, one for >>>>> each. >>>>> >>>>> Jeff >>>>> >>>>> -- >>>>> Jeff Hammond >>>>> jeff.science(a)gmail.com >>>>> http://jeffhammond.github.io/ >>> >>> >>> >>> -- >>> Jeff Hammond >>> jeff.science(a)gmail.com >>> http://jeffhammond.github.io/ >

1 0

Re: [mpiwg-rma] Questions about multiple overlapped windows
by Rajeev Thakur 09 Dec '14

09 Dec '14

> I am developing a program verification tool with memory consistency > models. As a first step, I try to formalize a part of MPI memory > consistency model, and enable it to handle one window. This paper might interest you then: http://htor.inf.ethz.ch/publications/index.php?pub=201 . It presents a formal model for MPI-3 RMA. Rajeev On Dec 9, 2014, at 3:59 AM, Tatsuya Abe <abetatsuya(a)gmail.com> wrote: > Hi Jeff and Rajeev, > >> My ticket would allow one to use MPI_WIN_ALLOCATE to get both >> intranode shared memory and internode RMA access. > >> See the text on lines 38-43 on pg 454 if it helps. >> >> See also ln 19-21 on pg 406. > ``However, concurrent communications to distinct, overlapping windows > may lead to undefined results.''[pg 406] > > Thank you for the detailed explanation and pointers. > > I am developing a program verification tool with memory consistency > models. As a first step, I try to formalize a part of MPI memory > consistency model, and enable it to handle one window. > >>>> Another reason is to allow ordered and unordered semantics. The >>>> option to control this happens during window construction and cannot >>>> be changed. If one wants one semantic (unordered) for performance and >>>> another (ordered) for correctness, then one needs two windows, one for >>>> each. > > It looks so interesting. I am wondering if you would give a > pointer/reference to usage of multiple windows for ordered and > unordered semantics. Thanks. > > Best, > Tatsuya > > > Jeff Hammond <jeff.science(a)gmail.com> wrote: >> On Mon, Dec 8, 2014 at 12:00 AM, Tatsuya Abe <abetatsuya(a)gmail.com> wrote: >>> Hi Jeff, >>> >>> Thank you so much for the reply. >>> >>>> A good example is using WIN_ALLOCATE_SHARED to create a shared memory >>>> window, then passing the resulting buffer into WIN_CREATE. This >>>> enables a window to support both intranode+interprocess load-store and >>>> internode RMA. >>> >>> Doesn't MPI_Win_allocate_shared support internode RMA, i.e., when I >>> create a window by using MPI_Win_allocate_shared, cannot I call >>> MPI_Put/MPI_Get for the window? I haven't found any description about >>> it in MPI 3.0 yet. >> >> WIN_ALLOCATE_SHARED is only valid on an MPI communicator corresponding >> to a shared memory domain, so no, it does not enable internode RMA >> unless you can do load-store between nodes. >> >> And yes, you can still use RMA function calls to access data in a >> shared memory window. This is essential, for example, if you want to >> use atomics, since some languages (e.g. C99) do not have a portable >> abstraction for these. >> >>> I am a little confused. Could you let me ask you more details? >>> >>> 1. Do you mean that another window must be created by MPI_Win_create, >>> in order to use MPI_Put/MPI_Get for a window created by >>> MPI_Win_allocate_shared? Is this why multiple windows are allowed >>> to be overlapped? >> >> Yes, but only for internode access, as noted above. >> >>> 2. Do you mean that the multiple overlapped windows (from >>> MPI_Win_allocate_shared and MPI_Win_create) become unnecessary in >>> this example if your ticket #397 passes? >> >> My ticket would allow one to use MPI_WIN_ALLOCATE to get both >> intranode shared memory and internode RMA access. >> >> Jeff >> >>> Tatsuya >>> >>> >>> >>> Jeff Hammond <jeff.science(a)gmail.com> wrote: >>>> On Sun, Dec 7, 2014 at 4:57 PM, Tatsuya Abe <abetatsuya(a)gmail.com> wrote: >>>>> Dear all, >>>>> >>>>> I am interested in memory consistency models, and studying program >>>>> verification with memory consistency models. I started to read MPI >>>>> 3.0 document (and Dr. Torsten Hoefler's tutorial slides). >>>>> http://www.eurompi2014.org/tutorials/hoefler-advanced-mpi-eurompi14.pdf >>>>> >>>>> I am wondering if you could give replies to my questions. >>>>> I have two questions about multiple overlapped windows: >>>>> >>>>> 1. (Assume that I create two overlapped windows.) What happens if I >>>>> do MPI_Put through one window and MPI_Get through the other window? >>>> >>>> It depends entirely on the synchronization epochs you use and whether >>>> the memory model provided is UNIFIED or SEPARATE. >>>> >>>>> 2. I am wondering what are multiple overlapped windows for. >>>> >>>> To work around obnoxious shortcomings of MPI RMA. >>>> >>>>> 1. I can find no description about the first question in MPI 3.0 >>>>> document. >>>>> >>>>> I found Torsten's slides (credit by RMA Working Group, MPI Forum), >>>>> which explicitly showed relations between operations (Load, Store, >>>>> Get, Put, and Acc) on pp.82--84. But, it seems an explanation >>>>> when exactly one window is created, >>>>> >>>>> For example, page 83 of the slide seems to claim that Put and >>>>> Get are allowed when they are not NOVL (non-overlapped) >>>>> through one window in separate semantics. >>>>> >>>>> One of my questions is ``are Put and Get allowed when they are >>>>> not NOVL (non-overlapped) through *two* *overlapped* windows >>>>> in separate semantics?'' I would like to get such table about >>>>> multiple windows. >>>> >>>> Within the unified model, one can reason about overlapping windows by >>>> recognizing that the private windows of the overlapping windows are >>>> all the same and thus when one synchronizes the public and private >>>> windows of each window with WIN_SYNC, then all of the public windows >>>> are the same as well. >>>> >>>>> 2. I can find no example about multiple overlapped windows except >>>>> Figure 11.1 in MPI 3.0 document. Also, I suspect that Put and Get >>>>> are allowed when they are not NOVL (non-overlapped) through two >>>>> overlapped windows in separate semantics. >>>>> >>>>> I cannot come up with any useful example code. Why are multiple >>>>> *and* overlapped windows allowed to be active? >>>> >>>> A good example is using WIN_ALLOCATE_SHARED to create a shared memory >>>> window, then passing the resulting buffer into WIN_CREATE. This >>>> enables a window to support both intranode+interprocess load-store and >>>> internode RMA. It is useful if one wants to reproduce the behavior of >>>> e.g. ARMCI in implementing Global Arrays. >>>> >>>> The aforementioned motivation will disappear if my ticket >>>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/397 is passed and >>>> part of MPI-4. >>>> >>>> Another reason is to allow ordered and unordered semantics. The >>>> option to control this happens during window construction and cannot >>>> be changed. If one wants one semantic (unordered) for performance and >>>> another (ordered) for correctness, then one needs two windows, one for >>>> each. >>>> >>>> Jeff >>>> >>>> -- >>>> Jeff Hammond >>>> jeff.science(a)gmail.com >>>> http://jeffhammond.github.io/ >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> http://jeffhammond.github.io/

1 0

Re: [mpiwg-rma] Questions about multiple overlapped windows
by Jeff Hammond 09 Dec '14

09 Dec '14

On Tue, Dec 9, 2014 at 1:59 AM, Tatsuya Abe <abetatsuya(a)gmail.com> wrote: > Hi Jeff and Rajeev, > >>My ticket would allow one to use MPI_WIN_ALLOCATE to get both >>intranode shared memory and internode RMA access. > >>See the text on lines 38-43 on pg 454 if it helps. >> >>See also ln 19-21 on pg 406. > ``However, concurrent communications to distinct, overlapping windows > may lead to undefined results.''[pg 406] > > Thank you for the detailed explanation and pointers. > > I am developing a program verification tool with memory consistency > models. As a first step, I try to formalize a part of MPI memory > consistency model, and enable it to handle one window. I would not worry about overlapping windows. Anyone using it is on their own anyways. >>>>Another reason is to allow ordered and unordered semantics. The >>>>option to control this happens during window construction and cannot >>>>be changed. If one wants one semantic (unordered) for performance and >>>>another (ordered) for correctness, then one needs two windows, one for >>>>each. > > It looks so interesting. I am wondering if you would give a > pointer/reference to usage of multiple windows for ordered and > unordered semantics. Thanks. There is none. It is merely something one can infer from the text of the standard. Jeff > Best, > Tatsuya > > > Jeff Hammond <jeff.science(a)gmail.com> wrote: >>On Mon, Dec 8, 2014 at 12:00 AM, Tatsuya Abe <abetatsuya(a)gmail.com> wrote: >>> Hi Jeff, >>> >>> Thank you so much for the reply. >>> >>>>A good example is using WIN_ALLOCATE_SHARED to create a shared memory >>>>window, then passing the resulting buffer into WIN_CREATE. This >>>>enables a window to support both intranode+interprocess load-store and >>>>internode RMA. >>> >>> Doesn't MPI_Win_allocate_shared support internode RMA, i.e., when I >>> create a window by using MPI_Win_allocate_shared, cannot I call >>> MPI_Put/MPI_Get for the window? I haven't found any description about >>> it in MPI 3.0 yet. >> >>WIN_ALLOCATE_SHARED is only valid on an MPI communicator corresponding >>to a shared memory domain, so no, it does not enable internode RMA >>unless you can do load-store between nodes. >> >>And yes, you can still use RMA function calls to access data in a >>shared memory window. This is essential, for example, if you want to >>use atomics, since some languages (e.g. C99) do not have a portable >>abstraction for these. >> >>> I am a little confused. Could you let me ask you more details? >>> >>> 1. Do you mean that another window must be created by MPI_Win_create, >>> in order to use MPI_Put/MPI_Get for a window created by >>> MPI_Win_allocate_shared? Is this why multiple windows are allowed >>> to be overlapped? >> >>Yes, but only for internode access, as noted above. >> >>> 2. Do you mean that the multiple overlapped windows (from >>> MPI_Win_allocate_shared and MPI_Win_create) become unnecessary in >>> this example if your ticket #397 passes? >> >>My ticket would allow one to use MPI_WIN_ALLOCATE to get both >>intranode shared memory and internode RMA access. >> >>Jeff >> >>> Tatsuya >>> >>> >>> >>> Jeff Hammond <jeff.science(a)gmail.com> wrote: >>>>On Sun, Dec 7, 2014 at 4:57 PM, Tatsuya Abe <abetatsuya(a)gmail.com> wrote: >>>>> Dear all, >>>>> >>>>> I am interested in memory consistency models, and studying program >>>>> verification with memory consistency models. I started to read MPI >>>>> 3.0 document (and Dr. Torsten Hoefler's tutorial slides). >>>>> http://www.eurompi2014.org/tutorials/hoefler-advanced-mpi-eurompi14.pdf >>>>> >>>>> I am wondering if you could give replies to my questions. >>>>> I have two questions about multiple overlapped windows: >>>>> >>>>> 1. (Assume that I create two overlapped windows.) What happens if I >>>>> do MPI_Put through one window and MPI_Get through the other window? >>>> >>>>It depends entirely on the synchronization epochs you use and whether >>>>the memory model provided is UNIFIED or SEPARATE. >>>> >>>>> 2. I am wondering what are multiple overlapped windows for. >>>> >>>>To work around obnoxious shortcomings of MPI RMA. >>>> >>>>> 1. I can find no description about the first question in MPI 3.0 >>>>> document. >>>>> >>>>> I found Torsten's slides (credit by RMA Working Group, MPI Forum), >>>>> which explicitly showed relations between operations (Load, Store, >>>>> Get, Put, and Acc) on pp.82--84. But, it seems an explanation >>>>> when exactly one window is created, >>>>> >>>>> For example, page 83 of the slide seems to claim that Put and >>>>> Get are allowed when they are not NOVL (non-overlapped) >>>>> through one window in separate semantics. >>>>> >>>>> One of my questions is ``are Put and Get allowed when they are >>>>> not NOVL (non-overlapped) through *two* *overlapped* windows >>>>> in separate semantics?'' I would like to get such table about >>>>> multiple windows. >>>> >>>>Within the unified model, one can reason about overlapping windows by >>>>recognizing that the private windows of the overlapping windows are >>>>all the same and thus when one synchronizes the public and private >>>>windows of each window with WIN_SYNC, then all of the public windows >>>>are the same as well. >>>> >>>>> 2. I can find no example about multiple overlapped windows except >>>>> Figure 11.1 in MPI 3.0 document. Also, I suspect that Put and Get >>>>> are allowed when they are not NOVL (non-overlapped) through two >>>>> overlapped windows in separate semantics. >>>>> >>>>> I cannot come up with any useful example code. Why are multiple >>>>> *and* overlapped windows allowed to be active? >>>> >>>>A good example is using WIN_ALLOCATE_SHARED to create a shared memory >>>>window, then passing the resulting buffer into WIN_CREATE. This >>>>enables a window to support both intranode+interprocess load-store and >>>>internode RMA. It is useful if one wants to reproduce the behavior of >>>>e.g. ARMCI in implementing Global Arrays. >>>> >>>>The aforementioned motivation will disappear if my ticket >>>>https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/397 is passed and >>>>part of MPI-4. >>>> >>>>Another reason is to allow ordered and unordered semantics. The >>>>option to control this happens during window construction and cannot >>>>be changed. If one wants one semantic (unordered) for performance and >>>>another (ordered) for correctness, then one needs two windows, one for >>>>each. >>>> >>>>Jeff >>>> >>>>-- >>>>Jeff Hammond >>>>jeff.science(a)gmail.com >>>>http://jeffhammond.github.io/ >> >> >> >>-- >>Jeff Hammond >>jeff.science(a)gmail.com >>http://jeffhammond.github.io/ -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpiwg-rma] Questions about multiple overlapped windows
by Tatsuya Abe 09 Dec '14

09 Dec '14

Hi Jeff and Rajeev, >My ticket would allow one to use MPI_WIN_ALLOCATE to get both >intranode shared memory and internode RMA access. >See the text on lines 38-43 on pg 454 if it helps. > >See also ln 19-21 on pg 406. ``However, concurrent communications to distinct, overlapping windows may lead to undefined results.''[pg 406] Thank you for the detailed explanation and pointers. I am developing a program verification tool with memory consistency models. As a first step, I try to formalize a part of MPI memory consistency model, and enable it to handle one window. >>>Another reason is to allow ordered and unordered semantics. The >>>option to control this happens during window construction and cannot >>>be changed. If one wants one semantic (unordered) for performance and >>>another (ordered) for correctness, then one needs two windows, one for >>>each. It looks so interesting. I am wondering if you would give a pointer/reference to usage of multiple windows for ordered and unordered semantics. Thanks. Best, Tatsuya Jeff Hammond <jeff.science(a)gmail.com> wrote: >On Mon, Dec 8, 2014 at 12:00 AM, Tatsuya Abe <abetatsuya(a)gmail.com> wrote: >> Hi Jeff, >> >> Thank you so much for the reply. >> >>>A good example is using WIN_ALLOCATE_SHARED to create a shared memory >>>window, then passing the resulting buffer into WIN_CREATE. This >>>enables a window to support both intranode+interprocess load-store and >>>internode RMA. >> >> Doesn't MPI_Win_allocate_shared support internode RMA, i.e., when I >> create a window by using MPI_Win_allocate_shared, cannot I call >> MPI_Put/MPI_Get for the window? I haven't found any description about >> it in MPI 3.0 yet. > >WIN_ALLOCATE_SHARED is only valid on an MPI communicator corresponding >to a shared memory domain, so no, it does not enable internode RMA >unless you can do load-store between nodes. > >And yes, you can still use RMA function calls to access data in a >shared memory window. This is essential, for example, if you want to >use atomics, since some languages (e.g. C99) do not have a portable >abstraction for these. > >> I am a little confused. Could you let me ask you more details? >> >> 1. Do you mean that another window must be created by MPI_Win_create, >> in order to use MPI_Put/MPI_Get for a window created by >> MPI_Win_allocate_shared? Is this why multiple windows are allowed >> to be overlapped? > >Yes, but only for internode access, as noted above. > >> 2. Do you mean that the multiple overlapped windows (from >> MPI_Win_allocate_shared and MPI_Win_create) become unnecessary in >> this example if your ticket #397 passes? > >My ticket would allow one to use MPI_WIN_ALLOCATE to get both >intranode shared memory and internode RMA access. > >Jeff > >> Tatsuya >> >> >> >> Jeff Hammond <jeff.science(a)gmail.com> wrote: >>>On Sun, Dec 7, 2014 at 4:57 PM, Tatsuya Abe <abetatsuya(a)gmail.com> wrote: >>>> Dear all, >>>> >>>> I am interested in memory consistency models, and studying program >>>> verification with memory consistency models. I started to read MPI >>>> 3.0 document (and Dr. Torsten Hoefler's tutorial slides). >>>> http://www.eurompi2014.org/tutorials/hoefler-advanced-mpi-eurompi14.pdf >>>> >>>> I am wondering if you could give replies to my questions. >>>> I have two questions about multiple overlapped windows: >>>> >>>> 1. (Assume that I create two overlapped windows.) What happens if I >>>> do MPI_Put through one window and MPI_Get through the other window? >>> >>>It depends entirely on the synchronization epochs you use and whether >>>the memory model provided is UNIFIED or SEPARATE. >>> >>>> 2. I am wondering what are multiple overlapped windows for. >>> >>>To work around obnoxious shortcomings of MPI RMA. >>> >>>> 1. I can find no description about the first question in MPI 3.0 >>>> document. >>>> >>>> I found Torsten's slides (credit by RMA Working Group, MPI Forum), >>>> which explicitly showed relations between operations (Load, Store, >>>> Get, Put, and Acc) on pp.82--84. But, it seems an explanation >>>> when exactly one window is created, >>>> >>>> For example, page 83 of the slide seems to claim that Put and >>>> Get are allowed when they are not NOVL (non-overlapped) >>>> through one window in separate semantics. >>>> >>>> One of my questions is ``are Put and Get allowed when they are >>>> not NOVL (non-overlapped) through *two* *overlapped* windows >>>> in separate semantics?'' I would like to get such table about >>>> multiple windows. >>> >>>Within the unified model, one can reason about overlapping windows by >>>recognizing that the private windows of the overlapping windows are >>>all the same and thus when one synchronizes the public and private >>>windows of each window with WIN_SYNC, then all of the public windows >>>are the same as well. >>> >>>> 2. I can find no example about multiple overlapped windows except >>>> Figure 11.1 in MPI 3.0 document. Also, I suspect that Put and Get >>>> are allowed when they are not NOVL (non-overlapped) through two >>>> overlapped windows in separate semantics. >>>> >>>> I cannot come up with any useful example code. Why are multiple >>>> *and* overlapped windows allowed to be active? >>> >>>A good example is using WIN_ALLOCATE_SHARED to create a shared memory >>>window, then passing the resulting buffer into WIN_CREATE. This >>>enables a window to support both intranode+interprocess load-store and >>>internode RMA. It is useful if one wants to reproduce the behavior of >>>e.g. ARMCI in implementing Global Arrays. >>> >>>The aforementioned motivation will disappear if my ticket >>>https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/397 is passed and >>>part of MPI-4. >>> >>>Another reason is to allow ordered and unordered semantics. The >>>option to control this happens during window construction and cannot >>>be changed. If one wants one semantic (unordered) for performance and >>>another (ordered) for correctness, then one needs two windows, one for >>>each. >>> >>>Jeff >>> >>>-- >>>Jeff Hammond >>>jeff.science(a)gmail.com >>>http://jeffhammond.github.io/ > > > >-- >Jeff Hammond >jeff.science(a)gmail.com >http://jeffhammond.github.io/

1 0

Re: [mpiwg-rma] Questions about multiple overlapped windows
by Rajeev Thakur 08 Dec '14

08 Dec '14

See the text on lines 38-43 on pg 454 if it helps. See also ln 19-21 on pg 406. Rajeev On Dec 7, 2014, at 6:57 PM, Tatsuya Abe <abetatsuya(a)gmail.com> wrote: > Dear all, > > I am interested in memory consistency models, and studying program > verification with memory consistency models. I started to read MPI > 3.0 document (and Dr. Torsten Hoefler's tutorial slides). > http://www.eurompi2014.org/tutorials/hoefler-advanced-mpi-eurompi14.pdf > > I am wondering if you could give replys to my questions. > I have two questions about multiple overlapped windows: > > 1. (Assume that I create two overlapped windows.) What happens if I > do MPI_Put through one window and MPI_Get through the other window? > > 2. I am wondering what are multiple overlapped windows for. > > > > 1. I can find no description about the first question in MPI 3.0 > document. > > I found Torsten's slides (credit by RMA Working Group, MPI Forum), > which explicitly showed relations between operations (Load, Store, > Get, Put, and Acc) on pp.82--84. But, it seems an explaination > when exactly one window is created, > > For example, page 83 of the slide seems to claim that Put and > Get are allowed when they are not NOVL (non-overlapped) > through one window in separate semantics. > > One of my questions is ``are Put and Get allowed when they are > not NOVL (non-overlapped) through *two* *overlapped* windows > in separate semantics?'' I would like to get such table about > multiple windows. > > > 2. I can find no example about multiple overlapped windows except > Figure 11.1 in MPI 3.0 document. Also, I suspect that Put and Get > are allowed when they are not NOVL (non-overlapped) through two > overlapped windows in separate semantics. > > I cannot come up with any useful example code. Why are multiple > *and* overlapped windows allowed to be active? > > Tatsuya > _______________________________________________ > mpiwg-rma mailing list > mpiwg-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-rma

1 0

Re: [mpiwg-rma] Questions about multiple overlapped windows
by Jeff Hammond 08 Dec '14

08 Dec '14

On Mon, Dec 8, 2014 at 12:00 AM, Tatsuya Abe <abetatsuya(a)gmail.com> wrote: > Hi Jeff, > > Thank you so much for the reply. > >>A good example is using WIN_ALLOCATE_SHARED to create a shared memory >>window, then passing the resulting buffer into WIN_CREATE. This >>enables a window to support both intranode+interprocess load-store and >>internode RMA. > > Doesn't MPI_Win_allocate_shared support internode RMA, i.e., when I > create a window by using MPI_Win_allocate_shared, cannot I call > MPI_Put/MPI_Get for the window? I haven't found any description about > it in MPI 3.0 yet. WIN_ALLOCATE_SHARED is only valid on an MPI communicator corresponding to a shared memory domain, so no, it does not enable internode RMA unless you can do load-store between nodes. And yes, you can still use RMA function calls to access data in a shared memory window. This is essential, for example, if you want to use atomics, since some languages (e.g. C99) do not have a portable abstraction for these. > I am a little confused. Could you let me ask you more details? > > 1. Do you mean that another window must be created by MPI_Win_create, > in order to use MPI_Put/MPI_Get for a window created by > MPI_Win_allocate_shared? Is this why multiple windows are allowed > to be overlapped? Yes, but only for internode access, as noted above. > 2. Do you mean that the multiple overlapped windows (from > MPI_Win_allocate_shared and MPI_Win_create) become unnecessary in > this example if your ticket #397 passes? My ticket would allow one to use MPI_WIN_ALLOCATE to get both intranode shared memory and internode RMA access. Jeff > Tatsuya > > > > Jeff Hammond <jeff.science(a)gmail.com> wrote: >>On Sun, Dec 7, 2014 at 4:57 PM, Tatsuya Abe <abetatsuya(a)gmail.com> wrote: >>> Dear all, >>> >>> I am interested in memory consistency models, and studying program >>> verification with memory consistency models. I started to read MPI >>> 3.0 document (and Dr. Torsten Hoefler's tutorial slides). >>> http://www.eurompi2014.org/tutorials/hoefler-advanced-mpi-eurompi14.pdf >>> >>> I am wondering if you could give replies to my questions. >>> I have two questions about multiple overlapped windows: >>> >>> 1. (Assume that I create two overlapped windows.) What happens if I >>> do MPI_Put through one window and MPI_Get through the other window? >> >>It depends entirely on the synchronization epochs you use and whether >>the memory model provided is UNIFIED or SEPARATE. >> >>> 2. I am wondering what are multiple overlapped windows for. >> >>To work around obnoxious shortcomings of MPI RMA. >> >>> 1. I can find no description about the first question in MPI 3.0 >>> document. >>> >>> I found Torsten's slides (credit by RMA Working Group, MPI Forum), >>> which explicitly showed relations between operations (Load, Store, >>> Get, Put, and Acc) on pp.82--84. But, it seems an explanation >>> when exactly one window is created, >>> >>> For example, page 83 of the slide seems to claim that Put and >>> Get are allowed when they are not NOVL (non-overlapped) >>> through one window in separate semantics. >>> >>> One of my questions is ``are Put and Get allowed when they are >>> not NOVL (non-overlapped) through *two* *overlapped* windows >>> in separate semantics?'' I would like to get such table about >>> multiple windows. >> >>Within the unified model, one can reason about overlapping windows by >>recognizing that the private windows of the overlapping windows are >>all the same and thus when one synchronizes the public and private >>windows of each window with WIN_SYNC, then all of the public windows >>are the same as well. >> >>> 2. I can find no example about multiple overlapped windows except >>> Figure 11.1 in MPI 3.0 document. Also, I suspect that Put and Get >>> are allowed when they are not NOVL (non-overlapped) through two >>> overlapped windows in separate semantics. >>> >>> I cannot come up with any useful example code. Why are multiple >>> *and* overlapped windows allowed to be active? >> >>A good example is using WIN_ALLOCATE_SHARED to create a shared memory >>window, then passing the resulting buffer into WIN_CREATE. This >>enables a window to support both intranode+interprocess load-store and >>internode RMA. It is useful if one wants to reproduce the behavior of >>e.g. ARMCI in implementing Global Arrays. >> >>The aforementioned motivation will disappear if my ticket >>https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/397 is passed and >>part of MPI-4. >> >>Another reason is to allow ordered and unordered semantics. The >>option to control this happens during window construction and cannot >>be changed. If one wants one semantic (unordered) for performance and >>another (ordered) for correctness, then one needs two windows, one for >>each. >> >>Jeff >> >>-- >>Jeff Hammond >>jeff.science(a)gmail.com >>http://jeffhammond.github.io/ -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpiwg-rma] Questions about multiple overlapped windows
by Tatsuya Abe 08 Dec '14

08 Dec '14

Hi Jeff, Thank you so much for the reply. >A good example is using WIN_ALLOCATE_SHARED to create a shared memory >window, then passing the resulting buffer into WIN_CREATE. This >enables a window to support both intranode+interprocess load-store and >internode RMA. Doesn't MPI_Win_allocate_shared support internode RMA, i.e., when I create a window by using MPI_Win_allocate_shared, cannot I call MPI_Put/MPI_Get for the window? I haven't found any description about it in MPI 3.0 yet. I am a little confused. Could you let me ask you more details? 1. Do you mean that another window must be created by MPI_Win_create, in order to use MPI_Put/MPI_Get for a window created by MPI_Win_allocate_shared? Is this why multiple windows are allowed to be overlapped? 2. Do you mean that the multiple overlapped windows (from MPI_Win_allocate_shared and MPI_Win_create) become unnecessary in this exapmple if your ticket #397 passes? Tatsuya Jeff Hammond <jeff.science(a)gmail.com> wrote: >On Sun, Dec 7, 2014 at 4:57 PM, Tatsuya Abe <abetatsuya(a)gmail.com> wrote: >> Dear all, >> >> I am interested in memory consistency models, and studying program >> verification with memory consistency models. I started to read MPI >> 3.0 document (and Dr. Torsten Hoefler's tutorial slides). >> http://www.eurompi2014.org/tutorials/hoefler-advanced-mpi-eurompi14.pdf >> >> I am wondering if you could give replies to my questions. >> I have two questions about multiple overlapped windows: >> >> 1. (Assume that I create two overlapped windows.) What happens if I >> do MPI_Put through one window and MPI_Get through the other window? > >It depends entirely on the synchronization epochs you use and whether >the memory model provided is UNIFIED or SEPARATE. > >> 2. I am wondering what are multiple overlapped windows for. > >To work around obnoxious shortcomings of MPI RMA. > >> 1. I can find no description about the first question in MPI 3.0 >> document. >> >> I found Torsten's slides (credit by RMA Working Group, MPI Forum), >> which explicitly showed relations between operations (Load, Store, >> Get, Put, and Acc) on pp.82--84. But, it seems an explanation >> when exactly one window is created, >> >> For example, page 83 of the slide seems to claim that Put and >> Get are allowed when they are not NOVL (non-overlapped) >> through one window in separate semantics. >> >> One of my questions is ``are Put and Get allowed when they are >> not NOVL (non-overlapped) through *two* *overlapped* windows >> in separate semantics?'' I would like to get such table about >> multiple windows. > >Within the unified model, one can reason about overlapping windows by >recognizing that the private windows of the overlapping windows are >all the same and thus when one synchronizes the public and private >windows of each window with WIN_SYNC, then all of the public windows >are the same as well. > >> 2. I can find no example about multiple overlapped windows except >> Figure 11.1 in MPI 3.0 document. Also, I suspect that Put and Get >> are allowed when they are not NOVL (non-overlapped) through two >> overlapped windows in separate semantics. >> >> I cannot come up with any useful example code. Why are multiple >> *and* overlapped windows allowed to be active? > >A good example is using WIN_ALLOCATE_SHARED to create a shared memory >window, then passing the resulting buffer into WIN_CREATE. This >enables a window to support both intranode+interprocess load-store and >internode RMA. It is useful if one wants to reproduce the behavior of >e.g. ARMCI in implementing Global Arrays. > >The aforementioned motivation will disappear if my ticket >https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/397 is passed and >part of MPI-4. > >Another reason is to allow ordered and unordered semantics. The >option to control this happens during window construction and cannot >be changed. If one wants one semantic (unordered) for performance and >another (ordered) for correctness, then one needs two windows, one for >each. > >Jeff > >-- >Jeff Hammond >jeff.science(a)gmail.com >http://jeffhammond.github.io/

1 0

Re: [mpiwg-rma] Questions about multiple overlapped windows
by Jeff Hammond 07 Dec '14

07 Dec '14

On Sun, Dec 7, 2014 at 4:57 PM, Tatsuya Abe <abetatsuya(a)gmail.com> wrote: > Dear all, > > I am interested in memory consistency models, and studying program > verification with memory consistency models. I started to read MPI > 3.0 document (and Dr. Torsten Hoefler's tutorial slides). > http://www.eurompi2014.org/tutorials/hoefler-advanced-mpi-eurompi14.pdf > > I am wondering if you could give replies to my questions. > I have two questions about multiple overlapped windows: > > 1. (Assume that I create two overlapped windows.) What happens if I > do MPI_Put through one window and MPI_Get through the other window? It depends entirely on the synchronization epochs you use and whether the memory model provided is UNIFIED or SEPARATE. > 2. I am wondering what are multiple overlapped windows for. To work around obnoxious shortcomings of MPI RMA. > 1. I can find no description about the first question in MPI 3.0 > document. > > I found Torsten's slides (credit by RMA Working Group, MPI Forum), > which explicitly showed relations between operations (Load, Store, > Get, Put, and Acc) on pp.82--84. But, it seems an explanation > when exactly one window is created, > > For example, page 83 of the slide seems to claim that Put and > Get are allowed when they are not NOVL (non-overlapped) > through one window in separate semantics. > > One of my questions is ``are Put and Get allowed when they are > not NOVL (non-overlapped) through *two* *overlapped* windows > in separate semantics?'' I would like to get such table about > multiple windows. Within the unified model, one can reason about overlapping windows by recognizing that the private windows of the overlapping windows are all the same and thus when one synchronizes the public and private windows of each window with WIN_SYNC, then all of the public windows are the same as well. > 2. I can find no example about multiple overlapped windows except > Figure 11.1 in MPI 3.0 document. Also, I suspect that Put and Get > are allowed when they are not NOVL (non-overlapped) through two > overlapped windows in separate semantics. > > I cannot come up with any useful example code. Why are multiple > *and* overlapped windows allowed to be active? A good example is using WIN_ALLOCATE_SHARED to create a shared memory window, then passing the resulting buffer into WIN_CREATE. This enables a window to support both intranode+interprocess load-store and internode RMA. It is useful if one wants to reproduce the behavior of e.g. ARMCI in implementing Global Arrays. The aforementioned motivation will disappear if my ticket https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/397 is passed and part of MPI-4. Another reason is to allow ordered and unordered semantics. The option to control this happens during window construction and cannot be changed. If one wants one semantic (unordered) for performance and another (ordered) for correctness, then one needs two windows, one for each. Jeff -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Questions about multiple overlapped windows
by Tatsuya Abe 07 Dec '14

07 Dec '14

Dear all, I am interested in memory consistency models, and studying program verification with memory consistency models. I started to read MPI 3.0 document (and Dr. Torsten Hoefler's tutorial slides). http://www.eurompi2014.org/tutorials/hoefler-advanced-mpi-eurompi14.pdf I am wondering if you could give replys to my questions. I have two questions about multiple overlapped windows: 1. (Assume that I create two overlapped windows.) What happens if I do MPI_Put through one window and MPI_Get through the other window? 2. I am wondering what are multiple overlapped windows for. 1. I can find no description about the first question in MPI 3.0 document. I found Torsten's slides (credit by RMA Working Group, MPI Forum), which explicitly showed relations between operations (Load, Store, Get, Put, and Acc) on pp.82--84. But, it seems an explaination when exactly one window is created, For example, page 83 of the slide seems to claim that Put and Get are allowed when they are not NOVL (non-overlapped) through one window in separate semantics. One of my questions is ``are Put and Get allowed when they are not NOVL (non-overlapped) through *two* *overlapped* windows in separate semantics?'' I would like to get such table about multiple windows. 2. I can find no example about multiple overlapped windows except Figure 11.1 in MPI 3.0 document. Also, I suspect that Put and Get are allowed when they are not NOVL (non-overlapped) through two overlapped windows in separate semantics. I cannot come up with any useful example code. Why are multiple *and* overlapped windows allowed to be active? Tatsuya

1 0