February 2011 - mpiwg-rma - mailman3.cels.anl.gov

Re: [Mpi3-rma] Draft Plenary Presentation for MPI RMA
by Torsten Hoefler 04 Feb '11

04 Feb '11

On Fri, Feb 04, 2011 at 03:40:09PM -0600, James Dinan wrote: > On 02/04/2011 03:11 PM, Torsten Hoefler wrote: >> On Fri, Feb 04, 2011 at 02:45:42PM -0600, James Dinan wrote: >>> Hi Torsten, >>> >>> On 02/04/2011 12:35 PM, Torsten Hoefler wrote: >>>>> Slide 22: UPC doesn't allow sharing data on the stack, only heap data >>>>> can be shared. In general, this is kind of an iffy thing, might >>>>> encounter some resistance to suggest doing this. Linux is going to >>>>> initially map the stack to the zero page which is marked copy-on-write, >>>>> so beyond the parts you have used, the stack doesn't exist. >>>> >>>> That's what registration is for. It has to be created then. Or am I >>>> missing something? >>> >>> UPC doesn't allow shared variables on the stack and I can't think of any >>> other PGAS model that does. So, the argument of registering and >>> exposing the stack doesn't seem like it holds water to me. >> Well, the current proposal allows this. One can register anything that >> can be dereferenced, including stack variables. If you think that should >> not be allowed than we need to fix the proposal. The slides simply >> explain the proposal. > > I don't think this should be forbidden. I just think it's a bad idea to > do it and we should not present it as a motivation for the work. It's > also not needed by UPC. Ok, so we agree :). >>> I believe Dan, et al.'s issue with MPI_Alloc_mem as a requirement is >>> that *static* shared UPC objects could not be put into windows. Static >>> distributed shared arrays are one of the most common use cases in UPC. >>> Adding win_register now allows you to register statically declared >>> arrays and expose them in a window. >> This is allowed too. > > I would prefer to see static arrays used as the motivating example on > slide 22 instead of the stack. It's needed by UPC and helpful in > general to C/Fortran programmers. I changed it a bit. I still find it convenient to target stack locations. I'm also often sending stack locations in my codes. I don't see what's wrong or bad with this. Yes, one needs to pay attention if the stack is valid. Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [Mpi3-rma] Draft Plenary Presentation for MPI RMA
by Jeff Hammond 04 Feb '11

04 Feb '11

If it is not obvious, ARMCI_Fence is bulk remote completion on a single target, not global ala shmem_quiet (?). Of course, ARMCI AllFence completes at all targets what was sent from the calling process. ARMCI is sequentially consistent between the source and target so ARMCI_Fence ensures that the remote completion is visible to everyone. Sorry for being pedantic. Maybe everyone already knew all this. Jeff On Fri, Feb 4, 2011 at 3:42 PM, James Dinan <dinan(a)mcs.anl.gov> wrote: > On 02/04/2011 03:10 PM, Torsten Hoefler wrote: >> >> On Fri, Feb 04, 2011 at 02:35:45PM -0600, Pavan Balaji wrote: >>> >>> On 02/04/2011 01:14 PM, Torsten Hoefler wrote: >>>>> >>>>> Slide 27: ARMCI has per-operation non-blocking calls. ï¿½Blocking calls >>>>> and waits give you local completion and only fence gives you remote >>>>> completion. >>>> >>>> Yes, ARMCI has per operation remote completion which we don't support at >>>> this point. Proposal 2 had this a while ago but we didn't find a strong >>>> use-case (as opposed to the per-operation local completion). Added a >>>> comment. >>> >>> No, ARMCI doesn't have per operation remote completion; only local >>> completion. Attached the slides I had distributed earlier about this for >>> your reference. >> >> Ok, I got this impression from our previous discussion. So Jim, what did >> your comment then mean? > > > I meant the same thing as what Pavan wrote, but had a language fail: ARMCI > gives per-operation local completion for non-blocking operations. ï¿½Blocking > operations are locally complete when they return. ï¿½Bulk remote completion is > accomplished through fence. > > ï¿½~Jim. > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > -- Jeff Hammond Argonne Leadership Computing Facility jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond

1 0

Re: [Mpi3-rma] Draft Plenary Presentation for MPI RMA
by James Dinan 04 Feb '11

04 Feb '11

On 02/04/2011 03:10 PM, Torsten Hoefler wrote: > On Fri, Feb 04, 2011 at 02:35:45PM -0600, Pavan Balaji wrote: >> >> On 02/04/2011 01:14 PM, Torsten Hoefler wrote: >>>> Slide 27: ARMCI has per-operation non-blocking calls. Blocking calls >>>> and waits give you local completion and only fence gives you remote >>>> completion. >>> Yes, ARMCI has per operation remote completion which we don't support at >>> this point. Proposal 2 had this a while ago but we didn't find a strong >>> use-case (as opposed to the per-operation local completion). Added a >>> comment. >> >> No, ARMCI doesn't have per operation remote completion; only local >> completion. Attached the slides I had distributed earlier about this for >> your reference. > Ok, I got this impression from our previous discussion. So Jim, what did > your comment then mean? I meant the same thing as what Pavan wrote, but had a language fail: ARMCI gives per-operation local completion for non-blocking operations. Blocking operations are locally complete when they return. Bulk remote completion is accomplished through fence. ~Jim.

1 0

Re: [Mpi3-rma] Draft Plenary Presentation for MPI RMA
by James Dinan 04 Feb '11

04 Feb '11

On 02/04/2011 03:11 PM, Torsten Hoefler wrote: > On Fri, Feb 04, 2011 at 02:45:42PM -0600, James Dinan wrote: >> Hi Torsten, >> >> On 02/04/2011 12:35 PM, Torsten Hoefler wrote: >>>> Slide 22: UPC doesn't allow sharing data on the stack, only heap data >>>> can be shared. In general, this is kind of an iffy thing, might >>>> encounter some resistance to suggest doing this. Linux is going to >>>> initially map the stack to the zero page which is marked copy-on-write, >>>> so beyond the parts you have used, the stack doesn't exist. >>> >>> That's what registration is for. It has to be created then. Or am I >>> missing something? >> >> UPC doesn't allow shared variables on the stack and I can't think of any >> other PGAS model that does. So, the argument of registering and >> exposing the stack doesn't seem like it holds water to me. > Well, the current proposal allows this. One can register anything that > can be dereferenced, including stack variables. If you think that should > not be allowed than we need to fix the proposal. The slides simply > explain the proposal. I don't think this should be forbidden. I just think it's a bad idea to do it and we should not present it as a motivation for the work. It's also not needed by UPC. >> I believe Dan, et al.'s issue with MPI_Alloc_mem as a requirement is >> that *static* shared UPC objects could not be put into windows. Static >> distributed shared arrays are one of the most common use cases in UPC. >> Adding win_register now allows you to register statically declared >> arrays and expose them in a window. > This is allowed too. I would prefer to see static arrays used as the motivating example on slide 22 instead of the stack. It's needed by UPC and helpful in general to C/Fortran programmers. ~Jim.

1 0

Re: [Mpi3-rma] Draft Plenary Presentation for MPI RMA
by Torsten Hoefler 04 Feb '11

04 Feb '11

On Fri, Feb 04, 2011 at 02:45:42PM -0600, James Dinan wrote: > Hi Torsten, > > On 02/04/2011 12:35 PM, Torsten Hoefler wrote: >>> Slide 22: UPC doesn't allow sharing data on the stack, only heap data >>> can be shared. In general, this is kind of an iffy thing, might >>> encounter some resistance to suggest doing this. Linux is going to >>> initially map the stack to the zero page which is marked copy-on-write, >>> so beyond the parts you have used, the stack doesn't exist. > > >> That's what registration is for. It has to be created then. Or am I >> missing something? > > UPC doesn't allow shared variables on the stack and I can't think of any > other PGAS model that does. So, the argument of registering and > exposing the stack doesn't seem like it holds water to me. Well, the current proposal allows this. One can register anything that can be dereferenced, including stack variables. If you think that should not be allowed than we need to fix the proposal. The slides simply explain the proposal. > I believe Dan, et al.'s issue with MPI_Alloc_mem as a requirement is > that *static* shared UPC objects could not be put into windows. Static > distributed shared arrays are one of the most common use cases in UPC. > Adding win_register now allows you to register statically declared > arrays and expose them in a window. This is allowed too. Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [Mpi3-rma] Draft Plenary Presentation for MPI RMA
by Torsten Hoefler 04 Feb '11

04 Feb '11

On Fri, Feb 04, 2011 at 02:35:45PM -0600, Pavan Balaji wrote: > > On 02/04/2011 01:14 PM, Torsten Hoefler wrote: >>> Slide 27: ARMCI has per-operation non-blocking calls. Blocking calls >>> and waits give you local completion and only fence gives you remote >>> completion. >> Yes, ARMCI has per operation remote completion which we don't support at >> this point. Proposal 2 had this a while ago but we didn't find a strong >> use-case (as opposed to the per-operation local completion). Added a >> comment. > > No, ARMCI doesn't have per operation remote completion; only local > completion. Attached the slides I had distributed earlier about this for > your reference. Ok, I got this impression from our previous discussion. So Jim, what did your comment then mean? Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [Mpi3-rma] Draft Plenary Presentation for MPI RMA
by James Dinan 04 Feb '11

04 Feb '11

Hi Torsten, On 02/04/2011 12:35 PM, Torsten Hoefler wrote: >> Slide 22: UPC doesn't allow sharing data on the stack, only heap data >> can be shared. In general, this is kind of an iffy thing, might >> encounter some resistance to suggest doing this. Linux is going to >> initially map the stack to the zero page which is marked copy-on-write, >> so beyond the parts you have used, the stack doesn't exist. > > That's what registration is for. It has to be created then. Or am I > missing something? UPC doesn't allow shared variables on the stack and I can't think of any other PGAS model that does. So, the argument of registering and exposing the stack doesn't seem like it holds water to me. I believe Dan, et al.'s issue with MPI_Alloc_mem as a requirement is that *static* shared UPC objects could not be put into windows. Static distributed shared arrays are one of the most common use cases in UPC. Adding win_register now allows you to register statically declared arrays and expose them in a window. ~Jim.

1 0

Re: [Mpi3-rma] Draft Plenary Presentation for MPI RMA
by Pavan Balaji 04 Feb '11

04 Feb '11

On 02/04/2011 01:14 PM, Torsten Hoefler wrote: >> Slide 27: ARMCI has per-operation non-blocking calls. Blocking calls >> and waits give you local completion and only fence gives you remote >> completion. > Yes, ARMCI has per operation remote completion which we don't support at > this point. Proposal 2 had this a while ago but we didn't find a strong > use-case (as opposed to the per-operation local completion). Added a > comment. No, ARMCI doesn't have per operation remote completion; only local completion. Attached the slides I had distributed earlier about this for your reference. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] Draft Plenary Presentation for MPI RMA
by Torsten Hoefler 04 Feb '11

04 Feb '11

Hi Jim, We addressed the issues below, see inline comments! > These look good. Here's some feedback: > > Slide 4: RAW (Read after Write), WAR, WAW are commonly used terms for > this, should we use those? The group has a veto-time until tomorrow. It'll be updated if nobody complains. > Slide 21: UPC allows one-sided allocation of distributed shared data > through the one-sided call "shared void *upc_global_alloc(size_t > nblocks, size_t nbytes)". I think Dan was pointing out that this is > impossible in MPI-2. (Still don't think we can support this in MPI-3.) Added a comment, not sure if there is a real application use-case (I can imagine some though). > Slide 22: UPC doesn't allow sharing data on the stack, only heap data > can be shared. In general, this is kind of an iffy thing, might > encounter some resistance to suggest doing this. Linux is going to > initially map the stack to the zero page which is marked copy-on-write, > so beyond the parts you have used, the stack doesn't exist. Also as a sub-bullet. > Slide 27: ARMCI has per-operation non-blocking calls. Blocking calls > and waits give you local completion and only fence gives you remote > completion. Yes, ARMCI has per operation remote completion which we don't support at this point. Proposal 2 had this a while ago but we didn't find a strong use-case (as opposed to the per-operation local completion). Added a comment. > Slide 30: Conflicting relaxed operations issued by a single source will > appear in program order when observed by that source. IIRC, a third > party need not observe this same order. I think that the result of > conflicting relaxed operations issued from different sources is > undefined. Added a clarifying comment. Please check the new version! Thanks & Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [Mpi3-rma] Draft Plenary Presentation for MPI RMA
by Torsten Hoefler 04 Feb '11

04 Feb '11

Hi Jim, > These look good. Here's some feedback: > > Slide 4: RAW (Read after Write), WAR, WAW are commonly used terms for > this, should we use those? This is my terminology. We can switch to this is nobody objects. I'll also update the proposal. Btw., the renaming from MPI_RMA_query to MPI_Win_query is still pending. If I don't hear anything by *tonight*, I will update the proposal tomorrow with the two changes above. > Slide 21: UPC allows one-sided allocation of distributed shared data > through the one-sided call "shared void *upc_global_alloc(size_t > nblocks, size_t nbytes)". I think Dan was pointing out that this is > impossible in MPI-2. (Still don't think we can support this in MPI-3.) You're right, allocation is either local or collective, never remote. > Slide 22: UPC doesn't allow sharing data on the stack, only heap data > can be shared. In general, this is kind of an iffy thing, might > encounter some resistance to suggest doing this. Linux is going to > initially map the stack to the zero page which is marked copy-on-write, > so beyond the parts you have used, the stack doesn't exist. That's what registration is for. It has to be created then. Or am I missing something? > Slide 30: Conflicting relaxed operations issued by a single source will > appear in program order when observed by that source. IIRC, a third > party need not observe this same order. I think that the result of > conflicting relaxed operations issued from different sources is > undefined. Yes Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0