November 2010 - mpiwg-rma - mailman3.cels.anl.gov

Re: [Mpi3-rma] Updated Proposal 1
by Pavan Balaji 29 Nov '10

29 Nov '10

On 11/29/2010 10:06 AM, Barrett, Brian W wrote: > I think we could do multi-element with the very loose element-wise > guarantee. At one point, there was discussion about strengthening > that guarantee, which would make the multi-element Get_accumulate > painful. The complain that was raised against multi-element Get_accumulate was that of reliability. If the network adapter does not have any buffering capability, the target buffer cannot be modified till the original data has been delivered to the origin. FWIW, the only requirement I had raised for multi-element Get_accumulate was for the NO_OP operation, where this is not an issue (in other words, element-wise atomic GET operation). -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] Updated Proposal 1
by Barrett, Brian W 29 Nov '10

29 Nov '10

On Nov 26, 2010, at 3:47 PM, Torsten Hoefler wrote: > Jeff, >>> 2) Keith and Brian, could you please elaborate on the arguments against >>> allowing multiple elements (count>1) in MPI_Get_accumulate? I remember >>> there was some discussion about buffering and failures if one wanted to >>> support it in hardware but I don't remember what the issues were. It >>> seems like one could simply pipeline the hardware operations or just >>> fall back to a software implementation if count is bigger than a certain >>> threshold. >> >> With pipelining, I assume atomicity is only per element? I do not see >> how you could realize anything else in hardware. I don't see any >> value in multi-element RMW if it is element-wise atomic. One could >> just as easily send multiple messages. > Correct, the current RMA doesn't offer more than element-wise atomicity > and we don't plan to extend this. You are right, the only benefit would > lie in reduced overheads which isn't all that much. I'm not sure if > full-message atomicity would be easy/fast to implement. I think we could do multi-element with the very loose element-wise guarantee. At one point, there was discussion about strengthening that guarantee, which would make the multi-element Get_accumulate painful. Brian -- Brian W. Barrett Dept. 1423: Scalable System Software Sandia National Laboratories

1 0

Re: [Mpi3-rma] Updated Proposal 1
by Jeff Hammond 28 Nov '10

28 Nov '10

>> What is the use case multi-element RMW anyways? ï¿½I'm definitely going >> to need message-atomic put, but I'll do the portable i.e. MPI version >> with RMW-lock + put + RMW-unlock. > > For dynamic windows, a portable pointer will be something like <process, > displacement>. ï¿½If you want to build lock-free data structures, you'd want > to do swap on pointers. ï¿½If you can't do both atomically, then you will need > to do things differently anyway. ï¿½Providing a count for this function could > be misleading, I'd suggest some text to clarify. That requires multi-element i.e. message-level atomicity. As Torsten said earlier, "the current RMA doesn't offer more than element-wise atomicity and we don't plan to extend this." The appropriate way to do message-level atomicity is active-messages or RMW-lock + RMA + RMW-unlock. > Jeff, unless you need trylock there are more efficient ways to do one-sided > locking on top of existing MPI-2 RMA that don't require polling over the > network. ï¿½I can point you to code if you're interested, I think it's in an > MPICH example somewhere. I'm going to want lock, trylock and unlock, so if the approach you alludes to only provides lock/unlock, then it doesn't work for me. Besides, I'm not interested in what one can hack on top of MPI-2. This discussion should be about the right answer for the right reason within MPI-3. Jeff -- Jeff Hammond Argonne Leadership Computing Facility jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond

1 0

Re: [Mpi3-rma] Updated Proposal 1
by James Dinan 28 Nov '10

28 Nov '10

Hi Torsten, > 1) Pavan and I still need to talk about a good strategy to handle the > lock-free vs. lock-all/shared naming/syntax issues. I know this is the last thing we will discuss, but just to put it in writing before I forget: I think you called "lock-free" "conflict-free" in a telecon. I thought the latter name was clearer. Best, ~Jim.

1 0

Re: [Mpi3-rma] Updated Proposal 1
by James Dinan 28 Nov '10

28 Nov '10

On 11/26/10 4:37 PM, Jeff Hammond wrote: > What is the use case multi-element RMW anyways? I'm definitely going > to need message-atomic put, but I'll do the portable i.e. MPI version > with RMW-lock + put + RMW-unlock. For dynamic windows, a portable pointer will be something like <process, displacement>. If you want to build lock-free data structures, you'd want to do swap on pointers. If you can't do both atomically, then you will need to do things differently anyway. Providing a count for this function could be misleading, I'd suggest some text to clarify. Jeff, unless you need trylock there are more efficient ways to do one-sided locking on top of existing MPI-2 RMA that don't require polling over the network. I can point you to code if you're interested, I think it's in an MPICH example somewhere. ~Jim.

1 0

Re: [Mpi3-rma] Updated Proposal 1
by Pavan Balaji 26 Nov '10

26 Nov '10

Torsten, RMA Query still combines unified public/private window semantics with progress semantics. We discussed this in the last telecon. The progress semantics are only required if the target process wants to read data put by the origin without calling an MPI function. This has no use case especially since MPI_Put followed by a local load from the same location gives you garbage based on MPI semantics. Since I believe there is no plan to add such semantics, it'll be best to get rid of the progress part and only have RMA_QUERY as follows: MPI_RMA_QUERY(win, model) Thanks, -- Pavan On 11/26/2010 01:46 PM, Torsten Hoefler wrote: > Hi All, > > We updated the items in proposal 1 that we discussed at the last > telecon. The updated proposal is attached to the Wiki. > > There are some open issues: > > 1) Pavan and I still need to talk about a good strategy to handle the > lock-free vs. lock-all/shared naming/syntax issues. > > 2) Keith and Brian, could you please elaborate on the arguments against > allowing multiple elements (count>1) in MPI_Get_accumulate? I remember > there was some discussion about buffering and failures if one wanted to > support it in hardware but I don't remember what the issues were. It > seems like one could simply pipeline the hardware operations or just > fall back to a software implementation if count is bigger than a certain > threshold. > > Thanks& All the Best, > Torsten > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] Updated Proposal 1
by Jeff Hammond 26 Nov '10

26 Nov '10

Polling on CAS is lame, but unless MPI-3 is going to add a blocking lock acquisition call, then that's the only general solution. Jeff On Fri, Nov 26, 2010 at 4:47 PM, Torsten Hoefler <htor(a)illinois.edu> wrote: > Jeff, >> > 2) Keith and Brian, could you please elaborate on the arguments against >> > allowing multiple elements (count>1) in MPI_Get_accumulate? I remember >> > there was some discussion about buffering and failures if one wanted to >> > support it in hardware but I don't remember what the issues were. It >> > seems like one could simply pipeline the hardware operations or just >> > fall back to a software implementation if count is bigger than a certain >> > threshold. >> >> With pipelining, I assume atomicity is only per element? ï¿½I do not see >> how you could realize anything else in hardware. ï¿½I don't see any >> value in multi-element RMW if it is element-wise atomic. ï¿½One could >> just as easily send multiple messages. > Correct, the current RMA doesn't offer more than element-wise atomicity > and we don't plan to extend this. You are right, the only benefit would > lie in reduced overheads which isn't all that much. I'm not sure if > full-message atomicity would be easy/fast to implement. > >> What is the use case multi-element RMW anyways? ï¿½I'm definitely going >> to need message-atomic put, but I'll do the portable i.e. MPI version >> with RMW-lock + put + RMW-unlock. > That was the use-case. Manual lock/unlock (+flushes etc.) as you > proposed is a valid workaround but seems to impose additional roundtrips > (you'd have to poll on CAS). > > All the Best, > ï¿½Torsten > > -- > ï¿½bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- > Torsten Hoefler ï¿½ ï¿½ ï¿½ ï¿½ | Performance Modeling and Simulation Lead > Blue Waters Directorate | University of Illinois (UIUC) > 1205 W Clark Street ï¿½ ï¿½ | Urbana, IL, 61801 > NCSA Building ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ | +01 (217) 244-7736 > _______________________________________________ > mpi3-rma mailing list > mpi3-rma(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma > -- Jeff Hammond Argonne Leadership Computing Facility jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond

1 0

Re: [Mpi3-rma] Updated Proposal 1
by Torsten Hoefler 26 Nov '10

26 Nov '10

Jeff, > > 2) Keith and Brian, could you please elaborate on the arguments against > > allowing multiple elements (count>1) in MPI_Get_accumulate? I remember > > there was some discussion about buffering and failures if one wanted to > > support it in hardware but I don't remember what the issues were. It > > seems like one could simply pipeline the hardware operations or just > > fall back to a software implementation if count is bigger than a certain > > threshold. > > With pipelining, I assume atomicity is only per element? I do not see > how you could realize anything else in hardware. I don't see any > value in multi-element RMW if it is element-wise atomic. One could > just as easily send multiple messages. Correct, the current RMA doesn't offer more than element-wise atomicity and we don't plan to extend this. You are right, the only benefit would lie in reduced overheads which isn't all that much. I'm not sure if full-message atomicity would be easy/fast to implement. > What is the use case multi-element RMW anyways? I'm definitely going > to need message-atomic put, but I'll do the portable i.e. MPI version > with RMW-lock + put + RMW-unlock. That was the use-case. Manual lock/unlock (+flushes etc.) as you proposed is a valid workaround but seems to impose additional roundtrips (you'd have to poll on CAS). All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [Mpi3-rma] Updated Proposal 1
by Jeff Hammond 26 Nov '10

26 Nov '10

Torsten, > 2) Keith and Brian, could you please elaborate on the arguments against > allowing multiple elements (count>1) in MPI_Get_accumulate? I remember > there was some discussion about buffering and failures if one wanted to > support it in hardware but I don't remember what the issues were. It > seems like one could simply pipeline the hardware operations or just > fall back to a software implementation if count is bigger than a certain > threshold. With pipelining, I assume atomicity is only per element? I do not see how you could realize anything else in hardware. I don't see any value in multi-element RMW if it is element-wise atomic. One could just as easily send multiple messages. What is the use case multi-element RMW anyways? I'm definitely going to need message-atomic put, but I'll do the portable i.e. MPI version with RMW-lock + put + RMW-unlock. Jeff -- Jeff Hammond Argonne Leadership Computing Facility jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond

1 0

[Mpi3-rma] Updated Proposal 1
by Torsten Hoefler 26 Nov '10

26 Nov '10

Hi All, We updated the items in proposal 1 that we discussed at the last telecon. The updated proposal is attached to the Wiki. There are some open issues: 1) Pavan and I still need to talk about a good strategy to handle the lock-free vs. lock-all/shared naming/syntax issues. 2) Keith and Brian, could you please elaborate on the arguments against allowing multiple elements (count>1) in MPI_Get_accumulate? I remember there was some discussion about buffering and failures if one wanted to support it in hardware but I don't remember what the issues were. It seems like one could simply pipeline the hardware operations or just fall back to a software implementation if count is bigger than a certain threshold. Thanks & All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0