December 2011 - mpiwg-rma - mailman3.cels.anl.gov

Re: [Mpi3-rma] Next RMA Telecon
by Pavan Balaji 26 Dec '11

26 Dec '11

On 12/26/2011 11:15 PM, Torsten Hoefler wrote: > On Mon, Dec 26, 2011 at 10:49:55PM -0600, Pavan Balaji wrote: > Right, Brian had some arguments why/how this would be either a > non-scalable implementation (in memory and time) or a burden on the > implementer. So we didn't want to enforce to have a good implementation > of this (since our only use-case was solved otherwise). Sure. I don't feel strongly about allowing exclusive locks right now. I just wanted to get a better understanding of the issue. But I do think that the lock_type argument should be retained, which is an orthogonal issue. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] Next RMA Telecon
by Torsten Hoefler 26 Dec '11

26 Dec '11

On Mon, Dec 26, 2011 at 10:49:55PM -0600, Pavan Balaji wrote: > > On 12/26/2011 10:45 PM, Torsten Hoefler wrote: >> No, the MPI library does not have to deal with silly use programs. If a >> user decides to write a non-scalable program, then the MPI library >> cannot do much. > > My question is about correctness, not performance scalability. It can > be slow, but it has to work. Also, if the number of processes is small, > it's not really a scalability issue either. I see, I saw only the scalability issue. Yes, it is a correct program. > My point is that, the user can still do this, so the MPI implementation > has to deal with it. So not providing Win_lock_all(EXCLUSIVE) is not > really reducing the implementation complexity. The lock_all call does > give the opportunity to make the implementation more scalable (and > possibly more complex), but that's a quality of implementation issue. > > So, what am I missing here? Right, Brian had some arguments why/how this would be either a non-scalable implementation (in memory and time) or a burden on the implementer. So we didn't want to enforce to have a good implementation of this (since our only use-case was solved otherwise). Thanks & Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [Mpi3-rma] Next RMA Telecon
by Pavan Balaji 26 Dec '11

26 Dec '11

On 12/26/2011 10:45 PM, Torsten Hoefler wrote: > No, the MPI library does not have to deal with silly use programs. If a > user decides to write a non-scalable program, then the MPI library > cannot do much. My question is about correctness, not performance scalability. It can be slow, but it has to work. Also, if the number of processes is small, it's not really a scalability issue either. My point is that, the user can still do this, so the MPI implementation has to deal with it. So not providing Win_lock_all(EXCLUSIVE) is not really reducing the implementation complexity. The lock_all call does give the opportunity to make the implementation more scalable (and possibly more complex), but that's a quality of implementation issue. So, what am I missing here? -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] Next RMA Telecon
by Torsten Hoefler 26 Dec '11

26 Dec '11

On Mon, Dec 26, 2011 at 10:36:28PM -0600, Pavan Balaji wrote: > > On 12/26/2011 10:31 PM, Torsten Hoefler wrote: >>> Also, the case I had in mind was using lock/unlock. If it is valid for >>> the user to do a for loop of Win_lock(exclusive), I'm not sure how we >>> are taking away the MPI implementation burden by not allowing >>> Win_lock_all(exclusive). >> A for loop is a silly user program, having a slow P^2 lock_all >> exclusive (in the extreme case) is a problem in the MPI lib. > > True. Hence the need for a lock_all(EXCLUSIVE). But if that's not > provided, there should at least be a fall back option, and the for loop > is that. > > Besides, silly or not, it's a valid MPI program and the MPI > implementation has to deal with it. No, the MPI library does not have to deal with silly use programs. If a user decides to write a non-scalable program, then the MPI library cannot do much. >>> More importantly -- even if we want to not allow exclusive locks in >>> WIN_LOCK_ALL, I would suggest retaining the lock_type parameter and say >>> that the behavior of EXCLUSIVE locks is undefined, so that it's future >>> proof in case we want to add it later. Also, implementations that want >>> to provide it can do so without random hacks or new API functions. >> Yes, I thought about this, however, our original #270 didn't have this >> feature and I decided to revert to the original. We can do a straw vote >> here and fix it quickly. I have no strong opinion. > > I'd prefer a straw vote on this. I don't see any additional burden on > implementors who don't want to allow it. But it's much cleaner for > implementations that decide to allow it. > >>> The idea is that a process gets a lock on a shared memory region and >>> uses it for remote RMA communication. >> Yes, that should work but the semantics of overlapping windows are >> hairy. > > I think they are quite well defined, and fairly intuitive once you think > through it. But that's just personal opinion. I agree, but I fear that 98% of our user-base (the people who did not spent dozens of hours discussing the semantics :-) ) may think differently (remember that there was a large and credible group in the Forum that wanted to completely abandon those semantics just two years ago :-) ). All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- "All that we need to make us really happy is something to be enthusiastic about." -- Charles Kingsley

1 0

Re: [Mpi3-rma] Next RMA Telecon
by Pavan Balaji 26 Dec '11

26 Dec '11

On 12/26/2011 10:31 PM, Torsten Hoefler wrote: >> Also, the case I had in mind was using lock/unlock. If it is valid for >> the user to do a for loop of Win_lock(exclusive), I'm not sure how we >> are taking away the MPI implementation burden by not allowing >> Win_lock_all(exclusive). > A for loop is a silly user program, having a slow P^2 lock_all > exclusive (in the extreme case) is a problem in the MPI lib. True. Hence the need for a lock_all(EXCLUSIVE). But if that's not provided, there should at least be a fall back option, and the for loop is that. Besides, silly or not, it's a valid MPI program and the MPI implementation has to deal with it. >> More importantly -- even if we want to not allow exclusive locks in >> WIN_LOCK_ALL, I would suggest retaining the lock_type parameter and say >> that the behavior of EXCLUSIVE locks is undefined, so that it's future >> proof in case we want to add it later. Also, implementations that want >> to provide it can do so without random hacks or new API functions. > Yes, I thought about this, however, our original #270 didn't have this > feature and I decided to revert to the original. We can do a straw vote > here and fix it quickly. I have no strong opinion. I'd prefer a straw vote on this. I don't see any additional burden on implementors who don't want to allow it. But it's much cleaner for implementations that decide to allow it. >> The idea is that a process gets a lock on a shared memory region and >> uses it for remote RMA communication. > Yes, that should work but the semantics of overlapping windows are > hairy. I think they are quite well defined, and fairly intuitive once you think through it. But that's just personal opinion. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] Next RMA Telecon
by Torsten Hoefler 26 Dec '11

26 Dec '11

On Mon, Dec 26, 2011 at 09:00:59PM -0600, Pavan Balaji wrote: > > On 12/26/2011 08:48 PM, Torsten Hoefler wrote: >> Why does: >> >> win_fence() >> if(rank == 0) { initialize_window(); MPI_Barrier(); } >> else MPI_Barrier(); >> win_fence() >> // all procs read written data here >> >> not work for Ron's use-case? Are you talking about another use-case? > > I see. I thought you were referring to using point-to-point/collectives > alone without any RMA calls. > > Also, the case I had in mind was using lock/unlock. If it is valid for > the user to do a for loop of Win_lock(exclusive), I'm not sure how we > are taking away the MPI implementation burden by not allowing > Win_lock_all(exclusive). A for loop is a silly user program, having a slow P^2 lock_all exclusive (in the extreme case) is a problem in the MPI lib. > More importantly -- even if we want to not allow exclusive locks in > WIN_LOCK_ALL, I would suggest retaining the lock_type parameter and say > that the behavior of EXCLUSIVE locks is undefined, so that it's future > proof in case we want to add it later. Also, implementations that want > to provide it can do so without random hacks or new API functions. Yes, I thought about this, however, our original #270 didn't have this feature and I decided to revert to the original. We can do a straw vote here and fix it quickly. I have no strong opinion. > Btw, I assume the following would be valid? > > Win_allocate_shared(shared_mem, &shared_win); > Win_create(shared_mem, &remote_win); /* overlapping window */ > > P0: > Win_lock_all(SHARED, &shared_win); > Recv(P1); > Barrier(); > > ALL_OTHER_PROCESSES_SHARING_SHARED_WIN: > Barrier(); > > REMOTE_PROCESS: > Win_lock(EXCLUSIVE, P0, remote_win); > Put(P0, remote_win); > Win_unlock(P0); > Send(P0); > > The idea is that a process gets a lock on a shared memory region and > uses it for remote RMA communication. Yes, that should work but the semantics of overlapping windows are hairy. All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [Mpi3-rma] Next RMA Telecon
by Pavan Balaji 26 Dec '11

26 Dec '11

On 12/26/2011 08:48 PM, Torsten Hoefler wrote: > Why does: > > win_fence() > if(rank == 0) { initialize_window(); MPI_Barrier(); } > else MPI_Barrier(); > win_fence() > // all procs read written data here > > not work for Ron's use-case? Are you talking about another use-case? I see. I thought you were referring to using point-to-point/collectives alone without any RMA calls. Also, the case I had in mind was using lock/unlock. If it is valid for the user to do a for loop of Win_lock(exclusive), I'm not sure how we are taking away the MPI implementation burden by not allowing Win_lock_all(exclusive). More importantly -- even if we want to not allow exclusive locks in WIN_LOCK_ALL, I would suggest retaining the lock_type parameter and say that the behavior of EXCLUSIVE locks is undefined, so that it's future proof in case we want to add it later. Also, implementations that want to provide it can do so without random hacks or new API functions. Btw, I assume the following would be valid? Win_allocate_shared(shared_mem, &shared_win); Win_create(shared_mem, &remote_win); /* overlapping window */ P0: Win_lock_all(SHARED, &shared_win); Recv(P1); Barrier(); ALL_OTHER_PROCESSES_SHARING_SHARED_WIN: Barrier(); REMOTE_PROCESS: Win_lock(EXCLUSIVE, P0, remote_win); Put(P0, remote_win); Win_unlock(P0); Send(P0); The idea is that a process gets a lock on a shared memory region and uses it for remote RMA communication. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] Next RMA Telecon
by Torsten Hoefler 26 Dec '11

26 Dec '11

On Mon, Dec 26, 2011 at 08:40:41PM -0600, Pavan Balaji wrote: > > On 12/26/2011 12:12 PM, Torsten Hoefler wrote: >> On Mon, Dec 26, 2011 at 12:22:38AM -0600, Pavan Balaji wrote: >>> >>> On 12/09/2011 03:36 PM, Torsten Hoefler wrote: >>>>> 3) allocate_shared proposal with MPI_Win_lock_all(exclusive) >>>> - remove MPI_Win_lock_all(exclusive) from the proposal and re-read >>> >>> Why was this removed? >> Brian and others felt that an implementation on a distributed-memory >> network would put a huge burden on the implementers and may cause other >> problems. A workaround offering essentially the same functionality by >> using point-to-point and collective synchronization (i.e., block p-1 >> processes in a barrier while one process updates the shared memory) was >> identified. > > Didn't we already discuss this in one of the working groups (though I > forget whether it was RMA or hybrid)? If I recall correctly, a non-RMA > solution was shot down because it doesn't provide the memory barriers > that a lock/unlock would do. > > Someone also pointed out that we could do a for loop of > Win_lock(exclusive) to achieve the same result, but it was not scalable. Why does: win_fence() if(rank == 0) { initialize_window(); MPI_Barrier(); } else MPI_Barrier(); win_fence() // all procs read written data here not work for Ron's use-case? Are you talking about another use-case? All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [Mpi3-rma] Next RMA Telecon
by Pavan Balaji 26 Dec '11

26 Dec '11

On 12/26/2011 12:12 PM, Torsten Hoefler wrote: > On Mon, Dec 26, 2011 at 12:22:38AM -0600, Pavan Balaji wrote: >> >> On 12/09/2011 03:36 PM, Torsten Hoefler wrote: >>>> 3) allocate_shared proposal with MPI_Win_lock_all(exclusive) >>> - remove MPI_Win_lock_all(exclusive) from the proposal and re-read >> >> Why was this removed? > Brian and others felt that an implementation on a distributed-memory > network would put a huge burden on the implementers and may cause other > problems. A workaround offering essentially the same functionality by > using point-to-point and collective synchronization (i.e., block p-1 > processes in a barrier while one process updates the shared memory) was > identified. Didn't we already discuss this in one of the working groups (though I forget whether it was RMA or hybrid)? If I recall correctly, a non-RMA solution was shot down because it doesn't provide the memory barriers that a lock/unlock would do. Someone also pointed out that we could do a for loop of Win_lock(exclusive) to achieve the same result, but it was not scalable. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-rma] Next RMA Telecon
by Torsten Hoefler 26 Dec '11

26 Dec '11

On Mon, Dec 26, 2011 at 12:22:38AM -0600, Pavan Balaji wrote: > > On 12/09/2011 03:36 PM, Torsten Hoefler wrote: >>> 3) allocate_shared proposal with MPI_Win_lock_all(exclusive) >> - remove MPI_Win_lock_all(exclusive) from the proposal and re-read > > Why was this removed? Brian and others felt that an implementation on a distributed-memory network would put a huge burden on the implementers and may cause other problems. A workaround offering essentially the same functionality by using point-to-point and collective synchronization (i.e., block p-1 processes in a barrier while one process updates the shared memory) was identified. All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0