February 2011 - mpiwg-hybridpm - mailman3.cels.anl.gov

Re: [Mpi3-hybridpm] New draft of the EI chapter uploaded
by Pavan Balaji 09 Feb '11

09 Feb '11

Hi Doug, On 02/09/2011 09:23 AM, Douglas Miller wrote: > Maybe we really are saying the same thing. The talk about making all MPI > call synchronzise might be throwing me off. I was thinking of this (the > 'balanced' INFO var) as being information that the user was giving to > the implementation telling it that the code between JOIN and LEAVE was > taking a "deliberate" approach towards progress, and that the user was > saying that they understand that the performance of their JOIN-LEAVE > block will directly relate to how much/frequently they make MPI calls > (or get to LEAVE). Not sure what the right words are, but I feel we must > make it more obvious to users why they want to use 'balanced' and how > they should select what code to put between JOIN and LEAVE in that case. Note that the synchronization between the threads can happen only when all threads are within MPI calls. So, I'm saying the same thing, but I agree the wording is not very clear. But we should avoid saying that you can *only* make MPI calls within JOIN/LEAVE. As long as the user is aware there *might be* a synchronization step between threads, (s)he can decide whether it's the right model for the application or not. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] New draft of the EI chapter uploaded
by Douglas Miller 09 Feb '11

09 Feb '11

Hi Pavan, Maybe we really are saying the same thing. The talk about making all MPI call synchronzise might be throwing me off. I was thinking of this (the 'balanced' INFO var) as being information that the user was giving to the implementation telling it that the code between JOIN and LEAVE was taking a "deliberate" approach towards progress, and that the user was saying that they understand that the performance of their JOIN-LEAVE block will directly relate to how much/frequently they make MPI calls (or get to LEAVE). Not sure what the right words are, but I feel we must make it more obvious to users why they want to use 'balanced' and how they should select what code to put between JOIN and LEAVE in that case. _______________________________________________ Douglas Miller BlueGene Messaging Development IBM Corp., Rochester, MN USA Bldg 030-2 A410 dougmill(a)us.ibm.com Douglas Miller/Rochester/IBM Pavan Balaji <balaji(a)mcs.anl.g ov> To Douglas Miller/Rochester/IBM@IBMUS 02/09/2011 09:05 cc AM mpi3-hybridpm(a)lists.mpi-forum.org Subject Re: [Mpi3-hybridpm] New draft of the EI chapter uploaded Hi Doug, On 02/09/2011 08:54 AM, Douglas Miller wrote: > The problem is that none of these INFO options represent the mode I was > discussing in our last meeting. These are different interpretations of > how helper threads will be used and implemented. > > I am not at all comfortable with this concept of adding synchronization > to all MPI calls. In fact, I'm not at all sure what it means to set > balanced to true and then go into computation. How do computation > threads get synchronized in those cases? It sounds the same as > balanced=false. This is based on what we had discussed in the call -- we even had a discussion on why the term was called "balanced" and Bronis clarified that the computation was balanced across threads allowing the MPI implementation to use all threads for any MPI call (this "using all threads" is what I'm calling "synchronizing"). Marc had also pointed out that each thread should independently be able to make progress to MPI_HELPER_TEAM_LEAVE even if a thread is expecting help from other threads thus requiring synchronization between them (e.g., you cannot wait on a pthread_barrier). Even this part is captured in the description that the threads might synchronize with other threads during an MPI call. I don't think we are saying two different things -- it's just a word-smithing issue but I think your description is restricting it more than you require by saying that the application can only call MPI operations. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] New draft of the EI chapter uploaded
by Pavan Balaji 09 Feb '11

09 Feb '11

Hi Doug, On 02/09/2011 08:54 AM, Douglas Miller wrote: > The problem is that none of these INFO options represent the mode I was > discussing in our last meeting. These are different interpretations of > how helper threads will be used and implemented. > > I am not at all comfortable with this concept of adding synchronization > to all MPI calls. In fact, I'm not at all sure what it means to set > balanced to true and then go into computation. How do computation > threads get synchronized in those cases? It sounds the same as > balanced=false. This is based on what we had discussed in the call -- we even had a discussion on why the term was called "balanced" and Bronis clarified that the computation was balanced across threads allowing the MPI implementation to use all threads for any MPI call (this "using all threads" is what I'm calling "synchronizing"). Marc had also pointed out that each thread should independently be able to make progress to MPI_HELPER_TEAM_LEAVE even if a thread is expecting help from other threads thus requiring synchronization between them (e.g., you cannot wait on a pthread_barrier). Even this part is captured in the description that the threads might synchronize with other threads during an MPI call. I don't think we are saying two different things -- it's just a word-smithing issue but I think your description is restricting it more than you require by saying that the application can only call MPI operations. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] New draft of the EI chapter uploaded
by Douglas Miller 09 Feb '11

09 Feb '11

The problem is that none of these INFO options represent the mode I was discussing in our last meeting. These are different interpretations of how helper threads will be used and implemented. I am not at all comfortable with this concept of adding synchronization to all MPI calls. In fact, I'm not at all sure what it means to set balanced to true and then go into computation. How do computation threads get synchronized in those cases? It sounds the same as balanced=false. _______________________________________________ Douglas Miller BlueGene Messaging Development IBM Corp., Rochester, MN USA Bldg 030-2 A410 dougmill(a)us.ibm.com Douglas Miller/Rochester/IBM Pavan Balaji <balaji(a)mcs.anl.g ov> To mpi3-hybridpm(a)lists.mpi-forum.org 02/08/2011 05:38 cc PM Douglas Miller/Rochester/IBM@IBMUS Subject Re: [Mpi3-hybridpm] New draft of the EI chapter uploaded Hi Doug, On 02/07/2011 07:31 AM, Douglas Miller wrote: > *balanced *- if set to true, the MPI implementation may assume that all > threads in the team will actively participate by making MPI calls (or > calling MPI_HELPER_TEAM_LEAVE). If false, the implementation must allow > for some threads spending the bulk of their time performing computation > or other non-MPI activities. With respect to the first point, threads don't need to "actively participate" from the user's perspective. The user just needs to be aware that all MPI calls might synchronize all or some of the threads. For the second point, I think this is incorrect. The threads spending the bulk of their time in computation is fine even in "balanced" mode. If the user's code is willing to accept the "synchronization overhead" during the MPI call, that's fine. For example, we already discussed that the following is valid: MPI_Helper_team_join() compute(random_time) pthread_barrier(); if (thread == 0) { MPI_SOMETHING(); } else { MPI_SOMETHING_ELSE(); } MPI_Helper_team_leave() But the following might be valid too (pthread_barrier is gone): MPI_Helper_team_join() compute(random_time) if (thread == 0) { MPI_SOMETHING(); } else { MPI_SOMETHING_ELSE(); } MPI_Helper_team_leave() In the second example, if the MPI implementation internally decides to synchronize the threads, that's fine, since the application would have done that anyway. This should be left to the user. The only thing that will be invalid is: MPI_Helper_team_join() compute(random_time) if (thread == 0) { MPI_SOMETHING(); } pthread_barrier(); MPI_Helper_team_leave() This is invalid because the MPI implementation is expecting to synchronize with other threads during the MPI_SOMETHING call, but it can't as all other threads are blocked in pthread_barrier(). > ...also, wouldn't setting 'singular' to true also imply/force 'balanced' > to true? It almost seems like we have a three-state value rather than > two booleans. From strict to weak: (singular), (!singular,balanced), > (!singular,!balanced). I'm not clear just how much difference there is > between singular (true) and balanced (true). Maybe we need only 'balanced'? Yes, 'singular' is a superset of 'balanced'. I'll point that out. But as we discussed in the telecon, 'singular' can give more benefit for implementations as they know that other MPI calls will not be scheduled, so it's easier to optimize. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] New draft of the EI chapter uploaded
by Pavan Balaji 08 Feb '11

08 Feb '11

Hi Doug, On 02/07/2011 07:31 AM, Douglas Miller wrote: > *balanced *- if set to true, the MPI implementation may assume that all > threads in the team will actively participate by making MPI calls (or > calling MPI_HELPER_TEAM_LEAVE). If false, the implementation must allow > for some threads spending the bulk of their time performing computation > or other non-MPI activities. With respect to the first point, threads don't need to "actively participate" from the user's perspective. The user just needs to be aware that all MPI calls might synchronize all or some of the threads. For the second point, I think this is incorrect. The threads spending the bulk of their time in computation is fine even in "balanced" mode. If the user's code is willing to accept the "synchronization overhead" during the MPI call, that's fine. For example, we already discussed that the following is valid: MPI_Helper_team_join() compute(random_time) pthread_barrier(); if (thread == 0) { MPI_SOMETHING(); } else { MPI_SOMETHING_ELSE(); } MPI_Helper_team_leave() But the following might be valid too (pthread_barrier is gone): MPI_Helper_team_join() compute(random_time) if (thread == 0) { MPI_SOMETHING(); } else { MPI_SOMETHING_ELSE(); } MPI_Helper_team_leave() In the second example, if the MPI implementation internally decides to synchronize the threads, that's fine, since the application would have done that anyway. This should be left to the user. The only thing that will be invalid is: MPI_Helper_team_join() compute(random_time) if (thread == 0) { MPI_SOMETHING(); } pthread_barrier(); MPI_Helper_team_leave() This is invalid because the MPI implementation is expecting to synchronize with other threads during the MPI_SOMETHING call, but it can't as all other threads are blocked in pthread_barrier(). > ...also, wouldn't setting 'singular' to true also imply/force 'balanced' > to true? It almost seems like we have a three-state value rather than > two booleans. From strict to weak: (singular), (!singular,balanced), > (!singular,!balanced). I'm not clear just how much difference there is > between singular (true) and balanced (true). Maybe we need only 'balanced'? Yes, 'singular' is a superset of 'balanced'. I'll point that out. But as we discussed in the telecon, 'singular' can give more benefit for implementations as they know that other MPI calls will not be scheduled, so it's easier to optimize. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] New draft of the EI chapter uploaded
by Douglas Miller 07 Feb '11

07 Feb '11

Just a couple comments on new things added. The text on the MPI_HELPER_TEAM_CREATE INFO seems unclear. Here's some proposed alternate text: balanced - if set to true, the MPI implementation may assume that all threads in the team will actively participate by making MPI calls (or calling MPI_HELPER_TEAM_LEAVE). If false, the implementation must allow for some threads spending the bulk of their time performing computation or other non-MPI activities. ...also, wouldn't setting 'singular' to true also imply/force 'balanced' to true? It almost seems like we have a three-state value rather than two booleans. From strict to weak: (singular), (!singular,balanced), (!singular,!balanced). I'm not clear just how much difference there is between singular (true) and balanced (true). Maybe we need only 'balanced'? ------- Also, I think MPI_HELPER_TEAM_YIELD could be clearer. Maybe: This call allows a thread to actively participate in the team while performing other activities not related to MPI. The frequency with which a thread calls this function affects how much the thread is able to participate. This function is typically needed only if MPI_HELPER_TEAM_CREATE was called with INFO having balanced and singular set to false (or MPI_INFO_NULL). (or is that last sentence an "advice to users"?) ------- thanks, _______________________________________________ Douglas Miller BlueGene Messaging Development IBM Corp., Rochester, MN USA Bldg 030-2 A410 dougmill(a)us.ibm.com Douglas Miller/Rochester/IBM Pavan Balaji <balaji(a)mcs.anl.g ov> To Sent by: mpi3-hybridpm(a)lists.mpi-forum.org mpi3-hybridpm-bou cc nces(a)lists.mpi-fo rum.org Subject [Mpi3-hybridpm] New draft of the EI chapter uploaded 02/05/2011 05:58 PM Please respond to mpi3-hybridpm@lis ts.mpi-forum.org All, I've uploaded v0.9 of the EI chapter to the wiki. This is close to the version we will be presenting to the Forum. Changes at this point, if any, will be minor (e.g., some more description of the functionality, etc.). Thanks, -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji _______________________________________________ Mpi3-hybridpm mailing list Mpi3-hybridpm(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm

1 0

Re: [Mpi3-hybridpm] New draft of the EI chapter uploaded
by Pavan Balaji 06 Feb '11

06 Feb '11

Hi Rajeev, On 02/06/2011 06:25 PM, Rajeev Thakur wrote: > MPI_Comm_shm_alloc probably needs an Advice to Users about the input > communicator argument to this function. Just passing any communicator > (e.g. MPI_COMM_WORLD) may not result in successful shared memory > allocation. One use case is the communicator should represent > processes on the same node. Other use cases could be implementation > defined. Not sure of the exact language to put here, but it would be > useful to make users aware of this issue in some way, rather than say > nothing at all. Right. There was also a suggestion that we provide a new predefined communicator called (MPI_COMM_NODE or MPI_COMM_SHM) that allow shared memory regions to be created on them, their dup's or subsets. We didn't decide on what the right thing to do is (could be as simple as an advice to users and say that MPI does not handle it). I'll add a discussion item for it. > This proposal is a shrunk version of earlier proposals, with only > helper threads and shared memory allocation functions included. > Endpoints are not included. Is the endpoints proposal going to be put > forth separately later? I hope it is in good shape because it was a > promising idea. The endpoints proposal is not removed. We were just bandwidth limited to bring text for all parts to this particular Forum meeting. We'll be adding it as soon as this meeting is done, hopefully by the next one. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi3-hybridpm] New draft of the EI chapter uploaded
by Rajeev Thakur 06 Feb '11

06 Feb '11

MPI_Comm_shm_alloc probably needs an Advice to Users about the input communicator argument to this function. Just passing any communicator (e.g. MPI_COMM_WORLD) may not result in successful shared memory allocation. One use case is the communicator should represent processes on the same node. Other use cases could be implementation defined. Not sure of the exact language to put here, but it would be useful to make users aware of this issue in some way, rather than say nothing at all. This proposal is a shrunk version of earlier proposals, with only helper threads and shared memory allocation functions included. Endpoints are not included. Is the endpoints proposal going to be put forth separately later? I hope it is in good shape because it was a promising idea. Rajeev On Feb 5, 2011, at 5:58 PM, Pavan Balaji wrote: > All, > > I've uploaded v0.9 of the EI chapter to the wiki. This is close to the version we will be presenting to the Forum. Changes at this point, if any, will be minor (e.g., some more description of the functionality, etc.). > > Thanks, > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > _______________________________________________ > Mpi3-hybridpm mailing list > Mpi3-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm

1 0

[Mpi3-hybridpm] Webex information for Hybrid plenary
by Pavan Balaji 06 Feb '11

06 Feb '11

Hi all, Attached is the webex information if you plan to call in for the Hybrid plenary on Wednesday (9AM to 11.30AM pacific). Thanks, -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

[Mpi3-hybridpm] New draft of the EI chapter uploaded
by Pavan Balaji 05 Feb '11

05 Feb '11

All, I've uploaded v0.9 of the EI chapter to the wiki. This is close to the version we will be presenting to the Forum. Changes at this point, if any, will be minor (e.g., some more description of the functionality, etc.). Thanks, -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0