May 2014 - mpiwg-hybridpm - mailman3.cels.anl.gov

Re: [mpiwg-hybridpm] Telecon 5/19 11am CT
by Balaji, Pavan 16 May '14

16 May '14

Thanks, Jim. All, I’ll not be able to join the meeting this Monday since I’ll be running a workshop at IPDPS at that time. — Pavan On May 15, 2014, at 11:16 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: > Hi All, > > Our regular Hybrid telecon will be on Monday 5/19 at 11am central. Webex call-in information is on the wiki: > > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Hybrid > > In preparation for the upcoming Forum meeting, I would like to prepare an update on endpoints to deliver during the Hybrid WG session on Tuesday. Currently, I am tracking the following open items for endpoints. It would be great to touch on these briefly at the upcoming meeting. > • Endpoints query function. This was split out into ticket #425 as a general function to query whether MPI processes exist in the same address space. > • Discussion of communicator comparison routines in the presence of MPI processes that share an address space. This appears to also be trending toward a more general change to support models where processes share an address space. > • Collective semantic for MPI_Comm_free. Currently we are converging on removing the special case for MPI_Comm_free and potentially creating a separate operation that can be used to free endpoints in a non-THREAD_MULTIPLE environment. > We also have ticket #411 going forward for a formal reading at the next meeting. Monday is the deadline for changes, so please review. > > Jeff, are you planning to do anything with #381 communicator info keys (or has that moved to a different WG)? > > Any other agenda items? > > Thanks, > ~Jim. > _______________________________________________ > mpiwg-hybridpm mailing list > mpiwg-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-hybridpm

1 0

Telecon 5/19 11am CT
by Jim Dinan 15 May '14

15 May '14

Hi All, Our regular Hybrid telecon will be on Monday 5/19 at 11am central. Webex call-in information is on the wiki: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Hybrid In preparation for the upcoming Forum meeting, I would like to prepare an update on endpoints to deliver during the Hybrid WG session on Tuesday. Currently, I am tracking the following open items for endpoints. It would be great to touch on these briefly at the upcoming meeting. 1. Endpoints query function. This was split out into ticket #425 as a general function to query whether MPI processes exist in the same address space. 2. Discussion of communicator comparison routines in the presence of MPI processes that share an address space. This appears to also be trending toward a more general change to support models where processes share an address space. 3. Collective semantic for MPI_Comm_free. Currently we are converging on removing the special case for MPI_Comm_free and potentially creating a separate operation that can be used to free endpoints in a non-THREAD_MULTIPLE environment. We also have ticket #411 going forward for a formal reading at the next meeting. Monday is the deadline for changes, so please review. Jeff, are you planning to do anything with #381 communicator info keys (or has that moved to a different WG)? Any other agenda items? Thanks, ~Jim.

1 0

Re: [mpiwg-hybridpm] End-point ambiguity with MPI_COMM_INTERCOMM_CREATE
by Daniel Holmes 14 May '14

14 May '14

Hi Jeff, I discovered this possible issue when implementing end-points in my MPI library. It has not been discussed in the WG yet. Cheers, Dan. On 13/05/2014 15:29, Jeff Squyres (jsquyres) wrote: > I'm sorry I missed the last hybrid call -- did this issue get discussed / incorporated into the endpoints proposal? > > > On May 7, 2014, at 1:02 PM, Daniel Holmes <dholmes(a)epcc.ed.ac.uk> wrote: > >> Hi all, >> >> The operation MPI_COMM_INTERCOMM_CREATE is defined in MPI-3 on pages 261-2. >> >> The peer communicator is supplied to enable communication between the group leaders. >> This requires knowledge of the local and remote ranks in peer_comm. >> The remote_leader is the "rank of the remote group leader in peer_comm". >> However, the local_leader is the "rank of local group leader in local_comm". >> This must be translated to the appropriate rank in peer_comm. >> >> Unfortunately, if peer_comm is an end-points communicator this rank may not be particularly well-defined. >> >> Proposal: change the definition of the local_leader parameter to be "rank of local group leader in peer_comm". >> >> --- >> >> In the simplest case, the tie can be broken by examining local parameter values: >> >> my_mcw_rank <- MPI_COMM_RANK(MPI_COMM_WORLD) >> ep_comm <- MPI_COMM_CREATE_ENDPOINTS(parent:=MPI_COMM_WORLD, my_num_ep:=2) >> if (my_mcw_rank == 10) { >> // ep_comm[0] represents rank 20, is local_leader rank 20 or rank 21? >> inter_comm <- MPI_COMM_INTERCOMM_CREATE(local_comm:=MPI_COMM_SELF, local_leader:=0, peer_comm:=ep_comm[0], remote_leader:=30, tag:=1) >> } else if (my_mcw_rank == 15) { >> // ep_comm[0] represents rank 30, is local_leader rank 30 or rank 31? >> inter_comm <- MPI_COMM_INTERCOMM_CREATE(local_comm:=MPI_COMM_SELF, local_leader:=0, peer_comm:=ep_comm[0], remote_leader:=20, tag:=1) >> } >> >> In the above code, the local_leader (rank 0 in MPI_COMM_SELF) does not identify a unique rank in the peer communicator. >> Either of the two locally available communicator handles could be used as the local_leader but the choice is ambiguous. >> In this case, the communicator handle supplied as peer_comm can be used to break the tie. >> >> In a more tricky case, collective communication using local_comm would be necessary to break the tie: >> >> my_mcw_rank <- MPI_COMM_RANK(MPI_COMM_WORLD) >> split_comm <- MPI_COMM_SPLIT(MPI_COMM_WORLD, my_mcw_rank % 3, my_mcw_rank) >> ep_comm <- MPI_COMM_CREATE_ENDPOINTS(parent:=MPI_COMM_WORLD, my_num_ep:=2) >> if (my_mcw_rank % 3 == 1) { >> // ep_comm[0] represents rank 2*my_mcw_rank, is local_leader (rank 3 in split_comm, rank 10 in mcw) rank 20 or rank 21 in peer_comm? >> inter_comm <- MPI_COMM_INTERCOMM_CREATE(local_comm:=split_comm, local_leader:=3, peer_comm:=ep_comm[0], remote_leader:=30, tag:=2) >> } else if (my_mcw_rank % 3 == 0) { >> // ep_comm[0] represents rank 2*my_mcw_rank, is local_leader (rank 6 in split_comm, rank 15 in mcw) rank 30 or rank 31 in peer_comm? >> inter_comm <- MPI_COMM_INTERCOMM_CREATE(local_comm:=split_comm, local_leader:=6, peer_comm:=ep_comm[0], remote_leader:=20, tag:=2) >> } >> >> This code splits comm_world into three groups with keys 0, 1 and 2. >> Then it attempts to create an inter-communicator from the groups with keys 0 and 1. >> It uses a peer communicator which has two end-points per rank in comm_world. >> The specification of the local group leader is ambiguous in peer comm because there are two valid answers. >> Processes that determine they are in the split group with key 1 (MCW ranks {1,4,7,10,13,...}), >> supply a local group leader identified by <split_comm, rank 3> >> which is unambiguously equivalent to <comm_world, rank 10> >> which could be either <ep_comm, rank 20> or <ep_comm, rank 21> >> This tie can be broken by broadcasting the rank of the peer_comm communicator handle supplied by the local group leader to all ranks in local_comm. >> >> Cheers, >> Dan. >> >> -- >> Dan Holmes >> Applications Consultant in HPC Research >> EPCC, The University of Edinburgh >> James Clerk Maxwell Building >> The Kings Buildings >> Mayfield Road >> Edinburgh, UK >> EH9 3JZ >> T: +44(0)131 651 3465 >> E:[email protected] >> >> *Please consider the environment before printing this email.* >> >> >> The University of Edinburgh is a charitable body, registered in >> Scotland, with registration number SC005336. >> >> _______________________________________________ >> mpiwg-hybridpm mailing list >> mpiwg-hybridpm(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-hybridpm > -- Dan Holmes Applications Consultant in HPC Research EPCC, The University of Edinburgh James Clerk Maxwell Building The Kings Buildings Mayfield Road Edinburgh, UK EH9 3JZ T: +44(0)131 651 3465 E: dholmes(a)epcc.ed.ac.uk *Please consider the environment before printing this email.* The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0

Re: [mpiwg-hybridpm] End-point ambiguity with MPI_COMM_INTERCOMM_CREATE
by Jeff Squyres (jsquyres) 13 May '14

13 May '14

I'm sorry I missed the last hybrid call -- did this issue get discussed / incorporated into the endpoints proposal? On May 7, 2014, at 1:02 PM, Daniel Holmes <dholmes(a)epcc.ed.ac.uk> wrote: > Hi all, > > The operation MPI_COMM_INTERCOMM_CREATE is defined in MPI-3 on pages 261-2. > > The peer communicator is supplied to enable communication between the group leaders. > This requires knowledge of the local and remote ranks in peer_comm. > The remote_leader is the "rank of the remote group leader in peer_comm". > However, the local_leader is the "rank of local group leader in local_comm". > This must be translated to the appropriate rank in peer_comm. > > Unfortunately, if peer_comm is an end-points communicator this rank may not be particularly well-defined. > > Proposal: change the definition of the local_leader parameter to be "rank of local group leader in peer_comm". > > --- > > In the simplest case, the tie can be broken by examining local parameter values: > > my_mcw_rank <- MPI_COMM_RANK(MPI_COMM_WORLD) > ep_comm <- MPI_COMM_CREATE_ENDPOINTS(parent:=MPI_COMM_WORLD, my_num_ep:=2) > if (my_mcw_rank == 10) { > // ep_comm[0] represents rank 20, is local_leader rank 20 or rank 21? > inter_comm <- MPI_COMM_INTERCOMM_CREATE(local_comm:=MPI_COMM_SELF, local_leader:=0, peer_comm:=ep_comm[0], remote_leader:=30, tag:=1) > } else if (my_mcw_rank == 15) { > // ep_comm[0] represents rank 30, is local_leader rank 30 or rank 31? > inter_comm <- MPI_COMM_INTERCOMM_CREATE(local_comm:=MPI_COMM_SELF, local_leader:=0, peer_comm:=ep_comm[0], remote_leader:=20, tag:=1) > } > > In the above code, the local_leader (rank 0 in MPI_COMM_SELF) does not identify a unique rank in the peer communicator. > Either of the two locally available communicator handles could be used as the local_leader but the choice is ambiguous. > In this case, the communicator handle supplied as peer_comm can be used to break the tie. > > In a more tricky case, collective communication using local_comm would be necessary to break the tie: > > my_mcw_rank <- MPI_COMM_RANK(MPI_COMM_WORLD) > split_comm <- MPI_COMM_SPLIT(MPI_COMM_WORLD, my_mcw_rank % 3, my_mcw_rank) > ep_comm <- MPI_COMM_CREATE_ENDPOINTS(parent:=MPI_COMM_WORLD, my_num_ep:=2) > if (my_mcw_rank % 3 == 1) { > // ep_comm[0] represents rank 2*my_mcw_rank, is local_leader (rank 3 in split_comm, rank 10 in mcw) rank 20 or rank 21 in peer_comm? > inter_comm <- MPI_COMM_INTERCOMM_CREATE(local_comm:=split_comm, local_leader:=3, peer_comm:=ep_comm[0], remote_leader:=30, tag:=2) > } else if (my_mcw_rank % 3 == 0) { > // ep_comm[0] represents rank 2*my_mcw_rank, is local_leader (rank 6 in split_comm, rank 15 in mcw) rank 30 or rank 31 in peer_comm? > inter_comm <- MPI_COMM_INTERCOMM_CREATE(local_comm:=split_comm, local_leader:=6, peer_comm:=ep_comm[0], remote_leader:=20, tag:=2) > } > > This code splits comm_world into three groups with keys 0, 1 and 2. > Then it attempts to create an inter-communicator from the groups with keys 0 and 1. > It uses a peer communicator which has two end-points per rank in comm_world. > The specification of the local group leader is ambiguous in peer comm because there are two valid answers. > Processes that determine they are in the split group with key 1 (MCW ranks {1,4,7,10,13,...}), > supply a local group leader identified by <split_comm, rank 3> > which is unambiguously equivalent to <comm_world, rank 10> > which could be either <ep_comm, rank 20> or <ep_comm, rank 21> > This tie can be broken by broadcasting the rank of the peer_comm communicator handle supplied by the local group leader to all ranks in local_comm. > > Cheers, > Dan. > > -- > Dan Holmes > Applications Consultant in HPC Research > EPCC, The University of Edinburgh > James Clerk Maxwell Building > The Kings Buildings > Mayfield Road > Edinburgh, UK > EH9 3JZ > T: +44(0)131 651 3465 > E:[email protected] > > *Please consider the environment before printing this email.* > > > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > _______________________________________________ > mpiwg-hybridpm mailing list > mpiwg-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-hybridpm -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

End-point ambiguity with MPI_COMM_INTERCOMM_CREATE
by Daniel Holmes 07 May '14

07 May '14

Hi all, The operation MPI_COMM_INTERCOMM_CREATE is defined in MPI-3 on pages 261-2. The peer communicator is supplied to enable communication between the group leaders. This requires knowledge of the local and remote ranks in peer_comm. The remote_leader is the "rank of the remote group leader in peer_comm". However, the local_leader is the "rank of local group leader in local_comm". This must be translated to the appropriate rank in peer_comm. Unfortunately, if peer_comm is an end-points communicator this rank may not be particularly well-defined. Proposal: change the definition of the local_leader parameter to be "rank of local group leader in peer_comm". --- In the simplest case, the tie can be broken by examining local parameter values: my_mcw_rank <- MPI_COMM_RANK(MPI_COMM_WORLD) ep_comm <- MPI_COMM_CREATE_ENDPOINTS(parent:=MPI_COMM_WORLD, my_num_ep:=2) if (my_mcw_rank == 10) { // ep_comm[0] represents rank 20, is local_leader rank 20 or rank 21? inter_comm <- MPI_COMM_INTERCOMM_CREATE(local_comm:=MPI_COMM_SELF, local_leader:=0, peer_comm:=ep_comm[0], remote_leader:=30, tag:=1) } else if (my_mcw_rank == 15) { // ep_comm[0] represents rank 30, is local_leader rank 30 or rank 31? inter_comm <- MPI_COMM_INTERCOMM_CREATE(local_comm:=MPI_COMM_SELF, local_leader:=0, peer_comm:=ep_comm[0], remote_leader:=20, tag:=1) } In the above code, the local_leader (rank 0 in MPI_COMM_SELF) does not identify a unique rank in the peer communicator. Either of the two locally available communicator handles could be used as the local_leader but the choice is ambiguous. In this case, the communicator handle supplied as peer_comm can be used to break the tie. In a more tricky case, collective communication using local_comm would be necessary to break the tie: my_mcw_rank <- MPI_COMM_RANK(MPI_COMM_WORLD) split_comm <- MPI_COMM_SPLIT(MPI_COMM_WORLD, my_mcw_rank % 3, my_mcw_rank) ep_comm <- MPI_COMM_CREATE_ENDPOINTS(parent:=MPI_COMM_WORLD, my_num_ep:=2) if (my_mcw_rank % 3 == 1) { // ep_comm[0] represents rank 2*my_mcw_rank, is local_leader (rank 3 in split_comm, rank 10 in mcw) rank 20 or rank 21 in peer_comm? inter_comm <- MPI_COMM_INTERCOMM_CREATE(local_comm:=split_comm, local_leader:=3, peer_comm:=ep_comm[0], remote_leader:=30, tag:=2) } else if (my_mcw_rank % 3 == 0) { // ep_comm[0] represents rank 2*my_mcw_rank, is local_leader (rank 6 in split_comm, rank 15 in mcw) rank 30 or rank 31 in peer_comm? inter_comm <- MPI_COMM_INTERCOMM_CREATE(local_comm:=split_comm, local_leader:=6, peer_comm:=ep_comm[0], remote_leader:=20, tag:=2) } This code splits comm_world into three groups with keys 0, 1 and 2. Then it attempts to create an inter-communicator from the groups with keys 0 and 1. It uses a peer communicator which has two end-points per rank in comm_world. The specification of the local group leader is ambiguous in peer comm because there are two valid answers. Processes that determine they are in the split group with key 1 (MCW ranks {1,4,7,10,13,...}), supply a local group leader identified by <split_comm, rank 3> which is unambiguously equivalent to <comm_world, rank 10> which could be either <ep_comm, rank 20> or <ep_comm, rank 21> This tie can be broken by broadcasting the rank of the peer_comm communicator handle supplied by the local group leader to all ranks in local_comm. Cheers, Dan. -- Dan Holmes Applications Consultant in HPC Research EPCC, The University of Edinburgh James Clerk Maxwell Building The Kings Buildings Mayfield Road Edinburgh, UK EH9 3JZ T: +44(0)131 651 3465 E:[email protected] *Please consider the environment before printing this email.* The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0

Re: [mpiwg-hybridpm] Hybrid WG telecon reminder: May 5th 11am central
by Jim Dinan 05 May '14

05 May '14

Minutes are posted: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Hybrid/notes-2014-05-… On Mon, May 5, 2014 at 10:58 AM, Jim Dinan <james.dinan(a)gmail.com> wrote: > Expanded agenda: > > 1. (Jim) Discussion of endpoints query function. We decided not to pursue > this at this time for two reasons: (1) it's hard to define (is it relative > to another communicator?) and (2) because knowing the number of endpoints > is not helpful, you need the handles and a routine that returns the handles > breaks MPI convention. > > 2. (Dan) Discussion of communicator comparison routines in the presence of > endpoints. > > 3. (Pavan) Move toward resolution of MPI_Comm_free. Currently, the > leading option is to leave MPI_Comm_free as a conventional collective > (MPI_THREAD_MULTIPLE mode is required to free an endpoints communicator). > We could pursue a lesser-collective MPI_Comm_endpoints_free (i.e. for > lesser threading models) in a future proposal. > > ~Jim. > > > On Sun, May 4, 2014 at 7:12 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > >> >> Agenda: Continue Endpoints discussion. >> >> https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Hybrid >> >> — Pavan >> >> _______________________________________________ >> mpiwg-hybridpm mailing list >> mpiwg-hybridpm(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-hybridpm >> > >

1 0

Re: [mpiwg-hybridpm] Hybrid WG telecon reminder: May 5th 11am central
by Jim Dinan 05 May '14

05 May '14

Expanded agenda: 1. (Jim) Discussion of endpoints query function. We decided not to pursue this at this time for two reasons: (1) it's hard to define (is it relative to another communicator?) and (2) because knowing the number of endpoints is not helpful, you need the handles and a routine that returns the handles breaks MPI convention. 2. (Dan) Discussion of communicator comparison routines in the presence of endpoints. 3. (Pavan) Move toward resolution of MPI_Comm_free. Currently, the leading option is to leave MPI_Comm_free as a conventional collective (MPI_THREAD_MULTIPLE mode is required to free an endpoints communicator). We could pursue a lesser-collective MPI_Comm_endpoints_free (i.e. for lesser threading models) in a future proposal. ~Jim. On Sun, May 4, 2014 at 7:12 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > Agenda: Continue Endpoints discussion. > > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Hybrid > > — Pavan > > _______________________________________________ > mpiwg-hybridpm mailing list > mpiwg-hybridpm(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-hybridpm >

1 0

Hybrid WG telecon reminder: May 5th 11am central
by Balaji, Pavan 04 May '14

04 May '14

Agenda: Continue Endpoints discussion. https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Hybrid — Pavan

1 0