- mpiwg-coll - mailman3.cels.anl.gov

Re: [mpiwg-coll] [Mpi-forum] Error/gap in MPI_NEIGHBOR_ALLTOALL/ALLGATHER
by Rolf Rabenseifner 28 Sep '19

28 Sep '19

Yes, if after seven years of MPI_NEIGHBOR_ALLTOALL, neither the user know whether their MPI library is wrong nor all implementors are sure how to implement this routine for 1 or 2 processes in cyclic Cartesian direction, then some wording is missing in the MPI standard. Best regards Rolf ----- Anthony Skjellum <skjellum(a)auburn.edu> wrote: > Rolf let’s open a Ticket > > Anthony Skjellum, PhD > 205-807-4968 > > > > On Sep 27, 2019, at 6:09 PM, Rolf Rabenseifner via mpi-forum <mpi-forum(a)lists.mpi-forum.org> wrote: > > > > Dear MPI collective WG, > > > > you may try to resolve the problem with a maybe wrong > > MPI specification for MPI_NEIGHBOR_ALLTOALL/ALLGATHER > > > > Dear MPI Forum member, > > > > you may own/use an MPI implementation that implements > > MPI_NEIGHBOR_ALLTOALL/ALLGATHER > > with race conditions if #nprocs in one dimension is > > only 1 or 2 and periodic==true > > > > The problem was reported as a bug of the OpenMPI library > > by Simone Chiochetti from DICAM at the University of Trento, > > but seems to be a bug in the MPI specification, > > or at least an advice to implementors is missing. > > > > I produced a set of animated slides. > > Please look at them in presentation mode with animation. > > > > Have fun with a problem that clearly prevents the use > > of MPI_NEIGHBOR_... routines with cyclic boundary condition > > if one wants to verify that mpirun -np 1 is doing > > the same as the sequential code. > > > > Best regards > > Rolf > > > > -- > > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de . > > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . > > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . > > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner . > > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) . > > <neighbor_mpi-3_bug.pptx> > > _______________________________________________ > > mpi-forum mailing list > > mpi-forum(a)lists.mpi-forum.org > > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de . High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner . Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) .

1 0

Error/gap in MPI_NEIGHBOR_ALLTOALL/ALLGATHER
by Rolf Rabenseifner 27 Sep '19

27 Sep '19

Dear MPI collective WG, you may try to resolve the problem with a maybe wrong MPI specification for MPI_NEIGHBOR_ALLTOALL/ALLGATHER Dear MPI Forum member, you may own/use an MPI implementation that implements MPI_NEIGHBOR_ALLTOALL/ALLGATHER with race conditions if #nprocs in one dimension is only 1 or 2 and periodic==true The problem was reported as a bug of the OpenMPI library by Simone Chiochetti from DICAM at the University of Trento, but seems to be a bug in the MPI specification, or at least an advice to implementors is missing. I produced a set of animated slides. Please look at them in presentation mode with animation. Have fun with a problem that clearly prevents the use of MPI_NEIGHBOR_... routines with cyclic boundary condition if one wants to verify that mpirun -np 1 is doing the same as the sequential code. Best regards Rolf -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de . High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 . University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 . Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner . Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307) .

1 0

Re: [mpiwg-coll] [Mpi-forum] Reminder: MPI Virtual Meeting THIS Wednesday Oct. 24.
by HOLMES Daniel 25 Oct '18

25 Oct '18

Hi Tony, What technology are we using to connect? The normal webex is still scheduled for Wed - so won’t work today. Cheers, Dan. — Dr Daniel Holmes PhD Applications Consultant in HPC Research d.holmes(a)epcc.ed.ac.uk<mailto:[email protected]> Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. — On 24 Oct 2018, at 13:42, Anthony Skjellum via mpiwg-coll <mpiwg-coll(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: With apologies, for those attending, we will do this next on Thursday October 25, November 1, and November 8---forgot about the shift of weekend vis a vis Israel and US/Europe, so Friday isn't good. No meeting on November 15, 22 (SC and Thanksgiving respectively). We will go back to Wednesday meetings on December 5, 12, 19, skipping December 26 and January 2. So first 2019 meeting will be January 9. Please attend today's virtual meeting on MPI-4. Tony On Wed, Oct 24, 2018 at 6:52 AM Tony Skjellum <skjellum(a)gmail.com<mailto:[email protected]>> wrote: Resend Anthony Skjellum, PhD 205-807-4968 On Oct 24, 2018, at 6:49 AM, Anthony Skjellum <skjellum(a)icloud.com<mailto:[email protected]>> wrote: Our weekly meeting of persistent and collective is hereby moved to 9am eastern USA time this Friday and all Friday’s till virtual meetings end ... Anthony Skjellum, PhD 205-807-4968 Begin forwarded message: From: Martin Schulz via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> Date: October 24, 2018 at 5:23:53 AM EDT To: Wesley Bland via mpi-forum <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: Martin Schulz <schulzm(a)in.tum.de<mailto:[email protected]>> Subject: [Mpi-forum] Reminder: MPI Virtual Meeting THIS Wednesday Oct. 24. Reply-To: Main MPI Forum mailing list <mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]>> Hi all, Just as a reminder, today (i.e., THIS Wednesday) we will have our next virtual meeting in this quarter. The webex information can be reached via https://www.mpi-forum.org/meetings/ The topic will be a discussion on how we want to proceed with the MPI standard in general, what our roadmap is and how much we want to stick to backwards compatibility. This topic come out of a discussion towards the end of the MPI Forum in Barcelona and was triggered on how to handle “BigCount”, but then quickly went much wider. We ended up with a large majority (of those present only, of course) wanting to investigate this further and possible opting for something we called a “fork” of the standard, i.e., having an MPI 3.x line and an MPI 4.x line, which would no longer be backwards compatible, but still composable. The meeting today is intended to be an open discussion to a) continue the discussion from Barcelona and to b) get the pulse from people who were not at the meeting. Thanks, Martin — Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching Email: schulzm(a)in.tum.de<mailto:[email protected]> _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum -- Anthony Skjellum, PhD skjellum(a)gmail.com<mailto:[email protected]> Cell: +1-205-807-4968 _______________________________________________ mpiwg-coll mailing list mpiwg-coll(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0

Re: [mpiwg-coll] [Mpi-forum] Reminder: MPI Virtual Meeting THIS Wednesday Oct. 24.
by Anthony Skjellum 24 Oct '18

24 Oct '18

With apologies, for those attending, we will do this next on Thursday October 25, November 1, and November 8---forgot about the shift of weekend vis a vis Israel and US/Europe, so Friday isn't good. No meeting on November 15, 22 (SC and Thanksgiving respectively). We will go back to Wednesday meetings on December 5, 12, 19, skipping December 26 and January 2. So first 2019 meeting will be January 9. Please attend today's virtual meeting on MPI-4. Tony On Wed, Oct 24, 2018 at 6:52 AM Tony Skjellum <skjellum(a)gmail.com> wrote: > Resend > > Anthony Skjellum, PhD > 205-807-4968 > > > On Oct 24, 2018, at 6:49 AM, Anthony Skjellum <skjellum(a)icloud.com> wrote: > > Our weekly meeting of persistent and collective is hereby moved to 9am > eastern USA time this Friday and all Friday’s till virtual meetings end ... > > > > Anthony Skjellum, PhD > 205-807-4968 > > > Begin forwarded message: > > *From:* Martin Schulz via mpi-forum <mpi-forum(a)lists.mpi-forum.org> > *Date:* October 24, 2018 at 5:23:53 AM EDT > *To:* Wesley Bland via mpi-forum <mpi-forum(a)lists.mpi-forum.org> > *Cc:* Martin Schulz <schulzm(a)in.tum.de> > *Subject:* *[Mpi-forum] Reminder: MPI Virtual Meeting THIS Wednesday Oct. > 24.* > *Reply-To:* Main MPI Forum mailing list <mpi-forum(a)lists.mpi-forum.org> > > Hi all, > > Just as a reminder, today (i.e., THIS Wednesday) we will have our next > virtual meeting in this quarter. The webex information can be reached via > > https://www.mpi-forum.org/meetings/ > > The topic will be a discussion on how we want to proceed with the MPI > standard in general, what our roadmap is and how much we want to stick to > backwards compatibility. This topic come out of a discussion towards the > end of the MPI Forum in Barcelona and was triggered on how to handle > “BigCount”, but then quickly went much wider. We ended up with a large > majority (of those present only, of course) wanting to investigate this > further and possible opting for something we called a “fork” of the > standard, i.e., having an MPI 3.x line and an MPI 4.x line, which would no > longer be backwards compatible, but still composable. The meeting today is > intended to be an open discussion to a) continue the discussion from > Barcelona and to b) get the pulse from people who were not at the meeting. > > Thanks, > Martin > > — > Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel > Systems > Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching > Email: schulzm(a)in.tum.de <schulzm(a)in.tum.de> > > > > > > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum > > -- Anthony Skjellum, PhD skjellum(a)gmail.com Cell: +1-205-807-4968

1 0

Re: [mpiwg-coll] [Mpi-forum] Reminder: MPI Virtual Meeting THIS Wednesday Oct. 24.
by Tony Skjellum 24 Oct '18

24 Oct '18

Resend Anthony Skjellum, PhD 205-807-4968 > On Oct 24, 2018, at 6:49 AM, Anthony Skjellum <skjellum(a)icloud.com> wrote: > > Our weekly meeting of persistent and collective is hereby moved to 9am eastern USA time this Friday and all Friday’s till virtual meetings end ... > > > > Anthony Skjellum, PhD > 205-807-4968 > > > Begin forwarded message: > >> From: Martin Schulz via mpi-forum <mpi-forum(a)lists.mpi-forum.org> >> Date: October 24, 2018 at 5:23:53 AM EDT >> To: Wesley Bland via mpi-forum <mpi-forum(a)lists.mpi-forum.org> >> Cc: Martin Schulz <schulzm(a)in.tum.de> >> Subject: [Mpi-forum] Reminder: MPI Virtual Meeting THIS Wednesday Oct. 24. >> Reply-To: Main MPI Forum mailing list <mpi-forum(a)lists.mpi-forum.org> >> >> Hi all, >> >> Just as a reminder, today (i.e., THIS Wednesday) we will have our next virtual meeting in this quarter. The webex information can be reached via >> >> https://www.mpi-forum.org/meetings/ >> >> The topic will be a discussion on how we want to proceed with the MPI standard in general, what our roadmap is and how much we want to stick to backwards compatibility. This topic come out of a discussion towards the end of the MPI Forum in Barcelona and was triggered on how to handle “BigCount”, but then quickly went much wider. We ended up with a large majority (of those present only, of course) wanting to investigate this further and possible opting for something we called a “fork” of the standard, i.e., having an MPI 3.x line and an MPI 4.x line, which would no longer be backwards compatible, but still composable. The meeting today is intended to be an open discussion to a) continue the discussion from Barcelona and to b) get the pulse from people who were not at the meeting. >> >> Thanks, >> Martin >> >> — >> Prof. Dr. Martin Schulz, Chair of Computer Architecture and Parallel Systems >> Department of Informatics, TU-Munich, Boltzmannstraße 3, D-85748 Garching >> Email: schulzm(a)in.tum.de >> >> >> >> >> >> >> _______________________________________________ >> mpi-forum mailing list >> mpi-forum(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum

1 0

Re: [mpiwg-coll] backward communication in neighborhood collectives
by Junchao Zhang 23 Jul '18

23 Jul '18

No, not at all. --Junchao Zhang On Mon, Jul 23, 2018 at 11:04 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > Is creating two communicators prohibitive? > > Sent from my iPhone > > On Jul 23, 2018, at 8:02 AM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: > > Jeff, > No. I do not want to send messages to other ranks (not in > comm_dist_graph). > My question is: I create a graph with MPI_Dist_graph_create_adjacent, > which specifies a communication direction. I can do MPI_Neighbor_alltoallv > on it. That is great. But I also want to do backward communication > (reverse the direction of the edges in graph). How can I do it without > creating a new communicator? > The background of this question is: In PETSc, we have an operation called > VecScatter, which scatters some entries of a parallel vector x to another > parallel vector y. Sometime, we want to reverse the operation, to scatter y > entries to x. > > > --Junchao Zhang > > On Mon, Jul 23, 2018 at 9:49 AM, Jeff Hammond <jeff.science(a)gmail.com> > wrote: > >> On Mon, Jul 23, 2018 at 7:45 AM, Junchao Zhang <jczhang(a)mcs.anl.gov> >> wrote: >> >>> Hello, Dan, >>> I am interested in MPI_Neighbor_alltoallv. From its arguments, I do not >>> see one can send a message to a neighbor not specified in the graph created >>> by for example, MPI_Dist_graph_create_adjacent. >>> >> >> Why would you expect this? The whole point of graph communicators is to >> specify the communication topology so that it can be optimized. If you >> want to communicate with another ranks, use the parent communicator that >> supports communication to all ranks. >> >> Jeff >> >> >>> --Junchao Zhang >>> >>> On Mon, Jul 23, 2018 at 5:51 AM, HOLMES Daniel <d.holmes(a)epcc.ed.ac.uk> >>> wrote: >>> >>>> Hi Junchao Zhang, >>>> >>>> My understanding of the current API for MPI-3.1 is that: >>>> >>>> 1) the virtual topology does not actually restrict communication via >>>> the communicator to the edges specified in the topology - messages can be >>>> sent along any edge in either direction, and even between pairs of >>>> processes for which no edge was specified. >>>> >>>> 2) the virtual topology can be specified as a symmetric graph - for >>>> every ‘forward’ edge (e.g. from A to B), the ‘backward’ edge (i.e. from B >>>> to A) can be included as well. >>>> >>>> 3) there is already language in the MPI Standard regarding how MPI >>>> handles symmetric and non-symmetric graph topologies for neighbourhood >>>> collective operations. >>>> >>>> Thus, there is no need to create two distributed graph topology >>>> communicators to achieve ‘forward and backward communication along the >>>> edges’. >>>> >>>> Cheers, >>>> Dan. >>>> — >>>> Dr Daniel Holmes PhD >>>> Applications Consultant in HPC Research >>>> d.holmes(a)epcc.ed.ac.uk >>>> Phone: +44 (0) 131 651 3465 >>>> Mobile: +44 (0) 7940 524 088 >>>> Address: Room 3415, JCMB, The King’s Buildings, Edinburgh, EH9 3FD >>>> — >>>> The University of Edinburgh is a charitable body, registered in >>>> Scotland, with registration number SC005336. >>>> — >>>> >>>> On 15 Jul 2018, at 10:42, Anthony Skjellum <tony(a)runtimecomputing.com> >>>> wrote: >>>> >>>> Hi, I just saw this. >>>> >>>> We definitely need to consider this concern. >>>> >>>> I also need to go review the APIs. >>>> >>>> Thanks, >>>> Tony >>>> >>>> >>>> On Sat, Jul 14, 2018 at 12:27 AM, Junchao Zhang <jczhang(a)mcs.anl.gov> >>>> wrote: >>>> >>>>> I want to try MPI neighborhood collectives. I have a communication >>>>> graph and want to do both forward and backward communication along the >>>>> edges. With current APIs, it looks I need to create two comm_dist_graphs. >>>>> It is a waste since MPI implementations can do similar optimizations for >>>>> the two. >>>>> Should MPI support this scenario? Thanks. >>>>> >>>>> --Junchao Zhang >>>>> >>>>> _______________________________________________ >>>>> mpiwg-coll mailing list >>>>> mpiwg-coll(a)lists.mpi-forum.org >>>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll >>>>> >>>>> >>>> >>>> >>>> -- >>>> Tony Skjellum, PhD >>>> RunTime Computing Solutions, LLC >>>> tony(a)runtimecomputing.com >>>> direct: +1-205-918-7514 >>>> cell: +1-205-807-4968 >>>> _______________________________________________ >>>> mpiwg-coll mailing list >>>> mpiwg-coll(a)lists.mpi-forum.org >>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll >>>> >>>> >>>> >>>> The University of Edinburgh is a charitable body, registered in >>>> Scotland, with registration number SC005336. >>>> >>>> _______________________________________________ >>>> mpiwg-coll mailing list >>>> mpiwg-coll(a)lists.mpi-forum.org >>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll >>>> >>>> >>> >>> _______________________________________________ >>> mpiwg-coll mailing list >>> mpiwg-coll(a)lists.mpi-forum.org >>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll >>> >>> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> http://jeffhammond.github.io/ >> >> _______________________________________________ >> mpiwg-coll mailing list >> mpiwg-coll(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll >> >> > _______________________________________________ > mpiwg-coll mailing list > mpiwg-coll(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll > > > _______________________________________________ > mpiwg-coll mailing list > mpiwg-coll(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll > >

1 0

Re: [mpiwg-coll] backward communication in neighborhood collectives
by Jeff Hammond 23 Jul '18

23 Jul '18

Is creating two communicators prohibitive? Sent from my iPhone > On Jul 23, 2018, at 8:02 AM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: > > Jeff, > No. I do not want to send messages to other ranks (not in comm_dist_graph). > My question is: I create a graph with MPI_Dist_graph_create_adjacent, which specifies a communication direction. I can do MPI_Neighbor_alltoallv on it. That is great. But I also want to do backward communication (reverse the direction of the edges in graph). How can I do it without creating a new communicator? > The background of this question is: In PETSc, we have an operation called VecScatter, which scatters some entries of a parallel vector x to another parallel vector y. Sometime, we want to reverse the operation, to scatter y entries to x. > > > --Junchao Zhang > >> On Mon, Jul 23, 2018 at 9:49 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>> On Mon, Jul 23, 2018 at 7:45 AM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: >>> Hello, Dan, >>> I am interested in MPI_Neighbor_alltoallv. From its arguments, I do not see one can send a message to a neighbor not specified in the graph created by for example, MPI_Dist_graph_create_adjacent. >> >> Why would you expect this? The whole point of graph communicators is to specify the communication topology so that it can be optimized. If you want to communicate with another ranks, use the parent communicator that supports communication to all ranks. >> >> Jeff >> >>> --Junchao Zhang >>> >>>> On Mon, Jul 23, 2018 at 5:51 AM, HOLMES Daniel <d.holmes(a)epcc.ed.ac.uk> wrote: >>>> Hi Junchao Zhang, >>>> >>>> My understanding of the current API for MPI-3.1 is that: >>>> >>>> 1) the virtual topology does not actually restrict communication via the communicator to the edges specified in the topology - messages can be sent along any edge in either direction, and even between pairs of processes for which no edge was specified. >>>> >>>> 2) the virtual topology can be specified as a symmetric graph - for every ‘forward’ edge (e.g. from A to B), the ‘backward’ edge (i.e. from B to A) can be included as well. >>>> >>>> 3) there is already language in the MPI Standard regarding how MPI handles symmetric and non-symmetric graph topologies for neighbourhood collective operations. >>>> >>>> Thus, there is no need to create two distributed graph topology communicators to achieve ‘forward and backward communication along the edges’. >>>> >>>> Cheers, >>>> Dan. >>>> — >>>> Dr Daniel Holmes PhD >>>> Applications Consultant in HPC Research >>>> d.holmes(a)epcc.ed.ac.uk >>>> Phone: +44 (0) 131 651 3465 >>>> Mobile: +44 (0) 7940 524 088 >>>> Address: Room 3415, JCMB, The King’s Buildings, Edinburgh, EH9 3FD >>>> — >>>> The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. >>>> — >>>> >>>>> On 15 Jul 2018, at 10:42, Anthony Skjellum <tony(a)runtimecomputing.com> wrote: >>>>> >>>>> Hi, I just saw this. >>>>> >>>>> We definitely need to consider this concern. >>>>> >>>>> I also need to go review the APIs. >>>>> >>>>> Thanks, >>>>> Tony >>>>> >>>>> >>>>>> On Sat, Jul 14, 2018 at 12:27 AM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: >>>>>> I want to try MPI neighborhood collectives. I have a communication graph and want to do both forward and backward communication along the edges. With current APIs, it looks I need to create two comm_dist_graphs. It is a waste since MPI implementations can do similar optimizations for the two. >>>>>> Should MPI support this scenario? Thanks. >>>>>> >>>>>> --Junchao Zhang >>>>>> >>>>>> _______________________________________________ >>>>>> mpiwg-coll mailing list >>>>>> mpiwg-coll(a)lists.mpi-forum.org >>>>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Tony Skjellum, PhD >>>>> RunTime Computing Solutions, LLC >>>>> tony(a)runtimecomputing.com >>>>> direct: +1-205-918-7514 >>>>> cell: +1-205-807-4968 >>>>> _______________________________________________ >>>>> mpiwg-coll mailing list >>>>> mpiwg-coll(a)lists.mpi-forum.org >>>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll >>>> >>>> >>>> The University of Edinburgh is a charitable body, registered in >>>> Scotland, with registration number SC005336. >>>> >>>> _______________________________________________ >>>> mpiwg-coll mailing list >>>> mpiwg-coll(a)lists.mpi-forum.org >>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll >>>> >>> >>> >>> _______________________________________________ >>> mpiwg-coll mailing list >>> mpiwg-coll(a)lists.mpi-forum.org >>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll >>> >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> http://jeffhammond.github.io/ >> >> _______________________________________________ >> mpiwg-coll mailing list >> mpiwg-coll(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll >> > > _______________________________________________ > mpiwg-coll mailing list > mpiwg-coll(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll

1 0

Re: [mpiwg-coll] backward communication in neighborhood collectives
by Junchao Zhang 23 Jul '18

23 Jul '18

I do not want to combine the forward/backward graphs. Creating two communicators is OK with me. My original thought was: with one graph, if MPI also supports its reverse operation, then MPI implementations can do symmetric optimizations and save some resources. --Junchao Zhang On Mon, Jul 23, 2018 at 10:35 AM, Jed Brown <jed(a)jedbrown.org> wrote: > For ghost value updates in domain decomposition PDE solvers (usually the > most performance-sensitive use of VecScatter), the graph will usually be > symmetric. It is often nonsymmetric for multilevel restriction and > prolongation and for some field-split methods in multiphysics > applications, in which case I think you'll want to make two neighborhood > communicators rather than the union of the forward and reverse/transpose > operations with some empty messages. > > Junchao Zhang <jczhang(a)mcs.anl.gov> writes: > > > Dan, > > Using your approach, I have to specify zero messages in arguments to > > MPI_NEIGHBOR_ALLTOALLV, > > because I only do communication in one direction in one time. Also, in > my > > graph, there may have edges A->B and B->A. They may have different > weights. > > Using your approach, it is quite confusing. > > One uses MPI_NEIGHBOR_ALLTOALLV to have better performance. We have to > > keep it in mind. > > > > --Junchao Zhang > > > > On Mon, Jul 23, 2018 at 9:59 AM, HOLMES Daniel <d.holmes(a)epcc.ed.ac.uk> > > wrote: > > > >> Hi Junchao Zhang, > >> > >> You are correct - the neighbourhood collectives send messages to (and > >> receive messages from) only neighbouring MPI processes, as defined by > the > >> virtual topology. However, it is possible to use point-to-point, > >> single-sided, or normal collective functions to send (and receive) > messages > >> between *any* MPI processes in a communicator, irrespective of the > >> presence/structure of a virtual topology. > >> > >> For your use-case (if I understand it correctly), you should specify a > >> symmetric graph - each process specifies to > MPI_DIST_GRAPH_CREATE_ADJACENT > >> all the processes it wishes to communicate with (both sending and > >> receiving) during subsequent calls to MPI_NEIGHBOR_ALLTOALLV. > >> > >> Cheers, > >> Dan. > >> — > >> Dr Daniel Holmes PhD > >> Applications Consultant in HPC Research > >> d.holmes(a)epcc.ed.ac.uk > >> Phone: +44 (0) 131 651 3465 > >> Mobile: +44 (0) 7940 524 088 > >> Address: Room 3415, JCMB, The King’s Buildings, Edinburgh, EH9 3FD > >> — > >> The University of Edinburgh is a charitable body, registered in > Scotland, > >> with registration number SC005336. > >> — > >> > >> On 23 Jul 2018, at 15:45, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: > >> > >> Hello, Dan, > >> I am interested in MPI_Neighbor_alltoallv. From its arguments, I do not > >> see one can send a message to a neighbor not specified in the graph > created > >> by for example, MPI_Dist_graph_create_adjacent. > >> > >> --Junchao Zhang > >> > >> On Mon, Jul 23, 2018 at 5:51 AM, HOLMES Daniel <d.holmes(a)epcc.ed.ac.uk> > >> wrote: > >> > >>> Hi Junchao Zhang, > >>> > >>> My understanding of the current API for MPI-3.1 is that: > >>> > >>> 1) the virtual topology does not actually restrict communication via > the > >>> communicator to the edges specified in the topology - messages can be > sent > >>> along any edge in either direction, and even between pairs of > processes for > >>> which no edge was specified. > >>> > >>> 2) the virtual topology can be specified as a symmetric graph - for > every > >>> ‘forward’ edge (e.g. from A to B), the ‘backward’ edge (i.e. from B to > A) > >>> can be included as well. > >>> > >>> 3) there is already language in the MPI Standard regarding how MPI > >>> handles symmetric and non-symmetric graph topologies for neighbourhood > >>> collective operations. > >>> > >>> Thus, there is no need to create two distributed graph topology > >>> communicators to achieve ‘forward and backward communication along the > >>> edges’. > >>> > >>> Cheers, > >>> Dan. > >>> — > >>> Dr Daniel Holmes PhD > >>> Applications Consultant in HPC Research > >>> d.holmes(a)epcc.ed.ac.uk > >>> Phone: +44 (0) 131 651 3465 > >>> Mobile: +44 (0) 7940 524 088 > >>> Address: Room 3415, JCMB, The King’s Buildings, Edinburgh, EH9 3FD > >>> — > >>> The University of Edinburgh is a charitable body, registered in > Scotland, > >>> with registration number SC005336. > >>> — > >>> > >>> On 15 Jul 2018, at 10:42, Anthony Skjellum <tony(a)runtimecomputing.com> > >>> wrote: > >>> > >>> Hi, I just saw this. > >>> > >>> We definitely need to consider this concern. > >>> > >>> I also need to go review the APIs. > >>> > >>> Thanks, > >>> Tony > >>> > >>> > >>> On Sat, Jul 14, 2018 at 12:27 AM, Junchao Zhang <jczhang(a)mcs.anl.gov> > >>> wrote: > >>> > >>>> I want to try MPI neighborhood collectives. I have a communication > >>>> graph and want to do both forward and backward communication along the > >>>> edges. With current APIs, it looks I need to create two > comm_dist_graphs. > >>>> It is a waste since MPI implementations can do similar optimizations > for > >>>> the two. > >>>> Should MPI support this scenario? Thanks. > >>>> > >>>> --Junchao Zhang > >>>> > >>>> _______________________________________________ > >>>> mpiwg-coll mailing list > >>>> mpiwg-coll(a)lists.mpi-forum.org > >>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll > >>>> > >>>> > >>> > >>> > >>> -- > >>> Tony Skjellum, PhD > >>> RunTime Computing Solutions, LLC > >>> tony(a)runtimecomputing.com > >>> direct: +1-205-918-7514 > >>> cell: +1-205-807-4968 > >>> _______________________________________________ > >>> mpiwg-coll mailing list > >>> mpiwg-coll(a)lists.mpi-forum.org > >>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll > >>> > >>> > >>> > >>> The University of Edinburgh is a charitable body, registered in > >>> Scotland, with registration number SC005336. > >>> > >>> _______________________________________________ > >>> mpiwg-coll mailing list > >>> mpiwg-coll(a)lists.mpi-forum.org > >>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll > >>> > >>> > >> _______________________________________________ > >> mpiwg-coll mailing list > >> mpiwg-coll(a)lists.mpi-forum.org > >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll > >> > >> > >> > >> The University of Edinburgh is a charitable body, registered in > >> Scotland, with registration number SC005336. > >> > >> _______________________________________________ > >> mpiwg-coll mailing list > >> mpiwg-coll(a)lists.mpi-forum.org > >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll > >> > >> > > _______________________________________________ > > mpiwg-coll mailing list > > mpiwg-coll(a)lists.mpi-forum.org > > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll >

1 0

Re: [mpiwg-coll] backward communication in neighborhood collectives
by Jed Brown 23 Jul '18

23 Jul '18

For ghost value updates in domain decomposition PDE solvers (usually the most performance-sensitive use of VecScatter), the graph will usually be symmetric. It is often nonsymmetric for multilevel restriction and prolongation and for some field-split methods in multiphysics applications, in which case I think you'll want to make two neighborhood communicators rather than the union of the forward and reverse/transpose operations with some empty messages. Junchao Zhang <jczhang(a)mcs.anl.gov> writes: > Dan, > Using your approach, I have to specify zero messages in arguments to > MPI_NEIGHBOR_ALLTOALLV, > because I only do communication in one direction in one time. Also, in my > graph, there may have edges A->B and B->A. They may have different weights. > Using your approach, it is quite confusing. > One uses MPI_NEIGHBOR_ALLTOALLV to have better performance. We have to > keep it in mind. > > --Junchao Zhang > > On Mon, Jul 23, 2018 at 9:59 AM, HOLMES Daniel <d.holmes(a)epcc.ed.ac.uk> > wrote: > >> Hi Junchao Zhang, >> >> You are correct - the neighbourhood collectives send messages to (and >> receive messages from) only neighbouring MPI processes, as defined by the >> virtual topology. However, it is possible to use point-to-point, >> single-sided, or normal collective functions to send (and receive) messages >> between *any* MPI processes in a communicator, irrespective of the >> presence/structure of a virtual topology. >> >> For your use-case (if I understand it correctly), you should specify a >> symmetric graph - each process specifies to MPI_DIST_GRAPH_CREATE_ADJACENT >> all the processes it wishes to communicate with (both sending and >> receiving) during subsequent calls to MPI_NEIGHBOR_ALLTOALLV. >> >> Cheers, >> Dan. >> — >> Dr Daniel Holmes PhD >> Applications Consultant in HPC Research >> d.holmes(a)epcc.ed.ac.uk >> Phone: +44 (0) 131 651 3465 >> Mobile: +44 (0) 7940 524 088 >> Address: Room 3415, JCMB, The King’s Buildings, Edinburgh, EH9 3FD >> — >> The University of Edinburgh is a charitable body, registered in Scotland, >> with registration number SC005336. >> — >> >> On 23 Jul 2018, at 15:45, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: >> >> Hello, Dan, >> I am interested in MPI_Neighbor_alltoallv. From its arguments, I do not >> see one can send a message to a neighbor not specified in the graph created >> by for example, MPI_Dist_graph_create_adjacent. >> >> --Junchao Zhang >> >> On Mon, Jul 23, 2018 at 5:51 AM, HOLMES Daniel <d.holmes(a)epcc.ed.ac.uk> >> wrote: >> >>> Hi Junchao Zhang, >>> >>> My understanding of the current API for MPI-3.1 is that: >>> >>> 1) the virtual topology does not actually restrict communication via the >>> communicator to the edges specified in the topology - messages can be sent >>> along any edge in either direction, and even between pairs of processes for >>> which no edge was specified. >>> >>> 2) the virtual topology can be specified as a symmetric graph - for every >>> ‘forward’ edge (e.g. from A to B), the ‘backward’ edge (i.e. from B to A) >>> can be included as well. >>> >>> 3) there is already language in the MPI Standard regarding how MPI >>> handles symmetric and non-symmetric graph topologies for neighbourhood >>> collective operations. >>> >>> Thus, there is no need to create two distributed graph topology >>> communicators to achieve ‘forward and backward communication along the >>> edges’. >>> >>> Cheers, >>> Dan. >>> — >>> Dr Daniel Holmes PhD >>> Applications Consultant in HPC Research >>> d.holmes(a)epcc.ed.ac.uk >>> Phone: +44 (0) 131 651 3465 >>> Mobile: +44 (0) 7940 524 088 >>> Address: Room 3415, JCMB, The King’s Buildings, Edinburgh, EH9 3FD >>> — >>> The University of Edinburgh is a charitable body, registered in Scotland, >>> with registration number SC005336. >>> — >>> >>> On 15 Jul 2018, at 10:42, Anthony Skjellum <tony(a)runtimecomputing.com> >>> wrote: >>> >>> Hi, I just saw this. >>> >>> We definitely need to consider this concern. >>> >>> I also need to go review the APIs. >>> >>> Thanks, >>> Tony >>> >>> >>> On Sat, Jul 14, 2018 at 12:27 AM, Junchao Zhang <jczhang(a)mcs.anl.gov> >>> wrote: >>> >>>> I want to try MPI neighborhood collectives. I have a communication >>>> graph and want to do both forward and backward communication along the >>>> edges. With current APIs, it looks I need to create two comm_dist_graphs. >>>> It is a waste since MPI implementations can do similar optimizations for >>>> the two. >>>> Should MPI support this scenario? Thanks. >>>> >>>> --Junchao Zhang >>>> >>>> _______________________________________________ >>>> mpiwg-coll mailing list >>>> mpiwg-coll(a)lists.mpi-forum.org >>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll >>>> >>>> >>> >>> >>> -- >>> Tony Skjellum, PhD >>> RunTime Computing Solutions, LLC >>> tony(a)runtimecomputing.com >>> direct: +1-205-918-7514 >>> cell: +1-205-807-4968 >>> _______________________________________________ >>> mpiwg-coll mailing list >>> mpiwg-coll(a)lists.mpi-forum.org >>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll >>> >>> >>> >>> The University of Edinburgh is a charitable body, registered in >>> Scotland, with registration number SC005336. >>> >>> _______________________________________________ >>> mpiwg-coll mailing list >>> mpiwg-coll(a)lists.mpi-forum.org >>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll >>> >>> >> _______________________________________________ >> mpiwg-coll mailing list >> mpiwg-coll(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll >> >> >> >> The University of Edinburgh is a charitable body, registered in >> Scotland, with registration number SC005336. >> >> _______________________________________________ >> mpiwg-coll mailing list >> mpiwg-coll(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll >> >> > _______________________________________________ > mpiwg-coll mailing list > mpiwg-coll(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll

1 0

Re: [mpiwg-coll] backward communication in neighborhood collectives
by Junchao Zhang 23 Jul '18

23 Jul '18

Dan, Using your approach, I have to specify zero messages in arguments to MPI_NEIGHBOR_ALLTOALLV, because I only do communication in one direction in one time. Also, in my graph, there may have edges A->B and B->A. They may have different weights. Using your approach, it is quite confusing. One uses MPI_NEIGHBOR_ALLTOALLV to have better performance. We have to keep it in mind. --Junchao Zhang On Mon, Jul 23, 2018 at 9:59 AM, HOLMES Daniel <d.holmes(a)epcc.ed.ac.uk> wrote: > Hi Junchao Zhang, > > You are correct - the neighbourhood collectives send messages to (and > receive messages from) only neighbouring MPI processes, as defined by the > virtual topology. However, it is possible to use point-to-point, > single-sided, or normal collective functions to send (and receive) messages > between *any* MPI processes in a communicator, irrespective of the > presence/structure of a virtual topology. > > For your use-case (if I understand it correctly), you should specify a > symmetric graph - each process specifies to MPI_DIST_GRAPH_CREATE_ADJACENT > all the processes it wishes to communicate with (both sending and > receiving) during subsequent calls to MPI_NEIGHBOR_ALLTOALLV. > > Cheers, > Dan. > — > Dr Daniel Holmes PhD > Applications Consultant in HPC Research > d.holmes(a)epcc.ed.ac.uk > Phone: +44 (0) 131 651 3465 > Mobile: +44 (0) 7940 524 088 > Address: Room 3415, JCMB, The King’s Buildings, Edinburgh, EH9 3FD > — > The University of Edinburgh is a charitable body, registered in Scotland, > with registration number SC005336. > — > > On 23 Jul 2018, at 15:45, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: > > Hello, Dan, > I am interested in MPI_Neighbor_alltoallv. From its arguments, I do not > see one can send a message to a neighbor not specified in the graph created > by for example, MPI_Dist_graph_create_adjacent. > > --Junchao Zhang > > On Mon, Jul 23, 2018 at 5:51 AM, HOLMES Daniel <d.holmes(a)epcc.ed.ac.uk> > wrote: > >> Hi Junchao Zhang, >> >> My understanding of the current API for MPI-3.1 is that: >> >> 1) the virtual topology does not actually restrict communication via the >> communicator to the edges specified in the topology - messages can be sent >> along any edge in either direction, and even between pairs of processes for >> which no edge was specified. >> >> 2) the virtual topology can be specified as a symmetric graph - for every >> ‘forward’ edge (e.g. from A to B), the ‘backward’ edge (i.e. from B to A) >> can be included as well. >> >> 3) there is already language in the MPI Standard regarding how MPI >> handles symmetric and non-symmetric graph topologies for neighbourhood >> collective operations. >> >> Thus, there is no need to create two distributed graph topology >> communicators to achieve ‘forward and backward communication along the >> edges’. >> >> Cheers, >> Dan. >> — >> Dr Daniel Holmes PhD >> Applications Consultant in HPC Research >> d.holmes(a)epcc.ed.ac.uk >> Phone: +44 (0) 131 651 3465 >> Mobile: +44 (0) 7940 524 088 >> Address: Room 3415, JCMB, The King’s Buildings, Edinburgh, EH9 3FD >> — >> The University of Edinburgh is a charitable body, registered in Scotland, >> with registration number SC005336. >> — >> >> On 15 Jul 2018, at 10:42, Anthony Skjellum <tony(a)runtimecomputing.com> >> wrote: >> >> Hi, I just saw this. >> >> We definitely need to consider this concern. >> >> I also need to go review the APIs. >> >> Thanks, >> Tony >> >> >> On Sat, Jul 14, 2018 at 12:27 AM, Junchao Zhang <jczhang(a)mcs.anl.gov> >> wrote: >> >>> I want to try MPI neighborhood collectives. I have a communication >>> graph and want to do both forward and backward communication along the >>> edges. With current APIs, it looks I need to create two comm_dist_graphs. >>> It is a waste since MPI implementations can do similar optimizations for >>> the two. >>> Should MPI support this scenario? Thanks. >>> >>> --Junchao Zhang >>> >>> _______________________________________________ >>> mpiwg-coll mailing list >>> mpiwg-coll(a)lists.mpi-forum.org >>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll >>> >>> >> >> >> -- >> Tony Skjellum, PhD >> RunTime Computing Solutions, LLC >> tony(a)runtimecomputing.com >> direct: +1-205-918-7514 >> cell: +1-205-807-4968 >> _______________________________________________ >> mpiwg-coll mailing list >> mpiwg-coll(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll >> >> >> >> The University of Edinburgh is a charitable body, registered in >> Scotland, with registration number SC005336. >> >> _______________________________________________ >> mpiwg-coll mailing list >> mpiwg-coll(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll >> >> > _______________________________________________ > mpiwg-coll mailing list > mpiwg-coll(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll > > > > The University of Edinburgh is a charitable body, registered in > Scotland, with registration number SC005336. > > _______________________________________________ > mpiwg-coll mailing list > mpiwg-coll(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll > >

1 0