For ghost value updates in domain decomposition PDE solvers (usually the most performance-sensitive use of VecScatter), the graph will usually be symmetric. It is often nonsymmetric for multilevel restriction and prolongation and for some field-split methods in multiphysics applications, in which case I think you'll want to make two neighborhood communicators rather than the union of the forward and reverse/transpose operations with some empty messages. Junchao Zhang <[email protected]> writes:

Dan, Using your approach, I have to specify zero messages in arguments to MPI_NEIGHBOR_ALLTOALLV, because I only do communication in one direction in one time. Also, in my graph, there may have edges A->B and B->A. They may have different weights. Using your approach, it is quite confusing. One uses MPI_NEIGHBOR_ALLTOALLV to have better performance. We have to keep it in mind.

--Junchao Zhang

On Mon, Jul 23, 2018 at 9:59 AM, HOLMES Daniel <[email protected]> wrote:

...

Hi Junchao Zhang,

You are correct - the neighbourhood collectives send messages to (and receive messages from) only neighbouring MPI processes, as defined by the virtual topology. However, it is possible to use point-to-point, single-sided, or normal collective functions to send (and receive) messages between *any* MPI processes in a communicator, irrespective of the presence/structure of a virtual topology.

For your use-case (if I understand it correctly), you should specify a symmetric graph - each process specifies to MPI_DIST_GRAPH_CREATE_ADJACENT all the processes it wishes to communicate with (both sending and receiving) during subsequent calls to MPI_NEIGHBOR_ALLTOALLV.

Cheers, Dan. — Dr Daniel Holmes PhD Applications Consultant in HPC Research [email protected] Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 3415, JCMB, The King’s Buildings, Edinburgh, EH9 3FD — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. —

On 23 Jul 2018, at 15:45, Junchao Zhang <[email protected]> wrote:

Hello, Dan, I am interested in MPI_Neighbor_alltoallv. From its arguments, I do not see one can send a message to a neighbor not specified in the graph created by for example, MPI_Dist_graph_create_adjacent.

--Junchao Zhang

On Mon, Jul 23, 2018 at 5:51 AM, HOLMES Daniel <[email protected]> wrote:

...

Hi Junchao Zhang,

My understanding of the current API for MPI-3.1 is that:

1) the virtual topology does not actually restrict communication via the communicator to the edges specified in the topology - messages can be sent along any edge in either direction, and even between pairs of processes for which no edge was specified.

2) the virtual topology can be specified as a symmetric graph - for every ‘forward’ edge (e.g. from A to B), the ‘backward’ edge (i.e. from B to A) can be included as well.

3) there is already language in the MPI Standard regarding how MPI handles symmetric and non-symmetric graph topologies for neighbourhood collective operations.

Thus, there is no need to create two distributed graph topology communicators to achieve ‘forward and backward communication along the edges’.

Cheers, Dan. — Dr Daniel Holmes PhD Applications Consultant in HPC Research [email protected] Phone: +44 (0) 131 651 3465 Mobile: +44 (0) 7940 524 088 Address: Room 3415, JCMB, The King’s Buildings, Edinburgh, EH9 3FD — The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. —

On 15 Jul 2018, at 10:42, Anthony Skjellum <[email protected]> wrote:

Hi, I just saw this.

We definitely need to consider this concern.

I also need to go review the APIs.

Thanks, Tony

On Sat, Jul 14, 2018 at 12:27 AM, Junchao Zhang <[email protected]> wrote:

...

I want to try MPI neighborhood collectives. I have a communication graph and want to do both forward and backward communication along the edges. With current APIs, it looks I need to create two comm_dist_graphs. It is a waste since MPI implementations can do similar optimizations for the two. Should MPI support this scenario? Thanks.

--Junchao Zhang

_______________________________________________ mpiwg-coll mailing list [email protected] https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll

-- Tony Skjellum, PhD RunTime Computing Solutions, LLC [email protected] direct: +1-205-918-7514 cell: +1-205-807-4968 _______________________________________________ mpiwg-coll mailing list [email protected] https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll

The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

_______________________________________________ mpiwg-coll mailing list [email protected] https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll

_______________________________________________ mpiwg-coll mailing list [email protected] https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll

The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

_______________________________________________ mpiwg-coll mailing list [email protected] https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll

_______________________________________________ mpiwg-coll mailing list [email protected] https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll