William, yes, I reported the bahavior of 5 different MPI existing MPI implementations. And I attached my test program that calls the MPI topology functions and reports the behavior of the MPI library to which it is compiled and linked. Using this test program, everybody can detect that different MPI implementations are giving quite different answers and therefore, there is a need for clarifying the MPI 1.1 standard. Best regards Rolf On Thu, 24 Jan 2008 05:02:47 +0800 "William Yu" <wyu_at_[hidden]> wrote:

Hi Rolf,

Just curious. Is this implemented already? Sounds like you have a reference implementation built against different MPIs.

Thanks.

________________ Reply Header ________________ Subject: [mpi-21] Ballot 4 - 0-dim MPI_CART_CREATE and MPI_CART_SUB Author: "Rolf Rabenseifner" <rabenseifner_at_[hidden]> Date: January 23rd 2008 5:44 pm

This is a discussion-point for MPI 2.1, Ballot 4.

This is a follow up to: Questions about Graph_create, Cart_crate, and Cart_coords in http://www.cs.uiuc.edu/homes/wgropp/projects/parallel/MPI/mpi-errata/index.h... with mail discussion in http://www.cs.uiuc.edu/homes/wgropp/projects/parallel/MPI/mpi-errata/discuss...

I'm starting a new 2nd stream for the zero-dim discussion. -------------------------------

This report shows significant differences of MPI implementations in the (extreme) case of zero-dimensional topologies!

Therefore clarifications may be necessary. (Although hopefully never an application may produce zero-dimensional Cartesian topologies!)

If there are no objections, I would produce clarifications for MPI-2.1 according to the behavior of mpich2 and MPI from IBM (except the rank -767705708, see below).

MPI-1.1 Sect. 6.5.6, page 187, routine MPI_CART_SUB defines on lines 38-42: If a cartesian topology has been created with MPI_CART_CREATE, the function MPI_CART_SUB can be used to partition the communicator group into subgroups that form lower-dimensional cartesian subgrids, and to build for each subgroup a communicator with the associated subgrid cartesian topology. (This function is closely related to MPI_COMM_SPLIT.)

The text clearly says, that the new communicator must (1) have a Cartesian topology associated (2) be lower-dimensional in the case of a subgrid

There is no restriction on the input array remain_dims. Therefore all MPI implementations (that I tested) allow that all entries in remain_dims are "false".

I tested several MPI libraries with creating a subgrid with MPI_Cart_sub(remain_dims=0) from a 1-dim Cartesian topology. I tested the subgrid communicator with MPI_Topo_test. In the case of MPI_CART, I used MPI_Cartdim_get to retrieve ndims, and MPI_Cart_get for further details:

- mpich2: MPI_CART, ndims=0, 1.0.3 MPI_Cart_get works, but keeps all OUT arguments unchanged

- IBM: MPI_CART, ndims=0, (on SP) MPI_Cart_get works, but keeps all OUT arguments unchanged

- OpenMPI: MPI_CART, ndims=1, 1.2.4 MPI_Cart_get works and returns dims=1, periods=0, coords=0 independent from process or periods in the original comm. (may be wrong because (2) is not fulfilled)

- NEC MP/EX: not MPI_CART (may be wrong because (1) is not fulfilled)

- Voltaire: not MPI_CART (mpich1) (may be wrong because (1) is not fulfilled)

With the implementations that return a correct zero-dim Cartesian topology, I tested further usage of this zero-dim communicator:

- MPI_Comm_size returns 1 and MPI_Comm_rank returns 0 because this communicator is like MPI_COMM_SELF, but with Cartesian topology associated.

- MPI_Cart_rank(IN ZeroDimComm, IN coords=0, OUT rank) Rationale. A zero-dim communicator has zero coords, i.e., this routine should not examine the coords input argument. --> mpich2: rank = 0 is returned (may be because this is the only existing rank in this communicator, this value may make sense, independent of the coord, that should not be analyzed) --> IBM: rank = -767705708 is returned (strange value, not MPI_PROC_NULL, not MPI_UNDEFINED)

- MPI_Cart_coords(IN ZeroDimComm, IN rank=0, OUT coords) Rationale. A zero-dim communicator has zero coords, i.e., this routine should not return anything in the coords output argument. --> mpich2 and IBM: coords is not modified (as expected)

- MPI_Cart_sub(IN ZeroDimComm, IN remain_dims=0, OUT subsubcomm) Rationale. A zero-dim communicator has zero dimensions, i.e., this routine should not examine remain_dims and the returned communicator should be again a zero-dim Cartesian communicator. --> mpich2 and IBM: subsubcomm is a zero-dim Cartesian communicator (as expected)

- MPI_Cart_shift(IN ZeroDimComm, IN direction=0, IN disp=1, OUT src, OUT dest) Rationale. This call is erroneous because in a zero-dim communicator, the direction=0 does not exist. mpich2 and IBM: They detect the error and abort.

(In OpenMPI, all these calls work as expected on a 1-dimensional topology on a "MPI_COMM_SELF")

The last test is not addressed by MPI-1.1: Is it possible to build a zero-dim Cartesian topology directly by calling MPI_Cart_create:

MPI_Cart_create(IN MPI_COMM_SELF, IN ndims=0, IN dims=1, IN Periods, IN reorder, OUT ZeroDimComm)

Results: All tested MPI implementations return an error and abort. (Same on MPI_COMM_WORLD)

The latter question was raised by Jesper Traeff.

Best regards Rolf

Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner_at_[hidden] High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

_______________________________________________ mpi-21 mailing list mpi-21_at_[hidden] http://lists.cs.uiuc.edu/mailman/listinfo/mpi-21

_______________________________________________ mpi-21 mailing list mpi-21_at_[hidden] http://lists.cs.uiuc.edu/mailman/listinfo/mpi-21

Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner_at_[hidden] High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)