Hi Jeff, I don't think it is a correct program. If the send is correctly canceled then the origin has to satisfy the destination with another send. The hang is an expected result. This is what the standard says (P102): "...or that the send is successfully cancelled, in which case no part of the message was received at the destination. Then, any matching receive has to be satisfied by another send." --Halim Abdelhalim Amer (Halim) Postdoctoral Appointee MCS Division Argonne National Laboratory On 6/4/15 9:21 AM, Jeff Hammond wrote:

I can't tell for sure if this is a correct program, but multiple members of the MPI Forum suggested it is.

If it is a correct program, it appears to expose a bug in MPICH, because the MPI_Waitall hangs.

Thanks,

Jeff

$ mpicc -g -Wall -std=c99 cancel-sucks.c && mpiexec -n 4 ./a.out

$ mpichversion MPICH Version: 3.2b1 MPICH Release date: unreleased development copy MPICH Device: ch3:nemesis MPICH configure: CC=gcc-4.9 CXX=g++-4.9 FC=gfortran-4.9 F77=gfortran-4.9 --enable-cxx --enable-fortran --enable-threads=runtime --enable-g=dbg --with-pm=hydra --prefix=/opt/mpich/dev/gcc/default --enable-wrapper-rpath --enable-static --enable-shared MPICH CC: gcc-4.9 -g -O2 MPICH CXX: g++-4.9 -g -O2 MPICH F77: gfortran-4.9 -g -O2 MPICH FC: gfortran-4.9 -g -O2

#include <stdio.h> #include <stdlib.h> #include <mpi.h>

const int n=1000;

int main(void) { MPI_Init(NULL,NULL);

int size, rank; MPI_Comm_size(MPI_COMM_WORLD, &size); MPI_Comm_rank(MPI_COMM_WORLD, &rank); if (size<2) { printf("You must use 2 or more processes!\n"); MPI_Finalize(); exit(1); }

MPI_Request reqs[2*n];

int target = (rank+1)%size; for (int i=0; i<n; i++) { MPI_Issend(NULL,0,MPI_BYTE,target,0,MPI_COMM_WORLD,&(reqs[i])); }

srand((unsigned)(rank+MPI_Wtime())); int slot = rand()%n; printf("Cancelling send %d.\n", slot); MPI_Cancel(&reqs[slot]);

#if 1 MPI_Barrier(MPI_COMM_WORLD); #endif

int origin = (rank==0) ? (size-1) : (rank-1); for (int i=0; i<n; i++) { MPI_Irecv(NULL,0,MPI_BYTE,origin,0,MPI_COMM_WORLD,&(reqs[n+i])); }

MPI_Status stats[2*n]; MPI_Waitall(2*n,reqs,stats);

for (int i=0; i<n; i++) { int flag; MPI_Test_cancelled(&(stats[i]),&flag); if (flag) { printf("Status %d indicates cancel was successful.\n", i); } }

MPI_Finalize(); return 0; }