June 2015 - devel - mailman3.cels.anl.gov

Re: [mpich-devel] MPICH hangs in MPI_Waitall when MPI_Cancel is used
by Halim Amer 04 Jun '15

04 Jun '15

Hi Jeff, I don't think it is a correct program. If the send is correctly canceled then the origin has to satisfy the destination with another send. The hang is an expected result. This is what the standard says (P102): "...or that the send is successfully cancelled, in which case no part of the message was received at the destination. Then, any matching receive has to be satisfied by another send." --Halim Abdelhalim Amer (Halim) Postdoctoral Appointee MCS Division Argonne National Laboratory On 6/4/15 9:21 AM, Jeff Hammond wrote: > I can't tell for sure if this is a correct program, but multiple > members of the MPI Forum suggested it is. > > If it is a correct program, it appears to expose a bug in MPICH, > because the MPI_Waitall hangs. > > Thanks, > > Jeff > > $ mpicc -g -Wall -std=c99 cancel-sucks.c && mpiexec -n 4 ./a.out > > $ mpichversion > MPICH Version: 3.2b1 > MPICH Release date: unreleased development copy > MPICH Device: ch3:nemesis > MPICH configure: CC=gcc-4.9 CXX=g++-4.9 FC=gfortran-4.9 > F77=gfortran-4.9 --enable-cxx --enable-fortran > --enable-threads=runtime --enable-g=dbg --with-pm=hydra > --prefix=/opt/mpich/dev/gcc/default --enable-wrapper-rpath > --enable-static --enable-shared > MPICH CC: gcc-4.9 -g -O2 > MPICH CXX: g++-4.9 -g -O2 > MPICH F77: gfortran-4.9 -g -O2 > MPICH FC: gfortran-4.9 -g -O2 > > > #include <stdio.h> > #include <stdlib.h> > #include <mpi.h> > > const int n=1000; > > int main(void) > { > MPI_Init(NULL,NULL); > > int size, rank; > MPI_Comm_size(MPI_COMM_WORLD, &size); > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > if (size<2) { > printf("You must use 2 or more processes!\n"); > MPI_Finalize(); > exit(1); > } > > MPI_Request reqs[2*n]; > > int target = (rank+1)%size; > for (int i=0; i<n; i++) { > MPI_Issend(NULL,0,MPI_BYTE,target,0,MPI_COMM_WORLD,&(reqs[i])); > } > > srand((unsigned)(rank+MPI_Wtime())); > int slot = rand()%n; > printf("Cancelling send %d.\n", slot); > MPI_Cancel(&reqs[slot]); > > #if 1 > MPI_Barrier(MPI_COMM_WORLD); > #endif > > int origin = (rank==0) ? (size-1) : (rank-1); > for (int i=0; i<n; i++) { > MPI_Irecv(NULL,0,MPI_BYTE,origin,0,MPI_COMM_WORLD,&(reqs[n+i])); > } > > MPI_Status stats[2*n]; > MPI_Waitall(2*n,reqs,stats); > > for (int i=0; i<n; i++) { > int flag; > MPI_Test_cancelled(&(stats[i]),&flag); > if (flag) { > printf("Status %d indicates cancel was successful.\n", i); > } > } > > MPI_Finalize(); > return 0; > } > >

1 0

MPICH hangs in MPI_Waitall when MPI_Cancel is used
by Jeff Hammond 04 Jun '15

04 Jun '15

I can't tell for sure if this is a correct program, but multiple members of the MPI Forum suggested it is. If it is a correct program, it appears to expose a bug in MPICH, because the MPI_Waitall hangs. Thanks, Jeff $ mpicc -g -Wall -std=c99 cancel-sucks.c && mpiexec -n 4 ./a.out $ mpichversion MPICH Version: 3.2b1 MPICH Release date: unreleased development copy MPICH Device: ch3:nemesis MPICH configure: CC=gcc-4.9 CXX=g++-4.9 FC=gfortran-4.9 F77=gfortran-4.9 --enable-cxx --enable-fortran --enable-threads=runtime --enable-g=dbg --with-pm=hydra --prefix=/opt/mpich/dev/gcc/default --enable-wrapper-rpath --enable-static --enable-shared MPICH CC: gcc-4.9 -g -O2 MPICH CXX: g++-4.9 -g -O2 MPICH F77: gfortran-4.9 -g -O2 MPICH FC: gfortran-4.9 -g -O2 #include <stdio.h> #include <stdlib.h> #include <mpi.h> const int n=1000; int main(void) { MPI_Init(NULL,NULL); int size, rank; MPI_Comm_size(MPI_COMM_WORLD, &size); MPI_Comm_rank(MPI_COMM_WORLD, &rank); if (size<2) { printf("You must use 2 or more processes!\n"); MPI_Finalize(); exit(1); } MPI_Request reqs[2*n]; int target = (rank+1)%size; for (int i=0; i<n; i++) { MPI_Issend(NULL,0,MPI_BYTE,target,0,MPI_COMM_WORLD,&(reqs[i])); } srand((unsigned)(rank+MPI_Wtime())); int slot = rand()%n; printf("Cancelling send %d.\n", slot); MPI_Cancel(&reqs[slot]); #if 1 MPI_Barrier(MPI_COMM_WORLD); #endif int origin = (rank==0) ? (size-1) : (rank-1); for (int i=0; i<n; i++) { MPI_Irecv(NULL,0,MPI_BYTE,origin,0,MPI_COMM_WORLD,&(reqs[n+i])); } MPI_Status stats[2*n]; MPI_Waitall(2*n,reqs,stats); for (int i=0; i<n; i++) { int flag; MPI_Test_cancelled(&(stats[i]),&flag); if (flag) { printf("Status %d indicates cancel was successful.\n", i); } } MPI_Finalize(); return 0; } -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-devel] Issue with too many calls to MPIR_Add_finalize
by Lisandro Dalcin 04 Jun '15

04 Jun '15

On 3 June 2015 at 21:57, Rob Latham <robl(a)mcs.anl.gov> wrote: > > > On 05/07/2015 07:45 AM, Lisandro Dalcin wrote: >> >> I'm running the testsuite of my Python wrappers with MPICH 3.2b2. The >> runs are aborting this way: > > > Sorry no one got back to you yet. I opened a ticket to track this issue > > https://trac.mpich.org/projects/mpich/ticket/2272 > Although I've not tested, this fix should be enough: diff --git a/src/mpi/init/finalize.c b/src/mpi/init/finalize.c index f71668b..afebd77 100644 --- a/src/mpi/init/finalize.c +++ b/src/mpi/init/finalize.c @@ -65,7 +65,7 @@ typedef struct Finalize_func_t { } Finalize_func_t; /* When full debugging is enabled, each MPI handle type has a finalize handler installed to detect unfreed handles. */ -#define MAX_FINALIZE_FUNC 32 +#define MAX_FINALIZE_FUNC 64 static Finalize_func_t fstack[MAX_FINALIZE_FUNC]; static int fstack_sp = 0; static int fstack_max_priority = 0; -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459

1 0

Re: [mpich-devel] Issue with too many calls to MPIR_Add_finalize
by Rob Latham 03 Jun '15

03 Jun '15

On 05/07/2015 07:45 AM, Lisandro Dalcin wrote: > I'm running the testsuite of my Python wrappers with MPICH 3.2b2. The > runs are aborting this way: Sorry no one got back to you yet. I opened a ticket to track this issue https://trac.mpich.org/projects/mpich/ticket/2272 ==rob > > $ python test/runtests.py -q > overflow in finalize stack! > internal ABORT - process 0 > [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=13 > : > system msg for write_line failure : Bad file descriptor > > It seems that the routine MPIR_Add_finalize() is being called more > than 32 times, exhausting the static allocation space of 32 entries. > I'm attaching a text file with a gdb sessions showing all the calls to > this routine. > > > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0