October 2015 - mpi-forum - mailman3.cels.anl.gov

Re: [Mpi-forum] user-defined reductions and re-entrancy
by William Gropp 20 Oct '15

20 Oct '15

The standard doesn’t explicitly address this. However, in the spirit of the standard, I would say yes, multiple threads may be used. The behavior of side effects of the function are not defined, nor is it required to apply to elements 0..n-1 in that order in an n-element vector (this is a vector on each process; the is/is-not commutative applies between processes). Rather than try to enumerate and disallow certain actions, I think this is the right solution - the action of the user function on the input parameters is defined; nothing else should be. An implementation might warn users that the function may be called by multiple threads concurrently (e.g., to use atomic memory operations to count invocations or increment “evil”). Bill William Gropp Director, Parallel Computing Institute Thomas M. Siebel Chair in Computer Science Chief Scientist, NCSA University of Illinois Urbana-Champaign On Oct 20, 2015, at 1:04 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > Thanks Jed for clarifying. I agree with your modified example. > > The fundamental question here remains: can an implementation internally call user-defined reduction operators from multiple threads at the same time? > > Jeff > > Sent from my iPhone > >> On Oct 20, 2015, at 10:50 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: >> >> "Balaji, Pavan" <balaji(a)anl.gov> writes: >> >>> You can't call MPI functions in there, but you can replace the abort with an exit(1). >> >> MPI-3.0 p184: "MPI_ABORT may be called inside the function in case of an error." >> >>> Each user function is called once per process, so "evil" will always be 1, isn't it? I'm not sure why reentrance is an issue here. >> >> Jeff said "user code increments evil once between every call to this >> function so that it should never fail in single-threaded execution". >> >> A self-contained rendition: >> >> void User_reduce(void* invec, void* inoutvec, int *len, MPI_Datatype *datatype) >> { >> static int evil = 0; >> double * in = (double*)invec; >> double * out = (double*)inoutvec; >> if (*datatype == MPI_DOUBLE && (++evil%2) == 1) { >> for (int i=0; i<*len; i++) out[i] = evil*in[i]; >> evil++; >> } else { >> MPI_Abort(1,MPI_COMM_WORLD); >> } >> } >> >> The results certainly depend on how an implementation segments a buffer >> and on which process the reduction is called. Additionally, calling the >> reduction from multiple threads may cause a non-deterministic abort. >> _______________________________________________ >> mpi-forum mailing list >> mpi-forum(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum

1 0

Re: [Mpi-forum] user-defined reductions and re-entrancy
by Jeff Squyres (jsquyres) 20 Oct '15

20 Oct '15

On Oct 20, 2015, at 2:04 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > Thanks Jed for clarifying. I agree with your modified example. > > The fundamental question here remains: can an implementation internally call user-defined reduction operators from multiple threads at the same time? I don't think that the MPI spec defines this. -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [Mpi-forum] user-defined reductions and re-entrancy
by Jeff Hammond 20 Oct '15

20 Oct '15

Thanks Jed for clarifying. I agree with your modified example. The fundamental question here remains: can an implementation internally call user-defined reduction operators from multiple threads at the same time? Jeff Sent from my iPhone > On Oct 20, 2015, at 10:50 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > > "Balaji, Pavan" <balaji(a)anl.gov> writes: > >> You can't call MPI functions in there, but you can replace the abort with an exit(1). > > MPI-3.0 p184: "MPI_ABORT may be called inside the function in case of an error." > >> Each user function is called once per process, so "evil" will always be 1, isn't it? I'm not sure why reentrance is an issue here. > > Jeff said "user code increments evil once between every call to this > function so that it should never fail in single-threaded execution". > > A self-contained rendition: > > void User_reduce(void* invec, void* inoutvec, int *len, MPI_Datatype *datatype) > { > static int evil = 0; > double * in = (double*)invec; > double * out = (double*)inoutvec; > if (*datatype == MPI_DOUBLE && (++evil%2) == 1) { > for (int i=0; i<*len; i++) out[i] = evil*in[i]; > evil++; > } else { > MPI_Abort(1,MPI_COMM_WORLD); > } > } > > The results certainly depend on how an implementation segments a buffer > and on which process the reduction is called. Additionally, calling the > reduction from multiple threads may cause a non-deterministic abort. > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum

1 0

Re: [Mpi-forum] user-defined reductions and re-entrancy
by Jed Brown 20 Oct '15

20 Oct '15

"Balaji, Pavan" <balaji(a)anl.gov> writes: > You can't call MPI functions in there, but you can replace the abort with an exit(1). MPI-3.0 p184: "MPI_ABORT may be called inside the function in case of an error." > Each user function is called once per process, so "evil" will always be 1, isn't it? I'm not sure why reentrance is an issue here. Jeff said "user code increments evil once between every call to this function so that it should never fail in single-threaded execution". A self-contained rendition: void User_reduce(void* invec, void* inoutvec, int *len, MPI_Datatype *datatype) { static int evil = 0; double * in = (double*)invec; double * out = (double*)inoutvec; if (*datatype == MPI_DOUBLE && (++evil%2) == 1) { for (int i=0; i<*len; i++) out[i] = evil*in[i]; evil++; } else { MPI_Abort(1,MPI_COMM_WORLD); } } The results certainly depend on how an implementation segments a buffer and on which process the reduction is called. Additionally, calling the reduction from multiple threads may cause a non-deterministic abort.

1 0

Re: [Mpi-forum] user-defined reductions and re-entrancy
by Balaji, Pavan 20 Oct '15

20 Oct '15

You can't call MPI functions in there, but you can replace the abort with an exit(1). Each user function is called once per process, so "evil" will always be 1, isn't it? I'm not sure why reentrance is an issue here. -- Pavan From: mpi-forum on behalf of Jeff Hammond Reply-To: Main MPI Forum mailing list Date: Tuesday, October 20, 2015 at 11:02 AM To: Main MPI Forum mailing list Subject: [Mpi-forum] user-defined reductions and re-entrancy Is there anything in the MPI standard preventing this sort of disgusting behavior? /* user code increments evil once between every call to this function so that it should never fail in single-threaded execution */ static int evil = 0; MPI_User_function(void* invec, void* inoutvec, int *len, MPI_Datatype *datatype) { evil++; double * in = (double*)invec; double * out = (double*)inoutvec; if (*datatype == MPI_DOUBLE && (evil%2)==0) { for (int i=0; i<*len; i++) out[i] = evil*in[i]; } else { MPI_Abort(1,MPI_COMM_WORLD); } } My goal is to illustrate a user-defined reduction that is not reentrant. If you dislike my example, then please mentally replace it with a better one of your own creation :-) The more general question is, are there any constraints on whether implementations may internally use multiple threads to call user-defined reductions? I do not see any, but perhaps this is an oversight on my part. Thanks, Jeff -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> http://jeffhammond.github.io/

1 0

user-defined reductions and re-entrancy
by Jeff Hammond 20 Oct '15

20 Oct '15

Is there anything in the MPI standard preventing this sort of disgusting behavior? /* user code increments evil once between every call to this function so that it should never fail in single-threaded execution */ static int evil = 0; MPI_User_function(void* invec, void* inoutvec, int *len, MPI_Datatype *datatype) { evil++; double * in = (double*)invec; double * out = (double*)inoutvec; if (*datatype == MPI_DOUBLE && (evil%2)==0) { for (int i=0; i<*len; i++) out[i] = evil*in[i]; } else { MPI_Abort(1,MPI_COMM_WORLD); } } My goal is to illustrate a user-defined reduction that is not reentrant. If you dislike my example, then please mentally replace it with a better one of your own creation :-) The more general question is, are there any constraints on whether implementations may internally use multiple threads to call user-defined reductions? I do not see any, but perhaps this is an oversight on my part. Thanks, Jeff -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

MPI program on Cluster: Errors
by Arsalan Shahid 07 Oct '15

07 Oct '15

Hello, We have started to run MPI programs on cluster. Currently, we have been facing some issues while running programs on cluster. As a start, we have setup Beowulf cluster on two PC's. Simple "hello world" program runs perfectly fine on cluster on both Master and slave nodes. But, while running MPI programs using communicators and message passing routines, we are getting errors as below. Moreover, if we run the same program with number of cores equal to 1-4, the code works fine. I wonder if someone can sort out this issue ? xenox@master:/nfs$ mpirun -np 8 -f machinefile ./MPMPI 4 Fatal error in PMPI_Barrier: A process has failed, error stack: PMPI_Barrier(428).........: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(335)....: Failure during collective MPIR_Barrier_impl(328)....: MPIR_Barrier(292).........: MPIR_Barrier_intra(149)...: barrier_smp_intra(94).....: MPIR_Barrier_impl(335)....: Failure during collective MPIR_Barrier_impl(328)....: MPIR_Barrier(292).........: MPIR_Barrier_intra(169)...: dequeue_and_set_error(865): Communication error with rank 4 barrier_smp_intra(109)....: MPIR_Bcast_impl(1458).....: MPIR_Bcast(1482)..........: MPIR_Bcast_intra(1291)....: MPIR_Bcast_binomial(309)..: Failure during collective Fatal error in PMPI_Barrier: Other MPI error, error stack: PMPI_Barrier(428).......: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(335)..: Failure during collective MPIR_Barrier_impl(328)..: MPIR_Barrier(292).......: MPIR_Barrier_intra(149).: barrier_smp_intra(109)..: MPIR_Bcast_impl(1458)...: MPIR_Bcast(1482)........: MPIR_Bcast_intra(1291)..: MPIR_Bcast_binomial(309): Failure during collective Fatal error in PMPI_Barrier: Other MPI error, error stack: PMPI_Barrier(428).......: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(335)..: Failure during collective MPIR_Barrier_impl(328)..: MPIR_Barrier(292).......: MPIR_Barrier_intra(149).: barrier_smp_intra(109)..: MPIR_Bcast_impl(1458)...: MPIR_Bcast(1482)........: MPIR_Bcast_intra(1291)..: MPIR_Bcast_binomial(309): Failure during collective Fatal error in PMPI_Barrier: Other MPI error, error stack: PMPI_Barrier(428).......: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(335)..: Failure during collective MPIR_Barrier_impl(328)..: MPIR_Barrier(292).......: MPIR_Barrier_intra(149).: barrier_smp_intra(109)..: MPIR_Bcast_impl(1458)...: MPIR_Bcast(1482)........: MPIR_Bcast_intra(1291)..: MPIR_Bcast_binomial(309): Failure during collective =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 15777 RUNNING AT master = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:1@slave1] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:886): assert (!closed) failed [proxy:0:1@slave1] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [proxy:0:1@slave1] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@master] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@master] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@master] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion [mpiexec@master] main (ui/mpich/mpiexec.c:336): process manager error waiting for completion -Regards -- *-Arsalan Shahid* * RA, ACAL Research centre * * Department of Electrical Engineering* * HITEC University-* *Taxila Cantt-Islamabad-Pakistan* * *

1 0

Slides from Bordeaux
by Jeff Squyres (jsquyres) 02 Oct '15

02 Oct '15

Please don't forget to send me your slides, if you haven't done so already. Thanks. -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0