February 2013 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] PROBLEM IN MPI CODE EXECUTION
by Jayesh Krishna 27 Feb '13

27 Feb '13

Hi, Did you try registering a valid Windows username/password with mpiexec (mpiexec -register)? Regards, Jayesh ----- Original Message ----- From: "pooja dubey" <pjdb22(a)gmail.com> To: discuss(a)mpich.org Sent: Tuesday, February 26, 2013 12:13:32 PM Subject: [mpich-discuss] PROBLEM IN MPI CODE EXECUTION whenever i try to execute a code, i shows the following error- Credentials for NISHA rejected connecting to nisha Aborting: Unable to connect to nisha please tell me how can i execute the code to see output. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Debugging mpi program using mpich
by Jim Dinan 27 Feb '13

27 Feb '13

On 2/27/13 3:03 AM, Mathieu Dutour wrote: > Thank you very much! That will be helpful. > Still what I meant was something a little different. > I would like to have a mode in mpi where every little error is reported. > "Like process 1 has send data of length 10 but process 0 needed only 9 > at line 400 and 500" > "mpi process request at line 240 has not been honored" > "process has left without mpi_finalize" > "mpi_sent uses MPI_FLOAT but corresponding mpi_recv uses MPI_DOUBLE" MPICH will already detect and report many common errors like the one you described. That said, our coverage could definitely use improvement and we're always glad to get reports of specific errors that should be, but are not detected. > Also deadly embrace could be detected by the mpi library I think. I look forward to your patch. ;) > All this kind of debugging would be extremely helpful in practice and I > would gladly sacrifice speed if that allows to avoid the long thankless > work of debugging. We definitely agree that this would be extremely useful. We have limited developer time available, but are always happy to partner with others in the development of such features. Best, ~Jim.

1 0

Re: [mpich-discuss] Affinity with MPICH_ASYNC_PROGRESS
by Dave Goodell 27 Feb '13

27 Feb '13

On Feb 27, 2013, at 3:19 AM CST, Pavan Balaji wrote: > On 02/25/2013 08:41 AM US Central Time, Jed Brown wrote: >> It sounds like you are binding a process. Suppose hydra binds my process >> to cpuset 0xf and my process pthread_creates a total of three threads. >> If I bind those threads to 0x1, 0x2, 0x4, we still have the >> MPICH_ASYNC_PROGRESS running with affinity 0xf, and as far as I know, >> there is no way to bind it to 0x8. >> >> It would be interesting to be able to specify this, both to place the >> progress threads close to NICs and to make sure that progress threads do >> not disrupt application threads (to reduce jitter and possibly improve >> performance). > > There's no plan to support per-thread binding. This requires mpich to > keep track of thread IDs, which AFAICT, we don't, unless I'm forgetting > some piece deep inside MPICH (Dave or Jim should correct me, if I'm wrong). We do use thread-ids in some cases, but we definitely aren't tracking the overall set of threads in existence or otherwise manipulating them. I think we mainly use thread-ids to deal with thread-local storage in some cases. -Dave

1 0

Re: [mpich-discuss] Affinity with MPICH_ASYNC_PROGRESS
by Pavan Balaji 27 Feb '13

27 Feb '13

On 02/25/2013 08:41 AM US Central Time, Jed Brown wrote: > It sounds like you are binding a process. Suppose hydra binds my process > to cpuset 0xf and my process pthread_creates a total of three threads. > If I bind those threads to 0x1, 0x2, 0x4, we still have the > MPICH_ASYNC_PROGRESS running with affinity 0xf, and as far as I know, > there is no way to bind it to 0x8. > > It would be interesting to be able to specify this, both to place the > progress threads close to NICs and to make sure that progress threads do > not disrupt application threads (to reduce jitter and possibly improve > performance). There's no plan to support per-thread binding. This requires mpich to keep track of thread IDs, which AFAICT, we don't, unless I'm forgetting some piece deep inside MPICH (Dave or Jim should correct me, if I'm wrong). -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Debugging mpi program using mpich
by Mathieu Dutour 27 Feb '13

27 Feb '13

Thank you very much! That will be helpful. Still what I meant was something a little different. I would like to have a mode in mpi where every little error is reported. "Like process 1 has send data of length 10 but process 0 needed only 9 at line 400 and 500" "mpi process request at line 240 has not been honored" "process has left without mpi_finalize" "mpi_sent uses MPI_FLOAT but corresponding mpi_recv uses MPI_DOUBLE" Also deadly embrace could be detected by the mpi library I think. All this kind of debugging would be extremely helpful in practice and I would gladly sacrifice speed if that allows to avoid the long thankless work of debugging. Mathieu On Tue, Feb 26, 2013 at 5:38 PM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: > You can run your application under valgrind with a debug build of MPICH > and you should be able to eventually find your leaks that way (though the > output might be a bit hard to read at first, since you will be tracking > leaks in memory associated with the MPI object, not the object itself). > > Just run your app like this: > > ----8<---- > mpiexec -n NUMPROCS valgrind /path/to/your/app > ----8<---- > > If the valgrind output gets too jumbled, you can separate them by passing > the "--log-file" option to Valgrind, like this: > > ----8<---- > mpiexec -n NUMPROCS valgrind --log-file='vg_out.%q{PMI_RANK}' /path/to/app > ----8<---- > > This will deposit one file per process, suffixed by the rank in > MPI_COMM_WORLD. > > -Dave > > On Feb 26, 2013, at 10:17 AM CST, Mathieu Dutour wrote: > > > Thank you, I will try TotalView. > > Now another question: > > ---In C we can run program with valgrind and it tells you exactly where > memory is lost, when you use unitialized values and so on. > > ---In fortran with ifort you can run with > > -warn interfaces,nouncalled -fpp -gen-interface -g -traceback -check > uninit -check bounds -check pointers > > and you can do the same. > > > > It would be extremely helpful to have similar tools in MPI that > sacrifice speed > > and allows you to find any error during runtime. > > > > Mathieu > > > > > > On Tue, Feb 26, 2013 at 5:01 PM, Dave Goodell <goodell(a)mcs.anl.gov> > wrote: > > On Feb 26, 2013, at 9:39 AM CST, Mathieu Dutour wrote: > > > > > I used mpich-3.0.1 with debugging options and the program ran correctly > > > but at the end returned some errors indicated later. > > > I thank mpich for finding those errors that other mpi implementation > did > > > not find but I wonder if there is a way to transform this into more > useful > > > debugging informations. > > > > High-quality patches to improve the output are welcome. We primarily > view these leak-checking messages as tools for us (the core developers of > MPICH), not for end-user consumption. So we probably won't spend any time > to change these messages ourselves. > > > > > Mathieu > > > > > > PS: The errors retuned after leaving: > > > leaked context IDs detected: mask=0x9d7380 mask[0]=0x3fffffff > > > In direct memory block for handle type GROUP, 3 handles are still > allocated > > > > […] > > > > In case you have not yet found your bug, these messages are indicating > that you are leaking MPI objects, especially communicators, groups, and > datatypes. It could be that they are leaked indirectly because you have > not completed an outstanding request (via MPI_Wait or similar), as > indicated by the lines with "REQUEST" in them. > > > > -Dave > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

suspend/resume an MPI process using SIGSTOP and SIGCONT
by Ali Bouaricha 26 Feb '13

26 Feb '13

Hello, I wanted to suspend a process in MPICH2 by using kill(procid, SIGSTOP) But it seems that when I do that, the whole run gets suspended. Is there any implication of kill(procid,SIGSTOP) on the remainder of processes? Please help. Thank you and Regards, Ali Bouaricha

1 0

PROBLEM IN MPI CODE EXECUTION
by pooja dubey 26 Feb '13

26 Feb '13

whenever i try to execute a code, i shows the following error- Credentials for NISHA rejected connecting to nisha Aborting: Unable to connect to nisha please tell me how can i execute the code to see output.

1 0

Re: [mpich-discuss] Debugging mpi program using mpich
by Dave Goodell 26 Feb '13

26 Feb '13

You can run your application under valgrind with a debug build of MPICH and you should be able to eventually find your leaks that way (though the output might be a bit hard to read at first, since you will be tracking leaks in memory associated with the MPI object, not the object itself). Just run your app like this: ----8<---- mpiexec -n NUMPROCS valgrind /path/to/your/app ----8<---- If the valgrind output gets too jumbled, you can separate them by passing the "--log-file" option to Valgrind, like this: ----8<---- mpiexec -n NUMPROCS valgrind --log-file='vg_out.%q{PMI_RANK}' /path/to/app ----8<---- This will deposit one file per process, suffixed by the rank in MPI_COMM_WORLD. -Dave On Feb 26, 2013, at 10:17 AM CST, Mathieu Dutour wrote: > Thank you, I will try TotalView. > Now another question: > ---In C we can run program with valgrind and it tells you exactly where memory is lost, when you use unitialized values and so on. > ---In fortran with ifort you can run with > -warn interfaces,nouncalled -fpp -gen-interface -g -traceback -check uninit -check bounds -check pointers > and you can do the same. > > It would be extremely helpful to have similar tools in MPI that sacrifice speed > and allows you to find any error during runtime. > > Mathieu > > > On Tue, Feb 26, 2013 at 5:01 PM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: > On Feb 26, 2013, at 9:39 AM CST, Mathieu Dutour wrote: > > > I used mpich-3.0.1 with debugging options and the program ran correctly > > but at the end returned some errors indicated later. > > I thank mpich for finding those errors that other mpi implementation did > > not find but I wonder if there is a way to transform this into more useful > > debugging informations. > > High-quality patches to improve the output are welcome. We primarily view these leak-checking messages as tools for us (the core developers of MPICH), not for end-user consumption. So we probably won't spend any time to change these messages ourselves. > > > Mathieu > > > > PS: The errors retuned after leaving: > > leaked context IDs detected: mask=0x9d7380 mask[0]=0x3fffffff > > In direct memory block for handle type GROUP, 3 handles are still allocated > > […] > > In case you have not yet found your bug, these messages are indicating that you are leaking MPI objects, especially communicators, groups, and datatypes. It could be that they are leaked indirectly because you have not completed an outstanding request (via MPI_Wait or similar), as indicated by the lines with "REQUEST" in them. > > -Dave > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Debugging mpi program using mpich
by Mathieu Dutour 26 Feb '13

26 Feb '13

Thank you, I will try TotalView. Now another question: ---In C we can run program with valgrind and it tells you exactly where memory is lost, when you use unitialized values and so on. ---In fortran with ifort you can run with -warn interfaces,nouncalled -fpp -gen-interface -g -traceback -check uninit -check bounds -check pointers and you can do the same. It would be extremely helpful to have similar tools in MPI that sacrifice speed and allows you to find any error during runtime. Mathieu On Tue, Feb 26, 2013 at 5:01 PM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: > On Feb 26, 2013, at 9:39 AM CST, Mathieu Dutour wrote: > > > I used mpich-3.0.1 with debugging options and the program ran correctly > > but at the end returned some errors indicated later. > > I thank mpich for finding those errors that other mpi implementation did > > not find but I wonder if there is a way to transform this into more > useful > > debugging informations. > > High-quality patches to improve the output are welcome. We primarily view > these leak-checking messages as tools for us (the core developers of > MPICH), not for end-user consumption. So we probably won't spend any time > to change these messages ourselves. > > > Mathieu > > > > PS: The errors retuned after leaving: > > leaked context IDs detected: mask=0x9d7380 mask[0]=0x3fffffff > > In direct memory block for handle type GROUP, 3 handles are still > allocated > > […] > > In case you have not yet found your bug, these messages are indicating > that you are leaking MPI objects, especially communicators, groups, and > datatypes. It could be that they are leaked indirectly because you have > not completed an outstanding request (via MPI_Wait or similar), as > indicated by the lines with "REQUEST" in them. > > -Dave > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Debugging mpi program using mpich
by Dave Goodell 26 Feb '13

26 Feb '13

On Feb 26, 2013, at 9:39 AM CST, Mathieu Dutour wrote: > I used mpich-3.0.1 with debugging options and the program ran correctly > but at the end returned some errors indicated later. > I thank mpich for finding those errors that other mpi implementation did > not find but I wonder if there is a way to transform this into more useful > debugging informations. High-quality patches to improve the output are welcome. We primarily view these leak-checking messages as tools for us (the core developers of MPICH), not for end-user consumption. So we probably won't spend any time to change these messages ourselves. > Mathieu > > PS: The errors retuned after leaving: > leaked context IDs detected: mask=0x9d7380 mask[0]=0x3fffffff > In direct memory block for handle type GROUP, 3 handles are still allocated […] In case you have not yet found your bug, these messages are indicating that you are leaking MPI objects, especially communicators, groups, and datatypes. It could be that they are leaked indirectly because you have not completed an outstanding request (via MPI_Wait or similar), as indicated by the lines with "REQUEST" in them. -Dave

1 0