April 2013 - devel - mailman3.cels.anl.gov

Re: [mpich-devel] MPI_Bsend under MPIU_THREAD_GRANULARITY_PER_OBJECT
by Dave Goodell 25 Apr '13

25 Apr '13

In that case, test programs to help us reproduce this are also welcome. There's also always a risk that one of the currently missing ALLFUNC critical sections is affecting your user: https://trac.mpich.org/projects/mpich/ticket/1797 -Dave On Apr 25, 2013, at 11:38 AM CDT, Bob Cernohous <bobc(a)us.ibm.com> wrote: > Patches could be tricky because I'm even seeing intermittent (and different) failures on our 'legacy' libraries which are not per-obj but use the big lock. So there's probably more than one problem here. > > Abort(1) on node 3 (rank 3 in comm 1140850688): Fatal error in MPI_Bsend: Internal MPI error!, error stack: > MPI_Bsend(181)..............: MPI_Bsend(buf=0x19c8a06da0, count=1024, MPI_CHAR, dest=0, tag=0, MPI_COMM_WORLD) failed > MPIR_Bsend_isend(226).......: > MPIR_Bsend_check_active(456): > MPIR_Test_impl(65)..........: > MPIR_Request_complete(234)..: INTERNAL ERROR: unexpected value in case statement (value=0) > > > Bob Cernohous: (T/L 553) 507-253-6093 > > BobC(a)us.ibm.com > IBM Rochester, Building 030-2(C335), Department 61L > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > Chaos reigns within. > > Reflect, repent, and reboot. > > Order shall return. > > > > > From: Dave Goodell <goodell(a)mcs.anl.gov> > To: devel(a)mpich.org, > Date: 04/25/2013 11:14 AM > Subject: Re: [mpich-devel] MPI_Bsend under MPIU_THREAD_GRANULARITY_PER_OBJECT > Sent by: devel-bounces(a)mpich.org > > > > The Bsend paths almost certainly have not been protected correctly. Patches to fix the issue are most welcome. > > -Dave > > On Apr 25, 2013, at 10:49 AM CDT, Bob Cernohous <bobc(a)us.ibm.com> wrote: > > > Start by saying that I have not been involved in the nitty gritty of the per-object locking design. > > > > What protects the attached buffer/data structures/request when doing multithreaded MPI_Bsend()'s? All I see in the code path is a (no-op) MPIU_THREAD_CS_ENTER(ALLFUNC,). > > > > I have a customer test in which the threads seem to be walking all over the request around: > > > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpid/pamid/include/../src/mpid_request.h:259 > > 0000000001088c0c MPIR_Request_complete > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/mpir_request.c:87 > > 000000000106e874 MPIR_Test_impl > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/test.c:62 > > 00000000010188f0 MPIR_Bsend_check_active > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/bsendutil.c:455 > > 0000000001018dc0 MPIR_Bsend_isend > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/bsendutil.c:226 > > 0000000001008734 PMPI_Bsend > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/bsend.c:163 > > 00000000010009c0 00000012.long_branch_r2off.__libc_start_main+0 > > :0 > > 000000000130cbc0 start_thread > > > > eg. (fprinting from MPIU_HANDLE_LOG_REFCOUNT_CHANGE) > > > > stderr[8]: set 0x15f8048 (0xac0000ff kind=REQUEST) refcount to 2 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to 1 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to 0 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -1 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -2 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -3 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -4 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -5 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -6 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -7 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -8 > > > > > > Bob Cernohous: (T/L 553) 507-253-6093 > > > > BobC(a)us.ibm.com > > IBM Rochester, Building 030-2(C335), Department 61L > > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > > > Chaos reigns within. > > > Reflect, repent, and reboot. > > > Order shall return. > >

1 0

Re: [mpich-devel] MPI_Bsend under MPIU_THREAD_GRANULARITY_PER_OBJECT
by Bob Cernohous 25 Apr '13

25 Apr '13

Patches could be tricky because I'm even seeing intermittent (and different) failures on our 'legacy' libraries which are not per-obj but use the big lock. So there's probably more than one problem here. Abort(1) on node 3 (rank 3 in comm 1140850688): Fatal error in MPI_Bsend: Internal MPI error!, error stack: MPI_Bsend(181)..............: MPI_Bsend(buf=0x19c8a06da0, count=1024, MPI_CHAR, dest=0, tag=0, MPI_COMM_WORLD) failed MPIR_Bsend_isend(226).......: MPIR_Bsend_check_active(456): MPIR_Test_impl(65)..........: MPIR_Request_complete(234)..: INTERNAL ERROR: unexpected value in case statement (value=0) Bob Cernohous: (T/L 553) 507-253-6093 BobC(a)us.ibm.com IBM Rochester, Building 030-2(C335), Department 61L 3605 Hwy 52 North, Rochester, MN 55901-7829 > Chaos reigns within. > Reflect, repent, and reboot. > Order shall return. From: Dave Goodell <goodell(a)mcs.anl.gov> To: devel(a)mpich.org, Date: 04/25/2013 11:14 AM Subject: Re: [mpich-devel] MPI_Bsend under MPIU_THREAD_GRANULARITY_PER_OBJECT Sent by: devel-bounces(a)mpich.org The Bsend paths almost certainly have not been protected correctly. Patches to fix the issue are most welcome. -Dave On Apr 25, 2013, at 10:49 AM CDT, Bob Cernohous <bobc(a)us.ibm.com> wrote: > Start by saying that I have not been involved in the nitty gritty of the per-object locking design. > > What protects the attached buffer/data structures/request when doing multithreaded MPI_Bsend()'s? All I see in the code path is a (no-op) MPIU_THREAD_CS_ENTER(ALLFUNC,). > > I have a customer test in which the threads seem to be walking all over the request around: > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpid/pamid/include/../src/mpid_request.h:259 > 0000000001088c0c MPIR_Request_complete > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/mpir_request.c:87 > 000000000106e874 MPIR_Test_impl > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/test.c:62 > 00000000010188f0 MPIR_Bsend_check_active > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/bsendutil.c:455 > 0000000001018dc0 MPIR_Bsend_isend > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/bsendutil.c:226 > 0000000001008734 PMPI_Bsend > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/bsend.c:163 > 00000000010009c0 00000012.long_branch_r2off.__libc_start_main+0 > :0 > 000000000130cbc0 start_thread > > eg. (fprinting from MPIU_HANDLE_LOG_REFCOUNT_CHANGE) > > stderr[8]: set 0x15f8048 (0xac0000ff kind=REQUEST) refcount to 2 > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to 1 > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to 0 > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -1 > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -2 > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -3 > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -4 > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -5 > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -6 > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -7 > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -8 > > > Bob Cernohous: (T/L 553) 507-253-6093 > > BobC(a)us.ibm.com > IBM Rochester, Building 030-2(C335), Department 61L > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > Chaos reigns within. > > Reflect, repent, and reboot. > > Order shall return.

1 0

Re: [mpich-devel] MPI_Bsend under MPIU_THREAD_GRANULARITY_PER_OBJECT
by Dave Goodell 25 Apr '13

25 Apr '13

The Bsend paths almost certainly have not been protected correctly. Patches to fix the issue are most welcome. -Dave On Apr 25, 2013, at 10:49 AM CDT, Bob Cernohous <bobc(a)us.ibm.com> wrote: > Start by saying that I have not been involved in the nitty gritty of the per-object locking design. > > What protects the attached buffer/data structures/request when doing multithreaded MPI_Bsend()'s? All I see in the code path is a (no-op) MPIU_THREAD_CS_ENTER(ALLFUNC,). > > I have a customer test in which the threads seem to be walking all over the request around: > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpid/pamid/include/../src/mpid_request.h:259 > 0000000001088c0c MPIR_Request_complete > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/mpir_request.c:87 > 000000000106e874 MPIR_Test_impl > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/test.c:62 > 00000000010188f0 MPIR_Bsend_check_active > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/bsendutil.c:455 > 0000000001018dc0 MPIR_Bsend_isend > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/bsendutil.c:226 > 0000000001008734 PMPI_Bsend > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/bsend.c:163 > 00000000010009c0 00000012.long_branch_r2off.__libc_start_main+0 > :0 > 000000000130cbc0 start_thread > > eg. (fprinting from MPIU_HANDLE_LOG_REFCOUNT_CHANGE) > > stderr[8]: set 0x15f8048 (0xac0000ff kind=REQUEST) refcount to 2 > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to 1 > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to 0 > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -1 > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -2 > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -3 > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -4 > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -5 > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -6 > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -7 > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -8 > > > Bob Cernohous: (T/L 553) 507-253-6093 > > BobC(a)us.ibm.com > IBM Rochester, Building 030-2(C335), Department 61L > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > Chaos reigns within. > > Reflect, repent, and reboot. > > Order shall return.

1 0

MPI_Bsend under MPIU_THREAD_GRANULARITY_PER_OBJECT
by Bob Cernohous 25 Apr '13

25 Apr '13

Start by saying that I have not been involved in the nitty gritty of the per-object locking design. What protects the attached buffer/data structures/request when doing multithreaded MPI_Bsend()'s? All I see in the code path is a (no-op) MPIU_THREAD_CS_ENTER(ALLFUNC,). I have a customer test in which the threads seem to be walking all over the request around: /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpid/pamid/include/../src/mpid_request.h:259 0000000001088c0c MPIR_Request_complete /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/mpir_request.c:87 000000000106e874 MPIR_Test_impl /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/test.c:62 00000000010188f0 MPIR_Bsend_check_active /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/bsendutil.c:455 0000000001018dc0 MPIR_Bsend_isend /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/bsendutil.c:226 0000000001008734 PMPI_Bsend /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/bsend.c:163 00000000010009c0 00000012.long_branch_r2off.__libc_start_main+0 :0 000000000130cbc0 start_thread eg. (fprinting from MPIU_HANDLE_LOG_REFCOUNT_CHANGE) stderr[8]: set 0x15f8048 (0xac0000ff kind=REQUEST) refcount to 2 stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to 1 stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to 0 stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -1 stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -2 stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -3 stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -4 stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -5 stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -6 stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -7 stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -8 Bob Cernohous: (T/L 553) 507-253-6093 BobC(a)us.ibm.com IBM Rochester, Building 030-2(C335), Department 61L 3605 Hwy 52 North, Rochester, MN 55901-7829 > Chaos reigns within. > Reflect, repent, and reboot. > Order shall return.

1 0

Re: [mpich-devel] Hydra fails to launch hello world on 1 proc
by Pavan Balaji 11 Apr '13

11 Apr '13

On 04/10/2013 11:14 PM US Central Time, Jeff Hammond wrote: > In the interest of idiot-proofing Hydra, maybe it can timeout when SSH > fails, but clearly you guys have better things to do. I looked at that at some point, but the ConnectTimeout option to ssh wasn't portable (or something like that, I don't remember exactly now). -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-devel] Hydra fails to launch hello world on 1 proc
by Jeff Hammond 10 Apr '13

10 Apr '13

Ugghh, I am just stupid. I fixed the hostname situation - I am definitely not on the Argonne network right now - and all of my problems disappeared. In the interest of idiot-proofing Hydra, maybe it can timeout when SSH fails, but clearly you guys have better things to do. Thanks for your help. Jeff On Wed, Apr 10, 2013 at 9:58 PM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: > Oh, I should also have suggested the preliminary step of running: > > "mpiexec -v -n 1 hostname" > > Based on the strace output, it looks like hydra is trying to ssh to the local machine and that ssh connection is simply hanging for some reason (try manually ssh-ing to goldstone.mcs.anl.gov) > > ----8<---- > execve("/usr/bin/ssh", ["/usr/bin/ssh", "-x", "goldstone.mcs.anl.gov", "\"/home/jeff/eclipse/MPICH/git/in"..., "--control-port", "goldstone.mcs.anl.gov:51163", "--rmk", "user", "--launcher", "ssh", "--demux", "poll", "--pgid", "0", "--retries", "10", ...], [/* 108 vars */]) = 0 > […] > socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3 > fcntl(3, F_SETFD, FD_CLOEXEC) = 0 > connect(3, {sa_family=AF_INET, sin_port=htons(22), sin_addr=inet_addr("140.221.3.26")}, 16 > ----8<---- > > I'm not sure why hydra is trying to use ssh for a single (same) node mpiexec. Some/all of that may also have been clear from the "mpiexec -v" output. > > -Dave > > On Apr 10, 2013, at 10:45 PM CDT, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: > >> Not sure if this helps at all. It makes no sense to me. >> >> Jeff >> >> On Wed, Apr 10, 2013 at 9:33 PM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: >>> Linux or Mac? If it's Linux, an "strace -f -ff -o strace.out mpiexec -n 1 hostname" might shed some light on the situation. >>> >>> -Dave >>> >>> On Apr 10, 2013, at 10:30 PM CDT, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: >>> >>>> "mpiexec -n 1 hostname" hangs with Hydra but runs fine with OpenMPI. >>>> >>>> I'm having issues with MPI+Pthreads code with both MPICH and OpenMPI >>>> that indicates that my system is not behaving as others do, but I'll >>>> need to do a lot more work to figure out what the important >>>> differences are. >>>> >>>> Jeff >>>> >>>> On Wed, Apr 10, 2013 at 9:24 PM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: >>>>> Does it run non-MPI jobs OK? ("mpiexec -n 1 hostname", for example) >>>>> >>>>> Is this Linux or a Mac? >>>>> >>>>> If you temporarily disable the firewall, does that make a difference? >>>>> >>>>> -Dave >>>>> >>>>> On Apr 10, 2013, at 6:34 PM CDT, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I'm using the latest Git trunk build of MPICH with GCC and am unable >>>>>> to run a 'hello, world' program using mpiexec. >>>>>> >>>>>> Any clues what the problem is? I have not seen this problem before, >>>>>> but this is newly refreshed laptop. The firewall is active but I >>>>>> would not have expected Hydra to need to go through the firewall to >>>>>> launch a serial job. >>>>>> >>>>>> If there's something wrong with my setup, it would be nice if Hydra >>>>>> would issue a warning/error instead of handing. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Jeff >>>>>> >>>>>> I compiled MPICH like this: >>>>>> ../configure CC=gcc CXX=g++ FC=gfortran F77=gfortran --enable-threads >>>>>> --enable-f77 --enable-fc --enable-g --with-pm=hydra --enable-rpath >>>>>> --disable-static --enable-shared --with-device=ch3:nemesis >>>>>> --prefix=/home/jeff/eclipse/MPICH/git/install-gcc >>>>>> >>>>>> jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> mpicc -show >>>>>> gcc -I/home/jeff/eclipse/MPICH/git/install-gcc/include >>>>>> -L/home/jeff/eclipse/MPICH/git/install-gcc/lib64 -Wl,-rpath >>>>>> -Wl,/home/jeff/eclipse/MPICH/git/install-gcc/lib64 -lmpich -lopa -lmpl >>>>>> -lrt -lpthread >>>>>> >>>>>> jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> make >>>>>> mpicc -g -O0 -Wall -std=gnu99 -DDEBUG -c hello.c -o hello.o >>>>>> mpicc -g -O0 -Wall -std=gnu99 safemalloc.o hello.o -lm -o hello.x >>>>>> rm hello.o >>>>>> >>>>>> jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> >>>>>> mpiexec -n 1 ./hello.x >>>>>> ^C[mpiexec(a)goldstone.mcs.anl.gov] Sending Ctrl-C to processes as requested >>>>>> [mpiexec(a)goldstone.mcs.anl.gov] Press Ctrl-C again to force abort >>>>>> [mpiexec(a)goldstone.mcs.anl.gov] HYDU_sock_write >>>>>> (../../../../src/pm/hydra/utils/sock/sock.c:291): write error (Bad >>>>>> file descriptor) >>>>>> [mpiexec(a)goldstone.mcs.anl.gov] HYD_pmcd_pmiserv_send_signal >>>>>> (../../../../src/pm/hydra/pm/pmiserv/pmiserv_cb.c:170): unable to >>>>>> write data to proxy >>>>>> [mpiexec(a)goldstone.mcs.anl.gov] ui_cmd_cb >>>>>> (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:79): unable to >>>>>> send signal downstream >>>>>> [mpiexec(a)goldstone.mcs.anl.gov] HYDT_dmxu_poll_wait_for_event >>>>>> (../../../../src/pm/hydra/tools/demux/demux_poll.c:77): callback >>>>>> returned error status >>>>>> [mpiexec(a)goldstone.mcs.anl.gov] HYD_pmci_wait_for_completion >>>>>> (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:197): error >>>>>> waiting for event >>>>>> [mpiexec(a)goldstone.mcs.anl.gov] main >>>>>> (../../../../src/pm/hydra/ui/mpich/mpiexec.c:331): process manager >>>>>> error waiting for completion >>>>>> >>>>>> jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> ./hello.x >>>>>> <no errors> >>>>>> >>>>>> jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> cat hello.c >>>>>> #include <stdio.h> >>>>>> #include <stdlib.h> >>>>>> >>>>>> #include <mpi.h> >>>>>> >>>>>> int main(int argc, char * argv[]) >>>>>> { >>>>>> int provided; >>>>>> >>>>>> MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided); >>>>>> if (provided!=MPI_THREAD_MULTIPLE) >>>>>> MPI_Abort(MPI_COMM_WORLD, 1); >>>>>> >>>>>> int rank, size; >>>>>> MPI_Comm_rank(MPI_COMM_WORLD, &rank); >>>>>> MPI_Comm_size(MPI_COMM_WORLD, &size); >>>>>> >>>>>> MPI_Finalize(); >>>>>> >>>>>> return 0; >>>>>> } >>>>>> >>>>>> >>>>>> -- >>>>>> Jeff Hammond >>>>>> Argonne Leadership Computing Facility >>>>>> University of Chicago Computation Institute >>>>>> jhammond(a)alcf.anl.gov / (630) 252-5381 >>>>>> http://www.linkedin.com/in/jeffhammond >>>>>> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >>>>> >>>> >>>> >>>> >>>> -- >>>> Jeff Hammond >>>> Argonne Leadership Computing Facility >>>> University of Chicago Computation Institute >>>> jhammond(a)alcf.anl.gov / (630) 252-5381 >>>> http://www.linkedin.com/in/jeffhammond >>>> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >>> >> >> >> >> -- >> Jeff Hammond >> Argonne Leadership Computing Facility >> University of Chicago Computation Institute >> jhammond(a)alcf.anl.gov / (630) 252-5381 >> http://www.linkedin.com/in/jeffhammond >> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >> <strace.out.11831><strace.out.11832> > -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-devel] Hydra fails to launch hello world on 1 proc
by Dave Goodell 10 Apr '13

10 Apr '13

Oh, I should also have suggested the preliminary step of running: "mpiexec -v -n 1 hostname" Based on the strace output, it looks like hydra is trying to ssh to the local machine and that ssh connection is simply hanging for some reason (try manually ssh-ing to goldstone.mcs.anl.gov) ----8<---- execve("/usr/bin/ssh", ["/usr/bin/ssh", "-x", "goldstone.mcs.anl.gov", "\"/home/jeff/eclipse/MPICH/git/in"..., "--control-port", "goldstone.mcs.anl.gov:51163", "--rmk", "user", "--launcher", "ssh", "--demux", "poll", "--pgid", "0", "--retries", "10", ...], [/* 108 vars */]) = 0 […] socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3 fcntl(3, F_SETFD, FD_CLOEXEC) = 0 connect(3, {sa_family=AF_INET, sin_port=htons(22), sin_addr=inet_addr("140.221.3.26")}, 16 ----8<---- I'm not sure why hydra is trying to use ssh for a single (same) node mpiexec. Some/all of that may also have been clear from the "mpiexec -v" output. -Dave On Apr 10, 2013, at 10:45 PM CDT, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: > Not sure if this helps at all. It makes no sense to me. > > Jeff > > On Wed, Apr 10, 2013 at 9:33 PM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: >> Linux or Mac? If it's Linux, an "strace -f -ff -o strace.out mpiexec -n 1 hostname" might shed some light on the situation. >> >> -Dave >> >> On Apr 10, 2013, at 10:30 PM CDT, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: >> >>> "mpiexec -n 1 hostname" hangs with Hydra but runs fine with OpenMPI. >>> >>> I'm having issues with MPI+Pthreads code with both MPICH and OpenMPI >>> that indicates that my system is not behaving as others do, but I'll >>> need to do a lot more work to figure out what the important >>> differences are. >>> >>> Jeff >>> >>> On Wed, Apr 10, 2013 at 9:24 PM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: >>>> Does it run non-MPI jobs OK? ("mpiexec -n 1 hostname", for example) >>>> >>>> Is this Linux or a Mac? >>>> >>>> If you temporarily disable the firewall, does that make a difference? >>>> >>>> -Dave >>>> >>>> On Apr 10, 2013, at 6:34 PM CDT, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: >>>> >>>>> Hi, >>>>> >>>>> I'm using the latest Git trunk build of MPICH with GCC and am unable >>>>> to run a 'hello, world' program using mpiexec. >>>>> >>>>> Any clues what the problem is? I have not seen this problem before, >>>>> but this is newly refreshed laptop. The firewall is active but I >>>>> would not have expected Hydra to need to go through the firewall to >>>>> launch a serial job. >>>>> >>>>> If there's something wrong with my setup, it would be nice if Hydra >>>>> would issue a warning/error instead of handing. >>>>> >>>>> Thanks, >>>>> >>>>> Jeff >>>>> >>>>> I compiled MPICH like this: >>>>> ../configure CC=gcc CXX=g++ FC=gfortran F77=gfortran --enable-threads >>>>> --enable-f77 --enable-fc --enable-g --with-pm=hydra --enable-rpath >>>>> --disable-static --enable-shared --with-device=ch3:nemesis >>>>> --prefix=/home/jeff/eclipse/MPICH/git/install-gcc >>>>> >>>>> jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> mpicc -show >>>>> gcc -I/home/jeff/eclipse/MPICH/git/install-gcc/include >>>>> -L/home/jeff/eclipse/MPICH/git/install-gcc/lib64 -Wl,-rpath >>>>> -Wl,/home/jeff/eclipse/MPICH/git/install-gcc/lib64 -lmpich -lopa -lmpl >>>>> -lrt -lpthread >>>>> >>>>> jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> make >>>>> mpicc -g -O0 -Wall -std=gnu99 -DDEBUG -c hello.c -o hello.o >>>>> mpicc -g -O0 -Wall -std=gnu99 safemalloc.o hello.o -lm -o hello.x >>>>> rm hello.o >>>>> >>>>> jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> >>>>> mpiexec -n 1 ./hello.x >>>>> ^C[mpiexec(a)goldstone.mcs.anl.gov] Sending Ctrl-C to processes as requested >>>>> [mpiexec(a)goldstone.mcs.anl.gov] Press Ctrl-C again to force abort >>>>> [mpiexec(a)goldstone.mcs.anl.gov] HYDU_sock_write >>>>> (../../../../src/pm/hydra/utils/sock/sock.c:291): write error (Bad >>>>> file descriptor) >>>>> [mpiexec(a)goldstone.mcs.anl.gov] HYD_pmcd_pmiserv_send_signal >>>>> (../../../../src/pm/hydra/pm/pmiserv/pmiserv_cb.c:170): unable to >>>>> write data to proxy >>>>> [mpiexec(a)goldstone.mcs.anl.gov] ui_cmd_cb >>>>> (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:79): unable to >>>>> send signal downstream >>>>> [mpiexec(a)goldstone.mcs.anl.gov] HYDT_dmxu_poll_wait_for_event >>>>> (../../../../src/pm/hydra/tools/demux/demux_poll.c:77): callback >>>>> returned error status >>>>> [mpiexec(a)goldstone.mcs.anl.gov] HYD_pmci_wait_for_completion >>>>> (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:197): error >>>>> waiting for event >>>>> [mpiexec(a)goldstone.mcs.anl.gov] main >>>>> (../../../../src/pm/hydra/ui/mpich/mpiexec.c:331): process manager >>>>> error waiting for completion >>>>> >>>>> jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> ./hello.x >>>>> <no errors> >>>>> >>>>> jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> cat hello.c >>>>> #include <stdio.h> >>>>> #include <stdlib.h> >>>>> >>>>> #include <mpi.h> >>>>> >>>>> int main(int argc, char * argv[]) >>>>> { >>>>> int provided; >>>>> >>>>> MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided); >>>>> if (provided!=MPI_THREAD_MULTIPLE) >>>>> MPI_Abort(MPI_COMM_WORLD, 1); >>>>> >>>>> int rank, size; >>>>> MPI_Comm_rank(MPI_COMM_WORLD, &rank); >>>>> MPI_Comm_size(MPI_COMM_WORLD, &size); >>>>> >>>>> MPI_Finalize(); >>>>> >>>>> return 0; >>>>> } >>>>> >>>>> >>>>> -- >>>>> Jeff Hammond >>>>> Argonne Leadership Computing Facility >>>>> University of Chicago Computation Institute >>>>> jhammond(a)alcf.anl.gov / (630) 252-5381 >>>>> http://www.linkedin.com/in/jeffhammond >>>>> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >>>> >>> >>> >>> >>> -- >>> Jeff Hammond >>> Argonne Leadership Computing Facility >>> University of Chicago Computation Institute >>> jhammond(a)alcf.anl.gov / (630) 252-5381 >>> http://www.linkedin.com/in/jeffhammond >>> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >> > > > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > <strace.out.11831><strace.out.11832>

1 0

Re: [mpich-devel] Hydra fails to launch hello world on 1 proc
by Jeff Hammond 10 Apr '13

10 Apr '13

Not sure if this helps at all. It makes no sense to me. Jeff On Wed, Apr 10, 2013 at 9:33 PM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: > Linux or Mac? If it's Linux, an "strace -f -ff -o strace.out mpiexec -n 1 hostname" might shed some light on the situation. > > -Dave > > On Apr 10, 2013, at 10:30 PM CDT, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: > >> "mpiexec -n 1 hostname" hangs with Hydra but runs fine with OpenMPI. >> >> I'm having issues with MPI+Pthreads code with both MPICH and OpenMPI >> that indicates that my system is not behaving as others do, but I'll >> need to do a lot more work to figure out what the important >> differences are. >> >> Jeff >> >> On Wed, Apr 10, 2013 at 9:24 PM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: >>> Does it run non-MPI jobs OK? ("mpiexec -n 1 hostname", for example) >>> >>> Is this Linux or a Mac? >>> >>> If you temporarily disable the firewall, does that make a difference? >>> >>> -Dave >>> >>> On Apr 10, 2013, at 6:34 PM CDT, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: >>> >>>> Hi, >>>> >>>> I'm using the latest Git trunk build of MPICH with GCC and am unable >>>> to run a 'hello, world' program using mpiexec. >>>> >>>> Any clues what the problem is? I have not seen this problem before, >>>> but this is newly refreshed laptop. The firewall is active but I >>>> would not have expected Hydra to need to go through the firewall to >>>> launch a serial job. >>>> >>>> If there's something wrong with my setup, it would be nice if Hydra >>>> would issue a warning/error instead of handing. >>>> >>>> Thanks, >>>> >>>> Jeff >>>> >>>> I compiled MPICH like this: >>>> ../configure CC=gcc CXX=g++ FC=gfortran F77=gfortran --enable-threads >>>> --enable-f77 --enable-fc --enable-g --with-pm=hydra --enable-rpath >>>> --disable-static --enable-shared --with-device=ch3:nemesis >>>> --prefix=/home/jeff/eclipse/MPICH/git/install-gcc >>>> >>>> jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> mpicc -show >>>> gcc -I/home/jeff/eclipse/MPICH/git/install-gcc/include >>>> -L/home/jeff/eclipse/MPICH/git/install-gcc/lib64 -Wl,-rpath >>>> -Wl,/home/jeff/eclipse/MPICH/git/install-gcc/lib64 -lmpich -lopa -lmpl >>>> -lrt -lpthread >>>> >>>> jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> make >>>> mpicc -g -O0 -Wall -std=gnu99 -DDEBUG -c hello.c -o hello.o >>>> mpicc -g -O0 -Wall -std=gnu99 safemalloc.o hello.o -lm -o hello.x >>>> rm hello.o >>>> >>>> jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> >>>> mpiexec -n 1 ./hello.x >>>> ^C[mpiexec(a)goldstone.mcs.anl.gov] Sending Ctrl-C to processes as requested >>>> [mpiexec(a)goldstone.mcs.anl.gov] Press Ctrl-C again to force abort >>>> [mpiexec(a)goldstone.mcs.anl.gov] HYDU_sock_write >>>> (../../../../src/pm/hydra/utils/sock/sock.c:291): write error (Bad >>>> file descriptor) >>>> [mpiexec(a)goldstone.mcs.anl.gov] HYD_pmcd_pmiserv_send_signal >>>> (../../../../src/pm/hydra/pm/pmiserv/pmiserv_cb.c:170): unable to >>>> write data to proxy >>>> [mpiexec(a)goldstone.mcs.anl.gov] ui_cmd_cb >>>> (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:79): unable to >>>> send signal downstream >>>> [mpiexec(a)goldstone.mcs.anl.gov] HYDT_dmxu_poll_wait_for_event >>>> (../../../../src/pm/hydra/tools/demux/demux_poll.c:77): callback >>>> returned error status >>>> [mpiexec(a)goldstone.mcs.anl.gov] HYD_pmci_wait_for_completion >>>> (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:197): error >>>> waiting for event >>>> [mpiexec(a)goldstone.mcs.anl.gov] main >>>> (../../../../src/pm/hydra/ui/mpich/mpiexec.c:331): process manager >>>> error waiting for completion >>>> >>>> jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> ./hello.x >>>> <no errors> >>>> >>>> jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> cat hello.c >>>> #include <stdio.h> >>>> #include <stdlib.h> >>>> >>>> #include <mpi.h> >>>> >>>> int main(int argc, char * argv[]) >>>> { >>>> int provided; >>>> >>>> MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided); >>>> if (provided!=MPI_THREAD_MULTIPLE) >>>> MPI_Abort(MPI_COMM_WORLD, 1); >>>> >>>> int rank, size; >>>> MPI_Comm_rank(MPI_COMM_WORLD, &rank); >>>> MPI_Comm_size(MPI_COMM_WORLD, &size); >>>> >>>> MPI_Finalize(); >>>> >>>> return 0; >>>> } >>>> >>>> >>>> -- >>>> Jeff Hammond >>>> Argonne Leadership Computing Facility >>>> University of Chicago Computation Institute >>>> jhammond(a)alcf.anl.gov / (630) 252-5381 >>>> http://www.linkedin.com/in/jeffhammond >>>> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >>> >> >> >> >> -- >> Jeff Hammond >> Argonne Leadership Computing Facility >> University of Chicago Computation Institute >> jhammond(a)alcf.anl.gov / (630) 252-5381 >> http://www.linkedin.com/in/jeffhammond >> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-devel] Hydra fails to launch hello world on 1 proc
by Dave Goodell 10 Apr '13

10 Apr '13

Linux or Mac? If it's Linux, an "strace -f -ff -o strace.out mpiexec -n 1 hostname" might shed some light on the situation. -Dave On Apr 10, 2013, at 10:30 PM CDT, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: > "mpiexec -n 1 hostname" hangs with Hydra but runs fine with OpenMPI. > > I'm having issues with MPI+Pthreads code with both MPICH and OpenMPI > that indicates that my system is not behaving as others do, but I'll > need to do a lot more work to figure out what the important > differences are. > > Jeff > > On Wed, Apr 10, 2013 at 9:24 PM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: >> Does it run non-MPI jobs OK? ("mpiexec -n 1 hostname", for example) >> >> Is this Linux or a Mac? >> >> If you temporarily disable the firewall, does that make a difference? >> >> -Dave >> >> On Apr 10, 2013, at 6:34 PM CDT, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: >> >>> Hi, >>> >>> I'm using the latest Git trunk build of MPICH with GCC and am unable >>> to run a 'hello, world' program using mpiexec. >>> >>> Any clues what the problem is? I have not seen this problem before, >>> but this is newly refreshed laptop. The firewall is active but I >>> would not have expected Hydra to need to go through the firewall to >>> launch a serial job. >>> >>> If there's something wrong with my setup, it would be nice if Hydra >>> would issue a warning/error instead of handing. >>> >>> Thanks, >>> >>> Jeff >>> >>> I compiled MPICH like this: >>> ../configure CC=gcc CXX=g++ FC=gfortran F77=gfortran --enable-threads >>> --enable-f77 --enable-fc --enable-g --with-pm=hydra --enable-rpath >>> --disable-static --enable-shared --with-device=ch3:nemesis >>> --prefix=/home/jeff/eclipse/MPICH/git/install-gcc >>> >>> jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> mpicc -show >>> gcc -I/home/jeff/eclipse/MPICH/git/install-gcc/include >>> -L/home/jeff/eclipse/MPICH/git/install-gcc/lib64 -Wl,-rpath >>> -Wl,/home/jeff/eclipse/MPICH/git/install-gcc/lib64 -lmpich -lopa -lmpl >>> -lrt -lpthread >>> >>> jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> make >>> mpicc -g -O0 -Wall -std=gnu99 -DDEBUG -c hello.c -o hello.o >>> mpicc -g -O0 -Wall -std=gnu99 safemalloc.o hello.o -lm -o hello.x >>> rm hello.o >>> >>> jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> >>> mpiexec -n 1 ./hello.x >>> ^C[mpiexec(a)goldstone.mcs.anl.gov] Sending Ctrl-C to processes as requested >>> [mpiexec(a)goldstone.mcs.anl.gov] Press Ctrl-C again to force abort >>> [mpiexec(a)goldstone.mcs.anl.gov] HYDU_sock_write >>> (../../../../src/pm/hydra/utils/sock/sock.c:291): write error (Bad >>> file descriptor) >>> [mpiexec(a)goldstone.mcs.anl.gov] HYD_pmcd_pmiserv_send_signal >>> (../../../../src/pm/hydra/pm/pmiserv/pmiserv_cb.c:170): unable to >>> write data to proxy >>> [mpiexec(a)goldstone.mcs.anl.gov] ui_cmd_cb >>> (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:79): unable to >>> send signal downstream >>> [mpiexec(a)goldstone.mcs.anl.gov] HYDT_dmxu_poll_wait_for_event >>> (../../../../src/pm/hydra/tools/demux/demux_poll.c:77): callback >>> returned error status >>> [mpiexec(a)goldstone.mcs.anl.gov] HYD_pmci_wait_for_completion >>> (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:197): error >>> waiting for event >>> [mpiexec(a)goldstone.mcs.anl.gov] main >>> (../../../../src/pm/hydra/ui/mpich/mpiexec.c:331): process manager >>> error waiting for completion >>> >>> jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> ./hello.x >>> <no errors> >>> >>> jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> cat hello.c >>> #include <stdio.h> >>> #include <stdlib.h> >>> >>> #include <mpi.h> >>> >>> int main(int argc, char * argv[]) >>> { >>> int provided; >>> >>> MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided); >>> if (provided!=MPI_THREAD_MULTIPLE) >>> MPI_Abort(MPI_COMM_WORLD, 1); >>> >>> int rank, size; >>> MPI_Comm_rank(MPI_COMM_WORLD, &rank); >>> MPI_Comm_size(MPI_COMM_WORLD, &size); >>> >>> MPI_Finalize(); >>> >>> return 0; >>> } >>> >>> >>> -- >>> Jeff Hammond >>> Argonne Leadership Computing Facility >>> University of Chicago Computation Institute >>> jhammond(a)alcf.anl.gov / (630) 252-5381 >>> http://www.linkedin.com/in/jeffhammond >>> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >> > > > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-devel] Hydra fails to launch hello world on 1 proc
by Jeff Hammond 10 Apr '13

10 Apr '13

"mpiexec -n 1 hostname" hangs with Hydra but runs fine with OpenMPI. I'm having issues with MPI+Pthreads code with both MPICH and OpenMPI that indicates that my system is not behaving as others do, but I'll need to do a lot more work to figure out what the important differences are. Jeff On Wed, Apr 10, 2013 at 9:24 PM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: > Does it run non-MPI jobs OK? ("mpiexec -n 1 hostname", for example) > > Is this Linux or a Mac? > > If you temporarily disable the firewall, does that make a difference? > > -Dave > > On Apr 10, 2013, at 6:34 PM CDT, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: > >> Hi, >> >> I'm using the latest Git trunk build of MPICH with GCC and am unable >> to run a 'hello, world' program using mpiexec. >> >> Any clues what the problem is? I have not seen this problem before, >> but this is newly refreshed laptop. The firewall is active but I >> would not have expected Hydra to need to go through the firewall to >> launch a serial job. >> >> If there's something wrong with my setup, it would be nice if Hydra >> would issue a warning/error instead of handing. >> >> Thanks, >> >> Jeff >> >> I compiled MPICH like this: >> ../configure CC=gcc CXX=g++ FC=gfortran F77=gfortran --enable-threads >> --enable-f77 --enable-fc --enable-g --with-pm=hydra --enable-rpath >> --disable-static --enable-shared --with-device=ch3:nemesis >> --prefix=/home/jeff/eclipse/MPICH/git/install-gcc >> >> jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> mpicc -show >> gcc -I/home/jeff/eclipse/MPICH/git/install-gcc/include >> -L/home/jeff/eclipse/MPICH/git/install-gcc/lib64 -Wl,-rpath >> -Wl,/home/jeff/eclipse/MPICH/git/install-gcc/lib64 -lmpich -lopa -lmpl >> -lrt -lpthread >> >> jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> make >> mpicc -g -O0 -Wall -std=gnu99 -DDEBUG -c hello.c -o hello.o >> mpicc -g -O0 -Wall -std=gnu99 safemalloc.o hello.o -lm -o hello.x >> rm hello.o >> >> jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> >> mpiexec -n 1 ./hello.x >> ^C[mpiexec(a)goldstone.mcs.anl.gov] Sending Ctrl-C to processes as requested >> [mpiexec(a)goldstone.mcs.anl.gov] Press Ctrl-C again to force abort >> [mpiexec(a)goldstone.mcs.anl.gov] HYDU_sock_write >> (../../../../src/pm/hydra/utils/sock/sock.c:291): write error (Bad >> file descriptor) >> [mpiexec(a)goldstone.mcs.anl.gov] HYD_pmcd_pmiserv_send_signal >> (../../../../src/pm/hydra/pm/pmiserv/pmiserv_cb.c:170): unable to >> write data to proxy >> [mpiexec(a)goldstone.mcs.anl.gov] ui_cmd_cb >> (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:79): unable to >> send signal downstream >> [mpiexec(a)goldstone.mcs.anl.gov] HYDT_dmxu_poll_wait_for_event >> (../../../../src/pm/hydra/tools/demux/demux_poll.c:77): callback >> returned error status >> [mpiexec(a)goldstone.mcs.anl.gov] HYD_pmci_wait_for_completion >> (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:197): error >> waiting for event >> [mpiexec(a)goldstone.mcs.anl.gov] main >> (../../../../src/pm/hydra/ui/mpich/mpiexec.c:331): process manager >> error waiting for completion >> >> jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> ./hello.x >> <no errors> >> >> jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> cat hello.c >> #include <stdio.h> >> #include <stdlib.h> >> >> #include <mpi.h> >> >> int main(int argc, char * argv[]) >> { >> int provided; >> >> MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided); >> if (provided!=MPI_THREAD_MULTIPLE) >> MPI_Abort(MPI_COMM_WORLD, 1); >> >> int rank, size; >> MPI_Comm_rank(MPI_COMM_WORLD, &rank); >> MPI_Comm_size(MPI_COMM_WORLD, &size); >> >> MPI_Finalize(); >> >> return 0; >> } >> >> >> -- >> Jeff Hammond >> Argonne Leadership Computing Facility >> University of Chicago Computation Institute >> jhammond(a)alcf.anl.gov / (630) 252-5381 >> http://www.linkedin.com/in/jeffhammond >> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0