April 2013 - devel - mailman3.cels.anl.gov

Re: [mpich-devel] Hydra fails to launch hello world on 1 proc
by Dave Goodell 10 Apr '13

10 Apr '13

Does it run non-MPI jobs OK? ("mpiexec -n 1 hostname", for example) Is this Linux or a Mac? If you temporarily disable the firewall, does that make a difference? -Dave On Apr 10, 2013, at 6:34 PM CDT, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: > Hi, > > I'm using the latest Git trunk build of MPICH with GCC and am unable > to run a 'hello, world' program using mpiexec. > > Any clues what the problem is? I have not seen this problem before, > but this is newly refreshed laptop. The firewall is active but I > would not have expected Hydra to need to go through the firewall to > launch a serial job. > > If there's something wrong with my setup, it would be nice if Hydra > would issue a warning/error instead of handing. > > Thanks, > > Jeff > > I compiled MPICH like this: > ../configure CC=gcc CXX=g++ FC=gfortran F77=gfortran --enable-threads > --enable-f77 --enable-fc --enable-g --with-pm=hydra --enable-rpath > --disable-static --enable-shared --with-device=ch3:nemesis > --prefix=/home/jeff/eclipse/MPICH/git/install-gcc > > jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> mpicc -show > gcc -I/home/jeff/eclipse/MPICH/git/install-gcc/include > -L/home/jeff/eclipse/MPICH/git/install-gcc/lib64 -Wl,-rpath > -Wl,/home/jeff/eclipse/MPICH/git/install-gcc/lib64 -lmpich -lopa -lmpl > -lrt -lpthread > > jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> make > mpicc -g -O0 -Wall -std=gnu99 -DDEBUG -c hello.c -o hello.o > mpicc -g -O0 -Wall -std=gnu99 safemalloc.o hello.o -lm -o hello.x > rm hello.o > > jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> > mpiexec -n 1 ./hello.x > ^C[mpiexec(a)goldstone.mcs.anl.gov] Sending Ctrl-C to processes as requested > [mpiexec(a)goldstone.mcs.anl.gov] Press Ctrl-C again to force abort > [mpiexec(a)goldstone.mcs.anl.gov] HYDU_sock_write > (../../../../src/pm/hydra/utils/sock/sock.c:291): write error (Bad > file descriptor) > [mpiexec(a)goldstone.mcs.anl.gov] HYD_pmcd_pmiserv_send_signal > (../../../../src/pm/hydra/pm/pmiserv/pmiserv_cb.c:170): unable to > write data to proxy > [mpiexec(a)goldstone.mcs.anl.gov] ui_cmd_cb > (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:79): unable to > send signal downstream > [mpiexec(a)goldstone.mcs.anl.gov] HYDT_dmxu_poll_wait_for_event > (../../../../src/pm/hydra/tools/demux/demux_poll.c:77): callback > returned error status > [mpiexec(a)goldstone.mcs.anl.gov] HYD_pmci_wait_for_completion > (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:197): error > waiting for event > [mpiexec(a)goldstone.mcs.anl.gov] main > (../../../../src/pm/hydra/ui/mpich/mpiexec.c:331): process manager > error waiting for completion > > jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> ./hello.x > <no errors> > > jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> cat hello.c > #include <stdio.h> > #include <stdlib.h> > > #include <mpi.h> > > int main(int argc, char * argv[]) > { > int provided; > > MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided); > if (provided!=MPI_THREAD_MULTIPLE) > MPI_Abort(MPI_COMM_WORLD, 1); > > int rank, size; > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > MPI_Comm_size(MPI_COMM_WORLD, &size); > > MPI_Finalize(); > > return 0; > } > > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Hydra fails to launch hello world on 1 proc
by Jeff Hammond 10 Apr '13

10 Apr '13

Hi, I'm using the latest Git trunk build of MPICH with GCC and am unable to run a 'hello, world' program using mpiexec. Any clues what the problem is? I have not seen this problem before, but this is newly refreshed laptop. The firewall is active but I would not have expected Hydra to need to go through the firewall to launch a serial job. If there's something wrong with my setup, it would be nice if Hydra would issue a warning/error instead of handing. Thanks, Jeff I compiled MPICH like this: ../configure CC=gcc CXX=g++ FC=gfortran F77=gfortran --enable-threads --enable-f77 --enable-fc --enable-g --with-pm=hydra --enable-rpath --disable-static --enable-shared --with-device=ch3:nemesis --prefix=/home/jeff/eclipse/MPICH/git/install-gcc jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> mpicc -show gcc -I/home/jeff/eclipse/MPICH/git/install-gcc/include -L/home/jeff/eclipse/MPICH/git/install-gcc/lib64 -Wl,-rpath -Wl,/home/jeff/eclipse/MPICH/git/install-gcc/lib64 -lmpich -lopa -lmpl -lrt -lpthread jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> make mpicc -g -O0 -Wall -std=gnu99 -DDEBUG -c hello.c -o hello.o mpicc -g -O0 -Wall -std=gnu99 safemalloc.o hello.o -lm -o hello.x rm hello.o jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> mpiexec -n 1 ./hello.x ^C[mpiexec(a)goldstone.mcs.anl.gov] Sending Ctrl-C to processes as requested [mpiexec(a)goldstone.mcs.anl.gov] Press Ctrl-C again to force abort [mpiexec(a)goldstone.mcs.anl.gov] HYDU_sock_write (../../../../src/pm/hydra/utils/sock/sock.c:291): write error (Bad file descriptor) [mpiexec(a)goldstone.mcs.anl.gov] HYD_pmcd_pmiserv_send_signal (../../../../src/pm/hydra/pm/pmiserv/pmiserv_cb.c:170): unable to write data to proxy [mpiexec(a)goldstone.mcs.anl.gov] ui_cmd_cb (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:79): unable to send signal downstream [mpiexec(a)goldstone.mcs.anl.gov] HYDT_dmxu_poll_wait_for_event (../../../../src/pm/hydra/tools/demux/demux_poll.c:77): callback returned error status [mpiexec(a)goldstone.mcs.anl.gov] HYD_pmci_wait_for_completion (../../../../src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:197): error waiting for event [mpiexec(a)goldstone.mcs.anl.gov] main (../../../../src/pm/hydra/ui/mpich/mpiexec.c:331): process manager error waiting for completion jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> ./hello.x <no errors> jeff@goldstone:~/eclipse/OSPRI/mcs.svn/trunk/tests/devices/mpi-pt> cat hello.c #include <stdio.h> #include <stdlib.h> #include <mpi.h> int main(int argc, char * argv[]) { int provided; MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided); if (provided!=MPI_THREAD_MULTIPLE) MPI_Abort(MPI_COMM_WORLD, 1); int rank, size; MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); MPI_Finalize(); return 0; } -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-devel] Regarding direct receive
by Jim Dinan 09 Apr '13

09 Apr '13

Hi Denis, I have a fair bit of experience with CH3, but not much in the netmods. This sounds like something that could be handled by the corresponding eager/rendezvous code via MPIDI_CH3U_Receive_data_found. Are you providing new eager/rendezvous implementations in your netmod? I don't have enough personal context to evaluate the overall structure, but your else case below looks reasonable to me. Best, ~Jim. On 4/9/13 6:54 AM, Kirjanov, Denis wrote: > Hello, > I'm writing a netmod and I have a trouble with receiving a messages directly to the user buffer: > > The relevant code looks like the following: > > Assuming we have a request with the one iov entry > /*************** > if (!VC_CH(vc)->recv_active) { > /* handle a new message using MPID_nem_handle_pkt() */ > } else { > MPIU_Memcpy(iov->MPID_IOV_BUF, &recv_buff->buff[0], bytes_recvd); > iov->MPID_IOV_BUF = (void *)((char *)iov->MPID_IOV_BUF + bytes_recvd); > iov->MPID_IOV_LEN -= bytes_recvd; > } > > if (iov->MPID_IOV_LEN) { > /* receive directly to the iov->MPID_IOV_BUF next time*/ > goto fn_exit; > } > > /* handle a recv request */ > } > ********************/ > The problem that I'm running an IMB program compiled with CHECK option and it claims about corrupted buffer. > Is this is a right way to handle packet chunks which will be saved in the user buffer pointer provided in the rreq? > > Thanks! >

1 0

Regarding direct receive
by Kirjanov, Denis 09 Apr '13

09 Apr '13

Hello, I'm writing a netmod and I have a trouble with receiving a messages directly to the user buffer: The relevant code looks like the following: Assuming we have a request with the one iov entry /*************** if (!VC_CH(vc)->recv_active) { /* handle a new message using MPID_nem_handle_pkt() */ } else { MPIU_Memcpy(iov->MPID_IOV_BUF, &recv_buff->buff[0], bytes_recvd); iov->MPID_IOV_BUF = (void *)((char *)iov->MPID_IOV_BUF + bytes_recvd); iov->MPID_IOV_LEN -= bytes_recvd; } if (iov->MPID_IOV_LEN) { /* receive directly to the iov->MPID_IOV_BUF next time*/ goto fn_exit; } /* handle a recv request */ } ********************/ The problem that I'm running an IMB program compiled with CHECK option and it claims about corrupted buffer. Is this is a right way to handle packet chunks which will be saved in the user buffer pointer provided in the rreq? Thanks!

1 0

help, we are trying to make [wmpiconfig.exe] program.
by seungkyun lim 04 Apr '13

04 Apr '13

[Specification of our environment] OS: Windows 7 (64bit) Two workstations: AMD FX 8150(64bit) Programming environment: Visual studio 2010 (64bit) I use MPICH2 64 bit in windows 64bit. and I've already finished sample test for communicate and Master PC and slave PCs. now, I need command which can search slave PCs. It can communicate with saster PC from CMD program Is there any command of CMD program to search a slaves PCs in workgroup from master PC? lastly, I'd appreciate if you let me know some materials to make wmpiconfig.exe thank you

1 0

Re: [mpich-devel] external32 (was: Re: binding a new MPI type with fortran)
by Dave Goodell 03 Apr '13

03 Apr '13

On Apr 3, 2013, at 1:30 PM CDT, "Rob Latham" <robl(a)mcs.anl.gov> wrote: > On Wed, Apr 03, 2013 at 01:07:09PM -0500, Bob Cernohous wrote: >>> From: "Rob Latham" <robl(a)mcs.anl.gov> >>> Once you re-sync with MPICH, this use case will work -- integer, >>> float, and double have the same size types in native and external32 >>> format. >> >> 're=sync' meaning we get to MPICH 3.0? or we pick up new work/fixes >> related to ticket 1754? > > former: whenver you pull in MPICH 3.0. > > If you really really need external 32 bits right now you can cherry pick > fec9f28d0b67a8e6912b1a309b2739a2551b83d2 > 064194da2a8f30f52dd512693b34dad4f209497d > and > d8eec5491fa0929d43da3086230795a2fb1c6545 > > The meat of the work in ticket 1754 is yet to be done. FWIW, you may have some of these commits already if you're up to 1.5 already: ----8<---- % for c in fec9f2 064194 d8eec5 ; do git describe --contains $c ; done v1.5~62 v1.5~57 v3.0.3~53 ----8<---- -Dave

1 0

Re: [mpich-devel] external32 (was: Re: binding a new MPI type with fortran)
by Rob Latham 03 Apr '13

03 Apr '13

On Wed, Apr 03, 2013 at 01:07:09PM -0500, Bob Cernohous wrote: > > From: "Rob Latham" <robl(a)mcs.anl.gov> > > Once you re-sync with MPICH, this use case will work -- integer, > > float, and double have the same size types in native and external32 > > format. > > 're=sync' meaning we get to MPICH 3.0? or we pick up new work/fixes > related to ticket 1754? former: whenver you pull in MPICH 3.0. If you really really need external 32 bits right now you can cherry pick fec9f28d0b67a8e6912b1a309b2739a2551b83d2 064194da2a8f30f52dd512693b34dad4f209497d and d8eec5491fa0929d43da3086230795a2fb1c6545 The meat of the work in ticket 1754 is yet to be done. ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-devel] external32 (was: Re: binding a new MPI type with fortran)
by Bob Cernohous 03 Apr '13

03 Apr '13

> From: "Rob Latham" <robl(a)mcs.anl.gov> > Once you re-sync with MPICH, this use case will work -- integer, > float, and double have the same size types in native and external32 > format. 're=sync' meaning we get to MPICH 3.0? or we pick up new work/fixes related to ticket 1754?

1 0

external32 (was: Re: binding a new MPI type with fortran)
by Rob Latham 03 Apr '13

03 Apr '13

On Wed, Apr 03, 2013 at 09:19:59AM -0500, Bob Cernohous wrote: > I've been out sick and haven't followed this conversation too closely, but > I've gotten a support note that seems somewhat related -- at least they're > asking about external32. Any comments from the experts before I try to > dig in? ROMIO supports external32? A quick search shows external32 code > is common, not ad_bg specific. external32 is indeed a part of the standard. for 15 years ROMIO had no external32 support, and no one seemed to care too much. Mostly, the role of external32 -- a portable file format -- is better played by pnetcdf and hdf5, which in addition to being portable are also self-describing. Intel contributed external32 support to ROMIO, but Dave and I are pretty sure it's not going to work when type-on-platform is different from type-in-external32 See the end of https://trac.mpich.org/projects/mpich/ticket/1754 > The MPI standard defines a portable data representation for MPI I/O, > see e.g. [1]. On BGQ, attempting to write data via MPI I/O with > the data representation set to 'external32', MPI does not report any > errors but the data written is not in the correct format. Furthermore > trying to read the data back with the same data representation fails > and does not reproduce the original data. Right. I think the Intel-contributed external32 work, warts and all, went into MPICH 3.0 sorry we did not return an error in older versions. that's a bug. > The example Fortran code does the following: > 1) write a few numbers (integer, float and double) to file via MPI I/O > in native format > 2) read the numbers back in native format > 3) write the same numbers to file in external32 format > 4) output MPI errors/success when using MPI I/O > 5) read the numbers back from file > > 1+2 work flawlessly > 3+4 show no errors and instead indicate MPI SUCCESS > 5 fails to reproduce correct numbers Once you re-sync with MPICH, this use case will work -- integer, float, and double have the same size types in native and external32 format. > The data written in 1) appears as big endian data as expected. The > data written in 3) seems to be little endian (for integer and float) > and strangely byte-swapped for doubles. > > [1] http://www.mpi-forum.org/docs/mpi21-report/node285.htm > > > Bob Cernohous: (T/L 553) 507-253-6093 > > BobC(a)us.ibm.com > IBM Rochester, Building 030-2(C335), Department 61L > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > Chaos reigns within. > > Reflect, repent, and reboot. > > Order shall return. > > > > > From: Dave Goodell <goodell(a)mcs.anl.gov> > To: Larry Baker <baker(a)usgs.gov>, > Cc: devel(a)mpich.org > Date: 04/02/2013 04:48 PM > Subject: Re: [mpich-devel] binding a new MPI type with fortran > Sent by: devel-bounces(a)mpich.org > > > > On Apr 2, 2013, at 4:34 PM CDT, Larry Baker <baker(a)usgs.gov> wrote: > > > Jeff referred me to the MPI 3.0 spec ( > http://mpi-forum.org/docs/mpi-3.0/mpi30-report.pdf) for data sizes. But, > even it is contradictory. For example, in section 3.2.2, it says an > MPI_INTEGER is a Fortran (default KIND) INTEGER. Fortran does not specify > the size of a (default KIND) INTEGER. However, Table 13.2 says it is 4 > bytes. It could just as easily be 8 bytes in a 64-bit execution > environment. In that case, Fortran requires that the size of all the > default KIND numeric types in Table 13.2 would double in size. But, not > necessarily pointers; pointers in Fortran do not have a specified size > (storage unit). > > Table 13.2 lists sizes for a specific kind of encoding of MPI datatypes > known as "external32". You should not use it as a reference for the > actual size of the MPI datatype, which must instead always match the > actual language type in the corresponding language binding. > > > The MPI 3.0 spec says if you send an MPI_INTEGER, you have to receive an > MPI_INTEGER. I didn't exhaustively search the MPI 3.0 spec, but you might > also have to make sure the default KIND INTEGER is the same for the sender > and receiver. This might be beyond the scope of MPI, and I do not know > how exhaustive the run-time tests can be to detect violations of the > rules. (A message would have to carry type metadata for the payload data > in addition to the payload data itself.) Section 17 had lots of > information about Fortran and C data sizes, including automatically > conversions. I didn't read enough to know if the spec covers the case of > exchanging default KIND INTEGERs between two executables with different > sizes. Maybe it is true that MPI mandates something like lexical scoping > rules, such that, for example, all members of an MPI application must > assign the same semantics to their fundamental data types -- the default > KIND INTEGER, etc., in the case of Fortran. > > Most users should ignore the rules regarding conversions on heterogeneous > systems. They don't make a lot of sense even if you're an MPI expert, > and: > > 1) almost no MPI implementations properly support this behavior > > 2) almost no user codes are written correctly to deal with it, even if #1 > were true > > -Dave > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-devel] binding a new MPI type with fortran
by Bob Cernohous 03 Apr '13

03 Apr '13

I've been out sick and haven't followed this conversation too closely, but I've gotten a support note that seems somewhat related -- at least they're asking about external32. Any comments from the experts before I try to dig in? ROMIO supports external32? A quick search shows external32 code is common, not ad_bg specific. --------- The MPI standard defines a portable data representation for MPI I/O, see e.g. [1]. On BGQ, attempting to write data via MPI I/O with the data representation set to 'external32', MPI does not report any errors but the data written is not in the correct format. Furthermore trying to read the data back with the same data representation fails and does not reproduce the original data. The example Fortran code does the following: 1) write a few numbers (integer, float and double) to file via MPI I/O in native format 2) read the numbers back in native format 3) write the same numbers to file in external32 format 4) output MPI errors/success when using MPI I/O 5) read the numbers back from file 1+2 work flawlessly 3+4 show no errors and instead indicate MPI SUCCESS 5 fails to reproduce correct numbers The data written in 1) appears as big endian data as expected. The data written in 3) seems to be little endian (for integer and float) and strangely byte-swapped for doubles. [1] http://www.mpi-forum.org/docs/mpi21-report/node285.htm Bob Cernohous: (T/L 553) 507-253-6093 BobC(a)us.ibm.com IBM Rochester, Building 030-2(C335), Department 61L 3605 Hwy 52 North, Rochester, MN 55901-7829 > Chaos reigns within. > Reflect, repent, and reboot. > Order shall return. From: Dave Goodell <goodell(a)mcs.anl.gov> To: Larry Baker <baker(a)usgs.gov>, Cc: devel(a)mpich.org Date: 04/02/2013 04:48 PM Subject: Re: [mpich-devel] binding a new MPI type with fortran Sent by: devel-bounces(a)mpich.org On Apr 2, 2013, at 4:34 PM CDT, Larry Baker <baker(a)usgs.gov> wrote: > Jeff referred me to the MPI 3.0 spec ( http://mpi-forum.org/docs/mpi-3.0/mpi30-report.pdf) for data sizes. But, even it is contradictory. For example, in section 3.2.2, it says an MPI_INTEGER is a Fortran (default KIND) INTEGER. Fortran does not specify the size of a (default KIND) INTEGER. However, Table 13.2 says it is 4 bytes. It could just as easily be 8 bytes in a 64-bit execution environment. In that case, Fortran requires that the size of all the default KIND numeric types in Table 13.2 would double in size. But, not necessarily pointers; pointers in Fortran do not have a specified size (storage unit). Table 13.2 lists sizes for a specific kind of encoding of MPI datatypes known as "external32". You should not use it as a reference for the actual size of the MPI datatype, which must instead always match the actual language type in the corresponding language binding. > The MPI 3.0 spec says if you send an MPI_INTEGER, you have to receive an MPI_INTEGER. I didn't exhaustively search the MPI 3.0 spec, but you might also have to make sure the default KIND INTEGER is the same for the sender and receiver. This might be beyond the scope of MPI, and I do not know how exhaustive the run-time tests can be to detect violations of the rules. (A message would have to carry type metadata for the payload data in addition to the payload data itself.) Section 17 had lots of information about Fortran and C data sizes, including automatically conversions. I didn't read enough to know if the spec covers the case of exchanging default KIND INTEGERs between two executables with different sizes. Maybe it is true that MPI mandates something like lexical scoping rules, such that, for example, all members of an MPI application must assign the same semantics to their fundamental data types -- the default KIND INTEGER, etc., in the case of Fortran. Most users should ignore the rules regarding conversions on heterogeneous systems. They don't make a lot of sense even if you're an MPI expert, and: 1) almost no MPI implementations properly support this behavior 2) almost no user codes are written correctly to deal with it, even if #1 were true -Dave

1 0