May 2013 - devel - mailman3.cels.anl.gov

Re: [mpich-devel] ROMIO collective i/o memory use
by Bob Cernohous 06 May '13

06 May '13

I agree and suggested: --------------------- It appears they don't have enough memory for an alltoallv exchange. Try '1'... * - BGMPIO_COMM - Define how data is exchanged on collective * reads and writes. Possible values: * - 0 - Use MPI_Alltoallv. * - 1 - Use MPI_Isend/MPI_Irecv. * - Default is 0. --------------------- but they didn't want a work around they wanted a 'fix for o(p) allocations'. There are o(p) allocations all over collective i/o from a quick glance. Just wanted some input from the experts about scaling romio. I haven't heard if the suggestion worked. Bob Cernohous: (T/L 553) 507-253-6093 BobC(a)us.ibm.com IBM Rochester, Building 030-2(C335), Department 61L 3605 Hwy 52 North, Rochester, MN 55901-7829 > Chaos reigns within. > Reflect, repent, and reboot. > Order shall return. devel-bounces(a)mpich.org wrote on 05/04/2013 09:43:10 PM: > From: "Rob Latham" <robl(a)mcs.anl.gov> > To: devel(a)mpich.org, > Cc: mpich2-dev(a)mcs.anl.gov > Date: 05/04/2013 09:48 PM > Subject: Re: [mpich-devel] ROMIO collective i/o memory use > Sent by: devel-bounces(a)mpich.org > > On Mon, Apr 29, 2013 at 10:28:01AM -0500, Bob Cernohous wrote: > > A customer (Argonne ;) is complaining about O(p) allocations in collective > > i/o. A collective read is failing at larger scale. > > > > Any thoughts or comments or advice? There appears to be lots of O(p) in > > ROMIO collective I/O. Plus a lot of (possibly large) aggregated data > > buffers. A quick search shows > > The O(p) allocations are a concern, sure. For two-phase, though, the > real problem lies in ADIOI_R_Exchange_data_alltoallv and > ADIOI_W_Exchange_data_alltoallv . The O(p) allocations are the least > of our worries! > > around line 1063 of ad_bg_rdcoll.c > > all_recv_buf = (char *) ADIOI_Malloc( rtail ); > > all_send_buf = (char *) ADIOI_Malloc( stail ); > > (rtail and stail are the sum of the receive and send arrays) > > ==rob > > > The common ROMIO read collective code: > > > > Find all "ADIOI_Malloc", Match case, Regular expression (UNIX) > > > > File Z:\bgq\comm\lib\dev\mpich2\src\mpi\romio\adio\common\ad_read_coll.c > > > > 124 38: st_offsets = (ADIO_Offset *) > > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > > > > 125 39: end_offsets = (ADIO_Offset *) > > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > > > > 317 44: *offset_list_ptr = (ADIO_Offset *) > > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > > > > 318 41: *len_list_ptr = (ADIO_Offset *) > > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > > > > 334 44: *offset_list_ptr = (ADIO_Offset *) > > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > > > > 335 41: *len_list_ptr = (ADIO_Offset *) > > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > > > > 436 18: ADIOI_Malloc((contig_access_count+1)*sizeof(ADIO_Offset)); > > > > 437 41: *len_list_ptr = (ADIO_Offset *) > > ADIOI_Malloc((contig_access_count+1)*sizeof(ADIO_Offset)); > > > > 573 37: if (ntimes) read_buf = (char *) ADIOI_Malloc(coll_bufsize); > > > > 578 21: count = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > 587 25: send_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > 590 25: recv_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > 598 25: start_pos = (int *) ADIOI_Malloc(nprocs*sizeof(int)); > > > > 739 32: tmp_buf = (char *) ADIOI_Malloc(for_next_iter); > > > > 744 33: read_buf = (char *) > > ADIOI_Malloc(for_next_iter+coll_bufsize); > > > > 805 9: ADIOI_Malloc((nprocs_send+nprocs_recv+1)*sizeof(MPI_Request)); > > > > 827 30: recv_buf = (char **) ADIOI_Malloc(nprocs * sizeof(char*)); > > > > 830 44: (char *) > > ADIOI_Malloc(recv_size[i]); > > > > 870 31: statuses = (MPI_Status *) > > ADIOI_Malloc((nprocs_send+nprocs_recv+1) * \ > > > > 988 35: curr_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > > sizeof(unsigned)); > > > > 989 35: done_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > > sizeof(unsigned)); > > > > 990 35: recv_buf_idx = (unsigned *) ADIOI_Malloc(nprocs * > > sizeof(unsigned)); > > > > Total found: 22 > > > > > > Our BG version of read collective: > > > > File Z:\bgq\comm\lib\dev\mpich2\src\mpi\romio\adio\ad_bg\ad_bg_rdcoll.c > > > > 179 40: st_offsets = (ADIO_Offset *) > > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > > > > 180 40: end_offsets = (ADIO_Offset *) > > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > > > > 183 43: bg_offsets0 = (ADIO_Offset *) > > ADIOI_Malloc(2*nprocs*sizeof(ADIO_Offset)); > > > > 184 43: bg_offsets = (ADIO_Offset *) > > ADIOI_Malloc(2*nprocs*sizeof(ADIO_Offset)); > > > > 475 37: if (ntimes) read_buf = (char *) ADIOI_Malloc(coll_bufsize); > > > > 480 21: count = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > 489 25: send_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > 492 25: recv_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > 500 25: start_pos = (int *) ADIOI_Malloc(nprocs*sizeof(int)); > > > > 676 32: tmp_buf = (char *) ADIOI_Malloc(for_next_iter); > > > > 681 33: read_buf = (char *) > > ADIOI_Malloc(for_next_iter+coll_bufsize); > > > > 761 9: ADIOI_Malloc((nprocs_send+nprocs_recv+1)*sizeof(MPI_Request)); > > > > 783 30: recv_buf = (char **) ADIOI_Malloc(nprocs * sizeof(char*)); > > > > 786 44: (char *) > > ADIOI_Malloc(recv_size[i]); > > > > 826 31: statuses = (MPI_Status *) > > ADIOI_Malloc((nprocs_send+nprocs_recv+1) * \ > > > > 944 35: curr_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > > sizeof(unsigned)); > > > > 945 35: done_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > > sizeof(unsigned)); > > > > 946 35: recv_buf_idx = (unsigned *) ADIOI_Malloc(nprocs * > > sizeof(unsigned)); > > > > 1058 23: rdispls = (int *) ADIOI_Malloc( nprocs * sizeof(int) ); > > > > 1063 29: all_recv_buf = (char *) ADIOI_Malloc( rtail ); > > > > 1064 26: recv_buf = (char **) ADIOI_Malloc(nprocs * sizeof(char *)); > > > > 1068 23: sdispls = (int *) ADIOI_Malloc( nprocs * sizeof(int) ); > > > > 1073 29: all_send_buf = (char *) ADIOI_Malloc( stail ); > > > > Total found: 23 > > > > > > Bob Cernohous: (T/L 553) 507-253-6093 > > > > BobC(a)us.ibm.com > > IBM Rochester, Building 030-2(C335), Department 61L > > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > > > Chaos reigns within. > > > Reflect, repent, and reboot. > > > Order shall return. > > -- > Rob Latham > Mathematics and Computer Science Division > Argonne National Lab, IL USA >

1 0

Re: [mpich-devel] ROMIO collective i/o memory use
by Rob Latham 04 May '13

04 May '13

On Mon, Apr 29, 2013 at 10:28:01AM -0500, Bob Cernohous wrote: > A customer (Argonne ;) is complaining about O(p) allocations in collective > i/o. A collective read is failing at larger scale. > > Any thoughts or comments or advice? There appears to be lots of O(p) in > ROMIO collective I/O. Plus a lot of (possibly large) aggregated data > buffers. A quick search shows The O(p) allocations are a concern, sure. For two-phase, though, the real problem lies in ADIOI_R_Exchange_data_alltoallv and ADIOI_W_Exchange_data_alltoallv . The O(p) allocations are the least of our worries! around line 1063 of ad_bg_rdcoll.c all_recv_buf = (char *) ADIOI_Malloc( rtail ); all_send_buf = (char *) ADIOI_Malloc( stail ); (rtail and stail are the sum of the receive and send arrays) ==rob > The common ROMIO read collective code: > > Find all "ADIOI_Malloc", Match case, Regular expression (UNIX) > > File Z:\bgq\comm\lib\dev\mpich2\src\mpi\romio\adio\common\ad_read_coll.c > > 124 38: st_offsets = (ADIO_Offset *) > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > > 125 39: end_offsets = (ADIO_Offset *) > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > > 317 44: *offset_list_ptr = (ADIO_Offset *) > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > > 318 41: *len_list_ptr = (ADIO_Offset *) > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > > 334 44: *offset_list_ptr = (ADIO_Offset *) > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > > 335 41: *len_list_ptr = (ADIO_Offset *) > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > > 436 18: ADIOI_Malloc((contig_access_count+1)*sizeof(ADIO_Offset)); > > 437 41: *len_list_ptr = (ADIO_Offset *) > ADIOI_Malloc((contig_access_count+1)*sizeof(ADIO_Offset)); > > 573 37: if (ntimes) read_buf = (char *) ADIOI_Malloc(coll_bufsize); > > 578 21: count = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > 587 25: send_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > 590 25: recv_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > 598 25: start_pos = (int *) ADIOI_Malloc(nprocs*sizeof(int)); > > 739 32: tmp_buf = (char *) ADIOI_Malloc(for_next_iter); > > 744 33: read_buf = (char *) > ADIOI_Malloc(for_next_iter+coll_bufsize); > > 805 9: ADIOI_Malloc((nprocs_send+nprocs_recv+1)*sizeof(MPI_Request)); > > 827 30: recv_buf = (char **) ADIOI_Malloc(nprocs * sizeof(char*)); > > 830 44: (char *) > ADIOI_Malloc(recv_size[i]); > > 870 31: statuses = (MPI_Status *) > ADIOI_Malloc((nprocs_send+nprocs_recv+1) * \ > > 988 35: curr_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > sizeof(unsigned)); > > 989 35: done_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > sizeof(unsigned)); > > 990 35: recv_buf_idx = (unsigned *) ADIOI_Malloc(nprocs * > sizeof(unsigned)); > > Total found: 22 > > > Our BG version of read collective: > > File Z:\bgq\comm\lib\dev\mpich2\src\mpi\romio\adio\ad_bg\ad_bg_rdcoll.c > > 179 40: st_offsets = (ADIO_Offset *) > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > > 180 40: end_offsets = (ADIO_Offset *) > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > > 183 43: bg_offsets0 = (ADIO_Offset *) > ADIOI_Malloc(2*nprocs*sizeof(ADIO_Offset)); > > 184 43: bg_offsets = (ADIO_Offset *) > ADIOI_Malloc(2*nprocs*sizeof(ADIO_Offset)); > > 475 37: if (ntimes) read_buf = (char *) ADIOI_Malloc(coll_bufsize); > > 480 21: count = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > 489 25: send_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > 492 25: recv_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > 500 25: start_pos = (int *) ADIOI_Malloc(nprocs*sizeof(int)); > > 676 32: tmp_buf = (char *) ADIOI_Malloc(for_next_iter); > > 681 33: read_buf = (char *) > ADIOI_Malloc(for_next_iter+coll_bufsize); > > 761 9: ADIOI_Malloc((nprocs_send+nprocs_recv+1)*sizeof(MPI_Request)); > > 783 30: recv_buf = (char **) ADIOI_Malloc(nprocs * sizeof(char*)); > > 786 44: (char *) > ADIOI_Malloc(recv_size[i]); > > 826 31: statuses = (MPI_Status *) > ADIOI_Malloc((nprocs_send+nprocs_recv+1) * \ > > 944 35: curr_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > sizeof(unsigned)); > > 945 35: done_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > sizeof(unsigned)); > > 946 35: recv_buf_idx = (unsigned *) ADIOI_Malloc(nprocs * > sizeof(unsigned)); > > 1058 23: rdispls = (int *) ADIOI_Malloc( nprocs * sizeof(int) ); > > 1063 29: all_recv_buf = (char *) ADIOI_Malloc( rtail ); > > 1064 26: recv_buf = (char **) ADIOI_Malloc(nprocs * sizeof(char *)); > > 1068 23: sdispls = (int *) ADIOI_Malloc( nprocs * sizeof(int) ); > > 1073 29: all_send_buf = (char *) ADIOI_Malloc( stail ); > > Total found: 23 > > > Bob Cernohous: (T/L 553) 507-253-6093 > > BobC(a)us.ibm.com > IBM Rochester, Building 030-2(C335), Department 61L > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > Chaos reigns within. > > Reflect, repent, and reboot. > > Order shall return. -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-devel] segfault calling neighbor collectives in communicator with no topology
by Dave Goodell 02 May '13

02 May '13

On May 2, 2013, at 11:27 AM CDT, Lisandro Dalcin <dalcinl(a)gmail.com> wrote: > On 2 May 2013 18:58, Dave Goodell <goodell(a)mcs.anl.gov> wrote: >> Thanks for letting us know. I've created a ticket to track this and commented on your suggestions there: >> >> https://trac.mpich.org/projects/mpich/ticket/1833#comment:1 >> > > Thanks. BTW, you are right. This was indeed bad API usage from the > user. All neighbor collectives should check that the communicator is > an INTRAcommunicator with cart|graph|distgraph topology attached and > otherwise generate a proper error (of class MPI_ERR_COMM?) Sounds right to me. There might be a more appropriate MPI_ERR_* class, but I'd have to re-read the list to be sure. -Dave

1 0

Re: [mpich-devel] segfault calling neighbor collectives in communicator with no topology
by Lisandro Dalcin 02 May '13

02 May '13

On 2 May 2013 18:58, Dave Goodell <goodell(a)mcs.anl.gov> wrote: > Thanks for letting us know. I've created a ticket to track this and commented on your suggestions there: > > https://trac.mpich.org/projects/mpich/ticket/1833#comment:1 > Thanks. BTW, you are right. This was indeed bad API usage from the user. All neighbor collectives should check that the communicator is an INTRAcommunicator with cart|graph|distgraph topology attached and otherwise generate a proper error (of class MPI_ERR_COMM?) -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169

1 0

Re: [mpich-devel] segfault calling neighbor collectives in communicator with no topology
by Dave Goodell 02 May '13

02 May '13

Thanks for letting us know. I've created a ticket to track this and commented on your suggestions there: https://trac.mpich.org/projects/mpich/ticket/1833#comment:1 -Dave On Apr 30, 2013, at 2:53 AM CDT, Lisandro Dalcin <dalcinl(a)gmail.com> wrote: > I'm adding support for the MPI-3 neighborhood collectives to mpi4py. > By mistake, I called a neighbor collective on COMM_SELF, and got a > segfault. After running under valgrind, I get the trace below. > > It seems that MPICH (running 3.0.4) is not checking the communicators > for a topology being attached. This should be fixed in > MPIR_Topo_canon_nhb_count() at src/mpi/topo/topoutil.c, adding a check > after the following line: > > topo_ptr = MPIR_Topology_get(comm_ptr); > > BTW, the same kind of check should also be added to MPIR_Topo_canon_nhb(). > > > ==14696== Invalid read of size 4 > ==14696== at 0xDE0ED39: MPIR_Topo_canon_nhb_count (topoutil.c:283) > ==14696== by 0xDFA870A: MPIR_Ineighbor_allgather_default > (inhb_allgather.c:50) > ==14696== by 0xDFA8B8B: MPIR_Ineighbor_allgather_impl (inhb_allgather.c:98) > ==14696== by 0xDFAE27C: MPIR_Neighbor_allgather_default (nhb_allgather.c:37) > ==14696== by 0xDFAE350: MPIR_Neighbor_allgather_impl (nhb_allgather.c:58) > ==14696== by 0xDFAE918: PMPI_Neighbor_allgather (nhb_allgather.c:155) > ==14696== by 0xDAD7B77: > __pyx_pw_6mpi4py_3MPI_9Intracomm_25Neighbor_allgather > (mpi4py.MPI.c:87767) > ==14696== by 0x31784DD280: PyEval_EvalFrameEx (in > /usr/lib64/libpython2.7.so.1.0) > ==14696== by 0x31784DCEF0: PyEval_EvalFrameEx (in > /usr/lib64/libpython2.7.so.1.0) > ==14696== by 0x31784DDCBE: PyEval_EvalCodeEx (in > /usr/lib64/libpython2.7.so.1.0) > ==14696== by 0x317846DA36: ??? (in /usr/lib64/libpython2.7.so.1.0) > ==14696== by 0x3178449C0D: PyObject_Call (in /usr/lib64/libpython2.7.so.1.0) > ==14696== Address 0x0 is not stack'd, malloc'd or (recently) free'd > > > -- > Lisandro Dalcin > --------------- > CIMEC (INTEC/CONICET-UNL) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > 3000 Santa Fe, Argentina > Tel: +54-342-4511594 (ext 1011) > Tel/Fax: +54-342-4511169

1 0

Re: [mpich-devel] Access to git
by William Gropp 30 Apr '13

30 Apr '13

Probably - I don't remember what I did, but its likely I generated a new public key. I've installed my key pair on MCS and was able to clone the writable directory. It looks like my keys disappeared from MCS. Bill William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Thomas M. Siebel Chair in Computer Science University of Illinois Urbana-Champaign On Apr 30, 2013, at 3:56 PM, Pavan Balaji wrote: > Bill, > > The git setup requires your ssh public key to be loaded on the server. > Are you using different matching private keys on MCS and your laptop? > > -- Pavan > > On 04/30/2013 01:25 PM US Central Time, William Gropp wrote: >> I believe that the access setup for git requires something special be >> done - I have no trouble using git on my Mac but not having a version >> that I can push from on MCS means I have to move corrections back to my >> home system. >> >> Bill >> >> William Gropp >> Director, Parallel Computing Institute >> Deputy Director for Research >> Institute for Advanced Computing Applications and Technologies >> Thomas M. Siebel Chair in Computer Science >> University of Illinois Urbana-Champaign >> >> >> >> >> On Apr 30, 2013, at 1:36 PM, Rajeev Thakur wrote: >> >>> It worked for me on /tmp on octopus just now. >>> >>> Rajeev >>> >>> On Apr 30, 2013, at 11:29 AM, William Gropp wrote: >>> >>>> I can't checkout mpich when I'm logged into MCS (e.g., from octopus). >>>> The git page says to send this email when that happens :) >>>> >>>> Bill >>>> >>>> William Gropp >>>> Director, Parallel Computing Institute >>>> Deputy Director for Research >>>> Institute for Advanced Computing Applications and Technologies >>>> Thomas M. Siebel Chair in Computer Science >>>> University of Illinois Urbana-Champaign >>>> >>>> >>>> >>>> >>> >> > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji

1 0