May 2013 - devel - mailman3.cels.anl.gov

Re: [mpich-devel] s/bcopy/memmove/
by Pavan Balaji 18 May '13

18 May '13

Jeff, Can you send the full changeset as a single git-formatted patch? -- Pavan On 05/18/2013 09:15 PM US Central Time, Jeff Hammond wrote: > (background: i previously replaced all instances of bzero with memset > but only ibm@mpich was copied) > > there was only one instance of bopy that wasn't macro-ed to memmove > already so i fixed it. patch attached. > > jeff > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

s/bcopy/memmove/
by Jeff Hammond 18 May '13

18 May '13

(background: i previously replaced all instances of bzero with memset but only ibm@mpich was copied) there was only one instance of bopy that wasn't macro-ed to memmove already so i fixed it. patch attached. jeff -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

Re: [mpich-devel] mpich.org is down?
by Jeff Hammond 09 May '13

09 May '13

Just thank the good lord that we're using Git now and you aren't completely foobarred by the lack of Argonne internets. Jeff On Thu, May 9, 2013 at 2:05 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > mpich.org is still unreachable though. Should be fixed soon. > > On May 9, 2013, at 2:04 PM, Antonio J. Peña wrote: > >> >> Dear Michael, >> >> We experienced a network issue at institutional level. It seems to be solved right now. >> >> Antonio >> >> >> On Thursday, May 09, 2013 11:03:30 AM Michael Blocksome wrote: >> I can't access the git repositories or the trac database on mpich.org. Any ideas? >> >> Michael Blocksome >> Blue Gene Messaging >> blocksom(a)us.ibm.com >> >> >> >> -- >> Antonio J. Peña >> Postdoctoral Appointee >> Mathematics and Computer Science Division >> Argonne National Laboratory >> 9700 South Cass Avenue, Bldg. 240, Of. 3148 >> Argonne, IL 60439-4847 >> (+1) 630-252-7928 >> apenya(a)mcs.anl.gov >> > -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

Re: [mpich-devel] mpich.org is down?
by Rajeev Thakur 09 May '13

09 May '13

mpich.org is still unreachable though. Should be fixed soon. On May 9, 2013, at 2:04 PM, Antonio J. Peña wrote: > > Dear Michael, > > We experienced a network issue at institutional level. It seems to be solved right now. > > Antonio > > > On Thursday, May 09, 2013 11:03:30 AM Michael Blocksome wrote: > I can't access the git repositories or the trac database on mpich.org. Any ideas? > > Michael Blocksome > Blue Gene Messaging > blocksom(a)us.ibm.com > > > > -- > Antonio J. Peña > Postdoctoral Appointee > Mathematics and Computer Science Division > Argonne National Laboratory > 9700 South Cass Avenue, Bldg. 240, Of. 3148 > Argonne, IL 60439-4847 > (+1) 630-252-7928 > apenya(a)mcs.anl.gov >

1 0

Re: [mpich-devel] mpich.org is down?
by Antonio J. Peña 09 May '13

09 May '13

Dear Michael, We experienced a network issue at institutional level. It seems to be solved right now. Antonio On Thursday, May 09, 2013 11:03:30 AM Michael Blocksome wrote: I can't access the git repositories or the trac database on mpich.org. Any ideas? Michael Blocksome Blue Gene Messaging blocksom(a)us.ibm.com -- Antonio J. Peña Postdoctoral Appointee Mathematics and Computer Science Division Argonne National Laboratory 9700 South Cass Avenue, Bldg. 240, Of. 3148 Argonne, IL 60439-4847 (+1) 630-252-7928 apenya(a)mcs.anl.gov

1 0

mpich.org is down?
by Michael Blocksome 09 May '13

09 May '13

I can't access the git repositories or the trac database on mpich.org. Any ideas? Michael Blocksome Blue Gene Messaging blocksom(a)us.ibm.com

1 0

Re: [mpich-devel] ROMIO collective i/o memory use
by Rob Latham 07 May '13

07 May '13

On Mon, May 06, 2013 at 03:05:10PM -0500, Bob Cernohous wrote: > Ouch. So you've seen the scaling failures at 8k... ranks? racks? > Kevin is failing at... 16 racks x 16 ranks per node... I think ... > so 256k ranks. MOAB in particular has fairly straightforward workloads that will exhaust memory (by attempting a 400 MiB memory allocation) at 2048 MPI processes. ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-devel] ROMIO collective i/o memory use
by Rob Latham 07 May '13

07 May '13

On Mon, May 06, 2013 at 01:41:07PM -0500, Bob Cernohous wrote: > I agree and suggested: > --------------------- > It appears they don't have enough memory for an alltoallv exchange. Try > '1'... > > * - BGMPIO_COMM - Define how data is exchanged on collective > > * reads and writes. Possible values: > > * - 0 - Use MPI_Alltoallv. > > * - 1 - Use MPI_Isend/MPI_Irecv. > > * - Default is 0. > > --------------------- > > but they didn't want a work around they wanted a 'fix for o(p) > allocations'. There are o(p) allocations all over collective i/o from a > quick glance. Just wanted some input from the experts about scaling > romio. I haven't heard if the suggestion worked. MOAB, a framework for "doing stuff to meshed data": simple experiment of reading and writing a mesh with 8 million "tetrahedron" elements. Setting BGMPIO_COMM to 1 serves to push the memory allocation problem only a little bit down the road. I could not run this test case until our Blue Gene /P came out of maintenance With default BG /P settings, MOAB cannot even read in the initial 8 million tet dataset with 2048 MPI processes. Set BGPMPIO_COMM to 1 and 2048 MPI processes work, but 4192 does not: I'm still working off of our /P but the code for /Q is the same: stderr[3260] Abort(-1) on node 3260: Unable to allocate non-contiguous buffer (Not enough space for file ) (that error message is misleading: it's a malloc that has returned NULL) 0x01a7e0ec raise ../nptl/sysdeps/unix/sysv/linux/raise.c:67 0x01a3ce40 abort /bgsys/drivers/V1R4M2_200_2010-100508P/ppc/toolchain/gnu/glibc-2.4/stdlib/abort.c:73 0x018a9df4 MPID_Abort /gpfs/home/robl/src/dcmf/BGP/IBM_V1R4M3/bgp/comm/lib/dev/mpich2/src/mpid/dcmfd/src/misc/mp id_abort.c:81 0x018ad4e8 MPIDI_DCMF_StartMsg /gpfs/home/robl/src/dcmf/BGP/IBM_V1R4M3/bgp/comm/lib/dev/mpich2/src/mpid/dcmfd/src/pt2pt/m pidi_startmessage.c:173 0x018abcb0 MPID_Isend /gpfs/home/robl/src/dcmf/BGP/IBM_V1R4M3/bgp/comm/lib/dev/mpich2/src/mpid/dcmfd/src/pt2pt/m pid_isend.c:105 0x01878c34 PMPI_Isend /gpfs/home/robl/src/dcmf/BGP/IBM_V1R4M3/bgp/comm/lib/dev/mpich2/src/mpi/pt2pt/isend.c:124 0x01890818 ADIOI_R_Exchange_data /gpfs/home/robl/src/dcmf/BGP/IBM_V1R4M3/bgp/comm/lib/dev/mpich2/src/mpi/romio/adio/ad_bgl/ ad_bgl_rdcoll.c:828 0x018929e0 ADIOI_Read_and_exch /gpfs/home/robl/src/dcmf/BGP/IBM_V1R4M3/bgp/comm/lib/dev/mpich2/src/mpi/romio/adio/ad_bgl/ ad_bgl_rdcoll.c:660 So, the suggestion to turn off the alltoallv optimization doesn't really work if we can't even run on one rack of /P I would say. ==rob > Bob Cernohous: (T/L 553) 507-253-6093 > > BobC(a)us.ibm.com > IBM Rochester, Building 030-2(C335), Department 61L > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > Chaos reigns within. > > Reflect, repent, and reboot. > > Order shall return. > > > devel-bounces(a)mpich.org wrote on 05/04/2013 09:43:10 PM: > > > From: "Rob Latham" <robl(a)mcs.anl.gov> > > To: devel(a)mpich.org, > > Cc: mpich2-dev(a)mcs.anl.gov > > Date: 05/04/2013 09:48 PM > > Subject: Re: [mpich-devel] ROMIO collective i/o memory use > > Sent by: devel-bounces(a)mpich.org > > > > On Mon, Apr 29, 2013 at 10:28:01AM -0500, Bob Cernohous wrote: > > > A customer (Argonne ;) is complaining about O(p) allocations in > collective > > > i/o. A collective read is failing at larger scale. > > > > > > Any thoughts or comments or advice? There appears to be lots of O(p) > in > > > ROMIO collective I/O. Plus a lot of (possibly large) aggregated data > > > buffers. A quick search shows > > > > The O(p) allocations are a concern, sure. For two-phase, though, the > > real problem lies in ADIOI_R_Exchange_data_alltoallv and > > ADIOI_W_Exchange_data_alltoallv . The O(p) allocations are the least > > of our worries! > > > > around line 1063 of ad_bg_rdcoll.c > > > > all_recv_buf = (char *) ADIOI_Malloc( rtail ); > > > > all_send_buf = (char *) ADIOI_Malloc( stail ); > > > > (rtail and stail are the sum of the receive and send arrays) > > > > ==rob > > > > > The common ROMIO read collective code: > > > > > > Find all "ADIOI_Malloc", Match case, Regular expression (UNIX) > > > > > > File > Z:\bgq\comm\lib\dev\mpich2\src\mpi\romio\adio\common\ad_read_coll.c > > > > > > 124 38: st_offsets = (ADIO_Offset *) > > > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > > > > > > 125 39: end_offsets = (ADIO_Offset *) > > > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > > > > > > 317 44: *offset_list_ptr = (ADIO_Offset *) > > > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > > > > > > 318 41: *len_list_ptr = (ADIO_Offset *) > > > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > > > > > > 334 44: *offset_list_ptr = (ADIO_Offset *) > > > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > > > > > > 335 41: *len_list_ptr = (ADIO_Offset *) > > > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > > > > > > 436 18: ADIOI_Malloc((contig_access_count+1)*sizeof(ADIO_Offset)); > > > > > > 437 41: *len_list_ptr = (ADIO_Offset *) > > > ADIOI_Malloc((contig_access_count+1)*sizeof(ADIO_Offset)); > > > > > > 573 37: if (ntimes) read_buf = (char *) > ADIOI_Malloc(coll_bufsize); > > > > > > 578 21: count = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > > > 587 25: send_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > > > 590 25: recv_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > > > 598 25: start_pos = (int *) ADIOI_Malloc(nprocs*sizeof(int)); > > > > > > 739 32: tmp_buf = (char *) ADIOI_Malloc(for_next_iter); > > > > > > 744 33: read_buf = (char *) > > > ADIOI_Malloc(for_next_iter+coll_bufsize); > > > > > > 805 9: > ADIOI_Malloc((nprocs_send+nprocs_recv+1)*sizeof(MPI_Request)); > > > > > > 827 30: recv_buf = (char **) ADIOI_Malloc(nprocs * > sizeof(char*)); > > > > > > 830 44: (char *) > > > ADIOI_Malloc(recv_size[i]); > > > > > > 870 31: statuses = (MPI_Status *) > > > ADIOI_Malloc((nprocs_send+nprocs_recv+1) * \ > > > > > > 988 35: curr_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > > > sizeof(unsigned)); > > > > > > 989 35: done_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > > > sizeof(unsigned)); > > > > > > 990 35: recv_buf_idx = (unsigned *) ADIOI_Malloc(nprocs * > > > sizeof(unsigned)); > > > > > > Total found: 22 > > > > > > > > > Our BG version of read collective: > > > > > > File > Z:\bgq\comm\lib\dev\mpich2\src\mpi\romio\adio\ad_bg\ad_bg_rdcoll.c > > > > > > 179 40: st_offsets = (ADIO_Offset *) > > > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > > > > > > 180 40: end_offsets = (ADIO_Offset *) > > > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > > > > > > 183 43: bg_offsets0 = (ADIO_Offset *) > > > ADIOI_Malloc(2*nprocs*sizeof(ADIO_Offset)); > > > > > > 184 43: bg_offsets = (ADIO_Offset *) > > > ADIOI_Malloc(2*nprocs*sizeof(ADIO_Offset)); > > > > > > 475 37: if (ntimes) read_buf = (char *) > ADIOI_Malloc(coll_bufsize); > > > > > > 480 21: count = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > > > 489 25: send_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > > > 492 25: recv_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > > > 500 25: start_pos = (int *) ADIOI_Malloc(nprocs*sizeof(int)); > > > > > > 676 32: tmp_buf = (char *) ADIOI_Malloc(for_next_iter); > > > > > > 681 33: read_buf = (char *) > > > ADIOI_Malloc(for_next_iter+coll_bufsize); > > > > > > 761 9: > ADIOI_Malloc((nprocs_send+nprocs_recv+1)*sizeof(MPI_Request)); > > > > > > 783 30: recv_buf = (char **) ADIOI_Malloc(nprocs * > sizeof(char*)); > > > > > > 786 44: (char *) > > > ADIOI_Malloc(recv_size[i]); > > > > > > 826 31: statuses = (MPI_Status *) > > > ADIOI_Malloc((nprocs_send+nprocs_recv+1) * \ > > > > > > 944 35: curr_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > > > sizeof(unsigned)); > > > > > > 945 35: done_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > > > sizeof(unsigned)); > > > > > > 946 35: recv_buf_idx = (unsigned *) ADIOI_Malloc(nprocs * > > > sizeof(unsigned)); > > > > > > 1058 23: rdispls = (int *) ADIOI_Malloc( nprocs * sizeof(int) ); > > > > > > 1063 29: all_recv_buf = (char *) ADIOI_Malloc( rtail ); > > > > > > 1064 26: recv_buf = (char **) ADIOI_Malloc(nprocs * sizeof(char > *)); > > > > > > 1068 23: sdispls = (int *) ADIOI_Malloc( nprocs * sizeof(int) ); > > > > > > 1073 29: all_send_buf = (char *) ADIOI_Malloc( stail ); > > > > > > Total found: 23 > > > > > > > > > Bob Cernohous: (T/L 553) 507-253-6093 > > > > > > BobC(a)us.ibm.com > > > IBM Rochester, Building 030-2(C335), Department 61L > > > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > > > > > Chaos reigns within. > > > > Reflect, repent, and reboot. > > > > Order shall return. > > > > -- > > Rob Latham > > Mathematics and Computer Science Division > > Argonne National Lab, IL USA > > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

unsubscribe
by Paul, Brent S 06 May '13

06 May '13

==================================================== Brent S. Paul Principal Engineer, Hydrodynamics and Acoustics R&D Department Alion Science and Technology (860) 462-1038 <mailto:[email protected]> bpaul(a)alionscience.com ====================================================

1 0

Re: [mpich-devel] ROMIO collective i/o memory use
by Bob Cernohous 06 May '13

06 May '13

I think my memory was off. Jeff actually put in a different alltoall involving... --------------------------------------------------------------------- The performance problem has been isolated to a single function in the MPI-IO ROMIO common directory, ADIOI_Calc_others_req(). This function was consuming 90% of the time between I/O syscalls. It was doing Isend/Irecv between all of the nodes, twice. I copied that function to a BG/L-specific module, rewrote it to use MPI_Alltoallv(), and changed MPI-IO collective write and read to call it by default, or when -env BGLMPIO_TUNEBLOCKING=1 is specified. The new function only consumed 16% of the time between I/O syscalls, and brought the total time in line with using MPI_Gather() and POSIX I/O. ... Both the performance fix and the memory leak fix went into the V1R3M2 DRV140_2007-070417. I compiled both testcases against that driver and ran them on a full rack. The performance was good (see below), and the memory leak was gone. Performance fix: [0] using 5120 blocks per task, and ntasks = 1024 ... [0] Time to write the file with MPI_File_write_all = 61.602 seconds, bandwidth = 0.340 MB/sec [0] file block size = 87600 [0] Time to write the file with posix write = 84.563 seconds, bandwidth = 0.248 MB/sec ------------------------------------------------------------------- Anyway, this is somewhat off the original topic of o(p) allocations except that it often seems to be a trade off between performance and memory. Bob Cernohous: (T/L 553) 507-253-6093 BobC(a)us.ibm.com IBM Rochester, Building 030-2(C335), Department 61L 3605 Hwy 52 North, Rochester, MN 55901-7829 > Chaos reigns within. > Reflect, repent, and reboot. > Order shall return. devel-bounces(a)mpich.org wrote on 05/06/2013 04:54:10 PM: > From: Bob Cernohous/Rochester/IBM@IBMUS > To: mpich2-dev(a)mcs.anl.gov, > Date: 05/06/2013 04:55 PM > Subject: Re: [mpich-devel] ROMIO collective i/o memory use > Sent by: devel-bounces(a)mpich.org > > I did a couple quick searches and this goes back to bg/l, not bg/p. > Jeff once investigated a problem and said > ---------------- > In general, for smaller blocks, MPI-IO performed better than POSIX > IO. For a midplane, they are about equal. For a rack, MPI-IO is > noticably slower. I am now suspecting the collective phase of MPI- > IO may be taking the time (#2 above). > > Specifying -env BGLMPIO_TUNEGATHER=0 did not significantly change > the 1 rack result. This controls using allgather (0) vs allreduce > (1 - the default) to communicate start and end offsets among the nodes. > > **Specifying -env BGLMPIO_COMM=1 made the 1 rack result twice as > slow. This controls using alltoallv (0 - the default) vs send/recv > (1) to do the consolidation phase. > > Specifying -env BGLMPIO_TUNEBLOCKING=0 made the 1 rack result so > slow that I cancelled the job. This controls whether to take psets > and GPFS into account (1 - the default) or not (0). > --------------- > I can't find the issue where he first implemented BGLMPIO_COMM. I > seem to remember it performed MUCH better on small scattered i/o > than send/recv. He had tables and numbers which I can't find right now. > > > Bob Cernohous: (T/L 553) 507-253-6093 > > BobC(a)us.ibm.com > IBM Rochester, Building 030-2(C335), Department 61L > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > Chaos reigns within. > > Reflect, repent, and reboot. > > Order shall return. > > > devel-bounces(a)mpich.org wrote on 05/06/2013 04:35:11 PM: > > > From: Jeff Hammond <jhammond(a)alcf.anl.gov> > > To: devel(a)mpich.org, > > Cc: mpich2-dev(a)mcs.anl.gov, devel-bounces(a)mpich.org > > Date: 05/06/2013 04:36 PM > > Subject: Re: [mpich-devel] ROMIO collective i/o memory use > > Sent by: devel-bounces(a)mpich.org > > > > Do alltoallv actually run faster than send-recv for the MPIO use case? > > For >1MB messages, is alltoallv noticeably faster than a well-written > > send-recv implantation? > > > > At least on BGQ, send-recv turns into a receiver-side PAMI_Rget for > > large messages; I would guess the optimized alltoallv implementation > > is rput-based at the SPI level. Other than overhead, they should run > > at the same speed, no? If execution overhead is not significant, then > > the implementation that minimizes memory usage should be the default. > > > > I suppose I should just write alltoallv using send-recv and see what > > the difference is... > > > > Jeff > > > > On Mon, May 6, 2013 at 3:05 PM, Bob Cernohous <bobc(a)us.ibm.com> wrote: > > > > > >> From: "Rob Latham" <robl(a)mcs.anl.gov> > > >> > > >> On Mon, May 06, 2013 at 02:30:15PM -0500, Bob Cernohous wrote: > > >> > > From: Rob Ross <rross(a)mcs.anl.gov> > > >> > > > > >> > > Should we consider this as interest in working on this problem on > > >> > > the IBM side :)? -- Rob > > >> > > > >> > Say what?! ;) > > >> > > >> RobR's excited that IBM's looking at the ROMIO piece of DCMF. We > > >> thought we were on our own with that one. > > >> > > >> > > >> > I was looking more for agreement that collective i/o is 'what it > > >> > is'... and maybe some idea if we just have some known limitations on > > >> > scaling it. Yes, that BG alltoallv is a bigger problem that we can > > >> > avoid > > >> > with an env var -- is that just going to have to be 'good enough'? (I > > >> > think that Jeff P wrote that on BG/P and got good performancewith that > > >> > alltoallv. Trading memory for performance, not unusual, and at least > > >> > it's > > >> > selectable.) > > >> > > >> I can't test while our Blue Gene is under maintenance. I know the > > >> environment variable selection helps only a little bit (like improves > > >> scaling from 4k to 8k maybe? don't have the notes offhand). > > > > > > Ouch. So you've seen the scaling failures at 8k... ranks? > racks? Kevin is > > > failing at... 16 racks x 16 ranks per node... I think ... so 256k ranks. > > > > > > > > -- > > Jeff Hammond > > Argonne Leadership Computing Facility > > University of Chicago Computation Institute > > jhammond(a)alcf.anl.gov / (630) 252-5381 > > http://www.linkedin.com/in/jeffhammond > > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > > ALCF docs: http://www.alcf.anl.gov/user-guides > >

1 0