May 2013 - devel - mailman3.cels.anl.gov

Re: [mpich-devel] ROMIO collective i/o memory use
by Jeff Hammond 06 May '13

06 May '13

There are at least a few cases where BGQ collective performance is _better_ with the MPICH built-in algorithms than the PAMI_Collective implementation, which suggests to me that the assumption that the BG MPIDI collectives are always faster no longer holds. This is partly because PAMI collectives are not necessarily written for BGQ but could also be because BGQ is less sensitive to certain types of contention than previous iterations were. I'm not that "post all isend/irecv pairs and waitall" is the right way to do this is reducing memory is the goal. There are a variety of alternatives but I'd have to run experiments and compare to alltoallv to know if anything of them are better. Jeff On Mon, May 6, 2013 at 4:44 PM, Rob Latham <robl(a)mcs.anl.gov> wrote: > On Mon, May 06, 2013 at 04:35:11PM -0500, Jeff Hammond wrote: >> Do alltoallv actually run faster than send-recv for the MPIO use case? >> For >1MB messages, is alltoallv noticeably faster than a well-written >> send-recv implantation? > > We only have data from Blue Gene /L: Hao Yu's paper only studied > scalability up to 1k mpi processes. Alltoallv got the implementation > an extra 100 MiB /sec or so. > > That paper cites almassi's ICS 2005 paper "MPI collective communication on > BlueGene/L systems", but surely there has been more recent work in > this area? > > What is a "well written send-recv implementation"? One that posts a > bunch of nonblocking sends/receives and lets the MPICH progress engine > figure it out? One that tries to schedule the send/recieves in some > way? > > ==rob > > -- > Rob Latham > Mathematics and Computer Science Division > Argonne National Lab, IL USA -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

Re: [mpich-devel] ROMIO collective i/o memory use
by Bob Cernohous 06 May '13

06 May '13

I did a couple quick searches and this goes back to bg/l, not bg/p. Jeff once investigated a problem and said ---------------- In general, for smaller blocks, MPI-IO performed better than POSIX IO. For a midplane, they are about equal. For a rack, MPI-IO is noticably slower. I am now suspecting the collective phase of MPI-IO may be taking the time (#2 above). Specifying -env BGLMPIO_TUNEGATHER=0 did not significantly change the 1 rack result. This controls using allgather (0) vs allreduce (1 - the default) to communicate start and end offsets among the nodes. **Specifying -env BGLMPIO_COMM=1 made the 1 rack result twice as slow. This controls using alltoallv (0 - the default) vs send/recv (1) to do the consolidation phase. Specifying -env BGLMPIO_TUNEBLOCKING=0 made the 1 rack result so slow that I cancelled the job. This controls whether to take psets and GPFS into account (1 - the default) or not (0). --------------- I can't find the issue where he first implemented BGLMPIO_COMM. I seem to remember it performed MUCH better on small scattered i/o than send/recv. He had tables and numbers which I can't find right now. Bob Cernohous: (T/L 553) 507-253-6093 BobC(a)us.ibm.com IBM Rochester, Building 030-2(C335), Department 61L 3605 Hwy 52 North, Rochester, MN 55901-7829 > Chaos reigns within. > Reflect, repent, and reboot. > Order shall return. devel-bounces(a)mpich.org wrote on 05/06/2013 04:35:11 PM: > From: Jeff Hammond <jhammond(a)alcf.anl.gov> > To: devel(a)mpich.org, > Cc: mpich2-dev(a)mcs.anl.gov, devel-bounces(a)mpich.org > Date: 05/06/2013 04:36 PM > Subject: Re: [mpich-devel] ROMIO collective i/o memory use > Sent by: devel-bounces(a)mpich.org > > Do alltoallv actually run faster than send-recv for the MPIO use case? > For >1MB messages, is alltoallv noticeably faster than a well-written > send-recv implantation? > > At least on BGQ, send-recv turns into a receiver-side PAMI_Rget for > large messages; I would guess the optimized alltoallv implementation > is rput-based at the SPI level. Other than overhead, they should run > at the same speed, no? If execution overhead is not significant, then > the implementation that minimizes memory usage should be the default. > > I suppose I should just write alltoallv using send-recv and see what > the difference is... > > Jeff > > On Mon, May 6, 2013 at 3:05 PM, Bob Cernohous <bobc(a)us.ibm.com> wrote: > > > >> From: "Rob Latham" <robl(a)mcs.anl.gov> > >> > >> On Mon, May 06, 2013 at 02:30:15PM -0500, Bob Cernohous wrote: > >> > > From: Rob Ross <rross(a)mcs.anl.gov> > >> > > > >> > > Should we consider this as interest in working on this problem on > >> > > the IBM side :)? -- Rob > >> > > >> > Say what?! ;) > >> > >> RobR's excited that IBM's looking at the ROMIO piece of DCMF. We > >> thought we were on our own with that one. > >> > >> > >> > I was looking more for agreement that collective i/o is 'what it > >> > is'... and maybe some idea if we just have some known limitations on > >> > scaling it. Yes, that BG alltoallv is a bigger problem that we can > >> > avoid > >> > with an env var -- is that just going to have to be 'good enough'? (I > >> > think that Jeff P wrote that on BG/P and got good performance with that > >> > alltoallv. Trading memory for performance, not unusual, and at least > >> > it's > >> > selectable.) > >> > >> I can't test while our Blue Gene is under maintenance. I know the > >> environment variable selection helps only a little bit (like improves > >> scaling from 4k to 8k maybe? don't have the notes offhand). > > > > Ouch. So you've seen the scaling failures at 8k... ranks? racks? Kevin is > > failing at... 16 racks x 16 ranks per node... I think ... so 256k ranks. > > > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > ALCF docs: http://www.alcf.anl.gov/user-guides >

1 0

Re: [mpich-devel] ROMIO collective i/o memory use
by Rob Latham 06 May '13

06 May '13

On Mon, May 06, 2013 at 04:35:11PM -0500, Jeff Hammond wrote: > Do alltoallv actually run faster than send-recv for the MPIO use case? > For >1MB messages, is alltoallv noticeably faster than a well-written > send-recv implantation? We only have data from Blue Gene /L: Hao Yu's paper only studied scalability up to 1k mpi processes. Alltoallv got the implementation an extra 100 MiB /sec or so. That paper cites almassi's ICS 2005 paper "MPI collective communication on BlueGene/L systems", but surely there has been more recent work in this area? What is a "well written send-recv implementation"? One that posts a bunch of nonblocking sends/receives and lets the MPICH progress engine figure it out? One that tries to schedule the send/recieves in some way? ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-devel] ROMIO collective i/o memory use
by Jeff Hammond 06 May '13

06 May '13

Do alltoallv actually run faster than send-recv for the MPIO use case? For >1MB messages, is alltoallv noticeably faster than a well-written send-recv implantation? At least on BGQ, send-recv turns into a receiver-side PAMI_Rget for large messages; I would guess the optimized alltoallv implementation is rput-based at the SPI level. Other than overhead, they should run at the same speed, no? If execution overhead is not significant, then the implementation that minimizes memory usage should be the default. I suppose I should just write alltoallv using send-recv and see what the difference is... Jeff On Mon, May 6, 2013 at 3:05 PM, Bob Cernohous <bobc(a)us.ibm.com> wrote: > >> From: "Rob Latham" <robl(a)mcs.anl.gov> >> >> On Mon, May 06, 2013 at 02:30:15PM -0500, Bob Cernohous wrote: >> > > From: Rob Ross <rross(a)mcs.anl.gov> >> > > >> > > Should we consider this as interest in working on this problem on >> > > the IBM side :)? -- Rob >> > >> > Say what?! ;) >> >> RobR's excited that IBM's looking at the ROMIO piece of DCMF. We >> thought we were on our own with that one. >> >> >> > I was looking more for agreement that collective i/o is 'what it >> > is'... and maybe some idea if we just have some known limitations on >> > scaling it. Yes, that BG alltoallv is a bigger problem that we can >> > avoid >> > with an env var -- is that just going to have to be 'good enough'? (I >> > think that Jeff P wrote that on BG/P and got good performance with that >> > alltoallv. Trading memory for performance, not unusual, and at least >> > it's >> > selectable.) >> >> I can't test while our Blue Gene is under maintenance. I know the >> environment variable selection helps only a little bit (like improves >> scaling from 4k to 8k maybe? don't have the notes offhand). > > Ouch. So you've seen the scaling failures at 8k... ranks? racks? Kevin is > failing at... 16 racks x 16 ranks per node... I think ... so 256k ranks. -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

Re: [mpich-devel] ROMIO collective i/o memory use
by Bob Cernohous 06 May '13

06 May '13

> From: "Rob Latham" <robl(a)mcs.anl.gov> > > On Mon, May 06, 2013 at 02:30:15PM -0500, Bob Cernohous wrote: > > > From: Rob Ross <rross(a)mcs.anl.gov> > > > > > > Should we consider this as interest in working on this problem on > > > the IBM side :)? -- Rob > > > > Say what?! ;) > > RobR's excited that IBM's looking at the ROMIO piece of DCMF. We > thought we were on our own with that one. > > > > I was looking more for agreement that collective i/o is 'what it > > is'... and maybe some idea if we just have some known limitations on > > scaling it. Yes, that BG alltoallv is a bigger problem that we can avoid > > with an env var -- is that just going to have to be 'good enough'? (I > > think that Jeff P wrote that on BG/P and got good performance with that > > alltoallv. Trading memory for performance, not unusual, and at least it's > > selectable.) > > I can't test while our Blue Gene is under maintenance. I know the > environment variable selection helps only a little bit (like improves > scaling from 4k to 8k maybe? don't have the notes offhand). Ouch. So you've seen the scaling failures at 8k... ranks? racks? Kevin is failing at... 16 racks x 16 ranks per node... I think ... so 256k ranks.

1 0

Re: [mpich-devel] ROMIO collective i/o memory use
by Rob Ross 06 May '13

06 May '13

Hi, Let's separate collective I/O the concept from collective I/O the implementation in ROMIO. Lots of things we *could* do: - switch to a different internal representation of noncontiguous regions, perhaps dramatically decreasing memory allocation when exchanging regions - switch to a different collective I/O algorithm that uses some hierarchy to reduce number of processes for which space must be allocated - don't globally coordinate, but rather coordinate more loosely to reduce communication and memory allocation requirements - … I'm sure we can come up with more ideas, some of which would be new research and some of which would be mostly engineering. All relatively major activities. Thus my question :). Rob On May 6, 2013, at 2:30 PM, Bob Cernohous wrote: > > > From: Rob Ross <rross(a)mcs.anl.gov> > > > > Should we consider this as interest in working on this problem on > > the IBM side :)? -- Rob > > Say what?! ;) > > Meaning we can get rid of all o(p) allocations? Not sure how you do internal collectives on behalf of the app/mpi-io without at least some of those. I was looking more for agreement that collective i/o is 'what it is'... and maybe some idea if we just have some known limitations on scaling it. Yes, that BG alltoallv is a bigger problem that we can avoid with an env var -- is that just going to have to be 'good enough'? (I think that Jeff P wrote that on BG/P and got good performance with that alltoallv. Trading memory for performance, not unusual, and at least it's selectable.)

1 0

Re: [mpich-devel] ROMIO collective i/o memory use
by Rob Latham 06 May '13

06 May '13

On Mon, May 06, 2013 at 02:30:15PM -0500, Bob Cernohous wrote: > > From: Rob Ross <rross(a)mcs.anl.gov> > > > > Should we consider this as interest in working on this problem on > > the IBM side :)? -- Rob > > Say what?! ;) RobR's excited that IBM's looking at the ROMIO piece of DCMF. We thought we were on our own with that one. > I was looking more for agreement that collective i/o is 'what it > is'... and maybe some idea if we just have some known limitations on > scaling it. Yes, that BG alltoallv is a bigger problem that we can avoid > with an env var -- is that just going to have to be 'good enough'? (I > think that Jeff P wrote that on BG/P and got good performance with that > alltoallv. Trading memory for performance, not unusual, and at least it's > selectable.) I can't test while our Blue Gene is under maintenance. I know the environment variable selection helps only a little bit (like improves scaling from 4k to 8k maybe? don't have the notes offhand). ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-devel] ROMIO collective i/o memory use
by Kevin Harms 06 May '13

06 May '13

sorry, haven't updated the PMR but it failed. Memory failure in PAMI. I will update the PMR today with the new results. kevin On May 6, 2013, at 1:41 PM, Bob Cernohous <bobc(a)us.ibm.com> wrote: > I agree and suggested: > --------------------- > It appears they don't have enough memory for an alltoallv exchange. Try '1'... > > * - BGMPIO_COMM - Define how data is exchanged on collective > * reads and writes. Possible values: > * - 0 - Use MPI_Alltoallv. > * - 1 - Use MPI_Isend/MPI_Irecv. > * - Default is 0. > --------------------- > > but they didn't want a work around they wanted a 'fix for o(p) allocations'. There are o(p) allocations all over collective i/o from a quick glance. Just wanted some input from the experts about scaling romio. I haven't heard if the suggestion worked. > > Bob Cernohous: (T/L 553) 507-253-6093 > > BobC(a)us.ibm.com > IBM Rochester, Building 030-2(C335), Department 61L > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > Chaos reigns within. > > Reflect, repent, and reboot. > > Order shall return. > > > devel-bounces(a)mpich.org wrote on 05/04/2013 09:43:10 PM: > > > From: "Rob Latham" <robl(a)mcs.anl.gov> > > To: devel(a)mpich.org, > > Cc: mpich2-dev(a)mcs.anl.gov > > Date: 05/04/2013 09:48 PM > > Subject: Re: [mpich-devel] ROMIO collective i/o memory use > > Sent by: devel-bounces(a)mpich.org > > > > On Mon, Apr 29, 2013 at 10:28:01AM -0500, Bob Cernohous wrote: > > > A customer (Argonne ;) is complaining about O(p) allocations in collective > > > i/o. A collective read is failing at larger scale. > > > > > > Any thoughts or comments or advice? There appears to be lots of O(p) in > > > ROMIO collective I/O. Plus a lot of (possibly large) aggregated data > > > buffers. A quick search shows > > > > The O(p) allocations are a concern, sure. For two-phase, though, the > > real problem lies in ADIOI_R_Exchange_data_alltoallv and > > ADIOI_W_Exchange_data_alltoallv . The O(p) allocations are the least > > of our worries! > > > > around line 1063 of ad_bg_rdcoll.c > > > > all_recv_buf = (char *) ADIOI_Malloc( rtail ); > > > > all_send_buf = (char *) ADIOI_Malloc( stail ); > > > > (rtail and stail are the sum of the receive and send arrays) > > > > ==rob > > > > > The common ROMIO read collective code: > > > > > > Find all "ADIOI_Malloc", Match case, Regular expression (UNIX) > > > > > > File Z:\bgq\comm\lib\dev\mpich2\src\mpi\romio\adio\common\ad_read_coll.c > > > > > > 124 38: st_offsets = (ADIO_Offset *) > > > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > > > > > > 125 39: end_offsets = (ADIO_Offset *) > > > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > > > > > > 317 44: *offset_list_ptr = (ADIO_Offset *) > > > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > > > > > > 318 41: *len_list_ptr = (ADIO_Offset *) > > > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > > > > > > 334 44: *offset_list_ptr = (ADIO_Offset *) > > > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > > > > > > 335 41: *len_list_ptr = (ADIO_Offset *) > > > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > > > > > > 436 18: ADIOI_Malloc((contig_access_count+1)*sizeof(ADIO_Offset)); > > > > > > 437 41: *len_list_ptr = (ADIO_Offset *) > > > ADIOI_Malloc((contig_access_count+1)*sizeof(ADIO_Offset)); > > > > > > 573 37: if (ntimes) read_buf = (char *) ADIOI_Malloc(coll_bufsize); > > > > > > 578 21: count = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > > > 587 25: send_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > > > 590 25: recv_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > > > 598 25: start_pos = (int *) ADIOI_Malloc(nprocs*sizeof(int)); > > > > > > 739 32: tmp_buf = (char *) ADIOI_Malloc(for_next_iter); > > > > > > 744 33: read_buf = (char *) > > > ADIOI_Malloc(for_next_iter+coll_bufsize); > > > > > > 805 9: ADIOI_Malloc((nprocs_send+nprocs_recv+1)*sizeof(MPI_Request)); > > > > > > 827 30: recv_buf = (char **) ADIOI_Malloc(nprocs * sizeof(char*)); > > > > > > 830 44: (char *) > > > ADIOI_Malloc(recv_size[i]); > > > > > > 870 31: statuses = (MPI_Status *) > > > ADIOI_Malloc((nprocs_send+nprocs_recv+1) * \ > > > > > > 988 35: curr_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > > > sizeof(unsigned)); > > > > > > 989 35: done_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > > > sizeof(unsigned)); > > > > > > 990 35: recv_buf_idx = (unsigned *) ADIOI_Malloc(nprocs * > > > sizeof(unsigned)); > > > > > > Total found: 22 > > > > > > > > > Our BG version of read collective: > > > > > > File Z:\bgq\comm\lib\dev\mpich2\src\mpi\romio\adio\ad_bg\ad_bg_rdcoll.c > > > > > > 179 40: st_offsets = (ADIO_Offset *) > > > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > > > > > > 180 40: end_offsets = (ADIO_Offset *) > > > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > > > > > > 183 43: bg_offsets0 = (ADIO_Offset *) > > > ADIOI_Malloc(2*nprocs*sizeof(ADIO_Offset)); > > > > > > 184 43: bg_offsets = (ADIO_Offset *) > > > ADIOI_Malloc(2*nprocs*sizeof(ADIO_Offset)); > > > > > > 475 37: if (ntimes) read_buf = (char *) ADIOI_Malloc(coll_bufsize); > > > > > > 480 21: count = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > > > 489 25: send_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > > > 492 25: recv_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > > > 500 25: start_pos = (int *) ADIOI_Malloc(nprocs*sizeof(int)); > > > > > > 676 32: tmp_buf = (char *) ADIOI_Malloc(for_next_iter); > > > > > > 681 33: read_buf = (char *) > > > ADIOI_Malloc(for_next_iter+coll_bufsize); > > > > > > 761 9: ADIOI_Malloc((nprocs_send+nprocs_recv+1)*sizeof(MPI_Request)); > > > > > > 783 30: recv_buf = (char **) ADIOI_Malloc(nprocs * sizeof(char*)); > > > > > > 786 44: (char *) > > > ADIOI_Malloc(recv_size[i]); > > > > > > 826 31: statuses = (MPI_Status *) > > > ADIOI_Malloc((nprocs_send+nprocs_recv+1) * \ > > > > > > 944 35: curr_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > > > sizeof(unsigned)); > > > > > > 945 35: done_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > > > sizeof(unsigned)); > > > > > > 946 35: recv_buf_idx = (unsigned *) ADIOI_Malloc(nprocs * > > > sizeof(unsigned)); > > > > > > 1058 23: rdispls = (int *) ADIOI_Malloc( nprocs * sizeof(int) ); > > > > > > 1063 29: all_recv_buf = (char *) ADIOI_Malloc( rtail ); > > > > > > 1064 26: recv_buf = (char **) ADIOI_Malloc(nprocs * sizeof(char *)); > > > > > > 1068 23: sdispls = (int *) ADIOI_Malloc( nprocs * sizeof(int) ); > > > > > > 1073 29: all_send_buf = (char *) ADIOI_Malloc( stail ); > > > > > > Total found: 23 > > > > > > > > > Bob Cernohous: (T/L 553) 507-253-6093 > > > > > > BobC(a)us.ibm.com > > > IBM Rochester, Building 030-2(C335), Department 61L > > > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > > > > > Chaos reigns within. > > > > Reflect, repent, and reboot. > > > > Order shall return. > > > > -- > > Rob Latham > > Mathematics and Computer Science Division > > Argonne National Lab, IL USA > >

1 0

Re: [mpich-devel] ROMIO collective i/o memory use
by Bob Cernohous 06 May '13

06 May '13

> From: Rob Ross <rross(a)mcs.anl.gov> > > Should we consider this as interest in working on this problem on > the IBM side :)? -- Rob Say what?! ;) Meaning we can get rid of all o(p) allocations? Not sure how you do internal collectives on behalf of the app/mpi-io without at least some of those. I was looking more for agreement that collective i/o is 'what it is'... and maybe some idea if we just have some known limitations on scaling it. Yes, that BG alltoallv is a bigger problem that we can avoid with an env var -- is that just going to have to be 'good enough'? (I think that Jeff P wrote that on BG/P and got good performance with that alltoallv. Trading memory for performance, not unusual, and at least it's selectable.)

1 0

Re: [mpich-devel] ROMIO collective i/o memory use
by Rob Ross 06 May '13

06 May '13

Should we consider this as interest in working on this problem on the IBM side :)? -- Rob On May 6, 2013, at 1:41 PM, Bob Cernohous wrote: > I agree and suggested: > --------------------- > It appears they don't have enough memory for an alltoallv exchange. Try '1'... > > * - BGMPIO_COMM - Define how data is exchanged on collective > * reads and writes. Possible values: > * - 0 - Use MPI_Alltoallv. > * - 1 - Use MPI_Isend/MPI_Irecv. > * - Default is 0. > --------------------- > > but they didn't want a work around they wanted a 'fix for o(p) allocations'. There are o(p) allocations all over collective i/o from a quick glance. Just wanted some input from the experts about scaling romio. I haven't heard if the suggestion worked. > > Bob Cernohous: (T/L 553) 507-253-6093 > > BobC(a)us.ibm.com > IBM Rochester, Building 030-2(C335), Department 61L > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > Chaos reigns within. > > Reflect, repent, and reboot. > > Order shall return. > > > devel-bounces(a)mpich.org wrote on 05/04/2013 09:43:10 PM: > > > From: "Rob Latham" <robl(a)mcs.anl.gov> > > To: devel(a)mpich.org, > > Cc: mpich2-dev(a)mcs.anl.gov > > Date: 05/04/2013 09:48 PM > > Subject: Re: [mpich-devel] ROMIO collective i/o memory use > > Sent by: devel-bounces(a)mpich.org > > > > On Mon, Apr 29, 2013 at 10:28:01AM -0500, Bob Cernohous wrote: > > > A customer (Argonne ;) is complaining about O(p) allocations in collective > > > i/o. A collective read is failing at larger scale. > > > > > > Any thoughts or comments or advice? There appears to be lots of O(p) in > > > ROMIO collective I/O. Plus a lot of (possibly large) aggregated data > > > buffers. A quick search shows > > > > The O(p) allocations are a concern, sure. For two-phase, though, the > > real problem lies in ADIOI_R_Exchange_data_alltoallv and > > ADIOI_W_Exchange_data_alltoallv . The O(p) allocations are the least > > of our worries! > > > > around line 1063 of ad_bg_rdcoll.c > > > > all_recv_buf = (char *) ADIOI_Malloc( rtail ); > > > > all_send_buf = (char *) ADIOI_Malloc( stail ); > > > > (rtail and stail are the sum of the receive and send arrays) > > > > ==rob > > > > > The common ROMIO read collective code: > > > > > > Find all "ADIOI_Malloc", Match case, Regular expression (UNIX) > > > > > > File Z:\bgq\comm\lib\dev\mpich2\src\mpi\romio\adio\common\ad_read_coll.c > > > > > > 124 38: st_offsets = (ADIO_Offset *) > > > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > > > > > > 125 39: end_offsets = (ADIO_Offset *) > > > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > > > > > > 317 44: *offset_list_ptr = (ADIO_Offset *) > > > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > > > > > > 318 41: *len_list_ptr = (ADIO_Offset *) > > > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > > > > > > 334 44: *offset_list_ptr = (ADIO_Offset *) > > > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > > > > > > 335 41: *len_list_ptr = (ADIO_Offset *) > > > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > > > > > > 436 18: ADIOI_Malloc((contig_access_count+1)*sizeof(ADIO_Offset)); > > > > > > 437 41: *len_list_ptr = (ADIO_Offset *) > > > ADIOI_Malloc((contig_access_count+1)*sizeof(ADIO_Offset)); > > > > > > 573 37: if (ntimes) read_buf = (char *) ADIOI_Malloc(coll_bufsize); > > > > > > 578 21: count = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > > > 587 25: send_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > > > 590 25: recv_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > > > 598 25: start_pos = (int *) ADIOI_Malloc(nprocs*sizeof(int)); > > > > > > 739 32: tmp_buf = (char *) ADIOI_Malloc(for_next_iter); > > > > > > 744 33: read_buf = (char *) > > > ADIOI_Malloc(for_next_iter+coll_bufsize); > > > > > > 805 9: ADIOI_Malloc((nprocs_send+nprocs_recv+1)*sizeof(MPI_Request)); > > > > > > 827 30: recv_buf = (char **) ADIOI_Malloc(nprocs * sizeof(char*)); > > > > > > 830 44: (char *) > > > ADIOI_Malloc(recv_size[i]); > > > > > > 870 31: statuses = (MPI_Status *) > > > ADIOI_Malloc((nprocs_send+nprocs_recv+1) * \ > > > > > > 988 35: curr_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > > > sizeof(unsigned)); > > > > > > 989 35: done_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > > > sizeof(unsigned)); > > > > > > 990 35: recv_buf_idx = (unsigned *) ADIOI_Malloc(nprocs * > > > sizeof(unsigned)); > > > > > > Total found: 22 > > > > > > > > > Our BG version of read collective: > > > > > > File Z:\bgq\comm\lib\dev\mpich2\src\mpi\romio\adio\ad_bg\ad_bg_rdcoll.c > > > > > > 179 40: st_offsets = (ADIO_Offset *) > > > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > > > > > > 180 40: end_offsets = (ADIO_Offset *) > > > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > > > > > > 183 43: bg_offsets0 = (ADIO_Offset *) > > > ADIOI_Malloc(2*nprocs*sizeof(ADIO_Offset)); > > > > > > 184 43: bg_offsets = (ADIO_Offset *) > > > ADIOI_Malloc(2*nprocs*sizeof(ADIO_Offset)); > > > > > > 475 37: if (ntimes) read_buf = (char *) ADIOI_Malloc(coll_bufsize); > > > > > > 480 21: count = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > > > 489 25: send_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > > > 492 25: recv_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > > > > > > 500 25: start_pos = (int *) ADIOI_Malloc(nprocs*sizeof(int)); > > > > > > 676 32: tmp_buf = (char *) ADIOI_Malloc(for_next_iter); > > > > > > 681 33: read_buf = (char *) > > > ADIOI_Malloc(for_next_iter+coll_bufsize); > > > > > > 761 9: ADIOI_Malloc((nprocs_send+nprocs_recv+1)*sizeof(MPI_Request)); > > > > > > 783 30: recv_buf = (char **) ADIOI_Malloc(nprocs * sizeof(char*)); > > > > > > 786 44: (char *) > > > ADIOI_Malloc(recv_size[i]); > > > > > > 826 31: statuses = (MPI_Status *) > > > ADIOI_Malloc((nprocs_send+nprocs_recv+1) * \ > > > > > > 944 35: curr_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > > > sizeof(unsigned)); > > > > > > 945 35: done_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > > > sizeof(unsigned)); > > > > > > 946 35: recv_buf_idx = (unsigned *) ADIOI_Malloc(nprocs * > > > sizeof(unsigned)); > > > > > > 1058 23: rdispls = (int *) ADIOI_Malloc( nprocs * sizeof(int) ); > > > > > > 1063 29: all_recv_buf = (char *) ADIOI_Malloc( rtail ); > > > > > > 1064 26: recv_buf = (char **) ADIOI_Malloc(nprocs * sizeof(char *)); > > > > > > 1068 23: sdispls = (int *) ADIOI_Malloc( nprocs * sizeof(int) ); > > > > > > 1073 29: all_send_buf = (char *) ADIOI_Malloc( stail ); > > > > > > Total found: 23 > > > > > > > > > Bob Cernohous: (T/L 553) 507-253-6093 > > > > > > BobC(a)us.ibm.com > > > IBM Rochester, Building 030-2(C335), Department 61L > > > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > > > > > Chaos reigns within. > > > > Reflect, repent, and reboot. > > > > Order shall return. > > > > -- > > Rob Latham > > Mathematics and Computer Science Division > > Argonne National Lab, IL USA > >

1 0