July 2013 - devel - mailman3.cels.anl.gov

Re: [mpich-devel] Collective i/o failure
by Jeff Hammond 25 Jul '13

25 Jul '13

MPI_Allreduce on an integer should be almost infinitely fast on BG so maybe an ifdef guard is all that is required to keep this from standing in the way of widespread acceptance of MPI-IO. Best, Jeff ----- Original Message ----- > From: "Rob Ross" <rross(a)mcs.anl.gov> > To: devel(a)mpich.org > Sent: Thursday, July 25, 2013 2:03:40 PM > Subject: Re: [mpich-devel] Collective i/o failure > > Just to reiterate the point that RobL made: adding collectives to > check for completion, etc. of other ranks adds overhead to the calls > when they are successful. This in turn makes people not use MPI-IO, > because it becomes slower, which is good for reducing bug reports, > but bad for encouraging use of standard interfaces. > > Rob > > On Jul 25, 2013, at 12:10 PM, Bob Cernohous wrote: > > > > From: "Rob Latham" <robl(a)mcs.anl.gov> > > > > > How did this single rank get a negative offset? Was there some > > > integer math that overflowed? > > > > That's for the app developer to figure out. My issue is that if > > all ranks had failed the write he probably would have started > > figuring that out a few days ago and I wouldn't have gotten > > involved :) It's the weird hw error that dragged me into this > > when the non-failing ranks entered allreduce in romio and the > > failing ranks entered allreduce in the app. > > > > Like I said : > > > > > > Just wondering if there's something I can fix here in addition > > > > to the > > > > application. > > > > Not the highest priority really. But I coincidentally just got > > another report (from ANL this time) that an app is hung with half > > the ranks in write_at_all and half the ranks in a later barrier. > > It could be something similar. I don't have enough information > > yet to know but I've suggested they look at errors from write. > > > > -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

Re: [mpich-devel] Collective i/o failure
by Rob Ross 25 Jul '13

25 Jul '13

Just to reiterate the point that RobL made: adding collectives to check for completion, etc. of other ranks adds overhead to the calls when they are successful. This in turn makes people not use MPI-IO, because it becomes slower, which is good for reducing bug reports, but bad for encouraging use of standard interfaces. Rob On Jul 25, 2013, at 12:10 PM, Bob Cernohous wrote: > > From: "Rob Latham" <robl(a)mcs.anl.gov> > > > How did this single rank get a negative offset? Was there some > > integer math that overflowed? > > That's for the app developer to figure out. My issue is that if all ranks had failed the write he probably would have started figuring that out a few days ago and I wouldn't have gotten involved :) It's the weird hw error that dragged me into this when the non-failing ranks entered allreduce in romio and the failing ranks entered allreduce in the app. > > Like I said : > > > > Just wondering if there's something I can fix here in addition to the > > > application. > > Not the highest priority really. But I coincidentally just got another report (from ANL this time) that an app is hung with half the ranks in write_at_all and half the ranks in a later barrier. It could be something similar. I don't have enough information yet to know but I've suggested they look at errors from write. >

1 0

Re: [mpich-devel] Collective i/o failure
by Bob Cernohous 25 Jul '13

25 Jul '13

> From: "Rob Latham" <robl(a)mcs.anl.gov> > How did this single rank get a negative offset? Was there some > integer math that overflowed? That's for the app developer to figure out. My issue is that if all ranks had failed the write he probably would have started figuring that out a few days ago and I wouldn't have gotten involved :) It's the weird hw error that dragged me into this when the non-failing ranks entered allreduce in romio and the failing ranks entered allreduce in the app. Like I said : > > Just wondering if there's something I can fix here in addition to the > > application. Not the highest priority really. But I coincidentally just got another report (from ANL this time) that an app is hung with half the ranks in write_at_all and half the ranks in a later barrier. It could be something similar. I don't have enough information yet to know but I've suggested they look at errors from write.

1 0

Re: [mpich-devel] Collective i/o failure
by Rob Latham 25 Jul '13

25 Jul '13

On Thu, Jul 25, 2013 at 09:16:22AM -0500, Bob Cernohous wrote: > Question. If a rank fails a collective i/o, does it make sense for all > ranks to fail that collective i/o? Or just the single rank? The problem > with just a single rank failing is the remaining ranks likely get hung in > some internal collective call. I'm wondering if having all ranks fail is > *better*? > > In write_all in particular: > > http://git.mpich.org/mpich.git/blob/HEAD:/src/mpi/romio/mpi-io/write_all.c#… > > > 89 if (file_ptr_type == ADIO_EXPLICIT_OFFSET && offset < 0) > 90 { > 91 error_code = MPIO_Err_create_code(MPI_SUCCESS, > MPIR_ERR_RECOVERABLE, > 92 myname, __LINE__, > MPI_ERR_ARG, > 93 "**iobadoffset", 0); > 94 error_code = MPIO_Err_return_file(adio_fh, error_code); > 95 goto fn_exit; > 96 } > 97 /* --END ERROR HANDLING-- */ > > We *could* allreduce the offset and fail all ranks if any rank has a > negative offset. Not sure that's the right answer. There are some places already in the I/O path where we collectively check the parameters, such as a check to ensure all processes open/create a file with the same AMODE. Adding more collectives to the I/O path might make sense, but should be done carefully. I recently added MPIO_CHECK_INFO_ALL to address an IBM-reported bug, so CHECK_OFFSET_ALL would not be without precedent. > What happened was a single rank failed, the remaining ranks entered a > (hardware-based) allreduce. How did this single rank get a negative offset? Was there some integer math that overflowed? > The failing rank happened to enter an > application (hardware-based) allreduce and you get a hardware detected > failure that no app developer is going to even try to decode and debug. So > I get the call. Without the hardware involved, it just hangs on two > independent allreduces. They might have figured that one out by looking > at where the (hundreds/thousands of) stacks were hung. Of course, they > didn't actually check the return from the write_all :) Although I'm not > sure how MPIR_ERR_RECOVERABLE it is at that point. In parallel-netcdf I chose to force the caller to check errors so that I could avoid an allreduce inside pnetcdf. That seemed like the right thing to do for pnetcdf. > Just wondering if there's something I can fix here in addition to the > application. > > Bob Cernohous: (T/L 553) 507-253-6093 > > BobC(a)us.ibm.com > IBM Rochester, Building 030-2(C335), Department 61L > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > Chaos reigns within. > > Reflect, repent, and reboot. > > Order shall return. -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Collective i/o failure
by Bob Cernohous 25 Jul '13

25 Jul '13

Question. If a rank fails a collective i/o, does it make sense for all ranks to fail that collective i/o? Or just the single rank? The problem with just a single rank failing is the remaining ranks likely get hung in some internal collective call. I'm wondering if having all ranks fail is *better*? In write_all in particular: http://git.mpich.org/mpich.git/blob/HEAD:/src/mpi/romio/mpi-io/write_all.c#… 89 if (file_ptr_type == ADIO_EXPLICIT_OFFSET && offset < 0) 90 { 91 error_code = MPIO_Err_create_code(MPI_SUCCESS, MPIR_ERR_RECOVERABLE, 92 myname, __LINE__, MPI_ERR_ARG, 93 "**iobadoffset", 0); 94 error_code = MPIO_Err_return_file(adio_fh, error_code); 95 goto fn_exit; 96 } 97 /* --END ERROR HANDLING-- */ We *could* allreduce the offset and fail all ranks if any rank has a negative offset. Not sure that's the right answer. What happened was a single rank failed, the remaining ranks entered a (hardware-based) allreduce. The failing rank happened to enter an application (hardware-based) allreduce and you get a hardware detected failure that no app developer is going to even try to decode and debug. So I get the call. Without the hardware involved, it just hangs on two independent allreduces. They might have figured that one out by looking at where the (hundreds/thousands of) stacks were hung. Of course, they didn't actually check the return from the write_all :) Although I'm not sure how MPIR_ERR_RECOVERABLE it is at that point. Just wondering if there's something I can fix here in addition to the application. Bob Cernohous: (T/L 553) 507-253-6093 BobC(a)us.ibm.com IBM Rochester, Building 030-2(C335), Department 61L 3605 Hwy 52 North, Rochester, MN 55901-7829 > Chaos reigns within. > Reflect, repent, and reboot. > Order shall return.

1 0

Re: [mpich-devel] tackling "large datatype" change
by Pavan Balaji 12 Jul '13

12 Jul '13

Not quite. I said that it works for some cases, but there are bugs. -- Pavan On 07/12/2013 10:32 AM, Jeff Hammond wrote: > Pavan indicated to me in another conversation that the internal overflow issues were fixed. You can fight with him about the correctness of that statement :-) > > Jeff > > ----- Original Message ----- >> From: "David Goodell (dgoodell)" <dgoodell(a)cisco.com> >> To: "<devel(a)mpich.org>" <devel(a)mpich.org> >> Sent: Friday, July 12, 2013 10:13:37 AM >> Subject: Re: [mpich-devel] tackling "large datatype" change >> >> Jeff, >> >> How could this possibly work correctly with broken (i.e., uses "int"s >> internally for certain calculations) datatype and communication >> engines? >> >> The hard part of large count types is not constructing the actual >> type, it's processing that type correctly throughout the entire >> stack... >> >> -Dave >> >> On Jul 12, 2013, at 9:57 AM, Jeff Hammond <jhammond(a)alcf.anl.gov> >> wrote: >> >>> Hi Rob, >>> >>> I started working on something to address this a few weeks ago. I >>> was going to write a portable (i.e. above MPI) implementation of >>> >>> MPIX_Type_contiguous_x(MPI_Count count, MPI_Datatype old_type, >>> MPI_Datatype *newtype) >>> >>> and then reimplement it efficiently in MPICH. I will also try to >>> get JeffS or DaveG to create it for OtherMPI so e.g. PETSc can use >>> it more broadly. >>> >>> Does this sound like an okay plan to you? I've not pushed the >>> latest code to Github but when I do, I'll send you the link. >>> >>> Best, >>> >>> Jeff >>> >>> >>> ----- Original Message ----- >>>> From: "Rob Latham" <robl(a)mcs.anl.gov> >>>> To: devel(a)mpich.org >>>> Sent: Friday, July 12, 2013 6:25:14 AM >>>> Subject: [mpich-devel] tackling "large datatype" change >>>> >>>> I've been trying to tackle tt #1742, #1890, and #1893 as part of >>>> some >>>> I/O work that uses large datatypes. >>>> >>>> Am I stepping on anyone's toes here? Pavan, I know i bugged you >>>> about these >>>> tickets. Hope I didn't waste my Thursday on this... >>>> >>>> Pavan told me there are two problems here, but I think there are >>>> really one solution to both: >>>> >>>> - large datatype means a datatype that describes more than 2 gigs >>>> of >>>> data. A million contigs of a million contigs, say >>>> >>>> - large count means datatypes that use MPI_Count to say how many >>>> elements they have: a contig of 3 billion MPI_BYTES, say >>>> >>>> Internally, though, there's no way to decouple those two problems. >>>> >>>> The changes are pervasive and make me really nervous. >>>> >>>> I've been sometimes pushing changes to the ticket-1742-bigio >>>> branch. >>>> >>>> Who can review datatype changes once I'm done?: >>>> >>>> ==rob >>>> >>>> -- >>>> Rob Latham >>>> Mathematics and Computer Science Division >>>> Argonne National Lab, IL USA >>>> >>> >>> -- >>> Jeff Hammond >>> Argonne Leadership Computing Facility >>> University of Chicago Computation Institute >>> jhammond(a)alcf.anl.gov / (630) 252-5381 >>> http://www.linkedin.com/in/jeffhammond >>> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond >>> ALCF docs: http://www.alcf.anl.gov/user-guides >> >> > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-devel] tackling "large datatype" change
by Rob Latham 12 Jul '13

12 Jul '13

On Fri, Jul 12, 2013 at 10:32:49AM -0500, Jeff Hammond wrote: > Pavan indicated to me in another conversation that the internal overflow issues were fixed. You can fight with him about the correctness of that statement :-) He might mean Dave's large-count branch from April. Oddly, Dave's large-count work and my work overlap in about one line. Neither branch is (yet) sufficient to pass the tests I've described. I think I'm going to want to incorporate large-count into ticket-1742-bigio but I can't get my head around 'rebase --onto' ==rob > > Jeff > > ----- Original Message ----- > > From: "David Goodell (dgoodell)" <dgoodell(a)cisco.com> > > To: "<devel(a)mpich.org>" <devel(a)mpich.org> > > Sent: Friday, July 12, 2013 10:13:37 AM > > Subject: Re: [mpich-devel] tackling "large datatype" change > > > > Jeff, > > > > How could this possibly work correctly with broken (i.e., uses "int"s > > internally for certain calculations) datatype and communication > > engines? > > > > The hard part of large count types is not constructing the actual > > type, it's processing that type correctly throughout the entire > > stack... > > > > -Dave > > > > On Jul 12, 2013, at 9:57 AM, Jeff Hammond <jhammond(a)alcf.anl.gov> > > wrote: > > > > > Hi Rob, > > > > > > I started working on something to address this a few weeks ago. I > > > was going to write a portable (i.e. above MPI) implementation of > > > > > > MPIX_Type_contiguous_x(MPI_Count count, MPI_Datatype old_type, > > > MPI_Datatype *newtype) > > > > > > and then reimplement it efficiently in MPICH. I will also try to > > > get JeffS or DaveG to create it for OtherMPI so e.g. PETSc can use > > > it more broadly. > > > > > > Does this sound like an okay plan to you? I've not pushed the > > > latest code to Github but when I do, I'll send you the link. > > > > > > Best, > > > > > > Jeff > > > > > > > > > ----- Original Message ----- > > >> From: "Rob Latham" <robl(a)mcs.anl.gov> > > >> To: devel(a)mpich.org > > >> Sent: Friday, July 12, 2013 6:25:14 AM > > >> Subject: [mpich-devel] tackling "large datatype" change > > >> > > >> I've been trying to tackle tt #1742, #1890, and #1893 as part of > > >> some > > >> I/O work that uses large datatypes. > > >> > > >> Am I stepping on anyone's toes here? Pavan, I know i bugged you > > >> about these > > >> tickets. Hope I didn't waste my Thursday on this... > > >> > > >> Pavan told me there are two problems here, but I think there are > > >> really one solution to both: > > >> > > >> - large datatype means a datatype that describes more than 2 gigs > > >> of > > >> data. A million contigs of a million contigs, say > > >> > > >> - large count means datatypes that use MPI_Count to say how many > > >> elements they have: a contig of 3 billion MPI_BYTES, say > > >> > > >> Internally, though, there's no way to decouple those two problems. > > >> > > >> The changes are pervasive and make me really nervous. > > >> > > >> I've been sometimes pushing changes to the ticket-1742-bigio > > >> branch. > > >> > > >> Who can review datatype changes once I'm done?: > > >> > > >> ==rob > > >> > > >> -- > > >> Rob Latham > > >> Mathematics and Computer Science Division > > >> Argonne National Lab, IL USA > > >> > > > > > > -- > > > Jeff Hammond > > > Argonne Leadership Computing Facility > > > University of Chicago Computation Institute > > > jhammond(a)alcf.anl.gov / (630) 252-5381 > > > http://www.linkedin.com/in/jeffhammond > > > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > > > ALCF docs: http://www.alcf.anl.gov/user-guides > > > > > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-devel] tackling "large datatype" change
by Jeff Hammond 12 Jul '13

12 Jul '13

Pavan indicated to me in another conversation that the internal overflow issues were fixed. You can fight with him about the correctness of that statement :-) Jeff ----- Original Message ----- > From: "David Goodell (dgoodell)" <dgoodell(a)cisco.com> > To: "<devel(a)mpich.org>" <devel(a)mpich.org> > Sent: Friday, July 12, 2013 10:13:37 AM > Subject: Re: [mpich-devel] tackling "large datatype" change > > Jeff, > > How could this possibly work correctly with broken (i.e., uses "int"s > internally for certain calculations) datatype and communication > engines? > > The hard part of large count types is not constructing the actual > type, it's processing that type correctly throughout the entire > stack... > > -Dave > > On Jul 12, 2013, at 9:57 AM, Jeff Hammond <jhammond(a)alcf.anl.gov> > wrote: > > > Hi Rob, > > > > I started working on something to address this a few weeks ago. I > > was going to write a portable (i.e. above MPI) implementation of > > > > MPIX_Type_contiguous_x(MPI_Count count, MPI_Datatype old_type, > > MPI_Datatype *newtype) > > > > and then reimplement it efficiently in MPICH. I will also try to > > get JeffS or DaveG to create it for OtherMPI so e.g. PETSc can use > > it more broadly. > > > > Does this sound like an okay plan to you? I've not pushed the > > latest code to Github but when I do, I'll send you the link. > > > > Best, > > > > Jeff > > > > > > ----- Original Message ----- > >> From: "Rob Latham" <robl(a)mcs.anl.gov> > >> To: devel(a)mpich.org > >> Sent: Friday, July 12, 2013 6:25:14 AM > >> Subject: [mpich-devel] tackling "large datatype" change > >> > >> I've been trying to tackle tt #1742, #1890, and #1893 as part of > >> some > >> I/O work that uses large datatypes. > >> > >> Am I stepping on anyone's toes here? Pavan, I know i bugged you > >> about these > >> tickets. Hope I didn't waste my Thursday on this... > >> > >> Pavan told me there are two problems here, but I think there are > >> really one solution to both: > >> > >> - large datatype means a datatype that describes more than 2 gigs > >> of > >> data. A million contigs of a million contigs, say > >> > >> - large count means datatypes that use MPI_Count to say how many > >> elements they have: a contig of 3 billion MPI_BYTES, say > >> > >> Internally, though, there's no way to decouple those two problems. > >> > >> The changes are pervasive and make me really nervous. > >> > >> I've been sometimes pushing changes to the ticket-1742-bigio > >> branch. > >> > >> Who can review datatype changes once I'm done?: > >> > >> ==rob > >> > >> -- > >> Rob Latham > >> Mathematics and Computer Science Division > >> Argonne National Lab, IL USA > >> > > > > -- > > Jeff Hammond > > Argonne Leadership Computing Facility > > University of Chicago Computation Institute > > jhammond(a)alcf.anl.gov / (630) 252-5381 > > http://www.linkedin.com/in/jeffhammond > > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > > ALCF docs: http://www.alcf.anl.gov/user-guides > > -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

Re: [mpich-devel] tackling "large datatype" change
by Rob Latham 12 Jul '13

12 Jul '13

On Fri, Jul 12, 2013 at 10:24:48AM -0500, Pavan Balaji wrote: > Rob, > > Sorry, I'll look into this next week. no need to apologize. I've been there -- twice. > Amount of time left is proportional to age of baby. > > The first solution works for some cases in MPICH, but there are > still many bugs. I think your ticket is to fix this part? > > The second solution is known not to work. OK, I don't think there's any way to decouple these two solutions, though. There are lots of places in the datatype/dataloop code where optimizations are applied : "oh these two contig types are next to each other. Let's combine them into a single contig type... with a count that overflows. " ==rob > -- Pavan > > On 07/12/2013 06:25 AM, Rob Latham wrote: > >I've been trying to tackle tt #1742, #1890, and #1893 as part of some > >I/O work that uses large datatypes. > > > >Am I stepping on anyone's toes here? Pavan, I know i bugged you about these > >tickets. Hope I didn't waste my Thursday on this... > > > >Pavan told me there are two problems here, but I think there are > >really one solution to both: > > > >- large datatype means a datatype that describes more than 2 gigs of > > data. A million contigs of a million contigs, say > > > >- large count means datatypes that use MPI_Count to say how many > > elements they have: a contig of 3 billion MPI_BYTES, say > > > >Internally, though, there's no way to decouple those two problems. > > > >The changes are pervasive and make me really nervous. > > > >I've been sometimes pushing changes to the ticket-1742-bigio branch. > > > >Who can review datatype changes once I'm done?: > > > >==rob > > > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-devel] tackling "large datatype" change
by Rob Latham 12 Jul '13

12 Jul '13

On Fri, Jul 12, 2013 at 03:13:37PM +0000, David Goodell (dgoodell) wrote: > Jeff, > > How could this possibly work correctly with broken (i.e., uses "int"s internally for certain calculations) datatype and communication engines? > > The hard part of large count types is not constructing the actual type, it's processing that type correctly throughout the entire stack... Exactly. And it's not "certain" calculations. Ints are used pervasively. So you have MPIX_Type_contiguous_x(2346319872, MPI_BYTE, &bigtype) How do you feed that to underlying MPI? You can make a contig of a million MPI_BYTES. Then you make a struct type of 2237 "megabyte" types and 655360 MPI_BYTE types. Now what? Oh, MPICH blows up. (patch 1: make mpich not blow up) OK, now you get a type with a size of -1948647424 and you go crazy finding all the places where "this much data" is stuffed into an int. I think that correctly summarizes my Thursday... Anyway, this is scutt work -- necesary and welcome, but not research by any means. I'd hate for anyone else to duplicate work in this area, hence my declaration of intent. I have no idea how to partition this if someone else wanted to chip away at the problem. ==rob > > -Dave > > On Jul 12, 2013, at 9:57 AM, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: > > > Hi Rob, > > > > I started working on something to address this a few weeks ago. I was going to write a portable (i.e. above MPI) implementation of > > > > MPIX_Type_contiguous_x(MPI_Count count, MPI_Datatype old_type, MPI_Datatype *newtype) > > > > and then reimplement it efficiently in MPICH. I will also try to get JeffS or DaveG to create it for OtherMPI so e.g. PETSc can use it more broadly. > > > > Does this sound like an okay plan to you? I've not pushed the latest code to Github but when I do, I'll send you the link. > > > > Best, > > > > Jeff > > > > > > ----- Original Message ----- > >> From: "Rob Latham" <robl(a)mcs.anl.gov> > >> To: devel(a)mpich.org > >> Sent: Friday, July 12, 2013 6:25:14 AM > >> Subject: [mpich-devel] tackling "large datatype" change > >> > >> I've been trying to tackle tt #1742, #1890, and #1893 as part of some > >> I/O work that uses large datatypes. > >> > >> Am I stepping on anyone's toes here? Pavan, I know i bugged you > >> about these > >> tickets. Hope I didn't waste my Thursday on this... > >> > >> Pavan told me there are two problems here, but I think there are > >> really one solution to both: > >> > >> - large datatype means a datatype that describes more than 2 gigs of > >> data. A million contigs of a million contigs, say > >> > >> - large count means datatypes that use MPI_Count to say how many > >> elements they have: a contig of 3 billion MPI_BYTES, say > >> > >> Internally, though, there's no way to decouple those two problems. > >> > >> The changes are pervasive and make me really nervous. > >> > >> I've been sometimes pushing changes to the ticket-1742-bigio branch. > >> > >> Who can review datatype changes once I'm done?: > >> > >> ==rob > >> > >> -- > >> Rob Latham > >> Mathematics and Computer Science Division > >> Argonne National Lab, IL USA > >> > > > > -- > > Jeff Hammond > > Argonne Leadership Computing Facility > > University of Chicago Computation Institute > > jhammond(a)alcf.anl.gov / (630) 252-5381 > > http://www.linkedin.com/in/jeffhammond > > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > > ALCF docs: http://www.alcf.anl.gov/user-guides > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0