May 2012 - mpiwg-io - mailman3.cels.anl.gov

Re: [MPI3-IO] [EXTERNAL] New MPI-IO routines
by Quincey Koziol 24 May '12

24 May '12

Hi all, On May 24, 2012, at 5:13 PM, Rob Ross wrote: > Hi, > > In terms of the broader context, I believe that messages to the same destination with matching tags are already ordered: > > --- > Messages are non-overtaking: If a sender sends two messages in succession to the same destination, and both match the same receive, then this operation cannot receive the second message if the first one is still pending. If a receiver posts two receives in succession, and both match the same message, then the second receive operation cannot be satisfied by this message, if the first one is still pending. This requirement facilitates matching of sends to receives. It guarantees that message-passing code is deterministic, if processes are single-threaded and the wildcard MPI_ANY_SOURCE is not used in receives. (Some of the calls described later, such as MPI_CANCEL or MPI_WAITANY, are additional sources of nondeterminism.) > --- > > So in the point-to-point context, I think this concept is already there. The I/O routines just happen to be weaker? Been a while since I last looked at this stuff though, so I may be misremembering. Hmm, I wonder if it's worthwhile to add a "mode flag" to the I/O routines (similar to the "atomic mode" flag) that makes the I/O operations "non-overtaking" also? That would be a nice alignment of functionality for the standard. Quincey > Rob > > On May 24, 2012, at 3:46 PM, Mohamad Chaarawi wrote: > >> Hi Jay, >> >> On 5/24/2012 3:08 PM, Lofstead, Gerald F II wrote: >>> I can see the usefulness of this beyond the SWMR case (essentially queue a >>> bunch of IO ops in the background and come back when they are all done) >> >> yes that is definitely useful in many cases, especially if the file manipulation routines are nonblocking too.. >> >>> and wonder is there a broader agenda that should be looked at for this? >>> For example, is the non-blocking, but ordered idea something useful for >>> messaging as well? Personally, I think so. Given that, is it worth >>> exploring this in a broader context to add the non-blocking, ordered style >>> to MPI as a whole? >> >> I am not aware of anyone exploring this beyond I/O. I can't answer the question if it's worthwhile to do this for ALL MPI, but I don't see why it would not be possible.. Datatype creation is something that I could think of, which would benefit well from this and would go nice with the I/O version of such routines. >> >>> >>> Other than an explicit queue mechanism to execute these in series, what >>> are the complexity problems you envision? It seems really straightforward. >> >> There's that, and there is the consistency semantics issue, where the library needs to make sure that the operations are atomic. >> >> Thanks, >> Mohamad >> >>> >>> Jay Lofstead >>> >>> On 5/24/12 12:39 PM, "Mohamad Chaarawi"<chaarawi(a)hdfgroup.org> wrote: >>> >>>> Hi all, >>>> >>>> We would like to get a discussion going on adding new routines to the >>>> standard that would support non-blocking but ordered access routines, >>>> and non-blocking ordered file manipulation routines. They would be two >>>> separate tickets but they share similar semantics. >>>> >>>> So basically what we would like is that when we call: >>>> 1) iwrite_at() >>>> 2) iread_at() >>>> 3) iwrite_at() >>>> ... >>>> 4) waitall() >>>> >>>> We would like to see the data accessed as if the calls were blocking. >>>> This could translate to something like: >>>> iwrite_at() >>>> iwait() >>>> iread_at() >>>> iwait() >>>> iwrite_at() >>>> iwait() >>>> >>>> There is ofcourse no such thing as iwait, but you get the idea :-) >>>> >>>> Our use case for this is the SWMR functionality that is being worked on, >>>> where the non-blocking version would require the MPI library to remember >>>> the order in which the operations were called; otherwise I don't see how >>>> we can enforce this ordering and at the same time keep the entire I/O >>>> access nonblocking. >>>> We probably need to rename the routines (something other than the i >>>> prefix) because the semantics are different from the current >>>> non-blocking routines. Maybe aread/awrite (a for asynchronous). >>>> >>>> Another feature we are considering is the non-blocking file manipulation >>>> routines. Similarly to the semantics described above, the library needs >>>> to remember the order they are called. >>>> iopen >>>> iset_view >>>> iwrite >>>> iclose >>>> ... >>>> waitall() >>>> >>>> Again since the semantics are different in the way other immediate >>>> routines are defined, we need to reconsider the naming. >>>> >>>> Any thoughts? >>>> >>>> Thanks, >>>> Mohamad >>>> >>>> _______________________________________________ >>>> MPI3-IO mailing list >>>> MPI3-IO(a)lists.mpi-forum.org >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io >>>> >>> >>> >>> _______________________________________________ >>> MPI3-IO mailing list >>> MPI3-IO(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io >> >> _______________________________________________ >> MPI3-IO mailing list >> MPI3-IO(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io > > > _______________________________________________ > MPI3-IO mailing list > MPI3-IO(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io

1 0

Re: [MPI3-IO] [EXTERNAL] New MPI-IO routines
by Rob Ross 24 May '12

24 May '12

Hey, I'm a big fan of not adding things to the standard that may be done reasonably easily above the MPI API. In this case, there is a strong argument for doing it above MPI-IO, and that argument is that the higher-level library almost certainly can more efficiently determine if operations would conflict or not. MPI-IO library is going to have to compare extent lists or have some (currently nonexistent) method of comparing datatypes and knowing if they overlap or not without unrolling them. Rob On May 24, 2012, at 4:05 PM, Dries Kimpe wrote: > > * Mohamad Chaarawi <chaarawi(a)hdfgroup.org> [2012-05-24 15:53:04]: > >> Hi Dries, > >> On 5/24/2012 3:28 PM, Dries Kimpe wrote: >>> Also, what (if anything) would prevent implementing this on top of MPI, >>> using existing the MPI_File_iread/iwrite routines? > >> How would you do this efficiently and asynchronously? >> you need to set atomic mode to get sequential consistency (which is not >> pretty and I don't think it is well supported). Even with atomic mode >> set, how would you guarantee serial execution order? > > For each call, detect if there is a dependency on a previously issued but > not yet completed operation by checking the active range list (see below). > > - If there is none, > issue normal MPI call (MPI_File_iread/iwrite) and record bytes touched in > the file (exactly, or approximation) in the active range list. > > - If there is, don't issue the operation until the operation(s) it depends > on are completed, instead store it on an ordered queue (basically the > parameters passed to the function). > > Using a combination of generalized requests (to hand out to the > application even if the underlying operation has not yet been started), > and a progress method (polling / thread) to start blocked operations when > the dependencies are satisfied should do it easily. > > Using poll: > - test any of the underlying MPI_File request handles. > Remove completed entries from the active range list. > > - Any time an operation completes, walk the queue of not-yet-started > operations and issue any operation which no longer is blocked. > > > Basically, exactly the same the MPI implementation would have to do if > this was provided by the MPI library. > > No need to enable atomic mode... > > Dries > _______________________________________________ > MPI3-IO mailing list > MPI3-IO(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io

1 0

Re: [MPI3-IO] [EXTERNAL] New MPI-IO routines
by Mohamad Chaarawi 24 May '12

24 May '12

Hi Dries, Ok I understood your original question differently, but anyway, yes I can create a library on top of MPI to do what you mentioned. I don't see why, for this case, it is better.. A lot of the information that is needed to determine whether an operation depends on the progress of another is present in the MPI library. Doing the same thing again at a higher layer (i.e. decoding datatypes, hand out generalized requests, etc..) isn't very encouraging. On 5/24/2012 4:29 PM, Dries Kimpe wrote: > And on the other hand, if this can be done equally well on top of MPI, it > should be done on top as that is the fastest way to provide this new > functionality to the general public (without having to wait for MPI-3.1) I agree with you there.. Thanks, Mohamad

1 0

Re: [MPI3-IO] [EXTERNAL] New MPI-IO routines
by Rob Ross 24 May '12

24 May '12

Hi, In terms of the broader context, I believe that messages to the same destination with matching tags are already ordered: --- Messages are non-overtaking: If a sender sends two messages in succession to the same destination, and both match the same receive, then this operation cannot receive the second message if the first one is still pending. If a receiver posts two receives in succession, and both match the same message, then the second receive operation cannot be satisfied by this message, if the first one is still pending. This requirement facilitates matching of sends to receives. It guarantees that message-passing code is deterministic, if processes are single-threaded and the wildcard MPI_ANY_SOURCE is not used in receives. (Some of the calls described later, such as MPI_CANCEL or MPI_WAITANY, are additional sources of nondeterminism.) --- So in the point-to-point context, I think this concept is already there. The I/O routines just happen to be weaker? Been a while since I last looked at this stuff though, so I may be misremembering. Rob On May 24, 2012, at 3:46 PM, Mohamad Chaarawi wrote: > Hi Jay, > > On 5/24/2012 3:08 PM, Lofstead, Gerald F II wrote: >> I can see the usefulness of this beyond the SWMR case (essentially queue a >> bunch of IO ops in the background and come back when they are all done) > > yes that is definitely useful in many cases, especially if the file manipulation routines are nonblocking too.. > >> and wonder is there a broader agenda that should be looked at for this? >> For example, is the non-blocking, but ordered idea something useful for >> messaging as well? Personally, I think so. Given that, is it worth >> exploring this in a broader context to add the non-blocking, ordered style >> to MPI as a whole? > > I am not aware of anyone exploring this beyond I/O. I can't answer the question if it's worthwhile to do this for ALL MPI, but I don't see why it would not be possible.. Datatype creation is something that I could think of, which would benefit well from this and would go nice with the I/O version of such routines. > >> >> Other than an explicit queue mechanism to execute these in series, what >> are the complexity problems you envision? It seems really straightforward. > > There's that, and there is the consistency semantics issue, where the library needs to make sure that the operations are atomic. > > Thanks, > Mohamad > >> >> Jay Lofstead >> >> On 5/24/12 12:39 PM, "Mohamad Chaarawi"<chaarawi(a)hdfgroup.org> wrote: >> >>> Hi all, >>> >>> We would like to get a discussion going on adding new routines to the >>> standard that would support non-blocking but ordered access routines, >>> and non-blocking ordered file manipulation routines. They would be two >>> separate tickets but they share similar semantics. >>> >>> So basically what we would like is that when we call: >>> 1) iwrite_at() >>> 2) iread_at() >>> 3) iwrite_at() >>> ... >>> 4) waitall() >>> >>> We would like to see the data accessed as if the calls were blocking. >>> This could translate to something like: >>> iwrite_at() >>> iwait() >>> iread_at() >>> iwait() >>> iwrite_at() >>> iwait() >>> >>> There is ofcourse no such thing as iwait, but you get the idea :-) >>> >>> Our use case for this is the SWMR functionality that is being worked on, >>> where the non-blocking version would require the MPI library to remember >>> the order in which the operations were called; otherwise I don't see how >>> we can enforce this ordering and at the same time keep the entire I/O >>> access nonblocking. >>> We probably need to rename the routines (something other than the i >>> prefix) because the semantics are different from the current >>> non-blocking routines. Maybe aread/awrite (a for asynchronous). >>> >>> Another feature we are considering is the non-blocking file manipulation >>> routines. Similarly to the semantics described above, the library needs >>> to remember the order they are called. >>> iopen >>> iset_view >>> iwrite >>> iclose >>> ... >>> waitall() >>> >>> Again since the semantics are different in the way other immediate >>> routines are defined, we need to reconsider the naming. >>> >>> Any thoughts? >>> >>> Thanks, >>> Mohamad >>> >>> _______________________________________________ >>> MPI3-IO mailing list >>> MPI3-IO(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io >>> >> >> >> _______________________________________________ >> MPI3-IO mailing list >> MPI3-IO(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io > > _______________________________________________ > MPI3-IO mailing list > MPI3-IO(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io

1 0

Re: [MPI3-IO] [EXTERNAL] New MPI-IO routines
by Dries Kimpe 24 May '12

24 May '12

* Rob Latham <robl(a)mcs.anl.gov> [2012-05-24 16:01:12]: [...] > See, now you have a list of options, and you show them to someone like > Dries and he says either "you forgot about XYZ" or "oh, i see now, all > of those are gross" That's a bit harsh :( My opinion is simply that, if the only thing we get from putting it inside the MPI library is to avoid the progress thread/poking issue, it might be better to try to provide a solution for the progress thread instead. After all, even if it is built into the MPI library, there will very likely be no progress unless somebody is calling into MPI. And on the other hand, if this can be done equally well on top of MPI, it should be done on top as that is the fastest way to provide this new functionality to the general public (without having to wait for MPI-3.1) Dries

1 0

Re: [MPI3-IO] [EXTERNAL] New MPI-IO routines
by Lofstead, Gerald F II 24 May '12

24 May '12

This really brings up why I asked about doing it more generally. I think this is worth doing, but it should probably be done in a broader context to get much wider discussion about the implications of offering an operating mode that is ordered asynchronous. This may address issues in other areas of the MPI spec. Jay On 5/24/12 3:01 PM, "Rob Latham" <robl(a)mcs.anl.gov> wrote: > >See, now you have a list of options, and you show them to someone like >Dries and he says either "you forgot about XYZ" or "oh, i see now, all >of those are gross" > >==rob > >-- >Rob Latham >Mathematics and Computer Science Division >Argonne National Lab, IL USA >_______________________________________________ >MPI3-IO mailing list >MPI3-IO(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io >

1 0

Re: [MPI3-IO] [EXTERNAL] New MPI-IO routines
by Dries Kimpe 24 May '12

24 May '12

* Mohamad Chaarawi <chaarawi(a)hdfgroup.org> [2012-05-24 15:53:04]: > Hi Dries, > On 5/24/2012 3:28 PM, Dries Kimpe wrote: > > Also, what (if anything) would prevent implementing this on top of MPI, > > using existing the MPI_File_iread/iwrite routines? > How would you do this efficiently and asynchronously? > you need to set atomic mode to get sequential consistency (which is not > pretty and I don't think it is well supported). Even with atomic mode > set, how would you guarantee serial execution order? For each call, detect if there is a dependency on a previously issued but not yet completed operation by checking the active range list (see below). - If there is none, issue normal MPI call (MPI_File_iread/iwrite) and record bytes touched in the file (exactly, or approximation) in the active range list. - If there is, don't issue the operation until the operation(s) it depends on are completed, instead store it on an ordered queue (basically the parameters passed to the function). Using a combination of generalized requests (to hand out to the application even if the underlying operation has not yet been started), and a progress method (polling / thread) to start blocked operations when the dependencies are satisfied should do it easily. Using poll: - test any of the underlying MPI_File request handles. Remove completed entries from the active range list. - Any time an operation completes, walk the queue of not-yet-started operations and issue any operation which no longer is blocked. Basically, exactly the same the MPI implementation would have to do if this was provided by the MPI library. No need to enable atomic mode... Dries

1 0

Re: [MPI3-IO] [EXTERNAL] New MPI-IO routines
by Rob Latham 24 May '12

24 May '12

On Thu, May 24, 2012 at 03:53:04PM -0500, Mohamad Chaarawi wrote: > Hi Dries, > > On 5/24/2012 3:28 PM, Dries Kimpe wrote: > >Also, what (if anything) would prevent implementing this on top of MPI, > >using existing the MPI_File_iread/iwrite routines? > > How would you do this efficiently and asynchronously? > you need to set atomic mode to get sequential consistency (which is > not pretty and I don't think it is well supported). Even with atomic > mode set, how would you guarantee serial execution order? Dries is asking how you could build a library on top of MPI_File_iread/iwrite to provide these additional semantics. Yes, iread/iwrite are independent and so offer no chance to coordinate. - Jay's thread-like event loop requiring periodic poking - If you set up an RMA window to buffer I/O, you run into trouble if the host processor goes off into a computational loop and the MPI library does not make background RMA progress. Said otherwise, this one also requires periodic poking - If you use something like zookeeper to hand out offsets, you have the problem of setting up another process or set of processes. - If you go back in time and run on an Intel Paragon you can use that file systems native shared file pointer support to get the same effect. See, now you have a list of options, and you show them to someone like Dries and he says either "you forgot about XYZ" or "oh, i see now, all of those are gross" ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [MPI3-IO] [EXTERNAL] New MPI-IO routines
by Lofstead, Gerald F II 24 May '12

24 May '12

You could also have a separate thread-like loop using a state machine that checks for each previous op to complete before proceeding to the next one. You'd have to call it explicitly at various points to force progress or put it in a literal thread. It is a bit ugly, but it can work. I have to do something similar, but in a different context currently. Jay On 5/24/12 2:53 PM, "Mohamad Chaarawi" <chaarawi(a)hdfgroup.org> wrote: >Hi Dries, > >On 5/24/2012 3:28 PM, Dries Kimpe wrote: >> Also, what (if anything) would prevent implementing this on top of MPI, >> using existing the MPI_File_iread/iwrite routines? > >How would you do this efficiently and asynchronously? >you need to set atomic mode to get sequential consistency (which is not >pretty and I don't think it is well supported). Even with atomic mode >set, how would you guarantee serial execution order? > >Thanks, >Mohamad >_______________________________________________ >MPI3-IO mailing list >MPI3-IO(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io >

1 0

Re: [MPI3-IO] [EXTERNAL] New MPI-IO routines
by Mohamad Chaarawi 24 May '12

24 May '12

Hi Dries, On 5/24/2012 3:28 PM, Dries Kimpe wrote: > Also, what (if anything) would prevent implementing this on top of MPI, > using existing the MPI_File_iread/iwrite routines? How would you do this efficiently and asynchronously? you need to set atomic mode to get sequential consistency (which is not pretty and I don't think it is well supported). Even with atomic mode set, how would you guarantee serial execution order? Thanks, Mohamad

1 0