- mpiwg-io - mailman3.cels.anl.gov

Re: [MPI3-IO] shared file pointer
by Dries Kimpe 02 Feb '12

02 Feb '12

* Quincey Koziol <koziol(a)hdfgroup.org> [2012-02-01 17:31:39]: [...] > > I guess the same technique (and attribute) could be used for the > > independent shared fp operations, but I'm really no fan of that. > > If only because when we do that, the independent shared fp update really > > becomes collective, as it will block until some other MPI rank completes > > the collective shared fp update. > > I would prefer to leave that as undefined ordering, because to me it > > really feels like doing two non-blocking receives using the same target > > buffer. > I'm neutral on it from a user (i.e. HDF5) perspective, since I > don't think anyone will actually use the feature. But, if someone > did, would an undefined ordering be an OK thing for them too? If > so, we can say that the order is undefined in the standard and > leave it at that... There might be some use in 'undefined ordering' as opposed to 'illegal' for those application that don't care about the ordering. Applications that do rely on the ordering can easily use the existing MPI functions to enforce ordering. So, the way I see it: 2 choices: 1) Say order is undefined in the standard (basically there's a precedent with the split collective versions). 2) Say it is illegal. The user can duplicate the file handle and easily implement their own version of what they need. (2) is easier for the implementor. Dries

1 0

Re: [MPI3-IO] Fault Tolerance & I/O Discussion
by Josh Hursey 02 Feb '12

02 Feb '12

We will meet Monday, Feb. 6 from 12-1 pm EST/New York to discuss I/O in the context of the fault tolerance proposal (or the Super Bowl if we get bored). We can use the following teleconf information: US Toll Free number: 877-801-8130 Toll number: 1-203-692-8690 Access Code: 1044056 The Run-Through Stabilization proposal can be found attached to the ticket: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/276 https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/ticket/276/FTWG-Pro… We will be primarily focusing on section 17.12 of that document. I will try to send out a reminder the day of. Thanks, Josh On Tue, Jan 31, 2012 at 5:04 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > Try the new link (I had to close the other poll due to a timezone problem): > http://www.doodle.com/yifhpi5emyyzrspa > > -- Josh > > > On Tue, Jan 31, 2012 at 4:41 PM, Mohamad Chaarawi <chaarawi(a)hdfgroup.org>wrote: > >> Hi Josh, >> >> When I click the link, it says the poll has been deleted. >> >> Thanks, >> Mohamad >> >> >> On 01/31/2012 11:52 AM, Josh Hursey wrote: >> >> (Cross posted to both the I/O and FT MPI-3 listservs) >> >> During the FT plenary session at the Jan. MPI Forum meeting there were >> some concerns about fault tolerance semantics in the I/O chapter. We did >> not have much time to fully discuss the additional semantics during the >> meeting. To make sure that we push towards a complete set of semantics >> useful for the I/O community in the next draft I would like to have a >> teleconf to discuss the I/O chapter of the FT proposal. Preferability in >> the next week and a half (so we have time to fine tune things before the >> next forum meeting). >> >> Below is a link to a doodle poll to find a good time for a teleconf. If >> you are interested in participating in this discussion, please fill this >> poll out by 2 PM Eastern on Thurs. Feb. 2 so we can set the date/time. >> http://www.doodle.com/s3hz9daeh8pn483m >> >> Thanks, >> Josh >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> >> >> _______________________________________________ >> MPI3-IO mailing [email protected]://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io >> >> >> >> _______________________________________________ >> MPI3-IO mailing list >> MPI3-IO(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io >> >> > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [MPI3-IO] shared file pointer
by Quincey Koziol 01 Feb '12

01 Feb '12

Hi Dries, On Feb 1, 2012, at 3:37 PM, Dries Kimpe wrote: > > > * Torsten Hoefler <htor(a)illinois.edu> [2012-02-01 15:18:41]: > >>> Because of the MPI_Iscan properties, a second MPI_File_iread_ordered could >>> not proceed before the first one finished the scan phase, hence they would >>> be properly ordered. >> Well, they *can* complete out of order (not match though!) you need to >> make sure the order it preserved. > > Good point. > > For this to work, we need an atomic Iscan + update fp. > In other words, we also have to order FP update operations. > But that is pretty easy to do, by using an attribute on the file handle. > > MPI_File_iread_ordered: > lock > -> get sequence number from file handle attribute, store number > with operation > -> add our number at the end of a list in attribute on file handle > unlock > > start and wait (non-blocking) for iscan > > if iscan completes: wait until the number at the front of the list > (in attribute is ours). > if it is: > update shared fp and remove from list > else > wait. > > do ifile_read_at. > > > Obviously, there are better ways of doing this. The main point is that we > need to introduce an extra dependency on the update for the shared fp so > that we wait until any earlier operation already updated the shared fp. > > I guess the same technique (and attribute) could be used for the > independent shared fp operations, but I'm really no fan of that. > > If only because when we do that, the independent shared fp update really > becomes collective, as it will block until some other MPI rank completes > the collective shared fp update. > > I would prefer to leave that as undefined ordering, because to me it > really feels like doing two non-blocking receives using the same target > buffer. I'm neutral on it from a user (i.e. HDF5) perspective, since I don't think anyone will actually use the feature. But, if someone did, would an undefined ordering be an OK thing for them too? If so, we can say that the order is undefined in the standard and leave it at that... Quincey

1 0

Re: [MPI3-IO] shared file pointer
by Dries Kimpe 01 Feb '12

01 Feb '12

* Torsten Hoefler <htor(a)illinois.edu> [2012-02-01 15:18:41]: > > Because of the MPI_Iscan properties, a second MPI_File_iread_ordered could > > not proceed before the first one finished the scan phase, hence they would > > be properly ordered. > Well, they *can* complete out of order (not match though!) you need to > make sure the order it preserved. Good point. For this to work, we need an atomic Iscan + update fp. In other words, we also have to order FP update operations. But that is pretty easy to do, by using an attribute on the file handle. MPI_File_iread_ordered: lock -> get sequence number from file handle attribute, store number with operation -> add our number at the end of a list in attribute on file handle unlock start and wait (non-blocking) for iscan if iscan completes: wait until the number at the front of the list (in attribute is ours). if it is: update shared fp and remove from list else wait. do ifile_read_at. Obviously, there are better ways of doing this. The main point is that we need to introduce an extra dependency on the update for the shared fp so that we wait until any earlier operation already updated the shared fp. I guess the same technique (and attribute) could be used for the independent shared fp operations, but I'm really no fan of that. If only because when we do that, the independent shared fp update really becomes collective, as it will block until some other MPI rank completes the collective shared fp update. I would prefer to leave that as undefined ordering, because to me it really feels like doing two non-blocking receives using the same target buffer. Dries

1 0

Re: [MPI3-IO] shared file pointer
by Torsten Hoefler 01 Feb '12

01 Feb '12

On Wed, Feb 01, 2012 at 03:16:30PM -0600, Dries Kimpe wrote: > > * Torsten Hoefler <htor(a)illinois.edu> [2012-02-01 15:04:41]: > > > > Torsten? (CC'd) > > There are obviously no filehandles in NBC. The scan operations match in > > issue-order and the buffers specified have to be disjunct (as for any > > nonblocking communication in MPI). I.e., the two NBCs do not influence > > each other from a correctness perspective. > > Meaning that the ordering issue for non-blocking collectives (but not > mixed case) is straigtforward, right? yes > Implementation of MPI_File_Iread_ordered: > MPI_Iscan (amount of local data) > <make progress> > When MPI_Iscan completes: > update shared file pointer <atomic: one rank can do this> > MPI_File_iread_at Not really --- one would have to "collect" all other (noncollective) operations that update the shared FP and make sure they update it *after* the scan completed. > As long as it is possible to adjust and get the shared file pointer > (which is what I'll be proposing in the ticket I've talked about to you > guys) it is possible to do this and not have 'undefined' behaviour. > > Also note that currently, ROMIO already has this functionality. > (As this is how MPI_File_read_shared is implemented). > > Because of the MPI_Iscan properties, a second MPI_File_iread_ordered could > not proceed before the first one finished the scan phase, hence they would > be properly ordered. Well, they *can* complete out of order (not match though!) you need to make sure the order it preserved. > Now, for mixing in indepentent shared FP operations, while not undefined, > the ordering would not be determined and will depend on what happens on > other nodes and other things. > > As in: > > (1) MPI_File_iread_ordered > ... > (2) MPI_File_read_shared > > (3) MPI_Wait () > > If (1) can complete the scan before the call to (2), (1) will appear to > logically execute before (2). In all other cases, (1) will logically > execute before (2). > > Dries -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [MPI3-IO] shared file pointer
by Dries Kimpe 01 Feb '12

01 Feb '12

* Torsten Hoefler <htor(a)illinois.edu> [2012-02-01 15:04:41]: > > Torsten? (CC'd) > There are obviously no filehandles in NBC. The scan operations match in > issue-order and the buffers specified have to be disjunct (as for any > nonblocking communication in MPI). I.e., the two NBCs do not influence > each other from a correctness perspective. Meaning that the ordering issue for non-blocking collectives (but not mixed case) is straigtforward, right? Implementation of MPI_File_Iread_ordered: MPI_Iscan (amount of local data) <make progress> When MPI_Iscan completes: update shared file pointer <atomic: one rank can do this> MPI_File_iread_at As long as it is possible to adjust and get the shared file pointer (which is what I'll be proposing in the ticket I've talked about to you guys) it is possible to do this and not have 'undefined' behaviour. Also note that currently, ROMIO already has this functionality. (As this is how MPI_File_read_shared is implemented). Because of the MPI_Iscan properties, a second MPI_File_iread_ordered could not proceed before the first one finished the scan phase, hence they would be properly ordered. Now, for mixing in indepentent shared FP operations, while not undefined, the ordering would not be determined and will depend on what happens on other nodes and other things. As in: (1) MPI_File_iread_ordered ... (2) MPI_File_read_shared (3) MPI_Wait () If (1) can complete the scan before the call to (2), (1) will appear to logically execute before (2). In all other cases, (1) will logically execute before (2). Dries

1 0

Re: [MPI3-IO] shared file pointer
by Torsten Hoefler 01 Feb '12

01 Feb '12

On Wed, Feb 01, 2012 at 02:48:16PM -0600, Quincey Koziol wrote: > > I would have to look up if this is the way the standard looks at it, but > > one possible way of reasoning about the non-blocking collectives is to > > consider them to be blocking from the time they are started up to the time > > they are completed by wait/test/... > > > > In this case, the example reduces to having two threads doing a > > MPI_File_read_ordered on the same file handle. > > > > It is perfectly valid to ask what happens in MPI 2.2 in this case. > > > > Clearly, for the split collectives, doing this is /not/ legal as the > > standard explicitly prohibits having more than one split collective on the > > same file handle at the same time. > > > > But really, somebody with better knowledge of the non-blocking collectives > > should answer what happens if two MPI_Iscan operations are executed on the > > same communicator. > > Torsten? (CC'd) There are obviously no filehandles in NBC. The scan operations match in issue-order and the buffers specified have to be disjunct (as for any nonblocking communication in MPI). I.e., the two NBCs do not influence each other from a correctness perspective. Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [MPI3-IO] shared file pointer
by Dries Kimpe 01 Feb '12

01 Feb '12

Agree with what you say below, except that I tend to make a distinction between: (1) undefined: -> anything can happen (with one possible example being having to read_ordered/shared calls returning data from the same fp position) (2) not fixing the ordering -> one of x possible orderings are possible, and exactly 1 of those will be chosen. I don't think we have to worry about (1). Dries * Quincey Koziol <koziol(a)hdfgroup.org> [2012-02-01 14:48:16]: > Hi Dries, > On Feb 1, 2012, at 2:39 PM, Dries Kimpe wrote: > > * Quincey Koziol <koziol(a)hdfgroup.org> [2012-02-01 13:38:32]: > > [...] > >>> In other words, the example above has -- according to the standard -- two > >>> outcomes: either the collective call logically executes before the > >>> independent, or the independent logically executes before the collective. > >> Is there a well-defined behavior in the standard for this? > > I'm not exactly sure what you're asking for, but at least for the split > > collectives, the standard literally leaves the implementor the choice of > > calling the blocking call in either the split collective start or end. > > Hence the two possible outcomes. > *bleh* > > Now, the fact that this is the case for the split collectives doesn't mean > > that this has to be the case for the non-blocking versions. > > It could also just be an oversight in the standard. > I'm leaning that way right now... :-) > >>> I'm not sure that doing 2 iread_ordered calls would be legal or -- if > >>> legal -- forced to act as if the shared file pointer was updated before > >>> the read_shared() call (or the 2nd iread_ordered for that matter). > >> Hmm, maybe nonblocking (both independent & collective) I/O on a shared file pointer doesn't make sense at all then? > > I would not go that far. It might not make sense to have _multiple_ > > outstanding non-blocking shared file pointer calls on a single MPI File > > handle, at least not if one wants to rely on a specific ordering. > > For example, it could be perfectly legal to have two outstanding > > non-blocking shared file pointer routines, while at the same time > > guaranteeing atomic shared file pointer updates but /not/ guaranteeing the > > ordering between the two. > But if they weren't ordered, the second operation could have undefined behavior, depending on when the file offset was changed. (Since it will depend on the current offset, for where it's operation gets performed) > > I can definitely think of an application scenario where this would make > > sense. > > I would have to look up if this is the way the standard looks at it, but > > one possible way of reasoning about the non-blocking collectives is to > > consider them to be blocking from the time they are started up to the time > > they are completed by wait/test/... > > In this case, the example reduces to having two threads doing a > > MPI_File_read_ordered on the same file handle. > > It is perfectly valid to ask what happens in MPI 2.2 in this case. > > Clearly, for the split collectives, doing this is /not/ legal as the > > standard explicitly prohibits having more than one split collective on the > > same file handle at the same time. > > But really, somebody with better knowledge of the non-blocking collectives > > should answer what happens if two MPI_Iscan operations are executed on the > > same communicator. > Torsten? (CC'd) > > If in this case, the standard can guarantee ordering, than it is trivial > > to do the same for MPI_File_iread_ordered. > Sure... > > Unfortunately, that still leaves the problematic case of mixing > > independent shared file pointer calls with non-blocking shared fp > > collectives. As I pointed out before, for split collectives, in MPI-2.2, > > the standard leaves what happens up to the implementation as the > > implementation is free to call the blocking call in the _begin or _end > > routine of the split collective. > > However, the behaviour is not undefined as in any case, there will be a > > logical ordering and the independent and split collective will not return > > the same data. > The behavior is definitely undefined from the application's perspective... :-/ > Quincey

1 0

Re: [MPI3-IO] shared file pointer
by Quincey Koziol 01 Feb '12

01 Feb '12

Hi Dries, On Feb 1, 2012, at 2:39 PM, Dries Kimpe wrote: > > * Quincey Koziol <koziol(a)hdfgroup.org> [2012-02-01 13:38:32]: > > [...] > >>> In other words, the example above has -- according to the standard -- two >>> outcomes: either the collective call logically executes before the >>> independent, or the independent logically executes before the collective. > >> Is there a well-defined behavior in the standard for this? > > I'm not exactly sure what you're asking for, but at least for the split > collectives, the standard literally leaves the implementor the choice of > calling the blocking call in either the split collective start or end. > > Hence the two possible outcomes. *bleh* > Now, the fact that this is the case for the split collectives doesn't mean > that this has to be the case for the non-blocking versions. > It could also just be an oversight in the standard. I'm leaning that way right now... :-) >>> I'm not sure that doing 2 iread_ordered calls would be legal or -- if >>> legal -- forced to act as if the shared file pointer was updated before >>> the read_shared() call (or the 2nd iread_ordered for that matter). > >> Hmm, maybe nonblocking (both independent & collective) I/O on a shared file pointer doesn't make sense at all then? > > I would not go that far. It might not make sense to have _multiple_ > outstanding non-blocking shared file pointer calls on a single MPI File > handle, at least not if one wants to rely on a specific ordering. > > For example, it could be perfectly legal to have two outstanding > non-blocking shared file pointer routines, while at the same time > guaranteeing atomic shared file pointer updates but /not/ guaranteeing the > ordering between the two. But if they weren't ordered, the second operation could have undefined behavior, depending on when the file offset was changed. (Since it will depend on the current offset, for where it's operation gets performed) > I can definitely think of an application scenario where this would make > sense. > > I would have to look up if this is the way the standard looks at it, but > one possible way of reasoning about the non-blocking collectives is to > consider them to be blocking from the time they are started up to the time > they are completed by wait/test/... > > In this case, the example reduces to having two threads doing a > MPI_File_read_ordered on the same file handle. > > It is perfectly valid to ask what happens in MPI 2.2 in this case. > > Clearly, for the split collectives, doing this is /not/ legal as the > standard explicitly prohibits having more than one split collective on the > same file handle at the same time. > > But really, somebody with better knowledge of the non-blocking collectives > should answer what happens if two MPI_Iscan operations are executed on the > same communicator. Torsten? (CC'd) > If in this case, the standard can guarantee ordering, than it is trivial > to do the same for MPI_File_iread_ordered. Sure... > Unfortunately, that still leaves the problematic case of mixing > independent shared file pointer calls with non-blocking shared fp > collectives. As I pointed out before, for split collectives, in MPI-2.2, > the standard leaves what happens up to the implementation as the > implementation is free to call the blocking call in the _begin or _end > routine of the split collective. > > However, the behaviour is not undefined as in any case, there will be a > logical ordering and the independent and split collective will not return > the same data. The behavior is definitely undefined from the application's perspective... :-/ Quincey

1 0

Re: [MPI3-IO] shared file pointer
by Dries Kimpe 01 Feb '12

01 Feb '12

* Quincey Koziol <koziol(a)hdfgroup.org> [2012-02-01 13:38:32]: [...] > > In other words, the example above has -- according to the standard -- two > > outcomes: either the collective call logically executes before the > > independent, or the independent logically executes before the collective. > Is there a well-defined behavior in the standard for this? I'm not exactly sure what you're asking for, but at least for the split collectives, the standard literally leaves the implementor the choice of calling the blocking call in either the split collective start or end. Hence the two possible outcomes. Now, the fact that his is the case for the split collectives doesn't mean that this has to be the case for the non-blocking versions. It could also just be an oversight in the standard. > > I'm not sure that doing 2 iread_ordered calls would be legal or -- if > > legal -- forced to act as if the shared file pointer was updated before > > the read_shared() call (or the 2nd iread_ordered for that matter). > Hmm, maybe nonblocking (both independent & collective) I/O on a shared file pointer doesn't make sense at all then? I would not go that far. It might not make sense to have _multiple_ outstanding non-blocking shared file pointer calls on a single MPI File handle, at least not if one wants to rely on a specific ordering. For example, it could be perfectly legal to have two outstanding non-blocking shared file pointer routines, while at the same time guaranteeing atomic shared file pointer updates but /not/ guaranteeing the ordering between the two. I can definitely think of an application scenario where this would make sense. I would have to look up if this is the way the standard looks at it, but one possible way of reasoning about the non-blocking collectives is to consider them to be blocking from the time they are started up to the time they are completed by wait/test/... In this case, the example reduces to having two threads doing a MPI_File_read_ordered on the same file handle. It is perfectly valid to ask what happens in MPI 2.2 in this case. Clearly, for the split collectives, doing this is /not/ legal as the standard explicitly prohibits having more than one split collective on the same file handle at the same time. But really, somebody with better knowledge of the non-blocking collectives should answer what happens if two MPI_Iscan operations are executed on the same communicator. If in this case, the standard can guarantee ordering, than it is trivial to do the same for MPI_File_iread_ordered. Unfortunately, that still leaves the problematic case of mixing independent shared file pointer calls with non-blocking shared fp collectives. As I pointed out before, for split collectives, in MPI-2.2, the standard leaves what happens up to the implementation as the implementation is free to call the blocking call in the _begin or _end routine of the split collective. However, the behaviour is not undefined as in any case, there will be a logical ordering and the independent and split collective will not return the same data. Dries

1 0