February 2012 - mpiwg-io - mailman3.cels.anl.gov

Re: [MPI3-IO] shared file pointer
by Dries Kimpe 01 Feb '12

01 Feb '12

* Torsten Hoefler <htor(a)illinois.edu> [2012-02-01 15:18:41]: > > Because of the MPI_Iscan properties, a second MPI_File_iread_ordered could > > not proceed before the first one finished the scan phase, hence they would > > be properly ordered. > Well, they *can* complete out of order (not match though!) you need to > make sure the order it preserved. Good point. For this to work, we need an atomic Iscan + update fp. In other words, we also have to order FP update operations. But that is pretty easy to do, by using an attribute on the file handle. MPI_File_iread_ordered: lock -> get sequence number from file handle attribute, store number with operation -> add our number at the end of a list in attribute on file handle unlock start and wait (non-blocking) for iscan if iscan completes: wait until the number at the front of the list (in attribute is ours). if it is: update shared fp and remove from list else wait. do ifile_read_at. Obviously, there are better ways of doing this. The main point is that we need to introduce an extra dependency on the update for the shared fp so that we wait until any earlier operation already updated the shared fp. I guess the same technique (and attribute) could be used for the independent shared fp operations, but I'm really no fan of that. If only because when we do that, the independent shared fp update really becomes collective, as it will block until some other MPI rank completes the collective shared fp update. I would prefer to leave that as undefined ordering, because to me it really feels like doing two non-blocking receives using the same target buffer. Dries

1 0

Re: [MPI3-IO] shared file pointer
by Torsten Hoefler 01 Feb '12

01 Feb '12

On Wed, Feb 01, 2012 at 03:16:30PM -0600, Dries Kimpe wrote: > > * Torsten Hoefler <htor(a)illinois.edu> [2012-02-01 15:04:41]: > > > > Torsten? (CC'd) > > There are obviously no filehandles in NBC. The scan operations match in > > issue-order and the buffers specified have to be disjunct (as for any > > nonblocking communication in MPI). I.e., the two NBCs do not influence > > each other from a correctness perspective. > > Meaning that the ordering issue for non-blocking collectives (but not > mixed case) is straigtforward, right? yes > Implementation of MPI_File_Iread_ordered: > MPI_Iscan (amount of local data) > <make progress> > When MPI_Iscan completes: > update shared file pointer <atomic: one rank can do this> > MPI_File_iread_at Not really --- one would have to "collect" all other (noncollective) operations that update the shared FP and make sure they update it *after* the scan completed. > As long as it is possible to adjust and get the shared file pointer > (which is what I'll be proposing in the ticket I've talked about to you > guys) it is possible to do this and not have 'undefined' behaviour. > > Also note that currently, ROMIO already has this functionality. > (As this is how MPI_File_read_shared is implemented). > > Because of the MPI_Iscan properties, a second MPI_File_iread_ordered could > not proceed before the first one finished the scan phase, hence they would > be properly ordered. Well, they *can* complete out of order (not match though!) you need to make sure the order it preserved. > Now, for mixing in indepentent shared FP operations, while not undefined, > the ordering would not be determined and will depend on what happens on > other nodes and other things. > > As in: > > (1) MPI_File_iread_ordered > ... > (2) MPI_File_read_shared > > (3) MPI_Wait () > > If (1) can complete the scan before the call to (2), (1) will appear to > logically execute before (2). In all other cases, (1) will logically > execute before (2). > > Dries -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [MPI3-IO] shared file pointer
by Dries Kimpe 01 Feb '12

01 Feb '12

* Torsten Hoefler <htor(a)illinois.edu> [2012-02-01 15:04:41]: > > Torsten? (CC'd) > There are obviously no filehandles in NBC. The scan operations match in > issue-order and the buffers specified have to be disjunct (as for any > nonblocking communication in MPI). I.e., the two NBCs do not influence > each other from a correctness perspective. Meaning that the ordering issue for non-blocking collectives (but not mixed case) is straigtforward, right? Implementation of MPI_File_Iread_ordered: MPI_Iscan (amount of local data) <make progress> When MPI_Iscan completes: update shared file pointer <atomic: one rank can do this> MPI_File_iread_at As long as it is possible to adjust and get the shared file pointer (which is what I'll be proposing in the ticket I've talked about to you guys) it is possible to do this and not have 'undefined' behaviour. Also note that currently, ROMIO already has this functionality. (As this is how MPI_File_read_shared is implemented). Because of the MPI_Iscan properties, a second MPI_File_iread_ordered could not proceed before the first one finished the scan phase, hence they would be properly ordered. Now, for mixing in indepentent shared FP operations, while not undefined, the ordering would not be determined and will depend on what happens on other nodes and other things. As in: (1) MPI_File_iread_ordered ... (2) MPI_File_read_shared (3) MPI_Wait () If (1) can complete the scan before the call to (2), (1) will appear to logically execute before (2). In all other cases, (1) will logically execute before (2). Dries

1 0

Re: [MPI3-IO] shared file pointer
by Torsten Hoefler 01 Feb '12

01 Feb '12

On Wed, Feb 01, 2012 at 02:48:16PM -0600, Quincey Koziol wrote: > > I would have to look up if this is the way the standard looks at it, but > > one possible way of reasoning about the non-blocking collectives is to > > consider them to be blocking from the time they are started up to the time > > they are completed by wait/test/... > > > > In this case, the example reduces to having two threads doing a > > MPI_File_read_ordered on the same file handle. > > > > It is perfectly valid to ask what happens in MPI 2.2 in this case. > > > > Clearly, for the split collectives, doing this is /not/ legal as the > > standard explicitly prohibits having more than one split collective on the > > same file handle at the same time. > > > > But really, somebody with better knowledge of the non-blocking collectives > > should answer what happens if two MPI_Iscan operations are executed on the > > same communicator. > > Torsten? (CC'd) There are obviously no filehandles in NBC. The scan operations match in issue-order and the buffers specified have to be disjunct (as for any nonblocking communication in MPI). I.e., the two NBCs do not influence each other from a correctness perspective. Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [MPI3-IO] shared file pointer
by Dries Kimpe 01 Feb '12

01 Feb '12

Agree with what you say below, except that I tend to make a distinction between: (1) undefined: -> anything can happen (with one possible example being having to read_ordered/shared calls returning data from the same fp position) (2) not fixing the ordering -> one of x possible orderings are possible, and exactly 1 of those will be chosen. I don't think we have to worry about (1). Dries * Quincey Koziol <koziol(a)hdfgroup.org> [2012-02-01 14:48:16]: > Hi Dries, > On Feb 1, 2012, at 2:39 PM, Dries Kimpe wrote: > > * Quincey Koziol <koziol(a)hdfgroup.org> [2012-02-01 13:38:32]: > > [...] > >>> In other words, the example above has -- according to the standard -- two > >>> outcomes: either the collective call logically executes before the > >>> independent, or the independent logically executes before the collective. > >> Is there a well-defined behavior in the standard for this? > > I'm not exactly sure what you're asking for, but at least for the split > > collectives, the standard literally leaves the implementor the choice of > > calling the blocking call in either the split collective start or end. > > Hence the two possible outcomes. > *bleh* > > Now, the fact that this is the case for the split collectives doesn't mean > > that this has to be the case for the non-blocking versions. > > It could also just be an oversight in the standard. > I'm leaning that way right now... :-) > >>> I'm not sure that doing 2 iread_ordered calls would be legal or -- if > >>> legal -- forced to act as if the shared file pointer was updated before > >>> the read_shared() call (or the 2nd iread_ordered for that matter). > >> Hmm, maybe nonblocking (both independent & collective) I/O on a shared file pointer doesn't make sense at all then? > > I would not go that far. It might not make sense to have _multiple_ > > outstanding non-blocking shared file pointer calls on a single MPI File > > handle, at least not if one wants to rely on a specific ordering. > > For example, it could be perfectly legal to have two outstanding > > non-blocking shared file pointer routines, while at the same time > > guaranteeing atomic shared file pointer updates but /not/ guaranteeing the > > ordering between the two. > But if they weren't ordered, the second operation could have undefined behavior, depending on when the file offset was changed. (Since it will depend on the current offset, for where it's operation gets performed) > > I can definitely think of an application scenario where this would make > > sense. > > I would have to look up if this is the way the standard looks at it, but > > one possible way of reasoning about the non-blocking collectives is to > > consider them to be blocking from the time they are started up to the time > > they are completed by wait/test/... > > In this case, the example reduces to having two threads doing a > > MPI_File_read_ordered on the same file handle. > > It is perfectly valid to ask what happens in MPI 2.2 in this case. > > Clearly, for the split collectives, doing this is /not/ legal as the > > standard explicitly prohibits having more than one split collective on the > > same file handle at the same time. > > But really, somebody with better knowledge of the non-blocking collectives > > should answer what happens if two MPI_Iscan operations are executed on the > > same communicator. > Torsten? (CC'd) > > If in this case, the standard can guarantee ordering, than it is trivial > > to do the same for MPI_File_iread_ordered. > Sure... > > Unfortunately, that still leaves the problematic case of mixing > > independent shared file pointer calls with non-blocking shared fp > > collectives. As I pointed out before, for split collectives, in MPI-2.2, > > the standard leaves what happens up to the implementation as the > > implementation is free to call the blocking call in the _begin or _end > > routine of the split collective. > > However, the behaviour is not undefined as in any case, there will be a > > logical ordering and the independent and split collective will not return > > the same data. > The behavior is definitely undefined from the application's perspective... :-/ > Quincey

1 0

Re: [MPI3-IO] shared file pointer
by Quincey Koziol 01 Feb '12

01 Feb '12

Hi Dries, On Feb 1, 2012, at 2:39 PM, Dries Kimpe wrote: > > * Quincey Koziol <koziol(a)hdfgroup.org> [2012-02-01 13:38:32]: > > [...] > >>> In other words, the example above has -- according to the standard -- two >>> outcomes: either the collective call logically executes before the >>> independent, or the independent logically executes before the collective. > >> Is there a well-defined behavior in the standard for this? > > I'm not exactly sure what you're asking for, but at least for the split > collectives, the standard literally leaves the implementor the choice of > calling the blocking call in either the split collective start or end. > > Hence the two possible outcomes. *bleh* > Now, the fact that this is the case for the split collectives doesn't mean > that this has to be the case for the non-blocking versions. > It could also just be an oversight in the standard. I'm leaning that way right now... :-) >>> I'm not sure that doing 2 iread_ordered calls would be legal or -- if >>> legal -- forced to act as if the shared file pointer was updated before >>> the read_shared() call (or the 2nd iread_ordered for that matter). > >> Hmm, maybe nonblocking (both independent & collective) I/O on a shared file pointer doesn't make sense at all then? > > I would not go that far. It might not make sense to have _multiple_ > outstanding non-blocking shared file pointer calls on a single MPI File > handle, at least not if one wants to rely on a specific ordering. > > For example, it could be perfectly legal to have two outstanding > non-blocking shared file pointer routines, while at the same time > guaranteeing atomic shared file pointer updates but /not/ guaranteeing the > ordering between the two. But if they weren't ordered, the second operation could have undefined behavior, depending on when the file offset was changed. (Since it will depend on the current offset, for where it's operation gets performed) > I can definitely think of an application scenario where this would make > sense. > > I would have to look up if this is the way the standard looks at it, but > one possible way of reasoning about the non-blocking collectives is to > consider them to be blocking from the time they are started up to the time > they are completed by wait/test/... > > In this case, the example reduces to having two threads doing a > MPI_File_read_ordered on the same file handle. > > It is perfectly valid to ask what happens in MPI 2.2 in this case. > > Clearly, for the split collectives, doing this is /not/ legal as the > standard explicitly prohibits having more than one split collective on the > same file handle at the same time. > > But really, somebody with better knowledge of the non-blocking collectives > should answer what happens if two MPI_Iscan operations are executed on the > same communicator. Torsten? (CC'd) > If in this case, the standard can guarantee ordering, than it is trivial > to do the same for MPI_File_iread_ordered. Sure... > Unfortunately, that still leaves the problematic case of mixing > independent shared file pointer calls with non-blocking shared fp > collectives. As I pointed out before, for split collectives, in MPI-2.2, > the standard leaves what happens up to the implementation as the > implementation is free to call the blocking call in the _begin or _end > routine of the split collective. > > However, the behaviour is not undefined as in any case, there will be a > logical ordering and the independent and split collective will not return > the same data. The behavior is definitely undefined from the application's perspective... :-/ Quincey

1 0

Re: [MPI3-IO] shared file pointer
by Dries Kimpe 01 Feb '12

01 Feb '12

* Quincey Koziol <koziol(a)hdfgroup.org> [2012-02-01 13:38:32]: [...] > > In other words, the example above has -- according to the standard -- two > > outcomes: either the collective call logically executes before the > > independent, or the independent logically executes before the collective. > Is there a well-defined behavior in the standard for this? I'm not exactly sure what you're asking for, but at least for the split collectives, the standard literally leaves the implementor the choice of calling the blocking call in either the split collective start or end. Hence the two possible outcomes. Now, the fact that his is the case for the split collectives doesn't mean that this has to be the case for the non-blocking versions. It could also just be an oversight in the standard. > > I'm not sure that doing 2 iread_ordered calls would be legal or -- if > > legal -- forced to act as if the shared file pointer was updated before > > the read_shared() call (or the 2nd iread_ordered for that matter). > Hmm, maybe nonblocking (both independent & collective) I/O on a shared file pointer doesn't make sense at all then? I would not go that far. It might not make sense to have _multiple_ outstanding non-blocking shared file pointer calls on a single MPI File handle, at least not if one wants to rely on a specific ordering. For example, it could be perfectly legal to have two outstanding non-blocking shared file pointer routines, while at the same time guaranteeing atomic shared file pointer updates but /not/ guaranteeing the ordering between the two. I can definitely think of an application scenario where this would make sense. I would have to look up if this is the way the standard looks at it, but one possible way of reasoning about the non-blocking collectives is to consider them to be blocking from the time they are started up to the time they are completed by wait/test/... In this case, the example reduces to having two threads doing a MPI_File_read_ordered on the same file handle. It is perfectly valid to ask what happens in MPI 2.2 in this case. Clearly, for the split collectives, doing this is /not/ legal as the standard explicitly prohibits having more than one split collective on the same file handle at the same time. But really, somebody with better knowledge of the non-blocking collectives should answer what happens if two MPI_Iscan operations are executed on the same communicator. If in this case, the standard can guarantee ordering, than it is trivial to do the same for MPI_File_iread_ordered. Unfortunately, that still leaves the problematic case of mixing independent shared file pointer calls with non-blocking shared fp collectives. As I pointed out before, for split collectives, in MPI-2.2, the standard leaves what happens up to the implementation as the implementation is free to call the blocking call in the _begin or _end routine of the split collective. However, the behaviour is not undefined as in any case, there will be a logical ordering and the independent and split collective will not return the same data. Dries

1 0

Re: [MPI3-IO] shared file pointer
by Quincey Koziol 01 Feb '12

01 Feb '12

Hi Dries, On Jan 31, 2012, at 3:50 PM, Dries Kimpe wrote: > My opinion: > > 1) Using non-blocking collectives in the implementation of the > non-blocking I/O functions is pretty straightforward. Yes it is a bit > harder (basically implementing a state machine) but that is a problem that > has been solved already in libnbc. OK, good. > 2) I'd have to take a closer look, but what do the I/O consistency rules > say here? What if the next one is write instead of read? I don't think that the read vs. write matters... > Also, in case of the split collectives, doesn't the standard state that > the call is considered complete when the end call returns? > Does it really state that the MPI_File_read_ordered_begin() needs to > update the shared file pointer? > > It says on page 422 that no collective I/O routines are permitted on a > file handle concurrently with a split collective. This still allows for > > MPI_File_read_ordered_begin() > MPI_File_read_shared () [but not read_ordered()] > MPI_File_read_ordered_end() > > However, it says that an implementation is free to implement any split > collective data access routine using the corresponding blocking collective > routine when either the begin or end call is issued. > > In other words, the example above has -- according to the standard -- two > outcomes: either the collective call logically executes before the > independent, or the independent logically executes before the collective. Is there a well-defined behavior in the standard for this? > I'm not sure that doing 2 iread_ordered calls would be legal or -- if > legal -- forced to act as if the shared file pointer was updated before > the read_shared() call (or the 2nd iread_ordered for that matter). Hmm, maybe nonblocking (both independent & collective) I/O on a shared file pointer doesn't make sense at all then? Quincey > As executing 2 non-blocking MPI_Iread_read_ordered() calls would translate > to 2 MPI_Iscan operations, what would happen using non-blocking > collectives in the following scenario? > > MPI_Iscan (comm) > MPI_Iscan (comm) > > Dries > > > * Moody, Adam T. <moody20(a)llnl.gov> [2012-01-27 10:29:32]: > >> Using MPI_Iallreduce certainly helps, and I think the progress rules of MPI would allow this to work. The downside is that you may need to track a long chain of dependencies. > >> MPI_File_iread_ordered() >> MPI_File_iread_ordered() >> MPI_File_read_shared() >> MPI_Waitall() > >> In this case, fortunately, the second iread_ordered() op does not need to wait on the iallreduce of the first to start its own iallreduce. It only needs to wait on the iallreduce of the first before it can actually start to read data. This is good, because otherwise it would force the second iread_ordered to block, which is something we should (probably must) avoid. > >> However, the read_shared needs to wait on the iallreduce for both outstanding iread_ordered ops before it can complete. Thus, MPI ends up tracking all of these dependencies. > >> Maybe that's not a big deal? Or maybe there's another way to implement this? It'd be good to have the ROMIO folks help out with this one. >> -Adam > >> ________________________________________ >> From: Mohamad Chaarawi [chaarawi(a)hdfgroup.org] >> Sent: Friday, January 27, 2012 9:25 AM >> To: Quincey Koziol >> Cc: Moody, Adam T.; Dries Kimpe >> Subject: Re: shared file pointer > >> Hi All, > >>>> As we discussed, I think our current non-blocking IO proposal may compound the problem of shared file pointers. Consider that you have a program like the following: > >>>> MPI_File_read_ordered_begin() >>>> MPI_File_read_shared() >>>> MPI_File_read_ordered_end() > >>>> Even though the collective may not return data until the end call, it does update the shared file pointer so that the individual read call in the middle uses the updated pointer value. The standard currently allows the begin call to be blocking (see MPI 2.2, page 422 lines 18-23). Given this, a scalable way to update the shared file pointer in the begin call is to execute an MPI_Allreduce() to update the pointer after tallying up the amount from each process. > >>>> However, with non-blocking collectives, we don't currently allow the initiation calls to be blocking. If we follow that convention, then MPI_File_iread_ordered() cannot block, which means we can't update the pointer using the Allreduce method like we can in the split versions. I think we should hash this out to be sure that it's possible to update the pointer in a scalable way. Or perhaps we need to say that MPI_File_iread_ordered() can block, but I don't think this would go over well. >>> Maybe there's a possible "out": could MPI_File_iread_ordered() be collective, but non synchronizing/blocking and use a nonblocking Allreduce operation? That way the implementation could record (internally) that the Allreduce was pending and wait for it to complete before the MPI_File_read_shared() proceeded (which wouldn't necessarily force the MPI_File_iread_ordered() to complete, just the Allreduce component, so that the shared file pointer address could be updated). Would that work? > >> I'm thinking on the same lines as Quincey mentioned. A non-blocking >> all-reduce would work, where MPI would record internally that this >> allreduce needs to complete before starting this blocking read_shared. >> The other option is to allow the iread_ordered to block for the >> allreduce so that the File pointer could be adjusted before returning. > >> Thanks, >> Mohamad > > >>> I've CC'd Mohamad and Dries, to bring them into the conversation, maybe they'll have more ideas. > >>> Quincey > >

1 0

Re: [MPI3-IO] Nonblocking Collective I/O Implementation for ticket #273
by Edgar Gabriel 31 Jan '12

31 Jan '12

On 1/31/2012 4:47 PM, Dries Kimpe wrote: > > * Mohamad Chaarawi <chaarawi(a)hdfgroup.org> [2012-01-27 15:04:41]: > > [...] > >> The reason is that the entire shared file pointer operations are not >> implemented within the MPI-I/O framework used. That being said, we can > > I'm sorry, I don't understand what you mean by the sentence above. The OMPIO framework, which has been used to implement the non-blocking collective I/O operations, does not support at the moment shared file pointer operations. Shared file pointers will be available around the May timeframe, but waiting for that would probably add an unnecessary delay for the specification. > >> provide the pseudo code for the operations, since in reality there is >> nothing fundamentally difficult if we have a shared file pointer and the >> other nonblocking routines available. > >> Would anyone on this list object to the idea of just providing the >> pseudo code implementation for the shared file pointer routines (i.e. >> postpone voting on the ticket till we have a prototype implementation >> which would likely not be in the MPI-3.0 time frame anymore)? > > Given the discussion about the non-blocking shared file pointer routines, > it is probably best to put the pseudocode to allow for discussion and make > sure we fully understand the consequences before getting a vote on the > ticket. > > While I agree that it would be possible for a user to implement collective > non-blocking shared file pointer routines, the same could be said for the > other I/O collectives, so in this case I'd prefer to provide those > routines. There is a precedence for what has been suggested in this email within the MPI standardization. The Forum agreed that an implementation of the non-blocking collective operations for inter-communicators is not necessary, since there is no fundamental, additional challenge compared to their intra-communicator counterparts. I think the same argument can be made here. There is nothing fundamentally difficult in the sequence of 1. get current position of shared file pointer on process 0 2. execute non-blocking scan operation where every process adds its own amount of data to be written/read to the current position of the shared file pointer 3. execute a non-blocking collective write_at operation with offsets as determined as a result of step 2. Step 2 has been implemented as part of the non-blocking collective operations, step 3 has been implemented as part of the non-blocking collective I/O operations. Step 1 has fundamentally been implemented a number of times. Thanks Edgar > > Dries > > > > _______________________________________________ > MPI3-IO mailing list > MPI3-IO(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io -- Edgar Gabriel Associate Professor Parallel Software Technologies Lab http://pstl.cs.uh.edu Department of Computer Science University of Houston Philip G. Hoffman Hall, Room 524 Houston, TX-77204, USA Tel: +1 (713) 743-3857 Fax: +1 (713) 743-3335

1 0