- mpiwg-io - mailman3.cels.anl.gov

Re: [MPI3-IO] shared file pointer
by Rob Latham 23 Feb '12

23 Feb '12

On Wed, Feb 22, 2012 at 05:25:04PM -0800, Adam T. Moody wrote: > I will say, though, given that we can now have multiple outstanding > non-blocking collective I/O calls, it would be really nice to be > able to do the following: > > MPI_File_iread_ordered() > MPI_File_iread_ordered() > MPI_File_iread_ordered() > MPI_Waitall() > > This provides a natural way for someone to read in three different > sections from a file -- just issue all the calls and sit back and > wait. However, this can only be done if the pointer is updated in > the initiation call. Now we're getting somewhere. Setting aside for a moment what the letter of MPI-2.2 says, Adam's use case is exactly in line with the spirit of shared file pointers. The I/O can happen whenever but the bookeeping -- updating the shared file pointer -- happens up front. ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [MPI3-IO] [Mpi3-ft] Fault Tolerance & I/O Discussion
by George Bosilca 22 Feb '12

22 Feb '12

The promised file. george. On Feb 23, 2012, at 00:28 , George Bosilca wrote: > Based on the discussion and opinions expressed during this call, the new FT proposal has been extended to include wording about the I/O. The model for I/O is very synchronous, providing a single function related to I/O. This function can be used to invalidate a file, preempting any subsequent operation on it. The file will then have to be closed, and is necessary reopened based on a new communicator (a smaller version of the original one). > > The current version if the FT proposal is attached to this email, and can be accessed in the working group wiki @ https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/User_Level_Failure_Mitiga…. > > We expect to have a final version of the proposal by Monday noon EST, as requested by the MPI_Forum to be able to have a first reading at the next meeting. We are looking forward to your comments. > > george. > > On Feb 14, 2012, at 21:56 , Josh Hursey wrote: > >> Just a reminder that we are going to meet at 11 am Eastern on Wednesday to continue our discussion. >> >> Thanks, >> Josh >> >> On Wed, Feb 8, 2012 at 1:24 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: >> We will meet Wednesday, Feb. 15 at 11 am Eastern to continue our discussion of fault tolerance in the I/O chapter. >> >> We can use the following teleconf information: >> US Toll Free number: 877-801-8130 >> Toll number: 1-203-692-8690 >> Access Code: 1044056 >> >> Thanks, >> Josh >> >> On Mon, Feb 6, 2012 at 2:40 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: >> We had a good discussion today. Attached are some notes that I took from the call. >> >> There were a few questions that we were discussing at the end of the call. As a result we are going to try to setup another teleconf. >> >> Below is a doodle poll to pick a date/time: >> http://www.doodle.com/zw83eht6hh8mwmic >> >> If you are interested in attending this teleconf, please fill out the poll by 2 pm Eastern on Wednesday, Feb. 8. >> >> In the mean time let us keep discussing these issues on the FT and I/O mailing lists. My preference would be to just discuss it on the FT mailing list as there are more FT folks over there, and then we would not distract the discussions about the other I/O tickets. >> >> Thanks, >> Josh >> >> On Mon, Feb 6, 2012 at 10:55 AM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: >> Just a reminder that we are meeting today at Noon Eastern. Enclosed are the call-in details and a link to the FT stabilization proposal. >> >> Thanks, >> Josh >> >> >> On Thu, Feb 2, 2012 at 2:46 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: >> We will meet Monday, Feb. 6 from 12-1 pm EST/New York to discuss I/O in the context of the fault tolerance proposal (or the Super Bowl if we get bored). >> >> We can use the following teleconf information: >> US Toll Free number: 877-801-8130 >> Toll number: 1-203-692-8690 >> Access Code: 1044056 >> >> The Run-Through Stabilization proposal can be found attached to the ticket: >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/276 >> https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/ticket/276/FTWG-Pro… >> >> We will be primarily focusing on section 17.12 of that document. I will try to send out a reminder the day of. >> >> Thanks, >> Josh >> >> On Tue, Jan 31, 2012 at 5:04 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: >> Try the new link (I had to close the other poll due to a timezone problem): >> http://www.doodle.com/yifhpi5emyyzrspa >> >> -- Josh >> >> >> On Tue, Jan 31, 2012 at 4:41 PM, Mohamad Chaarawi <chaarawi(a)hdfgroup.org> wrote: >> Hi Josh, >> >> When I click the link, it says the poll has been deleted. >> >> Thanks, >> Mohamad >> >> >> On 01/31/2012 11:52 AM, Josh Hursey wrote: >>> (Cross posted to both the I/O and FT MPI-3 listservs) >>> >>> During the FT plenary session at the Jan. MPI Forum meeting there were some concerns about fault tolerance semantics in the I/O chapter. We did not have much time to fully discuss the additional semantics during the meeting. To make sure that we push towards a complete set of semantics useful for the I/O community in the next draft I would like to have a teleconf to discuss the I/O chapter of the FT proposal. Preferability in the next week and a half (so we have time to fine tune things before the next forum meeting). >>> >>> Below is a link to a doodle poll to find a good time for a teleconf. If you are interested in participating in this discussion, please fill this poll out by 2 PM Eastern on Thurs. Feb. 2 so we can set the date/time. >>> http://www.doodle.com/s3hz9daeh8pn483m >>> >>> Thanks, >>> Josh >>> >>> -- >>> Joshua Hursey >>> Postdoctoral Research Associate >>> Oak Ridge National Laboratory >>> http://users.nccs.gov/~jjhursey >>> >>> >>> _______________________________________________ >>> MPI3-IO mailing list >>> MPI3-IO(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io >> >> >> _______________________________________________ >> MPI3-IO mailing list >> MPI3-IO(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io >> >> >> >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> >> >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> >> >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> >> >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> >> >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> >> >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> _______________________________________________ >> mpi3-ft mailing list >> mpi3-ft(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [MPI3-IO] [Mpi3-ft] Fault Tolerance & I/O Discussion
by George Bosilca 22 Feb '12

22 Feb '12

Based on the discussion and opinions expressed during this call, the new FT proposal has been extended to include wording about the I/O. The model for I/O is very synchronous, providing a single function related to I/O. This function can be used to invalidate a file, preempting any subsequent operation on it. The file will then have to be closed, and is necessary reopened based on a new communicator (a smaller version of the original one). The current version if the FT proposal is attached to this email, and can be accessed in the working group wiki @ https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/User_Level_Failure_Mitiga…. We expect to have a final version of the proposal by Monday noon EST, as requested by the MPI_Forum to be able to have a first reading at the next meeting. We are looking forward to your comments. george. On Feb 14, 2012, at 21:56 , Josh Hursey wrote: > Just a reminder that we are going to meet at 11 am Eastern on Wednesday to continue our discussion. > > Thanks, > Josh > > On Wed, Feb 8, 2012 at 1:24 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > We will meet Wednesday, Feb. 15 at 11 am Eastern to continue our discussion of fault tolerance in the I/O chapter. > > We can use the following teleconf information: > US Toll Free number: 877-801-8130 > Toll number: 1-203-692-8690 > Access Code: 1044056 > > Thanks, > Josh > > On Mon, Feb 6, 2012 at 2:40 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > We had a good discussion today. Attached are some notes that I took from the call. > > There were a few questions that we were discussing at the end of the call. As a result we are going to try to setup another teleconf. > > Below is a doodle poll to pick a date/time: > http://www.doodle.com/zw83eht6hh8mwmic > > If you are interested in attending this teleconf, please fill out the poll by 2 pm Eastern on Wednesday, Feb. 8. > > In the mean time let us keep discussing these issues on the FT and I/O mailing lists. My preference would be to just discuss it on the FT mailing list as there are more FT folks over there, and then we would not distract the discussions about the other I/O tickets. > > Thanks, > Josh > > On Mon, Feb 6, 2012 at 10:55 AM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > Just a reminder that we are meeting today at Noon Eastern. Enclosed are the call-in details and a link to the FT stabilization proposal. > > Thanks, > Josh > > > On Thu, Feb 2, 2012 at 2:46 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > We will meet Monday, Feb. 6 from 12-1 pm EST/New York to discuss I/O in the context of the fault tolerance proposal (or the Super Bowl if we get bored). > > We can use the following teleconf information: > US Toll Free number: 877-801-8130 > Toll number: 1-203-692-8690 > Access Code: 1044056 > > The Run-Through Stabilization proposal can be found attached to the ticket: > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/276 > https://svn.mpi-forum.org/trac/mpi-forum-web/attachment/ticket/276/FTWG-Pro… > > We will be primarily focusing on section 17.12 of that document. I will try to send out a reminder the day of. > > Thanks, > Josh > > On Tue, Jan 31, 2012 at 5:04 PM, Josh Hursey <jjhursey(a)open-mpi.org> wrote: > Try the new link (I had to close the other poll due to a timezone problem): > http://www.doodle.com/yifhpi5emyyzrspa > > -- Josh > > > On Tue, Jan 31, 2012 at 4:41 PM, Mohamad Chaarawi <chaarawi(a)hdfgroup.org> wrote: > Hi Josh, > > When I click the link, it says the poll has been deleted. > > Thanks, > Mohamad > > > On 01/31/2012 11:52 AM, Josh Hursey wrote: >> (Cross posted to both the I/O and FT MPI-3 listservs) >> >> During the FT plenary session at the Jan. MPI Forum meeting there were some concerns about fault tolerance semantics in the I/O chapter. We did not have much time to fully discuss the additional semantics during the meeting. To make sure that we push towards a complete set of semantics useful for the I/O community in the next draft I would like to have a teleconf to discuss the I/O chapter of the FT proposal. Preferability in the next week and a half (so we have time to fine tune things before the next forum meeting). >> >> Below is a link to a doodle poll to find a good time for a teleconf. If you are interested in participating in this discussion, please fill this poll out by 2 PM Eastern on Thurs. Feb. 2 so we can set the date/time. >> http://www.doodle.com/s3hz9daeh8pn483m >> >> Thanks, >> Josh >> >> -- >> Joshua Hursey >> Postdoctoral Research Associate >> Oak Ridge National Laboratory >> http://users.nccs.gov/~jjhursey >> >> >> _______________________________________________ >> MPI3-IO mailing list >> MPI3-IO(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io > > > _______________________________________________ > MPI3-IO mailing list > MPI3-IO(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io > > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft

1 0

Re: [MPI3-IO] shared file pointer
by Adam T. Moody 22 Feb '12

22 Feb '12

I will say, though, given that we can now have multiple outstanding non-blocking collective I/O calls, it would be really nice to be able to do the following: MPI_File_iread_ordered() MPI_File_iread_ordered() MPI_File_iread_ordered() MPI_Waitall() This provides a natural way for someone to read in three different sections from a file -- just issue all the calls and sit back and wait. However, this can only be done if the pointer is updated in the initiation call. -Adam Adam T. Moody wrote: >Hi Mohamad and Dries, >Yes, I see your point now about "using the corresponding blocking >collective routines when ... the end call is issued". I don't think >that's what the standard intended, but you're very right in that the >text says two different things. Some statements say the pointer is >updated by the call that initiates the operation, i.e., the _begin call, >but this says the opposite in that an implementation is allowed to do >all the work (including updating of the pointer) in the _end call. >Thus, it's not clear whether the pointer will always be updated after >returning from the _begin call. >-Adam > > >Mohamad Chaarawi wrote: > > > >>Hi Adam, >> >> >> >> >> >>>This statement says that an app can't know whether the begin call will >>>synchronize or not, so a portable app must assume that the call does >>>synchronize. However, the earlier statements say that regardless of >>>whether the MPI library implements the begin call as blocking or >>>non-blocking, the app is always guaranteed that the shared file >>>pointer will be updated upon return from the begin call. >>> >>> >>> >>> >>Yes but I agree with Dries that there is a contradiction, and it can be >>interpreted by a developer either way, i.e. the pointer can be either >>updated in the begin or end call.. >> >> >> >> >> >>>With split collectives, the "begin" call that initiates the operation >>>*can* block, but with non-blocking collectives (as currently defined), >>>the "i" call that initiates the operation *never* blocks. It's this >>>difference between split collectives and non-blocking collectives that >>>causes the difficulty here. To efficiently meet the requirements of >>>updating the shared file pointer, we'd really like to update the >>>pointer during the "i" call, but this would require the "i" call to >>>block. >>> >>> >>> >>> >>I do not have a strong opinion here, as we don't really use this >>feature.. But I can see how this could complicate things more to the >>user and the developer, which makes me more inclined to keep the >>ordering undefined. >>That said, we do want to start working on a ticket for new MPI-I/O >>features that would actually track order inside the implementation for >>nonblocking file access and manipulation routines (more like queuing).. >>We discussed that at the last Chicago meeting.. This is not MPI-3.0 >>bound though :) >> >>Thanks, >>Mohamad >> >> >>_______________________________________________ >>MPI3-IO mailing list >>MPI3-IO(a)lists.mpi-forum.org >>http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io >> >> >> >> > > > >

1 0

Re: [MPI3-IO] shared file pointer
by Adam T. Moody 22 Feb '12

22 Feb '12

Hi Mohamad and Dries, Yes, I see your point now about "using the corresponding blocking collective routines when ... the end call is issued". I don't think that's what the standard intended, but you're very right in that the text says two different things. Some statements say the pointer is updated by the call that initiates the operation, i.e., the _begin call, but this says the opposite in that an implementation is allowed to do all the work (including updating of the pointer) in the _end call. Thus, it's not clear whether the pointer will always be updated after returning from the _begin call. -Adam Mohamad Chaarawi wrote: >Hi Adam, > > > >>This statement says that an app can't know whether the begin call will >>synchronize or not, so a portable app must assume that the call does >>synchronize. However, the earlier statements say that regardless of >>whether the MPI library implements the begin call as blocking or >>non-blocking, the app is always guaranteed that the shared file >>pointer will be updated upon return from the begin call. >> >> > >Yes but I agree with Dries that there is a contradiction, and it can be >interpreted by a developer either way, i.e. the pointer can be either >updated in the begin or end call.. > > > >>With split collectives, the "begin" call that initiates the operation >>*can* block, but with non-blocking collectives (as currently defined), >>the "i" call that initiates the operation *never* blocks. It's this >>difference between split collectives and non-blocking collectives that >>causes the difficulty here. To efficiently meet the requirements of >>updating the shared file pointer, we'd really like to update the >>pointer during the "i" call, but this would require the "i" call to >>block. >> >> > >I do not have a strong opinion here, as we don't really use this >feature.. But I can see how this could complicate things more to the >user and the developer, which makes me more inclined to keep the >ordering undefined. >That said, we do want to start working on a ticket for new MPI-I/O >features that would actually track order inside the implementation for >nonblocking file access and manipulation routines (more like queuing).. >We discussed that at the last Chicago meeting.. This is not MPI-3.0 >bound though :) > >Thanks, >Mohamad > > >_______________________________________________ >MPI3-IO mailing list >MPI3-IO(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io > >

1 0

Re: [MPI3-IO] shared file pointer
by Mohamad Chaarawi 17 Feb '12

17 Feb '12

Hi Adam, > > This statement says that an app can't know whether the begin call will > synchronize or not, so a portable app must assume that the call does > synchronize. However, the earlier statements say that regardless of > whether the MPI library implements the begin call as blocking or > non-blocking, the app is always guaranteed that the shared file > pointer will be updated upon return from the begin call. Yes but I agree with Dries that there is a contradiction, and it can be interpreted by a developer either way, i.e. the pointer can be either updated in the begin or end call.. > > With split collectives, the "begin" call that initiates the operation > *can* block, but with non-blocking collectives (as currently defined), > the "i" call that initiates the operation *never* blocks. It's this > difference between split collectives and non-blocking collectives that > causes the difficulty here. To efficiently meet the requirements of > updating the shared file pointer, we'd really like to update the > pointer during the "i" call, but this would require the "i" call to > block. I do not have a strong opinion here, as we don't really use this feature.. But I can see how this could complicate things more to the user and the developer, which makes me more inclined to keep the ordering undefined. That said, we do want to start working on a ticket for new MPI-I/O features that would actually track order inside the implementation for nonblocking file access and manipulation routines (more like queuing).. We discussed that at the last Chicago meeting.. This is not MPI-3.0 bound though :) Thanks, Mohamad

1 0

Re: [MPI3-IO] shared file pointer
by Dries Kimpe 16 Feb '12

16 Feb '12

* Adam T. Moody <moody20(a)llnl.gov> [2012-02-16 14:08:43]: > >Contrast that with (page 422 in MPI-22 document): > >An implementation is free to implement any split collective data access > >routine using the corresponding blocking collective routine when either > >the begin call (e.g., MPI_FILE_READ_ALL_BEGIN) or the end call (e.g., > >MPI_FILE_READ_ALL_END) is issued. The begin and end calls are provided to > >allow the user and MPI implementation to optimize the collective > >operation. > >Note that I never claimed that your example was not allowed; I simply > >pointed out that, according to the text above, the end result is not > >deterministic. > >[ text remove ] > Hi Dries, > This statement says that an app can't know whether the begin call will > synchronize or not, so a portable app must assume that the call does > synchronize. However, the earlier statements say that regardless of > whether the MPI library implements the begin call as blocking or > non-blocking, the app is always guaranteed that the shared file pointer > will be updated upon return from the begin call. That's not how I read it. If an implementation is free to implement the split collective (i.e. read_ordered_begin) /USING THE CORRESPONDING BLOCKING COLLECTIVE ROUTINE/ (meaning read_ordered in this case) when the _end call is issued, this effectively means that the shared file pointer is *NOT* updated when the begin call is issued. There is conflicting text in the standard, but I don't see any reason why one interpretation has to be preferred over another. That being said, it doesn't really matter what there is in the standard right now. We're looking at changing the standard anyhow, and part of this change should be to resolve this conflict. > With split collectives, the "begin" call that initiates the operation > *can* block, but with non-blocking collectives (as currently defined), > the "i" call that initiates the operation *never* blocks. It's this > difference between split collectives and non-blocking collectives that > causes the difficulty here. To efficiently meet the requirements of > updating the shared file pointer, we'd really like to update the pointer > during the "i" call, but this would require the "i" call to block. > -Adam I disagree that updating the shared file pointer during the "i" call requires it to block. You have to be careful when using the term 'block'. In MPI terms, blocking means that when the operation returns, the 'resources' specified to the function can be reused (i.e. memory buffers). (See 2.4) You probably mean local vs non-local (assuming that cooperation of another rank is required to update the shared file pointer) -- which itself is *not* the same as collective vs non-collective. In any case, I don't see a technical reason -- even using your interpretation of the standard -- why the "i" call would have be non-local or blocking for that matter. So, it is perfectly possible to make MPI_File_read_shared non-local. A quick search didn't show any information about this call being local or non-local, so, even according to MPI-22, it *is* legal for MPI_File_read_shared to depend on some other rank calling MPI_File_iread_ordered (or similar). The standard does not require that for non-local calls, the dependent call needs to be the same (i.e. MPI_Send and MPI_Recv for example). Now, all the standard word play aside, and getting into my personal opinion as a user (and developer) of MPI: Yes, we can specify text in the standard saying that MPI_File_read_shared is non-local and depends on the completion (partial or not) of an already-started non-blocking or split-collective shared file pointer call. I simply feel it is: A) Unnecessary (since it is possible to open the file twice and avoid all overhead and complication of forcing the implementation to track this) B) Unnatural and confusing to the end user. C) Very unlikely to ever be an issue in the real world. I hope I clarified my position and arguments, so now I'll stay out of this discussion and leave it up to the wise people that actually attend the forum meetings to decide on this matter. I won't cry, no matter the outcome of this debate. :-) Dries

1 0

Re: [MPI3-IO] shared file pointer
by Adam T. Moody 16 Feb '12

16 Feb '12

Dries Kimpe wrote: >* Adam T. Moody <moody20(a)llnl.gov> [2012-02-16 12:43:35]: > > > >>Hi guys, >>My interpretation of the standard is that the following use of split >>collectives is ok: >> >> > > > >>MPI_File_read_ordered_begin() >>MPI_File_read_shared() >>MPI_File_read_ordered_end() >> >> > > > >>This understanding comes from the following: >> >> > > > >>p405, 17-9 >>"In general, each I/O operation leaves the file pointer pointing to the >>next data itme after the last one that is accessed by the operation. In >>a nonblocking or split collective operation, the pointer is updated by >>the call that initiates the I/O, possibly before the access completes." >> >> > > > >>and: >> >> > > > >>p417, 1-2 >>"After a shared file pointer operation is initiated, the shared file >>pointer is updated to point to the next etype after the last one that >>will be accessed." >> >> > >Contrast that with (page 422 in MPI-22 document): > >An implementation is free to implement any split collective data access >routine using the corresponding blocking collective routine when either >the begin call (e.g., MPI_FILE_READ_ALL_BEGIN) or the end call (e.g., >MPI_FILE_READ_ALL_END) is issued. The begin and end calls are provided to >allow the user and MPI implementation to optimize the collective >operation. > >Note that I never claimed that your example was not allowed; I simply >pointed out that, according to the text above, the end result is not >deterministic. > >[ text remove ] > > Hi Dries, This statement says that an app can't know whether the begin call will synchronize or not, so a portable app must assume that the call does synchronize. However, the earlier statements say that regardless of whether the MPI library implements the begin call as blocking or non-blocking, the app is always guaranteed that the shared file pointer will be updated upon return from the begin call. With split collectives, the "begin" call that initiates the operation *can* block, but with non-blocking collectives (as currently defined), the "i" call that initiates the operation *never* blocks. It's this difference between split collectives and non-blocking collectives that causes the difficulty here. To efficiently meet the requirements of updating the shared file pointer, we'd really like to update the pointer during the "i" call, but this would require the "i" call to block. -Adam > > >>However, to do this, the read_shared() call would need to block until >>all procs have called iread_ordered(). This does pose two difficulties: >> 1) It requires MPI to track any outstanding non-blocking collective >>calls so that it knows to wait on them. >> >> > >That is pretty easy. An attribute on the file handle can be used for that. > > > >> 2) It forces read_shared() into this limbo between being local / not >>local. It's supposed to be a local call, but in this context, it can't >>return until all other procs have called the collectives that come >>before it. So is it still "local"? >> >> > >This is pretty major if you ask me. > >Especially since an easy solution to avoid this problem is opening the >file twice (so you effectively get two shared file pointers). > >I'm wondering if there are any other functions already in this position? >The best I can come up with is MPI_Recv (and friends) and MPI_Send, >since technically those can block until the other side posts a matching >operation, but that is easily motivated in the standard. > >The text you quoted is interesting though, because at least it makes it >very clear that the shared file pointer is updated first, possibly before >the data is accessed. > >This clarifies what happens in a fault situation (read error or >otherwise): the shared file pointer is moved to point to the first byte >after the region that is *requested*, not after the region that is >actually successfully accessed. > >So, if the call fails to ready any data, the shared file pointer can still >point to the position it would have if the call succeeded. > > Dries > > >------------------------------------------------------------------------ > >_______________________________________________ >MPI3-IO mailing list >MPI3-IO(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io > >

1 0

Re: [MPI3-IO] shared file pointer
by Dries Kimpe 16 Feb '12

16 Feb '12

* Adam T. Moody <moody20(a)llnl.gov> [2012-02-16 12:43:35]: > Hi guys, > My interpretation of the standard is that the following use of split > collectives is ok: > MPI_File_read_ordered_begin() > MPI_File_read_shared() > MPI_File_read_ordered_end() > This understanding comes from the following: > p405, 17-9 > "In general, each I/O operation leaves the file pointer pointing to the > next data itme after the last one that is accessed by the operation. In > a nonblocking or split collective operation, the pointer is updated by > the call that initiates the I/O, possibly before the access completes." > and: > p417, 1-2 > "After a shared file pointer operation is initiated, the shared file > pointer is updated to point to the next etype after the last one that > will be accessed." Contrast that with (page 422 in MPI-22 document): An implementation is free to implement any split collective data access routine using the corresponding blocking collective routine when either the begin call (e.g., MPI_FILE_READ_ALL_BEGIN) or the end call (e.g., MPI_FILE_READ_ALL_END) is issued. The begin and end calls are provided to allow the user and MPI implementation to optimize the collective operation. Note that I never claimed that your example was not allowed; I simply pointed out that, according to the text above, the end result is not deterministic. [ text remove ] > However, to do this, the read_shared() call would need to block until > all procs have called iread_ordered(). This does pose two difficulties: > 1) It requires MPI to track any outstanding non-blocking collective > calls so that it knows to wait on them. That is pretty easy. An attribute on the file handle can be used for that. > 2) It forces read_shared() into this limbo between being local / not > local. It's supposed to be a local call, but in this context, it can't > return until all other procs have called the collectives that come > before it. So is it still "local"? This is pretty major if you ask me. Especially since an easy solution to avoid this problem is opening the file twice (so you effectively get two shared file pointers). I'm wondering if there are any other functions already in this position? The best I can come up with is MPI_Recv (and friends) and MPI_Send, since technically those can block until the other side posts a matching operation, but that is easily motivated in the standard. The text you quoted is interesting though, because at least it makes it very clear that the shared file pointer is updated first, possibly before the data is accessed. This clarifies what happens in a fault situation (read error or otherwise): the shared file pointer is moved to point to the first byte after the region that is *requested*, not after the region that is actually successfully accessed. So, if the call fails to ready any data, the shared file pointer can still point to the position it would have if the call succeeded. Dries

1 0

Re: [MPI3-IO] shared file pointer
by Adam T. Moody 16 Feb '12

16 Feb '12

Hi guys, My interpretation of the standard is that the following use of split collectives is ok: MPI_File_read_ordered_begin() MPI_File_read_shared() MPI_File_read_ordered_end() This understanding comes from the following: p405, 17-9 "In general, each I/O operation leaves the file pointer pointing to the next data itme after the last one that is accessed by the operation. In a nonblocking or split collective operation, the pointer is updated by the call that initiates the I/O, possibly before the access completes." and: p417, 1-2 "After a shared file pointer operation is initiated, the shared file pointer is updated to point to the next etype after the last one that will be accessed." Assuming this interpretation is correct, when removing split collectives from an existing app, then it's natural for the user to want to replace the above with: MPI_File_iread_ordered() MPI_File_read_shared() MPI_Wait() If possible, we should strive to allow this. I can see cases where it would be useful, e.g., rank 0 always reads some special fields, perhaps the header of the next shared array. And it's consistent with current behavior of the split collectives. However, to do this, the read_shared() call would need to block until all procs have called iread_ordered(). This does pose two difficulties: 1) It requires MPI to track any outstanding non-blocking collective calls so that it knows to wait on them. 2) It forces read_shared() into this limbo between being local / not local. It's supposed to be a local call, but in this context, it can't return until all other procs have called the collectives that come before it. So is it still "local"? Assuming #1 doesn't impose too big a burden on MPI implementations, then we need to decide whether the problem in #2 is a show stopper or not. -Adam Finally, there is Quincey Koziol wrote: >Hi Mohamad, > >On Feb 12, 2012, at 11:52 AM, Mohamad Chaarawi wrote: > >Hi All, > >So after laying out these choices, I see 3 options for the nbcio ticket: > >1) Keep it the way it is, and adding an advise to users saying that intermixing nonblocking collective shared fp operation and independent shared fp operations is undefined. > >2) Make the previous case illegal. > >3) remove the nonblocking ordered operations from this ticket (to be handled in a separate ticket) > >keep in mind, as mentioned previously, we can't provide a prototype implementation for those routines. From our side, if the forum accepts only a pseudo code implementation for those routines, we would like to proceed with option 1, otherwise we would go with option 3. > >Sounds good to me. > >Quincey > > >Any other opinions? > >Thanks, >Mohamad > >On 2/6/2012 1:19 PM, Dries Kimpe wrote: > >Yes, that's an accurate summary. > > Dries > >* Mohamad Chaarawi <chaarawi(a)hdfgroup.org><mailto:[email protected]> [2012-02-06 10:48:50]: > > > >Hi Dries, > > >I'm jumping late on this thread, but to summarize so far (and correct me >if I made a mistake understanding), we have two cases: > > >1) two non-blocking collective shared FP operations: > > > MPI_File_iread_ordered > MPI_File_iread_ordered > > >This will be ordered in the sense that the user will see that the first >operations will occur before the second one. > > >2) mixed collective and independent > > > MPI_File_iread_ordered > MPI_File_read_shared > > >Where the choices that you mentioned apply, right? > > > * make that case illegal > * make it undefined > > >As you mentioned that since the split collectives leave it as undefined, >makes me lean more towards keeping it that way. > > >Thanks, >Mohamad > > > > > > >On 02/02/2012 04:22 PM, Dries Kimpe wrote: > > >There might be some use in 'undefined ordering' as opposed to 'illegal' >for those application that don't care about the ordering. > > >Applications that do rely on the ordering can easily use the existing MPI >functions to enforce ordering. > > >So, the way I see it: 2 choices: >1) Say order is undefined in the standard (basically there's a precedent >with the split collective versions). >2) Say it is illegal. The user can duplicate the file handle and easily >implement their own version of what they need. > > >(2) is easier for the implementor. > > > > > > >_______________________________________________ >MPI3-IO mailing list >MPI3-IO(a)lists.mpi-forum.org<mailto:[email protected]> >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io > > > > >_______________________________________________ >MPI3-IO mailing list >MPI3-IO(a)lists.mpi-forum.org<mailto:[email protected]> >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io > > >_______________________________________________ >MPI3-IO mailing list >MPI3-IO(a)lists.mpi-forum.org<mailto:[email protected]> >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io > > > > >------------------------------------------------------------------------ > >_______________________________________________ >MPI3-IO mailing list >MPI3-IO(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io > >

1 0