- mpiwg-io - mailman3.cels.anl.gov

Re: [MPI3-IO] shared file pointer
by Quincey Koziol 01 Feb '12

01 Feb '12

Hi Dries, On Jan 31, 2012, at 3:50 PM, Dries Kimpe wrote: > My opinion: > > 1) Using non-blocking collectives in the implementation of the > non-blocking I/O functions is pretty straightforward. Yes it is a bit > harder (basically implementing a state machine) but that is a problem that > has been solved already in libnbc. OK, good. > 2) I'd have to take a closer look, but what do the I/O consistency rules > say here? What if the next one is write instead of read? I don't think that the read vs. write matters... > Also, in case of the split collectives, doesn't the standard state that > the call is considered complete when the end call returns? > Does it really state that the MPI_File_read_ordered_begin() needs to > update the shared file pointer? > > It says on page 422 that no collective I/O routines are permitted on a > file handle concurrently with a split collective. This still allows for > > MPI_File_read_ordered_begin() > MPI_File_read_shared () [but not read_ordered()] > MPI_File_read_ordered_end() > > However, it says that an implementation is free to implement any split > collective data access routine using the corresponding blocking collective > routine when either the begin or end call is issued. > > In other words, the example above has -- according to the standard -- two > outcomes: either the collective call logically executes before the > independent, or the independent logically executes before the collective. Is there a well-defined behavior in the standard for this? > I'm not sure that doing 2 iread_ordered calls would be legal or -- if > legal -- forced to act as if the shared file pointer was updated before > the read_shared() call (or the 2nd iread_ordered for that matter). Hmm, maybe nonblocking (both independent & collective) I/O on a shared file pointer doesn't make sense at all then? Quincey > As executing 2 non-blocking MPI_Iread_read_ordered() calls would translate > to 2 MPI_Iscan operations, what would happen using non-blocking > collectives in the following scenario? > > MPI_Iscan (comm) > MPI_Iscan (comm) > > Dries > > > * Moody, Adam T. <moody20(a)llnl.gov> [2012-01-27 10:29:32]: > >> Using MPI_Iallreduce certainly helps, and I think the progress rules of MPI would allow this to work. The downside is that you may need to track a long chain of dependencies. > >> MPI_File_iread_ordered() >> MPI_File_iread_ordered() >> MPI_File_read_shared() >> MPI_Waitall() > >> In this case, fortunately, the second iread_ordered() op does not need to wait on the iallreduce of the first to start its own iallreduce. It only needs to wait on the iallreduce of the first before it can actually start to read data. This is good, because otherwise it would force the second iread_ordered to block, which is something we should (probably must) avoid. > >> However, the read_shared needs to wait on the iallreduce for both outstanding iread_ordered ops before it can complete. Thus, MPI ends up tracking all of these dependencies. > >> Maybe that's not a big deal? Or maybe there's another way to implement this? It'd be good to have the ROMIO folks help out with this one. >> -Adam > >> ________________________________________ >> From: Mohamad Chaarawi [chaarawi(a)hdfgroup.org] >> Sent: Friday, January 27, 2012 9:25 AM >> To: Quincey Koziol >> Cc: Moody, Adam T.; Dries Kimpe >> Subject: Re: shared file pointer > >> Hi All, > >>>> As we discussed, I think our current non-blocking IO proposal may compound the problem of shared file pointers. Consider that you have a program like the following: > >>>> MPI_File_read_ordered_begin() >>>> MPI_File_read_shared() >>>> MPI_File_read_ordered_end() > >>>> Even though the collective may not return data until the end call, it does update the shared file pointer so that the individual read call in the middle uses the updated pointer value. The standard currently allows the begin call to be blocking (see MPI 2.2, page 422 lines 18-23). Given this, a scalable way to update the shared file pointer in the begin call is to execute an MPI_Allreduce() to update the pointer after tallying up the amount from each process. > >>>> However, with non-blocking collectives, we don't currently allow the initiation calls to be blocking. If we follow that convention, then MPI_File_iread_ordered() cannot block, which means we can't update the pointer using the Allreduce method like we can in the split versions. I think we should hash this out to be sure that it's possible to update the pointer in a scalable way. Or perhaps we need to say that MPI_File_iread_ordered() can block, but I don't think this would go over well. >>> Maybe there's a possible "out": could MPI_File_iread_ordered() be collective, but non synchronizing/blocking and use a nonblocking Allreduce operation? That way the implementation could record (internally) that the Allreduce was pending and wait for it to complete before the MPI_File_read_shared() proceeded (which wouldn't necessarily force the MPI_File_iread_ordered() to complete, just the Allreduce component, so that the shared file pointer address could be updated). Would that work? > >> I'm thinking on the same lines as Quincey mentioned. A non-blocking >> all-reduce would work, where MPI would record internally that this >> allreduce needs to complete before starting this blocking read_shared. >> The other option is to allow the iread_ordered to block for the >> allreduce so that the File pointer could be adjusted before returning. > >> Thanks, >> Mohamad > > >>> I've CC'd Mohamad and Dries, to bring them into the conversation, maybe they'll have more ideas. > >>> Quincey > >

1 0

Re: [MPI3-IO] Nonblocking Collective I/O Implementation for ticket #273
by Edgar Gabriel 31 Jan '12

31 Jan '12

On 1/31/2012 4:47 PM, Dries Kimpe wrote: > > * Mohamad Chaarawi <chaarawi(a)hdfgroup.org> [2012-01-27 15:04:41]: > > [...] > >> The reason is that the entire shared file pointer operations are not >> implemented within the MPI-I/O framework used. That being said, we can > > I'm sorry, I don't understand what you mean by the sentence above. The OMPIO framework, which has been used to implement the non-blocking collective I/O operations, does not support at the moment shared file pointer operations. Shared file pointers will be available around the May timeframe, but waiting for that would probably add an unnecessary delay for the specification. > >> provide the pseudo code for the operations, since in reality there is >> nothing fundamentally difficult if we have a shared file pointer and the >> other nonblocking routines available. > >> Would anyone on this list object to the idea of just providing the >> pseudo code implementation for the shared file pointer routines (i.e. >> postpone voting on the ticket till we have a prototype implementation >> which would likely not be in the MPI-3.0 time frame anymore)? > > Given the discussion about the non-blocking shared file pointer routines, > it is probably best to put the pseudocode to allow for discussion and make > sure we fully understand the consequences before getting a vote on the > ticket. > > While I agree that it would be possible for a user to implement collective > non-blocking shared file pointer routines, the same could be said for the > other I/O collectives, so in this case I'd prefer to provide those > routines. There is a precedence for what has been suggested in this email within the MPI standardization. The Forum agreed that an implementation of the non-blocking collective operations for inter-communicators is not necessary, since there is no fundamental, additional challenge compared to their intra-communicator counterparts. I think the same argument can be made here. There is nothing fundamentally difficult in the sequence of 1. get current position of shared file pointer on process 0 2. execute non-blocking scan operation where every process adds its own amount of data to be written/read to the current position of the shared file pointer 3. execute a non-blocking collective write_at operation with offsets as determined as a result of step 2. Step 2 has been implemented as part of the non-blocking collective operations, step 3 has been implemented as part of the non-blocking collective I/O operations. Step 1 has fundamentally been implemented a number of times. Thanks Edgar > > Dries > > > > _______________________________________________ > MPI3-IO mailing list > MPI3-IO(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io -- Edgar Gabriel Associate Professor Parallel Software Technologies Lab http://pstl.cs.uh.edu Department of Computer Science University of Houston Philip G. Hoffman Hall, Room 524 Houston, TX-77204, USA Tel: +1 (713) 743-3857 Fax: +1 (713) 743-3335

1 0

Re: [MPI3-IO] Nonblocking Collective I/O Implementation for ticket #273
by Dries Kimpe 31 Jan '12

31 Jan '12

* Mohamad Chaarawi <chaarawi(a)hdfgroup.org> [2012-01-27 15:04:41]: [...] > The reason is that the entire shared file pointer operations are not > implemented within the MPI-I/O framework used. That being said, we can I'm sorry, I don't understand what you mean by the sentence above. > provide the pseudo code for the operations, since in reality there is > nothing fundamentally difficult if we have a shared file pointer and the > other nonblocking routines available. > Would anyone on this list object to the idea of just providing the > pseudo code implementation for the shared file pointer routines (i.e. > postpone voting on the ticket till we have a prototype implementation > which would likely not be in the MPI-3.0 time frame anymore)? Given the discussion about the non-blocking shared file pointer routines, it is probably best to put the pseudocode to allow for discussion and make sure we fully understand the consequences before getting a vote on the ticket. While I agree that it would be possible for a user to implement collective non-blocking shared file pointer routines, the same could be said for the other I/O collectives, so in this case I'd prefer to provide those routines. Dries

1 0

Re: [MPI3-IO] Fault Tolerance & I/O Discussion
by Josh Hursey 31 Jan '12

31 Jan '12

Try the new link (I had to close the other poll due to a timezone problem): http://www.doodle.com/yifhpi5emyyzrspa -- Josh On Tue, Jan 31, 2012 at 4:41 PM, Mohamad Chaarawi <chaarawi(a)hdfgroup.org>wrote: > Hi Josh, > > When I click the link, it says the poll has been deleted. > > Thanks, > Mohamad > > > On 01/31/2012 11:52 AM, Josh Hursey wrote: > > (Cross posted to both the I/O and FT MPI-3 listservs) > > During the FT plenary session at the Jan. MPI Forum meeting there were > some concerns about fault tolerance semantics in the I/O chapter. We did > not have much time to fully discuss the additional semantics during the > meeting. To make sure that we push towards a complete set of semantics > useful for the I/O community in the next draft I would like to have a > teleconf to discuss the I/O chapter of the FT proposal. Preferability in > the next week and a half (so we have time to fine tune things before the > next forum meeting). > > Below is a link to a doodle poll to find a good time for a teleconf. If > you are interested in participating in this discussion, please fill this > poll out by 2 PM Eastern on Thurs. Feb. 2 so we can set the date/time. > http://www.doodle.com/s3hz9daeh8pn483m > > Thanks, > Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey > > > _______________________________________________ > MPI3-IO mailing [email protected]://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io > > > > _______________________________________________ > MPI3-IO mailing list > MPI3-IO(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io > > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [MPI3-IO] shared file pointer
by Dries Kimpe 31 Jan '12

31 Jan '12

My opinion: 1) Using non-blocking collectives in the implementation of the non-blocking I/O functions is pretty straightforward. Yes it is a bit harder (basically implementing a state machine) but that is a problem that has been solved already in libnbc. 2) I'd have to take a closer look, but what do the I/O consistency rules say here? What if the next one is write instead of read? Also, in case of the split collectives, doesn't the standard state that the call is considered complete when the end call returns? Does it really state that the MPI_File_read_ordered_begin() needs to update the shared file pointer? It says on page 422 that no collective I/O routines are permitted on a file handle concurrently with a split collective. This still allows for MPI_File_read_ordered_begin() MPI_File_read_shared () [but not read_ordered()] MPI_File_read_ordered_end() However, it says that an implementation is free to implement any split collective data access routine using the corresponding blocking collective routine when either the begin or end call is issued. In other words, the example above has -- according to the standard -- two outcomes: either the collective call logically executes before the independent, or the independent logically executes before the collective. I'm not sure that doing 2 iread_ordered calls would be legal or -- if legal -- forced to act as if the shared file pointer was updated before the read_shared() call (or the 2nd iread_ordered for that matter). As executing 2 non-blocking MPI_Iread_read_ordered() calls would translate to 2 MPI_Iscan operations, what would happen using non-blocking collectives in the following scenario? MPI_Iscan (comm) MPI_Iscan (comm) Dries * Moody, Adam T. <moody20(a)llnl.gov> [2012-01-27 10:29:32]: > Using MPI_Iallreduce certainly helps, and I think the progress rules of MPI would allow this to work. The downside is that you may need to track a long chain of dependencies. > MPI_File_iread_ordered() > MPI_File_iread_ordered() > MPI_File_read_shared() > MPI_Waitall() > In this case, fortunately, the second iread_ordered() op does not need to wait on the iallreduce of the first to start its own iallreduce. It only needs to wait on the iallreduce of the first before it can actually start to read data. This is good, because otherwise it would force the second iread_ordered to block, which is something we should (probably must) avoid. > However, the read_shared needs to wait on the iallreduce for both outstanding iread_ordered ops before it can complete. Thus, MPI ends up tracking all of these dependencies. > Maybe that's not a big deal? Or maybe there's another way to implement this? It'd be good to have the ROMIO folks help out with this one. > -Adam > ________________________________________ > From: Mohamad Chaarawi [chaarawi(a)hdfgroup.org] > Sent: Friday, January 27, 2012 9:25 AM > To: Quincey Koziol > Cc: Moody, Adam T.; Dries Kimpe > Subject: Re: shared file pointer > Hi All, > >> As we discussed, I think our current non-blocking IO proposal may compound the problem of shared file pointers. Consider that you have a program like the following: > >> MPI_File_read_ordered_begin() > >> MPI_File_read_shared() > >> MPI_File_read_ordered_end() > >> Even though the collective may not return data until the end call, it does update the shared file pointer so that the individual read call in the middle uses the updated pointer value. The standard currently allows the begin call to be blocking (see MPI 2.2, page 422 lines 18-23). Given this, a scalable way to update the shared file pointer in the begin call is to execute an MPI_Allreduce() to update the pointer after tallying up the amount from each process. > >> However, with non-blocking collectives, we don't currently allow the initiation calls to be blocking. If we follow that convention, then MPI_File_iread_ordered() cannot block, which means we can't update the pointer using the Allreduce method like we can in the split versions. I think we should hash this out to be sure that it's possible to update the pointer in a scalable way. Or perhaps we need to say that MPI_File_iread_ordered() can block, but I don't think this would go over well. > > Maybe there's a possible "out": could MPI_File_iread_ordered() be collective, but non synchronizing/blocking and use a nonblocking Allreduce operation? That way the implementation could record (internally) that the Allreduce was pending and wait for it to complete before the MPI_File_read_shared() proceeded (which wouldn't necessarily force the MPI_File_iread_ordered() to complete, just the Allreduce component, so that the shared file pointer address could be updated). Would that work? > I'm thinking on the same lines as Quincey mentioned. A non-blocking > all-reduce would work, where MPI would record internally that this > allreduce needs to complete before starting this blocking read_shared. > The other option is to allow the iread_ordered to block for the > allreduce so that the File pointer could be adjusted before returning. > Thanks, > Mohamad > > I've CC'd Mohamad and Dries, to bring them into the conversation, maybe they'll have more ideas. > > Quincey

1 0

Re: [MPI3-IO] Fault Tolerance & I/O Discussion
by Mohamad Chaarawi 31 Jan '12

31 Jan '12

Hi Josh, When I click the link, it says the poll has been deleted. Thanks, Mohamad On 01/31/2012 11:52 AM, Josh Hursey wrote: > (Cross posted to both the I/O and FT MPI-3 listservs) > > During the FT plenary session at the Jan. MPI Forum meeting there were > some concerns about fault tolerance semantics in the I/O chapter. We > did not have much time to fully discuss the additional semantics > during the meeting. To make sure that we push towards a complete set > of semantics useful for the I/O community in the next draft I would > like to have a teleconf to discuss the I/O chapter of the FT proposal. > Preferability in the next week and a half (so we have time to fine > tune things before the next forum meeting). > > Below is a link to a doodle poll to find a good time for a teleconf. > If you are interested in participating in this discussion, please fill > this poll out by 2 PM Eastern on Thurs. Feb. 2 so we can set the > date/time. > http://www.doodle.com/s3hz9daeh8pn483m > > Thanks, > Josh > > -- > Joshua Hursey > Postdoctoral Research Associate > Oak Ridge National Laboratory > http://users.nccs.gov/~jjhursey <http://users.nccs.gov/%7Ejjhursey> > > > _______________________________________________ > MPI3-IO mailing list > MPI3-IO(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-io

1 0

Re: [MPI3-IO] [Mpi3-ft] Fault Tolerance & I/O Discussion
by Josh Hursey 31 Jan '12

31 Jan '12

Sorry about that. I forgot to set the timezone when setting up the poll, and it looks like I cannot change it now. So I created a new poll. Please use the link below, it should now allow you to adjust the time zone local via a drop down menu at the top of the poll. http://www.doodle.com/yifhpi5emyyzrspa Thanks for letting me know. -- Josh On Tue, Jan 31, 2012 at 3:44 PM, Dries Kimpe <dkimpe(a)mcs.anl.gov> wrote: > What's the time zone for the times listed in the doodle poll? > Or does doodle automatically adjust? > > Dries > > * Josh Hursey <jjhursey(a)open-mpi.org> [2012-01-31 12:52:58]: > > > (Cross posted to both the I/O and FT MPI-3 listservs) > > > During the FT plenary session at the Jan. MPI Forum meeting there were > some > > concerns about fault tolerance semantics in the I/O chapter. We did not > > have much time to fully discuss the additional semantics during the > > meeting. To make sure that we push towards a complete set of semantics > > useful for the I/O community in the next draft I would like to have a > > teleconf to discuss the I/O chapter of the FT proposal. Preferability in > > the next week and a half (so we have time to fine tune things before the > > next forum meeting). > > > Below is a link to a doodle poll to find a good time for a teleconf. If > you > > are interested in participating in this discussion, please fill this poll > > out by 2 PM Eastern on Thurs. Feb. 2 so we can set the date/time. > > http://www.doodle.com/s3hz9daeh8pn483m > > > Thanks, > > Josh > > _______________________________________________ > mpi3-ft mailing list > mpi3-ft(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-ft > -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

Re: [MPI3-IO] [Mpi3-ft] Fault Tolerance & I/O Discussion
by Dries Kimpe 31 Jan '12

31 Jan '12

What's the time zone for the times listed in the doodle poll? Or does doodle automatically adjust? Dries * Josh Hursey <jjhursey(a)open-mpi.org> [2012-01-31 12:52:58]: > (Cross posted to both the I/O and FT MPI-3 listservs) > During the FT plenary session at the Jan. MPI Forum meeting there were some > concerns about fault tolerance semantics in the I/O chapter. We did not > have much time to fully discuss the additional semantics during the > meeting. To make sure that we push towards a complete set of semantics > useful for the I/O community in the next draft I would like to have a > teleconf to discuss the I/O chapter of the FT proposal. Preferability in > the next week and a half (so we have time to fine tune things before the > next forum meeting). > Below is a link to a doodle poll to find a good time for a teleconf. If you > are interested in participating in this discussion, please fill this poll > out by 2 PM Eastern on Thurs. Feb. 2 so we can set the date/time. > http://www.doodle.com/s3hz9daeh8pn483m > Thanks, > Josh

1 0

[MPI3-IO] Fault Tolerance & I/O Discussion
by Josh Hursey 31 Jan '12

31 Jan '12

(Cross posted to both the I/O and FT MPI-3 listservs) During the FT plenary session at the Jan. MPI Forum meeting there were some concerns about fault tolerance semantics in the I/O chapter. We did not have much time to fully discuss the additional semantics during the meeting. To make sure that we push towards a complete set of semantics useful for the I/O community in the next draft I would like to have a teleconf to discuss the I/O chapter of the FT proposal. Preferability in the next week and a half (so we have time to fine tune things before the next forum meeting). Below is a link to a doodle poll to find a good time for a teleconf. If you are interested in participating in this discussion, please fill this poll out by 2 PM Eastern on Thurs. Feb. 2 so we can set the date/time. http://www.doodle.com/s3hz9daeh8pn483m Thanks, Josh -- Joshua Hursey Postdoctoral Research Associate Oak Ridge National Laboratory http://users.nccs.gov/~jjhursey

1 0

[MPI3-IO] Nonblocking Collective I/O Implementation for ticket #273
by Mohamad Chaarawi 27 Jan '12

27 Jan '12

Hi All, We have the prototype implementation for the nonblocking collective I/O routines provided by Dr. Edgar Gabriel and his student (Vishwanath Venkatesan). The implementation provides a prototype for implementing the following routines: * MPI_File_iread_all * MPI_File_iwrite_all * MPI_File_iread_at_all * MPI_File_iwrite_at_all We did not implement, however the nonblocking counterpart for shared file pointer collective operations: * MPI_File_iread_ordered * MPI_File_iwrite_ordered The reason is that the entire shared file pointer operations are not implemented within the MPI-I/O framework used. That being said, we can provide the pseudo code for the operations, since in reality there is nothing fundamentally difficult if we have a shared file pointer and the other nonblocking routines available. Would anyone on this list object to the idea of just providing the pseudo code implementation for the shared file pointer routines (i.e. postpone voting on the ticket till we have a prototype implementation which would likely not be in the MPI-3.0 time frame anymore)? Thanks, Mohamad

1 0