May 2014 - mpi-forum - mailman3.cels.anl.gov

Re: [Mpi-forum] large count support not as easy as people seem to have thought
by Rob Latham 06 May '14

06 May '14

On 05/06/2014 04:14 PM, Jeff Hammond wrote: > On Tue, May 6, 2014 at 3:54 PM, Hjelm, Nathan T <hjelmn(a)lanl.gov> wrote: >> +1 for large counts. I find it just a bit ridiculous that this was punted on by the forum but I wasn't around for the discussion. Is this in issue forced on C/C++ by fortran? > > Unfortunately, I don't think we can blame this one on Fortran. Users > want to pass large counts from the C interface. That's the motivation > for BigMPI, at least. i imagine one day (once again?) the c int type will be 64 bits on a platform, right? or will c ints be 32 bits for ever? ==rob > > The Fortran users that want -i8 and friends to work with the MPI > Fortran interface are another issue that doesn't need new functions in > the standard, just a bunch of interface gymnastics that most > implementers won't enjoy. > > Best, > > Jeff > >> As I noted on this list last Friday, I am working on a higher-level >> library to support large counts for MPI communication functions >> (https://github.com/jeffhammond/BigMPI) >> >> In the course of actually trying to implement this the way the Forum >> contends it can be done - i.e. using derived-datatypes - I have found >> some issues that undermine the Forum's contention that it is so easy >> for users to do it that it doesn't need to be in the standard. >> >> To illustrate some of the issues that I have found, let us consider >> the large-count implementation of nonblocking reduce... >> >> # Example Use Case >> >> It is entirely reasonable to think that some quantum chemist will want >> to reduce a contiguous buffer of more than 2^31 doubles corresponding >> to the Fock matrix if they have a multithreaded code, since 16 GB is >> not an unreasonable amount of memory per node. >> >> # Issues >> >> Unlike the blocking case, where it is reasonable to chop up the data >> and performance multiple operations (e.g. >> https://github.com/jeffhammond/BigMPI/blob/master/src/reductions_x.c, >> assuming that untested code is correct), one must return a single >> request to the application if one is to implement MPIX_I(all)reduce_x >> with the same semantics as MPI_Iallreduce, as I aspire to do in >> BigMPI. >> >> Issue #1: chopping doesn't work for nonblocking. >> >> To do the large-count reduction in one nonblocking MPI call, a derived >> datatype is required. However, unlike in RMA, reductions cannot use >> built-in ops for user-defined datatypes, even if they are trivially >> composed of a large-count of built-in datatypes. See >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/338 and >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/34 for elaborate >> commentary on why this semantic mismatch is lame. >> >> Issue #2: cannot use built-in reduce ops. >> >> Once we rule out using built-in ops with our large-count datatypes, we >> must reimplement all of the reduction operations required. I find >> this to be nontrivial. I have not yet figured out how to get at the >> underlying datatype info in a simple manner. It appears that >> MPI_Type_get_envelope exists for this purpose, but it's a huge pain to >> have to call this function when all I need to know is the number of >> built-in datatypes so that I can apply my clever and use >> MPI_Reduce_local inside of my user-defined operation. >> >> Issue #3: implementing the user-defined reduce op isn't easy (in my opinion). >> >> Many MPI implementations optimize reductions. On Blue Gene/Q, MPI has >> explicitly vectorized intrinsic/assembly code. Unless >> MPI_Reduce_local hits that code path, I am losing a huge amount of >> performance in reductions when I go from 2^31 to 2^31+1 elements. I >> would not be surprised at all if user-defined ops+datatypes exercises >> suboptimal code paths in many MPI implementations, which means that >> the performance of nonblocking reductions is unnecessarily crippled. >> >> Issue #4: inability to use optimizations in the MPI implementation. >> >> # Conclusion >> >> I believe this problem is best addressed in one of two ways: >> >> 1) Approve the semantic changes requested in tickets 34 and 338 so >> that one can use built-in ops with homogeneous user-defined datatypes. >> This is my preference for multiple reasons. >> >> 2) Add large-count reductions to the standard. This means 8 new >> functions: blocking and nonblocking (all)reduce and >> reduce_scatter(_block). We don't need large-count functions for any >> other collectives because the datatype solution works just fine there, >> as I've already demonstrated in BigMPI >> (https://github.com/jeffhammond/BigMPI/blob/master/src/collectives_x.c) >> >> # Social Commentary >> >> From now on, when the Forum punts on things and says it's no problem >> for users to roll their own using the existing functionality in MPI, >> we should strive to be a bit more diligent and actually prototype that >> implementation in a manner that proves how easy it is for users. It >> turns out, writing code for some things is harder than just talking >> about them in a conference room... >> >> # Related >> >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/338#comment:9 >> captures some of this feedback in Trac. >> >> I created https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/423 for >> the reasons described therein. >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> _______________________________________________ >> mpi-forum mailing list >> mpi-forum(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum >> _______________________________________________ >> mpi-forum mailing list >> mpi-forum(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > > > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [Mpi-forum] large count support not as easy as people seem to have thought
by Jeff Hammond 06 May '14

06 May '14

On Tue, May 6, 2014 at 3:54 PM, Hjelm, Nathan T <hjelmn(a)lanl.gov> wrote: > +1 for large counts. I find it just a bit ridiculous that this was punted on by the forum but I wasn't around for the discussion. Is this in issue forced on C/C++ by fortran? Unfortunately, I don't think we can blame this one on Fortran. Users want to pass large counts from the C interface. That's the motivation for BigMPI, at least. The Fortran users that want -i8 and friends to work with the MPI Fortran interface are another issue that doesn't need new functions in the standard, just a bunch of interface gymnastics that most implementers won't enjoy. Best, Jeff > As I noted on this list last Friday, I am working on a higher-level > library to support large counts for MPI communication functions > (https://github.com/jeffhammond/BigMPI) > > In the course of actually trying to implement this the way the Forum > contends it can be done - i.e. using derived-datatypes - I have found > some issues that undermine the Forum's contention that it is so easy > for users to do it that it doesn't need to be in the standard. > > To illustrate some of the issues that I have found, let us consider > the large-count implementation of nonblocking reduce... > > # Example Use Case > > It is entirely reasonable to think that some quantum chemist will want > to reduce a contiguous buffer of more than 2^31 doubles corresponding > to the Fock matrix if they have a multithreaded code, since 16 GB is > not an unreasonable amount of memory per node. > > # Issues > > Unlike the blocking case, where it is reasonable to chop up the data > and performance multiple operations (e.g. > https://github.com/jeffhammond/BigMPI/blob/master/src/reductions_x.c, > assuming that untested code is correct), one must return a single > request to the application if one is to implement MPIX_I(all)reduce_x > with the same semantics as MPI_Iallreduce, as I aspire to do in > BigMPI. > > Issue #1: chopping doesn't work for nonblocking. > > To do the large-count reduction in one nonblocking MPI call, a derived > datatype is required. However, unlike in RMA, reductions cannot use > built-in ops for user-defined datatypes, even if they are trivially > composed of a large-count of built-in datatypes. See > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/338 and > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/34 for elaborate > commentary on why this semantic mismatch is lame. > > Issue #2: cannot use built-in reduce ops. > > Once we rule out using built-in ops with our large-count datatypes, we > must reimplement all of the reduction operations required. I find > this to be nontrivial. I have not yet figured out how to get at the > underlying datatype info in a simple manner. It appears that > MPI_Type_get_envelope exists for this purpose, but it's a huge pain to > have to call this function when all I need to know is the number of > built-in datatypes so that I can apply my clever and use > MPI_Reduce_local inside of my user-defined operation. > > Issue #3: implementing the user-defined reduce op isn't easy (in my opinion). > > Many MPI implementations optimize reductions. On Blue Gene/Q, MPI has > explicitly vectorized intrinsic/assembly code. Unless > MPI_Reduce_local hits that code path, I am losing a huge amount of > performance in reductions when I go from 2^31 to 2^31+1 elements. I > would not be surprised at all if user-defined ops+datatypes exercises > suboptimal code paths in many MPI implementations, which means that > the performance of nonblocking reductions is unnecessarily crippled. > > Issue #4: inability to use optimizations in the MPI implementation. > > # Conclusion > > I believe this problem is best addressed in one of two ways: > > 1) Approve the semantic changes requested in tickets 34 and 338 so > that one can use built-in ops with homogeneous user-defined datatypes. > This is my preference for multiple reasons. > > 2) Add large-count reductions to the standard. This means 8 new > functions: blocking and nonblocking (all)reduce and > reduce_scatter(_block). We don't need large-count functions for any > other collectives because the datatype solution works just fine there, > as I've already demonstrated in BigMPI > (https://github.com/jeffhammond/BigMPI/blob/master/src/collectives_x.c) > > # Social Commentary > > From now on, when the Forum punts on things and says it's no problem > for users to roll their own using the existing functionality in MPI, > we should strive to be a bit more diligent and actually prototype that > implementation in a manner that proves how easy it is for users. It > turns out, writing code for some things is harder than just talking > about them in a conference room... > > # Related > > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/338#comment:9 > captures some of this feedback in Trac. > > I created https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/423 for > the reasons described therein. > > -- > Jeff Hammond > jeff.science(a)gmail.com > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [Mpi-forum] Problem with Hotel Logistics for June Forum Meeting
by Hjelm, Nathan T 06 May '14

06 May '14

I am ok with an airport hotel as long as I don't have to go far for the forum. -Nathan ________________________________________ From: mpi-forum [mpi-forum-bounces(a)lists.mpi-forum.org] on behalf of Martin Schulz [schulzm(a)llnl.gov] Sent: Tuesday, May 06, 2014 12:37 PM To: Main mailing list Subject: [Mpi-forum] Problem with Hotel Logistics for June Forum Meeting Hi all, We are having some problems with the hotels for the June meeting – see Rich’s message below – and won’t be able to get a room block. How many people would be impacted by that, i.e., couldn’t find a decent hotel under these circumstances? In general, I think we have three options: * We can keep the meeting at Microsoft as usual, but this could mean for some people to potentially commute from further out to the meeting (worst case from O’Hare) * We can try to have the meeting at an airport hotel where the facilities are less ideal and we have to pay a higher cost from our leftover funds * ANL has previously offered to host, so if this still an option we could consider that as well, but this requires everyone to get rental cars Do people have preferences? Unless I hear a clear vote for one option, though, I think we should stick with option 1 to avoid confusion and have the least impact on budgets (forum and travelers). Thanks, Martin ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://scalability.llnl.gov/ CASC @ Lawrence Livermore National Laboratory, Livermore, USA From: "richardg(a)mellanox.com<mailto:[email protected]>" <richardg(a)mellanox.com<mailto:[email protected]>> Date: Tuesday, May 6, 2014 at 11:17 AM To: Anh Vo <Anh.Vo(a)microsoft.com<mailto:[email protected]>>, Martin Schulz <schulz6(a)llnl.gov<mailto:[email protected]>> Subject: RE: hotel for MPI Forum in June We have been looking into it, and l don’t think we will be able to come up with a hotel. There is some big meeting down town with hotels not having availability at least for blocks of rooms at a discount rate. For the fun of it I checked on gov rate availability in Marriott chains, and same issue (that is the rate we usually get). We could get hotels out towards the airport, but I do not think that is practical. One thing we could consider is having the meeting at the Spring Hill Suites out by the airport – not as nice a location, but last time we met there they gave us free meeting rooms for staying with them and for catering. Not sure if there is a room count requirement with it. Martin, what do you think ? Rich

1 0

Re: [Mpi-forum] large count support not as easy as people seem to have thought
by Hjelm, Nathan T 06 May '14

06 May '14

+1 for large counts. I find it just a bit ridiculous that this was punted on by the forum but I wasn't around for the discussion. Is this in issue forced on C/C++ by fortran? -Nathan ________________________________________ From: mpi-forum [mpi-forum-bounces(a)lists.mpi-forum.org] on behalf of Jeff Hammond [jeff.science(a)gmail.com] Sent: Tuesday, May 06, 2014 11:19 AM To: MPI Forum Subject: [Mpi-forum] large count support not as easy as people seem to have thought As I noted on this list last Friday, I am working on a higher-level library to support large counts for MPI communication functions (https://github.com/jeffhammond/BigMPI) In the course of actually trying to implement this the way the Forum contends it can be done - i.e. using derived-datatypes - I have found some issues that undermine the Forum's contention that it is so easy for users to do it that it doesn't need to be in the standard. To illustrate some of the issues that I have found, let us consider the large-count implementation of nonblocking reduce... # Example Use Case It is entirely reasonable to think that some quantum chemist will want to reduce a contiguous buffer of more than 2^31 doubles corresponding to the Fock matrix if they have a multithreaded code, since 16 GB is not an unreasonable amount of memory per node. # Issues Unlike the blocking case, where it is reasonable to chop up the data and performance multiple operations (e.g. https://github.com/jeffhammond/BigMPI/blob/master/src/reductions_x.c, assuming that untested code is correct), one must return a single request to the application if one is to implement MPIX_I(all)reduce_x with the same semantics as MPI_Iallreduce, as I aspire to do in BigMPI. Issue #1: chopping doesn't work for nonblocking. To do the large-count reduction in one nonblocking MPI call, a derived datatype is required. However, unlike in RMA, reductions cannot use built-in ops for user-defined datatypes, even if they are trivially composed of a large-count of built-in datatypes. See https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/338 and https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/34 for elaborate commentary on why this semantic mismatch is lame. Issue #2: cannot use built-in reduce ops. Once we rule out using built-in ops with our large-count datatypes, we must reimplement all of the reduction operations required. I find this to be nontrivial. I have not yet figured out how to get at the underlying datatype info in a simple manner. It appears that MPI_Type_get_envelope exists for this purpose, but it's a huge pain to have to call this function when all I need to know is the number of built-in datatypes so that I can apply my clever and use MPI_Reduce_local inside of my user-defined operation. Issue #3: implementing the user-defined reduce op isn't easy (in my opinion). Many MPI implementations optimize reductions. On Blue Gene/Q, MPI has explicitly vectorized intrinsic/assembly code. Unless MPI_Reduce_local hits that code path, I am losing a huge amount of performance in reductions when I go from 2^31 to 2^31+1 elements. I would not be surprised at all if user-defined ops+datatypes exercises suboptimal code paths in many MPI implementations, which means that the performance of nonblocking reductions is unnecessarily crippled. Issue #4: inability to use optimizations in the MPI implementation. # Conclusion I believe this problem is best addressed in one of two ways: 1) Approve the semantic changes requested in tickets 34 and 338 so that one can use built-in ops with homogeneous user-defined datatypes. This is my preference for multiple reasons. 2) Add large-count reductions to the standard. This means 8 new functions: blocking and nonblocking (all)reduce and reduce_scatter(_block). We don't need large-count functions for any other collectives because the datatype solution works just fine there, as I've already demonstrated in BigMPI (https://github.com/jeffhammond/BigMPI/blob/master/src/collectives_x.c) # Social Commentary >From now on, when the Forum punts on things and says it's no problem for users to roll their own using the existing functionality in MPI, we should strive to be a bit more diligent and actually prototype that implementation in a manner that proves how easy it is for users. It turns out, writing code for some things is harder than just talking about them in a conference room... # Related https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/338#comment:9 captures some of this feedback in Trac. I created https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/423 for the reasons described therein. -- Jeff Hammond jeff.science(a)gmail.com _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum

1 0

Re: [Mpi-forum] large count support not as easy as people seem to have thought
by Jeff Hammond 06 May '14

06 May '14

On Tue, May 6, 2014 at 3:06 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > On Tue, May 6, 2014 at 2:50 PM, Rob Latham <robl(a)mcs.anl.gov> wrote: >> >> >> On 05/06/2014 12:19 PM, Jeff Hammond wrote: >>> >>> Issue #2: cannot use built-in reduce ops. >>> >>> Once we rule out using built-in ops with our large-count datatypes, we >>> must reimplement all of the reduction operations required. I find >>> this to be nontrivial. I have not yet figured out how to get at the >>> underlying datatype info in a simple manner. It appears that >>> MPI_Type_get_envelope exists for this purpose, but it's a huge pain to >>> have to call this function when all I need to know is the number of >>> built-in datatypes so that I can apply my clever and use >>> MPI_Reduce_local inside of my user-defined operation. >> >> >> To determine the number of built-in datatypes, yes, one must recursively >> call MPI_Type_get_envelope and MPI_Type_get_contents. See the ROMIO >> datatype flattening code for just how much of a pain MPI_Type_get_contents >> and MPI_Type_get_envelope is. > > Indeed, I finished debugging my implementation of that for BigMPI a > few minutes ago > (https://github.com/jeffhammond/BigMPI/blob/master/src/type_contiguous_x.c) > It was obnoxious just to do it for the single case where I am parsing > my own datatype and thus know exactly what should be in it. It takes > 6 MPI calls to convert from the contiguous bigtype to a > (bigcount,basictype) pair, just as it takes 6 MPI calls to go the > other way. > >> You might find Rob Ross's libmpitypes approach a bit better, but that code >> is designed for applying functions to the data of a datatype, deliberately >> bypassing the get_envelope/get_contents information. > > I just need the decoder ring so I can write my own reduction > operations using MPI_Reduce_local so I think I'll avoid that > dependency. I'd rather be implementation agnostic anyways and I guess > libmpitypes is MPICH-oriented. And to add to the grossness, I can't just define one reduction operation (at least in a non-evil i.e. no-globals way) I have to define one of them for every built-in op that I want to support, because there is no way - short of packing the MPI_Op into a new variant of bigtype (which would totally break my abstraction and probably requires me to put it on the heap and pass as a pointer because I think MPI_Op is an opaque handle) - to pass the built-in operation passed to my MPIX_Reduce_x into the MPI_User_function that I register. TL;DR = large-count reductions are hard. We are really torturing users here. Jeff -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [Mpi-forum] large count support not as easy as people seem to have thought
by Jeff Hammond 06 May '14

06 May '14

On Tue, May 6, 2014 at 2:50 PM, Rob Latham <robl(a)mcs.anl.gov> wrote: > > > On 05/06/2014 12:19 PM, Jeff Hammond wrote: >> >> Issue #2: cannot use built-in reduce ops. >> >> Once we rule out using built-in ops with our large-count datatypes, we >> must reimplement all of the reduction operations required. I find >> this to be nontrivial. I have not yet figured out how to get at the >> underlying datatype info in a simple manner. It appears that >> MPI_Type_get_envelope exists for this purpose, but it's a huge pain to >> have to call this function when all I need to know is the number of >> built-in datatypes so that I can apply my clever and use >> MPI_Reduce_local inside of my user-defined operation. > > > To determine the number of built-in datatypes, yes, one must recursively > call MPI_Type_get_envelope and MPI_Type_get_contents. See the ROMIO > datatype flattening code for just how much of a pain MPI_Type_get_contents > and MPI_Type_get_envelope is. Indeed, I finished debugging my implementation of that for BigMPI a few minutes ago (https://github.com/jeffhammond/BigMPI/blob/master/src/type_contiguous_x.c) It was obnoxious just to do it for the single case where I am parsing my own datatype and thus know exactly what should be in it. It takes 6 MPI calls to convert from the contiguous bigtype to a (bigcount,basictype) pair, just as it takes 6 MPI calls to go the other way. > You might find Rob Ross's libmpitypes approach a bit better, but that code > is designed for applying functions to the data of a datatype, deliberately > bypassing the get_envelope/get_contents information. I just need the decoder ring so I can write my own reduction operations using MPI_Reduce_local so I think I'll avoid that dependency. I'd rather be implementation agnostic anyways and I guess libmpitypes is MPICH-oriented. Best, Jeff > -- > Rob Latham > Mathematics and Computer Science Division > Argonne National Lab, IL USA > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [Mpi-forum] large count support not as easy as people seem to have thought
by Rob Latham 06 May '14

06 May '14

On 05/06/2014 12:19 PM, Jeff Hammond wrote: > Issue #2: cannot use built-in reduce ops. > > Once we rule out using built-in ops with our large-count datatypes, we > must reimplement all of the reduction operations required. I find > this to be nontrivial. I have not yet figured out how to get at the > underlying datatype info in a simple manner. It appears that > MPI_Type_get_envelope exists for this purpose, but it's a huge pain to > have to call this function when all I need to know is the number of > built-in datatypes so that I can apply my clever and use > MPI_Reduce_local inside of my user-defined operation. To determine the number of built-in datatypes, yes, one must recursively call MPI_Type_get_envelope and MPI_Type_get_contents. See the ROMIO datatype flattening code for just how much of a pain MPI_Type_get_contents and MPI_Type_get_envelope is. You might find Rob Ross's libmpitypes approach a bit better, but that code is designed for applying functions to the data of a datatype, deliberately bypassing the get_envelope/get_contents information. ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Problem with Hotel Logistics for June Forum Meeting
by Martin Schulz 06 May '14

06 May '14

Hi all, We are having some problems with the hotels for the June meeting see Rich¹s message below and won¹t be able to get a room block. How many people would be impacted by that, i.e., couldn¹t find a decent hotel under these circumstances? In general, I think we have three options: * We can keep the meeting at Microsoft as usual, but this could mean for some people to potentially commute from further out to the meeting (worst case from O¹Hare) * We can try to have the meeting at an airport hotel where the facilities are less ideal and we have to pay a higher cost from our leftover funds * ANL has previously offered to host, so if this still an option we could consider that as well, but this requires everyone to get rental cars Do people have preferences? Unless I hear a clear vote for one option, though, I think we should stick with option 1 to avoid confusion and have the least impact on budgets (forum and travelers). Thanks, Martin ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://scalability.llnl.gov/ CASC @ Lawrence Livermore National Laboratory, Livermore, USA From: "richardg(a)mellanox.com" <richardg(a)mellanox.com> Date: Tuesday, May 6, 2014 at 11:17 AM To: Anh Vo <Anh.Vo(a)microsoft.com>, Martin Schulz <schulz6(a)llnl.gov> Subject: RE: hotel for MPI Forum in June We have been looking into it, and l don¹t think we will be able to come up with a hotel. There is some big meeting down town with hotels not having availability at least for blocks of rooms at a discount rate. For the fun of it I checked on gov rate availability in Marriott chains, and same issue (that is the rate we usually get). We could get hotels out towards the airport, but I do not think that is practical. One thing we could consider is having the meeting at the Spring Hill Suites out by the airport not as nice a location, but last time we met there they gave us free meeting rooms for staying with them and for catering. Not sure if there is a room count requirement with it. Martin, what do you think ? Rich

1 0

large count support not as easy as people seem to have thought
by Jeff Hammond 06 May '14

06 May '14

As I noted on this list last Friday, I am working on a higher-level library to support large counts for MPI communication functions (https://github.com/jeffhammond/BigMPI) In the course of actually trying to implement this the way the Forum contends it can be done - i.e. using derived-datatypes - I have found some issues that undermine the Forum's contention that it is so easy for users to do it that it doesn't need to be in the standard. To illustrate some of the issues that I have found, let us consider the large-count implementation of nonblocking reduce... # Example Use Case It is entirely reasonable to think that some quantum chemist will want to reduce a contiguous buffer of more than 2^31 doubles corresponding to the Fock matrix if they have a multithreaded code, since 16 GB is not an unreasonable amount of memory per node. # Issues Unlike the blocking case, where it is reasonable to chop up the data and performance multiple operations (e.g. https://github.com/jeffhammond/BigMPI/blob/master/src/reductions_x.c, assuming that untested code is correct), one must return a single request to the application if one is to implement MPIX_I(all)reduce_x with the same semantics as MPI_Iallreduce, as I aspire to do in BigMPI. Issue #1: chopping doesn't work for nonblocking. To do the large-count reduction in one nonblocking MPI call, a derived datatype is required. However, unlike in RMA, reductions cannot use built-in ops for user-defined datatypes, even if they are trivially composed of a large-count of built-in datatypes. See https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/338 and https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/34 for elaborate commentary on why this semantic mismatch is lame. Issue #2: cannot use built-in reduce ops. Once we rule out using built-in ops with our large-count datatypes, we must reimplement all of the reduction operations required. I find this to be nontrivial. I have not yet figured out how to get at the underlying datatype info in a simple manner. It appears that MPI_Type_get_envelope exists for this purpose, but it's a huge pain to have to call this function when all I need to know is the number of built-in datatypes so that I can apply my clever and use MPI_Reduce_local inside of my user-defined operation. Issue #3: implementing the user-defined reduce op isn't easy (in my opinion). Many MPI implementations optimize reductions. On Blue Gene/Q, MPI has explicitly vectorized intrinsic/assembly code. Unless MPI_Reduce_local hits that code path, I am losing a huge amount of performance in reductions when I go from 2^31 to 2^31+1 elements. I would not be surprised at all if user-defined ops+datatypes exercises suboptimal code paths in many MPI implementations, which means that the performance of nonblocking reductions is unnecessarily crippled. Issue #4: inability to use optimizations in the MPI implementation. # Conclusion I believe this problem is best addressed in one of two ways: 1) Approve the semantic changes requested in tickets 34 and 338 so that one can use built-in ops with homogeneous user-defined datatypes. This is my preference for multiple reasons. 2) Add large-count reductions to the standard. This means 8 new functions: blocking and nonblocking (all)reduce and reduce_scatter(_block). We don't need large-count functions for any other collectives because the datatype solution works just fine there, as I've already demonstrated in BigMPI (https://github.com/jeffhammond/BigMPI/blob/master/src/collectives_x.c) # Social Commentary >From now on, when the Forum punts on things and says it's no problem for users to roll their own using the existing functionality in MPI, we should strive to be a bit more diligent and actually prototype that implementation in a manner that proves how easy it is for users. It turns out, writing code for some things is harder than just talking about them in a conference room... # Related https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/338#comment:9 captures some of this feedback in Trac. I created https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/423 for the reasons described therein. -- Jeff Hammond jeff.science(a)gmail.com

1 0

BigMPI: MPI_Count support for communication functions
by Jeff Hammond 02 May '14

02 May '14

In reference to https://github.com/jeffhammond/BigMPI... # TL;DNR version: I have implemented large-count wrappers for p2p, bcast, (all){gather,scatter}, alltoall and rma, which are the most obvious functions where large-count support would be useful. This is very much a work-in-progress and the unit tests are incomplete. However, I want some people to critique it before I get too far along. # Technical details and associated commentary/background: Background: The Forum declined to add functions in MPI-3 to support large counts, instead encouraging users to work around the INT_MAX limitation of C int counts with datatypes and providing the minimum required functionality (i.e. datatype query functions) to make this possible. Motivation: Users don't like it when the Forum punts on things like this (e.g. http://gentryx.de/news_the_troubling_state_of_MPI.html) Apparently Pavan told some Europeans that I was going to solve this and so I had no choice but to defend my honor and actually finish that project. Interface: The API follows the pattern of MPI_Type_size_x w.r.t. "s/int/MPI_Count/" for the count type, but where I use MPIX since these functions are non-standard. Currently I support only C but Fortran is on the TODO list. Limitations: Even though MPI_Count might be 128b, I am only supporting 64b counts (because of MPI_Aint limitations and desire to use size_t in my unit tests), so BigMPI is not going to solve your problem if you want to communicate more than 8 EiB of data in a single message. If you have more than 8 EiB of memory and a 1+ PiB/s interconnect in your system, please let me know so that I can use it :-) Supported Functions: I believe that p2p, rma and bcast are the only functions worth supporting but I added some other collectives anyways. The other collectives clearly aren't scalable because they are going to move O(nproc*count) data, which is obviously a lot of memory in the case of e.g. allgather of >2GiB for more than a few dozen procs. The v-collectives are a pain because I have to duplicate the count vector, so I will not support these unless >1 users claim it is necessary. The w-collectives have always been ridiculous and I will not support them unless paid in real money to do it. # Technical details https://github.com/jeffhammond/BigMPI/blob/master/src/type_contiguous_x.c does the heavy lifting. It's pretty obvious how it works. Pavan tells me that the datatypes engine will turn this into a contiguous datatype internally and thus it will be efficient. MPI implementations need to be count-safe for this to work, but they need to be count-safe period if the Forum is serious about datatypes being the solution rather than MPI_Count everywhere. All of the communication functions follow the same pattern, which is clearly seen in MPIX_Send_x at the top of https://github.com/jeffhammond/BigMPI/blob/master/src/sendrecv_x.c. I've optimized for the common case of count<2^31 with a likely_if macro (because you clearly care about latency if your app ships around >2GiB payloads...). The most obvious optimization I can see doing is to implement MPIX_Type_contiguous_x using internals of MPICH and possibly some other implementation instead of calling eight MPI datatype functions. I already started working on MPIX_Type_contiguous_x in MPICH anyways and will try to get that upstream in the near future. I don't know enough OpenMPI guts to do that one right now but I'm sure somebody involved in that project can figure it out in short order. # Feedback and such I would really appreciate people telling me everything that I am doing wrong w.r.t. BigMPI (but not other things - there just isn't enough time :-) ). Please also help me prioritize further development (other than a complete set of unit tests) by telling me features are missing. And of course, let me know if the design is stupid or the implementation is buggy. While email works great, I am much more susceptible to developer shame if you file bug reports and feature requests on Github (https://github.com/jeffhammond/BigMPI/issues) Finally, please feel free to use the Github fork button and/or git-format-patch. IMPORTANT NOTE: You need to undefine BIGMPI_MAX_INT in src/Makefile and test/bigmpi/Makefile to do serious testing. I set this value artificially low to make for productive debugging and this is what is committed right now. # Future Work There is a large potential user base if I provide a Fortran 77/90 interface that works with -i8 (or equivalent). I do not care about satisfying the PMPI requirements here and will just drop from MPIX Fortran into the MPIX C functions. BigMPI does not do a great job of error checking, particularly to confirm the implementation is count-safe. I will add those soon. Obviously, I need to finish the unit tests. And so far, I have only run them with an artificial INT_MAX of 10^6 because my laptop becomes unresponsive if I allocate 4GB. I need to hope on a workstation and do proper >2^31 count testing. I apologize in advance if you beat me to this and find obvious bugs. The docs aren't complete but that's high on the priority list. # Humorous addendum Please, please refer to this as THE solution for Big Data w.r.t. MPI. I totally love Big Data hype. My proposed title for the Jeff Squyres Cisco blog post on this is "You can haz moar counts!" :-) # Conclusion Anyways, thanks in advance for any feedback. And a big thank you to Pavan for preliminary discussions and Andreas Schäfer for blog-shaming me into working on BigMPI again. Best, Jeff -- Jeff Hammond jeff.science(a)gmail.com

1 0