May 2014 - mpi-forum - mailman3.cels.anl.gov

Ticket #419 (errata): formal reading
by Jeff Squyres (jsquyres) 16 May '14

16 May '14

MPI Forum -- (Rolf R. is on vacation; I am sending out this T-2 week notice for him) Ticket 419 (errata) will be formally read in Chicago: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/419 It is a small correction to the definition of neighborhood collectives. -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

First Cut at June Agenda
by Martin Schulz 12 May '14

12 May '14

Hi all, A first agenda for the June meeting is available on the forum webpage. This is based on the feedback I got from the WG chairs so far. Please let me know if I missed anything or if anything is wrong. Also, note that the agenda contains several errata readings, ticket readings and votes (which are hereby announced). Please also double check that those are correct - if anything is missing, please let me know this week - the official cut-off is May 19th 2014 (i.e., next Monday two weeks before the forum meeting). Thanks! Martin ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://scalability.llnl.gov/ CASC @ Lawrence Livermore National Laboratory, Livermore, USA

1 0

Re: [Mpi-forum] Problem with Hotel Logistics for June Forum Meeting
by Martin Schulz 11 May '14

11 May '14

Thanks, Pavan, for checking! Martin ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://scalability.llnl.gov/ CASC @ Lawrence Livermore National Laboratory, Livermore, USA On 5/10/14, 11:48 PM, "Balaji, Pavan" <balaji(a)anl.gov> wrote: > >I checked at Argonne and our conference center is not available for >public events during those dates. Sorry. > > Pavan > >On May 11, 2014, at 1:32 AM, Martin Schulz <schulzm(a)llnl.gov> wrote: > >> Hi all, >> >> Based on the initial positive response, Rich looked into the hotel >>options around O¹Hare, but apparently that doesn¹t seem feasible. The >>quote he got at the hotel we have been using for previous events, and >>which used to be one of the more reasonable ones, was that we would have >>to buy food for $2500 per day, i.e., we would have to spend $10K for the >>four days, which is a bit too high (and that didn¹t even count power >>strips and projectors, yet). >> >> With that we are back to the Microsoft downtown option at least we >>know we have great facilities there, but unfortunately this means there >>won¹t be any help with hotels this time. I hope this doesn¹t cause any >>insurmountable problems for anyone - if so, please let me know. >> >> Thanks, >> >> Martin >> >> ________________________________________________________________________ >> Martin Schulz, schulzm(a)llnl.gov, http://scalability.llnl.gov/ >> CASC @ Lawrence Livermore National Laboratory, Livermore, USA >> >> >> From: Martin Schulz <schulzm(a)llnl.gov> >> Date: Tuesday, May 6, 2014 at 11:37 AM >> To: Main mailing list <mpi-forum(a)lists.mpi-forum.org> >> Subject: Problem with Hotel Logistics for June Forum Meeting >> >> Hi all, >> >> We are having some problems with the hotels for the June meeting see >>Rich¹s message below and won¹t be able to get a room block. How many >>people would be impacted by that, i.e., couldn¹t find a decent hotel >>under these circumstances? >> >> In general, I think we have three options: >> We can keep the meeting at Microsoft as usual, but this could mean >>for some people to potentially commute from further out to the meeting >>(worst case from O¹Hare) >> We can try to have the meeting at an airport hotel where the >>facilities are less ideal and we have to pay a higher cost from our >>leftover funds >> ANL has previously offered to host, so if this still an option we >>could consider that as well, but this requires everyone to get rental >>cars >> Do people have preferences? Unless I hear a clear vote for one option, >>though, I think we should stick with option 1 to avoid confusion and >>have the least impact on budgets (forum and travelers). >> >> Thanks, >> >> Martin >> >> >> ________________________________________________________________________ >> Martin Schulz, schulzm(a)llnl.gov, http://scalability.llnl.gov/ >> CASC @ Lawrence Livermore National Laboratory, Livermore, USA >> >> >> From: "richardg(a)mellanox.com" <richardg(a)mellanox.com> >> Date: Tuesday, May 6, 2014 at 11:17 AM >> To: Anh Vo <Anh.Vo(a)microsoft.com>, Martin Schulz <schulz6(a)llnl.gov> >> Subject: RE: hotel for MPI Forum in June >> >> We have been looking into it, and l don¹t think we will be able to come >>up with a hotel. There is some big meeting down town with hotels not >>having availability at least for blocks of rooms at a discount rate. >>For the fun of it I checked on gov rate availability in Marriott chains, >>and same issue (that is the rate we usually get). We could get hotels >>out towards the airport, but I do not think that is practical. >> >> One thing we could consider is having the meeting at the Spring Hill >>Suites out by the airport not as nice a location, but last time we met >>there they gave us free meeting rooms for staying with them and for >>catering. Not sure if there is a room count requirement with it. >>Martin, what do you think ? >> >> Rich >> >> _______________________________________________ >> mpi-forum mailing list >> mpi-forum(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > >_______________________________________________ >mpi-forum mailing list >mpi-forum(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum

1 0

Re: [Mpi-forum] Problem with Hotel Logistics for June Forum Meeting
by Martin Schulz 11 May '14

11 May '14

Hi all, Based on the initial positive response, Rich looked into the hotel options around O¹Hare, but apparently that doesn¹t seem feasible. The quote he got at the hotel we have been using for previous events, and which used to be one of the more reasonable ones, was that we would have to buy food for $2500 per day, i.e., we would have to spend $10K for the four days, which is a bit too high (and that didn¹t even count power strips and projectors, yet). With that we are back to the Microsoft downtown option at least we know we have great facilities there, but unfortunately this means there won¹t be any help with hotels this time. I hope this doesn¹t cause any insurmountable problems for anyone - if so, please let me know. Thanks, Martin ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://scalability.llnl.gov/ CASC @ Lawrence Livermore National Laboratory, Livermore, USA From: Martin Schulz <schulzm(a)llnl.gov> Date: Tuesday, May 6, 2014 at 11:37 AM To: Main mailing list <mpi-forum(a)lists.mpi-forum.org> Subject: Problem with Hotel Logistics for June Forum Meeting Hi all, We are having some problems with the hotels for the June meeting see Rich¹s message below and won¹t be able to get a room block. How many people would be impacted by that, i.e., couldn¹t find a decent hotel under these circumstances? In general, I think we have three options: * We can keep the meeting at Microsoft as usual, but this could mean for some people to potentially commute from further out to the meeting (worst case from O¹Hare) * We can try to have the meeting at an airport hotel where the facilities are less ideal and we have to pay a higher cost from our leftover funds * ANL has previously offered to host, so if this still an option we could consider that as well, but this requires everyone to get rental cars Do people have preferences? Unless I hear a clear vote for one option, though, I think we should stick with option 1 to avoid confusion and have the least impact on budgets (forum and travelers). Thanks, Martin ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://scalability.llnl.gov/ CASC @ Lawrence Livermore National Laboratory, Livermore, USA From: "richardg(a)mellanox.com" <richardg(a)mellanox.com> Date: Tuesday, May 6, 2014 at 11:17 AM To: Anh Vo <Anh.Vo(a)microsoft.com>, Martin Schulz <schulz6(a)llnl.gov> Subject: RE: hotel for MPI Forum in June We have been looking into it, and l don¹t think we will be able to come up with a hotel. There is some big meeting down town with hotels not having availability at least for blocks of rooms at a discount rate. For the fun of it I checked on gov rate availability in Marriott chains, and same issue (that is the rate we usually get). We could get hotels out towards the airport, but I do not think that is practical. One thing we could consider is having the meeting at the Spring Hill Suites out by the airport not as nice a location, but last time we met there they gave us free meeting rooms for staying with them and for catering. Not sure if there is a room count requirement with it. Martin, what do you think ? Rich

1 0

Re: [Mpi-forum] Problem with Hotel Logistics for June Forum Meeting
by Balaji, Pavan 10 May '14

10 May '14

I checked at Argonne and our conference center is not available for public events during those dates. Sorry. — Pavan On May 11, 2014, at 1:32 AM, Martin Schulz <schulzm(a)llnl.gov> wrote: > Hi all, > > Based on the initial positive response, Rich looked into the hotel options around O’Hare, but apparently that doesn’t seem feasible. The quote he got at the hotel we have been using for previous events, and which used to be one of the more reasonable ones, was that we would have to buy food for $2500 per day, i.e., we would have to spend $10K for the four days, which is a bit too high (and that didn’t even count power strips and projectors, yet). > > With that we are back to the Microsoft downtown option – at least we know we have great facilities there, but unfortunately this means there won’t be any help with hotels this time. I hope this doesn’t cause any insurmountable problems for anyone - if so, please let me know. > > Thanks, > > Martin > > ________________________________________________________________________ > Martin Schulz, schulzm(a)llnl.gov, http://scalability.llnl.gov/ > CASC @ Lawrence Livermore National Laboratory, Livermore, USA > > > From: Martin Schulz <schulzm(a)llnl.gov> > Date: Tuesday, May 6, 2014 at 11:37 AM > To: Main mailing list <mpi-forum(a)lists.mpi-forum.org> > Subject: Problem with Hotel Logistics for June Forum Meeting > > Hi all, > > We are having some problems with the hotels for the June meeting – see Rich’s message below – and won’t be able to get a room block. How many people would be impacted by that, i.e., couldn’t find a decent hotel under these circumstances? > > In general, I think we have three options: > • We can keep the meeting at Microsoft as usual, but this could mean for some people to potentially commute from further out to the meeting (worst case from O’Hare) > • We can try to have the meeting at an airport hotel where the facilities are less ideal and we have to pay a higher cost from our leftover funds > • ANL has previously offered to host, so if this still an option we could consider that as well, but this requires everyone to get rental cars > Do people have preferences? Unless I hear a clear vote for one option, though, I think we should stick with option 1 to avoid confusion and have the least impact on budgets (forum and travelers). > > Thanks, > > Martin > > > ________________________________________________________________________ > Martin Schulz, schulzm(a)llnl.gov, http://scalability.llnl.gov/ > CASC @ Lawrence Livermore National Laboratory, Livermore, USA > > > From: "richardg(a)mellanox.com" <richardg(a)mellanox.com> > Date: Tuesday, May 6, 2014 at 11:17 AM > To: Anh Vo <Anh.Vo(a)microsoft.com>, Martin Schulz <schulz6(a)llnl.gov> > Subject: RE: hotel for MPI Forum in June > > We have been looking into it, and l don’t think we will be able to come up with a hotel. There is some big meeting down town with hotels not having availability at least for blocks of rooms at a discount rate. For the fun of it I checked on gov rate availability in Marriott chains, and same issue (that is the rate we usually get). We could get hotels out towards the airport, but I do not think that is practical. > > One thing we could consider is having the meeting at the Spring Hill Suites out by the airport – not as nice a location, but last time we met there they gave us free meeting rooms for staying with them and for catering. Not sure if there is a room count requirement with it. Martin, what do you think ? > > Rich > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum

1 0

Re: [Mpi-forum] large count support not as easy as people seem to have thought
by Moody, Adam T. 06 May '14

06 May '14

Hi Jeff, Ok, thanks for the clarification. Constraining the uses makes sense. I had been considering the general case. And I was also worried about dealing with 128-bit file systems. I can imagine there may be some cases for really large datatypes in MPI I/O on 128-bit file systems, especially dealing with file views. BTW, I did convince myself that you can at least handlecontiguous types of an arbitrary length type(k) with an inductive proof. Base case: type(1) == this is a predefined type Inductive step: Assume all types 1-k can be created, now consider k+1: if k+1 is even type(k+1) == type_contiguous(2, type(k/2)) if k+1 is odd lens = {1,1} disps = {0, extent(type(1))} types = {type(1), type(k)} type(k+1) == type_struct(2, lens, disps, types) since extent(type(1)) is the extent of a predefined type, it is less than MPI_Aint -Adam ________________________________________ From: mpi-forum [mpi-forum-bounces(a)lists.mpi-forum.org] on behalf of Jeff Hammond [jeff.science(a)gmail.com] Sent: Tuesday, May 06, 2014 7:03 PM To: Main MPI Forum mailing list Subject: Re: [Mpi-forum] large count support not as easy as people seem to have thought Hi Adam, First, you've hit on an important shortcoming of BigMPI, which is that I do not document its limitations in enough detail, specifically w.r.t. int/MPI_Aint/MPI_Count. However, the README, which is on the home page, clearly states that BigMPI will not support counts larger than fit into MPI_Aint, since I believe this is impossible. I honestly do not see how 32b MPI_Aint is a valid issue with BigMPI. If you have a 32b address space, how are you going to allocate and index the buffer that spills the count? I am only supporting communication operations, not I/O ones, so there is no need to consider the global address space size. I suppose, in theory, you could have a 4GiB address space and be unable to do an in-place reduction on a 2^31+1 element array of MPI_CHAR, but does anyone care about this corner case? So, to be explicit, BigMPI is only going to support systems where sizeof(int)=4 and sizeof(MPI_Aint)=8, and where the user passes count that is less than 2^63. I believe this is the _only_ use case worth caring about. As I wrote in the README, if you've got a machine with more than 2^63 bytes of memory in it - even globally - please let me know so I can run NWChem on it :-) Finally, I do not believe anyone is going to build a supercomputer that has a 32b address space in the future. Blue Gene/P is going to be the last of its kind. There just isn't any rational for having 32b addresses if there are more than 4GiB of memory per node. It is already on my TODO list to add an autotools build system, which is going to allow me to detect all of the relevant type sizes and make sure the user doesn't try to do anything silly. Best, Jeff On Tue, May 6, 2014 at 8:37 PM, Adam T. Moody <moody20(a)llnl.gov> wrote: > Hi Jeff, > Thanks so much for taking on this challenge! > > I think things are even worse than you've noted. In fact, it may be > impossible. > > I had considered this a bit once before, and when I ran a gedankenexperiment > on my own MPI_Type_contiguous_x, I got caught up in all of the complexities > of portably* dealing with the various integer types. > > Just in this one function, you need to deal with int, MPI_Aint, and > MPI_Count. In your particular implementation, you have to be sure that you > don't overflow "int" in the call to MPI_Type_vector and you have to be sure > you don't overflow MPI_Aint in the call to MPI_Type_create_struct. > > https://github.com/jeffhammond/BigMPI/blob/master/src/type_contiguous_x.c#L… > > It gets ugly because MPI_Count can be much larger than the other two types. > MPI_Count has to be big enough to hold the largest value of int, MPI_Aint, > and MPI_Offset. MPI_Aint has to be large enough to represent every address > on a system, and int is of course often 32 bits but it's only guranteed to > be at least 16 bits as George pointed out. > > So now imagine MPI running on a very large system composed of many, many > small-memory cores. That's a very plausible future system design. On such > a system, it's possible that you may have 32-bit ints, 32-bit MPI_Aints > (small memory), but 128-bit MPI_Offsets. On such a system, MPI_Count / > (MAX_INT) would still be more than what fits in an int, so then you overflow > the first parameter to MPI_Type_vector. > > https://github.com/jeffhammond/BigMPI/blob/master/src/type_contiguous_x.c#L… > > Because MPI_Count can be much bigger than both int and MPI_Aint, I believe > this problem becomes very ugly (and perhaps impossible) to solve in a > portable way. Even implementing something as simple as > MPI_Type_contiguous_x is very cumbersome. > > I believe (but have not proved) that there are some large count values which > are *impossible* to represent with any combination of existing MPI calls due > to these limitations. Ultimately, you need to piece together arbitrary > types with something like MPI_Type_create_struct, but this takes an MPI_Aint > to express displacements from the front, so once you have types that are > well bigger than an MPI_Aint my guess is that there are probably some you > can't define at all. > -Adam > > > > Jeff Hammond wrote: > >> As I noted on this list last Friday, I am working on a higher-level >> library to support large counts for MPI communication functions >> (https://github.com/jeffhammond/BigMPI) >> >> In the course of actually trying to implement this the way the Forum >> contends it can be done - i.e. using derived-datatypes - I have found >> some issues that undermine the Forum's contention that it is so easy >> for users to do it that it doesn't need to be in the standard. >> >> To illustrate some of the issues that I have found, let us consider >> the large-count implementation of nonblocking reduce... >> >> # Example Use Case >> >> It is entirely reasonable to think that some quantum chemist will want >> to reduce a contiguous buffer of more than 2^31 doubles corresponding >> to the Fock matrix if they have a multithreaded code, since 16 GB is >> not an unreasonable amount of memory per node. >> >> # Issues >> >> Unlike the blocking case, where it is reasonable to chop up the data >> and performance multiple operations (e.g. >> https://github.com/jeffhammond/BigMPI/blob/master/src/reductions_x.c, >> assuming that untested code is correct), one must return a single >> request to the application if one is to implement MPIX_I(all)reduce_x >> with the same semantics as MPI_Iallreduce, as I aspire to do in >> BigMPI. >> >> Issue #1: chopping doesn't work for nonblocking. >> >> To do the large-count reduction in one nonblocking MPI call, a derived >> datatype is required. However, unlike in RMA, reductions cannot use >> built-in ops for user-defined datatypes, even if they are trivially >> composed of a large-count of built-in datatypes. See >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/338 and >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/34 for elaborate >> commentary on why this semantic mismatch is lame. >> >> Issue #2: cannot use built-in reduce ops. >> >> Once we rule out using built-in ops with our large-count datatypes, we >> must reimplement all of the reduction operations required. I find >> this to be nontrivial. I have not yet figured out how to get at the >> underlying datatype info in a simple manner. It appears that >> MPI_Type_get_envelope exists for this purpose, but it's a huge pain to >> have to call this function when all I need to know is the number of >> built-in datatypes so that I can apply my clever and use >> MPI_Reduce_local inside of my user-defined operation. >> >> Issue #3: implementing the user-defined reduce op isn't easy (in my >> opinion). >> >> Many MPI implementations optimize reductions. On Blue Gene/Q, MPI has >> explicitly vectorized intrinsic/assembly code. Unless >> MPI_Reduce_local hits that code path, I am losing a huge amount of >> performance in reductions when I go from 2^31 to 2^31+1 elements. I >> would not be surprised at all if user-defined ops+datatypes exercises >> suboptimal code paths in many MPI implementations, which means that >> the performance of nonblocking reductions is unnecessarily crippled. >> >> Issue #4: inability to use optimizations in the MPI implementation. >> >> # Conclusion >> >> I believe this problem is best addressed in one of two ways: >> >> 1) Approve the semantic changes requested in tickets 34 and 338 so >> that one can use built-in ops with homogeneous user-defined datatypes. >> This is my preference for multiple reasons. >> >> 2) Add large-count reductions to the standard. This means 8 new >> functions: blocking and nonblocking (all)reduce and >> reduce_scatter(_block). We don't need large-count functions for any >> other collectives because the datatype solution works just fine there, >> as I've already demonstrated in BigMPI >> (https://github.com/jeffhammond/BigMPI/blob/master/src/collectives_x.c) >> >> # Social Commentary >> >> From now on, when the Forum punts on things and says it's no problem >> for users to roll their own using the existing functionality in MPI, >> we should strive to be a bit more diligent and actually prototype that >> implementation in a manner that proves how easy it is for users. It >> turns out, writing code for some things is harder than just talking >> about them in a conference room... >> >> # Related >> >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/338#comment:9 >> captures some of this feedback in Trac. >> >> I created https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/423 for >> the reasons described therein. >> >> > > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Jeff Hammond jeff.science(a)gmail.com _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum

1 0

Re: [Mpi-forum] large count support not as easy as people seem to have thought
by Jeff Hammond 06 May '14

06 May '14

Hi Adam, First, you've hit on an important shortcoming of BigMPI, which is that I do not document its limitations in enough detail, specifically w.r.t. int/MPI_Aint/MPI_Count. However, the README, which is on the home page, clearly states that BigMPI will not support counts larger than fit into MPI_Aint, since I believe this is impossible. I honestly do not see how 32b MPI_Aint is a valid issue with BigMPI. If you have a 32b address space, how are you going to allocate and index the buffer that spills the count? I am only supporting communication operations, not I/O ones, so there is no need to consider the global address space size. I suppose, in theory, you could have a 4GiB address space and be unable to do an in-place reduction on a 2^31+1 element array of MPI_CHAR, but does anyone care about this corner case? So, to be explicit, BigMPI is only going to support systems where sizeof(int)=4 and sizeof(MPI_Aint)=8, and where the user passes count that is less than 2^63. I believe this is the _only_ use case worth caring about. As I wrote in the README, if you've got a machine with more than 2^63 bytes of memory in it - even globally - please let me know so I can run NWChem on it :-) Finally, I do not believe anyone is going to build a supercomputer that has a 32b address space in the future. Blue Gene/P is going to be the last of its kind. There just isn't any rational for having 32b addresses if there are more than 4GiB of memory per node. It is already on my TODO list to add an autotools build system, which is going to allow me to detect all of the relevant type sizes and make sure the user doesn't try to do anything silly. Best, Jeff On Tue, May 6, 2014 at 8:37 PM, Adam T. Moody <moody20(a)llnl.gov> wrote: > Hi Jeff, > Thanks so much for taking on this challenge! > > I think things are even worse than you've noted. In fact, it may be > impossible. > > I had considered this a bit once before, and when I ran a gedankenexperiment > on my own MPI_Type_contiguous_x, I got caught up in all of the complexities > of portably* dealing with the various integer types. > > Just in this one function, you need to deal with int, MPI_Aint, and > MPI_Count. In your particular implementation, you have to be sure that you > don't overflow "int" in the call to MPI_Type_vector and you have to be sure > you don't overflow MPI_Aint in the call to MPI_Type_create_struct. > > https://github.com/jeffhammond/BigMPI/blob/master/src/type_contiguous_x.c#L… > > It gets ugly because MPI_Count can be much larger than the other two types. > MPI_Count has to be big enough to hold the largest value of int, MPI_Aint, > and MPI_Offset. MPI_Aint has to be large enough to represent every address > on a system, and int is of course often 32 bits but it's only guranteed to > be at least 16 bits as George pointed out. > > So now imagine MPI running on a very large system composed of many, many > small-memory cores. That's a very plausible future system design. On such > a system, it's possible that you may have 32-bit ints, 32-bit MPI_Aints > (small memory), but 128-bit MPI_Offsets. On such a system, MPI_Count / > (MAX_INT) would still be more than what fits in an int, so then you overflow > the first parameter to MPI_Type_vector. > > https://github.com/jeffhammond/BigMPI/blob/master/src/type_contiguous_x.c#L… > > Because MPI_Count can be much bigger than both int and MPI_Aint, I believe > this problem becomes very ugly (and perhaps impossible) to solve in a > portable way. Even implementing something as simple as > MPI_Type_contiguous_x is very cumbersome. > > I believe (but have not proved) that there are some large count values which > are *impossible* to represent with any combination of existing MPI calls due > to these limitations. Ultimately, you need to piece together arbitrary > types with something like MPI_Type_create_struct, but this takes an MPI_Aint > to express displacements from the front, so once you have types that are > well bigger than an MPI_Aint my guess is that there are probably some you > can't define at all. > -Adam > > > > Jeff Hammond wrote: > >> As I noted on this list last Friday, I am working on a higher-level >> library to support large counts for MPI communication functions >> (https://github.com/jeffhammond/BigMPI) >> >> In the course of actually trying to implement this the way the Forum >> contends it can be done - i.e. using derived-datatypes - I have found >> some issues that undermine the Forum's contention that it is so easy >> for users to do it that it doesn't need to be in the standard. >> >> To illustrate some of the issues that I have found, let us consider >> the large-count implementation of nonblocking reduce... >> >> # Example Use Case >> >> It is entirely reasonable to think that some quantum chemist will want >> to reduce a contiguous buffer of more than 2^31 doubles corresponding >> to the Fock matrix if they have a multithreaded code, since 16 GB is >> not an unreasonable amount of memory per node. >> >> # Issues >> >> Unlike the blocking case, where it is reasonable to chop up the data >> and performance multiple operations (e.g. >> https://github.com/jeffhammond/BigMPI/blob/master/src/reductions_x.c, >> assuming that untested code is correct), one must return a single >> request to the application if one is to implement MPIX_I(all)reduce_x >> with the same semantics as MPI_Iallreduce, as I aspire to do in >> BigMPI. >> >> Issue #1: chopping doesn't work for nonblocking. >> >> To do the large-count reduction in one nonblocking MPI call, a derived >> datatype is required. However, unlike in RMA, reductions cannot use >> built-in ops for user-defined datatypes, even if they are trivially >> composed of a large-count of built-in datatypes. See >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/338 and >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/34 for elaborate >> commentary on why this semantic mismatch is lame. >> >> Issue #2: cannot use built-in reduce ops. >> >> Once we rule out using built-in ops with our large-count datatypes, we >> must reimplement all of the reduction operations required. I find >> this to be nontrivial. I have not yet figured out how to get at the >> underlying datatype info in a simple manner. It appears that >> MPI_Type_get_envelope exists for this purpose, but it's a huge pain to >> have to call this function when all I need to know is the number of >> built-in datatypes so that I can apply my clever and use >> MPI_Reduce_local inside of my user-defined operation. >> >> Issue #3: implementing the user-defined reduce op isn't easy (in my >> opinion). >> >> Many MPI implementations optimize reductions. On Blue Gene/Q, MPI has >> explicitly vectorized intrinsic/assembly code. Unless >> MPI_Reduce_local hits that code path, I am losing a huge amount of >> performance in reductions when I go from 2^31 to 2^31+1 elements. I >> would not be surprised at all if user-defined ops+datatypes exercises >> suboptimal code paths in many MPI implementations, which means that >> the performance of nonblocking reductions is unnecessarily crippled. >> >> Issue #4: inability to use optimizations in the MPI implementation. >> >> # Conclusion >> >> I believe this problem is best addressed in one of two ways: >> >> 1) Approve the semantic changes requested in tickets 34 and 338 so >> that one can use built-in ops with homogeneous user-defined datatypes. >> This is my preference for multiple reasons. >> >> 2) Add large-count reductions to the standard. This means 8 new >> functions: blocking and nonblocking (all)reduce and >> reduce_scatter(_block). We don't need large-count functions for any >> other collectives because the datatype solution works just fine there, >> as I've already demonstrated in BigMPI >> (https://github.com/jeffhammond/BigMPI/blob/master/src/collectives_x.c) >> >> # Social Commentary >> >> From now on, when the Forum punts on things and says it's no problem >> for users to roll their own using the existing functionality in MPI, >> we should strive to be a bit more diligent and actually prototype that >> implementation in a manner that proves how easy it is for users. It >> turns out, writing code for some things is harder than just talking >> about them in a conference room... >> >> # Related >> >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/338#comment:9 >> captures some of this feedback in Trac. >> >> I created https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/423 for >> the reasons described therein. >> >> > > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [Mpi-forum] large count support not as easy as people seem to have thought
by Adam T. Moody 06 May '14

06 May '14

Hi Jeff, Thanks so much for taking on this challenge! I think things are even worse than you've noted. In fact, it may be impossible. I had considered this a bit once before, and when I ran a gedankenexperiment on my own MPI_Type_contiguous_x, I got caught up in all of the complexities of portably* dealing with the various integer types. Just in this one function, you need to deal with int, MPI_Aint, and MPI_Count. In your particular implementation, you have to be sure that you don't overflow "int" in the call to MPI_Type_vector and you have to be sure you don't overflow MPI_Aint in the call to MPI_Type_create_struct. https://github.com/jeffhammond/BigMPI/blob/master/src/type_contiguous_x.c#L… It gets ugly because MPI_Count can be much larger than the other two types. MPI_Count has to be big enough to hold the largest value of int, MPI_Aint, and MPI_Offset. MPI_Aint has to be large enough to represent every address on a system, and int is of course often 32 bits but it's only guranteed to be at least 16 bits as George pointed out. So now imagine MPI running on a very large system composed of many, many small-memory cores. That's a very plausible future system design. On such a system, it's possible that you may have 32-bit ints, 32-bit MPI_Aints (small memory), but 128-bit MPI_Offsets. On such a system, MPI_Count / (MAX_INT) would still be more than what fits in an int, so then you overflow the first parameter to MPI_Type_vector. https://github.com/jeffhammond/BigMPI/blob/master/src/type_contiguous_x.c#L… Because MPI_Count can be much bigger than both int and MPI_Aint, I believe this problem becomes very ugly (and perhaps impossible) to solve in a portable way. Even implementing something as simple as MPI_Type_contiguous_x is very cumbersome. I believe (but have not proved) that there are some large count values which are *impossible* to represent with any combination of existing MPI calls due to these limitations. Ultimately, you need to piece together arbitrary types with something like MPI_Type_create_struct, but this takes an MPI_Aint to express displacements from the front, so once you have types that are well bigger than an MPI_Aint my guess is that there are probably some you can't define at all. -Adam Jeff Hammond wrote: >As I noted on this list last Friday, I am working on a higher-level >library to support large counts for MPI communication functions >(https://github.com/jeffhammond/BigMPI) > >In the course of actually trying to implement this the way the Forum >contends it can be done - i.e. using derived-datatypes - I have found >some issues that undermine the Forum's contention that it is so easy >for users to do it that it doesn't need to be in the standard. > >To illustrate some of the issues that I have found, let us consider >the large-count implementation of nonblocking reduce... > ># Example Use Case > >It is entirely reasonable to think that some quantum chemist will want >to reduce a contiguous buffer of more than 2^31 doubles corresponding >to the Fock matrix if they have a multithreaded code, since 16 GB is >not an unreasonable amount of memory per node. > ># Issues > >Unlike the blocking case, where it is reasonable to chop up the data >and performance multiple operations (e.g. >https://github.com/jeffhammond/BigMPI/blob/master/src/reductions_x.c, >assuming that untested code is correct), one must return a single >request to the application if one is to implement MPIX_I(all)reduce_x >with the same semantics as MPI_Iallreduce, as I aspire to do in >BigMPI. > >Issue #1: chopping doesn't work for nonblocking. > >To do the large-count reduction in one nonblocking MPI call, a derived >datatype is required. However, unlike in RMA, reductions cannot use >built-in ops for user-defined datatypes, even if they are trivially >composed of a large-count of built-in datatypes. See >https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/338 and >https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/34 for elaborate >commentary on why this semantic mismatch is lame. > >Issue #2: cannot use built-in reduce ops. > >Once we rule out using built-in ops with our large-count datatypes, we >must reimplement all of the reduction operations required. I find >this to be nontrivial. I have not yet figured out how to get at the >underlying datatype info in a simple manner. It appears that >MPI_Type_get_envelope exists for this purpose, but it's a huge pain to >have to call this function when all I need to know is the number of >built-in datatypes so that I can apply my clever and use >MPI_Reduce_local inside of my user-defined operation. > >Issue #3: implementing the user-defined reduce op isn't easy (in my opinion). > >Many MPI implementations optimize reductions. On Blue Gene/Q, MPI has >explicitly vectorized intrinsic/assembly code. Unless >MPI_Reduce_local hits that code path, I am losing a huge amount of >performance in reductions when I go from 2^31 to 2^31+1 elements. I >would not be surprised at all if user-defined ops+datatypes exercises >suboptimal code paths in many MPI implementations, which means that >the performance of nonblocking reductions is unnecessarily crippled. > >Issue #4: inability to use optimizations in the MPI implementation. > ># Conclusion > >I believe this problem is best addressed in one of two ways: > >1) Approve the semantic changes requested in tickets 34 and 338 so >that one can use built-in ops with homogeneous user-defined datatypes. > This is my preference for multiple reasons. > >2) Add large-count reductions to the standard. This means 8 new >functions: blocking and nonblocking (all)reduce and >reduce_scatter(_block). We don't need large-count functions for any >other collectives because the datatype solution works just fine there, >as I've already demonstrated in BigMPI >(https://github.com/jeffhammond/BigMPI/blob/master/src/collectives_x.c) > ># Social Commentary > >From now on, when the Forum punts on things and says it's no problem >for users to roll their own using the existing functionality in MPI, >we should strive to be a bit more diligent and actually prototype that >implementation in a manner that proves how easy it is for users. It >turns out, writing code for some things is harder than just talking >about them in a conference room... > ># Related > >https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/338#comment:9 >captures some of this feedback in Trac. > >I created https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/423 for >the reasons described therein. > > >

1 0

Re: [Mpi-forum] large count support not as easy as people seem to have thought
by Jed Brown 06 May '14

06 May '14

Rob Latham <robl(a)mcs.anl.gov> writes: > i imagine one day (once again?) the c int type will be 64 bits on a > platform, right? > > or will c ints be 32 bits for ever? 2^64 is quite a lot of memory. The popular 64-bit systems use LP64 or LLP64, both of which have 32-bit ints. I don't foresee any major player having incentive to promote int to 64-bits. I think it's worth fixing this interface problem _and_ fixing the MPI_Op support so that libraries can build robust abstractions. (I have example use cases in PETSc that caused a bit of discussion including the creation of ticket #338, and now Jeff has another.)

1 0

Re: [Mpi-forum] large count support not as easy as people seem to have thought
by George Bosilca 06 May '14

06 May '14

The size of an int is implementation dependent with a minimum size requirement (16b).It is not imposed by any standard. A nice summary is available @ http://en.wikipedia.org/wiki/C_data_types George. On Tue, May 6, 2014 at 5:34 PM, Rob Latham <robl(a)mcs.anl.gov> wrote: > > > On 05/06/2014 04:14 PM, Jeff Hammond wrote: >> >> On Tue, May 6, 2014 at 3:54 PM, Hjelm, Nathan T <hjelmn(a)lanl.gov> wrote: >>> >>> +1 for large counts. I find it just a bit ridiculous that this was punted >>> on by the forum but I wasn't around for the discussion. Is this in issue >>> forced on C/C++ by fortran? >> >> >> Unfortunately, I don't think we can blame this one on Fortran. Users >> want to pass large counts from the C interface. That's the motivation >> for BigMPI, at least. > > > i imagine one day (once again?) the c int type will be 64 bits on a > platform, right? > > or will c ints be 32 bits for ever? > > ==rob > > >> >> The Fortran users that want -i8 and friends to work with the MPI >> Fortran interface are another issue that doesn't need new functions in >> the standard, just a bunch of interface gymnastics that most >> implementers won't enjoy. >> >> Best, >> >> Jeff >> >>> As I noted on this list last Friday, I am working on a higher-level >>> library to support large counts for MPI communication functions >>> (https://github.com/jeffhammond/BigMPI) >>> >>> In the course of actually trying to implement this the way the Forum >>> contends it can be done - i.e. using derived-datatypes - I have found >>> some issues that undermine the Forum's contention that it is so easy >>> for users to do it that it doesn't need to be in the standard. >>> >>> To illustrate some of the issues that I have found, let us consider >>> the large-count implementation of nonblocking reduce... >>> >>> # Example Use Case >>> >>> It is entirely reasonable to think that some quantum chemist will want >>> to reduce a contiguous buffer of more than 2^31 doubles corresponding >>> to the Fock matrix if they have a multithreaded code, since 16 GB is >>> not an unreasonable amount of memory per node. >>> >>> # Issues >>> >>> Unlike the blocking case, where it is reasonable to chop up the data >>> and performance multiple operations (e.g. >>> https://github.com/jeffhammond/BigMPI/blob/master/src/reductions_x.c, >>> assuming that untested code is correct), one must return a single >>> request to the application if one is to implement MPIX_I(all)reduce_x >>> with the same semantics as MPI_Iallreduce, as I aspire to do in >>> BigMPI. >>> >>> Issue #1: chopping doesn't work for nonblocking. >>> >>> To do the large-count reduction in one nonblocking MPI call, a derived >>> datatype is required. However, unlike in RMA, reductions cannot use >>> built-in ops for user-defined datatypes, even if they are trivially >>> composed of a large-count of built-in datatypes. See >>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/338 and >>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/34 for elaborate >>> commentary on why this semantic mismatch is lame. >>> >>> Issue #2: cannot use built-in reduce ops. >>> >>> Once we rule out using built-in ops with our large-count datatypes, we >>> must reimplement all of the reduction operations required. I find >>> this to be nontrivial. I have not yet figured out how to get at the >>> underlying datatype info in a simple manner. It appears that >>> MPI_Type_get_envelope exists for this purpose, but it's a huge pain to >>> have to call this function when all I need to know is the number of >>> built-in datatypes so that I can apply my clever and use >>> MPI_Reduce_local inside of my user-defined operation. >>> >>> Issue #3: implementing the user-defined reduce op isn't easy (in my >>> opinion). >>> >>> Many MPI implementations optimize reductions. On Blue Gene/Q, MPI has >>> explicitly vectorized intrinsic/assembly code. Unless >>> MPI_Reduce_local hits that code path, I am losing a huge amount of >>> performance in reductions when I go from 2^31 to 2^31+1 elements. I >>> would not be surprised at all if user-defined ops+datatypes exercises >>> suboptimal code paths in many MPI implementations, which means that >>> the performance of nonblocking reductions is unnecessarily crippled. >>> >>> Issue #4: inability to use optimizations in the MPI implementation. >>> >>> # Conclusion >>> >>> I believe this problem is best addressed in one of two ways: >>> >>> 1) Approve the semantic changes requested in tickets 34 and 338 so >>> that one can use built-in ops with homogeneous user-defined datatypes. >>> This is my preference for multiple reasons. >>> >>> 2) Add large-count reductions to the standard. This means 8 new >>> functions: blocking and nonblocking (all)reduce and >>> reduce_scatter(_block). We don't need large-count functions for any >>> other collectives because the datatype solution works just fine there, >>> as I've already demonstrated in BigMPI >>> (https://github.com/jeffhammond/BigMPI/blob/master/src/collectives_x.c) >>> >>> # Social Commentary >>> >>> From now on, when the Forum punts on things and says it's no problem >>> for users to roll their own using the existing functionality in MPI, >>> we should strive to be a bit more diligent and actually prototype that >>> implementation in a manner that proves how easy it is for users. It >>> turns out, writing code for some things is harder than just talking >>> about them in a conference room... >>> >>> # Related >>> >>> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/338#comment:9 >>> captures some of this feedback in Trac. >>> >>> I created https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/423 for >>> the reasons described therein. >>> >>> -- >>> Jeff Hammond >>> jeff.science(a)gmail.com >>> _______________________________________________ >>> mpi-forum mailing list >>> mpi-forum(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum >>> _______________________________________________ >>> mpi-forum mailing list >>> mpi-forum(a)lists.mpi-forum.org >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum >> >> >> >> > > -- > Rob Latham > Mathematics and Computer Science Division > Argonne National Lab, IL USA > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum

1 0