October 2011 - mpiwg-coll - mailman3.cels.anl.gov

[mpi3-coll] Missing reduce_local and reduce_scatter_block in create_op text
by Adam T. Moody 28 Oct '11

28 Oct '11

Hi Torsten, There are a few lists that are missing MPI_REDUCE_LOCAL and MPI_REDUCE_SCATTER_BLOCK. 1) Section 5.9.2 List of predefined ops missing MPI_REDUCE_LOCAL, MPI_ALLREDUCE_SCATTER_BLOCK, and all non-blocking variants. 2) Description for MPI_OP_CREATE missing MPI_REDUCE_LOCAL, MPI_REDUCE_SCATTER_BLOCK, and all non-blocking variants. I guess we may not need to list the non-blocking variants in this list, but we should definitely add REDUCE_LOCAL and REDUCE_SCATTER_BLOCK. -Adam

1 0

Re: [mpi3-coll] Scalable Vector Collectives & Next Telecon
by Torsten Hoefler 27 Oct '11

27 Oct '11

Sorry for the double email -- just to make this clear: we are planning a reading in the January meeting, so please take the review seriously! Thanks & Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

[mpi3-coll] Scalable Vector Collectives & Next Telecon
by Torsten Hoefler 27 Oct '11

27 Oct '11

Hello Coll-WG, Christian and I finished the draft of the scalable collectives proposal. We incorporated all changes that we discussed during the last Forum meeting. The two major changes were: 1) added MPI_(I)Alltoalldw 2) removed the displs arguments The newest version of the chapter including the proposal is attached to ticket 264 (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/264) Everybody in the chapter committee (the signup sheet we did on Tuesday), please review the chapter carefully and sign off at the ticket. We will schedule a telecon in the week of either 11/28, 12/5, or 12/12. If you have a certain preference, please let me know, otherwise, I'll pick a random week for a doodle. Thanks & All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [mpi3-coll] Telecon to discuss DV-collectives (Alltoalldv)
by Torsten Hoefler 14 Oct '11

14 Oct '11

Just a quick reminder -- the telecon is going on now! Torsten On Thu, Oct 13, 2011 at 07:28:09PM -0500, Torsten Hoefler wrote: > Hello Coll-WG, > > The doodle converged to have the call tomorrow 3:30-4pm Central time. > > We will be discussing Ticket 264 and the addition of alltoalldv to it > (the detailed interface). Please review the email discussions from > today. The ticket is https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/264 . > > The call-in information is: > > Scheduled Start Time: 03:30 PM Central Daylight Time > Dial-in Number: 1-940-287-4000?(Texas) > Participant Access Code: 8966928 > > Talk to you tomorrow & Best, > Torsten Hoefler > > -- > bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- > Torsten Hoefler | Performance Modeling and Simulation Lead > Blue Waters Directorate | University of Illinois (UIUC) > 1205 W Clark Street | Urbana, IL, 61801 > NCSA Building | +01 (217) 244-7736 > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Niemand k�me auf die Idee, Tinte mit Tinte abzuwaschen; nur Blut soll immer wieder mit Blut abgewaschen werden. [Marie von Ebner-Eschenbach]

1 0

Re: [mpi3-coll] Telecon to discuss DV-collectives (Alltoalldv)
by Torsten Hoefler 13 Oct '11

13 Oct '11

Hello Coll-WG, The doodle converged to have the call tomorrow 3:30-4pm Central time. We will be discussing Ticket 264 and the addition of alltoalldv to it (the detailed interface). Please review the email discussions from today. The ticket is https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/264 . The call-in information is: Scheduled Start Time: 03:30 PM Central Daylight Time Dial-in Number: 1-940-287-4000�(Texas) Participant Access Code: 8966928 Talk to you tomorrow & Best, Torsten Hoefler -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [mpi3-coll] Telecon to discuss DV-collectives (Alltoalldv)
by Torsten Hoefler 13 Oct '11

13 Oct '11

Hi Alexander, > To the 2nd item: we had a comparable proposal for the conventional V > ops in the past (see > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/23) It seems like this proposal requires still all P entries in all arrays and thus has the \Omega(P) time and space overhead that we're trying to push down to O(k) (while we assume that k=O(log(P)) at very large scale). All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [mpi3-coll] Telecon to discuss DV-collectives (Alltoalldv)
by Torsten Hoefler 13 Oct '11

13 Oct '11

On Thu, Oct 13, 2011 at 10:55:47AM -0700, Adam T. Moody wrote: > A couple more things... :-) -- more comments below! > In Santorini, Rich brought up a couple of concerns that should be > considered. For one, he suggested that a slightly more general > interface might be better in which you specify a base count for all > processes, and then provide a list for processes that are different than > that count. This could subsume the interface I listed below if you set > the base count to be 0 and then list each non-zero item. The nice thing > about the base count approach is that it nicely handles the "mostly > regular" case, in which nearly all procs have the same amount of data > but only a few have a little more or a little less. For example, this > interface might look something like the following (added basecount and > removed displacements which need some thought here): > > MPI_Alltoalldv( > sendbuf, sbasecount, nsends, sendranks[], sendcounts[], sendtype, /* > O(sbasecount*P + k) list */ > recvbuf, rbasecount, nrecvs, recvranks[], recvcounts[], rectype, /* > O(rbasecount*P + k) list */ > comm > ); > > In the above, you would send/receive basecount data items from all > procs, except for a few ranks, whose counts are listed explicitly in > O(k) lists. Setting sbasecount/rbasecount=0 essentially reduces this > interface to the one below (ignoring displacements). Yes, that may be interesting for the use-case I have in mind (a 3d FFT on a circular (cutoff) region), however, may be too limited for other use cases (which I don't know). > The other thing Rich was concerned about was whether an interface could > be specified in such a way to reduce communication of the distributed > count values. One extension that might help here is to allow the > application to provide minimum and maximum count values that would apply > globally across all processes. For example, with a maximum count value > in gatherdv, you could set up a tree expecting the maxcount from all > children but just receive less during each step. On the otherhand, if > the min and max values are far apart, the implementation might fall back > to something more dynamic so it doesn't allocate a bunch of temporary > memory that it'll never use. If an application can't specify minimum or > maximum values, it could always pass MPI_UNDEFINED for the min/max > values. Yes, such tricks may simplify some implementation options. I'm not 100% sure how well they integrate with the current MPI syntax and semantics :-). But I guess recv is also only specifying a maximum. Keep in mind that this discussion may bring us then back to the displs[] array. Thanks & Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [mpi3-coll] Telecon to discuss DV-collectives (Alltoalldv)
by Torsten Hoefler 13 Oct '11

13 Oct '11

Hi Adam, > Soon after we decided to request alltoallv to be added to the dv ticket, > I realized there is one important difference between this and the > dynamic sparse data exchange (DSDE) case. With alltoallv, the receiver > knows which ranks it will recieve data from, but it doesn't with DSDE. > > I think for alltoalldv, you just need each process to provide two lists: > a send list and receive list. Where the current API looks like this: > > MPI_Alltoallv( > sendbuf, sendcounts[], sdispls[], sendtype, /* O(P) list */ > recvbuf, recvcounts[], rdispls[], rectype, /* O(P) list */ > comm > ); > > Provide a new O(k) interface like so (have to add a count to each list > to give its length, and a list of ranks): > > MPI_Alltoalldv( > nsends, sendbuf, sendranks[], sendcounts[], sdispls[], sendtype, /* > O(k) list */ > nrecvs, recvbuf, recvranks[], recvcounts[], rdispls[], rectype, /* > O(k) list */ > comm > ); Yes, this would probably be the simplest solution, however, it requires the user to specify k and the receiver list. As we know, this is sometimes hard. So I was hoping to solve the more general problem of DSDE with this interface. However, I agree that this departs from MPI semantics and since it requires dynamic memory allocation, it may not be a good choice for MPI. Bu I was hoping to discuss those issues at the telecon. Putting one of those interfaces into the draft document is simple (once we agree on how much we want to provide :-). Thanks & All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [mpi3-coll] Telecon to discuss DV-collectives (Alltoalldv)
by Supalov, Alexander 13 Oct '11

13 Oct '11

Hi, To the 2nd item: we had a comparable proposal for the conventional V ops in the past (see https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/23) Best regards. Alexander -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Adam T. Moody Sent: Thursday, October 13, 2011 7:56 PM To: MPI-3 Collective Subgroup Discussions Subject: Re: [mpi3-coll] Telecon to discuss DV-collectives (Alltoalldv) A couple more things... In Santorini, Rich brought up a couple of concerns that should be considered. For one, he suggested that a slightly more general interface might be better in which you specify a base count for all processes, and then provide a list for processes that are different than that count. This could subsume the interface I listed below if you set the base count to be 0 and then list each non-zero item. The nice thing about the base count approach is that it nicely handles the "mostly regular" case, in which nearly all procs have the same amount of data but only a few have a little more or a little less. For example, this interface might look something like the following (added basecount and removed displacements which need some thought here): MPI_Alltoalldv( sendbuf, sbasecount, nsends, sendranks[], sendcounts[], sendtype, /* O(sbasecount*P + k) list */ recvbuf, rbasecount, nrecvs, recvranks[], recvcounts[], rectype, /* O(rbasecount*P + k) list */ comm ); In the above, you would send/receive basecount data items from all procs, except for a few ranks, whose counts are listed explicitly in O(k) lists. Setting sbasecount/rbasecount=0 essentially reduces this interface to the one below (ignoring displacements). The other thing Rich was concerned about was whether an interface could be specified in such a way to reduce communication of the distributed count values. One extension that might help here is to allow the application to provide minimum and maximum count values that would apply globally across all processes. For example, with a maximum count value in gatherdv, you could set up a tree expecting the maxcount from all children but just receive less during each step. On the otherhand, if the min and max values are far apart, the implementation might fall back to something more dynamic so it doesn't allocate a bunch of temporary memory that it'll never use. If an application can't specify minimum or maximum values, it could always pass MPI_UNDEFINED for the min/max values. -Adam Adam T. Moody wrote: >Hi Torsten, >Soon after we decided to request alltoallv to be added to the dv ticket, >I realized there is one important difference between this and the >dynamic sparse data exchange (DSDE) case. With alltoallv, the receiver >knows which ranks it will recieve data from, but it doesn't with DSDE. > >I think for alltoalldv, you just need each process to provide two lists: >a send list and receive list. Where the current API looks like this: > >MPI_Alltoallv( > sendbuf, sendcounts[], sdispls[], sendtype, /* O(P) list */ > recvbuf, recvcounts[], rdispls[], rectype, /* O(P) list */ > comm >); > >Provide a new O(k) interface like so (have to add a count to each list >to give its length, and a list of ranks): > >MPI_Alltoalldv( > nsends, sendbuf, sendranks[], sendcounts[], sdispls[], sendtype, /* >O(k) list */ > nrecvs, recvbuf, recvranks[], recvcounts[], rdispls[], rectype, /* >O(k) list */ > comm >); > >I think the interface you're pondering would solve the tougher problem >of DSDE. >-Adam > > >Torsten Hoefler wrote: > > > >>Hello Coll-WG, >> >>At the last meeting, we decided to push the scalable (dv) collective >>proposal further towards a reading. The present forum members were >>rather clearly supporting the proposal by straw-vote. >> >>We also decided to include alltoalldv in the ticket, a call where every >>sender specifies the destinations it sends to as a list. We did not >>discuss the specification of the receive buffer though. If we force this >>to be if size P blocks (for P processes in the comm, and a block being >>count*sizeof(extent datatype)), then we're back to non-scalable again. I >>see the following alternatives: >> >>1) MPI allocates memory for the received blocks and returns a list of >> nodes where it received from and the allocated buffer with the >> received data >>2) the user allocates a buffer of size N (<=P) and provides it to the >> MPI library, the library fills the buffer and returns a list of >> source nodes. If a process received from more than N nodes, the call >> fails (MSG truncated). >>3) the user specifies a callback function for each received block :-) >> >>I prefer 3, however, this has the same issues as active messages and >>other callbacks and will most likely be discussed to death. 2 seems thus >>most reasonable. Does anybody have another proposal? >> >>We may want to split the ticket into two parts (separating out >>alltoalldv). >> >>I think we should have a quick (~30 mins) telecon to discuss this >>matter. Please indicate your availability in the following doodle before >>Friday 10/7 if you're interested to participate in the discussion. >> >>http://www.doodle.com/4wkqnsgi8nfhfdw3 >> >>The ticket is https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/264 . >> >>Thanks & Best, >> Torsten Hoefler >> >> >> >> >> > >_______________________________________________ >mpi3-coll mailing list >mpi3-coll(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll > > _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll -------------------------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen, Deutschland Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr./VAT Registration No.: DE129385895 Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052

1 0

Re: [mpi3-coll] Telecon to discuss DV-collectives (Alltoalldv)
by Adam T. Moody 13 Oct '11

13 Oct '11

A couple more things... In Santorini, Rich brought up a couple of concerns that should be considered. For one, he suggested that a slightly more general interface might be better in which you specify a base count for all processes, and then provide a list for processes that are different than that count. This could subsume the interface I listed below if you set the base count to be 0 and then list each non-zero item. The nice thing about the base count approach is that it nicely handles the "mostly regular" case, in which nearly all procs have the same amount of data but only a few have a little more or a little less. For example, this interface might look something like the following (added basecount and removed displacements which need some thought here): MPI_Alltoalldv( sendbuf, sbasecount, nsends, sendranks[], sendcounts[], sendtype, /* O(sbasecount*P + k) list */ recvbuf, rbasecount, nrecvs, recvranks[], recvcounts[], rectype, /* O(rbasecount*P + k) list */ comm ); In the above, you would send/receive basecount data items from all procs, except for a few ranks, whose counts are listed explicitly in O(k) lists. Setting sbasecount/rbasecount=0 essentially reduces this interface to the one below (ignoring displacements). The other thing Rich was concerned about was whether an interface could be specified in such a way to reduce communication of the distributed count values. One extension that might help here is to allow the application to provide minimum and maximum count values that would apply globally across all processes. For example, with a maximum count value in gatherdv, you could set up a tree expecting the maxcount from all children but just receive less during each step. On the otherhand, if the min and max values are far apart, the implementation might fall back to something more dynamic so it doesn't allocate a bunch of temporary memory that it'll never use. If an application can't specify minimum or maximum values, it could always pass MPI_UNDEFINED for the min/max values. -Adam Adam T. Moody wrote: >Hi Torsten, >Soon after we decided to request alltoallv to be added to the dv ticket, >I realized there is one important difference between this and the >dynamic sparse data exchange (DSDE) case. With alltoallv, the receiver >knows which ranks it will recieve data from, but it doesn't with DSDE. > >I think for alltoalldv, you just need each process to provide two lists: >a send list and receive list. Where the current API looks like this: > >MPI_Alltoallv( > sendbuf, sendcounts[], sdispls[], sendtype, /* O(P) list */ > recvbuf, recvcounts[], rdispls[], rectype, /* O(P) list */ > comm >); > >Provide a new O(k) interface like so (have to add a count to each list >to give its length, and a list of ranks): > >MPI_Alltoalldv( > nsends, sendbuf, sendranks[], sendcounts[], sdispls[], sendtype, /* >O(k) list */ > nrecvs, recvbuf, recvranks[], recvcounts[], rdispls[], rectype, /* >O(k) list */ > comm >); > >I think the interface you're pondering would solve the tougher problem >of DSDE. >-Adam > > >Torsten Hoefler wrote: > > > >>Hello Coll-WG, >> >>At the last meeting, we decided to push the scalable (dv) collective >>proposal further towards a reading. The present forum members were >>rather clearly supporting the proposal by straw-vote. >> >>We also decided to include alltoalldv in the ticket, a call where every >>sender specifies the destinations it sends to as a list. We did not >>discuss the specification of the receive buffer though. If we force this >>to be if size P blocks (for P processes in the comm, and a block being >>count*sizeof(extent datatype)), then we're back to non-scalable again. I >>see the following alternatives: >> >>1) MPI allocates memory for the received blocks and returns a list of >> nodes where it received from and the allocated buffer with the >> received data >>2) the user allocates a buffer of size N (<=P) and provides it to the >> MPI library, the library fills the buffer and returns a list of >> source nodes. If a process received from more than N nodes, the call >> fails (MSG truncated). >>3) the user specifies a callback function for each received block :-) >> >>I prefer 3, however, this has the same issues as active messages and >>other callbacks and will most likely be discussed to death. 2 seems thus >>most reasonable. Does anybody have another proposal? >> >>We may want to split the ticket into two parts (separating out >>alltoalldv). >> >>I think we should have a quick (~30 mins) telecon to discuss this >>matter. Please indicate your availability in the following doodle before >>Friday 10/7 if you're interested to participate in the discussion. >> >>http://www.doodle.com/4wkqnsgi8nfhfdw3 >> >>The ticket is https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/264 . >> >>Thanks & Best, >> Torsten Hoefler >> >> >> >> >> > >_______________________________________________ >mpi3-coll mailing list >mpi3-coll(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll > >

1 0