- mpiwg-coll - mailman3.cels.anl.gov

Re: [mpi3-coll] Telecon to discuss DV-collectives (Alltoalldv)
by Torsten Hoefler 13 Oct '11

13 Oct '11

Hi Adam, > Soon after we decided to request alltoallv to be added to the dv ticket, > I realized there is one important difference between this and the > dynamic sparse data exchange (DSDE) case. With alltoallv, the receiver > knows which ranks it will recieve data from, but it doesn't with DSDE. > > I think for alltoalldv, you just need each process to provide two lists: > a send list and receive list. Where the current API looks like this: > > MPI_Alltoallv( > sendbuf, sendcounts[], sdispls[], sendtype, /* O(P) list */ > recvbuf, recvcounts[], rdispls[], rectype, /* O(P) list */ > comm > ); > > Provide a new O(k) interface like so (have to add a count to each list > to give its length, and a list of ranks): > > MPI_Alltoalldv( > nsends, sendbuf, sendranks[], sendcounts[], sdispls[], sendtype, /* > O(k) list */ > nrecvs, recvbuf, recvranks[], recvcounts[], rdispls[], rectype, /* > O(k) list */ > comm > ); Yes, this would probably be the simplest solution, however, it requires the user to specify k and the receiver list. As we know, this is sometimes hard. So I was hoping to solve the more general problem of DSDE with this interface. However, I agree that this departs from MPI semantics and since it requires dynamic memory allocation, it may not be a good choice for MPI. Bu I was hoping to discuss those issues at the telecon. Putting one of those interfaces into the draft document is simple (once we agree on how much we want to provide :-). Thanks & All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [mpi3-coll] Telecon to discuss DV-collectives (Alltoalldv)
by Supalov, Alexander 13 Oct '11

13 Oct '11

Hi, To the 2nd item: we had a comparable proposal for the conventional V ops in the past (see https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/23) Best regards. Alexander -----Original Message----- From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Adam T. Moody Sent: Thursday, October 13, 2011 7:56 PM To: MPI-3 Collective Subgroup Discussions Subject: Re: [mpi3-coll] Telecon to discuss DV-collectives (Alltoalldv) A couple more things... In Santorini, Rich brought up a couple of concerns that should be considered. For one, he suggested that a slightly more general interface might be better in which you specify a base count for all processes, and then provide a list for processes that are different than that count. This could subsume the interface I listed below if you set the base count to be 0 and then list each non-zero item. The nice thing about the base count approach is that it nicely handles the "mostly regular" case, in which nearly all procs have the same amount of data but only a few have a little more or a little less. For example, this interface might look something like the following (added basecount and removed displacements which need some thought here): MPI_Alltoalldv( sendbuf, sbasecount, nsends, sendranks[], sendcounts[], sendtype, /* O(sbasecount*P + k) list */ recvbuf, rbasecount, nrecvs, recvranks[], recvcounts[], rectype, /* O(rbasecount*P + k) list */ comm ); In the above, you would send/receive basecount data items from all procs, except for a few ranks, whose counts are listed explicitly in O(k) lists. Setting sbasecount/rbasecount=0 essentially reduces this interface to the one below (ignoring displacements). The other thing Rich was concerned about was whether an interface could be specified in such a way to reduce communication of the distributed count values. One extension that might help here is to allow the application to provide minimum and maximum count values that would apply globally across all processes. For example, with a maximum count value in gatherdv, you could set up a tree expecting the maxcount from all children but just receive less during each step. On the otherhand, if the min and max values are far apart, the implementation might fall back to something more dynamic so it doesn't allocate a bunch of temporary memory that it'll never use. If an application can't specify minimum or maximum values, it could always pass MPI_UNDEFINED for the min/max values. -Adam Adam T. Moody wrote: >Hi Torsten, >Soon after we decided to request alltoallv to be added to the dv ticket, >I realized there is one important difference between this and the >dynamic sparse data exchange (DSDE) case. With alltoallv, the receiver >knows which ranks it will recieve data from, but it doesn't with DSDE. > >I think for alltoalldv, you just need each process to provide two lists: >a send list and receive list. Where the current API looks like this: > >MPI_Alltoallv( > sendbuf, sendcounts[], sdispls[], sendtype, /* O(P) list */ > recvbuf, recvcounts[], rdispls[], rectype, /* O(P) list */ > comm >); > >Provide a new O(k) interface like so (have to add a count to each list >to give its length, and a list of ranks): > >MPI_Alltoalldv( > nsends, sendbuf, sendranks[], sendcounts[], sdispls[], sendtype, /* >O(k) list */ > nrecvs, recvbuf, recvranks[], recvcounts[], rdispls[], rectype, /* >O(k) list */ > comm >); > >I think the interface you're pondering would solve the tougher problem >of DSDE. >-Adam > > >Torsten Hoefler wrote: > > > >>Hello Coll-WG, >> >>At the last meeting, we decided to push the scalable (dv) collective >>proposal further towards a reading. The present forum members were >>rather clearly supporting the proposal by straw-vote. >> >>We also decided to include alltoalldv in the ticket, a call where every >>sender specifies the destinations it sends to as a list. We did not >>discuss the specification of the receive buffer though. If we force this >>to be if size P blocks (for P processes in the comm, and a block being >>count*sizeof(extent datatype)), then we're back to non-scalable again. I >>see the following alternatives: >> >>1) MPI allocates memory for the received blocks and returns a list of >> nodes where it received from and the allocated buffer with the >> received data >>2) the user allocates a buffer of size N (<=P) and provides it to the >> MPI library, the library fills the buffer and returns a list of >> source nodes. If a process received from more than N nodes, the call >> fails (MSG truncated). >>3) the user specifies a callback function for each received block :-) >> >>I prefer 3, however, this has the same issues as active messages and >>other callbacks and will most likely be discussed to death. 2 seems thus >>most reasonable. Does anybody have another proposal? >> >>We may want to split the ticket into two parts (separating out >>alltoalldv). >> >>I think we should have a quick (~30 mins) telecon to discuss this >>matter. Please indicate your availability in the following doodle before >>Friday 10/7 if you're interested to participate in the discussion. >> >>http://www.doodle.com/4wkqnsgi8nfhfdw3 >> >>The ticket is https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/264 . >> >>Thanks & Best, >> Torsten Hoefler >> >> >> >> >> > >_______________________________________________ >mpi3-coll mailing list >mpi3-coll(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll > > _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll -------------------------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen, Deutschland Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr./VAT Registration No.: DE129385895 Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052

1 0

Re: [mpi3-coll] Telecon to discuss DV-collectives (Alltoalldv)
by Adam T. Moody 13 Oct '11

13 Oct '11

A couple more things... In Santorini, Rich brought up a couple of concerns that should be considered. For one, he suggested that a slightly more general interface might be better in which you specify a base count for all processes, and then provide a list for processes that are different than that count. This could subsume the interface I listed below if you set the base count to be 0 and then list each non-zero item. The nice thing about the base count approach is that it nicely handles the "mostly regular" case, in which nearly all procs have the same amount of data but only a few have a little more or a little less. For example, this interface might look something like the following (added basecount and removed displacements which need some thought here): MPI_Alltoalldv( sendbuf, sbasecount, nsends, sendranks[], sendcounts[], sendtype, /* O(sbasecount*P + k) list */ recvbuf, rbasecount, nrecvs, recvranks[], recvcounts[], rectype, /* O(rbasecount*P + k) list */ comm ); In the above, you would send/receive basecount data items from all procs, except for a few ranks, whose counts are listed explicitly in O(k) lists. Setting sbasecount/rbasecount=0 essentially reduces this interface to the one below (ignoring displacements). The other thing Rich was concerned about was whether an interface could be specified in such a way to reduce communication of the distributed count values. One extension that might help here is to allow the application to provide minimum and maximum count values that would apply globally across all processes. For example, with a maximum count value in gatherdv, you could set up a tree expecting the maxcount from all children but just receive less during each step. On the otherhand, if the min and max values are far apart, the implementation might fall back to something more dynamic so it doesn't allocate a bunch of temporary memory that it'll never use. If an application can't specify minimum or maximum values, it could always pass MPI_UNDEFINED for the min/max values. -Adam Adam T. Moody wrote: >Hi Torsten, >Soon after we decided to request alltoallv to be added to the dv ticket, >I realized there is one important difference between this and the >dynamic sparse data exchange (DSDE) case. With alltoallv, the receiver >knows which ranks it will recieve data from, but it doesn't with DSDE. > >I think for alltoalldv, you just need each process to provide two lists: >a send list and receive list. Where the current API looks like this: > >MPI_Alltoallv( > sendbuf, sendcounts[], sdispls[], sendtype, /* O(P) list */ > recvbuf, recvcounts[], rdispls[], rectype, /* O(P) list */ > comm >); > >Provide a new O(k) interface like so (have to add a count to each list >to give its length, and a list of ranks): > >MPI_Alltoalldv( > nsends, sendbuf, sendranks[], sendcounts[], sdispls[], sendtype, /* >O(k) list */ > nrecvs, recvbuf, recvranks[], recvcounts[], rdispls[], rectype, /* >O(k) list */ > comm >); > >I think the interface you're pondering would solve the tougher problem >of DSDE. >-Adam > > >Torsten Hoefler wrote: > > > >>Hello Coll-WG, >> >>At the last meeting, we decided to push the scalable (dv) collective >>proposal further towards a reading. The present forum members were >>rather clearly supporting the proposal by straw-vote. >> >>We also decided to include alltoalldv in the ticket, a call where every >>sender specifies the destinations it sends to as a list. We did not >>discuss the specification of the receive buffer though. If we force this >>to be if size P blocks (for P processes in the comm, and a block being >>count*sizeof(extent datatype)), then we're back to non-scalable again. I >>see the following alternatives: >> >>1) MPI allocates memory for the received blocks and returns a list of >> nodes where it received from and the allocated buffer with the >> received data >>2) the user allocates a buffer of size N (<=P) and provides it to the >> MPI library, the library fills the buffer and returns a list of >> source nodes. If a process received from more than N nodes, the call >> fails (MSG truncated). >>3) the user specifies a callback function for each received block :-) >> >>I prefer 3, however, this has the same issues as active messages and >>other callbacks and will most likely be discussed to death. 2 seems thus >>most reasonable. Does anybody have another proposal? >> >>We may want to split the ticket into two parts (separating out >>alltoalldv). >> >>I think we should have a quick (~30 mins) telecon to discuss this >>matter. Please indicate your availability in the following doodle before >>Friday 10/7 if you're interested to participate in the discussion. >> >>http://www.doodle.com/4wkqnsgi8nfhfdw3 >> >>The ticket is https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/264 . >> >>Thanks & Best, >> Torsten Hoefler >> >> >> >> >> > >_______________________________________________ >mpi3-coll mailing list >mpi3-coll(a)lists.mpi-forum.org >http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll > >

1 0

Re: [mpi3-coll] Telecon to discuss DV-collectives (Alltoalldv)
by Adam T. Moody 13 Oct '11

13 Oct '11

Hi Torsten, Soon after we decided to request alltoallv to be added to the dv ticket, I realized there is one important difference between this and the dynamic sparse data exchange (DSDE) case. With alltoallv, the receiver knows which ranks it will recieve data from, but it doesn't with DSDE. I think for alltoalldv, you just need each process to provide two lists: a send list and receive list. Where the current API looks like this: MPI_Alltoallv( sendbuf, sendcounts[], sdispls[], sendtype, /* O(P) list */ recvbuf, recvcounts[], rdispls[], rectype, /* O(P) list */ comm ); Provide a new O(k) interface like so (have to add a count to each list to give its length, and a list of ranks): MPI_Alltoalldv( nsends, sendbuf, sendranks[], sendcounts[], sdispls[], sendtype, /* O(k) list */ nrecvs, recvbuf, recvranks[], recvcounts[], rdispls[], rectype, /* O(k) list */ comm ); I think the interface you're pondering would solve the tougher problem of DSDE. -Adam Torsten Hoefler wrote: >Hello Coll-WG, > >At the last meeting, we decided to push the scalable (dv) collective >proposal further towards a reading. The present forum members were >rather clearly supporting the proposal by straw-vote. > >We also decided to include alltoalldv in the ticket, a call where every >sender specifies the destinations it sends to as a list. We did not >discuss the specification of the receive buffer though. If we force this >to be if size P blocks (for P processes in the comm, and a block being >count*sizeof(extent datatype)), then we're back to non-scalable again. I >see the following alternatives: > >1) MPI allocates memory for the received blocks and returns a list of > nodes where it received from and the allocated buffer with the > received data >2) the user allocates a buffer of size N (<=P) and provides it to the > MPI library, the library fills the buffer and returns a list of > source nodes. If a process received from more than N nodes, the call > fails (MSG truncated). >3) the user specifies a callback function for each received block :-) > >I prefer 3, however, this has the same issues as active messages and >other callbacks and will most likely be discussed to death. 2 seems thus >most reasonable. Does anybody have another proposal? > >We may want to split the ticket into two parts (separating out >alltoalldv). > >I think we should have a quick (~30 mins) telecon to discuss this >matter. Please indicate your availability in the following doodle before >Friday 10/7 if you're interested to participate in the discussion. > >http://www.doodle.com/4wkqnsgi8nfhfdw3 > >The ticket is https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/264 . > >Thanks & Best, > Torsten Hoefler > > >

1 0

[mpi3-coll] Telecon to discuss DV-collectives (Alltoalldv)
by Torsten Hoefler 03 Oct '11

03 Oct '11

Hello Coll-WG, At the last meeting, we decided to push the scalable (dv) collective proposal further towards a reading. The present forum members were rather clearly supporting the proposal by straw-vote. We also decided to include alltoalldv in the ticket, a call where every sender specifies the destinations it sends to as a list. We did not discuss the specification of the receive buffer though. If we force this to be if size P blocks (for P processes in the comm, and a block being count*sizeof(extent datatype)), then we're back to non-scalable again. I see the following alternatives: 1) MPI allocates memory for the received blocks and returns a list of nodes where it received from and the allocated buffer with the received data 2) the user allocates a buffer of size N (<=P) and provides it to the MPI library, the library fills the buffer and returns a list of source nodes. If a process received from more than N nodes, the call fails (MSG truncated). 3) the user specifies a callback function for each received block :-) I prefer 3, however, this has the same issues as active messages and other callbacks and will most likely be discussed to death. 2 seems thus most reasonable. Does anybody have another proposal? We may want to split the ticket into two parts (separating out alltoalldv). I think we should have a quick (~30 mins) telecon to discuss this matter. Please indicate your availability in the following doodle before Friday 10/7 if you're interested to participate in the discussion. http://www.doodle.com/4wkqnsgi8nfhfdw3 The ticket is https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/264 . Thanks & Best, Torsten Hoefler -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [mpi3-coll] Meeting in Helsinki
by Torsten Hoefler 03 Sep '09

03 Sep '09

Hello again, we have another schedule change. It seems like we need more time for MPI 2.2. Thus, we moved the meeting to tomorrow morning 9am Helsinki time. It'll be 8am German time and 2am Eastern. Please join (the guys on Pacific time will actually have a reasonable time :). All the Best, Torsten The new information is: Topic: MPI-3 Collective WG meeting Date: Friday, September 4, 2009 Time: 9:00 am, Northern Europe Summer Time (Helsinki, GMT+03:00) Meeting Number: 208 343 013 Meeting Password: mpi3 ------------------------------------------------------- To join the online meeting ------------------------------------------------------- 1. Go to https://cisco.webex.com/ciscosales/j.php?ED=124941567&UID=1174472542&PW=NNT… 2. Enter your name and email address. 3. Enter the meeting password: mpi3 4. Click "Join Now". ------------------------------------------------------- To join the teleconference only ------------------------------------------------------- 1. Dial into Cisco WebEx (view all Global Access Numbers at [2]http://cisco.com/en/US/about/doing_business/conferencing/index.html 2. Press 3 to attend the meeting. 3. Follow the prompts to enter the Meeting Number (listed above) or Access Code followed by the # sign. San Jose, CA: +1.408.525.6800 RTP: +1.919.392.3330 US/Canada: +1.866.432.9903 United Kingdom: +44.20.8824.0117 India: +91.80.4350.1111 Germany: +49.619.6773.9002 Japan: +81.3.5763.9394 China: +86.10.8515.5666 -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Fellow Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Meeting in Helsinki
by Torsten Hoefler 03 Sep '09

03 Sep '09

Hello Group, so we will have a webex meeting for the collectives group. Due to different MPI 2.2-related delays, we scheduled the meeting to 3:30-4:30 local time (please attend!). This would be 2:30 German time and 8:30 Eastern time. The access information is: Topic: MPI Forum MPI-3 Collectives WG Date: Thursday, September 3, 2009 Time: 3:30 pm, Northern Europe Summer Time (Helsinki, GMT+03:00) Meeting Number: 202 336 638 Meeting Password: mpi3 Web Link: ------------------------------------------------------- To join the online meeting ------------------------------------------------------- 1. Go to https://cisco.webex.com/ciscosales/j.php?ED=124769332&UID=1172973142&PW=NNz… 2. Enter your name and email address. 3. Enter the meeting password: mpi3 4. Click "Join Now". Phone: ------------------------------------------------------- To join the teleconference only ------------------------------------------------------- 1. Dial into Cisco WebEx (view all Global Access Numbers at [2]http://cisco.com/en/US/about/doing_business/conferencing/index.html 2. Press 3 to attend the meeting. 3. Follow the prompts to enter the Meeting Number (listed above) or Access Code followed by the # sign. San Jose, CA: +1.408.525.6800 RTP: +1.919.392.3330 US/Canada: +1.866.432.9903 United Kingdom: +44.20.8824.0117 India: +91.80.4350.1111 Germany: +49.619.6773.9002 Japan: +81.3.5763.9394 China: +86.10.8515.5666 All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Fellow Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] Meeting in Helsinki
by Jesper-Larsson Träff 26 Aug '09

26 Aug '09

Hi all, I'll not attend the Forum meeting (but will be in Helsinki for the conference), but would in principle be interested in joining a teleconf. Sometime in the afternoon would work fine for me Jesper On Mon, Aug 24, 2009 at 01:59:57PM -0400, Torsten Hoefler wrote: > Hello Collectives-WG, > I think we should get back to our regular schedule after all MPI-2.2 > goals have been reached. I propose to use the meeting in Helsinki to > discuss our ongoing efforts and decide what we want to focus on. > > However, I am not sure how many people are planning to attend the > meeting. Could you please let me know if you're going to attend? I am > wondering if it would be useful to try to schedule a telecon (or webex) > while during the meeting in Helsinki. Would this be possible during our > scheduled time (9/3, 1pm-3pm)? This seems to be pretty early for the CA > time. Rich, can we re-schedule the meeting in case we decide to have a > telecon? > > I put a tentative agenda at: > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/TeleconSept > > I also updated the WG's goals in the Wiki > (https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/CollectivesWikiPage) > they have been there since the very beginning. I added an itemized list. > Please feel free to extend or correct the current goals. > > Thanks and all the Best, > Torsten > > -- > bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- > Torsten Hoefler | Postdoctoral Fellow > Open Systems Lab | Indiana University > 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA > Lindley Hall Room 135 | +01 (812) 855-3608 > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll

1 0

[mpi3-coll] Meeting in Helsinki
by Torsten Hoefler 24 Aug '09

24 Aug '09

Hello Collectives-WG, I think we should get back to our regular schedule after all MPI-2.2 goals have been reached. I propose to use the meeting in Helsinki to discuss our ongoing efforts and decide what we want to focus on. However, I am not sure how many people are planning to attend the meeting. Could you please let me know if you're going to attend? I am wondering if it would be useful to try to schedule a telecon (or webex) while during the meeting in Helsinki. Would this be possible during our scheduled time (9/3, 1pm-3pm)? This seems to be pretty early for the CA time. Rich, can we re-schedule the meeting in case we decide to have a telecon? I put a tentative agenda at: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/TeleconSept I also updated the WG's goals in the Wiki (https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/CollectivesWikiPage) they have been there since the very beginning. I added an itemized list. Please feel free to extend or correct the current goals. Thanks and all the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Fellow Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

[mpi3-coll] Sameer Kumar is out of the office.
by Sameer Kumar 03 Aug '09

03 Aug '09

I will be out of the office starting 07/30/2009 and will not return until 08/20/2009. I will only have very limited email access. I will respond to your message when I return.

1 0