August 2009 - mpiwg-coll - mailman3.cels.anl.gov

Re: [mpi3-coll] Meeting in Helsinki
by Jesper-Larsson Träff 26 Aug '09

26 Aug '09

Hi all, I'll not attend the Forum meeting (but will be in Helsinki for the conference), but would in principle be interested in joining a teleconf. Sometime in the afternoon would work fine for me Jesper On Mon, Aug 24, 2009 at 01:59:57PM -0400, Torsten Hoefler wrote: > Hello Collectives-WG, > I think we should get back to our regular schedule after all MPI-2.2 > goals have been reached. I propose to use the meeting in Helsinki to > discuss our ongoing efforts and decide what we want to focus on. > > However, I am not sure how many people are planning to attend the > meeting. Could you please let me know if you're going to attend? I am > wondering if it would be useful to try to schedule a telecon (or webex) > while during the meeting in Helsinki. Would this be possible during our > scheduled time (9/3, 1pm-3pm)? This seems to be pretty early for the CA > time. Rich, can we re-schedule the meeting in case we decide to have a > telecon? > > I put a tentative agenda at: > https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/TeleconSept > > I also updated the WG's goals in the Wiki > (https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/CollectivesWikiPage) > they have been there since the very beginning. I added an itemized list. > Please feel free to extend or correct the current goals. > > Thanks and all the Best, > Torsten > > -- > bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- > Torsten Hoefler | Postdoctoral Fellow > Open Systems Lab | Indiana University > 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA > Lindley Hall Room 135 | +01 (812) 855-3608 > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll

1 0

[mpi3-coll] Meeting in Helsinki
by Torsten Hoefler 24 Aug '09

24 Aug '09

Hello Collectives-WG, I think we should get back to our regular schedule after all MPI-2.2 goals have been reached. I propose to use the meeting in Helsinki to discuss our ongoing efforts and decide what we want to focus on. However, I am not sure how many people are planning to attend the meeting. Could you please let me know if you're going to attend? I am wondering if it would be useful to try to schedule a telecon (or webex) while during the meeting in Helsinki. Would this be possible during our scheduled time (9/3, 1pm-3pm)? This seems to be pretty early for the CA time. Rich, can we re-schedule the meeting in case we decide to have a telecon? I put a tentative agenda at: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/TeleconSept I also updated the WG's goals in the Wiki (https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/CollectivesWikiPage) they have been there since the very beginning. I added an itemized list. Please feel free to extend or correct the current goals. Thanks and all the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Fellow Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

[mpi3-coll] Sameer Kumar is out of the office.
by Sameer Kumar 03 Aug '09

03 Aug '09

I will be out of the office starting 07/30/2009 and will not return until 08/20/2009. I will only have very limited email access. I will respond to your message when I return.

1 0

Re: [mpi3-coll] array parameters in nonblocking collectives
by Graham, Richard L. 01 Aug '09

01 Aug '09

The only issue I would have with this is that this forces the users to keep these objects in scope for the duration of the communication, which sort of defeats the intent of the nonblocking collectives. Having said that, the same issue holds for the data, and I would not suggest that we keep a copy of the data, so I realize I am advocating an inconsistent view point, so this may not be so good an idea. Rich On 8/1/09 11:33 AM, "Anthony Skjellum" <tony(a)runtimecomputing.com> wrote: It appears to me that mpi should temporarily "control"(own) the passed by reference data structures from the point the call is made to the time of completion... Users should accept this as part of the contact and shouldn't assume they own this data during the call's lifetime... From a state transition diagram perspective, ownership passes to and from mpi at ifunction and wait call, respectively. Tony Skjellum, PhD RunTime Computing Solutions, LLC tony(a)runtimecomputing.com direct: +1-205-314-3595 cell: +1-205-807-4968 -----Original Message----- From: "Rajeev Thakur" <thakur(a)mcs.anl.gov> Date: Sat, 1 Aug 2009 10:18:41 To: 'MPI-3 Collective Subgroup Discussions'<mpi3-coll(a)lists.mpi-forum.org> Subject: Re: [mpi3-coll] array parameters in nonblocking collectives _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll

1 0

Re: [mpi3-coll] array parameters in nonblocking collectives
by Torsten Hoefler 01 Aug '09

01 Aug '09

On Sat, Aug 01, 2009 at 10:54:20AM -0500, Rajeev Thakur wrote: > Torsten, > What does libNBC do in this case? hah, I forgot to say that :). LibNBC generates a collective schedule that encodes all steps in the init (MPI_I*) call. Thus, it simply copies the arguments in the schedule. This technique allows for tree-like (or butterfly) methods to be used in vector variants reducing the needed space per process to O(log(P)). However, they can become arbitrarily slow (log(P) large messages can be serialized at one process) with an adversarial vector distribution. Thus, I implemented the less scalable method that needs O(P) space per process (it'll eliminate all zeros) which can not loose parallelism or degenerate. Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Fellow Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] array parameters in nonblocking collectives
by Rajeev Thakur 01 Aug '09

01 Aug '09

Torsten, What does libNBC do in this case? Rajeev > -----Original Message----- > From: mpi3-coll-bounces(a)lists.mpi-forum.org > [mailto:[email protected]] On Behalf Of > Torsten Hoefler > Sent: Saturday, August 01, 2009 10:46 AM > To: MPI-3 Collective Subgroup Discussions > Subject: Re: [mpi3-coll] array parameters in nonblocking collectives > > Hello Bin, > thanks for bringing this up Bin, it seems like an important issue. > > I can see both sides. In general, I would say that all vector > collectives are not scalable anyway because they require > \Theta(P) memory per node and \Theta(P^2) total. Shifting > this by a constant factor probably doesn't matter too much > for asymptotic (large-scale) analysis. However, there is > certainly a border-zone where something could be gained with > transferring the ownership of the arrays. > > We will certainly discuss this issue in Helsinki and at the > next telecon! > > For the future, I would generally expect that, at large > scale, such communications are rather sparse. So we should > either talk about a sparse interface to vector collectives or > its static variant, the graph topology collectives. Those > options will also be discussed asap. Both options would > potentially reduce the total memory consumption (assuming > sparse communications) from \Theta(P^2) to O(P) or O(P > log(P)) depending on the problem. > > All the Best, > Torsten > > -- > bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- > Torsten Hoefler | Postdoctoral Fellow > Open Systems Lab | Indiana University > 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley > Hall Room 135 | +01 (812) 855-3608 > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll >

1 0

Re: [mpi3-coll] array parameters in nonblocking collectives
by Torsten Hoefler 01 Aug '09

01 Aug '09

Hello Bin, thanks for bringing this up Bin, it seems like an important issue. I can see both sides. In general, I would say that all vector collectives are not scalable anyway because they require \Theta(P) memory per node and \Theta(P^2) total. Shifting this by a constant factor probably doesn't matter too much for asymptotic (large-scale) analysis. However, there is certainly a border-zone where something could be gained with transferring the ownership of the arrays. We will certainly discuss this issue in Helsinki and at the next telecon! For the future, I would generally expect that, at large scale, such communications are rather sparse. So we should either talk about a sparse interface to vector collectives or its static variant, the graph topology collectives. Those options will also be discussed asap. Both options would potentially reduce the total memory consumption (assuming sparse communications) from \Theta(P^2) to O(P) or O(P log(P)) depending on the problem. All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Fellow Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] array parameters in nonblocking collectives
by Graham, Richard L. 01 Aug '09

01 Aug '09

For a period of time this will double the memory use for these arrays, so this will not change the order of magnitude of the memory requirements. If we are concerned about scalability, we should address this specifically. Rich ________________________________ From: mpi3-coll-bounces(a)lists.mpi-forum.org To: 'MPI-3 Collective Subgroup Discussions' Sent: Sat Aug 01 11:18:41 2009 Subject: Re: [mpi3-coll] array parameters in nonblocking collectives That means the MPI implementation would need to make a copy of potentially large lists. Reference counting won't work here. And the user could easily issue several nonblocking calls in a row. Ouch. Rajeev ________________________________ From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Graham, Richard L. Sent: Saturday, August 01, 2009 9:57 AM To: 'mpi3-coll(a)lists.mpi-forum.org' Subject: Re: [mpi3-coll] array parameters in nonblocking collectives I would expect that the user arguments shouild not be expected to be preserved in the user's arrays between posting the collectives and completing them. Rich ________________________________ From: mpi3-coll-bounces(a)lists.mpi-forum.org To: Torsten Hoefler Cc: MPI-3 Collective Subgroup Discussions Sent: Sat Aug 01 09:56:50 2009 Subject: [mpi3-coll] array parameters in nonblocking collectives Hi, I have one question about the array parameters (sendcounts, recvcounts, sdispls and rdispls) in nonblocking collectives such as MPI_IALLTOALLV. Are callers allowed to modify/destroy those arrays upon the return of the calls, or they must wait until after the MPI_WAIT call? I could not find the correct behavior defined in the latest proposal (nbc-proposal-rev-6.2.pdf) and do not remember having discussion on such. I think this issue is quite unique to nonblocking collective communication calls since arrays in other MPI nonblocking calls are arrays of MPI objects, for which the correct handling is well defined. So it needs to be clarified. Am I missing something? Thanks Bin Jia Communication Protocol Development Advanced Clustering Technology Team, IBM STG Poughkeepsie, NY 12601 Tel: 845-433-6301(T/L 293-)

1 0

Re: [mpi3-coll] array parameters in nonblocking collectives
by Anthony Skjellum 01 Aug '09

01 Aug '09

It appears to me that mpi should temporarily "control"(own) the passed by reference data structures from the point the call is made to the time of completion... Users should accept this as part of the contact and shouldn't assume they own this data during the call's lifetime... From a state transition diagram perspective, ownership passes to and from mpi at ifunction and wait call, respectively. Tony Skjellum, PhD RunTime Computing Solutions, LLC tony(a)runtimecomputing.com direct: +1-205-314-3595 cell: +1-205-807-4968 -----Original Message----- From: "Rajeev Thakur" <thakur(a)mcs.anl.gov> Date: Sat, 1 Aug 2009 10:18:41 To: 'MPI-3 Collective Subgroup Discussions'<mpi3-coll(a)lists.mpi-forum.org> Subject: Re: [mpi3-coll] array parameters in nonblocking collectives _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll

1 0

Re: [mpi3-coll] array parameters in nonblocking collectives
by Rajeev Thakur 01 Aug '09

01 Aug '09

That means the MPI implementation would need to make a copy of potentially large lists. Reference counting won't work here. And the user could easily issue several nonblocking calls in a row. Ouch. Rajeev _____ From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Graham, Richard L. Sent: Saturday, August 01, 2009 9:57 AM To: 'mpi3-coll(a)lists.mpi-forum.org' Subject: Re: [mpi3-coll] array parameters in nonblocking collectives I would expect that the user arguments shouild not be expected to be preserved in the user's arrays between posting the collectives and completing them. Rich _____ From: mpi3-coll-bounces(a)lists.mpi-forum.org To: Torsten Hoefler Cc: MPI-3 Collective Subgroup Discussions Sent: Sat Aug 01 09:56:50 2009 Subject: [mpi3-coll] array parameters in nonblocking collectives Hi, I have one question about the array parameters (sendcounts, recvcounts, sdispls and rdispls) in nonblocking collectives such as MPI_IALLTOALLV. Are callers allowed to modify/destroy those arrays upon the return of the calls, or they must wait until after the MPI_WAIT call? I could not find the correct behavior defined in the latest proposal (nbc-proposal-rev-6.2.pdf) and do not remember having discussion on such. I think this issue is quite unique to nonblocking collective communication calls since arrays in other MPI nonblocking calls are arrays of MPI objects, for which the correct handling is well defined. So it needs to be clarified. Am I missing something? Thanks Bin Jia Communication Protocol Development Advanced Clustering Technology Team, IBM STG Poughkeepsie, NY 12601 Tel: 845-433-6301(T/L 293-)

1 0