- mpiwg-coll - mailman3.cels.anl.gov

Re: [mpi3-coll] array parameters in nonblocking collectives
by Graham, Richard L. 01 Aug '09

01 Aug '09

The only issue I would have with this is that this forces the users to keep these objects in scope for the duration of the communication, which sort of defeats the intent of the nonblocking collectives. Having said that, the same issue holds for the data, and I would not suggest that we keep a copy of the data, so I realize I am advocating an inconsistent view point, so this may not be so good an idea. Rich On 8/1/09 11:33 AM, "Anthony Skjellum" <tony(a)runtimecomputing.com> wrote: It appears to me that mpi should temporarily "control"(own) the passed by reference data structures from the point the call is made to the time of completion... Users should accept this as part of the contact and shouldn't assume they own this data during the call's lifetime... From a state transition diagram perspective, ownership passes to and from mpi at ifunction and wait call, respectively. Tony Skjellum, PhD RunTime Computing Solutions, LLC tony(a)runtimecomputing.com direct: +1-205-314-3595 cell: +1-205-807-4968 -----Original Message----- From: "Rajeev Thakur" <thakur(a)mcs.anl.gov> Date: Sat, 1 Aug 2009 10:18:41 To: 'MPI-3 Collective Subgroup Discussions'<mpi3-coll(a)lists.mpi-forum.org> Subject: Re: [mpi3-coll] array parameters in nonblocking collectives _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll

1 0

Re: [mpi3-coll] array parameters in nonblocking collectives
by Torsten Hoefler 01 Aug '09

01 Aug '09

On Sat, Aug 01, 2009 at 10:54:20AM -0500, Rajeev Thakur wrote: > Torsten, > What does libNBC do in this case? hah, I forgot to say that :). LibNBC generates a collective schedule that encodes all steps in the init (MPI_I*) call. Thus, it simply copies the arguments in the schedule. This technique allows for tree-like (or butterfly) methods to be used in vector variants reducing the needed space per process to O(log(P)). However, they can become arbitrarily slow (log(P) large messages can be serialized at one process) with an adversarial vector distribution. Thus, I implemented the less scalable method that needs O(P) space per process (it'll eliminate all zeros) which can not loose parallelism or degenerate. Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Fellow Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] array parameters in nonblocking collectives
by Rajeev Thakur 01 Aug '09

01 Aug '09

Torsten, What does libNBC do in this case? Rajeev > -----Original Message----- > From: mpi3-coll-bounces(a)lists.mpi-forum.org > [mailto:[email protected]] On Behalf Of > Torsten Hoefler > Sent: Saturday, August 01, 2009 10:46 AM > To: MPI-3 Collective Subgroup Discussions > Subject: Re: [mpi3-coll] array parameters in nonblocking collectives > > Hello Bin, > thanks for bringing this up Bin, it seems like an important issue. > > I can see both sides. In general, I would say that all vector > collectives are not scalable anyway because they require > \Theta(P) memory per node and \Theta(P^2) total. Shifting > this by a constant factor probably doesn't matter too much > for asymptotic (large-scale) analysis. However, there is > certainly a border-zone where something could be gained with > transferring the ownership of the arrays. > > We will certainly discuss this issue in Helsinki and at the > next telecon! > > For the future, I would generally expect that, at large > scale, such communications are rather sparse. So we should > either talk about a sparse interface to vector collectives or > its static variant, the graph topology collectives. Those > options will also be discussed asap. Both options would > potentially reduce the total memory consumption (assuming > sparse communications) from \Theta(P^2) to O(P) or O(P > log(P)) depending on the problem. > > All the Best, > Torsten > > -- > bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- > Torsten Hoefler | Postdoctoral Fellow > Open Systems Lab | Indiana University > 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley > Hall Room 135 | +01 (812) 855-3608 > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll >

1 0

Re: [mpi3-coll] array parameters in nonblocking collectives
by Torsten Hoefler 01 Aug '09

01 Aug '09

Hello Bin, thanks for bringing this up Bin, it seems like an important issue. I can see both sides. In general, I would say that all vector collectives are not scalable anyway because they require \Theta(P) memory per node and \Theta(P^2) total. Shifting this by a constant factor probably doesn't matter too much for asymptotic (large-scale) analysis. However, there is certainly a border-zone where something could be gained with transferring the ownership of the arrays. We will certainly discuss this issue in Helsinki and at the next telecon! For the future, I would generally expect that, at large scale, such communications are rather sparse. So we should either talk about a sparse interface to vector collectives or its static variant, the graph topology collectives. Those options will also be discussed asap. Both options would potentially reduce the total memory consumption (assuming sparse communications) from \Theta(P^2) to O(P) or O(P log(P)) depending on the problem. All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Fellow Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0

Re: [mpi3-coll] array parameters in nonblocking collectives
by Graham, Richard L. 01 Aug '09

01 Aug '09

For a period of time this will double the memory use for these arrays, so this will not change the order of magnitude of the memory requirements. If we are concerned about scalability, we should address this specifically. Rich ________________________________ From: mpi3-coll-bounces(a)lists.mpi-forum.org To: 'MPI-3 Collective Subgroup Discussions' Sent: Sat Aug 01 11:18:41 2009 Subject: Re: [mpi3-coll] array parameters in nonblocking collectives That means the MPI implementation would need to make a copy of potentially large lists. Reference counting won't work here. And the user could easily issue several nonblocking calls in a row. Ouch. Rajeev ________________________________ From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Graham, Richard L. Sent: Saturday, August 01, 2009 9:57 AM To: 'mpi3-coll(a)lists.mpi-forum.org' Subject: Re: [mpi3-coll] array parameters in nonblocking collectives I would expect that the user arguments shouild not be expected to be preserved in the user's arrays between posting the collectives and completing them. Rich ________________________________ From: mpi3-coll-bounces(a)lists.mpi-forum.org To: Torsten Hoefler Cc: MPI-3 Collective Subgroup Discussions Sent: Sat Aug 01 09:56:50 2009 Subject: [mpi3-coll] array parameters in nonblocking collectives Hi, I have one question about the array parameters (sendcounts, recvcounts, sdispls and rdispls) in nonblocking collectives such as MPI_IALLTOALLV. Are callers allowed to modify/destroy those arrays upon the return of the calls, or they must wait until after the MPI_WAIT call? I could not find the correct behavior defined in the latest proposal (nbc-proposal-rev-6.2.pdf) and do not remember having discussion on such. I think this issue is quite unique to nonblocking collective communication calls since arrays in other MPI nonblocking calls are arrays of MPI objects, for which the correct handling is well defined. So it needs to be clarified. Am I missing something? Thanks Bin Jia Communication Protocol Development Advanced Clustering Technology Team, IBM STG Poughkeepsie, NY 12601 Tel: 845-433-6301(T/L 293-)

1 0

Re: [mpi3-coll] array parameters in nonblocking collectives
by Anthony Skjellum 01 Aug '09

01 Aug '09

It appears to me that mpi should temporarily "control"(own) the passed by reference data structures from the point the call is made to the time of completion... Users should accept this as part of the contact and shouldn't assume they own this data during the call's lifetime... From a state transition diagram perspective, ownership passes to and from mpi at ifunction and wait call, respectively. Tony Skjellum, PhD RunTime Computing Solutions, LLC tony(a)runtimecomputing.com direct: +1-205-314-3595 cell: +1-205-807-4968 -----Original Message----- From: "Rajeev Thakur" <thakur(a)mcs.anl.gov> Date: Sat, 1 Aug 2009 10:18:41 To: 'MPI-3 Collective Subgroup Discussions'<mpi3-coll(a)lists.mpi-forum.org> Subject: Re: [mpi3-coll] array parameters in nonblocking collectives _______________________________________________ mpi3-coll mailing list mpi3-coll(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll

1 0

Re: [mpi3-coll] array parameters in nonblocking collectives
by Rajeev Thakur 01 Aug '09

01 Aug '09

That means the MPI implementation would need to make a copy of potentially large lists. Reference counting won't work here. And the user could easily issue several nonblocking calls in a row. Ouch. Rajeev _____ From: mpi3-coll-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Graham, Richard L. Sent: Saturday, August 01, 2009 9:57 AM To: 'mpi3-coll(a)lists.mpi-forum.org' Subject: Re: [mpi3-coll] array parameters in nonblocking collectives I would expect that the user arguments shouild not be expected to be preserved in the user's arrays between posting the collectives and completing them. Rich _____ From: mpi3-coll-bounces(a)lists.mpi-forum.org To: Torsten Hoefler Cc: MPI-3 Collective Subgroup Discussions Sent: Sat Aug 01 09:56:50 2009 Subject: [mpi3-coll] array parameters in nonblocking collectives Hi, I have one question about the array parameters (sendcounts, recvcounts, sdispls and rdispls) in nonblocking collectives such as MPI_IALLTOALLV. Are callers allowed to modify/destroy those arrays upon the return of the calls, or they must wait until after the MPI_WAIT call? I could not find the correct behavior defined in the latest proposal (nbc-proposal-rev-6.2.pdf) and do not remember having discussion on such. I think this issue is quite unique to nonblocking collective communication calls since arrays in other MPI nonblocking calls are arrays of MPI objects, for which the correct handling is well defined. So it needs to be clarified. Am I missing something? Thanks Bin Jia Communication Protocol Development Advanced Clustering Technology Team, IBM STG Poughkeepsie, NY 12601 Tel: 845-433-6301(T/L 293-)

1 0

Re: [mpi3-coll] array parameters in nonblocking collectives
by Graham, Richard L. 01 Aug '09

01 Aug '09

I would expect that the user arguments shouild not be expected to be preserved in the user's arrays between posting the collectives and completing them. Rich ________________________________ From: mpi3-coll-bounces(a)lists.mpi-forum.org To: Torsten Hoefler Cc: MPI-3 Collective Subgroup Discussions Sent: Sat Aug 01 09:56:50 2009 Subject: [mpi3-coll] array parameters in nonblocking collectives Hi, I have one question about the array parameters (sendcounts, recvcounts, sdispls and rdispls) in nonblocking collectives such as MPI_IALLTOALLV. Are callers allowed to modify/destroy those arrays upon the return of the calls, or they must wait until after the MPI_WAIT call? I could not find the correct behavior defined in the latest proposal (nbc-proposal-rev-6.2.pdf) and do not remember having discussion on such. I think this issue is quite unique to nonblocking collective communication calls since arrays in other MPI nonblocking calls are arrays of MPI objects, for which the correct handling is well defined. So it needs to be clarified. Am I missing something? Thanks Bin Jia Communication Protocol Development Advanced Clustering Technology Team, IBM STG Poughkeepsie, NY 12601 Tel: 845-433-6301(T/L 293-)

1 0

[mpi3-coll] array parameters in nonblocking collectives
by Bin Jia 01 Aug '09

01 Aug '09

Hi, I have one question about the array parameters (sendcounts, recvcounts, sdispls and rdispls) in nonblocking collectives such as MPI_IALLTOALLV. Are callers allowed to modify/destroy those arrays upon the return of the calls, or they must wait until after the MPI_WAIT call? I could not find the correct behavior defined in the latest proposal (nbc-proposal-rev-6.2.pdf) and do not remember having discussion on such. I think this issue is quite unique to nonblocking collective communication calls since arrays in other MPI nonblocking calls are arrays of MPI objects, for which the correct handling is well defined. So it needs to be clarified. Am I missing something? Thanks Bin Jia Communication Protocol Development Advanced Clustering Technology Team, IBM STG Poughkeepsie, NY 12601 Tel: 845-433-6301(T/L 293-)

1 0

[mpi3-coll] March Teleconference
by Torsten Hoefler 19 Mar '09

19 Mar '09

Hello, I heard no further comments and thus I scheduled the teleconference for today, March 19th 11am EDT, 8am PDT, 16h CET. !!!! Be careful with the time zones, the US is on daylight savings time and Europe not, so my last announcement was wrong and the telecon will be at European CET 4pm/16h !!!! The reference time is 11am EDT, you can check the current time there at: http://www.timeanddate.com/worldclock/city.html?n=105 The call information is: Conf Id: 153970 Date: 3/19/2009 Begin Time: 11:00:00 AM Eastern Time (US & Canada) Phone Number: (812) 856-3600 PIN: 002118# I hope all can make it! All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Postdoctoral Researcher Open Systems Lab | Indiana University 150 S. Woodlawn Ave. | Bloomington, IN, 474045, USA Lindley Hall Room 135 | +01 (812) 855-3608

1 0