December 2016 - mpi-forum - mailman3.cels.anl.gov

Re: [Mpi-forum] MPI_Comm_split_type question
by Jeff Hammond 15 Dec '16

15 Dec '16

MPI_Comm_split calls Allgather internally in many implementations... Jeff Sent from my iPhone > On Dec 15, 2016, at 5:15 PM, Martin Schulz <schulzm(a)llnl.gov> wrote: > > Hi Guillaume, > > Just to add to George’s suggestion and following the same idea – instead of the Allgather, which could cause scalability problems, you could just do a Reduce or Allreduce with all processes with rank 0 in the new communicator contributing 1 and the rest 0 and then use an add as the reduction operator. > > Martin > > > ________________________________________________________________________ > Martin Schulz, schulzm(a)llnl.gov, http://scalability.llnl.gov/ > CASC @ Lawrence Livermore National Laboratory, Livermore, USA > > > From: <mpi-forum-bounces(a)lists.mpi-forum.org> on behalf of George Bosilca <bosilca(a)icl.utk.edu> > Reply-To: Main mailing list <mpi-forum(a)lists.mpi-forum.org> > Date: Thursday, December 15, 2016 at 11:41 AM > To: Main mailing list <mpi-forum(a)lists.mpi-forum.org> > Subject: Re: [Mpi-forum] MPI_Comm_split_type question > > Guillaume, > > If I understand you correctly you are trying to figure out how many unique groups (and therefore communicators) were globally created as a result of an MPI_Comm_split_type operation. Unfortunately, depending on the type passed to this function (there is a single one defined by MPI, but in Open MPI we support several extensions), the resulting communicator can be created without the need for global knowledge, and thus it is impossible to know how many have communicators have been create in total. > > To extend on what JeffH proposed, you can count the resulting number of communicators by doing an MPI_Allgather on the initial communicator on a data where each participant provide their rank in the newly created communicator (then returned by the MPI_Comm_split* operation), and then counting the number of 0. > > George. > > > >> On Dec 15, 2016, at 14:05 , Jeff Hammond <jeff.science(a)gmail.com> wrote: >> >> >> >> On Thu, Dec 15, 2016 at 10:29 AM, Guillaume Mercier <guillaume.mercier(a)u-bordeaux.fr> wrote: >>> >>> Hi Jeff, >>> >>>> On 15/12/2016 18:48, Jeff Hammond wrote: >>>> The number of output communicators from MPI_Comm_split(_type) is always >>>> one. >>> >>> Yes, obviously. >>> >>>> Different ranks may get different outputs, but one cannot transfer >>>> a communicator object from one rank to another. >>> >>> That's my issue actually. >>> >>>> If you want to do how many total output communicators there are, you can >>>> perform an MPI_Allgather on the color arguments and see how many unique >>>> values there are. >>> >>> I'm not sure about that: tae the case of MPI_Comm_split_type with >>> MPI_COMM_TYPE_SHARED. According to tha standards :"this type splits >>> the communicator into subcommunicators each of which can create a shared memory region". >>> So, there is only one color, but several subcommunicators. >>> Or am I understanding this the wrong way? >> >> Each rank gets its own subcommunicator object that captures the ranks of the shared memory domain, as supported by MPI_Win_allocate_shared. >> >> I'm still not sure I understand what the issue is. What are you trying to do? >> >> Jeff >> >>> Regards >>> Guillaume >>> >>> _______________________________________________ >>> mpi-forum mailing list >>> mpi-forum(a)lists.mpi-forum.org >>> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> http://jeffhammond.github.io/ >> _______________________________________________ >> mpi-forum mailing list >> mpi-forum(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum

1 0

Re: [Mpi-forum] MPI_Comm_split_type question
by Martin Schulz 15 Dec '16

15 Dec '16

Hi Guillaume, Just to add to George¹s suggestion and following the same idea instead of the Allgather, which could cause scalability problems, you could just do a Reduce or Allreduce with all processes with rank 0 in the new communicator contributing 1 and the rest 0 and then use an add as the reduction operator. Martin ________________________________________________________________________ Martin Schulz, schulzm(a)llnl.gov, http://scalability.llnl.gov/ CASC @ Lawrence Livermore National Laboratory, Livermore, USA From: <mpi-forum-bounces(a)lists.mpi-forum.org> on behalf of George Bosilca <bosilca(a)icl.utk.edu> Reply-To: Main mailing list <mpi-forum(a)lists.mpi-forum.org> Date: Thursday, December 15, 2016 at 11:41 AM To: Main mailing list <mpi-forum(a)lists.mpi-forum.org> Subject: Re: [Mpi-forum] MPI_Comm_split_type question Guillaume, If I understand you correctly you are trying to figure out how many unique groups (and therefore communicators) were globally created as a result of an MPI_Comm_split_type operation. Unfortunately, depending on the type passed to this function (there is a single one defined by MPI, but in Open MPI we support several extensions), the resulting communicator can be created without the need for global knowledge, and thus it is impossible to know how many have communicators have been create in total. To extend on what JeffH proposed, you can count the resulting number of communicators by doing an MPI_Allgather on the initial communicator on a data where each participant provide their rank in the newly created communicator (then returned by the MPI_Comm_split* operation), and then counting the number of 0. George. > On Dec 15, 2016, at 14:05 , Jeff Hammond <jeff.science(a)gmail.com> wrote: > > > > On Thu, Dec 15, 2016 at 10:29 AM, Guillaume Mercier > <guillaume.mercier(a)u-bordeaux.fr> wrote: >> >> Hi Jeff, >> >> On 15/12/2016 18:48, Jeff Hammond wrote: >>> The number of output communicators from MPI_Comm_split(_type) is always >>> one. >> >> Yes, obviously. >> >>> Different ranks may get different outputs, but one cannot transfer >>> a communicator object from one rank to another. >> >> That's my issue actually. >> >>> If you want to do how many total output communicators there are, you can >>> perform an MPI_Allgather on the color arguments and see how many unique >>> values there are. >> >> I'm not sure about that: tae the case of MPI_Comm_split_type with >> MPI_COMM_TYPE_SHARED. According to tha standards :"this type splits >> the communicator into subcommunicators each of which can create a shared >> memory region". >> So, there is only one color, but several subcommunicators. >> Or am I understanding this the wrong way? >> > > Each rank gets its own subcommunicator object that captures the ranks of the > shared memory domain, as supported by MPI_Win_allocate_shared. > > I'm still not sure I understand what the issue is. What are you trying to do? > > Jeff > >> Regards >> Guillaume >> >> _______________________________________________ >> mpi-forum mailing list >> mpi-forum(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum

1 0

Re: [Mpi-forum] MPI_Comm_split_type question
by Guillaume Mercier 15 Dec '16

15 Dec '16

George, Thanks, I didn't think about that but it seems to solve my problem. Thanks. Guillaume On 15/12/2016 20:41, George Bosilca wrote: > Guillaume, > > If I understand you correctly you are trying to figure out how many > unique groups (and therefore communicators) were globally created as a > result of an MPI_Comm_split_type operation. Unfortunately, depending on > the type passed to this function (there is a single one defined by MPI, > but in Open MPI we support several extensions), the resulting > communicator can be created without the need for global knowledge, and > thus it is impossible to know how many have communicators have been > create in total. > > To extend on what JeffH proposed, you can count the resulting number of > communicators by doing an MPI_Allgather on the initial communicator on a > data where each participant provide their rank in the newly created > communicator (then returned by the MPI_Comm_split* operation), and then > counting the number of 0. > > George. > > > >> On Dec 15, 2016, at 14:05 , Jeff Hammond <jeff.science(a)gmail.com >> <mailto:[email protected]>> wrote: >> >> >> >> On Thu, Dec 15, 2016 at 10:29 AM, Guillaume >> Mercier <guillaume.mercier(a)u-bordeaux.fr >> <mailto:[email protected]>> wrote: >> >> >> Hi Jeff, >> >> On 15/12/2016 18:48, Jeff Hammond wrote: >> >> The number of output communicators from MPI_Comm_split(_type) >> is always >> one. >> >> >> Yes, obviously. >> >> Different ranks may get different outputs, but one cannot transfer >> a communicator object from one rank to another. >> >> >> That's my issue actually. >> >> If you want to do how many total output communicators there >> are, you can >> perform an MPI_Allgather on the color arguments and see how >> many unique >> values there are. >> >> >> I'm not sure about that: tae the case of MPI_Comm_split_type with >> MPI_COMM_TYPE_SHARED. According to tha standards :"this type splits >> the communicator into subcommunicators each of which can create a >> shared memory region". >> So, there is only one color, but several subcommunicators. >> Or am I understanding this the wrong way? >> >> >> Each rank gets its own subcommunicator object that captures the ranks >> of the shared memory domain, as supported by MPI_Win_allocate_shared. >> >> I'm still not sure I understand what the issue is. What are you >> trying to do? >> >> Jeff >> >> >> Regards >> Guillaume >> >> _______________________________________________ >> mpi-forum mailing list >> mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> >> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum >> <https://lists.mpi-forum.org/mailman/listinfo/mpi-forum> >> >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com <mailto:[email protected]> >> http://jeffhammond.github.io/ >> _______________________________________________ >> mpi-forum mailing list >> mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> >> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum > > > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum >

1 0

Re: [Mpi-forum] MPI_Comm_split_type question
by George Bosilca 15 Dec '16

15 Dec '16

Guillaume, If I understand you correctly you are trying to figure out how many unique groups (and therefore communicators) were globally created as a result of an MPI_Comm_split_type operation. Unfortunately, depending on the type passed to this function (there is a single one defined by MPI, but in Open MPI we support several extensions), the resulting communicator can be created without the need for global knowledge, and thus it is impossible to know how many have communicators have been create in total. To extend on what JeffH proposed, you can count the resulting number of communicators by doing an MPI_Allgather on the initial communicator on a data where each participant provide their rank in the newly created communicator (then returned by the MPI_Comm_split* operation), and then counting the number of 0. George. > On Dec 15, 2016, at 14:05 , Jeff Hammond <jeff.science(a)gmail.com> wrote: > > > > On Thu, Dec 15, 2016 at 10:29 AM, Guillaume Mercier <guillaume.mercier(a)u-bordeaux.fr <mailto:[email protected]>> wrote: > > Hi Jeff, > > On 15/12/2016 18:48, Jeff Hammond wrote: > The number of output communicators from MPI_Comm_split(_type) is always > one. > > Yes, obviously. > > Different ranks may get different outputs, but one cannot transfer > a communicator object from one rank to another. > > That's my issue actually. > > If you want to do how many total output communicators there are, you can > perform an MPI_Allgather on the color arguments and see how many unique > values there are. > > I'm not sure about that: tae the case of MPI_Comm_split_type with > MPI_COMM_TYPE_SHARED. According to tha standards :"this type splits > the communicator into subcommunicators each of which can create a shared memory region". > So, there is only one color, but several subcommunicators. > Or am I understanding this the wrong way? > > > Each rank gets its own subcommunicator object that captures the ranks of the shared memory domain, as supported by MPI_Win_allocate_shared. > > I'm still not sure I understand what the issue is. What are you trying to do? > > Jeff > > Regards > Guillaume > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum <https://lists.mpi-forum.org/mailman/listinfo/mpi-forum> > > > > -- > Jeff Hammond > jeff.science(a)gmail.com <mailto:[email protected]> > http://jeffhammond.github.io/ <http://jeffhammond.github.io/>_______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum <https://lists.mpi-forum.org/mailman/listinfo/mpi-forum>

1 0

Re: [Mpi-forum] MPI_Comm_split_type question
by Jeff Hammond 15 Dec '16

15 Dec '16

On Thu, Dec 15, 2016 at 10:29 AM, Guillaume Mercier < guillaume.mercier(a)u-bordeaux.fr> wrote: > > Hi Jeff, > > On 15/12/2016 18:48, Jeff Hammond wrote: > >> The number of output communicators from MPI_Comm_split(_type) is always >> one. >> > > Yes, obviously. > > Different ranks may get different outputs, but one cannot transfer >> a communicator object from one rank to another. >> > > That's my issue actually. > > If you want to do how many total output communicators there are, you can >> perform an MPI_Allgather on the color arguments and see how many unique >> values there are. >> > > I'm not sure about that: tae the case of MPI_Comm_split_type with > MPI_COMM_TYPE_SHARED. According to tha standards :"this type splits > the communicator into subcommunicators each of which can create a shared > memory region". > So, there is only one color, but several subcommunicators. > Or am I understanding this the wrong way? > > Each rank gets its own subcommunicator object that captures the ranks of the shared memory domain, as supported by MPI_Win_allocate_shared. I'm still not sure I understand what the issue is. What are you trying to do? Jeff > Regards > Guillaume > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [Mpi-forum] MPI_Comm_split_type question
by Guillaume Mercier 15 Dec '16

15 Dec '16

Hi Jeff, On 15/12/2016 18:48, Jeff Hammond wrote: > The number of output communicators from MPI_Comm_split(_type) is always > one. Yes, obviously. > Different ranks may get different outputs, but one cannot transfer > a communicator object from one rank to another. That's my issue actually. > If you want to do how many total output communicators there are, you can > perform an MPI_Allgather on the color arguments and see how many unique > values there are. I'm not sure about that: tae the case of MPI_Comm_split_type with MPI_COMM_TYPE_SHARED. According to tha standards :"this type splits the communicator into subcommunicators each of which can create a shared memory region". So, there is only one color, but several subcommunicators. Or am I understanding this the wrong way? Regards Guillaume

1 0

Re: [Mpi-forum] MPI_Comm_split_type question
by Jeff Hammond 15 Dec '16

15 Dec '16

The number of output communicators from MPI_Comm_split(_type) is always one. Different ranks may get different outputs, but one cannot transfer a communicator object from one rank to another. If you want to do how many total output communicators there are, you can perform an MPI_Allgather on the color arguments and see how many unique values there are. Best, Jeff On Thu, Dec 15, 2016 at 9:32 AM, Guillaume Mercier < guillaume.mercier(a)u-bordeaux.fr> wrote: > > > Hello, > > When using the MPI_Comm_split_type function, the number of new > communicators created is unknown. However this information > could be useful to distribute data across various "domains" > (where each domain would be represented by such a communicator). > Is there a simple (or even a not-so-simple) way to get this > information? > > Thanks. > Guillaume Mercier > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

MPI_Comm_split_type question
by Guillaume Mercier 15 Dec '16

15 Dec '16

Hello, When using the MPI_Comm_split_type function, the number of new communicators created is unknown. However this information could be useful to distribute data across various "domains" (where each domain would be represented by such a communicator). Is there a simple (or even a not-so-simple) way to get this information? Thanks. Guillaume Mercier

1 0

Re: [Mpi-forum] Interest of changing lower bound in MPI_Type_create_resized
by Thakur, Rajeev 09 Dec '16

09 Dec '16

You are right. It is buf + i*extent, not lower_bound + i*extent. This stuff is confusing. Right now, I am not coming up with a good example of where a non-zero lower bound makes a difference. Rajeev > On Dec 9, 2016, at 9:31 AM, lecas(a)idris.fr <dimitri.lecas(a)idris.fr> wrote: > > Le 12/08/2016 à 03:14 PM, Thakur, Rajeev a écrit : >> When MPI sends a message, the second datatype is taken from the location (lower bound + extent), the third datatype from (lower bound + 2*extent), and so on. So the lower bound does affect what is sent. >> >> Rajeev >> > > This is not what i understand from the chapter 4.1.11 Use of General > Datatypes in Communication > > If a datatype have n entries and we are sending count datatype > > i=0,..., count-1 > j=0,..,n-1 > > So the address in memory to get each entries is : addr(i,j) = > buf+extent*i+disp(j) > > There is nowhere the usage of lower bound for having location of next > datatype. > buf is the argument in send/recv function. > > > >>> On Dec 8, 2016, at 7:54 AM, lecas(a)idris.fr <dimitri.lecas(a)idris.fr> wrote: >>> >>> Hi, >>> >>> In MPI_Type_create_resized we can change the extent and the lower bound. >>> >>> I don't understand what is the interest of changing the lower bound, >>> since for sending message, only the extent is used. >>> >>> Is there an example where the value for lower bound in >>> MPI_Type_create_resized have a real impact ? >>> >>> Thanks >>> >>> -- >>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >>> Dimitri Lecas >>> >>> >>> _______________________________________________ >>> mpi-forum mailing list >>> mpi-forum(a)lists.mpi-forum.org >>> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum >> _______________________________________________ >> mpi-forum mailing list >> mpi-forum(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum >> > > > -- > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Dimitri Lecas > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org https://lists.mpi-forum.org/mailman/listinfo/mpi-forum

1 0

Re: [Mpi-forum] Interest of changing lower bound in MPI_Type_create_resized
by lecas＠idris.fr 09 Dec '16

09 Dec '16

Le 12/08/2016 à 03:14 PM, Thakur, Rajeev a écrit : > When MPI sends a message, the second datatype is taken from the location (lower bound + extent), the third datatype from (lower bound + 2*extent), and so on. So the lower bound does affect what is sent. > > Rajeev > This is not what i understand from the chapter 4.1.11 Use of General Datatypes in Communication If a datatype have n entries and we are sending count datatype i=0,..., count-1 j=0,..,n-1 So the address in memory to get each entries is : addr(i,j) = buf+extent*i+disp(j) There is nowhere the usage of lower bound for having location of next datatype. buf is the argument in send/recv function. >> On Dec 8, 2016, at 7:54 AM, lecas(a)idris.fr <dimitri.lecas(a)idris.fr> wrote: >> >> Hi, >> >> In MPI_Type_create_resized we can change the extent and the lower bound. >> >> I don't understand what is the interest of changing the lower bound, >> since for sending message, only the extent is used. >> >> Is there an example where the value for lower bound in >> MPI_Type_create_resized have a real impact ? >> >> Thanks >> >> -- >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> Dimitri Lecas >> >> >> _______________________________________________ >> mpi-forum mailing list >> mpi-forum(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum > -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Dimitri Lecas ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org https://lists.mpi-forum.org/mailman/listinfo/mpi-forum

1 0