July 2013 - devel - mailman3.cels.anl.gov

Re: [mpich-devel] tackling "large datatype" change
by Pavan Balaji 12 Jul '13

12 Jul '13

Rob, Sorry, I'll look into this next week. Amount of time left is proportional to age of baby. The first solution works for some cases in MPICH, but there are still many bugs. I think your ticket is to fix this part? The second solution is known not to work. -- Pavan On 07/12/2013 06:25 AM, Rob Latham wrote: > I've been trying to tackle tt #1742, #1890, and #1893 as part of some > I/O work that uses large datatypes. > > Am I stepping on anyone's toes here? Pavan, I know i bugged you about these > tickets. Hope I didn't waste my Thursday on this... > > Pavan told me there are two problems here, but I think there are > really one solution to both: > > - large datatype means a datatype that describes more than 2 gigs of > data. A million contigs of a million contigs, say > > - large count means datatypes that use MPI_Count to say how many > elements they have: a contig of 3 billion MPI_BYTES, say > > Internally, though, there's no way to decouple those two problems. > > The changes are pervasive and make me really nervous. > > I've been sometimes pushing changes to the ticket-1742-bigio branch. > > Who can review datatype changes once I'm done?: > > ==rob > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-devel] tackling "large datatype" change
by David Goodell (dgoodell) 12 Jul '13

12 Jul '13

Jeff, How could this possibly work correctly with broken (i.e., uses "int"s internally for certain calculations) datatype and communication engines? The hard part of large count types is not constructing the actual type, it's processing that type correctly throughout the entire stack... -Dave On Jul 12, 2013, at 9:57 AM, Jeff Hammond <jhammond(a)alcf.anl.gov> wrote: > Hi Rob, > > I started working on something to address this a few weeks ago. I was going to write a portable (i.e. above MPI) implementation of > > MPIX_Type_contiguous_x(MPI_Count count, MPI_Datatype old_type, MPI_Datatype *newtype) > > and then reimplement it efficiently in MPICH. I will also try to get JeffS or DaveG to create it for OtherMPI so e.g. PETSc can use it more broadly. > > Does this sound like an okay plan to you? I've not pushed the latest code to Github but when I do, I'll send you the link. > > Best, > > Jeff > > > ----- Original Message ----- >> From: "Rob Latham" <robl(a)mcs.anl.gov> >> To: devel(a)mpich.org >> Sent: Friday, July 12, 2013 6:25:14 AM >> Subject: [mpich-devel] tackling "large datatype" change >> >> I've been trying to tackle tt #1742, #1890, and #1893 as part of some >> I/O work that uses large datatypes. >> >> Am I stepping on anyone's toes here? Pavan, I know i bugged you >> about these >> tickets. Hope I didn't waste my Thursday on this... >> >> Pavan told me there are two problems here, but I think there are >> really one solution to both: >> >> - large datatype means a datatype that describes more than 2 gigs of >> data. A million contigs of a million contigs, say >> >> - large count means datatypes that use MPI_Count to say how many >> elements they have: a contig of 3 billion MPI_BYTES, say >> >> Internally, though, there's no way to decouple those two problems. >> >> The changes are pervasive and make me really nervous. >> >> I've been sometimes pushing changes to the ticket-1742-bigio branch. >> >> Who can review datatype changes once I'm done?: >> >> ==rob >> >> -- >> Rob Latham >> Mathematics and Computer Science Division >> Argonne National Lab, IL USA >> > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

Re: [mpich-devel] tackling "large datatype" change
by Jeff Hammond 12 Jul '13

12 Jul '13

Hi Rob, I started working on something to address this a few weeks ago. I was going to write a portable (i.e. above MPI) implementation of MPIX_Type_contiguous_x(MPI_Count count, MPI_Datatype old_type, MPI_Datatype *newtype) and then reimplement it efficiently in MPICH. I will also try to get JeffS or DaveG to create it for OtherMPI so e.g. PETSc can use it more broadly. Does this sound like an okay plan to you? I've not pushed the latest code to Github but when I do, I'll send you the link. Best, Jeff ----- Original Message ----- > From: "Rob Latham" <robl(a)mcs.anl.gov> > To: devel(a)mpich.org > Sent: Friday, July 12, 2013 6:25:14 AM > Subject: [mpich-devel] tackling "large datatype" change > > I've been trying to tackle tt #1742, #1890, and #1893 as part of some > I/O work that uses large datatypes. > > Am I stepping on anyone's toes here? Pavan, I know i bugged you > about these > tickets. Hope I didn't waste my Thursday on this... > > Pavan told me there are two problems here, but I think there are > really one solution to both: > > - large datatype means a datatype that describes more than 2 gigs of > data. A million contigs of a million contigs, say > > - large count means datatypes that use MPI_Count to say how many > elements they have: a contig of 3 billion MPI_BYTES, say > > Internally, though, there's no way to decouple those two problems. > > The changes are pervasive and make me really nervous. > > I've been sometimes pushing changes to the ticket-1742-bigio branch. > > Who can review datatype changes once I'm done?: > > ==rob > > -- > Rob Latham > Mathematics and Computer Science Division > Argonne National Lab, IL USA > -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

tackling "large datatype" change
by Rob Latham 12 Jul '13

12 Jul '13

I've been trying to tackle tt #1742, #1890, and #1893 as part of some I/O work that uses large datatypes. Am I stepping on anyone's toes here? Pavan, I know i bugged you about these tickets. Hope I didn't waste my Thursday on this... Pavan told me there are two problems here, but I think there are really one solution to both: - large datatype means a datatype that describes more than 2 gigs of data. A million contigs of a million contigs, say - large count means datatypes that use MPI_Count to say how many elements they have: a contig of 3 billion MPI_BYTES, say Internally, though, there's no way to decouple those two problems. The changes are pervasive and make me really nervous. I've been sometimes pushing changes to the ticket-1742-bigio branch. Who can review datatype changes once I'm done?: ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

--program-suffix misbehaiving
by Wesley Bland 10 Jul '13

10 Jul '13

I'm seeing some bad behavior when configuring with --program-suffix (unless I'm misunderstanding how this is supposed to work). When using this, mpich doesn't generate the mpiexec symlinks anymore (or really what should probably mpiexec-[sufffix]). Instead, it skips straight to mpiexec.hydra-[suffix]. This is a bit confusing, but not a big deal. The bigger problem is that it doesn't correctly call hydra_pmi_proxy to launch the jobs. These executables are also named hydra_pmi_proxy-[suffix], which results in this nastiness: wbland@bb20:tmp$ mpiexec.hydra-branch -n 2 hostname [mpiexec@bb20] HYDU_create_process (/home/wbland/Repositories/mpich/src/pm/hydra/utils/launch/launch.c:75): execvp error on file /home/wbland/tools/mpich-branch/bin/hydra_pmi_proxy (No such file or directory) [mpiexec@bb20] HYD_pmcd_pmiserv_proxy_init_cb (/home/wbland/Repositories/mpich/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:505): assert (!closed) failed [mpiexec@bb20] HYDT_dmxu_poll_wait_for_event (/home/wbland/Repositories/mpich/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [mpiexec@bb20] HYD_pmci_wait_for_completion (/home/wbland/Repositories/mpich/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec@bb20] main (/home/wbland/Repositories/mpich/src/pm/hydra/ui/mpich/mpiexec.c:331): process manager error waiting for completion Before I file a ticket, I want to make sure that this isn't expected behavior somehow (especially the first part). Thanks, Wesley

1 0

Re: [mpich-devel] find out what processes are on the same node
by William Gropp 05 Jul '13

05 Jul '13

I also have an O(log Nproc) implementation that I've been using for some time. Bill William Gropp Director, Parallel Computing Institute Deputy Director for Research Institute for Advanced Computing Applications and Technologies Thomas M. Siebel Chair in Computer Science University of Illinois Urbana-Champaign On Jul 3, 2013, at 11:02 AM, Jeff Hammond wrote: > https://wiki.alcf.anl.gov/parts/index.php/Implementing_a_Symmetric_Heap has an implementation of MPE_Comm_split_node that does a very simple O(Nproc) implementation of MPI_Comm_split_type(type=SHARED) using MPI_Get_processor_name. It also shows the non-portable implementation for Cray and Blue Gene that one can use until MPI-3 is deployed on those systems. > > Jim Dinan wrote an O(log Nproc) implementation of MPI_Comm_split_type(type=SHARED) that I have somewhere but I don't like it because it's confusing to my small brain. > > Jeff > > ----- Original Message ----- > From: "Rob Latham" <robl(a)mcs.anl.gov> > To: devel(a)mpich.org > Sent: Wednesday, July 3, 2013 10:53:15 AM > Subject: Re: [mpich-devel] find out what processes are on the same node > > On Wed, Jul 03, 2013 at 10:39:55AM -0500, Tatiana V. Martsinkevich wrote: >> Hello, >> >> I have a couple of questions: >> >> 1) is there a way to find out what other processes reside in the same >> node as my_rank from within MPICH? > > MPI_GET_PROCESSOR_NAME might be what you are looking for. > > ==rob > > -- > Rob Latham > Mathematics and Computer Science Division > Argonne National Lab, IL USA > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

Re: [mpich-devel] find out what processes are on the same node
by Tatiana V. Martsinkevich 03 Jul '13

03 Jul '13

Jeff, Rob, thank you! MPI_Comm_split_type() is what I was looking for. Tatiana 2013/7/3 Jeff Hammond <jhammond(a)alcf.anl.gov>: > https://wiki.alcf.anl.gov/parts/index.php/Implementing_a_Symmetric_Heap has an implementation of MPE_Comm_split_node that does a very simple O(Nproc) implementation of MPI_Comm_split_type(type=SHARED) using MPI_Get_processor_name. It also shows the non-portable implementation for Cray and Blue Gene that one can use until MPI-3 is deployed on those systems. > > Jim Dinan wrote an O(log Nproc) implementation of MPI_Comm_split_type(type=SHARED) that I have somewhere but I don't like it because it's confusing to my small brain. > > Jeff > > ----- Original Message ----- > From: "Rob Latham" <robl(a)mcs.anl.gov> > To: devel(a)mpich.org > Sent: Wednesday, July 3, 2013 10:53:15 AM > Subject: Re: [mpich-devel] find out what processes are on the same node > > On Wed, Jul 03, 2013 at 10:39:55AM -0500, Tatiana V. Martsinkevich wrote: >> Hello, >> >> I have a couple of questions: >> >> 1) is there a way to find out what other processes reside in the same >> node as my_rank from within MPICH? > > MPI_GET_PROCESSOR_NAME might be what you are looking for. > > ==rob > > -- > Rob Latham > Mathematics and Computer Science Division > Argonne National Lab, IL USA > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

Re: [mpich-devel] find out what processes are on the same node
by Jeff Hammond 03 Jul '13

03 Jul '13

https://wiki.alcf.anl.gov/parts/index.php/Implementing_a_Symmetric_Heap has an implementation of MPE_Comm_split_node that does a very simple O(Nproc) implementation of MPI_Comm_split_type(type=SHARED) using MPI_Get_processor_name. It also shows the non-portable implementation for Cray and Blue Gene that one can use until MPI-3 is deployed on those systems. Jim Dinan wrote an O(log Nproc) implementation of MPI_Comm_split_type(type=SHARED) that I have somewhere but I don't like it because it's confusing to my small brain. Jeff ----- Original Message ----- From: "Rob Latham" <robl(a)mcs.anl.gov> To: devel(a)mpich.org Sent: Wednesday, July 3, 2013 10:53:15 AM Subject: Re: [mpich-devel] find out what processes are on the same node On Wed, Jul 03, 2013 at 10:39:55AM -0500, Tatiana V. Martsinkevich wrote: > Hello, > > I have a couple of questions: > > 1) is there a way to find out what other processes reside in the same > node as my_rank from within MPICH? MPI_GET_PROCESSOR_NAME might be what you are looking for. ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

Re: [mpich-devel] find out what processes are on the same node
by Rob Latham 03 Jul '13

03 Jul '13

On Wed, Jul 03, 2013 at 10:39:55AM -0500, Tatiana V. Martsinkevich wrote: > Hello, > > I have a couple of questions: > > 1) is there a way to find out what other processes reside in the same > node as my_rank from within MPICH? MPI_GET_PROCESSOR_NAME might be what you are looking for. ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-devel] find out what processes are on the same node
by Jeff Hammond 03 Jul '13

03 Jul '13

Check out the new MPI-3 function MPI_Comm_split_type that is supported by MPICH-3. Best, Jeff ----- Original Message ----- From: "Tatiana V. Martsinkevich" <tanya.manekineko(a)gmail.com> To: devel(a)mpich.org Sent: Wednesday, July 3, 2013 10:39:55 AM Subject: [mpich-devel] find out what processes are on the same node Hello, I have a couple of questions: 1) is there a way to find out what other processes reside in the same node as my_rank from within MPICH? I found this function: MPIR_Comm_is_node_aware(comm_ptr) and if it's true I can get access (although not always?) to comm_ptr->node_comm. I wonder what are the situations when I won't be able to know what processes are on the same node ( i.e. comm_ptr->node_comm is null) and is there any other way to get the info then? 2) MPIR_Comm_is_node_aware() returns true if comm->hierarchy_kind == MPID_HIERARCHY_PARENT and apparently there are the following hierarchies: /* flat, parent, node, or node_roots */ Could someone please explain what are each of the hierarchy? regards, -- Tatiana V. Martsinkevich PhD student INRIA Saclay, University of Paris Sud XI -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0