- mpiwg-coll - mailman3.cels.anl.gov

Re: [mpiwg-coll] deprecating MPI_GRAPH_CREATE?
by Jeff Hammond 01 Dec '14

01 Dec '14

But you're not going to write that proposal? You seem more qualified than most... Jeff Sent from my iPhone > On Dec 1, 2014, at 7:51 AM, Torsten Hoefler <htor(a)inf.ethz.ch> wrote: > > Jeff, > > There is no such proposal yet (afaik), I'd support a carefully crafted one and I think I wouldn't be alone. > > Thanks & Best, > Torsten > >> On 01.12.2014 06:41, Jeff Hammond wrote: >> Regarding http://lists.mpi-forum.org/mpiwg-coll/2014/03/0289.php, is >> there a proposal to deprecate MPI_GRAPH_CREATE? It seems that there >> are many issues with it and they are deliberately not getting fixed >> because a subset of the issues with it make it too horrible to fix. >> >> I find this situation distasteful as it leads to a poor user >> experience. Let's do the right thing and put MPI_GRAPH_CREATE out of >> our misery sooner rather than later. >> >> Thanks, >> >> Jeff > > -- > ### qreharg rug ebs fv crryF --- http://htor.inf.ethz.ch/ --- > Torsten Hoefler | Assistant Professor > Dept. of Computer Science | ETH Zürich > Universitätsstrasse 6 | Zurich-8092, Switzerland > CAB E 64.1 | Phone: +41 76 309 79 29 > _______________________________________________ > mpiwg-coll mailing list > mpiwg-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-coll

1 0

Re: [mpiwg-coll] deprecating MPI_GRAPH_CREATE?
by Torsten Hoefler 01 Dec '14

01 Dec '14

Jeff, There is no such proposal yet (afaik), I'd support a carefully crafted one and I think I wouldn't be alone. Thanks & Best, Torsten On 01.12.2014 06:41, Jeff Hammond wrote: > Regarding http://lists.mpi-forum.org/mpiwg-coll/2014/03/0289.php, is > there a proposal to deprecate MPI_GRAPH_CREATE? It seems that there > are many issues with it and they are deliberately not getting fixed > because a subset of the issues with it make it too horrible to fix. > > I find this situation distasteful as it leads to a poor user > experience. Let's do the right thing and put MPI_GRAPH_CREATE out of > our misery sooner rather than later. > > Thanks, > > Jeff > -- ### qreharg rug ebs fv crryF --- http://htor.inf.ethz.ch/ --- Torsten Hoefler | Assistant Professor Dept. of Computer Science | ETH Zï¿½rich Universitï¿½tsstrasse 6 | Zurich-8092, Switzerland CAB E 64.1 | Phone: +41 76 309 79 29

1 0

deprecating MPI_GRAPH_CREATE?
by Jeff Hammond 30 Nov '14

30 Nov '14

Regarding http://lists.mpi-forum.org/mpiwg-coll/2014/03/0289.php, is there a proposal to deprecate MPI_GRAPH_CREATE? It seems that there are many issues with it and they are deliberately not getting fixed because a subset of the issues with it make it too horrible to fix. I find this situation distasteful as it leads to a poor user experience. Let's do the right thing and put MPI_GRAPH_CREATE out of our misery sooner rather than later. Thanks, Jeff -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpiwg-coll] MPI-3.1 ticket 0 change (was ticket 420)
by Daniel Holmes 27 May '14

27 May '14

Hi Torsten, I have looked at both the pdf and the diff. Both look good to me. Cheers, Dan. On 22/05/2014 19:48, Torsten Hoefler wrote: > All, > > Rolf, Martin and I agreed to push this ticket through as ticket 0 since > this is uncontroversial and very minor (and a clear bug in an example, > so no need for discussion or errata). > > A compiled version of MPI-3.1 including this change can be found at > http://htor.inf.ethz.ch/sec/mpi31-report-420.pdf and the pdf diff is at > http://htor.inf.ethz.ch/sec/mpi31-report-420.diff . > > The original report/ticket can be found at > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/420 . > > Please review the change so that I can commit it into MPI 3.1. I'd ask > Rolf, Daniel, Martin, and Jim to review since I'm not sure about the > chapter committee. In addition, everybody on this mailinglist should > double-check. Thanks! > > Deadline for this trivial review is next Friday 5/30. > > Thanks a lot & All the Best, > Torsten > -- Dan Holmes Applications Consultant in HPC Research EPCC, The University of Edinburgh James Clerk Maxwell Building The Kings Buildings Mayfield Road Edinburgh, UK EH9 3JZ T: +44(0)131 651 3465 E: dholmes(a)epcc.ed.ac.uk *Please consider the environment before printing this email.* The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.

1 0

MPI-3.1 ticket 0 change (was ticket 420)
by Torsten Hoefler 22 May '14

22 May '14

All, Rolf, Martin and I agreed to push this ticket through as ticket 0 since this is uncontroversial and very minor (and a clear bug in an example, so no need for discussion or errata). A compiled version of MPI-3.1 including this change can be found at http://htor.inf.ethz.ch/sec/mpi31-report-420.pdf and the pdf diff is at http://htor.inf.ethz.ch/sec/mpi31-report-420.diff . The original report/ticket can be found at https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/420 . Please review the change so that I can commit it into MPI 3.1. I'd ask Rolf, Daniel, Martin, and Jim to review since I'm not sure about the chapter committee. In addition, everybody on this mailinglist should double-check. Thanks! Deadline for this trivial review is next Friday 5/30. Thanks a lot & All the Best, Torsten -- ### qreharg rug ebs fv crryF ------------- http://www.unixer.de/ ----- Torsten Hoefler | Assistant Professor Dept. of Computer Science | ETH Zï¿½rich Universitï¿½tsstrasse 6 | Zurich-8092, Switzerland CAB E 64.1 | Phone: +41 76 309 79 29

1 0

Re: [mpiwg-coll] Question re neighbourhood collectives and graph topologies
by Rolf Rabenseifner 31 Mar '14

31 Mar '14

Dear all, after Torsten's positive review, I stored all (solution + review) in the new ticket #419: https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/419 If technical or English corrections are needed, or if all is okay, I would be grateful if you could write your review as soon as possible to the ticket. Who is at the next meeting and could read the ticket at the meeting? As one-reading-one-vote MPI-3.0 errata ticket, it should still fit to the MPI-3.1 schedule. Best regards Rolf ----- Original Message ----- > From: "Torsten Hoefler" <htor(a)illinois.edu> > To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > Cc: "MPI-3 Collective Subgroup Discussions" <mpiwg-coll(a)lists.mpi-forum.org>, "David Henty" <d.henty(a)epcc.ed.ac.uk>, > "Martin Schulz" <schulzm(a)llnl.gov>, "Purushotham V. Bangalore" <puri(a)uab.edu>, "Shinji Sumimoto" > <s-sumi(a)labs.fujitsu.com>, "Dave Goodell" <dgoodell(a)cisco.com>, "Adam Moody" <moody20(a)llnl.gov> > Sent: Sunday, March 30, 2014 10:10:17 PM > Subject: Re: Fwd: Question re neighbourhood collectives and graph topologies > > Hi Rolf, > > On Tue, Mar 25, 2014 at 12:25:51PM +0100, Rolf Rabenseifner wrote: > > Based on David Henty's question, my analysis seems to imply > > that the definition of the neighborhood collectives for > > MPI_GRAPH_CREATE general graph topologies is fully broken, > > i.e., undefined. > > > > I CC'ed the MPI-3.0 topology chapter committee and send this to the > > MPI-next collective working group. > > > > Reason for my analysis: > > > > MPI-3.0 p295:25-26 allows multiple edges and a non-symmetric > > adjacency matrix. > > > > MPI-3.0 p295:26-27 tells that an adjacency edge from "process" to > > "neighbor" does not imply a communication direction, neither > > process=source nor process=destination. > Yes, this is from the original MPI-1 text and is probably confusing. > > > MPI-3.0 p306:46-p307:3 clearly defines that MPI_GRAPH_NEIGHBORS > > return > > exactly the same adjacency information as defined in > > MPI_GRAPH_CREATE. > > > > MPI-305 p307:11-34 (Example 7.5) uses a non-symmetric adjacency > > Matrix between process 2 and 3: > > > > The three edges between nodes 2 and 3 are: > > 1. process 2 -- neighbor 3 > > 2. process 3 -- neighbor 2 > > 3. process 3 -- neighbor 2 > This is truly horrible! > > > MPI-3.0 p315:11-13 defines: > > > > For a general graph topology, created with MPI_GRAPH_CREATE, > > the order of neighbors in the send and receive buffers is > > defined as the sequence of neighbors as returned by > > MPI_GRAPH_NEIGHBORS. > > > > For process rank 2, the relevant edges returned by > > MPI_GRAPH_NEIGHBORS is one edge to neighbor 3 > > and therefore one send buffer and one receive buffer > > will be defined. > > > > For process rank 3, the relevant edges returned by > > MPI_GRAPH_NEIGHBORS are two edges to neighbor 2 > > and therefore two send buffers and two receive buffers > > will be defined. > > > > This does not match and will cause broken communication! > Yes, we did not check the non-scalable MPI-1 interface for additional > brokenness. We wanted to deprecate it but didn't for various reasons. > Instead, we put strong wording advising to never use it. > > I mean: DO NOT USE MPI_GRAPH_CREATE EVER! :-) > > Yet, we should fix this inconsistency with an erratum. > > > Please do not think that three buffers will be defined, > > because you have to look at the total example, i.e. > > - in process 2 > > -- one send buffer for communication with rank 3 and > > -- one receive buffer for communication with rank 3 > > are used, and > > - in process 3 > > -- three send buffers for communication with ranks 0, 3, 3 and > > -- three receive buffers for communication with rank 0, 3, 3 > > ares used. > > > > The number of buffers between the processes 2 and 3 do not match. > Yes > > > Proposal: > > We should restrict the current definition of collective neighbor > > communication on general graph topologies on those topologies > > that define for each pair of processes on both processes the same > > amount of edges between this pair of processes. > > (This implies a symmetric adjacency matrix, but the adjacency > > information, which also includes the sequence of the edges, > > needs not to be symmetric.) > Well, this seems a bit like a hack, but since nobody should use this > interface anyway, I have no strong feelings. > > > Proposed solution: > > ------------------ > > > > MPI-3.0 page 315 lines 11-14 read > > > > For a general graph topology, created with MPI_GRAPH_CREATE, > > the order of neighbors in the send and receive buffers is > > defined as the sequence of neighbors as returned by > > MPI_GRAPH_NEIGHBORS. Note that general graph topologies > > should generally be replaced by the distributed graph > > topologies. > > > > but should read > > > > For a general graph topology, created with MPI_GRAPH_CREATE, > > | the use of neighborhood collective communication is > > | restricted to adjacency matrices with the number of edges > > | between any two processes is defined to be the same for both > > | processes (i.e., with a symmetric adjacency matrix). > > | In this case, > > the order of neighbors in the send and receive buffers is > > defined as the sequence of neighbors as returned by > > MPI_GRAPH_NEIGHBORS. Note that general graph topologies > > should generally be replaced by the distributed graph > > topologies. > That is ok with me. Rolf, feel free to push this erratum forward, > I'll > support it and we can discuss it in the collectives WG. > > Thanks & All the Best, > Torsten > > -- > ### qreharg rug ebs fv crryF ----------- http://htor.inf.ethz.ch/ > ----- > Torsten Hoefler | Assistant Professor > Dept. of Computer Science | ETH Zürich > Universitätsstrasse 6 | Zurich-8092, Switzerland > CAB E 64.1 | Phone: +41 76 309 79 29 > -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-coll] Fwd: Question re neighbourhood collectives and graph topologies
by Torsten Hoefler 30 Mar '14

30 Mar '14

Hi Rolf, On Tue, Mar 25, 2014 at 12:25:51PM +0100, Rolf Rabenseifner wrote: > Based on David Henty's question, my analysis seems to imply > that the definition of the neighborhood collectives for > MPI_GRAPH_CREATE general graph topologies is fully broken, > i.e., undefined. > > I CC'ed the MPI-3.0 topology chapter committee and send this to the > MPI-next collective working group. > > Reason for my analysis: > > MPI-3.0 p295:25-26 allows multiple edges and a non-symmetric > adjacency matrix. > > MPI-3.0 p295:26-27 tells that an adjacency edge from "process" to > "neighbor" does not imply a communication direction, neither > process=source nor process=destination. Yes, this is from the original MPI-1 text and is probably confusing. > MPI-3.0 p306:46-p307:3 clearly defines that MPI_GRAPH_NEIGHBORS return > exactly the same adjacency information as defined in MPI_GRAPH_CREATE. > > MPI-305 p307:11-34 (Example 7.5) uses a non-symmetric adjacency > Matrix between process 2 and 3: > > The three edges between nodes 2 and 3 are: > 1. process 2 -- neighbor 3 > 2. process 3 -- neighbor 2 > 3. process 3 -- neighbor 2 This is truly horrible! > MPI-3.0 p315:11-13 defines: > > For a general graph topology, created with MPI_GRAPH_CREATE, > the order of neighbors in the send and receive buffers is > defined as the sequence of neighbors as returned by > MPI_GRAPH_NEIGHBORS. > > For process rank 2, the relevant edges returned by > MPI_GRAPH_NEIGHBORS is one edge to neighbor 3 > and therefore one send buffer and one receive buffer > will be defined. > > For process rank 3, the relevant edges returned by > MPI_GRAPH_NEIGHBORS are two edges to neighbor 2 > and therefore two send buffers and two receive buffers > will be defined. > > This does not match and will cause broken communication! Yes, we did not check the non-scalable MPI-1 interface for additional brokenness. We wanted to deprecate it but didn't for various reasons. Instead, we put strong wording advising to never use it. I mean: DO NOT USE MPI_GRAPH_CREATE EVER! :-) Yet, we should fix this inconsistency with an erratum. > Please do not think that three buffers will be defined, > because you have to look at the total example, i.e. > - in process 2 > -- one send buffer for communication with rank 3 and > -- one receive buffer for communication with rank 3 > are used, and > - in process 3 > -- three send buffers for communication with ranks 0, 3, 3 and > -- three receive buffers for communication with rank 0, 3, 3 > ares used. > > The number of buffers between the processes 2 and 3 do not match. Yes > Proposal: > We should restrict the current definition of collective neighbor > communication on general graph topologies on those topologies > that define for each pair of processes on both processes the same > amount of edges between this pair of processes. > (This implies a symmetric adjacency matrix, but the adjacency > information, which also includes the sequence of the edges, > needs not to be symmetric.) Well, this seems a bit like a hack, but since nobody should use this interface anyway, I have no strong feelings. > Proposed solution: > ------------------ > > MPI-3.0 page 315 lines 11-14 read > > For a general graph topology, created with MPI_GRAPH_CREATE, > the order of neighbors in the send and receive buffers is > defined as the sequence of neighbors as returned by > MPI_GRAPH_NEIGHBORS. Note that general graph topologies > should generally be replaced by the distributed graph topologies. > > but should read > > For a general graph topology, created with MPI_GRAPH_CREATE, > | the use of neighborhood collective communication is > | restricted to adjacency matrices with the number of edges > | between any two processes is defined to be the same for both > | processes (i.e., with a symmetric adjacency matrix). > | In this case, > the order of neighbors in the send and receive buffers is > defined as the sequence of neighbors as returned by > MPI_GRAPH_NEIGHBORS. Note that general graph topologies > should generally be replaced by the distributed graph topologies. That is ok with me. Rolf, feel free to push this erratum forward, I'll support it and we can discuss it in the collectives WG. Thanks & All the Best, Torsten -- ### qreharg rug ebs fv crryF ----------- http://htor.inf.ethz.ch/ ----- Torsten Hoefler | Assistant Professor Dept. of Computer Science | ETH Zï¿½rich Universitï¿½tsstrasse 6 | Zurich-8092, Switzerland CAB E 64.1 | Phone: +41 76 309 79 29

1 0

Fwd: Question re neighbourhood collectives and graph topologies
by Rolf Rabenseifner 25 Mar '14

25 Mar '14

Based on David Henty's question, my analysis seems to imply that the definition of the neighborhood collectives for MPI_GRAPH_CREATE general graph topologies is fully broken, i.e., undefined. I CC'ed the MPI-3.0 topology chapter committee and send this to the MPI-next collective working group. Reason for my analysis: MPI-3.0 p295:25-26 allows multiple edges and a non-symmetric adjacency matrix. MPI-3.0 p295:26-27 tells that an adjacency edge from "process" to "neighbor" does not imply a communication direction, neither process=source nor process=destination. MPI-3.0 p306:46-p307:3 clearly defines that MPI_GRAPH_NEIGHBORS return exactly the same adjacency information as defined in MPI_GRAPH_CREATE. MPI-305 p307:11-34 (Example 7.5) uses a non-symmetric adjacency Matrix between process 2 and 3: The three edges between nodes 2 and 3 are: 1. process 2 -- neighbor 3 2. process 3 -- neighbor 2 3. process 3 -- neighbor 2 MPI-3.0 p315:11-13 defines: For a general graph topology, created with MPI_GRAPH_CREATE, the order of neighbors in the send and receive buffers is defined as the sequence of neighbors as returned by MPI_GRAPH_NEIGHBORS. For process rank 2, the relevant edges returned by MPI_GRAPH_NEIGHBORS is one edge to neighbor 3 and therefore one send buffer and one receive buffer will be defined. For process rank 3, the relevant edges returned by MPI_GRAPH_NEIGHBORS are two edges to neighbor 2 and therefore two send buffers and two receive buffers will be defined. This does not match and will cause broken communication! Please do not think that three buffers will be defined, because you have to look at the total example, i.e. - in process 2 -- one send buffer for communication with rank 3 and -- one receive buffer for communication with rank 3 are used, and - in process 3 -- three send buffers for communication with ranks 0, 3, 3 and -- three receive buffers for communication with rank 0, 3, 3 ares used. The number of buffers between the processes 2 and 3 do not match. Proposal: We should restrict the current definition of collective neighbor communication on general graph topologies on those topologies that define for each pair of processes on both processes the same amount of edges between this pair of processes. (This implies a symmetric adjacency matrix, but the adjacency information, which also includes the sequence of the edges, needs not to be symmetric.) Proposed solution: ------------------ MPI-3.0 page 315 lines 11-14 read For a general graph topology, created with MPI_GRAPH_CREATE, the order of neighbors in the send and receive buffers is defined as the sequence of neighbors as returned by MPI_GRAPH_NEIGHBORS. Note that general graph topologies should generally be replaced by the distributed graph topologies. but should read For a general graph topology, created with MPI_GRAPH_CREATE, | the use of neighborhood collective communication is | restricted to adjacency matrices with the number of edges | between any two processes is defined to be the same for both | processes (i.e., with a symmetric adjacency matrix). | In this case, the order of neighbors in the send and receive buffers is defined as the sequence of neighbors as returned by MPI_GRAPH_NEIGHBORS. Note that general graph topologies should generally be replaced by the distributed graph topologies. Best regards Rolf ----- Forwarded Message ----- From: "David Henty" <d.henty(a)epcc.ed.ac.uk> To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> Sent: Monday, March 24, 2014 7:09:00 PM Subject: Question re neighbourhood collectives and graph topologies Rolf, I see you were involved in the definition for the new scalable graph creation functions in MPI 3.0 so I wondered of you could answer a question re their use in neighbourhood collectives. Page 295 of the standard says for MPI_Graph_create that "The definition of a node-neighbor edge does not imply a direction of the communication." This made sense to me when graphs could only be used to look up the ranks of your neighbours: it was presumably up to the user how these were interpreted, e.g. if they implied a send or a receive. However, the new neighbourhood collective operations infer the communications from the graph and do the sends and receives automatically. For the new distributed creation routines this seems clear, e.g. they talk explicitly about "source and destination" or "indegree and outdegree". However, it isn't clear to me how this works with the old graph creation routine. Is it correct to assume that the explicit "neighbours" of each rank are the "out" ones, and the "in" ones are the links from all other processes who specify my rank as a neighbour? This seems the obvious definition to me, but it seems to conflict with the statement that "The definition of a node-neighbor edge does not imply a direction of the communication.". I see you have some pseudocode on how to translate between the different representations in the appendix of your 2010 paper, but I am being lazy and thinking it is quicker (for me if not for you) to you ask you rather than understand the pseudocode! Thanks in advance, David -- The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336. -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)

1 0

Re: [mpiwg-coll] ticket 353
by Jeff Hammond 22 Oct '13

22 Oct '13

Thanks for the clarification. Sorry for misreading the standard. Jeff On Tue, Oct 22, 2013 at 11:37 AM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > MPI_REDUCE_LOCAL is not the same as MPI_REDUCE on MPI_COMM_SELF. The latter just copies inbuf to outbuf without doing any op. Reduce_local performs an op between inbuf and inoutbuf. If inbuf is missing, it doesn't make sense to call Reduce_local. > > On Oct 22, 2013, at 9:51 AM, Jeff Hammond wrote: > >> https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/353 is trivial but >> perhaps I am the only person that sees the point. Can the WG give a >> thumbs up or down on proceeding? >> >> Thanks, >> >> Jeff >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> _______________________________________________ >> mpiwg-coll mailing list >> mpiwg-coll(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-coll > > _______________________________________________ > mpiwg-coll mailing list > mpiwg-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-coll -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpiwg-coll] ticket 353
by Rajeev Thakur 22 Oct '13

22 Oct '13

MPI_REDUCE_LOCAL is not the same as MPI_REDUCE on MPI_COMM_SELF. The latter just copies inbuf to outbuf without doing any op. Reduce_local performs an op between inbuf and inoutbuf. If inbuf is missing, it doesn't make sense to call Reduce_local. On Oct 22, 2013, at 9:51 AM, Jeff Hammond wrote: > https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/353 is trivial but > perhaps I am the only person that sees the point. Can the WG give a > thumbs up or down on proceeding? > > Thanks, > > Jeff > > -- > Jeff Hammond > jeff.science(a)gmail.com > _______________________________________________ > mpiwg-coll mailing list > mpiwg-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-coll

1 0