November 2011 - mpiwg-coll - mailman3.cels.anl.gov

Re: [mpi3-coll] Reduction in neighborhood collectives, persistence?
by Jed Brown 27 Nov '11

27 Nov '11

Thanks Torsten. Untrimmed quoting because I'm Cc'ing petsc-maint. On Sun, Nov 27, 2011 at 18:59, Torsten Hoefler <htor(a)illinois.edu> wrote: > Dear Jed, > > Thanks for your interest :-). Comments inline. > > > I was looking through the collectives document from > > [1]http://svn.mpi-forum.org/trac2/mpi-forum-web/ticket/258 There > appear to > > be some missing functions such as MPI_Neighbor_allreduce(), > > MPI_Neighbor_allgatherw(), and others. > One rule for including new functions into MPI-3.0 was to present a > string use-case to the MPI Forum. I am the author of the neighborhood > collectives and found use-cases for the proposed functions but failed to > find good use-cases for MPI_Neighbor_allreduce() and > MPI_Neighbor_allgatherw(). > > All use-cases I have are in Hoefler, Lorenzen, Lumsdaine: "Sparse > Non-Blocking Collectives in Quantum Mechanical Calculations"; > http://www.unixer.de/publications/index.php?pub=70 . > > In particular, the Forum decided against MPI_Neighbor_allgatherw() > because there is no MPI_(All)Gatherw() in MPI-2.2. Once somebody finds > it necessary to add those functions to the dense collectives, they will > be added to the sparse collectives as well. If you have a strong > use-case where the datatypes are performance-critical, then please let > us know. > Okay, I actually think the "w" variant is more useful for sparse collectives. I have a natural use case relating to unstructured mesh management, but I currently prefer to use one-sided comm there because it's more of a setup operation in the applications I'm targeting. > > The MPI_Neighbor_allreducew() text actually still exists in the draft > proposal (commented out :-). But as mentioned, I (as a computer > scientist) was lacking application use-cases. > > > There is a comment on that ticket about dropping reduce due to lack > > of use cases. Well, off the top of my head, MPI_Neighbor_allreducew() > > is ideal for: > > 1. sparse matrix-vector multiplication using column-oriented > distribution > > (or transpose multiplication with row-oriented distribution, or > symmetric > > formats) > > 2. the update in symmetric additive Schwarz > > 3. basic linear algebraic operations involving partially assembled > matrix > > formats such as those found in non-overlapping domain decomposition > > methods > > 4. basic linear algebraic operations involving nested or bordered matrix > > formats such as show up in multiphysics applications or when solving > > optimization problems > > 5. finite element residual evaluation using a non-overlapping element > > partition (the most common way to implement) > > 6. finite volume or discontinuous Galerkin flux accumulation using > > non-overlapping face partition (cell/element partition with redundant > flux > > computation is more common, but both are used) > > If provided, we would use it immediately in PETSc for the first four > > (through our VecScatter object which is also used in examples for > > scenarios 5 and 6). Is this a sufficient number/importance of use cases > to > > justify neighbor reduce? (Did anyone actually try to come up with use > > cases?) > Well, I was working with applications that couldn't benefit from those > use-cases and I don't know the details of PETSc. We need to make a > strong point for the MPI Forum to support the new functions. I could > provide a reference implementation of those functions if you would be > willing to plug them into PETSc to show that the provided semantics are > useful. > If you provide a reference implementation, I can make PETSc's VecScatter support it. We have many applications that use exactly that operation in a performance-sensitive way, so they will benefit if the performance is better than we get with the other alternatives (we currently have runtime-selectable implementations using persistent point-to-point, one-sided, dense MPI_Alltoallw() with most entries empty, and a couple others mostly intended for debugging). > > > MPI_Neighbor_allgatherw() is much less essential, but it's still > > convenient to avoid packing. (Packing by the caller isn't hard, but that > > argument makes all "w" versions option. It looks inconsistent to skip > just > > this one case. Also, the performance of packing can be delicate in a > NUMA > > environment, so it would be good to do it all through a common > mechanism.) > Yes, I couldn't agree more. But the Forum requires use-cases (in this > case, an example that provides higher performance by using those new > functions). > > > Of course all of these should have non-blocking variants (we never use > the > > blocking versions in PETSc). > Sure. > > > I hope it's not too late to get these in. > For MPI-3.0, it may be but MPI-3.1 should come forward soon. > > > Also, what happened to persistent collectives (including these > > neighborhood versions)? This page looks pretty > > bare: [2]https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/PersColl > > We would be especially interested in persistent versions of the > > neighborhood "v" and "w" variants because it lets us build a persistent > > handle so that the MPI implementation can determine all message sizes > once > > in setup instead of needing to rebuild it in each call (of which there > may > > be thousands or millions). In my opinion, not offering it is an implicit > > declaration that you think there is no setup that can be amortized > across > > calls. This seems unlikely to me in the case of neighbor collectives. > I agree but all proposals regarding persistence are moved to the > persistence working group headed by Anthony Skjellum. I am not sure > about the status. > > All the Best, > Torsten > > -- > bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- > Torsten Hoefler | Performance Modeling and Simulation Lead > Blue Waters Directorate | University of Illinois (UIUC) > 1205 W Clark Street | Urbana, IL, 61801 > NCSA Building | +01 (217) 244-7736 > _______________________________________________ > mpi3-coll mailing list > mpi3-coll(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-coll >

1 0

Re: [mpi3-coll] Reduction in neighborhood collectives, persistence?
by Torsten Hoefler 27 Nov '11

27 Nov '11

Dear Jed, Thanks for your interest :-). Comments inline. > I was looking through the collectives document from > [1]http://svn.mpi-forum.org/trac2/mpi-forum-web/ticket/258 There appear to > be some missing functions such as MPI_Neighbor_allreduce(), > MPI_Neighbor_allgatherw(), and others. One rule for including new functions into MPI-3.0 was to present a string use-case to the MPI Forum. I am the author of the neighborhood collectives and found use-cases for the proposed functions but failed to find good use-cases for MPI_Neighbor_allreduce() and MPI_Neighbor_allgatherw(). All use-cases I have are in Hoefler, Lorenzen, Lumsdaine: "Sparse Non-Blocking Collectives in Quantum Mechanical Calculations"; http://www.unixer.de/publications/index.php?pub=70 . In particular, the Forum decided against MPI_Neighbor_allgatherw() because there is no MPI_(All)Gatherw() in MPI-2.2. Once somebody finds it necessary to add those functions to the dense collectives, they will be added to the sparse collectives as well. If you have a strong use-case where the datatypes are performance-critical, then please let us know. The MPI_Neighbor_allreducew() text actually still exists in the draft proposal (commented out :-). But as mentioned, I (as a computer scientist) was lacking application use-cases. > There is a comment on that ticket about dropping reduce due to lack > of use cases. Well, off the top of my head, MPI_Neighbor_allreducew() > is ideal for: > 1. sparse matrix-vector multiplication using column-oriented distribution > (or transpose multiplication with row-oriented distribution, or symmetric > formats) > 2. the update in symmetric additive Schwarz > 3. basic linear algebraic operations involving partially assembled matrix > formats such as those found in non-overlapping domain decomposition > methods > 4. basic linear algebraic operations involving nested or bordered matrix > formats such as show up in multiphysics applications or when solving > optimization problems > 5. finite element residual evaluation using a non-overlapping element > partition (the most common way to implement) > 6. finite volume or discontinuous Galerkin flux accumulation using > non-overlapping face partition (cell/element partition with redundant flux > computation is more common, but both are used) > If provided, we would use it immediately in PETSc for the first four > (through our VecScatter object which is also used in examples for > scenarios 5 and 6).ï¿½Is this a sufficient number/importance of use cases to > justify neighbor reduce? (Did anyone actually try to come up with use > cases?) Well, I was working with applications that couldn't benefit from those use-cases and I don't know the details of PETSc. We need to make a strong point for the MPI Forum to support the new functions. I could provide a reference implementation of those functions if you would be willing to plug them into PETSc to show that the provided semantics are useful. > MPI_Neighbor_allgatherw() is much less essential, but it's still > convenient to avoid packing. (Packing by the caller isn't hard, but that > argument makes all "w" versions option. It looks inconsistent to skip just > this one case. Also, the performance of packing can be delicate in a NUMA > environment, so it would be good to do it all through a common mechanism.) Yes, I couldn't agree more. But the Forum requires use-cases (in this case, an example that provides higher performance by using those new functions). > Of course all of these should have non-blocking variants (we never use the > blocking versions in PETSc). Sure. > I hope it's not too late to get these in. For MPI-3.0, it may be but MPI-3.1 should come forward soon. > Also, what happened to persistent collectives (including these > neighborhood versions)? This page looks pretty > bare:ï¿½[2]https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/PersColl > We would be especially interested in persistent versions of the > neighborhood "v" and "w" variants because it lets us build a persistent > handle so that the MPI implementation can determine all message sizes once > in setup instead of needing to rebuild it in each call (of which there may > be thousands or millions). In my opinion, not offering it is an implicit > declaration that you think there is no setup that can be amortized across > calls. This seems unlikely to me in the case of neighbor collectives. I agree but all proposals regarding persistence are moved to the persistence working group headed by Anthony Skjellum. I am not sure about the status. All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

[mpi3-coll] Reduction in neighborhood collectives, persistence?
by Jed Brown 24 Nov '11

24 Nov '11

I was looking through the collectives document from http://svn.mpi-forum.org/trac2/mpi-forum-web/ticket/258 There appear to be some missing functions such as MPI_Neighbor_allreduce(), MPI_Neighbor_allgatherw(), and others. There is a comment on that ticket about dropping reduce due to lack of use cases. Well, off the top of my head, MPI_Neighbor_allreducew() is ideal for: 1. sparse matrix-vector multiplication using column-oriented distribution (or transpose multiplication with row-oriented distribution, or symmetric formats) 2. the update in symmetric additive Schwarz 3. basic linear algebraic operations involving partially assembled matrix formats such as those found in non-overlapping domain decomposition methods 4. basic linear algebraic operations involving nested or bordered matrix formats such as show up in multiphysics applications or when solving optimization problems 5. finite element residual evaluation using a non-overlapping element partition (the most common way to implement) 6. finite volume or discontinuous Galerkin flux accumulation using non-overlapping face partition (cell/element partition with redundant flux computation is more common, but both are used) If provided, we would use it immediately in PETSc for the first four (through our VecScatter object which is also used in examples for scenarios 5 and 6). Is this a sufficient number/importance of use cases to justify neighbor reduce? (Did anyone actually try to come up with use cases?) MPI_Neighbor_allgatherw() is much less essential, but it's still convenient to avoid packing. (Packing by the caller isn't hard, but that argument makes all "w" versions option. It looks inconsistent to skip just this one case. Also, the performance of packing can be delicate in a NUMA environment, so it would be good to do it all through a common mechanism.) Of course all of these should have non-blocking variants (we never use the blocking versions in PETSc). I hope it's not too late to get these in. Also, what happened to persistent collectives (including these neighborhood versions)? This page looks pretty bare: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/PersColl We would be especially interested in persistent versions of the neighborhood "v" and "w" variants because it lets us build a persistent handle so that the MPI implementation can determine all message sizes once in setup instead of needing to rebuild it in each call (of which there may be thousands or millions). In my opinion, not offering it is an implicit declaration that you think there is no setup that can be amortized across calls. This seems unlikely to me in the case of neighbor collectives.

1 0

[mpi3-coll] Next Telecon
by Torsten Hoefler 22 Nov '11

22 Nov '11

Dear Collective & Topology Working Group, We would like to schedule a telecon to discuss the scalable vector collective proposal #264 and issues with MPI_Icomm_dup #168 (if any). Please fill in your availability in the following doodle http://www.doodle.com/d8b6g88zips52g8n by the end of this week! Both proposals are scheduled for a reading during the next Forum. Please review both proposals and post comments. Thanks & Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [mpi3-coll] [MPI Forum] Ticket #168 MPI_Icomm_dup
by Torsten Hoefler 22 Nov '11

22 Nov '11

Jim, We talked about it and you agreed after a short explanation. Let me summarize our discussion for the others here. > I have a couple questions about this proposal: > > Why MPI_Icomm_dup() and not MPI_Icomm_create()? It seems like > MPI_Icomm_create would be more general while still providing the same > capability. Comm_create is a generalization of comm_dup and may thus be a more general solution. However, the only use-case that we know is for libraries (comm_dup) and comm_dup provides due to it's limited semantics an easier and potentially better-performing solution. > Why is this crucial for nonblocking libraries? It seems like a > synchronous initialization routine for such a library should be > sufficient from a functionality point of view. Is this primarily a > performance argument? Maybe you can point me to a specific passage in > the paper? It's in Section 3.3 in http://www.unixer.de/publications/index.php?pub=135 . A synchronous initialization would be necessary for *each* communicator *and* needs to propagate upwards in stacked libraries (forcing synchronous initialization of all levels in the call-stack above). This does not compose and can easily create a mess. All the Best, Torsten -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [mpi3-coll] [MPI Forum] Ticket #168 MPI_Icomm_dup
by James Dinan 21 Nov '11

21 Nov '11

Hi Torsten, I have a couple questions about this proposal: Why MPI_Icomm_dup() and not MPI_Icomm_create()? It seems like MPI_Icomm_create would be more general while still providing the same capability. Why is this crucial for nonblocking libraries? It seems like a synchronous initialization routine for such a library should be sufficient from a functionality point of view. Is this primarily a performance argument? Maybe you can point me to a specific passage in the paper? Best, ~Jim. On 11/13/11 9:25 AM, Torsten Hoefler wrote: > Hi Collective WG members and "Groups, Contexts, and Communicators" Committee, > > We discussed the functionality of a nonblocking communicator duplication > during MPI-2.2 and deferred the ticket (#164) to MPI-3 because we didn't > want to add too much at that point. > > This functionality is crucial for the implementation of nonblocking > libraries (and the library concept in general) in MPI. For a detailed > discussion see Hoefler, Snir: "Writing Parallel Libraries with MPI - > Common Practice, Issues, and Extensions". > > I beefed up the ticket to make it a full MPI-3 ticket and I would like > to have a reading at the January meeting. Everybody who is in the "To" > field of this message, please review this ticket in the context of the > groups and communicators and collectives chapters and comment (also > indicate when it's ok). Everybody else on the ML is invited to review > too. > > The collective operations and topologies WG will have another telecon > where this ticket is discussed soon. > > Thanks& Best, > Torsten > > PS: please check if I got your username in the CC field right, and > correct it if it is wrong (I didn't find an index for those). >

1 0

[mpi3-coll] [MPI Forum] Ticket #168 MPI_Icomm_dup
by Torsten Hoefler 13 Nov '11

13 Nov '11

Hi Collective WG members and "Groups, Contexts, and Communicators" Committee, We discussed the functionality of a nonblocking communicator duplication during MPI-2.2 and deferred the ticket (#164) to MPI-3 because we didn't want to add too much at that point. This functionality is crucial for the implementation of nonblocking libraries (and the library concept in general) in MPI. For a detailed discussion see Hoefler, Snir: "Writing Parallel Libraries with MPI - Common Practice, Issues, and Extensions". I beefed up the ticket to make it a full MPI-3 ticket and I would like to have a reading at the January meeting. Everybody who is in the "To" field of this message, please review this ticket in the context of the groups and communicators and collectives chapters and comment (also indicate when it's ok). Everybody else on the ML is invited to review too. The collective operations and topologies WG will have another telecon where this ticket is discussed soon. Thanks & Best, Torsten PS: please check if I got your username in the CC field right, and correct it if it is wrong (I didn't find an index for those). -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Re: [mpi3-coll] Missing reduce_local and reduce_scatter_block in create_op text
by Torsten Hoefler 12 Nov '11

12 Nov '11

Hi Adam, > There are a few lists that are missing MPI_REDUCE_LOCAL and > MPI_REDUCE_SCATTER_BLOCK. > > 1) Section 5.9.2 > List of predefined ops missing MPI_REDUCE_LOCAL, > MPI_ALLREDUCE_SCATTER_BLOCK, and all non-blocking variants. > > 2) Description for MPI_OP_CREATE missing MPI_REDUCE_LOCAL, > MPI_REDUCE_SCATTER_BLOCK, and all non-blocking variants. > > I guess we may not need to list the non-blocking variants in this list, > but we should definitely add REDUCE_LOCAL and REDUCE_SCATTER_BLOCK. Good catch -- thanks. I fixed both in mpi-3.0-minutiae (clear Ticket 0). I also mentioned the nonblocking variants. All the Best, Torsten

1 0

Re: [mpi3-coll] [Mpi-forum] MPI_Comm_create_group Meeting
by James Dinan 08 Nov '11

08 Nov '11

Hi All, Notes from yesterday's meeting are attached. Please note that we are moving all future discussion of this ticket to the mpi3-coll mailing list. Best, ~Jim. In attendance: Adam Moody, David Goodell, Jim Dinan, Pavan Balaji, Fab Tiller, Bronis de Supinski, Jeff Hammond Item 1: Name of the routine. We decided to stick with MPI_Comm_create group as the best way of matching the MPI_CLASS_ACTION_SUBSET naming convention. Item 2: Tag versus color. We decided to use a tag, which is defined to be in a collective tag space that is distinct from the point-to-point tag space. The collective tag will follow all MPI tag semantics, however it will not conflict with any point-to-point tags (including MPI_ANY_TAG). This essentially exposes the expected implementation in the standard, providing more flexible semantics to the user. In addition, it allows an implementation to make more efficient use of the tag space by defining tag conflicts across tagged collective operations (currently only MPI_Intercomm_create and MPI_Comm_create_group, but we could define others in the future). In a separate ticket, we can update MPI_Intercomm_create to specify that its tag is also in the collective tag space. This will be backward compatible, but will make MPI_Intercomm_create considerably easier to use because its tags will no longer conflict with point-to-point operations. Tasks: * Jim will work on a revision of the text that follows the above semantics. * The group will reconvene after Thanksgiving to review new text. * The group will move discussion to the collectives mailing list. On 11/4/11 11:07 AM, James Dinan wrote: > Based on the doodle poll, it looks like Monday at 3pm CT will be the > best time to meet. We'll use the hybrid WG telecon bridge: > > Domestic dial number: 866-654-6744 > International dial number: 517-308-8226 > Passcode: 6086006 > > See you then, > ~Jim. > > On 11/1/11 2:33 PM, James Dinan wrote: >> Hi All, >> >> I would like to form a group to review the MPI_Comm_create_group() >> proposal (#286) for a formal reading at the next Forum meeting. If you >> are interested in joining the discussion, could you please fill in your >> availability for a one hour conference call next week: >> >> http://www.doodle.com/9tganzq2hqiqy7kt >> >> We have a couple issues to discuss: >> >> 1.) The name of the function: MPI_Comm_create_group, >> MPI_Group_comm_create, MPI_Comm_create_from_group, ... >> >> 2.) The tag argument: Should we use a tag or a "color" to establish a >> safe communication conduit on the parent communicator. As discussed at >> the Forum, using a tag could conflict with the use of MPI_ANY_TAG on the >> parent communicator. A color would avoid this issue, however it could >> complicate tag matching. >> >> 3.) Changes to the text. >> >> Thanks, >> ~Jim. >> >> PS- I've attached a copy of the current text with comments from the >> formal reading at the last Forum meeting (in Acrobat: Comments -> >> Comments View -> Show Comments List for the full list). >> >> >> _______________________________________________ >> mpi-forum mailing list >> mpi-forum(a)lists.mpi-forum.org >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum

1 0