February 2012 - mpi-forum - mailman3.cels.anl.gov

Re: [Mpi-forum] Agenda corrections needed - Re: March MPI Forum meeting
by Torsten Hoefler 23 Feb '12

23 Feb '12

All, It turns out that I have to be in Washington on the Tuesday of the Forum. I will send one of my students to vote and take notes, however, it would be great if we could move the things for which my presence is required (i.e., me reading things). I'll be around on Mon until 7pm and Wed full day. Sorry, Torsten On Sat, Feb 18, 2012 at 11:48:08PM +0100, Rolf Rabenseifner wrote: > Dear Forum members, > > because it is two weeks before the meeting, I would like to have > the agenda finalized. > > I would like to have the following changes. > I'm not sure whether Rich may add this to the agenda this weekend, > therefore please take this email as a formal notation of additional > tickets that should be handled in Chicago. > Because the changes are small, the time-slots should be still okay. > > - Please can you move existing #294 > -- from voting block Tuesday 10:30-11:30am > -- to reading block Tuesday 12:30-4:30pm > > - Please add > #196 Change MPI_OFFSET external32 size to 16 Bytes (Ritzdorf, Rabenseifner) > #264 Scalable Variants of vector collectives (Hï¿½fler, Siebert) > formal re-reading of small changes marked with 264.1 > to the block Tuesday 12:30-4:30pm > (Both should need only a few minutes because the changes are small) > > - At last meeting we tried to find people who proceed with tickets > with authors not coming to the meeting. > We put me on the list for #158 and #187, i.e. in the agenda: > -- Traeff/Rabenseifner #158 > -- Ritzdorf/rabenseifner #187 > > - On Monday, Jeff and the forum should be prepared for a short > formal vote for all ticket 0, therefore please subsitute > Tuesday 8:45-9:30am: > "Existing ticket 0 -" > by > "Existing ticket 0 (reading and vote) -" > > - Please put also > #321 MPI_WAIT/TEST language unclear about > non blocking I/O, Generalized, or collective requests. > into the ticket-0 list. > > - Please add (e.g. at the end of the agenda): > > "If there is time, formal reading of tickets > - Use of [] in output arrays ticket #125 > - Consistent use of [] for input arrays ticket #126 > - "same arguments" to MPI_Reduce ticket #218 " > > Best regards > Rolf > > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- bash$ :(){ :|:&};: --------------------- http://www.unixer.de/ ----- Torsten Hoefler | Performance Modeling and Simulation Lead Blue Waters Directorate | University of Illinois (UIUC) 1205 W Clark Street | Urbana, IL, 61801 NCSA Building | +01 (217) 244-7736

1 0

Dinner Wednesday March 7
by Bronis R. de Supinski 22 Feb '12

22 Feb '12

All: I now see that the agenda has us ending at 3:30PM on Wednesday. Since we are going so late the other days, I thought we had also agreed to go until 5PM. As a result, I am flying back on Thursday morning. If anyone else is staying until Thursday, perhaps they would like to join me at my favorite restaurant, La Petit Folie (http://www.lapetitefolie.com) A friend of mine is the owner/chef so I have been often. Please let me know if you are interested. Thanks, Bronis

1 0

ticker 321 ready for review
by Fab Tillier 20 Feb '12

20 Feb '12

Hi folks, I know there's debate as to whether 321 should or shouldn't be handled by the chapter authors or as a formal ticket. While that debate rages, I thought I'd let everyone know that the changes are ready for review if you care to. I will be ready for a formal reading of this at the upcoming meeting should one be needed. You can find the PDF here: https://svn.mpi-forum.org/trac/mpi-forum-web/raw-attachment/ticket/321/tick… Please keep in mind that this mail is not intended to serve as a discussion point about Forum process. It is intended to give those interested in the changes an opportunity to read them, regardless of how we proceed. Cheers, -Fab

1 0

Ticket 278 ready for review
by Fab Tillier 20 Feb '12

20 Feb '12

Hi Folks, I have completed the fixes to ticket 278 (https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/278) requested by the Forum at the last meeting, and have attached an updated PDF to the ticket ready for review: https://svn.mpi-forum.org/trac/mpi-forum-web/raw-attachment/ticket/278/tick… This ticket also addresses issues from ticket 12 and 102, and I have asked the respective owners to review the changes, so with any luck those two tickets will be closed as duplicates of this one too. I'll be performing a formal reading at the March meeting. Let me know if you have any issues/comments with the changes. Thanks, -Fab

1 0

Re: [Mpi-forum] MPI-IO for mesh based simulations beyond 2 billion elements
by Rajeev Thakur 20 Feb '12

20 Feb '12

> Rajeev points out that if it's a 1d array just use > MPI_File_read_at_all or MPI_File_write_at_all. The offsets in those > calls take MPI_Offset types. That is without creating a subarray type. Rajeev On Feb 20, 2012, at 2:09 PM, Rob Latham wrote: > On Mon, Feb 20, 2012 at 09:26:28AM -0600, Rob Latham wrote: >> On Sun, Feb 19, 2012 at 10:51:42AM -0600, Rajeev Thakur wrote: >>> Harald, >>> If all you have is a one-dimensional array, and each process reads or writes a large contiguous chunk of it, you don't need to use Type_create_subarray or even use collective I/O. You can just have each process call MPI_File_read_at or write_at with the right offset. The subarray type is more useful if you have a multidimensional array and need to read a subsection of it that is not contiguously located in the file. >> >> Hi Rajeev: that's not entirely true any longer. On BlueGene, >> collective I/O will carry out aggregation and alignment to block >> boundaries. On Lustre, collective I/O can conjure up a workload such >> that all operations from a particular client go to one server. >> >> So, anything that allows collective I/O, even for large requests, we >> should try to encourage. >> >> 32 bit MPI_Aint on 32 bit platforms isn't perfect (I think there's >> some problem in the Fortran cases?) but it's allowed us to actually >> get work done on BlueGene. > > oh, gosh, right, 64 bit MPI_Aint on a platform with 32 bit integers. > > Rajeev points out that if it's a 1d array just use > MPI_File_read_at_all or MPI_File_write_at_all. The offsets in those > calls take MPI_Offset types. > > ==rob > > -- > Rob Latham > Mathematics and Computer Science Division > Argonne National Lab, IL USA > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum

1 0

Re: [Mpi-forum] MPI-IO for mesh based simulations beyond 2 billion elements
by Rob Latham 20 Feb '12

20 Feb '12

On Mon, Feb 20, 2012 at 09:26:28AM -0600, Rob Latham wrote: > On Sun, Feb 19, 2012 at 10:51:42AM -0600, Rajeev Thakur wrote: > > Harald, > > If all you have is a one-dimensional array, and each process reads or writes a large contiguous chunk of it, you don't need to use Type_create_subarray or even use collective I/O. You can just have each process call MPI_File_read_at or write_at with the right offset. The subarray type is more useful if you have a multidimensional array and need to read a subsection of it that is not contiguously located in the file. > > Hi Rajeev: that's not entirely true any longer. On BlueGene, > collective I/O will carry out aggregation and alignment to block > boundaries. On Lustre, collective I/O can conjure up a workload such > that all operations from a particular client go to one server. > > So, anything that allows collective I/O, even for large requests, we > should try to encourage. > > 32 bit MPI_Aint on 32 bit platforms isn't perfect (I think there's > some problem in the Fortran cases?) but it's allowed us to actually > get work done on BlueGene. oh, gosh, right, 64 bit MPI_Aint on a platform with 32 bit integers. Rajeev points out that if it's a 1d array just use MPI_File_read_at_all or MPI_File_write_at_all. The offsets in those calls take MPI_Offset types. ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [Mpi-forum] MPI-IO for mesh based simulations beyond 2 billion elements
by Douglas Miller 20 Feb '12

20 Feb '12

Keep in mind, BG/P is a 64-bit MPI_AINT on a 32-bit platform. _______________________________________________ Douglas Miller BlueGene Messaging Development IBM Corp., Rochester, MN USA Bldg 030-2 A401 dougmill(a)us.ibm.com Douglas Miller/Rochester/IBM Rob Latham <robl(a)mcs.anl.gov > To Sent by: Main MPI Forum mailing list mpi-forum-bounces <mpi-forum(a)lists.mpi-forum.org>, @lists.mpi-forum. cc org Harald Klimach <h.klimach(a)grs-sim.de> Subject 02/20/2012 09:26 Re: [Mpi-forum] MPI-IO for mesh AM based simulations beyond 2 billion elements Please respond to Main MPI Forum mailing list <mpi-forum@lists. mpi-forum.org> On Sun, Feb 19, 2012 at 10:51:42AM -0600, Rajeev Thakur wrote: > Harald, > If all you have is a one-dimensional array, and each process reads or writes a large contiguous chunk of it, you don't need to use Type_create_subarray or even use collective I/O. You can just have each process call MPI_File_read_at or write_at with the right offset. The subarray type is more useful if you have a multidimensional array and need to read a subsection of it that is not contiguously located in the file. Hi Rajeev: that's not entirely true any longer. On BlueGene, collective I/O will carry out aggregation and alignment to block boundaries. On Lustre, collective I/O can conjure up a workload such that all operations from a particular client go to one server. So, anything that allows collective I/O, even for large requests, we should try to encourage. 32 bit MPI_Aint on 32 bit platforms isn't perfect (I think there's some problem in the Fortran cases?) but it's allowed us to actually get work done on BlueGene. ==rob > > Rajeev > > > On Feb 19, 2012, at 4:23 AM, Rolf Rabenseifner wrote: > > > Dear Harald, dear I/O and datatype committee members, > > > > Indeed, you need a workaround. Our LARGE COUNT #265 is not enough. > > The workaround (instead of MPI_TYPE_CREATE_SUBARRAY) is: > > > > 1. Call MPI_FILE_SET_VIEW(fh, 0, vectype, vectype, datarep, MPI_INFO_NULL, iError) > > 2. Call MPI_FILE_GET_TYPE_EXTENT(fh, vectype, vectype_byte_extent, iError) > > 3. Byte_elemOffset = elemOff (as in your source code) * vectype_byte_extent > > Byte_GlobalSize = globElems (as in your source code) * vectype_byte_extent > > 4. Call MPI_Type_contiguous(locElems, vectype, chunktype, iError) > > 5. Call MPI_TYPE_CREATE_RESIZED(chunktype,-Byte_elemOffset, > > Byte_GlobalSize, ftype) > > (I hope that I understood thisroutine correctly > > and here, a negativ lb is needed.) > > > > 6. call MPI_File_set_view( fh, 0_MPI_OFFSET_KIND, & > >> & vectype, ftype, "native", & > >> & MPI_INFO_NULL, iError ) > > > > Any comments by specialists? > > > > Best regards > > Rolf > > > > ----- Original Message ----- > >> From: "Harald Klimach" <h.klimach(a)grs-sim.de> > >> To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > >> Sent: Sunday, February 19, 2012 10:05:40 AM > >> Subject: MPI-IO for mesh based simulations beyond 2 billion elements > >> > >> Dear Rolf, > >> > >> Mesh based simulations are an important backbone of CFD. The mesh > >> sizes we are capable > >> to compute on new HPC systems steadily increases even for complex > >> geometrical layouts. > >> For simple geometries, much larger domains then 2 billion elements > >> where already done > >> 10 years ago, see > >> M. Yokokawa, K. Itakura, A. Uno, T. Ishihara and Y. Kaneda, > >> "16.4-TFlops Direct Numerical Simulation of Turbulence by a Fourier > >> Spectral Method on the Earth Simulator", Proceedings of the 2002 > >> ACM/IEEE Conference on Supercomputing, Baltimore MD (2002). > >> To describe the spatial domain around complex geometries usually some > >> unstructured > >> approach has to be followed to allow a sparse representation. This > >> results in a one-dimensional > >> list of all elements which is distributed across all processes in a > >> parallel simulation. > >> Therefore the natural representation of all elements in each partition > >> is perfectly > >> described by an one-dimensional MPI subarray datatype. > >> This would also be the most natural choice to describe the MPI File > >> view for each > >> partition to enable dumping and reading of results and restart data. > >> However, due to the interface definition of MPI_Type_create_subarray, > >> the global > >> problem size would be limited to roughly 2 billion elements by MPI. > >> > >> Here is the actual implementation we are facing in our Code: > >> We use a Lattice-Boltzmann representation, where each element is > >> described by > >> 19 MPI_DOUBLE_PRECISION values: > >> > >> call MPI_Type_contiguous(19, MPI_DOUBLE_PRECISION, vectype, iError) > >> call MPI_Type_commit( vectype, iError ) > >> > >> This type is then used to build up the subarray description of each > >> partition: > >> > >> call MPI_Type_create_subarray( 1, [ globElems ], [ locElems ], [ > >> elemOff ], & > >> & MPI_ORDER_FORTRAN, vectype, ftype, iError) > >> call MPI_Type_commit( ftype, iError ) > >> > >> Where globElems is the global number of Elements, locElems the number > >> of > >> Elements in the process local partition, and elemOff the Offset of the > >> process > >> local partition in terms of elements. This type is then used to set > >> the view > >> for the local process on a file opened for MPI-IO: > >> > >> call MPI_File_open( MPI_COMM_WORLD, & > >> & 'restart.dat', & > >> & MPI_MODE_WRONLY+MPI_MODE_CREATE, & > >> & MPI_INFO_NULL, fhandle, & > >> & iError ) > >> > >> call MPI_File_set_view( fhandle, 0_MPI_OFFSET_KIND, & > >> & vectype, ftype, "native", & > >> & MPI_INFO_NULL, iError ) > >> > >> Finally the data is written to disk with collective write: > >> > >> call MPI_File_write_all(fhandle, data, locElems, vectype, iostatus, > >> iError) > >> > >> The global mesh is sorted according to a space-filling curve and to > >> provide > >> mostly equal distribution on all processes, the local number of > >> elements is > >> computed by the following integer arithmetic: > >> > >> locElems = globElems / nProcs > >> if (myrank < mod(globElems, nProcs) locElems = locElems + 1 > >> > >> Where nProcs is the number of processes in the communicator and myRank > >> the rank of the local partition in the communicator. > >> Note, that for differently sized elements, the workload varies and the > >> elements > >> shouldn't be distributed equally for work load balance. And with > >> different > >> partitioning strategies other distributions might arise. However in > >> any case it > >> is not generally possible to find common discriminators greater than > >> one > >> across all partitions. > >> Thus it is not easily possible to find a larger etype, which spans > >> more than a > >> single element. > >> > >> What is the recommended implementation for efficient parallel IO > >> within MPI > >> in such a scenario? > >> > >> Thanks a lot, > >> Harald > >> > >> -- > >> Dipl.-Ing. Harald Klimach > >> > >> German Research School for > >> Simulation Sciences GmbH > >> Schinkelstr. 2a > >> 52062 Aachen | Germany > >> > >> Tel +49 241 / 80-99758 > >> Fax +49 241 / 806-99758 > >> Web www.grs-sim.de > >> JID: haraldkl(a)jabber.ccc.de > >> > >> Members: Forschungszentrum Jülich GmbH | RWTH Aachen University > >> Registered in the commercial register of the local court of > >> Düren (Amtsgericht Düren) under registration number HRB 5268 > >> Registered office: Jülich > >> Executive board: Prof. Marek Behr Ph.D. | Dr. Norbert Drewes > > > > -- > > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > > Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) > > > > _______________________________________________ > > mpi-forum mailing list > > mpi-forum(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum

1 0

Re: [Mpi-forum] Chapter committees
by Pavan Balaji 20 Feb '12

20 Feb '12

I was supposed to be on there as well. Already sent a bunch of reminders about it. Who's *really* handling this? On 02/20/2012 10:59 AM, Graham, Richard L. wrote: > This is the updated list: > > Groups, Contexts, and Communicators : Bill Gropp(c), Puri Bangalore, Bronis de Supinski, James Dinan, Dave Goodell, Josh Hursey, Adam Moody > On Feb 20, 2012, at 6:54 PM, Jeffrey Squyres wrote: > > http://meetings.mpi-forum.org/mpi3.0_chapter_wgs.php > > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [Mpi-forum] Chapter committees
by Jeffrey Squyres 20 Feb '12

20 Feb '12

Thanks. On Feb 20, 2012, at 11:59 AM, Graham, Richard L. wrote: > This is the updated list: > > Groups, Contexts, and Communicators : Bill Gropp(c), Puri Bangalore, Bronis de Supinski, James Dinan, Dave Goodell, Josh Hursey, Adam Moody > On Feb 20, 2012, at 6:54 PM, Jeffrey Squyres wrote: > > http://meetings.mpi-forum.org/mpi3.0_chapter_wgs.php > > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [Mpi-forum] Chapter committees
by Graham, Richard L. 20 Feb '12

20 Feb '12

This is the updated list: Groups, Contexts, and Communicators : Bill Gropp(c), Puri Bangalore, Bronis de Supinski, James Dinan, Dave Goodell, Josh Hursey, Adam Moody On Feb 20, 2012, at 6:54 PM, Jeffrey Squyres wrote: http://meetings.mpi-forum.org/mpi3.0_chapter_wgs.php

1 0