February 2012 - mpi-forum - mailman3.cels.anl.gov

Chapter committees
by Jeffrey Squyres 20 Feb '12

20 Feb '12

I still only see the old chapter committee list on the web site: http://meetings.mpi-forum.org/mpi3.0_chapter_wgs.php Who is on the chapter committee for groups, contexts, communicators, and caching? (I have a new ticket for a minor issue and want to know who to put on the author list) -- Jeff Squyres jsquyres(a)cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/

1 0

Re: [Mpi-forum] MPI-2.2 errata tickets - may be urgent
by Dave Goodell 20 Feb '12

20 Feb '12

On Feb 18, 2012, at 4:35 PM CST, Rolf Rabenseifner wrote: > Rich, > > please can you add #196 to the agenda. > It is short and should still fit into the Tuesday 12:30-4:30pm slot. > It really needs reading + 2 votes. #196 will not go quickly and should not be handled entirely in 1 meeting. > I agree with Dave on >>>> #166, #170, #172, #177, #179, #181, #183, #186, #191, #197, >>>> #201, and #202 are all trivial text changes > (I withdrew #200 because it is in #260) > > #178 is also trivial. > > #198 needs discussion but is also trivial, i.e., can be done in May. > Or as you said: >>>> My personal view is that this ticket is a waste of time. Also, it's purely a formatting issue. No need to deal with it as a ticket. > With the C++ tickets, I've a little bit different view: > #192 is trivial and should be added into MPI-2.2-Errata > because it cannot be fixed in MPI-3.0 if we remove all C++. > > #59 and #190 are C++ interface changes and require 2 votes and > should be rejected because C++ is deprecated. Fine by me. > #176 should be handled by the I/O chapter committee, but > I expect that the current text is okay (but a bit strange) > and the ticket should be therefore withdrawn. > > #261 should also be withdrawn, because in many other arguments > we have the same meaning of OUT. > > #193, and #199 are already withdrawn. > > And #171 bhas already passed. -Dave

1 0

Re: [Mpi-forum] MPI-IO for mesh based simulations beyond 2 billion elements
by Rob Latham 20 Feb '12

20 Feb '12

On Sun, Feb 19, 2012 at 10:51:42AM -0600, Rajeev Thakur wrote: > Harald, > If all you have is a one-dimensional array, and each process reads or writes a large contiguous chunk of it, you don't need to use Type_create_subarray or even use collective I/O. You can just have each process call MPI_File_read_at or write_at with the right offset. The subarray type is more useful if you have a multidimensional array and need to read a subsection of it that is not contiguously located in the file. Hi Rajeev: that's not entirely true any longer. On BlueGene, collective I/O will carry out aggregation and alignment to block boundaries. On Lustre, collective I/O can conjure up a workload such that all operations from a particular client go to one server. So, anything that allows collective I/O, even for large requests, we should try to encourage. 32 bit MPI_Aint on 32 bit platforms isn't perfect (I think there's some problem in the Fortran cases?) but it's allowed us to actually get work done on BlueGene. ==rob > > Rajeev > > > On Feb 19, 2012, at 4:23 AM, Rolf Rabenseifner wrote: > > > Dear Harald, dear I/O and datatype committee members, > > > > Indeed, you need a workaround. Our LARGE COUNT #265 is not enough. > > The workaround (instead of MPI_TYPE_CREATE_SUBARRAY) is: > > > > 1. Call MPI_FILE_SET_VIEW(fh, 0, vectype, vectype, datarep, MPI_INFO_NULL, iError) > > 2. Call MPI_FILE_GET_TYPE_EXTENT(fh, vectype, vectype_byte_extent, iError) > > 3. Byte_elemOffset = elemOff (as in your source code) * vectype_byte_extent > > Byte_GlobalSize = globElems (as in your source code) * vectype_byte_extent > > 4. Call MPI_Type_contiguous(locElems, vectype, chunktype, iError) > > 5. Call MPI_TYPE_CREATE_RESIZED(chunktype,-Byte_elemOffset, > > Byte_GlobalSize, ftype) > > (I hope that I understood thisroutine correctly > > and here, a negativ lb is needed.) > > > > 6. call MPI_File_set_view( fh, 0_MPI_OFFSET_KIND, & > >> & vectype, ftype, "native", & > >> & MPI_INFO_NULL, iError ) > > > > Any comments by specialists? > > > > Best regards > > Rolf > > > > ----- Original Message ----- > >> From: "Harald Klimach" <h.klimach(a)grs-sim.de> > >> To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > >> Sent: Sunday, February 19, 2012 10:05:40 AM > >> Subject: MPI-IO for mesh based simulations beyond 2 billion elements > >> > >> Dear Rolf, > >> > >> Mesh based simulations are an important backbone of CFD. The mesh > >> sizes we are capable > >> to compute on new HPC systems steadily increases even for complex > >> geometrical layouts. > >> For simple geometries, much larger domains then 2 billion elements > >> where already done > >> 10 years ago, see > >> M. Yokokawa, K. Itakura, A. Uno, T. Ishihara and Y. Kaneda, > >> "16.4-TFlops Direct Numerical Simulation of Turbulence by a Fourier > >> Spectral Method on the Earth Simulator", Proceedings of the 2002 > >> ACM/IEEE Conference on Supercomputing, Baltimore MD (2002). > >> To describe the spatial domain around complex geometries usually some > >> unstructured > >> approach has to be followed to allow a sparse representation. This > >> results in a one-dimensional > >> list of all elements which is distributed across all processes in a > >> parallel simulation. > >> Therefore the natural representation of all elements in each partition > >> is perfectly > >> described by an one-dimensional MPI subarray datatype. > >> This would also be the most natural choice to describe the MPI File > >> view for each > >> partition to enable dumping and reading of results and restart data. > >> However, due to the interface definition of MPI_Type_create_subarray, > >> the global > >> problem size would be limited to roughly 2 billion elements by MPI. > >> > >> Here is the actual implementation we are facing in our Code: > >> We use a Lattice-Boltzmann representation, where each element is > >> described by > >> 19 MPI_DOUBLE_PRECISION values: > >> > >> call MPI_Type_contiguous(19, MPI_DOUBLE_PRECISION, vectype, iError) > >> call MPI_Type_commit( vectype, iError ) > >> > >> This type is then used to build up the subarray description of each > >> partition: > >> > >> call MPI_Type_create_subarray( 1, [ globElems ], [ locElems ], [ > >> elemOff ], & > >> & MPI_ORDER_FORTRAN, vectype, ftype, iError) > >> call MPI_Type_commit( ftype, iError ) > >> > >> Where globElems is the global number of Elements, locElems the number > >> of > >> Elements in the process local partition, and elemOff the Offset of the > >> process > >> local partition in terms of elements. This type is then used to set > >> the view > >> for the local process on a file opened for MPI-IO: > >> > >> call MPI_File_open( MPI_COMM_WORLD, & > >> & 'restart.dat', & > >> & MPI_MODE_WRONLY+MPI_MODE_CREATE, & > >> & MPI_INFO_NULL, fhandle, & > >> & iError ) > >> > >> call MPI_File_set_view( fhandle, 0_MPI_OFFSET_KIND, & > >> & vectype, ftype, "native", & > >> & MPI_INFO_NULL, iError ) > >> > >> Finally the data is written to disk with collective write: > >> > >> call MPI_File_write_all(fhandle, data, locElems, vectype, iostatus, > >> iError) > >> > >> The global mesh is sorted according to a space-filling curve and to > >> provide > >> mostly equal distribution on all processes, the local number of > >> elements is > >> computed by the following integer arithmetic: > >> > >> locElems = globElems / nProcs > >> if (myrank < mod(globElems, nProcs) locElems = locElems + 1 > >> > >> Where nProcs is the number of processes in the communicator and myRank > >> the rank of the local partition in the communicator. > >> Note, that for differently sized elements, the workload varies and the > >> elements > >> shouldn't be distributed equally for work load balance. And with > >> different > >> partitioning strategies other distributions might arise. However in > >> any case it > >> is not generally possible to find common discriminators greater than > >> one > >> across all partitions. > >> Thus it is not easily possible to find a larger etype, which spans > >> more than a > >> single element. > >> > >> What is the recommended implementation for efficient parallel IO > >> within MPI > >> in such a scenario? > >> > >> Thanks a lot, > >> Harald > >> > >> -- > >> Dipl.-Ing. Harald Klimach > >> > >> German Research School for > >> Simulation Sciences GmbH > >> Schinkelstr. 2a > >> 52062 Aachen | Germany > >> > >> Tel +49 241 / 80-99758 > >> Fax +49 241 / 806-99758 > >> Web www.grs-sim.de > >> JID: haraldkl(a)jabber.ccc.de > >> > >> Members: Forschungszentrum Jï¿½lich GmbH | RWTH Aachen University > >> Registered in the commercial register of the local court of > >> Dï¿½ren (Amtsgericht Dï¿½ren) under registration number HRB 5268 > >> Registered office: Jï¿½lich > >> Executive board: Prof. Marek Behr Ph.D. | Dr. Norbert Drewes > > > > -- > > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > > Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) > > > > _______________________________________________ > > mpi-forum mailing list > > mpi-forum(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [Mpi-forum] MPI-IO for mesh based simulations beyond 2 billion elements
by Supalov, Alexander 20 Feb '12

20 Feb '12

Hi Rolf, Intel MPI uses a 32-bit MPI_Aint and MPI_ADDRESS_KIND in the IA32 version of the library, in line with the ilp32 model. Best regards. Alexander -----Original Message----- From: mpi-forum-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Rolf Rabenseifner Sent: Sunday, February 19, 2012 12:22 PM To: Douglas Miller Cc: Main MPI Forum mailing list; Harald Klimach Subject: Re: [Mpi-forum] MPI-IO for mesh based simulations beyond 2 billion elements Doug and all other implementors, is there any platform and MPI library at IBM and elsewhere that has an MPI_Aint and INTEGER(KIND=MPI_ADDRESS_KIND) less than 64 bit? Harald, further details below. Best regards Rolf ----- Original Message ----- > From: "Harald Klimach" <h.klimach(a)grs-sim.de> > To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > Cc: "Main MPI Forum mailing list" <mpi-forum(a)lists.mpi-forum.org> > Sent: Sunday, February 19, 2012 11:55:35 AM > Subject: Re: MPI-IO for mesh based simulations beyond 2 billion elements > Dear Rolf, > > thanks a lot for your quick response and guidance. > MPI_Type_create_resized defines LB and Extent as MPI_ADDRESS_KIND. > What is this going to be on 32-Bit systems like the BlueGene? As far > as I > understand it, my chances for larger numbers are better with > MPI_OFFSET_KIND? > Thus, to me it looks like it is not possible to use the MPI Type > facilities for > large data sets in order to describe the IO layout for one-dimensional > data. > Instead I have to use the disp parameter in MPI_File_set_view. Is > there any > disadvantage attached to this, especially when doing collective IO? > Could MPI-Implementations exploit the data provided in the types > better than > individual disp in the view? With the resized datatype, the MPI library can detect that all filetypes collectively provided in MPI_SET_FILE_VIEW are disjunct! This may allow optimizations that are not possible with defining different disp in MPI_SET_FILE_VIEW; here, the first process will see the whole file, the all other ranks the whole file minus the portion really accessed by the processes with smaller ranks. Rolf > Thanks, > Harald > > Am 19.02.2012 um 11:23 schrieb Rolf Rabenseifner: > > > Dear Harald, dear I/O and datatype committee members, > > > > Indeed, you need a workaround. Our LARGE COUNT #265 is not enough. > > The workaround (instead of MPI_TYPE_CREATE_SUBARRAY) is: > > > > 1. Call MPI_FILE_SET_VIEW(fh, 0, vectype, vectype, datarep, > > MPI_INFO_NULL, iError) > > 2. Call MPI_FILE_GET_TYPE_EXTENT(fh, vectype, vectype_byte_extent, > > iError) > > 3. Byte_elemOffset = elemOff (as in your source code) * > > vectype_byte_extent > > Byte_GlobalSize = globElems (as in your source code) * > > vectype_byte_extent > > 4. Call MPI_Type_contiguous(locElems, vectype, chunktype, iError) > > 5. Call MPI_TYPE_CREATE_RESIZED(chunktype,-Byte_elemOffset, > > Byte_GlobalSize, ftype) > > (I hope that I understood thisroutine correctly > > and here, a negativ lb is needed.) > > > > 6. call MPI_File_set_view( fh, 0_MPI_OFFSET_KIND, & > >> & vectype, ftype, "native", & > >> & MPI_INFO_NULL, iError ) > > > > Any comments by specialists? > > > > Best regards > > Rolf > > > > ----- Original Message ----- > >> From: "Harald Klimach" <h.klimach(a)grs-sim.de> > >> To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > >> Sent: Sunday, February 19, 2012 10:05:40 AM > >> Subject: MPI-IO for mesh based simulations beyond 2 billion > >> elements > >> > >> Dear Rolf, > >> > >> Mesh based simulations are an important backbone of CFD. The mesh > >> sizes we are capable > >> to compute on new HPC systems steadily increases even for complex > >> geometrical layouts. > >> For simple geometries, much larger domains then 2 billion elements > >> where already done > >> 10 years ago, see > >> M. Yokokawa, K. Itakura, A. Uno, T. Ishihara and Y. Kaneda, > >> "16.4-TFlops Direct Numerical Simulation of Turbulence by a Fourier > >> Spectral Method on the Earth Simulator", Proceedings of the 2002 > >> ACM/IEEE Conference on Supercomputing, Baltimore MD (2002). > >> To describe the spatial domain around complex geometries usually > >> some > >> unstructured > >> approach has to be followed to allow a sparse representation. This > >> results in a one-dimensional > >> list of all elements which is distributed across all processes in a > >> parallel simulation. > >> Therefore the natural representation of all elements in each > >> partition > >> is perfectly > >> described by an one-dimensional MPI subarray datatype. > >> This would also be the most natural choice to describe the MPI File > >> view for each > >> partition to enable dumping and reading of results and restart > >> data. > >> However, due to the interface definition of > >> MPI_Type_create_subarray, > >> the global > >> problem size would be limited to roughly 2 billion elements by MPI. > >> > >> Here is the actual implementation we are facing in our Code: > >> We use a Lattice-Boltzmann representation, where each element is > >> described by > >> 19 MPI_DOUBLE_PRECISION values: > >> > >> call MPI_Type_contiguous(19, MPI_DOUBLE_PRECISION, vectype, iError) > >> call MPI_Type_commit( vectype, iError ) > >> > >> This type is then used to build up the subarray description of each > >> partition: > >> > >> call MPI_Type_create_subarray( 1, [ globElems ], [ locElems ], [ > >> elemOff ], & > >> & MPI_ORDER_FORTRAN, vectype, ftype, iError) > >> call MPI_Type_commit( ftype, iError ) > >> > >> Where globElems is the global number of Elements, locElems the > >> number > >> of > >> Elements in the process local partition, and elemOff the Offset of > >> the > >> process > >> local partition in terms of elements. This type is then used to set > >> the view > >> for the local process on a file opened for MPI-IO: > >> > >> call MPI_File_open( MPI_COMM_WORLD, & > >> & 'restart.dat', & > >> & MPI_MODE_WRONLY+MPI_MODE_CREATE, & > >> & MPI_INFO_NULL, fhandle, & > >> & iError ) > >> > >> call MPI_File_set_view( fhandle, 0_MPI_OFFSET_KIND, & > >> & vectype, ftype, "native", & > >> & MPI_INFO_NULL, iError ) > >> > >> Finally the data is written to disk with collective write: > >> > >> call MPI_File_write_all(fhandle, data, locElems, vectype, iostatus, > >> iError) > >> > >> The global mesh is sorted according to a space-filling curve and to > >> provide > >> mostly equal distribution on all processes, the local number of > >> elements is > >> computed by the following integer arithmetic: > >> > >> locElems = globElems / nProcs > >> if (myrank < mod(globElems, nProcs) locElems = locElems + 1 > >> > >> Where nProcs is the number of processes in the communicator and > >> myRank > >> the rank of the local partition in the communicator. > >> Note, that for differently sized elements, the workload varies and > >> the > >> elements > >> shouldn't be distributed equally for work load balance. And with > >> different > >> partitioning strategies other distributions might arise. However in > >> any case it > >> is not generally possible to find common discriminators greater > >> than > >> one > >> across all partitions. > >> Thus it is not easily possible to find a larger etype, which spans > >> more than a > >> single element. > >> > >> What is the recommended implementation for efficient parallel IO > >> within MPI > >> in such a scenario? > >> > >> Thanks a lot, > >> Harald > >> > >> -- > >> Dipl.-Ing. Harald Klimach > >> > >> German Research School for > >> Simulation Sciences GmbH > >> Schinkelstr. 2a > >> 52062 Aachen | Germany > >> > >> Tel +49 241 / 80-99758 > >> Fax +49 241 / 806-99758 > >> Web www.grs-sim.de > >> JID: haraldkl(a)jabber.ccc.de > >> > >> Members: Forschungszentrum Jülich GmbH | RWTH Aachen University > >> Registered in the commercial register of the local court of > >> Düren (Amtsgericht Düren) under registration number HRB 5268 > >> Registered office: Jülich > >> Executive board: Prof. Marek Behr Ph.D. | Dr. Norbert Drewes > > > > -- > > Dr. Rolf Rabenseifner . . . . . . . . . .. email > > rabenseifner(a)hlrs.de > > High Performance Computing Center (HLRS) . phone > > ++49(0)711/685-65530 > > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > > 685-65832 > > Head of Dpmt Parallel Computing . . . > > www.hlrs.de/people/rabenseifner > > Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) > > -- > Dipl.-Ing. Harald Klimach > > German Research School for > Simulation Sciences GmbH > Schinkelstr. 2a > 52062 Aachen | Germany > > Tel +49 241 / 80-99758 > Fax +49 241 / 806-99758 > Web www.grs-sim.de > JID: haraldkl(a)jabber.ccc.de > > Members: Forschungszentrum Jülich GmbH | RWTH Aachen University > Registered in the commercial register of the local court of > Düren (Amtsgericht Düren) under registration number HRB 5268 > Registered office: Jülich > Executive board: Prof. Marek Behr Ph.D. | Dr. Norbert Drewes -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) _______________________________________________ mpi-forum mailing list mpi-forum(a)lists.mpi-forum.org http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -------------------------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen, Deutschland Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr./VAT Registration No.: DE129385895 Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052

1 0

Re: [Mpi-forum] MPI-IO for mesh based simulations beyond 2 billion elements
by Rajeev Thakur 19 Feb '12

19 Feb '12

Harald, If all you have is a one-dimensional array, and each process reads or writes a large contiguous chunk of it, you don't need to use Type_create_subarray or even use collective I/O. You can just have each process call MPI_File_read_at or write_at with the right offset. The subarray type is more useful if you have a multidimensional array and need to read a subsection of it that is not contiguously located in the file. Rajeev On Feb 19, 2012, at 4:23 AM, Rolf Rabenseifner wrote: > Dear Harald, dear I/O and datatype committee members, > > Indeed, you need a workaround. Our LARGE COUNT #265 is not enough. > The workaround (instead of MPI_TYPE_CREATE_SUBARRAY) is: > > 1. Call MPI_FILE_SET_VIEW(fh, 0, vectype, vectype, datarep, MPI_INFO_NULL, iError) > 2. Call MPI_FILE_GET_TYPE_EXTENT(fh, vectype, vectype_byte_extent, iError) > 3. Byte_elemOffset = elemOff (as in your source code) * vectype_byte_extent > Byte_GlobalSize = globElems (as in your source code) * vectype_byte_extent > 4. Call MPI_Type_contiguous(locElems, vectype, chunktype, iError) > 5. Call MPI_TYPE_CREATE_RESIZED(chunktype,-Byte_elemOffset, > Byte_GlobalSize, ftype) > (I hope that I understood thisroutine correctly > and here, a negativ lb is needed.) > > 6. call MPI_File_set_view( fh, 0_MPI_OFFSET_KIND, & >> & vectype, ftype, "native", & >> & MPI_INFO_NULL, iError ) > > Any comments by specialists? > > Best regards > Rolf > > ----- Original Message ----- >> From: "Harald Klimach" <h.klimach(a)grs-sim.de> >> To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> >> Sent: Sunday, February 19, 2012 10:05:40 AM >> Subject: MPI-IO for mesh based simulations beyond 2 billion elements >> >> Dear Rolf, >> >> Mesh based simulations are an important backbone of CFD. The mesh >> sizes we are capable >> to compute on new HPC systems steadily increases even for complex >> geometrical layouts. >> For simple geometries, much larger domains then 2 billion elements >> where already done >> 10 years ago, see >> M. Yokokawa, K. Itakura, A. Uno, T. Ishihara and Y. Kaneda, >> "16.4-TFlops Direct Numerical Simulation of Turbulence by a Fourier >> Spectral Method on the Earth Simulator", Proceedings of the 2002 >> ACM/IEEE Conference on Supercomputing, Baltimore MD (2002). >> To describe the spatial domain around complex geometries usually some >> unstructured >> approach has to be followed to allow a sparse representation. This >> results in a one-dimensional >> list of all elements which is distributed across all processes in a >> parallel simulation. >> Therefore the natural representation of all elements in each partition >> is perfectly >> described by an one-dimensional MPI subarray datatype. >> This would also be the most natural choice to describe the MPI File >> view for each >> partition to enable dumping and reading of results and restart data. >> However, due to the interface definition of MPI_Type_create_subarray, >> the global >> problem size would be limited to roughly 2 billion elements by MPI. >> >> Here is the actual implementation we are facing in our Code: >> We use a Lattice-Boltzmann representation, where each element is >> described by >> 19 MPI_DOUBLE_PRECISION values: >> >> call MPI_Type_contiguous(19, MPI_DOUBLE_PRECISION, vectype, iError) >> call MPI_Type_commit( vectype, iError ) >> >> This type is then used to build up the subarray description of each >> partition: >> >> call MPI_Type_create_subarray( 1, [ globElems ], [ locElems ], [ >> elemOff ], & >> & MPI_ORDER_FORTRAN, vectype, ftype, iError) >> call MPI_Type_commit( ftype, iError ) >> >> Where globElems is the global number of Elements, locElems the number >> of >> Elements in the process local partition, and elemOff the Offset of the >> process >> local partition in terms of elements. This type is then used to set >> the view >> for the local process on a file opened for MPI-IO: >> >> call MPI_File_open( MPI_COMM_WORLD, & >> & 'restart.dat', & >> & MPI_MODE_WRONLY+MPI_MODE_CREATE, & >> & MPI_INFO_NULL, fhandle, & >> & iError ) >> >> call MPI_File_set_view( fhandle, 0_MPI_OFFSET_KIND, & >> & vectype, ftype, "native", & >> & MPI_INFO_NULL, iError ) >> >> Finally the data is written to disk with collective write: >> >> call MPI_File_write_all(fhandle, data, locElems, vectype, iostatus, >> iError) >> >> The global mesh is sorted according to a space-filling curve and to >> provide >> mostly equal distribution on all processes, the local number of >> elements is >> computed by the following integer arithmetic: >> >> locElems = globElems / nProcs >> if (myrank < mod(globElems, nProcs) locElems = locElems + 1 >> >> Where nProcs is the number of processes in the communicator and myRank >> the rank of the local partition in the communicator. >> Note, that for differently sized elements, the workload varies and the >> elements >> shouldn't be distributed equally for work load balance. And with >> different >> partitioning strategies other distributions might arise. However in >> any case it >> is not generally possible to find common discriminators greater than >> one >> across all partitions. >> Thus it is not easily possible to find a larger etype, which spans >> more than a >> single element. >> >> What is the recommended implementation for efficient parallel IO >> within MPI >> in such a scenario? >> >> Thanks a lot, >> Harald >> >> -- >> Dipl.-Ing. Harald Klimach >> >> German Research School for >> Simulation Sciences GmbH >> Schinkelstr. 2a >> 52062 Aachen | Germany >> >> Tel +49 241 / 80-99758 >> Fax +49 241 / 806-99758 >> Web www.grs-sim.de >> JID: haraldkl(a)jabber.ccc.de >> >> Members: Forschungszentrum Jï¿½lich GmbH | RWTH Aachen University >> Registered in the commercial register of the local court of >> Dï¿½ren (Amtsgericht Dï¿½ren) under registration number HRB 5268 >> Registered office: Jï¿½lich >> Executive board: Prof. Marek Behr Ph.D. | Dr. Norbert Drewes > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum

1 0

Re: [Mpi-forum] MPI-IO for mesh based simulations beyond 2 billion elements
by Jeff Hammond 19 Feb '12

19 Feb '12

On Blue Gene/P, MPI_Aint is a 64-bit integer specifically to address this issue, even though addresses are only 32-bit integers. For background and details, please see http://lists.alcf.anl.gov/pipermail/dcmf/2008-February/000060.html and http://lists.mcs.anl.gov/pipermail/mpich-discuss/2009-July/005315.html. Jeff On Sun, Feb 19, 2012 at 6:21 AM, Rolf Rabenseifner <rabenseifner(a)hlrs.de> wrote: > Doug and all other implementors, > > is there any platform and MPI library at IBM and elsewhere that has > an MPI_Aint and INTEGER(KIND=MPI_ADDRESS_KIND) less than 64 bit? > > Harald, further details below. > > Best regards > Rolf > > ----- Original Message ----- >> From: "Harald Klimach" <h.klimach(a)grs-sim.de> >> To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> >> Cc: "Main MPI Forum mailing list" <mpi-forum(a)lists.mpi-forum.org> >> Sent: Sunday, February 19, 2012 11:55:35 AM >> Subject: Re: MPI-IO for mesh based simulations beyond 2 billion elements >> Dear Rolf, >> >> thanks a lot for your quick response and guidance. >> MPI_Type_create_resized defines LB and Extent as MPI_ADDRESS_KIND. >> What is this going to be on 32-Bit systems like the BlueGene? As far >> as I >> understand it, my chances for larger numbers are better with >> MPI_OFFSET_KIND? >> Thus, to me it looks like it is not possible to use the MPI Type >> facilities for >> large data sets in order to describe the IO layout for one-dimensional >> data. >> Instead I have to use the disp parameter in MPI_File_set_view. Is >> there any >> disadvantage attached to this, especially when doing collective IO? >> Could MPI-Implementations exploit the data provided in the types >> better than >> individual disp in the view? > > With the resized datatype, the MPI library can detect that all > filetypes collectively provided in MPI_SET_FILE_VIEW are disjunct! > This may allow optimizations that are not possible with > defining different disp in MPI_SET_FILE_VIEW; here, the first process > will see the whole file, the all other ranks the whole file minus > the portion really accessed by the processes with smaller ranks. > > Rolf > >> Thanks, >> Harald >> >> Am 19.02.2012 um 11:23 schrieb Rolf Rabenseifner: >> >> > Dear Harald, dear I/O and datatype committee members, >> > >> > Indeed, you need a workaround. Our LARGE COUNT #265 is not enough. >> > The workaround (instead of MPI_TYPE_CREATE_SUBARRAY) is: >> > >> > 1. Call MPI_FILE_SET_VIEW(fh, 0, vectype, vectype, datarep, >> > MPI_INFO_NULL, iError) >> > 2. Call MPI_FILE_GET_TYPE_EXTENT(fh, vectype, vectype_byte_extent, >> > iError) >> > 3. Byte_elemOffset = elemOff (as in your source code) * >> > vectype_byte_extent >> > ï¿½ ï¿½Byte_GlobalSize = globElems (as in your source code) * >> > ï¿½ ï¿½vectype_byte_extent >> > 4. Call MPI_Type_contiguous(locElems, vectype, chunktype, iError) >> > 5. Call MPI_TYPE_CREATE_RESIZED(chunktype,-Byte_elemOffset, >> > ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ ï¿½ Byte_GlobalSize, ftype) >> > ï¿½ ï¿½(I hope that I understood thisroutine correctly >> > ï¿½ ï¿½ and here, a negativ lb is needed.) >> > >> > 6. call MPI_File_set_view( fh, 0_MPI_OFFSET_KIND, & >> >> & vectype, ftype, "native", & >> >> & MPI_INFO_NULL, iError ) >> > >> > Any comments by specialists? >> > >> > Best regards >> > Rolf >> > >> > ----- Original Message ----- >> >> From: "Harald Klimach" <h.klimach(a)grs-sim.de> >> >> To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> >> >> Sent: Sunday, February 19, 2012 10:05:40 AM >> >> Subject: MPI-IO for mesh based simulations beyond 2 billion >> >> elements >> >> >> >> Dear Rolf, >> >> >> >> Mesh based simulations are an important backbone of CFD. The mesh >> >> sizes we are capable >> >> to compute on new HPC systems steadily increases even for complex >> >> geometrical layouts. >> >> For simple geometries, much larger domains then 2 billion elements >> >> where already done >> >> 10 years ago, see >> >> M. Yokokawa, K. Itakura, A. Uno, T. Ishihara and Y. Kaneda, >> >> "16.4-TFlops Direct Numerical Simulation of Turbulence by a Fourier >> >> Spectral Method on the Earth Simulator", Proceedings of the 2002 >> >> ACM/IEEE Conference on Supercomputing, Baltimore MD (2002). >> >> To describe the spatial domain around complex geometries usually >> >> some >> >> unstructured >> >> approach has to be followed to allow a sparse representation. This >> >> results in a one-dimensional >> >> list of all elements which is distributed across all processes in a >> >> parallel simulation. >> >> Therefore the natural representation of all elements in each >> >> partition >> >> is perfectly >> >> described by an one-dimensional MPI subarray datatype. >> >> This would also be the most natural choice to describe the MPI File >> >> view for each >> >> partition to enable dumping and reading of results and restart >> >> data. >> >> However, due to the interface definition of >> >> MPI_Type_create_subarray, >> >> the global >> >> problem size would be limited to roughly 2 billion elements by MPI. >> >> >> >> Here is the actual implementation we are facing in our Code: >> >> We use a Lattice-Boltzmann representation, where each element is >> >> described by >> >> 19 MPI_DOUBLE_PRECISION values: >> >> >> >> call MPI_Type_contiguous(19, MPI_DOUBLE_PRECISION, vectype, iError) >> >> call MPI_Type_commit( vectype, iError ) >> >> >> >> This type is then used to build up the subarray description of each >> >> partition: >> >> >> >> call MPI_Type_create_subarray( 1, [ globElems ], [ locElems ], [ >> >> elemOff ], & >> >> & MPI_ORDER_FORTRAN, vectype, ftype, iError) >> >> call MPI_Type_commit( ftype, iError ) >> >> >> >> Where globElems is the global number of Elements, locElems the >> >> number >> >> of >> >> Elements in the process local partition, and elemOff the Offset of >> >> the >> >> process >> >> local partition in terms of elements. This type is then used to set >> >> the view >> >> for the local process on a file opened for MPI-IO: >> >> >> >> call MPI_File_open( MPI_COMM_WORLD, & >> >> & 'restart.dat', & >> >> & MPI_MODE_WRONLY+MPI_MODE_CREATE, & >> >> & MPI_INFO_NULL, fhandle, & >> >> & iError ) >> >> >> >> call MPI_File_set_view( fhandle, 0_MPI_OFFSET_KIND, & >> >> & vectype, ftype, "native", & >> >> & MPI_INFO_NULL, iError ) >> >> >> >> Finally the data is written to disk with collective write: >> >> >> >> call MPI_File_write_all(fhandle, data, locElems, vectype, iostatus, >> >> iError) >> >> >> >> The global mesh is sorted according to a space-filling curve and to >> >> provide >> >> mostly equal distribution on all processes, the local number of >> >> elements is >> >> computed by the following integer arithmetic: >> >> >> >> locElems = globElems / nProcs >> >> if (myrank < mod(globElems, nProcs) locElems = locElems + 1 >> >> >> >> Where nProcs is the number of processes in the communicator and >> >> myRank >> >> the rank of the local partition in the communicator. >> >> Note, that for differently sized elements, the workload varies and >> >> the >> >> elements >> >> shouldn't be distributed equally for work load balance. And with >> >> different >> >> partitioning strategies other distributions might arise. However in >> >> any case it >> >> is not generally possible to find common discriminators greater >> >> than >> >> one >> >> across all partitions. >> >> Thus it is not easily possible to find a larger etype, which spans >> >> more than a >> >> single element. >> >> >> >> What is the recommended implementation for efficient parallel IO >> >> within MPI >> >> in such a scenario? >> >> >> >> Thanks a lot, >> >> Harald >> >> >> >> -- >> >> Dipl.-Ing. Harald Klimach >> >> >> >> German Research School for >> >> Simulation Sciences GmbH >> >> Schinkelstr. 2a >> >> 52062 Aachen | Germany >> >> >> >> Tel +49 241 / 80-99758 >> >> Fax +49 241 / 806-99758 >> >> Web www.grs-sim.de >> >> JID: haraldkl(a)jabber.ccc.de >> >> >> >> Members: Forschungszentrum Jï¿½lich GmbH | RWTH Aachen University >> >> Registered in the commercial register of the local court of >> >> Dï¿½ren (Amtsgericht Dï¿½ren) under registration number HRB 5268 >> >> Registered office: Jï¿½lich >> >> Executive board: Prof. Marek Behr Ph.D. | Dr. Norbert Drewes >> > >> > -- >> > Dr. Rolf Rabenseifner . . . . . . . . . .. email >> > rabenseifner(a)hlrs.de >> > High Performance Computing Center (HLRS) . phone >> > ++49(0)711/685-65530 >> > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / >> > 685-65832 >> > Head of Dpmt Parallel Computing . . . >> > www.hlrs.de/people/rabenseifner >> > Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) >> >> -- >> Dipl.-Ing. Harald Klimach >> >> German Research School for >> Simulation Sciences GmbH >> Schinkelstr. 2a >> 52062 Aachen | Germany >> >> Tel +49 241 / 80-99758 >> Fax +49 241 / 806-99758 >> Web www.grs-sim.de >> JID: haraldkl(a)jabber.ccc.de >> >> Members: Forschungszentrum Jï¿½lich GmbH | RWTH Aachen University >> Registered in the commercial register of the local court of >> Dï¿½ren (Amtsgericht Dï¿½ren) under registration number HRB 5268 >> Registered office: Jï¿½lich >> Executive board: Prof. Marek Behr Ph.D. | Dr. Norbert Drewes > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) > > _______________________________________________ > mpi-forum mailing list > mpi-forum(a)lists.mpi-forum.org > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/old/index.php/User:Jhammond

1 0

Re: [Mpi-forum] MPI-IO for mesh based simulations beyond 2 billion elements
by Rolf Rabenseifner 19 Feb '12

19 Feb '12

Doug and all other implementors, is there any platform and MPI library at IBM and elsewhere that has an MPI_Aint and INTEGER(KIND=MPI_ADDRESS_KIND) less than 64 bit? Harald, further details below. Best regards Rolf ----- Original Message ----- > From: "Harald Klimach" <h.klimach(a)grs-sim.de> > To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > Cc: "Main MPI Forum mailing list" <mpi-forum(a)lists.mpi-forum.org> > Sent: Sunday, February 19, 2012 11:55:35 AM > Subject: Re: MPI-IO for mesh based simulations beyond 2 billion elements > Dear Rolf, > > thanks a lot for your quick response and guidance. > MPI_Type_create_resized defines LB and Extent as MPI_ADDRESS_KIND. > What is this going to be on 32-Bit systems like the BlueGene? As far > as I > understand it, my chances for larger numbers are better with > MPI_OFFSET_KIND? > Thus, to me it looks like it is not possible to use the MPI Type > facilities for > large data sets in order to describe the IO layout for one-dimensional > data. > Instead I have to use the disp parameter in MPI_File_set_view. Is > there any > disadvantage attached to this, especially when doing collective IO? > Could MPI-Implementations exploit the data provided in the types > better than > individual disp in the view? With the resized datatype, the MPI library can detect that all filetypes collectively provided in MPI_SET_FILE_VIEW are disjunct! This may allow optimizations that are not possible with defining different disp in MPI_SET_FILE_VIEW; here, the first process will see the whole file, the all other ranks the whole file minus the portion really accessed by the processes with smaller ranks. Rolf > Thanks, > Harald > > Am 19.02.2012 um 11:23 schrieb Rolf Rabenseifner: > > > Dear Harald, dear I/O and datatype committee members, > > > > Indeed, you need a workaround. Our LARGE COUNT #265 is not enough. > > The workaround (instead of MPI_TYPE_CREATE_SUBARRAY) is: > > > > 1. Call MPI_FILE_SET_VIEW(fh, 0, vectype, vectype, datarep, > > MPI_INFO_NULL, iError) > > 2. Call MPI_FILE_GET_TYPE_EXTENT(fh, vectype, vectype_byte_extent, > > iError) > > 3. Byte_elemOffset = elemOff (as in your source code) * > > vectype_byte_extent > > Byte_GlobalSize = globElems (as in your source code) * > > vectype_byte_extent > > 4. Call MPI_Type_contiguous(locElems, vectype, chunktype, iError) > > 5. Call MPI_TYPE_CREATE_RESIZED(chunktype,-Byte_elemOffset, > > Byte_GlobalSize, ftype) > > (I hope that I understood thisroutine correctly > > and here, a negativ lb is needed.) > > > > 6. call MPI_File_set_view( fh, 0_MPI_OFFSET_KIND, & > >> & vectype, ftype, "native", & > >> & MPI_INFO_NULL, iError ) > > > > Any comments by specialists? > > > > Best regards > > Rolf > > > > ----- Original Message ----- > >> From: "Harald Klimach" <h.klimach(a)grs-sim.de> > >> To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > >> Sent: Sunday, February 19, 2012 10:05:40 AM > >> Subject: MPI-IO for mesh based simulations beyond 2 billion > >> elements > >> > >> Dear Rolf, > >> > >> Mesh based simulations are an important backbone of CFD. The mesh > >> sizes we are capable > >> to compute on new HPC systems steadily increases even for complex > >> geometrical layouts. > >> For simple geometries, much larger domains then 2 billion elements > >> where already done > >> 10 years ago, see > >> M. Yokokawa, K. Itakura, A. Uno, T. Ishihara and Y. Kaneda, > >> "16.4-TFlops Direct Numerical Simulation of Turbulence by a Fourier > >> Spectral Method on the Earth Simulator", Proceedings of the 2002 > >> ACM/IEEE Conference on Supercomputing, Baltimore MD (2002). > >> To describe the spatial domain around complex geometries usually > >> some > >> unstructured > >> approach has to be followed to allow a sparse representation. This > >> results in a one-dimensional > >> list of all elements which is distributed across all processes in a > >> parallel simulation. > >> Therefore the natural representation of all elements in each > >> partition > >> is perfectly > >> described by an one-dimensional MPI subarray datatype. > >> This would also be the most natural choice to describe the MPI File > >> view for each > >> partition to enable dumping and reading of results and restart > >> data. > >> However, due to the interface definition of > >> MPI_Type_create_subarray, > >> the global > >> problem size would be limited to roughly 2 billion elements by MPI. > >> > >> Here is the actual implementation we are facing in our Code: > >> We use a Lattice-Boltzmann representation, where each element is > >> described by > >> 19 MPI_DOUBLE_PRECISION values: > >> > >> call MPI_Type_contiguous(19, MPI_DOUBLE_PRECISION, vectype, iError) > >> call MPI_Type_commit( vectype, iError ) > >> > >> This type is then used to build up the subarray description of each > >> partition: > >> > >> call MPI_Type_create_subarray( 1, [ globElems ], [ locElems ], [ > >> elemOff ], & > >> & MPI_ORDER_FORTRAN, vectype, ftype, iError) > >> call MPI_Type_commit( ftype, iError ) > >> > >> Where globElems is the global number of Elements, locElems the > >> number > >> of > >> Elements in the process local partition, and elemOff the Offset of > >> the > >> process > >> local partition in terms of elements. This type is then used to set > >> the view > >> for the local process on a file opened for MPI-IO: > >> > >> call MPI_File_open( MPI_COMM_WORLD, & > >> & 'restart.dat', & > >> & MPI_MODE_WRONLY+MPI_MODE_CREATE, & > >> & MPI_INFO_NULL, fhandle, & > >> & iError ) > >> > >> call MPI_File_set_view( fhandle, 0_MPI_OFFSET_KIND, & > >> & vectype, ftype, "native", & > >> & MPI_INFO_NULL, iError ) > >> > >> Finally the data is written to disk with collective write: > >> > >> call MPI_File_write_all(fhandle, data, locElems, vectype, iostatus, > >> iError) > >> > >> The global mesh is sorted according to a space-filling curve and to > >> provide > >> mostly equal distribution on all processes, the local number of > >> elements is > >> computed by the following integer arithmetic: > >> > >> locElems = globElems / nProcs > >> if (myrank < mod(globElems, nProcs) locElems = locElems + 1 > >> > >> Where nProcs is the number of processes in the communicator and > >> myRank > >> the rank of the local partition in the communicator. > >> Note, that for differently sized elements, the workload varies and > >> the > >> elements > >> shouldn't be distributed equally for work load balance. And with > >> different > >> partitioning strategies other distributions might arise. However in > >> any case it > >> is not generally possible to find common discriminators greater > >> than > >> one > >> across all partitions. > >> Thus it is not easily possible to find a larger etype, which spans > >> more than a > >> single element. > >> > >> What is the recommended implementation for efficient parallel IO > >> within MPI > >> in such a scenario? > >> > >> Thanks a lot, > >> Harald > >> > >> -- > >> Dipl.-Ing. Harald Klimach > >> > >> German Research School for > >> Simulation Sciences GmbH > >> Schinkelstr. 2a > >> 52062 Aachen | Germany > >> > >> Tel +49 241 / 80-99758 > >> Fax +49 241 / 806-99758 > >> Web www.grs-sim.de > >> JID: haraldkl(a)jabber.ccc.de > >> > >> Members: Forschungszentrum Jülich GmbH | RWTH Aachen University > >> Registered in the commercial register of the local court of > >> Düren (Amtsgericht Düren) under registration number HRB 5268 > >> Registered office: Jülich > >> Executive board: Prof. Marek Behr Ph.D. | Dr. Norbert Drewes > > > > -- > > Dr. Rolf Rabenseifner . . . . . . . . . .. email > > rabenseifner(a)hlrs.de > > High Performance Computing Center (HLRS) . phone > > ++49(0)711/685-65530 > > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / > > 685-65832 > > Head of Dpmt Parallel Computing . . . > > www.hlrs.de/people/rabenseifner > > Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) > > -- > Dipl.-Ing. Harald Klimach > > German Research School for > Simulation Sciences GmbH > Schinkelstr. 2a > 52062 Aachen | Germany > > Tel +49 241 / 80-99758 > Fax +49 241 / 806-99758 > Web www.grs-sim.de > JID: haraldkl(a)jabber.ccc.de > > Members: Forschungszentrum Jülich GmbH | RWTH Aachen University > Registered in the commercial register of the local court of > Düren (Amtsgericht Düren) under registration number HRB 5268 > Registered office: Jülich > Executive board: Prof. Marek Behr Ph.D. | Dr. Norbert Drewes -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

MPI-IO for mesh based simulations beyond 2 billion elements
by Rolf Rabenseifner 19 Feb '12

19 Feb '12

Dear Harald, dear I/O and datatype committee members, Indeed, you need a workaround. Our LARGE COUNT #265 is not enough. The workaround (instead of MPI_TYPE_CREATE_SUBARRAY) is: 1. Call MPI_FILE_SET_VIEW(fh, 0, vectype, vectype, datarep, MPI_INFO_NULL, iError) 2. Call MPI_FILE_GET_TYPE_EXTENT(fh, vectype, vectype_byte_extent, iError) 3. Byte_elemOffset = elemOff (as in your source code) * vectype_byte_extent Byte_GlobalSize = globElems (as in your source code) * vectype_byte_extent 4. Call MPI_Type_contiguous(locElems, vectype, chunktype, iError) 5. Call MPI_TYPE_CREATE_RESIZED(chunktype,-Byte_elemOffset, Byte_GlobalSize, ftype) (I hope that I understood thisroutine correctly and here, a negativ lb is needed.) 6. call MPI_File_set_view( fh, 0_MPI_OFFSET_KIND, & > & vectype, ftype, "native", & > & MPI_INFO_NULL, iError ) Any comments by specialists? Best regards Rolf ----- Original Message ----- > From: "Harald Klimach" <h.klimach(a)grs-sim.de> > To: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > Sent: Sunday, February 19, 2012 10:05:40 AM > Subject: MPI-IO for mesh based simulations beyond 2 billion elements > > Dear Rolf, > > Mesh based simulations are an important backbone of CFD. The mesh > sizes we are capable > to compute on new HPC systems steadily increases even for complex > geometrical layouts. > For simple geometries, much larger domains then 2 billion elements > where already done > 10 years ago, see > M. Yokokawa, K. Itakura, A. Uno, T. Ishihara and Y. Kaneda, > "16.4-TFlops Direct Numerical Simulation of Turbulence by a Fourier > Spectral Method on the Earth Simulator", Proceedings of the 2002 > ACM/IEEE Conference on Supercomputing, Baltimore MD (2002). > To describe the spatial domain around complex geometries usually some > unstructured > approach has to be followed to allow a sparse representation. This > results in a one-dimensional > list of all elements which is distributed across all processes in a > parallel simulation. > Therefore the natural representation of all elements in each partition > is perfectly > described by an one-dimensional MPI subarray datatype. > This would also be the most natural choice to describe the MPI File > view for each > partition to enable dumping and reading of results and restart data. > However, due to the interface definition of MPI_Type_create_subarray, > the global > problem size would be limited to roughly 2 billion elements by MPI. > > Here is the actual implementation we are facing in our Code: > We use a Lattice-Boltzmann representation, where each element is > described by > 19 MPI_DOUBLE_PRECISION values: > > call MPI_Type_contiguous(19, MPI_DOUBLE_PRECISION, vectype, iError) > call MPI_Type_commit( vectype, iError ) > > This type is then used to build up the subarray description of each > partition: > > call MPI_Type_create_subarray( 1, [ globElems ], [ locElems ], [ > elemOff ], & > & MPI_ORDER_FORTRAN, vectype, ftype, iError) > call MPI_Type_commit( ftype, iError ) > > Where globElems is the global number of Elements, locElems the number > of > Elements in the process local partition, and elemOff the Offset of the > process > local partition in terms of elements. This type is then used to set > the view > for the local process on a file opened for MPI-IO: > > call MPI_File_open( MPI_COMM_WORLD, & > & 'restart.dat', & > & MPI_MODE_WRONLY+MPI_MODE_CREATE, & > & MPI_INFO_NULL, fhandle, & > & iError ) > > call MPI_File_set_view( fhandle, 0_MPI_OFFSET_KIND, & > & vectype, ftype, "native", & > & MPI_INFO_NULL, iError ) > > Finally the data is written to disk with collective write: > > call MPI_File_write_all(fhandle, data, locElems, vectype, iostatus, > iError) > > The global mesh is sorted according to a space-filling curve and to > provide > mostly equal distribution on all processes, the local number of > elements is > computed by the following integer arithmetic: > > locElems = globElems / nProcs > if (myrank < mod(globElems, nProcs) locElems = locElems + 1 > > Where nProcs is the number of processes in the communicator and myRank > the rank of the local partition in the communicator. > Note, that for differently sized elements, the workload varies and the > elements > shouldn't be distributed equally for work load balance. And with > different > partitioning strategies other distributions might arise. However in > any case it > is not generally possible to find common discriminators greater than > one > across all partitions. > Thus it is not easily possible to find a larger etype, which spans > more than a > single element. > > What is the recommended implementation for efficient parallel IO > within MPI > in such a scenario? > > Thanks a lot, > Harald > > -- > Dipl.-Ing. Harald Klimach > > German Research School for > Simulation Sciences GmbH > Schinkelstr. 2a > 52062 Aachen | Germany > > Tel +49 241 / 80-99758 > Fax +49 241 / 806-99758 > Web www.grs-sim.de > JID: haraldkl(a)jabber.ccc.de > > Members: Forschungszentrum Jülich GmbH | RWTH Aachen University > Registered in the commercial register of the local court of > Düren (Amtsgericht Düren) under registration number HRB 5268 > Registered office: Jülich > Executive board: Prof. Marek Behr Ph.D. | Dr. Norbert Drewes -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

Re: [Mpi-forum] Agenda corrections needed - Re: March MPI Forum meeting
by Rolf Rabenseifner 18 Feb '12

18 Feb '12

Dear Forum members, because it is two weeks before the meeting, I would like to have the agenda finalized. I would like to have the following changes. I'm not sure whether Rich may add this to the agenda this weekend, therefore please take this email as a formal notation of additional tickets that should be handled in Chicago. Because the changes are small, the time-slots should be still okay. - Please can you move existing #294 -- from voting block Tuesday 10:30-11:30am -- to reading block Tuesday 12:30-4:30pm - Please add #196 Change MPI_OFFSET external32 size to 16 Bytes (Ritzdorf, Rabenseifner) #264 Scalable Variants of vector collectives (Höfler, Siebert) formal re-reading of small changes marked with 264.1 to the block Tuesday 12:30-4:30pm (Both should need only a few minutes because the changes are small) - At last meeting we tried to find people who proceed with tickets with authors not coming to the meeting. We put me on the list for #158 and #187, i.e. in the agenda: -- Traeff/Rabenseifner #158 -- Ritzdorf/rabenseifner #187 - On Monday, Jeff and the forum should be prepared for a short formal vote for all ticket 0, therefore please subsitute Tuesday 8:45-9:30am: "Existing ticket 0 -" by "Existing ticket 0 (reading and vote) -" - Please put also #321 MPI_WAIT/TEST language unclear about non blocking I/O, Generalized, or collective requests. into the ticket-0 list. - Please add (e.g. at the end of the agenda): "If there is time, formal reading of tickets - Use of [] in output arrays ticket #125 - Consistent use of [] for input arrays ticket #126 - "same arguments" to MPI_Reduce ticket #218 " Best regards Rolf -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0

Re: [Mpi-forum] MPI-2.2 errata tickets - may be urgent
by Rolf Rabenseifner 18 Feb '12

18 Feb '12

Rich, please can you add #196 to the agenda. It is short and should still fit into the Tuesday 12:30-4:30pm slot. It really needs reading + 2 votes. I agree with Dave on > > > #166, #170, #172, #177, #179, #181, #183, #186, #191, #197, > > > #201, and #202 are all trivial text changes (I withdrew #200 because it is in #260) #178 is also trivial. #198 needs discussion but is also trivial, i.e., can be done in May. Or as you said: > > > My personal view is that this ticket is a waste of time. With the C++ tickets, I've a little bit different view: #192 is trivial and should be added into MPI-2.2-Errata because it cannot be fixed in MPI-3.0 if we remove all C++. #59 and #190 are C++ interface changes and require 2 votes and should be rejected because C++ is deprecated. #176 should be handled by the I/O chapter committee, but I expect that the current text is okay (but a bit strange) and the ticket should be therefore withdrawn. #261 should also be withdrawn, because in many other arguments we have the same meaning of OUT. #193, and #199 are already withdrawn. And #171 bhas already passed. Rolf ----- Original Message ----- > From: "Rolf Rabenseifner" <rabenseifner(a)hlrs.de> > To: "Main MPI Forum mailing list" <mpi-forum(a)lists.mpi-forum.org> > Sent: Saturday, February 18, 2012 8:31:52 PM > Subject: Re: [Mpi-forum] MPI-2.2 errata tickets - may be urgent > Yes, we need to do the changes for 3.0. > I'll add the Errata-Mark in the ticket that we can put it > also into the errata pdf for 2.2. > > We should keep the trivial things outside the errata list, > i.e., to change them all into MPI-3.0 tickets. > > Ticket 321 may be also added into the errata list > because we never want to see an 2.2-compliant library > handling MPI_Wait for I/O-requests wrong. > > We should not spend time to an 2.2-errata discussion, > but we should formally read the necessary (and very small) > tickets in Chicago. > > Rolf > > ----- Original Message ----- > > From: "William Gropp" <wgropp(a)illinois.edu> > > To: "Main MPI Forum mailing list" <mpi-forum(a)lists.mpi-forum.org> > > Sent: Saturday, February 18, 2012 1:17:16 AM > > Subject: Re: [Mpi-forum] MPI-2.2 errata tickets - may be urgent > > I agree with Dave on these, particularly the trivial text changes. > > > > The way that these were handled for MPI 2.0 errata was by providing > > a > > web page and forum for discussion, followed by creating a detailed > > document with the changes (which we already have for previous 2.2 > > errata), which was then voted on, as a group and once, through an > > email vote of the Forum (this preceded the current sequence of > > meetings). For anything that is an errata item, this process has > > worked well. > > > > I think it is too late to make the change in 196 for MPI 2.2, but it > > should be considered for 3.0 (by the chapter committee). > > > > Bill > > > > On Feb 17, 2012, at 12:59 PM, Dave Goodell wrote: > > > > > ==Executive summary== > > > > > > Only #196 and #198 might possibly need to be discussed by and > > > voted > > > upon by the whole Forum. However, neither is an especially > > > pressing > > > or interesting issue. All of the rest of the tickets just require > > > that the chapter authors are diligent and correct all of the small > > > issues contained in these tickets for their respective chapters. > > > > > > > > > ==Detailed review== > > > > > > #166, #170, #172, #177, #179, #181, #183, #186, #191, #197, #200, > > > #201, and #202 are all trivial text changes that should be > > > absorbed > > > by the appropriate chapter committees. ***We must not waste our > > > time > > > reading these and voting on them in the whole Forum*** > > > > > > As long as C++ is deprecated then we should avoid taking any > > > action > > > on #59, #190, or #192. However #190 and #192 are trivial if the > > > relevant chapter committees want to fix them. > > > > > > #199 is a very small (sort of) problem w.r.t. our inconsistent > > > IN/OUT/INOUT usage. I'm inclined to say that we should leave this > > > up > > > to the chapter committee as well. Fab could do us all a favor by > > > withdrawing it again. > > > > > > #176, #178, and #261 aren't trivial, but they can easily be > > > handled > > > by a short discussion within the relevant chapter committees. > > > > > > I've withdrawn #193 as a duplicate of #201. > > > > > > #196 might warrant some discussion, although I personally think > > > that > > > the issue is pointless. > > > > > > #198 is something that needs to be handled by chapter > > > authors/committees, although the correct action to take would need > > > to be decided by a beauty contest at the forum level. My personal > > > view is that this ticket is a waste of time. > > > > > > -Dave > > > > > > On Feb 17, 2012, at 11:33 AM CST, Rolf Rabenseifner wrote: > > > > > >> Rich and all Forum members, > > >> > > >> it looks like that the whole Forum forgot that there are also > > >> MPI-2.2 errata tickets: > > >> > > >> https://svn.mpi-forum.org/trac/mpi-forum-web/query?status=accepted&status=a… > > >> > > >> 1 is passed > > >> 6 are waiting for reviews > > >> 17 seems to need some work > > >> > > >> Who may volonteer to look at these 23 issues = bugs in the > > >> current > > >> MPI-2.2 > > >> > > >> As long as they are all ticket 0 they can be done in Japan. > > >> > > >> 5 tickets are "correction to the standard" and usually need > > >> 1 reading + 2 vote meetings. > > >> This means that some of these tickets are urgent in the sense > > >> that they should be scheduled and read in the March meeting. > > >> > > >> The forum should be responsible to remove known bugs. > > > > > > > > > _______________________________________________ > > > mpi-forum mailing list > > > mpi-forum(a)lists.mpi-forum.org > > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > > > > William Gropp > > Director, Parallel Computing Institute > > Deputy Director for Research > > Institute for Advanced Computing Applications and Technologies > > Paul and Cynthia Saylor Professor of Computer Science > > University of Illinois Urbana-Champaign > > > > > > > > > > _______________________________________________ > > mpi-forum mailing list > > mpi-forum(a)lists.mpi-forum.org > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum > > -- > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) -- Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner(a)hlrs.de High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30)

1 0