Puri -- are you involved with the MPI-3 hybrid working group? I have not been following it closely -- are they already talking about such issues? On Jan 22, 2010, at 2:47 PM, Bill Long wrote:

Craig Rasmussen wrote:

...

This is a good question that Bill Long should respond to.

Bill has long claimed that Cray's coarray implementation works fine with their MPI library. I'm not sure what this means precisely. Certainly there is no problem with using the MPI API with a coarray program that I can think of because MPI would just do send/recv using local memory (but I can imagine some interesting race conditions if users aren't careful). I would guess that mpirun would have to know it is running a coarray program so that the two runtimes don't interfere with each other.

It is not necessarily the case that there would be "two runtimes". Vendors have done the work for MPI to launch and execute a parallel program consisting of replicated instances of the program running concurrently and communicating with each other. Fortran 2008 coarrays use the same basic model. There is some additional infrastructure needed for Fortran (and UPC), but this is incremental, not conflicting. It is quite reasonable, though not required by the Fortran standard, that the vendor would just enhance their current runtime and use it for all of MPI, Fortran 2008, and UPC. Cray calls the program launcher 'aprun' rather than 'mpirun', promoting the idea that the same infrastructure is used for all. Such reuse saves development costs, which the bean counters like, and has the desirable side effect of facilitating execution of mixed codes that include any combination of MPI, Fortran 2008, or UPC (or SHMEM, for that matter). I don't think any of the relevant standards specify this sort of commonality, but I expect customers to effectively require this capability.

...

I think running a hybrid MPI and coarray program will be common as MPI has more functionality at the moment (and probably always as it is library based).

I agree, but I think 'hybrid' needs some clarification. I see two, rather distinct cases:

1) Multi-level parallelism. Today this is characterized by MPI programs that use OpenMP within each rank to implement an additional layer of local, shared-memory parallelism. This has been around for years. It is quite reasonable to expect the use of OpenMP within a Fortran 2008 image in just the same way. Several years ago I saw a comparison of running N MPI ranks with M-way OpenMP parallelism within each rank and the same code with OpenMP turned off and run only with MPI using N*M ranks (same total number of processors). In the past, the all-MPI version tended to have better performance. However, with the more recent trends in core counts within a socket (4, 6, 8, 12, ... hundreds for a GPU) the balance is turning the other way. I would expect to see more local, shared-memory parallelism within an MPI rank/Fortran image. It might be from OpenMP (well established, mostly portable), automatic detection by the compiler (my preference), the new do concurrent construct in Fortran 2008 (also fine), or even CUDA/OpenCL (which I don't consider reasonable options for 'normal' programmers, but OK for BLAS-type library writers).

2) Co-existing parallelism at the same level. This would entail using Fortran 2008 parallelism with coarrays, or UPC, as an alternate mechanism to implement parallelism at the same level as MPI. There is a one-to-one correspondence between the MPI rank and Fortran image / UPC process. This could be going on without the programmer's knowledge (for example, parts of the Scalapack library might be written using coarrays), or by the programmer explicitly mixing the methods in the same program.

Assuming that Craig's comment was really about option (2), I would expect that this will be very common. MPI has been around for many years, it has proven to be the most successful scheme so far for parallelism that does not reply on shared memory, and hundreds (thousands?) of programmer-centuries have been invested in creating huge MPI-based codes. With its combination of success and inertia, MPI is not going away for a very very long time, if ever. And it is not likely that a large MPI code will suddenly be scrapped and rewritten entirely with Fortran 2008. Instead, initially some section of a code might be rewritten using coarrays, leaving the rest of the code alone. This has already happened for users with early access to coarrays. This is a quite reasonable scheme for gradually evolving a code. (An potential candidate that comes to mind is in WRF, where the transfer of a very large data structure is coded as hundreds of calls to 'pack' routines followed by a bulk MPI Send call for the resulting buffer, and on the receive side a mirror set of 'unpack' routines. The coarray alternative is a single assignment statement.)

As suggested by the end of Craig's comment, the reverse evolution is also possible. A new code might be developed using coarrays, but might call a highly optimized MPI routine for a particular collective communication pattern. This could be explicit, or hidden in a set of wrapper library routines that internally take advantage of the work that has gone into MPI optimization. The Fortran committee has discussed the inclusion of collective intrinsics in the standard (the topic is currently deferred to a TR). One school of though is that at least the bulk of these should be provided by a LaPack-style library outside the language spec. If we go that route (currently a minority view), a vendor might very well leverage the existing MPI work inside the library.

Cheers, Bill

...

-craig

On Jan 21, 2010, at 6:42 PM, Purushotham Bangalore wrote:

...

The other issue that I remember after the meeting: Since co-arrays will be part of the F08 standard, are they any potential issues that we need to be aware of? Or will the solutions from the Hybrid programming WG solve any potential problems that might arise?

Thanks Puri ----- Original Message ----- From: "Jeff Squyres" <[email protected]> To: "Craig Rasmussen" <[email protected]> Cc: "Craig Rasmussen" <[email protected]>, "Purushotham Bangalore" <[email protected]> Sent: Thursday, January 21, 2010 7:04:05 PM Subject: One more MPI Fortran thing...

Craig -- I forgot to mention this on the phone today: Puri asked in the Atlanta meeting today if we need to say anything about F08 in MPI-3. I.e., do we need a paragraph/page/section/whatever about any issues that we expect might occur with using the explicit interface with F08 compilers?

-- Jeff Squyres [email protected]

_______________________________________________ mpi3-fortran mailing list [email protected] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran

-- Bill Long [email protected] Fortran Technical Support & voice: 651-605-9024 Bioinformatics Software Development fax: 651-605-9142 Cray Inc./Cray Plaza, Suite 210/380 Jackson St./St. Paul, MN 55101

_______________________________________________ mpi3-fortran mailing list [email protected] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-fortran

-- Jeff Squyres [email protected]