Hi all, I've got a problem which I think might be a candidate for a fix in the 2.2 standard: Is it legal to make MPI calls in the 'delete' callback for an attribute attached to MPI_COMM_SELF which is being freed during a call to MPI_Finalize? Section 4.8 in the 2.0 standard doesn't really say, I think. Here's the text:
4.8 Allowing User Functions at Process Termination
There are times in which it would be convenient to have actions happen when an MPI process ï¬nishes. For example, a routine may do initializations that are useful until the MPI job (or that part of the job that being terminated in the case of dynamically created processes) is ï¬nished. This can be accomplished in MPI-2 by attaching an attribute to MPI_COMM_SELF with a callback function. When MPI_FINALIZE is called, it will ï¬rst execute the equivalent of an MPI_COMM_FREE on MPI_COMM_SELF. This will cause the delete callback function to be executed on all keys associated with MPI_COMM_SELF, in an arbitrary order. If no key has been attached to MPI_COMM_SELF, then no callback is invoked. The “freeing†of MPI_COMM_SELF occurs before any other parts of MPI are affected. Thus, for example, calling MPI_FINALIZED will return false in any of these callback functions. Once done with MPI_COMM_SELF, the order and rest of the actions taken by MPI_FINALIZE is not speciï¬ed.
Advice to implementors. Since attributes can be added from any supported language, the MPI implementation needs to remember the creating language so the correct callback is made. (End of advice to implementors.)
Here's my use case: we cache modified file information in the HDF5 library and need to be able to flush that data to the file if the user calls MPI_Finalize without calling the HDF5 routines to close (or flush) that file first. It looks like adding an attribute onto MPI_COMM_SELF is the right thing to do, and the standard seems to imply that it's possible to make MPI calls from that callback, but it's not certain to me. (Obviously, the HDF5 library needs to make I/ O calls and possible may need to exchange data or invoke a barrier, etc) Is it reasonable to add a sentence to the effect that "all MPI operations (except MPI_Finalize and MPI_Init) are possible from the attribute's 'delete' callback"? Here's a copy of section 4.8, with text to that effect worked in:
4.8 Allowing User Functions at Process Termination
There are times in which it would be convenient to have actions happen when an MPI process ï¬nishes. For example, a routine may do initializations that are useful until the MPI job (or that part of the job that being terminated in the case of dynamically created processes) is ï¬nished. This can be accomplished in MPI-2 by attaching an attribute to MPI_COMM_SELF with a callback function. When MPI_FINALIZE is called, it will ï¬rst execute the equivalent of an MPI_COMM_FREE on MPI_COMM_SELF. This will cause the delete callback function to be executed on all keys associated with MPI_COMM_SELF, in an arbitrary order. If no key has been attached to MPI_COMM_SELF, then no callback is invoked. The “freeing†of MPI_COMM_SELF occurs before any other parts of MPI are affected and all MPI operations (except MPI_Finalize and MPI_Init) are available to the attribute's 'delete' callback. Thus, for example, calling MPI_FINALIZED will return false in any of these callback functions. Once done with MPI_COMM_SELF, the order and rest of the actions taken by MPI_FINALIZE is not speciï¬ed.
Advice to implementors. Since attributes can be added from any supported language, the MPI implementation needs to remember the creating language so the correct callback is made. (End of advice to implementors.)
Thanks, Quincey Koziol The HDF Group