Hi all, I took another pass at updating the QMPI API based on our discussion in the working group today. I wasn’t going to post this on the pull request yet since I haven’t yet updated the implementation, but I wanted to start circulating it to get additional feedback. Please let me know if you think I misunderstood something or you think something should change. Thanks, Wes QMPI API Tool Loading The tool should use a system specific option to initialize the library when it's loaded. GCC and other compiler use the __attribute__((constructor) trick to accomplish this. The same can be used during finalization (if simple interception of MPI_FINALIZE is insufficient. Registration When the tool is loaded, it needs to be registered with the MPI library. This happens with this function: int QMPI_Register_tool_name(char * tool_name, void (* init_function_ptr)(int tool_id)); tool_name - [IN] Character string - A unique string representing the name of the tool init_function_ptr - [IN] Function pointer - Pointer to a function that MPI will called before MPI is initialized This function allows a tool to “announce” itself to MPICH, but does not insert a tool into the QMPI call stack. The tool_name provided needs to match the one provided in the environment variable describing the tool order (described later). At some point before MPI is initialized, it will call all of the callback functions provided via init_function_ptr. These functions may be called at any point between the calling of QMPI_Register_tool and executing MPI_Init. For MPICH, this will probably be implemented in the wrapper code that gets executed between when an application calls MPI_Init and when the QMPI tools that intercept MPI_Init are called. The callback function pointer looks like this: void init_function_ptr(int tool_id); tool_id - [OUT] Integer - A unique identifier for the tool This callback function is called one time for each instance of the tool that was requested via the environment variable specifying the order and quantity of tools. Each instance of the tool receives a unique tool_id. These tool IDs are not necessarily monotonically increasing and cannot be assumed to do so. Inside this callback function, the tool should register any tool-specific storage it will to use as well as any MPI functions it plans to intercept. This happens through the following two functions. int QMPI_Register_tool_storage(int tool_id, void *tool_storage); tool_id - [IN] Integer - A unique identifier for the tool tool_storage - [IN] Address - A pointer to an address where the tool has data being stored for its particular instance This function allows the tool to provide a storage object (tool_storage) which lets the tool store information specific to its instance of itself (in case there are multiple copies of a tool). The specific instance of the tool is identified with the tool_id value and should be matched with the one returned by the callback function that the tool registered with MPI. int QMPI_Register_tool_function(int tool_id, enum QMPI_Functions_enum function_enum, void (* function_ptr)(void)) tool_id - [IN] Integer - A unique identifier for the tool function_enum - [IN] Enum value - An enumerated value specifying which MPI function is being registered for interception. function_ptr - [IN] Function pointer - A pointer to a function that should be called when an application calls the specified function. This function accepts the parameter tool_id as the specific instance of a tool and should be matched with the one returned by the callback function that the tool registered with MPI. function_enum is used to specify which MPI function is being registered. The enum values are defined in the enum QMPI_Functions_enum in mpi.h and generally are of this form: enum QMPI_Functions_enum { MPI_ABORT_T, MPI_ACCUMULATE_T, ... snip ... MPI_WTIME_T, MPI_LAST_FUNC }; function_ptr should be a pointer to the tool's interception function pointer. This function pointer will be the same signature as the MPI function, with an additional QMPI_Context argument in the first position as a way for tools to query information about the specific MPI procedure being intercepted when the callback functions are called. The type signature for each MPI function is defined in mpi.h in the form of QMPI_<function_name>_t (e.g., QMPI_Send_t(QMPI_Context context, const void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm)). Interception When the tool's interception function is called, it can get any tool-specific information from the context object. The fields in the context object can be retrieved with these functions: QMPI_Context_get_storage(QMPI_Context context, int tool_id, void **storage); context - [IN] Opaque handle - A handle to a context object containing tool and function-specific information tool_id - [IN] Integer - A unique identifier for the tool storage - [OUT] Address - A pointer to an address where the tool has data being stored for its particular instance Get the back storage object the tool provided to QMPI_Register_tool_storage that matches tool_id. QMPI_Context_get_calling_address(QMPI_Context context, int tool_id, void *calling_address); context - [IN] Opaque handle - A handle to a context object containing tool and function-specific information tool_id - [IN] Integer - A unique identifier for the tool calling_address - [OUT] Address - The address of the application the called the MPI function. The the address of the point where the application originally called the MPI function before entering the QMPI tool stack. If both a PMPI and QMPI tool are registered and active at the same time, the address will be that of the PMPI function, rather than the application. Calling Additional QMPI Tools When the tool is done with its own interception, it should call this function to determine the next tool in the QMPI stack (which the tool itself is responsible for calling): int QMPI_Get_function(int tool_id, enum QMPI_Functions_enum function_enum, void (** function_ptr)(void), QMPI_Context * next_tool_context); tool_id - [IN] Integer - A unique identifier for the tool function_enum - [IN] Enum value - An enumerated value specifying which MPI function is being registered for interception. function_ptr - [OUT] Function pointer - A pointer to a function that should be called when an application calls the specified function. next_tool_context - [OUT] Opaque handle - A handle to a context object containing tool and function-specific information This function also uses tool_id and function_enum to determine the current tool's ID and the enum value for the function being queried. It also provides a function pointer in a similar format to the registration function. Finally, the function returns the context object of the next tool in the call stack which should be used when calling the returned function.