Hi all, I know QMPI wasn’t on the agenda for todays WG call, but I did finish a major update to the prototype with most of the things we discussed at the last two calls. I wanted to get that in front of people with my latest thoughts to try to make progress in the meantime. You can see everything on the pull request here: https://github.com/pmodels/mpich/pull/4715 <https://github.com/pmodels/mpich/pull/4715> And the sample tool here: https://gist.github.com/wesbland/065de74b3582913c3945099af5cf5130 <https://gist.github.com/wesbland/065de74b3582913c3945099af5cf5130> I’ll pull out some of the relevant bits. First, here’s what’s changed since last time we spoke: Correct the use of function pointers to use void(*)(void) instead of void *. Add a tool-defined context object that is provided to MPI on registration. Use a system-specific way of registering the tool rather than using dlopen and dlsym (e.g. using _attribute__((constructor))). Create an MPI-level function to register each tool function rather that using a struct of function pointers. Create an MPI-level function to get the next function in the stack rather than querying the array of structs of function pointers. Create an MPI-exposed enum to let tools specify which MPI function they are referencing when registering or querying a function. Create MPI-exposed typedefs for each function to allow the tool to use them if desired. Basically the entire API needed to change to reflect the changes we discussed, specifically handling a void * context for the tool and not using a “magic” registration function. I’ve documented the current version of the API on the pull request. Here are some of the issues that I noted as I was putting this together: 1. I’m not using an enum to determine where to store a particular pointer and to retrieve it quickly (as opposed to string parsing or if statements). One of my concerns is whether using this enum will cause similar problems to forward compatibility that we discussed with the struct of function pointers. For example, if the tool is compiled with MPI 5 QMPI and MPI_LAST_FUNC is 400 and is run against MPI 5.1 QMPI where MPI_LAST_FUNC is 410, that’s an issue. We need to be specific about saying that the user should never use MPI_LAST_FUNC and MPI libraries should separate the MPI_ functions and the MPIX_X functions that they also expose to avoid future collisions. 2. I’ve added typedefs for each MPI function in mpi.h primarily as a convenience, but that may or may not be strictly necessary. If we end up with an API like this, we will want to consider whether that’s required. 3. I’m using the __attribute__((constructor)) method of having libraries bootstrap themselves (which is more portable than I realized). There are other options for compilers like MSVC, but I don’t know if there’s a solution for *everyone* that we can point to to avoid having to standardize a registration function. One option is to create an *optional* function that could be loaded up if it were needed. 4. In addition to figuring out how to bootstrap libraries, it’s also not clear how to specify the ordering of loading libraries (and how to call them). At the moment, I’m using essentially a stack (MPI is 0, first tool is 1, etc.) and when starting the call stack, I call that last tool first and go backward. This is somewhat arbitrary and is as much a function of the way I implemented things as anything else. I know that there has been discussion of more complex ordering systems in future versions of QMPI and maybe for now what I have here is fine, but it would be good to think through whether this precludes those options in the future. Sorry for the long email. If you have any thoughts, feel free to share them and I can start considering those between now and next Thursday. Thanks, Wes