- mpiwg-tools - mailman3.cels.anl.gov

QMPI API (After 2020-10-08 Discussion)
by Wesley Bland 08 Oct '20

08 Oct '20

Hi all, I took another pass at updating the QMPI API based on our discussion in the working group today. I wasn’t going to post this on the pull request yet since I haven’t yet updated the implementation, but I wanted to start circulating it to get additional feedback. Please let me know if you think I misunderstood something or you think something should change. Thanks, Wes QMPI API Tool Loading The tool should use a system specific option to initialize the library when it's loaded. GCC and other compiler use the __attribute__((constructor) trick to accomplish this. The same can be used during finalization (if simple interception of MPI_FINALIZE is insufficient. Registration When the tool is loaded, it needs to be registered with the MPI library. This happens with this function: int QMPI_Register_tool_name(char * tool_name, void (* init_function_ptr)(int tool_id)); tool_name - [IN] Character string - A unique string representing the name of the tool init_function_ptr - [IN] Function pointer - Pointer to a function that MPI will called before MPI is initialized This function allows a tool to “announce” itself to MPICH, but does not insert a tool into the QMPI call stack. The tool_name provided needs to match the one provided in the environment variable describing the tool order (described later). At some point before MPI is initialized, it will call all of the callback functions provided via init_function_ptr. These functions may be called at any point between the calling of QMPI_Register_tool and executing MPI_Init. For MPICH, this will probably be implemented in the wrapper code that gets executed between when an application calls MPI_Init and when the QMPI tools that intercept MPI_Init are called. The callback function pointer looks like this: void init_function_ptr(int tool_id); tool_id - [OUT] Integer - A unique identifier for the tool This callback function is called one time for each instance of the tool that was requested via the environment variable specifying the order and quantity of tools. Each instance of the tool receives a unique tool_id. These tool IDs are not necessarily monotonically increasing and cannot be assumed to do so. Inside this callback function, the tool should register any tool-specific storage it will to use as well as any MPI functions it plans to intercept. This happens through the following two functions. int QMPI_Register_tool_storage(int tool_id, void *tool_storage); tool_id - [IN] Integer - A unique identifier for the tool tool_storage - [IN] Address - A pointer to an address where the tool has data being stored for its particular instance This function allows the tool to provide a storage object (tool_storage) which lets the tool store information specific to its instance of itself (in case there are multiple copies of a tool). The specific instance of the tool is identified with the tool_id value and should be matched with the one returned by the callback function that the tool registered with MPI. int QMPI_Register_tool_function(int tool_id, enum QMPI_Functions_enum function_enum, void (* function_ptr)(void)) tool_id - [IN] Integer - A unique identifier for the tool function_enum - [IN] Enum value - An enumerated value specifying which MPI function is being registered for interception. function_ptr - [IN] Function pointer - A pointer to a function that should be called when an application calls the specified function. This function accepts the parameter tool_id as the specific instance of a tool and should be matched with the one returned by the callback function that the tool registered with MPI. function_enum is used to specify which MPI function is being registered. The enum values are defined in the enum QMPI_Functions_enum in mpi.h and generally are of this form: enum QMPI_Functions_enum { MPI_ABORT_T, MPI_ACCUMULATE_T, ... snip ... MPI_WTIME_T, MPI_LAST_FUNC }; function_ptr should be a pointer to the tool's interception function pointer. This function pointer will be the same signature as the MPI function, with an additional QMPI_Context argument in the first position as a way for tools to query information about the specific MPI procedure being intercepted when the callback functions are called. The type signature for each MPI function is defined in mpi.h in the form of QMPI_<function_name>_t (e.g., QMPI_Send_t(QMPI_Context context, const void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm)). Interception When the tool's interception function is called, it can get any tool-specific information from the context object. The fields in the context object can be retrieved with these functions: QMPI_Context_get_storage(QMPI_Context context, int tool_id, void **storage); context - [IN] Opaque handle - A handle to a context object containing tool and function-specific information tool_id - [IN] Integer - A unique identifier for the tool storage - [OUT] Address - A pointer to an address where the tool has data being stored for its particular instance Get the back storage object the tool provided to QMPI_Register_tool_storage that matches tool_id. QMPI_Context_get_calling_address(QMPI_Context context, int tool_id, void *calling_address); context - [IN] Opaque handle - A handle to a context object containing tool and function-specific information tool_id - [IN] Integer - A unique identifier for the tool calling_address - [OUT] Address - The address of the application the called the MPI function. The the address of the point where the application originally called the MPI function before entering the QMPI tool stack. If both a PMPI and QMPI tool are registered and active at the same time, the address will be that of the PMPI function, rather than the application. Calling Additional QMPI Tools When the tool is done with its own interception, it should call this function to determine the next tool in the QMPI stack (which the tool itself is responsible for calling): int QMPI_Get_function(int tool_id, enum QMPI_Functions_enum function_enum, void (** function_ptr)(void), QMPI_Context * next_tool_context); tool_id - [IN] Integer - A unique identifier for the tool function_enum - [IN] Enum value - An enumerated value specifying which MPI function is being registered for interception. function_ptr - [OUT] Function pointer - A pointer to a function that should be called when an application calls the specified function. next_tool_context - [OUT] Opaque handle - A handle to a context object containing tool and function-specific information This function also uses tool_id and function_enum to determine the current tool's ID and the enum value for the function being queried. It also provides a function pointer in a similar format to the registration function. Finally, the function returns the context object of the next tool in the call stack which should be used when calling the returned function.

1 0

Meeting reminder: Oct 8, 2020 (today)
by Marc-André Hermanns 08 Oct '20

08 Oct '20

Hi all, This is a reminder of our regularly scheduled call today: Oct 8 at 8 am Pacific / 11 am Eastern / 5pm CEST. Webex Info: https://github.com/mpiwg-tools/mpi-standard/wiki *NOTE*: The WebEx information moved to a non-public wiki. You have to be logged into Github *and* have to have access to the private Tools WG repositories. If you need access contact me. Agenda: ------- - QMPI Cheers, Marc-Andre -- Dr. rer. nat. Marc-André Hermanns IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 52074 Aachen Phone: +49 241 80-24381 hermanns(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpiwg-tools] Meeting reminder: Oct 1, 2020 (today - in 15 minutes)
by Marc-André Hermanns 01 Oct '20

01 Oct '20

Hi all, the notes are up on the Wiki: https://github.com/mpiwg-tools/tools-issues/wiki/Notes-2020-10-01 As noted on the call, feel free to correct or expand as you see fit. Cheers, Marc-Andre On 01.10.20 16:49, Marc-André Hermanns via mpiwg-tools wrote: > Hi all, > > > This is a reminder of our regularly scheduled call today: > > Oct 1 at 8 am Pacific / 11 am Eastern / 5pm CEST. > > Webex Info: > https://github.com/mpiwg-tools/mpi-standard/wiki > > *NOTE*: The WebEx information moved to a non-public wiki. You have to > be logged into Github *and* have to have access to the private Tools > WG repositories. If you need access contact me. > > Agenda: > ------- > - QMPI > > Note: Due to MPI Forum and ongoing OpenMP LC meeting there was only > little progress on the design discussion and keep people will not be > able to join today. However, as there is a bit on a time crunch for > Intel's QMPI prototype regarding feature, I at least wanted to briefly > discuss some key open questions, so we could all take away some > homework for next week ;-) > > > Cheers, > Marc-Andre > > > _______________________________________________ > mpiwg-tools mailing list > mpiwg-tools(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-tools > -- Dr. rer. nat. Marc-André Hermanns IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 52074 Aachen Phone: +49 241 80-24381 hermanns(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Meeting reminder: Oct 1, 2020 (today - in 15 minutes)
by Marc-André Hermanns 01 Oct '20

01 Oct '20

Hi all, This is a reminder of our regularly scheduled call today: Oct 1 at 8 am Pacific / 11 am Eastern / 5pm CEST. Webex Info: https://github.com/mpiwg-tools/mpi-standard/wiki *NOTE*: The WebEx information moved to a non-public wiki. You have to be logged into Github *and* have to have access to the private Tools WG repositories. If you need access contact me. Agenda: ------- - QMPI Note: Due to MPI Forum and ongoing OpenMP LC meeting there was only little progress on the design discussion and keep people will not be able to join today. However, as there is a bit on a time crunch for Intel's QMPI prototype regarding feature, I at least wanted to briefly discuss some key open questions, so we could all take away some homework for next week ;-) Cheers, Marc-Andre -- Dr. rer. nat. Marc-André Hermanns IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 52074 Aachen Phone: +49 241 80-24381 hermanns(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

No call today
by Marc-André Hermanns 24 Sep '20

24 Sep '20

Dear all, I have a conflict today, and as mentioned last week the EuroMPI Workshop Extreme-scale Scientific Software Stack Forum takes place as well, where Martin has a talk that at least partly overlaps with our call, I would cancel for today. For anyone wanting to join on short notice follow the registration link at: https://eurompi.github.io/program.html There you'll get a link to a zoom meeting. Next week is the MPI Forum virtual meeting. For anyone wanting to join there, check the logistics page at https://meetings.mpi-forum.org/ Cheers, Marc-Andre -- Dr. rer. nat. Marc-André Hermanns IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 52074 Aachen Phone: +49 241 80-24381 hermanns(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Meeting reminder: Sep 17, 2020 (today)
by Marc-André Hermanns 17 Sep '20

17 Sep '20

Hi all, This is a reminder of our regularly scheduled call tomorrow: Sep 17 at 8 am Pacific / 11 am Eastern / 5pm CEST. Webex Info: https://github.com/mpiwg-tools/mpi-standard/wiki *NOTE*: The WebEx information moved to a non-public wiki. You have to be logged into Github *and* have to have access to the private Tools WG repositories. If you need access contact me. Agenda: ------- - QMPI Cheers, Marc-Andre -- Dr. rer. nat. Marc-André Hermanns IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 52074 Aachen Phone: +49 241 80-24381 hermanns(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Can't Attend Tomorrow's Meeting
by Wesley Bland 16 Sep '20

16 Sep '20

Hi all, I won’t be able to attend tomorrow’s Tools WG meeting due to a conflict, but feel free to discuss QMPI without me. Would it be possible for someone to hit the record button so I can watch the discussion afterward? Last time, there wasn’t much discussion on the pieces of the API I’d implemented apart from the ongoing discussion of specifying ordering of tools. Given that we won’t be meeting for the next few weeks because of EuroMPI, I’d be happy to hear any feedback that anyone has on the API as it currently stands. Thanks, Wes

1 0

No call today: Sep 10, 2020
by Marc-André Hermanns 10 Sep '20

10 Sep '20

Dear all, I am not feeling well today and won't make the call. I would therefore propose to skip the call today and continue next week. Cheers, Marc-Andre -- Dr. rer. nat. Marc-André Hermanns IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 52074 Aachen Phone: +49 241 80-24381 hermanns(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Meeting reminder: Sep 03, 2020 (today)
by Marc-André Hermanns 03 Sep '20

03 Sep '20

Hi all, This is a reminder of our regularly scheduled call tomorrow: Sep 03 at 8 am Pacific / 11 am Eastern / 5pm CEST. Webex Info: https://github.com/mpiwg-tools/mpi-standard/wiki *NOTE*: The WebEx information moved to a non-public wiki. You have to be logged into Github *and* have to have access to the private Tools WG repositories. If you need access contact me. Agenda: ------- - QMPI Cheers, Marc-Andre -- Dr. rer. nat. Marc-André Hermanns IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 52074 Aachen Phone: +49 241 80-24381 hermanns(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

2020-08-27 Update for QMPI Implementation
by Wesley Bland 27 Aug '20

27 Aug '20

Hi all, I know QMPI wasn’t on the agenda for todays WG call, but I did finish a major update to the prototype with most of the things we discussed at the last two calls. I wanted to get that in front of people with my latest thoughts to try to make progress in the meantime. You can see everything on the pull request here: https://github.com/pmodels/mpich/pull/4715 <https://github.com/pmodels/mpich/pull/4715> And the sample tool here: https://gist.github.com/wesbland/065de74b3582913c3945099af5cf5130 <https://gist.github.com/wesbland/065de74b3582913c3945099af5cf5130> I’ll pull out some of the relevant bits. First, here’s what’s changed since last time we spoke: Correct the use of function pointers to use void(*)(void) instead of void *. Add a tool-defined context object that is provided to MPI on registration. Use a system-specific way of registering the tool rather than using dlopen and dlsym (e.g. using _attribute__((constructor))). Create an MPI-level function to register each tool function rather that using a struct of function pointers. Create an MPI-level function to get the next function in the stack rather than querying the array of structs of function pointers. Create an MPI-exposed enum to let tools specify which MPI function they are referencing when registering or querying a function. Create MPI-exposed typedefs for each function to allow the tool to use them if desired. Basically the entire API needed to change to reflect the changes we discussed, specifically handling a void * context for the tool and not using a “magic” registration function. I’ve documented the current version of the API on the pull request. Here are some of the issues that I noted as I was putting this together: 1. I’m not using an enum to determine where to store a particular pointer and to retrieve it quickly (as opposed to string parsing or if statements). One of my concerns is whether using this enum will cause similar problems to forward compatibility that we discussed with the struct of function pointers. For example, if the tool is compiled with MPI 5 QMPI and MPI_LAST_FUNC is 400 and is run against MPI 5.1 QMPI where MPI_LAST_FUNC is 410, that’s an issue. We need to be specific about saying that the user should never use MPI_LAST_FUNC and MPI libraries should separate the MPI_ functions and the MPIX_X functions that they also expose to avoid future collisions. 2. I’ve added typedefs for each MPI function in mpi.h primarily as a convenience, but that may or may not be strictly necessary. If we end up with an API like this, we will want to consider whether that’s required. 3. I’m using the __attribute__((constructor)) method of having libraries bootstrap themselves (which is more portable than I realized). There are other options for compilers like MSVC, but I don’t know if there’s a solution for *everyone* that we can point to to avoid having to standardize a registration function. One option is to create an *optional* function that could be loaded up if it were needed. 4. In addition to figuring out how to bootstrap libraries, it’s also not clear how to specify the ordering of loading libraries (and how to call them). At the moment, I’m using essentially a stack (MPI is 0, first tool is 1, etc.) and when starting the call stack, I call that last tool first and go backward. This is somewhat arbitrary and is as much a function of the way I implemented things as anything else. I know that there has been discussion of more complex ordering systems in future versions of QMPI and maybe for now what I have here is fine, but it would be good to think through whether this precludes those options in the future. Sorry for the long email. If you have any thoughts, feel free to share them and I can start considering those between now and next Thursday. Thanks, Wes

1 0