- mpiwg-tools - mailman3.cels.anl.gov

PMPI and sessions init
by Joachim Protze 05 Nov '20

05 Nov '20

Hi all, if I understood the discussion about MPI_Session_init and PMPI right, I don't think there exists an issue. By ld-preloading the PMPI tool, the tool can also intercept a call to MPI_Session_init, which might come from a library constructor before main. Even a function called from the library constructor of a static library can be intercepted by a ld-preloaded tool. For libraries, which are explicitly loaded by the MPI runtime during initialization, this would also work. For others, it depends on the ordering of calling the library constructor (might work by chance :). This was one of my main motivation of having such an environmental variable. Best Joachim -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Meeting reminder: Nov 5, 2020 (today)
by Marc-André Hermanns 05 Nov '20

05 Nov '20

Hi all, This is a reminder of our regularly scheduled call today: Nov 5 at 8 am Pacific / 11 am Eastern / *5pm* CEST. Webex Info: https://github.com/mpiwg-tools/mpi-standard/wiki *NOTE*: The WebEx information moved to a non-public wiki. You have to be logged into Github *and* have to have access to the private Tools WG repositories. If you need access contact me. Agenda: ------- - QMPI Cheers, Marc-Andre -- Dr. rer. nat. Marc-André Hermanns IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 52074 Aachen Phone: +49 241 80-24381 hermanns(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpiwg-tools] Notes on first-party unwinding
by Wesley Bland 30 Oct '20

30 Oct '20

Thanks Bill. We’ll take a look at this for our implementation. It does seem like this is going to be the best strategy, though these sorts of implementation details probably won’t make much difference for the specification side of things. Thanks, Wes > On Oct 29, 2020, at 11:40 AM, William Williams via mpiwg-tools <mpiwg-tools(a)lists.mpi-forum.org> wrote: > > Hi all, > > Per our discussion at the end of the call today, I wanted to get a few things written down while I’m still thinking about them, based on my past experience with stack walking: > > * Writing the helper macro for appropriate inline assembly to get a frame pointer is trivial per-platform but you do need to force the frame pointer to exist. GNU even provides intrinsics to get RA/FP portably: https://gcc.gnu.org/onlinedocs/gcc/Return-Address.html <https://gcc.gnu.org/onlinedocs/gcc/Return-Address.html>. Though that’s not something I’ve worked with personally and I don’t know how portable “portable” is there. > * As Joachim pointed out, name lookup/matching is what makes things at all expensive. It’s generally worth a fair bit of precomputation and hashing so that we can make fast decisions based on the return address alone. > * If we can make the simplifying assumption that a single tool may not use the PMPI and QMPI interfaces, we can probably use library-sized address ranges to decide how to classify a return address—which tool owns it and thereby how it should appear/not appear on the stack. (Well, subject to the further assumption that a tool library contains MPI wrapper functions and things called by those wrapper functions. Is that something we feel we can require of tools?) > > Cheers, > Bill > > -- > Bill Williams > Wiss. Mitarbeiter > ZIH-IAK, TU-Dresden > Falkenbrunnen, Room 009 > Chemnitzer Straße 50 > 01187 Dresden > Phone: +49 351 463-39976 > e-mail: william.williams(a)mailbox.tu-dresden.de <mailto:[email protected]> > > > > > > > _______________________________________________ > mpiwg-tools mailing list > mpiwg-tools(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-tools

1 0

Notes on first-party unwinding
by William Williams 29 Oct '20

29 Oct '20

Hi all, Per our discussion at the end of the call today, I wanted to get a few things written down while I’m still thinking about them, based on my past experience with stack walking: * Writing the helper macro for appropriate inline assembly to get a frame pointer is trivial per-platform but you do need to force the frame pointer to exist. GNU even provides intrinsics to get RA/FP portably: https://gcc.gnu.org/onlinedocs/gcc/Return-Address.html <https://gcc.gnu.org/onlinedocs/gcc/Return-Address.html>. Though that’s not something I’ve worked with personally and I don’t know how portable “portable” is there. * As Joachim pointed out, name lookup/matching is what makes things at all expensive. It’s generally worth a fair bit of precomputation and hashing so that we can make fast decisions based on the return address alone. * If we can make the simplifying assumption that a single tool may not use the PMPI and QMPI interfaces, we can probably use library-sized address ranges to decide how to classify a return address—which tool owns it and thereby how it should appear/not appear on the stack. (Well, subject to the further assumption that a tool library contains MPI wrapper functions and things called by those wrapper functions. Is that something we feel we can require of tools?) Cheers, Bill -- Bill Williams Wiss. Mitarbeiter ZIH-IAK, TU-Dresden Falkenbrunnen, Room 009 Chemnitzer Straße 50 01187 Dresden Phone: +49 351 463-39976 e-mail: william.williams(a)mailbox.tu-dresden.de

1 0

Meeting reminder: Oct 29, 2020 (today) -- 1 hour earlier in Europe!
by Marc-André Hermanns 29 Oct '20

29 Oct '20

Hi all, This is a reminder of our regularly scheduled call today: Oct 29 at 8 am Pacific / 11 am Eastern / *4pm* CEST. Webex Info: https://github.com/mpiwg-tools/mpi-standard/wiki *NOTE*: The WebEx information moved to a non-public wiki. You have to be logged into Github *and* have to have access to the private Tools WG repositories. If you need access contact me. Agenda: ------- - QMPI Cheers, Marc-Andre -- Dr. rer. nat. Marc-André Hermanns IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 52074 Aachen Phone: +49 241 80-24381 hermanns(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpiwg-tools] Problem with API From 2020-10-15 Conversation
by Wesley Bland 19 Oct '20

19 Oct '20

I’m not sure such a tool makes a lot of sense given that we’re not strongly specifying the mechanism for determining tool order. If we want to do that, then having a mapping from environment variable to tool ID would make more sense. Also, even if you have tool ID, that’s probably not enough to go on since you’ll also need to know whether a particular tool intercepted a particular function. If you call the QMPI_Get_next_function(my_tool_id=1, …) and expect to get the function for tool_id=2, that doesn’t work if tool 2 didn’t register an interception for that function. You may get an interception function for tool 3, or none at all and you go all the way back to MPI. > On Oct 16, 2020, at 10:09 AM, Joachim Protze <protze(a)itc.rwth-aachen.de> wrote: > > Hi Wesley, Martin, > > I still don't see the need for splitting the function, or what this > would give us. > I attached the slides from yesterday for illustration. Slide 3 also > sketches, how I would emulate the PnMPI_Xy_NewStack function. > > Whoever implements the function (Supertool?) will need to know the > ordering of the tools and apply a mapping to the stacks. That's why I > propose a function like: > > QMPI_Get_tool_ids (int count, int * array_of_tool_ids); > > to query the tool IDs according to their ordering in the QMPI_TOOLS env > variable. The supertool can then build a map for the lookup of stacks: > {"stack1": 12, "stack2": 13} > A position map: > pos = {0:1, 45:2, 12:3, 3:4, 5:5, ...} > As well as a map for the end of the stacks: > stacklimit = {12: 6, 13: 9} > > > I also propose to have a predefined tool-ID (QMPI_MPI_ID) to allow for > querying the actual MPI function. > > > Best > Joachim > > > > Am 16.10.20 um 16:17 schrieb Wesley Bland via mpiwg-tools: >> Hi all, >> >> In yesterday’s call, we discussed separating the current QMPI_Get_function call (that takes the current tool_id and an enum value to get the function in the call stack) into two function calls. The first would determine the next tool in the stack and the second would get the desired function for that tool. So they would look something like this: >> >> QMPI_Get_next_tool(int tool_id, int num_next_tools, int *next_tool_ids); >> QMPI_Get_tool_fn(int tool_id, enum QMPI_Functions_enum function_enum, void (*function_ptr)(void), QMPI_Context *context); >> >> However, as I went to implement that today, I realized that the reason this is a single function today is that not all tools implement all functions. So if you have a tool stack that looks like this: >> >> A - B - C >> >> and tools A and C intercept MPI_SEND and tool B does not, when tool A asks for the next tool to implement MPI_SEND, it should get back the function for tool C. Unfortunately, if we don’t ask for a specific function, MPI doesn’t have a way to figure out whether it should return B or C. >> >> So there’s two options here. >> >> 1) Keep the two functions combined. This makes it clear in the current, linear case which tool comes next. However, it doesn’t move us toward the potential future of having a mechanism (internal or external) to make dynamic decisions about the next tool. >> >> 2) Add an enum value to the QMPI_Get_next_tool function so it is only searching for tools that implement the particular function. This does complicate the search functionality a little bit more, but not significantly. The main drawback I see here is just the confusion of passing that enum value into both functions. If we’re not providing this decision functionality in QMPI v1, it’s weird that we have this placeholder for it now. However, that’s already true with the array of next tools that will always be 1, so maybe it’s not a big deal. >> >> I would imagine that 2 is the best option here to support the PnMPI use case, but I’m happy to hear feedback. >> >> Thanks, >> Wes >> >> >> _______________________________________________ >> mpiwg-tools mailing list >> mpiwg-tools(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-tools >> > > > -- > Dipl.-Inf. Joachim Protze > > IT Center > Group: High Performance Computing > Division: Computational Science and Engineering > RWTH Aachen University > Seffenter Weg 23 > D 52074 Aachen (Germany) > Tel: +49 241 80- 24765 > Fax: +49 241 80-624765 > protze(a)itc.rwth-aachen.de > www.itc.rwth-aachen.de > <QMPI-stacks.pdf>

1 0

Re: [mpiwg-tools] Problem with API From 2020-10-15 Conversation
by Joachim Protze 16 Oct '20

16 Oct '20

Hi Wesley, Martin, I still don't see the need for splitting the function, or what this would give us. I attached the slides from yesterday for illustration. Slide 3 also sketches, how I would emulate the PnMPI_Xy_NewStack function. Whoever implements the function (Supertool?) will need to know the ordering of the tools and apply a mapping to the stacks. That's why I propose a function like: QMPI_Get_tool_ids (int count, int * array_of_tool_ids); to query the tool IDs according to their ordering in the QMPI_TOOLS env variable. The supertool can then build a map for the lookup of stacks: {"stack1": 12, "stack2": 13} A position map: pos = {0:1, 45:2, 12:3, 3:4, 5:5, ...} As well as a map for the end of the stacks: stacklimit = {12: 6, 13: 9} I also propose to have a predefined tool-ID (QMPI_MPI_ID) to allow for querying the actual MPI function. Best Joachim Am 16.10.20 um 16:17 schrieb Wesley Bland via mpiwg-tools: > Hi all, > > In yesterday’s call, we discussed separating the current QMPI_Get_function call (that takes the current tool_id and an enum value to get the function in the call stack) into two function calls. The first would determine the next tool in the stack and the second would get the desired function for that tool. So they would look something like this: > > QMPI_Get_next_tool(int tool_id, int num_next_tools, int *next_tool_ids); > QMPI_Get_tool_fn(int tool_id, enum QMPI_Functions_enum function_enum, void (*function_ptr)(void), QMPI_Context *context); > > However, as I went to implement that today, I realized that the reason this is a single function today is that not all tools implement all functions. So if you have a tool stack that looks like this: > > A - B - C > > and tools A and C intercept MPI_SEND and tool B does not, when tool A asks for the next tool to implement MPI_SEND, it should get back the function for tool C. Unfortunately, if we don’t ask for a specific function, MPI doesn’t have a way to figure out whether it should return B or C. > > So there’s two options here. > > 1) Keep the two functions combined. This makes it clear in the current, linear case which tool comes next. However, it doesn’t move us toward the potential future of having a mechanism (internal or external) to make dynamic decisions about the next tool. > > 2) Add an enum value to the QMPI_Get_next_tool function so it is only searching for tools that implement the particular function. This does complicate the search functionality a little bit more, but not significantly. The main drawback I see here is just the confusion of passing that enum value into both functions. If we’re not providing this decision functionality in QMPI v1, it’s weird that we have this placeholder for it now. However, that’s already true with the array of next tools that will always be 1, so maybe it’s not a big deal. > > I would imagine that 2 is the best option here to support the PnMPI use case, but I’m happy to hear feedback. > > Thanks, > Wes > > > _______________________________________________ > mpiwg-tools mailing list > mpiwg-tools(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-tools > -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Problem with API From 2020-10-15 Conversation
by Wesley Bland 16 Oct '20

16 Oct '20

Hi all, In yesterday’s call, we discussed separating the current QMPI_Get_function call (that takes the current tool_id and an enum value to get the function in the call stack) into two function calls. The first would determine the next tool in the stack and the second would get the desired function for that tool. So they would look something like this: QMPI_Get_next_tool(int tool_id, int num_next_tools, int *next_tool_ids); QMPI_Get_tool_fn(int tool_id, enum QMPI_Functions_enum function_enum, void (*function_ptr)(void), QMPI_Context *context); However, as I went to implement that today, I realized that the reason this is a single function today is that not all tools implement all functions. So if you have a tool stack that looks like this: A - B - C and tools A and C intercept MPI_SEND and tool B does not, when tool A asks for the next tool to implement MPI_SEND, it should get back the function for tool C. Unfortunately, if we don’t ask for a specific function, MPI doesn’t have a way to figure out whether it should return B or C. So there’s two options here. 1) Keep the two functions combined. This makes it clear in the current, linear case which tool comes next. However, it doesn’t move us toward the potential future of having a mechanism (internal or external) to make dynamic decisions about the next tool. 2) Add an enum value to the QMPI_Get_next_tool function so it is only searching for tools that implement the particular function. This does complicate the search functionality a little bit more, but not significantly. The main drawback I see here is just the confusion of passing that enum value into both functions. If we’re not providing this decision functionality in QMPI v1, it’s weird that we have this placeholder for it now. However, that’s already true with the array of next tools that will always be 1, so maybe it’s not a big deal. I would imagine that 2 is the best option here to support the PnMPI use case, but I’m happy to hear feedback. Thanks, Wes

1 0

Re: [mpiwg-tools] QMPI API (After 2020-10-08 Discussion)
by Wesley Bland 13 Oct '20

13 Oct '20

> On Oct 13, 2020, at 4:19 AM, Joachim Protze <protze(a)itc.rwth-aachen.de> wrote: > > Hi Wesley, > > most of the functions are as expected, just the last part makes me > wonder how this should work. > > Am 08.10.20 um 22:54 schrieb Wesley Bland via mpiwg-tools: >> >> int QMPI_Register_tool_name(char * tool_name, void (* >> init_function_ptr)(int tool_id)); >> >> tool_name - [IN] Character string - A unique string representing the >> name of the tool >> init_function_ptr - [IN] Function pointer - Pointer to a function that >> MPI will called before MPI is initialized >> > > Since you start operating with function pointers, I'm wondering whether > all the functions below should also be function pointers (with some > lookup mechanism) rather than public interface functions. I’m not sure what the benefit of that would be and there are significant performance drawbacks to taking functions that could otherwise be inlined and forcing them to be function pointers. For some of these functions, it may not make a lot of difference, but for those in the critical path, it makes a huge difference. > > Also, is the tool always guaranteed to find the QMPI_Register_tool_name > function? I don't remember the name, so I didn't find documentation, but > I think that some linkers support encapsulated name spaces for dynamic > libraries. This is a function that would be provided by MPI, like any other function, so I’m not sure what problem you’re describing. > > >> tool_storage - [IN] Address - A pointer to an address where the tool has >> data being stored for its particular instance >> >> This function allows the tool to provide a storage object (tool_storage) >> which lets the tool store information specific to its instance of itself >> (in case there are multiple copies of a tool). The specific instance of >> the tool is identified with the tool_id value and should be matched with >> the one returned by the callback function that the tool registered with MPI. > > From my perspective, this structure holds all state which a particular > instance of a tool has. I agree as long as what you mean is any state that the tool manages. None of the MPI-managed state goes in here. This is what we were calling the context object up to now, but I’m proposing that we call it “storage" and call the thing that contains it “context". > >> *Interception* >> >> When the tool's interception function is called, it can get any >> tool-specific information from the context object. The fields in the >> context object can be retrieved with these functions: >> >> QMPI_Context_get_storage(QMPI_Context context, int tool_id, void **storage); >> >> context - [IN] Opaque handle - A handle to a context object containing >> tool and function-specific information >> tool_id - [IN] Integer - A unique identifier for the tool >> storage - [OUT] Address - A pointer to an address where the tool has >> data being stored for its particular instance >> >> Get the back storage object the tool provided >> to QMPI_Register_tool_storage that matches tool_id. > > Where does this tool_id come from? Is it passed in as an argument to the > interception function? The tool does not know the id. Good point. I forgot to give an example of what the interception functions look like. I believe the intention after the call last week was to have two additional fields at the beginning of every MPI function. So, as an example, an MPI_SEND interception function would look like this: MPI_Send_example(QMPI_Context context, int tool_id, const void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm); The reason the tool_id is included in the argument list and not embedded in the context object is because the context object does not need to change from tool to tool on the way down the stack. It can contain an array of all storage objects, indexed by the tool_id. The context object only changes if a tool calls a function other than the one it is already in (e.g., an MPI_Allreduce calls MPI_Send/MPI_Recv). > >> >> *Calling Additional QMPI Tools* >> >> When the tool is done with its own interception, it should call this >> function to determine the next tool in the QMPI stack (which the tool >> itself is responsible for calling): >> >> int QMPI_Get_function(int tool_id, enum QMPI_Functions_enum >> function_enum, void (** function_ptr)(void), QMPI_Context * >> next_tool_context); >> >> tool_id - [IN] Integer - A unique identifier for the tool >> function_enum - [IN] Enum value - An enumerated value specifying which >> MPI function is being registered for interception. >> function_ptr - [OUT] Function pointer - A pointer to a function that >> should be called when an application calls the specified function. >> next_tool_context - [OUT] Opaque handle - A handle to a context object >> containing tool and function-specific information >> >> This function also uses tool_id and function_enum to determine the >> current tool's ID and the enum value for the function being queried. It >> also provides a function pointer in a similar format to the registration >> function. Finally, the function returns the context object of the next >> tool in the call stack which should be used when calling the returned >> function. > > Why does this function not need the current QMPI_Context context? > I would store a pointer to the function-table in this context and > implement QMPI_Get_function as a macro accessing this table. That is one way to do it that would need the current context object. In my implementation, I’ve been keeping that function table in MPI's memory so there was no need to put it in the context object. The function table doesn’t change from function to function so I didn’t see a reason to make multiple copies of it. > I assume the storage for next_tool_context is stack memory from the > caller (the tool)? Yes, I think that makes the most sense. So essentially, each tool holds the context object for the next tool in its own stack memory until it comes back up the stack and the memory is released. The alternative is for MPI to be responsible for allocating and garbage collecting these objects. > I would assume, that his function also provides the next tool id, to > pass it with the function call. You’re right. I forgot to include that in this function signature. I’ll add that to my documentation. > > Alternatively, the tool ID could be a field in next_tool_context. Then > we need a function to query the tool ID from the context. Also, > QMPI_Context_get_storage does not need the tool_id as input in this case. I’m not sure this works for the reason I described above with the MPI_Send_example function. Thanks, Wes > > > Best > Joachim > > > > -- > Dipl.-Inf. Joachim Protze > > IT Center > Group: High Performance Computing > Division: Computational Science and Engineering > RWTH Aachen University > Seffenter Weg 23 > D 52074 Aachen (Germany) > Tel: +49 241 80- 24765 > Fax: +49 241 80-624765 > protze(a)itc.rwth-aachen.de > www.itc.rwth-aachen.de >

1 0

Re: [mpiwg-tools] QMPI API (After 2020-10-08 Discussion)
by Joachim Protze 13 Oct '20

13 Oct '20

Hi Wesley, most of the functions are as expected, just the last part makes me wonder how this should work. Am 08.10.20 um 22:54 schrieb Wesley Bland via mpiwg-tools: > > int QMPI_Register_tool_name(char * tool_name, void (* > init_function_ptr)(int tool_id)); > > tool_name - [IN] Character string - A unique string representing the > name of the tool > init_function_ptr - [IN] Function pointer - Pointer to a function that > MPI will called before MPI is initialized > Since you start operating with function pointers, I'm wondering whether all the functions below should also be function pointers (with some lookup mechanism) rather than public interface functions. Also, is the tool always guaranteed to find the QMPI_Register_tool_name function? I don't remember the name, so I didn't find documentation, but I think that some linkers support encapsulated name spaces for dynamic libraries. > tool_storage - [IN] Address - A pointer to an address where the tool has > data being stored for its particular instance > > This function allows the tool to provide a storage object (tool_storage) > which lets the tool store information specific to its instance of itself > (in case there are multiple copies of a tool). The specific instance of > the tool is identified with the tool_id value and should be matched with > the one returned by the callback function that the tool registered with MPI. From my perspective, this structure holds all state which a particular instance of a tool has. > *Interception* > > When the tool's interception function is called, it can get any > tool-specific information from the context object. The fields in the > context object can be retrieved with these functions: > > QMPI_Context_get_storage(QMPI_Context context, int tool_id, void **storage); > > context - [IN] Opaque handle - A handle to a context object containing > tool and function-specific information > tool_id - [IN] Integer - A unique identifier for the tool > storage - [OUT] Address - A pointer to an address where the tool has > data being stored for its particular instance > > Get the back storage object the tool provided > to QMPI_Register_tool_storage that matches tool_id. Where does this tool_id come from? Is it passed in as an argument to the interception function? The tool does not know the id. > > *Calling Additional QMPI Tools* > > When the tool is done with its own interception, it should call this > function to determine the next tool in the QMPI stack (which the tool > itself is responsible for calling): > > int QMPI_Get_function(int tool_id, enum QMPI_Functions_enum > function_enum, void (** function_ptr)(void), QMPI_Context * > next_tool_context); > > tool_id - [IN] Integer - A unique identifier for the tool > function_enum - [IN] Enum value - An enumerated value specifying which > MPI function is being registered for interception. > function_ptr - [OUT] Function pointer - A pointer to a function that > should be called when an application calls the specified function. > next_tool_context - [OUT] Opaque handle - A handle to a context object > containing tool and function-specific information > > This function also uses tool_id and function_enum to determine the > current tool's ID and the enum value for the function being queried. It > also provides a function pointer in a similar format to the registration > function. Finally, the function returns the context object of the next > tool in the call stack which should be used when calling the returned > function. Why does this function not need the current QMPI_Context context? I would store a pointer to the function-table in this context and implement QMPI_Get_function as a macro accessing this table. I assume the storage for next_tool_context is stack memory from the caller (the tool)? I would assume, that his function also provides the next tool id, to pass it with the function call. Alternatively, the tool ID could be a field in next_tool_context. Then we need a function to query the tool ID from the context. Also, QMPI_Context_get_storage does not need the tool_id as input in this case. Best Joachim -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0