January 2021 - mpiwg-tools - mailman3.cels.anl.gov

Meeting reminder: Jan 28, 2020 (tomorrow)
by Marc-André Hermanns 27 Jan '21

27 Jan '21

Hi all, This is a reminder of our regularly scheduled call tomorrow: Jan 28 at 8 am Pacific / 11 am Eastern / 5pm CEST. Meeting Details: https://github.com/mpiwg-tools/mpi-standard/wiki/Meeting-Information *NOTE*: I have updated the meeting details and switched to *ZOOM* for the meeting platform. Let me know if something does not work out for you. Agenda: ------- - Issue #433 [1] and how to resolve it Cheers, Marc-Andre -- Dr. rer. nat. Marc-André Hermanns IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 52074 Aachen Phone: +49 241 80-24381 hermanns(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpiwg-tools] question about MPI_T_EVENT_SET_DROPPED_HANDLER
by Raffenetti, Kenneth J. 26 Jan '21

26 Jan '21

To me, option 1 makes the most sense from both an implementation and user perspective. Explicit and unsurprising. But yes, we may be constrained on time. Worst, IMO, is option 3, which on the user side would require some complex logic to even know what user_data pointer the dropped handler is providing. Options 2 and 4 are somewhere in the middle. Not ideal, but might be an effective hedge. Ken On 1/26/21, 11:29 AM, "Marc-André Hermanns" <hermanns(a)itc.rwth-aachen.de> wrote: There could be different solutions for this situation, and each of them has a problem in our current situation: 1) Add a "user_data" element to the "dropped_handler" registration call. 2) If the MPI implementation does not support a certain level at all, have the callback registration function return MPI_T_ERR_NOT_SUPPORTED and refuse the registration in this case. 3) Use the 'user_data' registered with a callback at a safety level closest to the one where the dropped handler is invoked with. 4) Just accept that the dropped handler will never be called in this case. Option 1 would be in symmetry with the "free" handler registration, but would need an API change and a no-no vote to be included in 4.0. Is that likely to happen? Option 2 would be possible, yet MPI_T_ERR_NOT_SUPPORTED is not yet listed in Table 15.7 on page 778 in the correct row (it exists, but is listed only for MPI_T_SOURCE_*) ... We might handle this as an oversight and clarify it basically with the release of MPI 4.0 and include it in 4.1. Option 3 feels odd, as the we assume something about the characteristics of 'user_data' for a safety level that might not be the case and might not be safe. Option 4 would be possible, yet lacks the feedback to the user that something went wrong. What's your opinion? Are there other/better ways to deal with the situation?

1 0

Re: [mpiwg-tools] question about MPI_T_EVENT_SET_DROPPED_HANDLER
by Marc-André Hermanns 26 Jan '21

26 Jan '21

Hi Ken, > Oh, so if I am reading your clarification correctly above, you are saying I should not invoke the dropped handler until a callback for that safety level has been registered. It is possible then that I might never call the dropped handler. Should I be managing dropped event counts for all safety levels? > > Let me see if I can give a more concrete example. Let's say I have an event registration handle allocated, and a callback registered for the lowest safety level (NONE). I also have a dropped handler registered. The event itself is generating instances for higher safety levels (MPI_RESTRICTED, THREAD_SAFE), but because there are no handlers registered for those levels, the events are being dropped. > > Question 1: Should the implementation be recording those dropped events for higher safety levels? The text leads me to believe yes. > Question 2: Should the implementation delay calling the dropped handler until a callback satisfying the safety level of the dropped events has been registered? Your clarification leads me to believe yes. > Question 3: If I register a MPI_RESTRICTED callback, will the dropped handler then execute to inform me of all the dropped events at that safety level? > Question 4: If I instead register a THREAD_SAFE callback, will the dropped handler execute to inform me of all the dropped events at that safety level **as well as MPI_RESTRICTED**? Let me see whether I can answer you questions, as some are indeed unclear with the current text/defined behavior. 1) The implementation should (where possible) record whenever an event could not be recorded and had to be dropped. 2) The implementation does not have to keep separate counters for each level. 3) The cb_safety argument of the handler is to define "what is possible for the tool inside this handler at the time of invocation" and does not imply the safety level events were dropped. Specifically, the implementation does not have to invoke the handler with the same safety level as where the event was dropped but (in the words of Capt. Jack Sparrow) "wait for the opportune moment" to invoke the handle with the highest leeway for the tool manageable. As higher level callbacks can fill in for lower levels (i.e., ASYNC_SIGNAL_SAFE can be invoked with level THREAD_SAFE) the implementation can always safely use the one with the lowest level still higher or equal to the level used when invoking the dropped handler. If an event is registered for a level that the implementation supports in general, there should be states in the execution where the dropped handler can be invoked with the matching safety level, and the implementation can delay until then. I hope this already clarifies most of the questions, yet you pointed out a severe under-specification that we need to fix: ---\/---\/---\/--- here be dragons ---\/---\/---\/---\/--- What happens when a user only registers a callback for NONE, and the library does not support these in the first place? Then, the library will never get to a place where it can issue the callback handler with a user provided user_data pointer. There could be different solutions for this situation, and each of them has a problem in our current situation: 1) Add a "user_data" element to the "dropped_handler" registration call. 2) If the MPI implementation does not support a certain level at all, have the callback registration function return MPI_T_ERR_NOT_SUPPORTED and refuse the registration in this case. 3) Use the 'user_data' registered with a callback at a safety level closest to the one where the dropped handler is invoked with. 4) Just accept that the dropped handler will never be called in this case. Option 1 would be in symmetry with the "free" handler registration, but would need an API change and a no-no vote to be included in 4.0. Is that likely to happen? Option 2 would be possible, yet MPI_T_ERR_NOT_SUPPORTED is not yet listed in Table 15.7 on page 778 in the correct row (it exists, but is listed only for MPI_T_SOURCE_*) ... We might handle this as an oversight and clarify it basically with the release of MPI 4.0 and include it in 4.1. Option 3 feels odd, as the we assume something about the characteristics of 'user_data' for a safety level that might not be the case and might not be safe. Option 4 would be possible, yet lacks the feedback to the user that something went wrong. What's your opinion? Are there other/better ways to deal with the situation? Are there other opinions/ideas in the group on how to resolve this? Cheers, Marc-Andre -- Dr. rer. nat. Marc-André Hermanns IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 52074 Aachen Phone: +49 241 80-24381 hermanns(a)itc.rwth-aachen.de www.itc.rwth-aachen.de Social Media Kanäle des IT Centers: https://blog.rwth-aachen.de/itc/ https://www.facebook.com/itcenterrwth https://www.linkedin.com/company/itcenterrwth https://twitter.com/ITCenterRWTH https://www.youtube.com/channel/UCKKDJJukeRwO0LP-ac8x8rQ

1 0

Re: [mpiwg-tools] question about MPI_T_EVENT_SET_DROPPED_HANDLER
by Raffenetti, Kenneth J. 26 Jan '21

26 Jan '21

On 1/25/21, 3:07 PM, "Raffenetti, Kenneth J." <raffenet(a)mcs.anl.gov> wrote: Oh, so if I am reading your clarification correctly above, you are saying I should not invoke the dropped handler until a callback for that safety level has been registered. It is possible then that I might never call the dropped handler. Should I be managing dropped event counts for all safety levels? Let me see if I can give a more concrete example. Let's say I have an event registration handle allocated, and a callback registered for the lowest safety level (NONE). I also have a dropped handler registered. The event itself is generating instances for higher safety levels (MPI_RESTRICTED, THREAD_SAFE), but because there are no handlers registered for those levels, the events are being dropped. Question 1: Should the implementation be recording those dropped events for higher safety levels? The text leads me to believe yes. Question 2: Should the implementation delay calling the dropped handler until a callback satisfying the safety level of the dropped events has been registered? Your clarification leads me to believe yes. Question 3: If I register a MPI_RESTRICTED callback, will the dropped handler then execute to inform me of all the dropped events at that safety level? Question 4: If I instead register a THREAD_SAFE callback, will the dropped handler execute to inform me of all the dropped events at that safety level **as well as MPI_RESTRICTED**? Ken

1 0

Re: [mpiwg-tools] question about MPI_T_EVENT_SET_DROPPED_HANDLER
by Raffenetti, Kenneth J. 25 Jan '21

25 Jan '21

On 1/25/21, 2:49 PM, "mpiwg-tools on behalf of Raffenetti, Kenneth J. via mpiwg-tools" <mpiwg-tools-bounces(a)lists.mpi-forum.org on behalf of mpiwg-tools(a)lists.mpi-forum.org> wrote: On 1/25/21, 12:44 PM, "mpiwg-tools on behalf of Raffenetti, Kenneth J. via mpiwg-tools" <mpiwg-tools-bounces(a)lists.mpi-forum.org on behalf of mpiwg-tools(a)lists.mpi-forum.org> wrote: On 1/25/21, 11:46 AM, "Marc-André Hermanns" <hermanns(a)itc.rwth-aachen.de> wrote: For me (although not explicitly defined in the text) that would imply that the dropped callback is only invoked once the library is in a state where a callback (and user_data) for the same level is defined and could be invoked. Does this make sense and clarify your concerns? Do you think this need more clarification in the text? I see. Yes, that makes sense for my implementation issue. I may have spoken too soon. I'm looking at pg 763, ln 37 (Handling Dropped Events): If an event occurs...[and] no callback function meeting the required callback safety level is registered, the event data may be dropped. To discover such data loss, the user can set a handler function for a specific event-registration handle. If a callback function meeting the required safety level is not registered, how do I decide which user buffer to provide to the dropped handler? Oh, so if I am reading your clarification correctly above, you are saying I should not invoke the dropped handler until a callback for that safety level has been registered. It is possible then that I might never call the dropped handler. Should I be managing dropped event counts for all safety levels? Ken

1 0

Re: [mpiwg-tools] question about MPI_T_EVENT_SET_DROPPED_HANDLER
by Raffenetti, Kenneth J. 25 Jan '21

25 Jan '21

On 1/25/21, 12:44 PM, "mpiwg-tools on behalf of Raffenetti, Kenneth J. via mpiwg-tools" <mpiwg-tools-bounces(a)lists.mpi-forum.org on behalf of mpiwg-tools(a)lists.mpi-forum.org> wrote: On 1/25/21, 11:46 AM, "Marc-André Hermanns" <hermanns(a)itc.rwth-aachen.de> wrote: For me (although not explicitly defined in the text) that would imply that the dropped callback is only invoked once the library is in a state where a callback (and user_data) for the same level is defined and could be invoked. Does this make sense and clarify your concerns? Do you think this need more clarification in the text? I see. Yes, that makes sense for my implementation issue. I may have spoken too soon. I'm looking at pg 763, ln 37 (Handling Dropped Events): If an event occurs...[and] no callback function meeting the required callback safety level is registered, the event data may be dropped. To discover such data loss, the user can set a handler function for a specific event-registration handle. If a callback function meeting the required safety level is not registered, how do I decide which user buffer to provide to the dropped handler? Ken

1 0

Re: [mpiwg-tools] question about MPI_T_EVENT_SET_DROPPED_HANDLER
by Raffenetti, Kenneth J. 25 Jan '21

25 Jan '21

On 1/25/21, 11:46 AM, "Marc-André Hermanns" <hermanns(a)itc.rwth-aachen.de> wrote: For me (although not explicitly defined in the text) that would imply that the dropped callback is only invoked once the library is in a state where a callback (and user_data) for the same level is defined and could be invoked. Does this make sense and clarify your concerns? Do you think this need more clarification in the text? I see. Yes, that makes sense for my implementation issue. Ken

1 0

Re: [mpiwg-tools] question about MPI_T_EVENT_SET_DROPPED_HANDLER
by Marc-André Hermanns 25 Jan '21

25 Jan '21

Hi Ken, > I am working on implementing MPI_T events in MPICH and came across > a potential issue. MPI_T_EVENT_SET_DROPPED_HANDLER does not take a > user_data argument, but the callback signature and standard text > suggest that it should be there.> > The argument user_data is the pointer to user-allocated memory that > was passed to the MPI implementation during callback registration.> > Am I correct in understanding that user_data is specified for each > callback registration? I.e. if a user registers callbacks for > different safety levels, those callbacks can be associated with > different user_data buffers. If so, it seems the dropped handler > registration is missing this functionality. I think page 768, lines 33-37 clarifies the behavior: "The argument cb_safety describes the safety requirements the callback must fulfill in the current invocation. [...] The argument user_data is the pointer to the user-allocated memory that was passed to the MPI implementation during callback registration." So I would pass the pointer that is associated with the safety level that would apply to the invocation. For me (although not explicitly defined in the text) that would imply that the dropped callback is only invoked once the library is in a state where a callback (and user_data) for the same level is defined and could be invoked. Does this make sense and clarify your concerns? Do you think this need more clarification in the text? What is the opinion of the others in the group? Does this sound reasonable? Is it underspecified? Cheers, Marc-Andre -- Dr. rer. nat. Marc-André Hermanns IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 52074 Aachen Phone: +49 241 80-24381 hermanns(a)itc.rwth-aachen.de www.itc.rwth-aachen.de Social Media Kanäle des IT Centers: https://blog.rwth-aachen.de/itc/ https://www.facebook.com/itcenterrwth https://www.linkedin.com/company/itcenterrwth https://twitter.com/ITCenterRWTH https://www.youtube.com/channel/UCKKDJJukeRwO0LP-ac8x8rQ

1 0

question about MPI_T_EVENT_SET_DROPPED_HANDLER
by Raffenetti, Kenneth J. 22 Jan '21

22 Jan '21

Hi, I am working on implementing MPI_T events in MPICH and came across a potential issue. MPI_T_EVENT_SET_DROPPED_HANDLER does not take a user_data argument, but the callback signature and standard text suggest that it should be there. The argument user_data is the pointer to user-allocated memory that was passed to the MPI implementation during callback registration. Am I correct in understanding that user_data is specified for each callback registration? I.e. if a user registers callbacks for different safety levels, those callbacks can be associated with different user_data buffers. If so, it seems the dropped handler registration is missing this functionality. Thanks, Ken

1 0

Meeting reminder: Jan 21, 2020 (today)
by Marc-André Hermanns 21 Jan '21

21 Jan '21

Hi all, This is a reminder of our regularly scheduled call today: Jan 21 at 8 am Pacific / 11 am Eastern / 5pm CEST. Meeting Details: https://github.com/mpiwg-tools/mpi-standard/wiki/Meeting-Information *NOTE*: I have updated the meeting details and switched to *ZOOM* for the meeting platform. Let me know if something does not work out for you. Agenda: ------- - Query calls for handle meta data Wesley has a conflict today, and won't be able to join. The ongoing handle introspection is currently not prepared enough to continue yet. Joachim brought up a scenario, where the ability to query whether a request is persistent or not would be helpful. So today we could discuss about specific meta data that would be interesting to query on the user level. Cheers, Marc-Andre -- Dr. rer. nat. Marc-André Hermanns IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 52074 Aachen Phone: +49 241 80-24381 hermanns(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0