Hi Ken,
Oh, so if I am reading your clarification correctly above, you are saying I should not invoke the dropped handler until a callback for that safety level has been registered. It is possible then that I might never call the dropped handler. Should I be managing dropped event counts for all safety levels?
Let me see if I can give a more concrete example. Let's say I have an event registration handle allocated, and a callback registered for the lowest safety level (NONE). I also have a dropped handler registered. The event itself is generating instances for higher safety levels (MPI_RESTRICTED, THREAD_SAFE), but because there are no handlers registered for those levels, the events are being dropped.
Question 1: Should the implementation be recording those dropped events for higher safety levels? The text leads me to believe yes. Question 2: Should the implementation delay calling the dropped handler until a callback satisfying the safety level of the dropped events has been registered? Your clarification leads me to believe yes. Question 3: If I register a MPI_RESTRICTED callback, will the dropped handler then execute to inform me of all the dropped events at that safety level? Question 4: If I instead register a THREAD_SAFE callback, will the dropped handler execute to inform me of all the dropped events at that safety level **as well as MPI_RESTRICTED**?
Let me see whether I can answer you questions, as some are indeed unclear with the current text/defined behavior. 1) The implementation should (where possible) record whenever an event could not be recorded and had to be dropped. 2) The implementation does not have to keep separate counters for each level. 3) The cb_safety argument of the handler is to define "what is possible for the tool inside this handler at the time of invocation" and does not imply the safety level events were dropped. Specifically, the implementation does not have to invoke the handler with the same safety level as where the event was dropped but (in the words of Capt. Jack Sparrow) "wait for the opportune moment" to invoke the handle with the highest leeway for the tool manageable. As higher level callbacks can fill in for lower levels (i.e., ASYNC_SIGNAL_SAFE can be invoked with level THREAD_SAFE) the implementation can always safely use the one with the lowest level still higher or equal to the level used when invoking the dropped handler. If an event is registered for a level that the implementation supports in general, there should be states in the execution where the dropped handler can be invoked with the matching safety level, and the implementation can delay until then. I hope this already clarifies most of the questions, yet you pointed out a severe under-specification that we need to fix: ---\/---\/---\/--- here be dragons ---\/---\/---\/---\/--- What happens when a user only registers a callback for NONE, and the library does not support these in the first place? Then, the library will never get to a place where it can issue the callback handler with a user provided user_data pointer. There could be different solutions for this situation, and each of them has a problem in our current situation: 1) Add a "user_data" element to the "dropped_handler" registration call. 2) If the MPI implementation does not support a certain level at all, have the callback registration function return MPI_T_ERR_NOT_SUPPORTED and refuse the registration in this case. 3) Use the 'user_data' registered with a callback at a safety level closest to the one where the dropped handler is invoked with. 4) Just accept that the dropped handler will never be called in this case. Option 1 would be in symmetry with the "free" handler registration, but would need an API change and a no-no vote to be included in 4.0. Is that likely to happen? Option 2 would be possible, yet MPI_T_ERR_NOT_SUPPORTED is not yet listed in Table 15.7 on page 778 in the correct row (it exists, but is listed only for MPI_T_SOURCE_*) ... We might handle this as an oversight and clarify it basically with the release of MPI 4.0 and include it in 4.1. Option 3 feels odd, as the we assume something about the characteristics of 'user_data' for a safety level that might not be the case and might not be safe. Option 4 would be possible, yet lacks the feedback to the user that something went wrong. What's your opinion? Are there other/better ways to deal with the situation? Are there other opinions/ideas in the group on how to resolve this? Cheers, Marc-Andre -- Dr. rer. nat. Marc-André Hermanns IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 52074 Aachen Phone: +49 241 80-24381 [email protected] www.itc.rwth-aachen.de Social Media Kanäle des IT Centers: https://blog.rwth-aachen.de/itc/ https://www.facebook.com/itcenterrwth https://www.linkedin.com/company/itcenterrwth https://twitter.com/ITCenterRWTH https://www.youtube.com/channel/UCKKDJJukeRwO0LP-ac8x8rQ