If dropped messages are exported by MPI to the application, any recovery
mechanism by the application would have no need to place the lost message
back into the original sequence order. The application will know that the
message is dropped and none of the APIs in the transactional messages
proposal allow the application to put a message into some specific spot
in the message order. It is informed of the drop and it may do whatever
it wants. Importantly, if MPI notices that a message is corrupted and
tells the application that it is dropped, it will also be able to perform
correct matching with that gap in the sequence number. In fact, that
requirement must be included in the spec in order to make sure that we
don't have any ambiguities. If a message is dropped, it should be counted
as delivered when the application is informed of the drop.
As for high-level recovery techniques, I can see a number of uses for
message drop information. It may be that the message is some sort of
periodic notification from the master to the slaves and dropped messages
are irrelevant. It may also be that the application is in fact using some
sort of protocol to overcome such failures. The point is that such
notifications can be useful and the main question for us is the
appropriate balance between clean semantics and the performance hit. The
fully transactional option is going to be expensive but very convenient.
However, we have a number of other options that will be quite useful for
use with higher-level recovery protocols and quite cheap. I think that we
should gravitate towards those but I thought it important to include a
variety of options for people to toss around.
Greg Bronevetsky
Post-Doctoral Researcher
1028 Building 451
Lawrence Livermore National Lab
(925) 424-5756
bronevetsky1@llnl.gov
Actually,
it is most likely that MPI implementations that don’t try to deal
with
dropped messages, can’t even detect that such event have
occurred. For
such implementation I would expect them to be able to detect a
problem with
failed communications only if the low-level library they use to
implement
the communications, such as some OS bypass library, returns an
error when
trying to post some sort of communications, or if the run-time used
by MPI
detects a fail process, and propagates this information to the rest
of the
processes in the application.
The ONLY layer that can handle any sort of recovery from a live
communications failure -
i.e. w/o some sort of check-point restart with or with out message
logging – is the
MPI implementation itself. The application reposting a send
can’t take get around the
lost data, because of the MPI message ordering requirements, unless
the implementation
totally relies on another library to satisfy the MPI ordering
requirements (i.e. it does not
generate some sort of message sequence number) and the message lost
is the last one
that was sent. MPI is not allowed to attempt any matching if
there is a gap in the
sequence number.