Dick, I think there are two independent questions here -- one is about the potential performance improvement, and another is about the ease of use (or rather complexity of use) for application writers. To some degree, I agree with both the concerns, but here are some notes for each one. With respect to the first, though latency improvement is probably only marginal (especially on fast processors), the degradation in the message rate is usually quite high. I've implemented this in MPICH2 and placed a copy here: https://svn.mcs.anl.gov/repos/mpi/mpich2/branches/dev/dyn_thread_level. Based on this implementation, I did some measurements using a single threaded application, but using MPI_Init_thread() vs. MPI_Init(). Experiment 1: Each process sends to MPI_PROC_NULL (emulating an infinitely fast network) MPI_Init -- 29.94 million messages per second MPI_Init_thread -- 15.86 million messages per second Difference -- almost 2X Experiment 2: Two processes communicating over TCP MPI_Init -- 3.947 million messages per second MPI_Init_thread -- 3.231 million messages per second Difference -- about 20% For fast networks, the difference might be somewhere in between. With respect to the second question, I agree that we are placing a lot of faith in the application developers for this, but this is probably not too different from trusting the application in that only one thread will call a Barrier or a Bcast, for example. If multiple threads call barriers in applications where only one thread is supposed to call, bad things can happen, but we trust that the user will not do that. Basically, there is a potential for bugs here, but I don't think that's any more than what is already present. If there are any concerns, or if you think we can do this in a better way, please do let me know. I'm open to editing this (even substantially if needed). Thanks. -- Pavan On 09/04/2008 09:35 AM, Richard Treumann wrote:

Pavan,

Can you explain what improvements you expect if MPI added this?

If an MPI implementation that is capable of supporting MPI_THREAD_MULTIPLE is running in MPI_THREAD_SINGLE mode, that probably means all protection against multiple threads is being bypassed at rather modest latency savings. The savings probably come from branching over lock operations. Ongoing checking for concurrent threads making MPI calls is expensive enough so doing it would eat up much of the savings from branching over lock operations. Basically, the MPI_THREAD_MULTIPLE capable MPI running in MPI_THREAD_SINGLE mode is probably defenseless against multi thread applications.

What this means for MPI_Set_thread_level(int required, int * provided);

is that the application would need to take full responsibility for making sure there is only one thread making MPI calls at the moment a mode switch is requested. If the application gets it wrong there is probably nothing (affordable) the MPI implementation can do to detect the danger. Applications that turned thread safety on and off but were misusing the call would never get an error message and might run without problems 99 times out of 100. 1% of the runs would have mysterious failures and the failures might each look quite different.

It is certainly possible for an MPI implementation that only supports MPI_THREAD_SINGLE to do some things faster by using simpler global data structures but such an MPI implementation could not honor a runtime request to convert to MPI_THREAD_MULTIPLE. It may even be possible for an MPI implementation at INIT time to chose simple vs. thread safe data structures but again, a switch once everything is up and running is probably not practical.

This does not seem to me to offer enough payoff to justify the dangers.

Dick

Dick Treumann - MPI Team IBM Systems & Technology Group Dept X2ZA / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363

mpi-22-bounces_at_[hidden] wrote on 09/04/2008 04:30:23 AM:

...

[image removed]

[Mpi-22] Dynamic thread levels

Pavan Balaji

to:

mpi-22

09/04/2008 04:39 AM

Sent by:

mpi-22-bounces_at_[hidden]

Please respond to "MPI 2.2"

Hi all,

I would like to propose an addition of a function call to dynamically modify the thread level required by the application, instead of at MPI_Init_thread() time. I'm sending this over email for comments (attached with this email); I'll upload it on the wiki this evening.

-- Pavan

-- Pavan Balaji http://www.mcs.anl.gov/~balaji Dynamic Thread Levels

Author: Pavan Balaji, Argonne National Laboratory

Background:

MPI 2.1 standard allows users to set thread level only at MPI_Init_thread time. This means that for applications that dynamically create threads (e.g., hybrid MPI + OpenMP applications), even non-threaded portions of the application have to rely on the maximum thread level used in the application (e.g., MULTIPLE).

Proposal:

Add an additional function call to dynamically set thread-level during the application.

int MPI_Set_thread_level(int required, int * provided);

Rational:

The requirement to specify the thread-level at MPI_Init_thread time is too restrictive for applications that perform a small amount of communication requiring a high-level of thread support. For correctness, the standard requires all of the code to follow the same thread-level, and provides the applications with no way to give the MPI library more information about their behavior.

Impact on MPI implementations:

Most MPI implementations already provide runtime support for thread-level, i.e., locks are compiled in, but whether they are invoked or not is decided at runtime. For implementations that choose not to respect this option, MPI_Set_thread_level() can just set the provided level to the current level, by ignoring the required level specified by the user.

Impact on MPI applications:

Existing MPI applications do not need to be modified at all. But newer applications can benefit from this additional functionality. _______________________________________________ mpi-22 mailing list mpi-22_at_[hidden] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22

------------------------------------------------------------------------

_______________________________________________ mpi-22 mailing list mpi-22_at_[hidden] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22

-- Pavan Balaji http://www.mcs.anl.gov/~balaji