Hello Bronis, the idea was not to eliminate PMPI per se. But rather allow in the standard a well-defined approach, with which function calls (due to the PMPI-requirement) can be macroized and additionally allowing the user to pass more information to the library. This should enable the compiler to do "a better job"... - The users that do not want it, do not specify the pre-processor flag and may happily use PMPI-based tools. - The users that do care, may specify it -- and depending on the implementation/hardware may see performance improvements. If the MPI-implementation does not yet process these macros, well so be it. Therefore I see this approach as the last step before deployment of an application. The proposed approach is just for discussion and by no means was meant to be solution for "the subset" -- however it may be a starting point and pre-processor based information a valid way, that goes hand-in-hand with other (MPI_Info-based?) subset-approaches... Thanks, Rainer On Friday 07 March 2008 17:20, Bronis R. de Supinski wrote:

...

...

A properly designed subsetting approach will NOT try to partition the profiling interface into its own subset. Instead, the interface must be partitioned into different subsets, with each PMPI function in the same subset as the corresponding MPI function.

I'm not sure that I understand your rationale here (perhaps we can discuss more in Chicago?).

Sure, we can discuss it in Chicago. However, The rationale is simple enough. The user of a subset wants to be able to profile the functionality in a given subset.

...

In my view of the world, at least one way to effect subsetting could be something that is dynamically loaded (or not) at run-time. Hence, you could have PMPI or not, depending on options given to mpirun (for example). I'm obviously waving my hands a bit here, but that's the premise I was assuming (details TBD, of course).

I have no objection to your premise. However, this proposal is specifically designed to prevent dynamic expansion of the feature set to include the PMPI calls (or at least to doing so effectively since there are no MPI calls between which the tool could interpose). In fact, the ability to change the subset dynamically is beyond what I was requiring (but is desirable). All I stated was that the expansion should not require recompilation in order to include the profiling interface for what is included in the subset.

If you (or anyone else) can design a mechanism that allows using the macros that Rainer wants to enable without precluding that interposition dynamically then I offer no opposition to the idea. However, I would also suggest that such a mechanism does not need any change to the standard; it is already compliant.

...

From your later mail:

...

Also, while I see your point that only a small set of stuff needs to be placed into mpi.h, that does not address the real question: What is the actual application level benefit? Why do I care if I can get MPI_Comm_size or MPI_Comm_rank inlined? Good programming practice does not call them frequently enough for any performance gain to matter. Alternatively, suppose I can squash some stuff out of the MPI_Send call path. Will it really matter to real applications? It seems unlikely, particularly relative to the cost of lost flexibility for understanding communication behavior and performance.

Rainer's proposal came from joint discussions with NEC where (correct me if I'm wrong, Hubert) select functions *are* [optionally?] inlined in mpi.h because the cost of function calls on their platform is so high.

While one can argue whether the standard should avoid precluding the exploitation of as yet unbuilt clever hardware mechanisms, it seems pretty clear that it is not the job of the standard to accommodate poorly designed hardware. If the cost of function calls is so excessively high that ONE prevents effective use of the hardware then the vendor needs to reconsider their design.

In any event, it appears that you are stating that a solution was found: offer a non-compliant alternative implementation. This is a far more appropriate solution in this case. Since I don't believe Rainer was proposing that the PMPI interface should be optional (just somehow not included in a base subset), the implementor still must support it. That means no implementation effort is saved by saying they could have a compliant implementation that does not always support the interface (but always offers the option to include that support). The implementor is always free to offer non-compliant alternatives; it is up the users whether they want to build with it.

...

Additionally, I'm always wary of the term "real applications" -- what a "real" application is to a US DOE lab is very different than what a "real application" is to, say, someone in the oil and gas industry. :-) The scale difference alone is enough to make the latency gains by inlining select MPI functions important (meaning: non- US-DOE-labs tend to run at [much] smaller scale; fabric latency may be constrained to a single switch, and therefore *can* see benefit from inlining).

As I understand Rainer's proposal, the inlining aspects also predicated on apps that enter production and are never changed. Perhaps this is not a pattern used much at DOE labs, but it is a pattern I see with many customers.

I'm not going to get into a pissing match with you over what applications do what. You seem to almost be using "US DOE lab" pejoratively. What I can say without the slightest hesitation is that recompilation is often a significant barrier to tool acceptance for application users and those users are NOT limited to one site or even one type of site (yes, I know users who are not at "DOE labs").

My opposition to changes that would force tools to require recompilation is borne from experience.

Bronis _______________________________________________ mpi3-subsetting mailing list mpi3-subsetting_at_[hidden] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-subsetting

-- ---------------------------------------------------------------- Dipl.-Inf. Rainer Keller http://www.hlrs.de/people/keller HLRS Tel: ++49 (0)711-685 6 5858 Nobelstrasse 19 Fax: ++49 (0)711-685 6 5832 70550 Stuttgart email: keller_at_[hidden] Germany AIM/Skype:rusraink