Jeff: Re: [snip]

...

A properly designed subsetting approach will NOT try to partition the profiling interface into its own subset. Instead, the interface must be partitioned into different subsets, with each PMPI function in the same subset as the corresponding MPI function.

I'm not sure that I understand your rationale here (perhaps we can discuss more in Chicago?).

Sure, we can discuss it in Chicago. However, The rationale is simple enough. The user of a subset wants to be able to profile the functionality in a given subset.

In my view of the world, at least one way to effect subsetting could be something that is dynamically loaded (or not) at run-time. Hence, you could have PMPI or not, depending on options given to mpirun (for example). I'm obviously waving my hands a bit here, but that's the premise I was assuming (details TBD, of course).

I have no objection to your premise. However, this proposal is specifically designed to prevent dynamic expansion of the feature set to include the PMPI calls (or at least to doing so effectively since there are no MPI calls between which the tool could interpose). In fact, the ability to change the subset dynamically is beyond what I was requiring (but is desirable). All I stated was that the expansion should not require recompilation in order to include the profiling interface for what is included in the subset. If you (or anyone else) can design a mechanism that allows using the macros that Rainer wants to enable without precluding that interposition dynamically then I offer no opposition to the idea. However, I would also suggest that such a mechanism does not need any change to the standard; it is already compliant.

From your later mail:

...

Also, while I see your point that only a small set of stuff needs to be placed into mpi.h, that does not address the real question: What is the actual application level benefit? Why do I care if I can get MPI_Comm_size or MPI_Comm_rank inlined? Good programming practice does not call them frequently enough for any performance gain to matter. Alternatively, suppose I can squash some stuff out of the MPI_Send call path. Will it really matter to real applications? It seems unlikely, particularly relative to the cost of lost flexibility for understanding communication behavior and performance.

Rainer's proposal came from joint discussions with NEC where (correct me if I'm wrong, Hubert) select functions *are* [optionally?] inlined in mpi.h because the cost of function calls on their platform is so high.

While one can argue whether the standard should avoid precluding the exploitation of as yet unbuilt clever hardware mechanisms, it seems pretty clear that it is not the job of the standard to accommodate poorly designed hardware. If the cost of function calls is so excessively high that ONE prevents effective use of the hardware then the vendor needs to reconsider their design. In any event, it appears that you are stating that a solution was found: offer a non-compliant alternative implementation. This is a far more appropriate solution in this case. Since I don't believe Rainer was proposing that the PMPI interface should be optional (just somehow not included in a base subset), the implementor still must support it. That means no implementation effort is saved by saying they could have a compliant implementation that does not always support the interface (but always offers the option to include that support). The implementor is always free to offer non-compliant alternatives; it is up the users whether they want to build with it.

Additionally, I'm always wary of the term "real applications" -- what a "real" application is to a US DOE lab is very different than what a "real application" is to, say, someone in the oil and gas industry. :-) The scale difference alone is enough to make the latency gains by inlining select MPI functions important (meaning: non- US-DOE-labs tend to run at [much] smaller scale; fabric latency may be constrained to a single switch, and therefore *can* see benefit from inlining).

As I understand Rainer's proposal, the inlining aspects also predicated on apps that enter production and are never changed. Perhaps this is not a pattern used much at DOE labs, but it is a pattern I see with many customers.

I'm not going to get into a pissing match with you over what applications do what. You seem to almost be using "US DOE lab" pejoratively. What I can say without the slightest hesitation is that recompilation is often a significant barrier to tool acceptance for application users and those users are NOT limited to one site or even one type of site (yes, I know users who are not at "DOE labs"). My opposition to changes that would force tools to require recompilation is borne from experience. Bronis