On 2/29/08 11:30 AM, "Supalov, Alexander" <alexander.supalov_at_[hidden]> wrote:

I see. Sorry for explaining the obvious. I guess the progress engine may take a hit every time there are either an MPI_ANY_SOURCE Recv or (thanks to Rich) multiple paths between the processes. Hence, all transfers are potentially affected.

With any_source the message can come from anyone, so the cost really depends on the mpi's queuing strategy, so the actual cost is very implementation specific. What ever the cost is, there are more potential sources, so at 100k there are 100k potential sources. The queuing could always have the unexpected messages cached in a single queue, but then all matching would be more expensive, vs. more of a hierarchical queue structure .... For expected messages there can also be an increase in matching costs, but again this is implementation specific. The other cost is that matching really has to be done at the destination - just a practical need - try to cancel 100k posted receives, after one match has been made, and make sure that only one proc has done the match.

Cancellation is a sticky matter. Some fabrics won't let you do this, so a cancel will always misfire.

Is this the case on the receive side ? The cancellation that Richard is mentioning is a receive side cancellation. I don't remember a network with this limitation, but I could very well be wrong on this one - I suppose it can also depend on how you do the matching. Rich

-----Original Message----- From: mpi3-subsetting-bounces_at_[hidden] [mailto:mpi3-subsetting-bounces_at_[hidden]] On Behalf Of Richard Barrett Sent: Friday, February 29, 2008 5:02 PM To: mpi3-subsetting_at_[hidden] Subject: [Mpi3-subsetting] MPI_ANY_SOURCE

...

...

Now, to one of your questions. An MPI_ANY_SOURCE

Although I appreciate the discussion, my intent (uh-oh!) in bring this up to let you know I "accept" the problem, yet ask for the capability anyway, but in a manner that keeps it from presenting problems everywhere. Or maybe I'm under-estimating what I was once told: the use of MPI_ANY_SOURCE anywhere means it is a problem everywhere, ie in _every_ function involved in transmitting data?

If that is the case, but I still _really_ wanted to use -lmpi_subset, I could do this: suppose a pe knows it will receive data from m pes. It could post numpe non-blocking receives, complete m, discover who they're from, then cancel the rest. Now I'm thinking I've created a bigger problem: when running acros numpes=100k cores, but m is say 10. True?

Barring some sort of workaround, excluding codes that "need" MPI_ANY_SOURCE seems to meaningfully reduce the number of codes that could use -lmpi_subset.

...

3. I (basically) understand the adverse performance effects of allowing promiscuous receives (MPI_ANY_SOURCE). However, this is a powerful capability for many codes, and used only in moderation, eg for setting up communication requirements (such as communication partners in unstructured, semi-structured, and dynamic mesh computations). In this case the sender knows its partner, but the receiver does not. A reduction(sum) is used to let each process know the number of communication partners from which it will receive data, the process posts that many promiscuous receives, which when satisfied lets it from then on specify the sender. So would it be possible to include this capability in a separate function, say the blocking send/recv, but not allow it in the non-blocking version?

Richard