On Tue, May 6, 2014 at 3:06 PM, Jeff Hammond <[email protected]> wrote:

On Tue, May 6, 2014 at 2:50 PM, Rob Latham <[email protected]> wrote:

...

On 05/06/2014 12:19 PM, Jeff Hammond wrote:

...

Issue #2: cannot use built-in reduce ops.

Once we rule out using built-in ops with our large-count datatypes, we must reimplement all of the reduction operations required. I find this to be nontrivial. I have not yet figured out how to get at the underlying datatype info in a simple manner. It appears that MPI_Type_get_envelope exists for this purpose, but it's a huge pain to have to call this function when all I need to know is the number of built-in datatypes so that I can apply my clever and use MPI_Reduce_local inside of my user-defined operation.

To determine the number of built-in datatypes, yes, one must recursively call MPI_Type_get_envelope and MPI_Type_get_contents. See the ROMIO datatype flattening code for just how much of a pain MPI_Type_get_contents and MPI_Type_get_envelope is.

Indeed, I finished debugging my implementation of that for BigMPI a few minutes ago (https://github.com/jeffhammond/BigMPI/blob/master/src/type_contiguous_x.c). It was obnoxious just to do it for the single case where I am parsing my own datatype and thus know exactly what should be in it. It takes 6 MPI calls to convert from the contiguous bigtype to a (bigcount,basictype) pair, just as it takes 6 MPI calls to go the other way.

...

You might find Rob Ross's libmpitypes approach a bit better, but that code is designed for applying functions to the data of a datatype, deliberately bypassing the get_envelope/get_contents information.

I just need the decoder ring so I can write my own reduction operations using MPI_Reduce_local so I think I'll avoid that dependency. I'd rather be implementation agnostic anyways and I guess libmpitypes is MPICH-oriented.

And to add to the grossness, I can't just define one reduction operation (at least in a non-evil i.e. no-globals way) I have to define one of them for every built-in op that I want to support, because there is no way - short of packing the MPI_Op into a new variant of bigtype (which would totally break my abstraction and probably requires me to put it on the heap and pass as a pointer because I think MPI_Op is an opaque handle) - to pass the built-in operation passed to my MPIX_Reduce_x into the MPI_User_function that I register. TL;DR = large-count reductions are hard. We are really torturing users here. Jeff -- Jeff Hammond [email protected]