Really? A 4 year old BGP question!? Do I remember it ... Paul Coffman is doing some ROMIO work. Let's get him involved and see how this fits in to what he's doing now on Q. It's good info for him anyway. "Rob Latham" <[email protected]> wrote on 10/17/2013 10:32:40 AM:

From: "Rob Latham" <[email protected]> To: [email protected], Cc: Bob Cernohous/Rochester/IBM@IBMUS Date: 10/17/2013 10:33 AM Subject: Re: [mpich2-dev] And yet another ROMIO performance question

On Wed, Sep 16, 2009 at 02:57:45PM -0500, Bob Cernohous wrote:

...

Same customer noticed that the file system was doing much more i/o than MPIIO on collective reads.

Hi Bob. Did you ever get an acceptable answer to this 4 year old question?

Have you ever had anyone respond to a 4 year old message? Let's see how well lotus notes handles /that/.

Sorry it took me so long to give you an answer. By the time I figured out what was going on, I forgot you needed an answer. In my meager defense, my daughter was 6 weeks old back then....

What happens is two-phase implements data sieving. Normally, this is good. On Blue Gene, we turn on two-phase all the time (the hint is "enable", not "automatic"), even when accesses are not overlapping.

So, consider a file domain where the first and last bytes are written. (e.g. a parallel-netcdf application is reading from a file with record variables). ROMIO will issue one read for the entire file domain.

Writes are even worse: ROMIO will read in the entire file domain, modify the regions, and write it out.

Not sure the best way to fix this. Probably stick in a few peephole optimizations, like if there's only one access in a file domain, just go ahead and issue that one access.

A "real" fix would have to build up a basic performance model so we could answer the question of where the tradeoff of "big request with extra wasted data" beats "many small requests, but no wasted data".

==rob

-- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA