Hi Jeff, Thank you so much for the update! I already posted in Stack Overflow - no help yet! I will check with our Cray machine support staff and I will shoot an email to MPICH too! Really, thanks for the response! Best, Catherine On Tue, Jul 10, 2018 at 2:21 PM, Jeff Hammond <[email protected]> wrote:

Hi Catherine,

It sounds like this is an implementation issue. This email list is for discussion of the MPI standard itself. While many of us have a great deal of implementation expertise, you are likely to get the best response from the user list associated with the implementation you are using:

MPICH <[email protected]> Open-MPI <[email protected]> MVAPICH2 <[email protected]>

If you are using Cray MPI, you'll need to contact Cray support, perhaps via the staff that support your Cray machine locally. For Intel MPI, start with https://software.intel.com/en-us/forums/intel- clusters-and-hpc-technology/. I don't know about SGI or NEC support, unfortunately. You may also have good luck with StackOverflow - there are quite a few MPI experts there.

I'll note that most of the implementations of MPI I/O are based on ROMIO, which is part of MPICH, so you might want to start with the MPICH user list.

Best,

Jeff

On Tue, Jul 10, 2018 at 11:01 AM, Catherine Jenifer Rajam Rajendran < [email protected]> wrote:

...

Hi All,

I am trying to write in the same binary file using MPI. I set the offset for each process in the beginning as per the rank. Then the following code snippet in C runs. All MPI process executes and computes the value and it writes to the exact offset as set.

The problem I am facing is, say, out of 32 Process, one process is executed in 2 hours. Rest of the process keeps running for more than 24 hours, The thing is, it computes the values as expected but it takes so much time. It seems like a deadlock situation, each process waits for some resource. But, I am not sharing/communicating between the processes. I am just using MPI_File_write_at to write at a specific location in the binary file.

I need to mention that each process computes huge amount of data so storing it temporarily seemed inappropriate. I want to write the output in single file as number of processes is increased depending on input data. Number of computations are evenly distributed to all process. So, why does process takes different time interval to finish its job?!

for(i=1;i<=limit;i++) { for(j=i+1;j<=limit;j++) { if(my_rank == step%num_cpus) { Calc = Calculation(); buf[0] = (double)Calc; MPI_File_write_at(outFile, OUT_ofst, buf, 1, MPI_DOUBLE, &status); Calc = 0.0; OUT_ofst += num_cpus*MPI_File_write_at(sizeof(double)); count++; } step++; } }

I am new to MPI and I guess people must have had similar issues while executing in MPI. Can anyone help me out please! I can provide more details if needed.

Thanks, Catherine

_______________________________________________ mpi-forum mailing list [email protected] https://lists.mpi-forum.org/mailman/listinfo/mpi-forum

-- Jeff Hammond [email protected] http://jeffhammond.github.io/

_______________________________________________ mpi-forum mailing list [email protected] https://lists.mpi-forum.org/mailman/listinfo/mpi-forum