Just something that can be structured into a tree on which paths or
inodes we declare the already existing MPI-IO files.
Not necessarily with all the constrains of POSIX.
Basically introducing simple inodes, implemented with supported
protocols / transports.
Axel
On 07.01.19 16:02, William Gropp wrote:
> Are we assuming a POSIX filesystem (a bad idea in my opinion)? Are we
> assuming a more restricted set of file system structure operations,
> along with weaker consistency semantics (matching the MPI-IO approach
> rather than POSIX)?
>
> Bill
>
> William Gropp
> Director and Chief Scientist, NCSA
> Thomas M. Siebel Chair in Computer Science
> University of Illinois Urbana-Champaign
>
>
>
>
>
>
>> On Jan 7, 2019, at 8:28 AM, Axel Huebl via mpi-comments
>> <mpi-comments(a)lists.mpi-forum.org
>> <mailto:[email protected]>> wrote:
>>
>> Dear MPI-Forum,
>>
>>
>> should we extend the MPI-I/O interfaces for directory operations?
>> Especially a "mkdir -p" equivalent could be useful.
>>
>> In many scientific apps that create output at scale, an output
>> structure needs to be created on demand.
>>
>> One can of course decide to do so with e.g. Linux `mkdir` but this can
>> be tricky:
>>
>> If all ranks do it, one will race the others and it's unnecessary
>> expensive in terms of operations.
>>
>> If a "root" rank does it, there is no guarantee that e.g. a
>> MPI_Barrier which syncs the "MPI-World" will also cause the
>> "filesystem-world" to be up-to-date. For example a mkdir on a slow NFS
>> local node has bot yet propagated to the other compute nodes.
>>
>> Comments, feedback and overlooked already existing solutions welcome!
>>
>>
>> Best regards,
>> Axel Huebl
>> --
>>
>> Axel Huebl
>> Phone: +49 351 260 3582
>> Institute of Radiation Physics
>> http://www.hzdr.de/crp
>> Helmholtz-Zentrum Dresden - Rossendorf (HZDR)
>> Bautzner Landstr. 400 | 01328 Dresden | Germany
>> Board of Directors:
>> Prof. Dr. Dr. h. c. Roland Sauerbrey, Dr. Ulrich Breuer
>> Company Registration Number VR 1693, Amtsgericht Dresden
>> _______________________________________________
>> mpi-comments mailing list
>> mpi-comments(a)lists.mpi-forum.org
>> https://lists.mpi-forum.org/mailman/listinfo/mpi-comments
>
--
Axel Huebl
Phone: +49 351 260 3582
Institute of Radiation Physics
http://www.hzdr.de/crp
Helmholtz-Zentrum Dresden - Rossendorf (HZDR)
Bautzner Landstr. 400 | 01328 Dresden | Germany
Board of Directors:
Prof. Dr. Dr. h. c. Roland Sauerbrey, Dr. Ulrich Breuer
Company Registration Number VR 1693, Amtsgericht Dresden
Are we assuming a POSIX filesystem (a bad idea in my opinion)? Are we assuming a more restricted set of file system structure operations, along with weaker consistency semantics (matching the MPI-IO approach rather than POSIX)?
Bill
William Gropp
Director and Chief Scientist, NCSA
Thomas M. Siebel Chair in Computer Science
University of Illinois Urbana-Champaign
> On Jan 7, 2019, at 8:28 AM, Axel Huebl via mpi-comments <mpi-comments(a)lists.mpi-forum.org> wrote:
>
> Dear MPI-Forum,
>
>
> should we extend the MPI-I/O interfaces for directory operations? Especially a "mkdir -p" equivalent could be useful.
>
> In many scientific apps that create output at scale, an output structure needs to be created on demand.
>
> One can of course decide to do so with e.g. Linux `mkdir` but this can be tricky:
>
> If all ranks do it, one will race the others and it's unnecessary expensive in terms of operations.
>
> If a "root" rank does it, there is no guarantee that e.g. a MPI_Barrier which syncs the "MPI-World" will also cause the "filesystem-world" to be up-to-date. For example a mkdir on a slow NFS local node has bot yet propagated to the other compute nodes.
>
> Comments, feedback and overlooked already existing solutions welcome!
>
>
> Best regards,
> Axel Huebl
> --
>
> Axel Huebl
> Phone: +49 351 260 3582
> Institute of Radiation Physics
> http://www.hzdr.de/crp
> Helmholtz-Zentrum Dresden - Rossendorf (HZDR)
> Bautzner Landstr. 400 | 01328 Dresden | Germany
> Board of Directors:
> Prof. Dr. Dr. h. c. Roland Sauerbrey, Dr. Ulrich Breuer
> Company Registration Number VR 1693, Amtsgericht Dresden
> _______________________________________________
> mpi-comments mailing list
> mpi-comments(a)lists.mpi-forum.org
> https://lists.mpi-forum.org/mailman/listinfo/mpi-comments
Dear MPI-Forum,
should we extend the MPI-I/O interfaces for directory operations?
Especially a "mkdir -p" equivalent could be useful.
In many scientific apps that create output at scale, an output structure
needs to be created on demand.
One can of course decide to do so with e.g. Linux `mkdir` but this can
be tricky:
If all ranks do it, one will race the others and it's unnecessary
expensive in terms of operations.
If a "root" rank does it, there is no guarantee that e.g. a MPI_Barrier
which syncs the "MPI-World" will also cause the "filesystem-world" to be
up-to-date. For example a mkdir on a slow NFS local node has bot yet
propagated to the other compute nodes.
Comments, feedback and overlooked already existing solutions welcome!
Best regards,
Axel Huebl
--
Axel Huebl
Phone: +49 351 260 3582
Institute of Radiation Physics
http://www.hzdr.de/crp
Helmholtz-Zentrum Dresden - Rossendorf (HZDR)
Bautzner Landstr. 400 | 01328 Dresden | Germany
Board of Directors:
Prof. Dr. Dr. h. c. Roland Sauerbrey, Dr. Ulrich Breuer
Company Registration Number VR 1693, Amtsgericht Dresden
