December 2014 - devel - mailman3.cels.anl.gov

Re: [mpich-devel] --enable-wrapper-rpath not working
by Kenneth Raffenetti 18 Dec '14

18 Dec '14

On 12/18/2014 07:36 AM, Jeff Hammond wrote: > On Wed, Dec 17, 2014 at 2:00 PM, Kenneth Raffenetti > <raffenet(a)mcs.anl.gov> wrote: >> We've got an open bug related to using the Intel Fortran compiler on OSX. >> >> http://trac.mpich.org/projects/mpich/ticket/2024 >> >> What's missing from that bug description is that it appears that the >> convenience libraries (mpl, romio, etc.) are not properly embedded when >> linking with ifort on OSX. Your error output below is consistent with this >> issue. Is it possible to disable fortran in your MPICH build in this >> instance? That would at least confirm the source of the problem. > > I don't know that you understand what you're asking. Asking a quantum > chemist to disable Fortran is like asking Bret Michaels to disable > bandanas and groupies with low morals! > > Anyways, I can do this, and I even have a special hook so that I can > compile NWChem on Mac without MPI Fortran support because I often use > MPI built with LLVM. > > It is rather obnoxious that a Fortran issue would break the C/C++ > wrappers though. Priority on this bug dropped after we had trouble renewing our Intel licenses for OSX. I'll poke at the source to see if we can get them active again. Ken

1 0

Re: [mpich-devel] --enable-wrapper-rpath not working
by Jeff Hammond 18 Dec '14

18 Dec '14

On Wed, Dec 17, 2014 at 2:00 PM, Kenneth Raffenetti <raffenet(a)mcs.anl.gov> wrote: > We've got an open bug related to using the Intel Fortran compiler on OSX. > > http://trac.mpich.org/projects/mpich/ticket/2024 > > What's missing from that bug description is that it appears that the > convenience libraries (mpl, romio, etc.) are not properly embedded when > linking with ifort on OSX. Your error output below is consistent with this > issue. Is it possible to disable fortran in your MPICH build in this > instance? That would at least confirm the source of the problem. I don't know that you understand what you're asking. Asking a quantum chemist to disable Fortran is like asking Bret Michaels to disable bandanas and groupies with low morals! Anyways, I can do this, and I even have a special hook so that I can compile NWChem on Mac without MPI Fortran support because I often use MPI built with LLVM. It is rather obnoxious that a Fortran issue would break the C/C++ wrappers though. Jeff -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-devel] --enable-wrapper-rpath not working
by Kenneth Raffenetti 17 Dec '14

17 Dec '14

We've got an open bug related to using the Intel Fortran compiler on OSX. http://trac.mpich.org/projects/mpich/ticket/2024 What's missing from that bug description is that it appears that the convenience libraries (mpl, romio, etc.) are not properly embedded when linking with ifort on OSX. Your error output below is consistent with this issue. Is it possible to disable fortran in your MPICH build in this instance? That would at least confirm the source of the problem. Ken On 12/17/2014 02:28 PM, Jeff Hammond wrote: > This is what I do right now to get it working: > > ../configure --prefix=/Users/jrhammon/Work/MADNESS/install-intel > CC=icc CXX=icpc --with-tbb=/opt/intel/composer_xe_2015/tbb --with-mkl > CPPFLAGS=-I/opt/mpich/dev/intel/default/include > LDFLAGS=/opt/mpich/dev/intel/default/lib/libmpicxx.a > /opt/mpich/dev/intel/default/lib/libmpi.a > /opt/mpich/dev/intel/default/lib/libpmpi.a MPICC=icc MPICXX=icpc > > If I do the more straightforward invocation: > > ../configure --prefix=/Users/jrhammon/Work/MADNESS/install-intel > CC=icc CXX=icpc --with-tbb=/opt/intel/composer_xe_2015/tbb --with-mkl > MPICC=/opt/mpich/dev/intel/default/bin/mpicc > MPICXX=/opt/mpich/dev/intel/default/bin/mpicxx > > Then configure fails like this: > > mpicc > configure:5840: checking that the > /opt/mpich/dev/intel/default/bin/mpicc linker works > configure:5853: /opt/mpich/dev/intel/default/bin/mpicc -o conftest -g > -O2 conftest.c >&5 > Undefined symbols for architecture x86_64: > "_MPIO_Err_return_file", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPIR_ROMIO_Get_file_errhand", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPIR_ROMIO_Set_file_errhand", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPIX_File_iread_all", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPIX_File_iread_at_all", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPIX_File_iwrite_all", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPIX_File_iwrite_at_all", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_c2f", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_close", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_delete", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_f2c", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_get_amode", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_get_atomicity", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_get_byte_offset", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_get_group", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_get_info", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_get_position", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_get_position_shared", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_get_size", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_get_type_extent", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_get_view", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_iread", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_iread_at", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_iread_shared", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_iwrite", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_iwrite_at", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_iwrite_shared", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_open", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_preallocate", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_read", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_read_all", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_read_all_begin", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_read_all_end", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_read_at", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_read_at_all", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_read_at_all_begin", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_read_at_all_end", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_read_ordered", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_read_ordered_begin", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_read_ordered_end", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_read_shared", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_seek", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_seek_shared", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_set_atomicity", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_set_info", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_set_size", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_set_view", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_sync", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_write", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_write_all", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_write_all_begin", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_write_all_end", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_write_at", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_write_at_all", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_write_at_all_begin", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_write_at_all_end", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_write_ordered", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_write_ordered_begin", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_write_ordered_end", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_File_write_shared", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPI_Register_datarep", referenced from: > import-atom in libmpi.dylib > import-atom in libpmpi.dylib > "_MPL_TrSetMaxMem", referenced from: > import-atom in libpmpi.dylib > "_MPL_env2bool", referenced from: > import-atom in libpmpi.dylib > "_MPL_env2int", referenced from: > import-atom in libpmpi.dylib > "_MPL_env2range", referenced from: > import-atom in libpmpi.dylib > "_MPL_env2str", referenced from: > import-atom in libpmpi.dylib > "_MPL_large_readv", referenced from: > import-atom in libpmpi.dylib > "_MPL_large_writev", referenced from: > import-atom in libpmpi.dylib > "_MPL_strncpy", referenced from: > import-atom in libpmpi.dylib > "_MPL_trDebugLevel", referenced from: > import-atom in libpmpi.dylib > "_MPL_trcalloc", referenced from: > import-atom in libpmpi.dylib > "_MPL_trdump", referenced from: > import-atom in libpmpi.dylib > "_MPL_trfree", referenced from: > import-atom in libpmpi.dylib > "_MPL_trid", referenced from: > import-atom in libpmpi.dylib > "_MPL_trinit", referenced from: > import-atom in libpmpi.dylib > "_MPL_trlevel", referenced from: > import-atom in libpmpi.dylib > "_MPL_trmalloc", referenced from: > import-atom in libpmpi.dylib > "_MPL_trrealloc", referenced from: > import-atom in libpmpi.dylib > "_MPL_trspace", referenced from: > import-atom in libpmpi.dylib > "_MPL_trstrdup", referenced from: > import-atom in libpmpi.dylib > "_MPL_trvalid", referenced from: > import-atom in libpmpi.dylib > "_PMPIX_File_iread_all", referenced from: > import-atom in libpmpi.dylib > "_PMPIX_File_iread_at_all", referenced from: > import-atom in libpmpi.dylib > "_PMPIX_File_iwrite_all", referenced from: > import-atom in libpmpi.dylib > "_PMPIX_File_iwrite_at_all", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_close", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_delete", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_get_amode", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_get_atomicity", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_get_byte_offset", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_get_group", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_get_info", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_get_position", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_get_position_shared", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_get_size", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_get_type_extent", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_get_view", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_iread", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_iread_at", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_iread_shared", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_iwrite", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_iwrite_at", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_iwrite_shared", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_open", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_preallocate", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_read", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_read_all", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_read_all_begin", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_read_all_end", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_read_at", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_read_at_all", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_read_at_all_begin", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_read_at_all_end", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_read_ordered", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_read_ordered_begin", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_read_ordered_end", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_read_shared", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_seek", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_seek_shared", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_set_atomicity", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_set_info", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_set_size", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_set_view", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_sync", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_write", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_write_all", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_write_all_begin", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_write_all_end", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_write_at", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_write_at_all", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_write_at_all_begin", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_write_at_all_end", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_write_ordered", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_write_ordered_begin", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_write_ordered_end", referenced from: > import-atom in libpmpi.dylib > "_PMPI_File_write_shared", referenced from: > import-atom in libpmpi.dylib > "_PMPI_Register_datarep", referenced from: > import-atom in libpmpi.dylib > ld: symbol(s) not found for architecture x86_64 > > Recall: > > /opt/mpich/dev/intel/default/bin/mpicc -show > icc -Wl,-flat_namespace -I/opt/mpich/dev/intel/default/include > -L/opt/mpich/dev/intel/default/lib -lmpi -lpmpi > > Thanks, > > Jeff > > On Wed, Dec 17, 2014 at 12:18 PM, Kenneth Raffenetti > <raffenet(a)mcs.anl.gov> wrote: >> AFAICT, the Darwin linker includes the full path to all dynamic libraries >> needed by the library by default. Rpath flags shouldn't be strictly >> necessary. >> >> mpich-mac1:pt2pt raffenet$ otool -L large_message >> large_message: >> /Users/raffenet/mpich/i/lib/libmpi.0.dylib (compatibility version >> 1.0.0, current version 1.0.0) >> /Users/raffenet/mpich/i/lib/libpmpi.0.dylib (compatibility version >> 1.0.0, current version 1.0.0) >> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current >> version 1213.0.0) >> >> The concept of rpath was added in OSX 10.4 apparently, but I don't see much >> in the way of adoption in other open source projects. Are your binaries >> running into library not found issues? >> >> Ken >> >> >> On 12/17/2014 11:41 AM, Jeff Hammond wrote: >>> >>> Am I doing something wrong here? Why is the rpath option not being >>> added to the wrapper script? This issue makes it impossible for me to >>> use MPICH on my laptop without excessive tedium. >>> >>> Thanks, >>> >>> Jeff >>> >>> mobl:build-intel jrhammon$ /opt/mpich/dev/intel/default/bin/mpichversion >>> MPICH Version: 3.2a2 >>> MPICH Release date: unreleased development copy >>> MPICH Device: ch3:nemesis >>> MPICH configure: CC=icc CXX=icpc FC=ifort F77=ifort >>> --enable-fortran >>> --enable-threads=runtime --enable-g=dbg --with-pm=hydra >>> --prefix=/opt/mpich/dev/intel/default --enable-wrapper-rpath >>> --enable-static --enable-shared >>> MPICH CC: icc -g -O2 >>> MPICH CXX: icpc -g -O2 >>> MPICH F77: ifort -g -O2 >>> MPICH FC: ifort -g -O2 >>> >>> mobl:build-intel jrhammon$ /opt/mpich/dev/intel/default/bin/mpicc -show >>> icc -Wl,-flat_namespace -I/opt/mpich/dev/intel/default/include >>> -L/opt/mpich/dev/intel/default/lib -lmpi -lpmpi >>> >> _______________________________________________ >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/devel > > >

1 0

Re: [mpich-devel] --enable-wrapper-rpath not working
by Jeff Hammond 17 Dec '14

17 Dec '14

This is what I do right now to get it working: ../configure --prefix=/Users/jrhammon/Work/MADNESS/install-intel CC=icc CXX=icpc --with-tbb=/opt/intel/composer_xe_2015/tbb --with-mkl CPPFLAGS=-I/opt/mpich/dev/intel/default/include LDFLAGS=/opt/mpich/dev/intel/default/lib/libmpicxx.a /opt/mpich/dev/intel/default/lib/libmpi.a /opt/mpich/dev/intel/default/lib/libpmpi.a MPICC=icc MPICXX=icpc If I do the more straightforward invocation: ../configure --prefix=/Users/jrhammon/Work/MADNESS/install-intel CC=icc CXX=icpc --with-tbb=/opt/intel/composer_xe_2015/tbb --with-mkl MPICC=/opt/mpich/dev/intel/default/bin/mpicc MPICXX=/opt/mpich/dev/intel/default/bin/mpicxx Then configure fails like this: mpicc configure:5840: checking that the /opt/mpich/dev/intel/default/bin/mpicc linker works configure:5853: /opt/mpich/dev/intel/default/bin/mpicc -o conftest -g -O2 conftest.c >&5 Undefined symbols for architecture x86_64: "_MPIO_Err_return_file", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPIR_ROMIO_Get_file_errhand", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPIR_ROMIO_Set_file_errhand", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPIX_File_iread_all", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPIX_File_iread_at_all", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPIX_File_iwrite_all", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPIX_File_iwrite_at_all", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_c2f", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_close", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_delete", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_f2c", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_get_amode", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_get_atomicity", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_get_byte_offset", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_get_group", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_get_info", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_get_position", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_get_position_shared", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_get_size", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_get_type_extent", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_get_view", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_iread", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_iread_at", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_iread_shared", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_iwrite", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_iwrite_at", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_iwrite_shared", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_open", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_preallocate", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_read", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_read_all", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_read_all_begin", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_read_all_end", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_read_at", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_read_at_all", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_read_at_all_begin", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_read_at_all_end", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_read_ordered", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_read_ordered_begin", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_read_ordered_end", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_read_shared", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_seek", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_seek_shared", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_set_atomicity", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_set_info", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_set_size", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_set_view", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_sync", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_write", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_write_all", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_write_all_begin", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_write_all_end", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_write_at", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_write_at_all", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_write_at_all_begin", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_write_at_all_end", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_write_ordered", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_write_ordered_begin", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_write_ordered_end", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_File_write_shared", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPI_Register_datarep", referenced from: import-atom in libmpi.dylib import-atom in libpmpi.dylib "_MPL_TrSetMaxMem", referenced from: import-atom in libpmpi.dylib "_MPL_env2bool", referenced from: import-atom in libpmpi.dylib "_MPL_env2int", referenced from: import-atom in libpmpi.dylib "_MPL_env2range", referenced from: import-atom in libpmpi.dylib "_MPL_env2str", referenced from: import-atom in libpmpi.dylib "_MPL_large_readv", referenced from: import-atom in libpmpi.dylib "_MPL_large_writev", referenced from: import-atom in libpmpi.dylib "_MPL_strncpy", referenced from: import-atom in libpmpi.dylib "_MPL_trDebugLevel", referenced from: import-atom in libpmpi.dylib "_MPL_trcalloc", referenced from: import-atom in libpmpi.dylib "_MPL_trdump", referenced from: import-atom in libpmpi.dylib "_MPL_trfree", referenced from: import-atom in libpmpi.dylib "_MPL_trid", referenced from: import-atom in libpmpi.dylib "_MPL_trinit", referenced from: import-atom in libpmpi.dylib "_MPL_trlevel", referenced from: import-atom in libpmpi.dylib "_MPL_trmalloc", referenced from: import-atom in libpmpi.dylib "_MPL_trrealloc", referenced from: import-atom in libpmpi.dylib "_MPL_trspace", referenced from: import-atom in libpmpi.dylib "_MPL_trstrdup", referenced from: import-atom in libpmpi.dylib "_MPL_trvalid", referenced from: import-atom in libpmpi.dylib "_PMPIX_File_iread_all", referenced from: import-atom in libpmpi.dylib "_PMPIX_File_iread_at_all", referenced from: import-atom in libpmpi.dylib "_PMPIX_File_iwrite_all", referenced from: import-atom in libpmpi.dylib "_PMPIX_File_iwrite_at_all", referenced from: import-atom in libpmpi.dylib "_PMPI_File_close", referenced from: import-atom in libpmpi.dylib "_PMPI_File_delete", referenced from: import-atom in libpmpi.dylib "_PMPI_File_get_amode", referenced from: import-atom in libpmpi.dylib "_PMPI_File_get_atomicity", referenced from: import-atom in libpmpi.dylib "_PMPI_File_get_byte_offset", referenced from: import-atom in libpmpi.dylib "_PMPI_File_get_group", referenced from: import-atom in libpmpi.dylib "_PMPI_File_get_info", referenced from: import-atom in libpmpi.dylib "_PMPI_File_get_position", referenced from: import-atom in libpmpi.dylib "_PMPI_File_get_position_shared", referenced from: import-atom in libpmpi.dylib "_PMPI_File_get_size", referenced from: import-atom in libpmpi.dylib "_PMPI_File_get_type_extent", referenced from: import-atom in libpmpi.dylib "_PMPI_File_get_view", referenced from: import-atom in libpmpi.dylib "_PMPI_File_iread", referenced from: import-atom in libpmpi.dylib "_PMPI_File_iread_at", referenced from: import-atom in libpmpi.dylib "_PMPI_File_iread_shared", referenced from: import-atom in libpmpi.dylib "_PMPI_File_iwrite", referenced from: import-atom in libpmpi.dylib "_PMPI_File_iwrite_at", referenced from: import-atom in libpmpi.dylib "_PMPI_File_iwrite_shared", referenced from: import-atom in libpmpi.dylib "_PMPI_File_open", referenced from: import-atom in libpmpi.dylib "_PMPI_File_preallocate", referenced from: import-atom in libpmpi.dylib "_PMPI_File_read", referenced from: import-atom in libpmpi.dylib "_PMPI_File_read_all", referenced from: import-atom in libpmpi.dylib "_PMPI_File_read_all_begin", referenced from: import-atom in libpmpi.dylib "_PMPI_File_read_all_end", referenced from: import-atom in libpmpi.dylib "_PMPI_File_read_at", referenced from: import-atom in libpmpi.dylib "_PMPI_File_read_at_all", referenced from: import-atom in libpmpi.dylib "_PMPI_File_read_at_all_begin", referenced from: import-atom in libpmpi.dylib "_PMPI_File_read_at_all_end", referenced from: import-atom in libpmpi.dylib "_PMPI_File_read_ordered", referenced from: import-atom in libpmpi.dylib "_PMPI_File_read_ordered_begin", referenced from: import-atom in libpmpi.dylib "_PMPI_File_read_ordered_end", referenced from: import-atom in libpmpi.dylib "_PMPI_File_read_shared", referenced from: import-atom in libpmpi.dylib "_PMPI_File_seek", referenced from: import-atom in libpmpi.dylib "_PMPI_File_seek_shared", referenced from: import-atom in libpmpi.dylib "_PMPI_File_set_atomicity", referenced from: import-atom in libpmpi.dylib "_PMPI_File_set_info", referenced from: import-atom in libpmpi.dylib "_PMPI_File_set_size", referenced from: import-atom in libpmpi.dylib "_PMPI_File_set_view", referenced from: import-atom in libpmpi.dylib "_PMPI_File_sync", referenced from: import-atom in libpmpi.dylib "_PMPI_File_write", referenced from: import-atom in libpmpi.dylib "_PMPI_File_write_all", referenced from: import-atom in libpmpi.dylib "_PMPI_File_write_all_begin", referenced from: import-atom in libpmpi.dylib "_PMPI_File_write_all_end", referenced from: import-atom in libpmpi.dylib "_PMPI_File_write_at", referenced from: import-atom in libpmpi.dylib "_PMPI_File_write_at_all", referenced from: import-atom in libpmpi.dylib "_PMPI_File_write_at_all_begin", referenced from: import-atom in libpmpi.dylib "_PMPI_File_write_at_all_end", referenced from: import-atom in libpmpi.dylib "_PMPI_File_write_ordered", referenced from: import-atom in libpmpi.dylib "_PMPI_File_write_ordered_begin", referenced from: import-atom in libpmpi.dylib "_PMPI_File_write_ordered_end", referenced from: import-atom in libpmpi.dylib "_PMPI_File_write_shared", referenced from: import-atom in libpmpi.dylib "_PMPI_Register_datarep", referenced from: import-atom in libpmpi.dylib ld: symbol(s) not found for architecture x86_64 Recall: /opt/mpich/dev/intel/default/bin/mpicc -show icc -Wl,-flat_namespace -I/opt/mpich/dev/intel/default/include -L/opt/mpich/dev/intel/default/lib -lmpi -lpmpi Thanks, Jeff On Wed, Dec 17, 2014 at 12:18 PM, Kenneth Raffenetti <raffenet(a)mcs.anl.gov> wrote: > AFAICT, the Darwin linker includes the full path to all dynamic libraries > needed by the library by default. Rpath flags shouldn't be strictly > necessary. > > mpich-mac1:pt2pt raffenet$ otool -L large_message > large_message: > /Users/raffenet/mpich/i/lib/libmpi.0.dylib (compatibility version > 1.0.0, current version 1.0.0) > /Users/raffenet/mpich/i/lib/libpmpi.0.dylib (compatibility version > 1.0.0, current version 1.0.0) > /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current > version 1213.0.0) > > The concept of rpath was added in OSX 10.4 apparently, but I don't see much > in the way of adoption in other open source projects. Are your binaries > running into library not found issues? > > Ken > > > On 12/17/2014 11:41 AM, Jeff Hammond wrote: >> >> Am I doing something wrong here? Why is the rpath option not being >> added to the wrapper script? This issue makes it impossible for me to >> use MPICH on my laptop without excessive tedium. >> >> Thanks, >> >> Jeff >> >> mobl:build-intel jrhammon$ /opt/mpich/dev/intel/default/bin/mpichversion >> MPICH Version: 3.2a2 >> MPICH Release date: unreleased development copy >> MPICH Device: ch3:nemesis >> MPICH configure: CC=icc CXX=icpc FC=ifort F77=ifort >> --enable-fortran >> --enable-threads=runtime --enable-g=dbg --with-pm=hydra >> --prefix=/opt/mpich/dev/intel/default --enable-wrapper-rpath >> --enable-static --enable-shared >> MPICH CC: icc -g -O2 >> MPICH CXX: icpc -g -O2 >> MPICH F77: ifort -g -O2 >> MPICH FC: ifort -g -O2 >> >> mobl:build-intel jrhammon$ /opt/mpich/dev/intel/default/bin/mpicc -show >> icc -Wl,-flat_namespace -I/opt/mpich/dev/intel/default/include >> -L/opt/mpich/dev/intel/default/lib -lmpi -lpmpi >> > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-devel] --enable-wrapper-rpath not working
by Kenneth Raffenetti 17 Dec '14

17 Dec '14

AFAICT, the Darwin linker includes the full path to all dynamic libraries needed by the library by default. Rpath flags shouldn't be strictly necessary. mpich-mac1:pt2pt raffenet$ otool -L large_message large_message: /Users/raffenet/mpich/i/lib/libmpi.0.dylib (compatibility version 1.0.0, current version 1.0.0) /Users/raffenet/mpich/i/lib/libpmpi.0.dylib (compatibility version 1.0.0, current version 1.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0) The concept of rpath was added in OSX 10.4 apparently, but I don't see much in the way of adoption in other open source projects. Are your binaries running into library not found issues? Ken On 12/17/2014 11:41 AM, Jeff Hammond wrote: > Am I doing something wrong here? Why is the rpath option not being > added to the wrapper script? This issue makes it impossible for me to > use MPICH on my laptop without excessive tedium. > > Thanks, > > Jeff > > mobl:build-intel jrhammon$ /opt/mpich/dev/intel/default/bin/mpichversion > MPICH Version: 3.2a2 > MPICH Release date: unreleased development copy > MPICH Device: ch3:nemesis > MPICH configure: CC=icc CXX=icpc FC=ifort F77=ifort --enable-fortran > --enable-threads=runtime --enable-g=dbg --with-pm=hydra > --prefix=/opt/mpich/dev/intel/default --enable-wrapper-rpath > --enable-static --enable-shared > MPICH CC: icc -g -O2 > MPICH CXX: icpc -g -O2 > MPICH F77: ifort -g -O2 > MPICH FC: ifort -g -O2 > > mobl:build-intel jrhammon$ /opt/mpich/dev/intel/default/bin/mpicc -show > icc -Wl,-flat_namespace -I/opt/mpich/dev/intel/default/include > -L/opt/mpich/dev/intel/default/lib -lmpi -lpmpi >

1 0

--enable-wrapper-rpath not working
by Jeff Hammond 17 Dec '14

17 Dec '14

Am I doing something wrong here? Why is the rpath option not being added to the wrapper script? This issue makes it impossible for me to use MPICH on my laptop without excessive tedium. Thanks, Jeff mobl:build-intel jrhammon$ /opt/mpich/dev/intel/default/bin/mpichversion MPICH Version: 3.2a2 MPICH Release date: unreleased development copy MPICH Device: ch3:nemesis MPICH configure: CC=icc CXX=icpc FC=ifort F77=ifort --enable-fortran --enable-threads=runtime --enable-g=dbg --with-pm=hydra --prefix=/opt/mpich/dev/intel/default --enable-wrapper-rpath --enable-static --enable-shared MPICH CC: icc -g -O2 MPICH CXX: icpc -g -O2 MPICH F77: ifort -g -O2 MPICH FC: ifort -g -O2 mobl:build-intel jrhammon$ /opt/mpich/dev/intel/default/bin/mpicc -show icc -Wl,-flat_namespace -I/opt/mpich/dev/intel/default/include -L/opt/mpich/dev/intel/default/lib -lmpi -lpmpi -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-devel] Performance with MPICH
by Lu, Huiwei 04 Dec '14

04 Dec '14

Hi, Arthur, This devel list is used for discuss with development of the MPICH runtime itself. For general questions about how to write parallel programs, please refer to MPI tutorials or try stackoverflow. My personal favorite tutorials are: https://computing.llnl.gov/tutorials/parallel_comp/ http://mpitutorial.com Thanks, — Huiwei > On Dec 4, 2014, at 5:31 AM, arthur.neto(a)ccee.org.br wrote: > > Hi > > I have a problem when I run my program using a server with 16 processor time is the same using 4 server there is some parameter of performace I can do > > > <Mail Attachment.jpeg> Câmara de Comercialização de Energia Elétrica > Arthur Domingos Neto > Gerência de Tecnologia da Informação > e-mail: arthur.neto(a)ccee.org.br > tel: (55) (11) - 3175-6049 > fax: (55) (11) - 3175-6676 > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel >

1 0

Re: [mpich-devel] Performance with MPICH
by "Antonio J. Peña" 04 Dec '14

04 Dec '14

Hi Arthur, The performance issue you are mentioning is very likely to be specific to your code. Unless you identify a performance problem in MPICH, we are not able to help you here. You may look for assistance in a forum specialized in your specific application domain, or in a general-purpose coding forum such as stackoverflow. Best, Antonio On 12/04/2014 05:31 AM, arthur.neto(a)ccee.org.br wrote: > Hi > > I have a problem when I run my program using a server with 16 > processor time is the same using 4 server there is some parameter of > performace I can do > > > *Câmara de Comercialização de Energia Elétrica > Arthur Domingos Neto > Gerência de Tecnologia da Informação * > e-mail: arthur.neto(a)ccee.org.br > tel: (55) (11) - 3175-6049 > fax: (55) (11) - 3175-6676 > > > > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel -- Antonio J. Peña Postdoctoral Appointee Mathematics and Computer Science Division Argonne National Laboratory 9700 South Cass Avenue, Bldg. 240, Of. 3148 Argonne, IL 60439-4847 apenya(a)mcs.anl.gov www.mcs.anl.gov/~apenya

1 0

Re: [mpich-devel] Fatal error in PMPI_Barrier
by Giuseppe Congiu 04 Dec '14

04 Dec '14

Thanks again Rob, I will ! Giuseppe On 4 December 2014 at 14:51, Rob Latham <robl(a)mcs.anl.gov> wrote: > > > On 12/04/2014 08:16 AM, Giuseppe Congiu wrote: > >> Hello Rob, >> >> I think I may have just found what the problem is: >> >> In ADIOI_GEN_IfileSync() I am passing args to MPI_Grequest_start() to >> later get how much data has been written to global file system in the >> query function and also return the error_code. The problem is that I am >> freeing args in the pthread routine (which is not reported in the >> previous email btw). This causes MPI_Wait() catching fire when the >> pthread has already completed and args freed (which happens when I have >> many segments). I fixed the code and now it seems to work : ) >> >> I am sorry I have bothered you with a silly problem just related to a >> lack of attention from my side. Nevertheless, talking to you has helped >> me finding the solution in a few mins after wandering for days. >> > > My old office mate had a little rubber duck he'd keep on his desk. When > faced with a programming problem, he'd talk to "debugging ducky" and sort > out the state. > > I'm glad I could be your debugging ducky. I am always glad to come > across folks brave enough to hack on the ROMIO code. Please keep me > updated with your research. > > ==rob > > >> Best Regards, >> >> Giuseppe >> >> On 4 December 2014 at 13:14, Giuseppe Congiu >> <giuseppe.congiu(a)seagate.com <mailto:[email protected]>> wrote: >> >> >> On 3 December 2014 at 21:19, Rob Latham <robl(a)mcs.anl.gov >> <mailto:[email protected]>> wrote: >> >> Sorry not to have responded to you sooner. Between a big >> conference and US thanksgiving, a lot of us were out of the >> office for the last few weeks. >> >> >> Hello Rob, I can totally understand, I have been also busy with >> other stuff in the last few weeks. Nevertheless, I cannot deny I >> have been looking forward for a reply : ) >> >> Welcome. You're digging into ROMIO and Generalized Requests, so >> you've picked two fairly obscure areas of MPI to work on! >> >> As it happens, I am the world's expert in both ROMIO and >> Generalized requests. (the population size for that domain is >> exceedingly small...) >> >> >> Then you are the blessing that I have been waiting for : ) >> >> I think this approach can greatly benefit collective I/O -- I >> observe on some systems that the communication cost is not >> hurting two-phase collective I/O but instead it is the >> synchronization: if one I/O aggregator is taking longer to do >> I/O than the others, all N processes pay a price waiting for the >> laggard. >> >> >> Yes, this is exactly the point. Communication cost does not seem to >> be a problem, on the other hand the MPI_Allreduce() at the end of >> the collective write is a real pain. By writing to local devices, >> the I/O response time variation among aggregators can be greatly >> reduced from what I observed. >> >> before we get too far, we need to talk a bit about generalized >> requests and their "super-strict" progress model. Can you >> share your query, free, and cancel functions? >> >> >> The call back functions are quite simple, they don't do anything >> special: >> >> int ADIOI_GEN_Ifile_sync_query_fn( void *extra_state, >> MPI_Status *status ) >> { >> ARGS *state = (ARGS*)extra_state; >> >> MPI_Status_set_cancelled( status, 0 ); >> MPI_Status_set_elements( status, MPI_BYTE, >> state->bytes_xfered ); >> status->MPI_SOURCE = MPI_UNDEFINED; >> status->MPI_TAG = MPI_UNDEFINED; >> return state->error_code; >> } >> >> int ADIOI_GEN_Ifile_sync_free_fn( void *extra_state ) >> { >> return MPI_SUCCESS; >> } >> >> int ADIOI_GEN_Ifile_sync_cancel_fn( void *extra_state, int >> complete ) >> { >> return MPI_SUCCESS; >> } >> >> Which part of this ends up in a generalized request? >> >> >> The function that ends up in a generalized request is the >> synchronization function. ADIOI_GEN_IfileSync(), which is also >> pretty simple. It just reads the locally written file domains and >> writes them to the global file. >> >> MPICH is going to set those hints to "automatic", but you have >> overridden the defaults? (it's the right override in most >> cases, so good! unless you did not, in which case we should >> double check that you are not mixing vendor MPICH and your own >> MPICH) >> >> I am setting the MPI hints using a IOR configuration file, follows a >> config file snippet: >> >> IOR_HINT__MPI__cb_buffer_size=16777216 >> IOR_HINT__MPI__cb_nodes=1 >> IOR_HINT__MPI__romio_cb_read=enable >> IOR_HINT__MPI__romio_cb_write=enable >> IOR_HINT__MPI__local_cache=enable >> IOR_HINT__MPI__local_cache_path=/tmp/ior_tmp_file >> IOR_HINT__MPI__local_cache_flush_flag=flush_immediate >> IOR_HINT__MPI__local_cache_discard_flag=enable >> IOR_HINT__MPI__romio_no_indep_rw=true >> >> the collective buffer size by default is 16MB so the hint in this >> case is superfluous (same goes for cb_nodes=1 since I am using only >> one node). >> >> MPIR_Barrier_intra(169): >> PSIlogger: Child with rank 5 exited with status 1. >> mpid_irecv_done(101)...: read from socket failed - request >> state:recv(pde)done >> wait entry: 2a1ef18, 21f2790, 2a1e8e8, 21f4488 >> Fatal error in PMPI_Wait: Other MPI error, error stack: >> PMPI_Wait(180)..........: MPI_Wait(request=0x21f4488, >> status=0x21f2138) >> failed >> MPIR_Wait_impl(77)......: >> MPIR_Grequest_query(447): user request query function >> returned error >> code 1601 >> >> >> I wonder if your query is doing more than it's supposed to do.... >> >> >> Maybe you can tell me. I basically used Chapter 12 - External >> Interfaces - (from MPI specifications published September 21, 2012) >> as reference for the generalized request interface. >> I just want to remind that this problem is showing itself only when >> I am writing more than 4 segments in IOR. >> >> Does anyone have a clue of what is going wrong here? >> >> >> We're going to need to see some more code, I think... >> >> >> Follows the function that starts the generalized request to >> synchronize the file: >> >> int ADIOI_GEN_IfileSync( ADIO_File fd, int count, MPI_Datatype >> datatype, >> int file_ptr_type, ADIO_Offset offset, >> ADIO_Request *request ) >> { >> pthread_t thread; >> pthread_attr_t attr; >> int rc; >> ARGS *args; >> >> /* copy args */ >> args = (ARGS *)ADIOI_Malloc(sizeof(ARGS)); >> args->fd = fd; >> args->count = count; >> args->datatype = datatype; >> args->file_ptr_type = file_ptr_type; >> args->offset = offset; >> args->request = request; >> >> /* start generalized request */ >> MPI_Grequest_start( &ADIOI_GEN_Ifile_sync_query_fn, >> &ADIOI_GEN_Ifile_sync_free_fn, >> &ADIOI_GEN_Ifile_sync_cancel_fn, >> args, >> request ); >> >> /* spawn a new thread to handle the request */ >> pthread_attr_init( &attr ); >> pthread_attr_setdetachstate( &attr, PTHREAD_CREATE_DETACHED >> ); >> rc = pthread_create( &thread, &attr, >> ADIOI_GEN_Ifile_sync_thread, args ); >> pthread_attr_destroy( &attr ); >> >> /* --Error handling starts here-- */ >> //TODO: do something >> /* --End of Error handling-- */ >> return MPI_SUCCESS; >> } >> >> BTW, I just realized I forgot to free ARGS in the query function. I >> will fix that but I don't think it is causing the problem. Am I wrong? >> >> Thanks for your help, >> >> -- >> Giuseppe Congiu *·* Research Engineer II >> Seagate Technology, LLC >> office: +44 (0)23 9249 6082 <tel:%2B44%20%280%2923%209249%206082> >> *·* mobile: >> www.seagate.com <http://www.seagate.com> >> >> >> >> >> -- >> Giuseppe Congiu *·* Research Engineer II >> Seagate Technology, LLC >> office: +44 (0)23 9249 6082 *·* mobile: >> www.seagate.com <http://www.seagate.com> >> > > -- > Rob Latham > Mathematics and Computer Science Division > Argonne National Lab, IL USA > -- Giuseppe Congiu *·* Research Engineer II Seagate Technology, LLC office: +44 (0)23 9249 6082 *·* mobile: www.seagate.com

1 0

Re: [mpich-devel] Fatal error in PMPI_Barrier
by Rob Latham 04 Dec '14

04 Dec '14

On 12/04/2014 08:16 AM, Giuseppe Congiu wrote: > Hello Rob, > > I think I may have just found what the problem is: > > In ADIOI_GEN_IfileSync() I am passing args to MPI_Grequest_start() to > later get how much data has been written to global file system in the > query function and also return the error_code. The problem is that I am > freeing args in the pthread routine (which is not reported in the > previous email btw). This causes MPI_Wait() catching fire when the > pthread has already completed and args freed (which happens when I have > many segments). I fixed the code and now it seems to work : ) > > I am sorry I have bothered you with a silly problem just related to a > lack of attention from my side. Nevertheless, talking to you has helped > me finding the solution in a few mins after wandering for days. My old office mate had a little rubber duck he'd keep on his desk. When faced with a programming problem, he'd talk to "debugging ducky" and sort out the state. I'm glad I could be your debugging ducky. I am always glad to come across folks brave enough to hack on the ROMIO code. Please keep me updated with your research. ==rob > > Best Regards, > > Giuseppe > > On 4 December 2014 at 13:14, Giuseppe Congiu > <giuseppe.congiu(a)seagate.com <mailto:[email protected]>> wrote: > > > On 3 December 2014 at 21:19, Rob Latham <robl(a)mcs.anl.gov > <mailto:[email protected]>> wrote: > > Sorry not to have responded to you sooner. Between a big > conference and US thanksgiving, a lot of us were out of the > office for the last few weeks. > > > Hello Rob, I can totally understand, I have been also busy with > other stuff in the last few weeks. Nevertheless, I cannot deny I > have been looking forward for a reply : ) > > Welcome. You're digging into ROMIO and Generalized Requests, so > you've picked two fairly obscure areas of MPI to work on! > > As it happens, I am the world's expert in both ROMIO and > Generalized requests. (the population size for that domain is > exceedingly small...) > > > Then you are the blessing that I have been waiting for : ) > > I think this approach can greatly benefit collective I/O -- I > observe on some systems that the communication cost is not > hurting two-phase collective I/O but instead it is the > synchronization: if one I/O aggregator is taking longer to do > I/O than the others, all N processes pay a price waiting for the > laggard. > > > Yes, this is exactly the point. Communication cost does not seem to > be a problem, on the other hand the MPI_Allreduce() at the end of > the collective write is a real pain. By writing to local devices, > the I/O response time variation among aggregators can be greatly > reduced from what I observed. > > before we get too far, we need to talk a bit about generalized > requests and their "super-strict" progress model. Can you > share your query, free, and cancel functions? > > > The call back functions are quite simple, they don't do anything > special: > > int ADIOI_GEN_Ifile_sync_query_fn( void *extra_state, > MPI_Status *status ) > { > ARGS *state = (ARGS*)extra_state; > > MPI_Status_set_cancelled( status, 0 ); > MPI_Status_set_elements( status, MPI_BYTE, > state->bytes_xfered ); > status->MPI_SOURCE = MPI_UNDEFINED; > status->MPI_TAG = MPI_UNDEFINED; > return state->error_code; > } > > int ADIOI_GEN_Ifile_sync_free_fn( void *extra_state ) > { > return MPI_SUCCESS; > } > > int ADIOI_GEN_Ifile_sync_cancel_fn( void *extra_state, int > complete ) > { > return MPI_SUCCESS; > } > > Which part of this ends up in a generalized request? > > > The function that ends up in a generalized request is the > synchronization function. ADIOI_GEN_IfileSync(), which is also > pretty simple. It just reads the locally written file domains and > writes them to the global file. > > MPICH is going to set those hints to "automatic", but you have > overridden the defaults? (it's the right override in most > cases, so good! unless you did not, in which case we should > double check that you are not mixing vendor MPICH and your own > MPICH) > > I am setting the MPI hints using a IOR configuration file, follows a > config file snippet: > > IOR_HINT__MPI__cb_buffer_size=16777216 > IOR_HINT__MPI__cb_nodes=1 > IOR_HINT__MPI__romio_cb_read=enable > IOR_HINT__MPI__romio_cb_write=enable > IOR_HINT__MPI__local_cache=enable > IOR_HINT__MPI__local_cache_path=/tmp/ior_tmp_file > IOR_HINT__MPI__local_cache_flush_flag=flush_immediate > IOR_HINT__MPI__local_cache_discard_flag=enable > IOR_HINT__MPI__romio_no_indep_rw=true > > the collective buffer size by default is 16MB so the hint in this > case is superfluous (same goes for cb_nodes=1 since I am using only > one node). > > MPIR_Barrier_intra(169): > PSIlogger: Child with rank 5 exited with status 1. > mpid_irecv_done(101)...: read from socket failed - request > state:recv(pde)done > wait entry: 2a1ef18, 21f2790, 2a1e8e8, 21f4488 > Fatal error in PMPI_Wait: Other MPI error, error stack: > PMPI_Wait(180)..........: MPI_Wait(request=0x21f4488, > status=0x21f2138) > failed > MPIR_Wait_impl(77)......: > MPIR_Grequest_query(447): user request query function > returned error > code 1601 > > > I wonder if your query is doing more than it's supposed to do.... > > > Maybe you can tell me. I basically used Chapter 12 - External > Interfaces - (from MPI specifications published September 21, 2012) > as reference for the generalized request interface. > I just want to remind that this problem is showing itself only when > I am writing more than 4 segments in IOR. > > Does anyone have a clue of what is going wrong here? > > > We're going to need to see some more code, I think... > > > Follows the function that starts the generalized request to > synchronize the file: > > int ADIOI_GEN_IfileSync( ADIO_File fd, int count, MPI_Datatype > datatype, > int file_ptr_type, ADIO_Offset offset, > ADIO_Request *request ) > { > pthread_t thread; > pthread_attr_t attr; > int rc; > ARGS *args; > > /* copy args */ > args = (ARGS *)ADIOI_Malloc(sizeof(ARGS)); > args->fd = fd; > args->count = count; > args->datatype = datatype; > args->file_ptr_type = file_ptr_type; > args->offset = offset; > args->request = request; > > /* start generalized request */ > MPI_Grequest_start( &ADIOI_GEN_Ifile_sync_query_fn, > &ADIOI_GEN_Ifile_sync_free_fn, > &ADIOI_GEN_Ifile_sync_cancel_fn, > args, > request ); > > /* spawn a new thread to handle the request */ > pthread_attr_init( &attr ); > pthread_attr_setdetachstate( &attr, PTHREAD_CREATE_DETACHED ); > rc = pthread_create( &thread, &attr, > ADIOI_GEN_Ifile_sync_thread, args ); > pthread_attr_destroy( &attr ); > > /* --Error handling starts here-- */ > //TODO: do something > /* --End of Error handling-- */ > return MPI_SUCCESS; > } > > BTW, I just realized I forgot to free ARGS in the query function. I > will fix that but I don't think it is causing the problem. Am I wrong? > > Thanks for your help, > > -- > Giuseppe Congiu *·* Research Engineer II > Seagate Technology, LLC > office: +44 (0)23 9249 6082 <tel:%2B44%20%280%2923%209249%206082> > *·* mobile: > www.seagate.com <http://www.seagate.com> > > > > > -- > Giuseppe Congiu *·* Research Engineer II > Seagate Technology, LLC > office: +44 (0)23 9249 6082 *·* mobile: > www.seagate.com <http://www.seagate.com> -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0