Hi Eric, 1 hour is normal for running the entire MPICH test suite. It looks like your failures are either dynamic process (e.g., spawn) or fault tolerance related tests. They should be irrelevant to the PETSc issue. Min On 2018/03/27 21:09, Eric Chamberland wrote:

Hi Min and Matthew,

In fact, I just ran an "hello word" on 2 processes and it works.  I do not have a more complicated example without PETSc since I have a Petsc-based source code...

However, I just tried to launch "make testing" into the mpich directory and it ended with some failed tests  and it was very long: about an hour.  Is it normal?

Please, see the attached file: summary.xml

Thanks,

Eric

On 27/03/18 05:04 PM, Matthew Knepley wrote:

...

On Tue, Mar 27, 2018 at 4:59 PM, Min Si <[email protected] <mailto:[email protected]>> wrote:

Hi Eric,

It will be great if you could give us a simple MPI program (not with PETSc) to reproduce this issue. If this is a problem happens only when PETSc is involved, the PETSc team can give you more suggestions.

Hi Min,

It is really easy to run PETSc at ANL. I am sure one of us can help if you cannot reproduce this bug on your own.

  Thanks,

     Matt

Thanks, Min

On 2018/03/27 15:38, Eric Chamberland wrote:

Hi,

since more than 2 weeks that the master branch of mpich is still and it can be reproduced with a simple "make test" after a fresh installation of PETSc...

Is anyone testing it?

Is it supposed to be working?

Just tell me if I should "follow" another mpich branch please.

Thanks,

Eric

On 14/03/18 03:35 AM, Eric Chamberland wrote:

Hi,

fwiw, the actual mpich/master branch doesn't passes the PETSc "make test" after a fresh installation...  It hangs just afer the 1 MPI process test, meaning it is locked into the 2 process test:

make PETSC_DIR=/pmi/cmpbib/compilation_BIB_dernier_mpich/COMPILE_AUTO/mpich-3.x-debug/petsc-3.8.3-debug PETSC_ARCH=arch-linux2-c-debug test Running test examples to verify correct installation Using PETSC_DIR=/pmi/cmpbib/compilation_BIB_dernier_mpich/COMPILE_AUTO/mpich-3.x-debug/petsc-3.8.3-debug and PETSC_ARCH=arch-linux2-c-debug C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI process

^Cmakefile:151: recipe for target 'test' failed make: [test] Interrupt (ignored)

thanks,

Eric

On 13/03/18 08:07 AM, Eric Chamberland wrote:

Hi,

each night we are testing mpich/master with our petsc-based code. I don't know if PETSc team is doing the same thing with mpich/master?   (Maybe it is a good idea?)

Everything was fine (except the issue https://github.com/pmodels/mpich/issues/2892 <https://github.com/pmodels/mpich/issues/2892>) up to commit 7b8d64debd, but since commit mpich:a8a2b30fd21), I have a segfault on a any parallel nightly test.

For example, a 2 process test ends at almost different execution points:

rank 0:

#003: /lib64/libpthread.so.0(+0xf870) [0x7f25bf908870] #004: /pmi/cmpbib/compilation_BIB_dernier_mpich/COMPILE_AUTO/BIB/bin/BIBMEFGD.opt() [0x64a788] #005: /lib64/libc.so.6(+0x35140) [0x7f25bca18140] #006: /lib64/libc.so.6(__poll+0x2d) [0x7f25bcabfbfd] #007: /opt/mpich-3.x_debug/lib/libmpi.so.0(+0x1e4cc9) [0x7f25bd90ccc9] #008: /opt/mpich-3.x_debug/lib/libmpi.so.0(+0x1ea55c) [0x7f25bd91255c] #009: /opt/mpich-3.x_debug/lib/libmpi.so.0(+0xba657) [0x7f25bd7e2657] #010: /opt/mpich-3.x_debug/lib/libmpi.so.0(PMPI_Waitall+0xe3) [0x7f25bd7e3343] #011: /opt/petsc-3.8.3_debug_mpich-3.x_debug/lib/libpetsc.so.3.8(PetscGatherMessageLengths+0x654) [0x7f25c4bb3193] #012: /opt/petsc-3.8.3_debug_mpich-3.x_debug/lib/libpetsc.so.3.8(VecScatterCreate_PtoS+0x859) [0x7f25c4e82d7f] #013: /opt/petsc-3.8.3_debug_mpich-3.x_debug/lib/libpetsc.so.3.8(VecScatterCreate+0x5684) [0x7f25c4e4d055] #014: /opt/petsc-3.8.3_debug_mpich-3.x_debug/lib/libpetsc.so.3.8(VecCreateGhostWithArray+0x688) [0x7f25c4e01a39] #015: /opt/petsc-3.8.3_debug_mpich-3.x_debug/lib/libpetsc.so.3.8(VecCreateGhost+0x179) [0x7f25c4e020f6]

rank 1:

#002: /pmi/cmpbib/compilation_BIB_dernier_mpich/COMPILE_AUTO/GIREF/lib/libgiref_opt_Util.so(traitementSignal+0x2bd0) [0x7f62df8e7310] #003: /lib64/libc.so.6(+0x35140) [0x7f62d3bc9140] #004: /lib64/libc.so.6(__poll+0x2d) [0x7f62d3c70bfd] #005: /opt/mpich-3.x_debug/lib/libmpi.so.0(+0x1e4cc9) [0x7f62d4abdcc9] #006: /opt/mpich-3.x_debug/lib/libmpi.so.0(+0x1ea55c) [0x7f62d4ac355c] #007: /opt/mpich-3.x_debug/lib/libmpi.so.0(+0x12c9c5) [0x7f62d4a059c5] #008: /opt/mpich-3.x_debug/lib/libmpi.so.0(+0x12e102) [0x7f62d4a07102] #009: /opt/mpich-3.x_debug/lib/libmpi.so.0(+0xf17a1) [0x7f62d49ca7a1] #010: /opt/mpich-3.x_debug/lib/libmpi.so.0(+0x3facf) [0x7f62d4918acf] #011: /opt/mpich-3.x_debug/lib/libmpi.so.0(+0x3fc3d) [0x7f62d4918c3d] #012: /opt/mpich-3.x_debug/lib/libmpi.so.0(+0xf18d8) [0x7f62d49ca8d8] #013: /opt/mpich-3.x_debug/lib/libmpi.so.0(+0x3fb88) [0x7f62d4918b88] #014: /opt/mpich-3.x_debug/lib/libmpi.so.0(+0x3fc3d) [0x7f62d4918c3d] #015: /opt/mpich-3.x_debug/lib/libmpi.so.0(MPI_Barrier+0x27b) [0x7f62d4918edb] #016: /opt/petsc-3.8.3_debug_mpich-3.x_debug/lib/libpetsc.so.3.8(PetscCommGetNewTag+0x3ff) [0x7f62dbceb055] #017: /opt/petsc-3.8.3_debug_mpich-3.x_debug/lib/libpetsc.so.3.8(PetscObjectGetNewTag+0x15d) [0x7f62dbceaadb] #018: /opt/petsc-3.8.3_debug_mpich-3.x_debug/lib/libpetsc.so.3.8(VecScatterCreateCommon_PtoS+0x1ee) [0x7f62dc03625c] #019: /opt/petsc-3.8.3_debug_mpich-3.x_debug/lib/libpetsc.so.3.8(VecScatterCreate_PtoS+0x29c4) [0x7f62dc035eea] #020: /opt/petsc-3.8.3_debug_mpich-3.x_debug/lib/libpetsc.so.3.8(VecScatterCreate+0x5684) [0x7f62dbffe055] #021: /opt/petsc-3.8.3_debug_mpich-3.x_debug/lib/libpetsc.so.3.8(VecCreateGhostWithArray+0x688) [0x7f62dbfb2a39] #022: /opt/petsc-3.8.3_debug_mpich-3.x_debug/lib/libpetsc.so.3.8(VecCreateGhost+0x179) [0x7f62dbfb30f6]

Have some other users (PETSc users?) reported problem?

Thanks,

Eric

ps: usual informations:

mpich logs: http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2018.03.12.05h39m54s_conf... <http://www.giref.ulaval.ca/%7Ecmpgiref/dernier_mpich/2018.03.12.05h39m54s_config.log>

http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2018.03.12.05h39m54s_conf... <http://www.giref.ulaval.ca/%7Ecmpgiref/dernier_mpich/2018.03.12.05h39m54s_config.system>

http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2018.03.12.05h39m54s_mpic... <http://www.giref.ulaval.ca/%7Ecmpgiref/dernier_mpich/2018.03.12.05h39m54s_mpich_version.txt>

http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2018.03.12.05h39m54s_c.tx... <http://www.giref.ulaval.ca/%7Ecmpgiref/dernier_mpich/2018.03.12.05h39m54s_c.txt>

http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2018.03.12.05h39m54s_m.tx... <http://www.giref.ulaval.ca/%7Ecmpgiref/dernier_mpich/2018.03.12.05h39m54s_m.txt>

http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2018.03.12.05h39m54s_mi.t... <http://www.giref.ulaval.ca/%7Ecmpgiref/dernier_mpich/2018.03.12.05h39m54s_mi.txt>

http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2018.03.12.05h39m54s_open... <http://www.giref.ulaval.ca/%7Ecmpgiref/dernier_mpich/2018.03.12.05h39m54s_openmpa_config.log>

http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2018.03.12.05h39m54s_mpl_... <http://www.giref.ulaval.ca/%7Ecmpgiref/dernier_mpich/2018.03.12.05h39m54s_mpl_config.log>

http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2018.03.12.05h39m54s_pm_h... <http://www.giref.ulaval.ca/%7Ecmpgiref/dernier_mpich/2018.03.12.05h39m54s_pm_hydra_config.log>

http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2018.03.12.05h39m54s_pm_h... <http://www.giref.ulaval.ca/%7Ecmpgiref/dernier_mpich/2018.03.12.05h39m54s_pm_hydra_tools_topo_config.log>

http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2018.03.12.05h39m54s_mpie... <http://www.giref.ulaval.ca/%7Ecmpgiref/dernier_mpich/2018.03.12.05h39m54s_mpiexec_info.txt>

Petsc logs: http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2018.03.12.05h39m54s_conf... <http://www.giref.ulaval.ca/%7Ecmpgiref/dernier_mpich/2018.03.12.05h39m54s_configure.log>

http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2018.03.12.05h39m54s_make... <http://www.giref.ulaval.ca/%7Ecmpgiref/dernier_mpich/2018.03.12.05h39m54s_make.log>

http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2018.03.12.05h39m54s_defa... <http://www.giref.ulaval.ca/%7Ecmpgiref/dernier_mpich/2018.03.12.05h39m54s_default.log>

http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2018.03.12.05h39m54s_RDic... <http://www.giref.ulaval.ca/%7Ecmpgiref/dernier_mpich/2018.03.12.05h39m54s_RDict.log>

http://www.giref.ulaval.ca/~cmpgiref/dernier_mpich/2018.03.12.05h39m54s_CMak... <http://www.giref.ulaval.ca/%7Ecmpgiref/dernier_mpich/2018.03.12.05h39m54s_CMakeLists.txt>

_______________________________________________ discuss mailing list [email protected] <mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss <https://lists.mpich.org/mailman/listinfo/discuss>

-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/%7Emk51/>

_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss