This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via c9d0fb481b235dc6e2a5f3974e12f233b03384ed (commit) via 67e1318ea71b4136a9e3b0eb77d84c7fdde3a7fa (commit) via e1ddafb8b2b7301b84e77c11918faf89a5b583d4 (commit) via 729dd94332ba2e360bfba75ab9db2a56aae19e76 (commit) via 7f1e00c834d6bf85195b587d64855eaca1a48a5e (commit) via ba0fdd3595b09f0033c849eb7d0a0167f6cf9636 (commit) via ecbe04f11e73de7505dac9b46db8849431bb5e31 (commit) from ab86ff374c430a00ee9065e8b1b95df27087f204 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/c9d0fb481b235dc6e2a5f3974e12f233b0... commit c9d0fb481b235dc6e2a5f3974e12f233b03384ed Author: Yanfei Guo <[email protected]> Date: Fri Oct 16 11:06:54 2015 -0500 maint/jenkins: add timeout script for multinode jobs We use this script to calculate the timeout for SLURM jobs based on the configuration, netmod, etc. This help improve the backfilling on SLURM. No reviewer. diff --git a/maint/jenkins/multinode/timeout.conf b/maint/jenkins/multinode/timeout.conf new file mode 100644 index 0000000..b841c05 --- /dev/null +++ b/maint/jenkins/multinode/timeout.conf @@ -0,0 +1,23 @@ +# Contitional Timeout settings +# +# Syntax (similar to a cron file): +# [job_name] [compiler] [jenkins_configure] [netmod] [n_nodes] [ppn] [timeout in min] +# Other conditions only allows exact matches. +# +# Examples: +# mxm gnu debug * * sed -i "..." +# This will apply the set the XFAIL when the job is "mpich-master-mxm" or +# "mpich-review-mxm", the compiler is "gnu", and the jenkins_configure is +# "debug". +# +# master-ubuntu * * * ubuntu32 sed -i "..." +# This will apply the set the XFAIL when the job is "mpich-master-ubuntu" and +# the running queue is "ubuntu32". +# +mpich-master-multinode * * mxm 2 * 60 +mpich-master-multinode * * mxm 4 * 60 +mpich-master-multinode * * mxm 8 * 70 +mpich-master-multinode * * tcp * * 120 +mpich-master-multinode * * ofi * * 150 +mpich-master-multinode * * portals4 * * 90 +* * * * * * 120 diff --git a/maint/jenkins/set-timeout.sh b/maint/jenkins/set-timeout.sh new file mode 100755 index 0000000..1554632 --- /dev/null +++ b/maint/jenkins/set-timeout.sh @@ -0,0 +1,102 @@ +#!/bin/zsh + +jobname="" +compiler="" +jenkins_configure="" +n_nodes="1" +ppn="1" +netmod="" + +TIMEOUT_CONF="maint/jenkins/multinode/timeout.conf" + +##################################################################### +## Initialization +##################################################################### + +while getopts ":f:j:c:o:n:p:m:" opt; do + case "$opt" in + j) + jobname=$OPTARG ;; + c) + compiler=$OPTARG ;; + o) + jenkins_configure=$OPTARG ;; + n) + n_nodes=$OPTARG ;; + p) + ppn=$OPTARG ;; + m) + netmod=$OPTARG ;; + f) + TIMEOUT_CONF=$OPTARG ;; + \?) + echo "Invalid option: -$OPTARG" >&2 + exit 1 + esac +done + +##################################################################### +## Main ( +##################################################################### + +if test ! -f "$TIMEOUT_CONF" ; then + echo "Cannot find $TIMEOUT_CONF. No TIMEOUT will be calculated" + exit 0 +fi + +TimeoutCond() { + local job="$1" + local comp="$2" + local option="$3" + local nmod="$4" + local _n_nodes="$5" + local _ppn="$6" + + local state=0 + + if [[ ! "$job" == "*" ]]; then + # clean up jobname and do substring match + if [[ ! "${jobname%%,*}" == *$job* ]]; then state=1; fi + fi + + if [[ ! "$comp" == "*" ]]; then + if [[ ! "$compiler" == "$comp" ]]; then state=1; fi + fi + + if [[ ! "$option" == "*" ]]; then + if [[ ! "$jenkins_configure" == "$option" ]]; then state=1; fi + fi + + if [[ ! "$nmod" == "*" ]]; then + if [[ ! "$netmod" == "$nmod" ]]; then state=1; fi + fi + + if [[ ! "$_n_nodes" == "*" ]]; then + if [[ ! "$n_nodes" == "$_n_nodes" ]]; then state=1; fi + fi + + if [[ ! "$_ppn" == "*" ]]; then + if [[ ! "$ppn" == "$_ppn" ]]; then state=1; fi + fi + + echo "$state" +} + +while read -r line; do + #clean leading whitespaces + line=$(echo "$line" | sed "s/^ *//g") + line=$(echo "$line" | sed "s/ *$//g") + # echo $line + # skip comment line + if test -x "$line" -o "${line:1}" = "#" ; then + continue + fi + + arr=( $(echo $line) ) + if [[ "0" == $(TimeoutCond "${arr[1]}" "${arr[2]}" "${arr[3]}" "${arr[4]}" "${arr[5]}" "${arr[6]}") ]]; then + echo "${arr[@]:6}" + exit 0 + fi +done < "$TIMEOUT_CONF" + +exit 0 http://git.mpich.org/mpich.git/commitdiff/67e1318ea71b4136a9e3b0eb77d84c7fdd... commit 67e1318ea71b4136a9e3b0eb77d84c7fdde3a7fa Author: Yanfei Guo <[email protected]> Date: Fri Oct 16 11:00:58 2015 -0500 maint/jenkins: xfail test failing on Portals 4 bug The Portals 4 reference implementation has a bug that makes recv cancel operations unreliable in threaded cases. We modified to the threaded alltoall test to consistently reproduce the failure. We have xfailed it when running with Portals 4 to not dirty the Jenkins results. Now we apply the same xfail for any job that uses portals4 netmod. No reviewer. diff --git a/maint/jenkins/xfail.conf b/maint/jenkins/xfail.conf index 119f8ae..934685b 100644 --- a/maint/jenkins/xfail.conf +++ b/maint/jenkins/xfail.conf @@ -23,4 +23,4 @@ # For each build, all applied XFAILS will be summaried in # ${SOURCE}/apply-xfail.sh # -portals4 * * * * sed -i "s+\(^alltoall .*\)+\1 xfail=ticket0+g" test/mpi/threads/pt2pt/testlist +* * * portals4 * sed -i "s+\(^alltoall .*\)+\1 xfail=ticket0+g" test/mpi/threads/pt2pt/testlist http://git.mpich.org/mpich.git/commitdiff/e1ddafb8b2b7301b84e77c11918faf89a5... commit e1ddafb8b2b7301b84e77c11918faf89a5b583d4 Author: Yanfei Guo <[email protected]> Date: Wed Oct 14 14:31:06 2015 -0500 maint/jenkins: build mpich-tarball on /sandbox Creating mpich tarball on local disk instead of NFS. No reviewer. diff --git a/maint/jenkins/test-worker-tarball.sh b/maint/jenkins/test-worker-tarball.sh index 300631b..13fa912 100755 --- a/maint/jenkins/test-worker-tarball.sh +++ b/maint/jenkins/test-worker-tarball.sh @@ -9,11 +9,25 @@ export CXX=g++ export FC=gfortran cd $WORKSPACE -git clean -x -d -f +TMP_WORKSPACE=$(mktemp -d /sandbox/jenkins.tmp.XXXXXXXX) mkdir build -cd build + +cp -a $WORKSPACE/* $TMP_WORKSPACE/ +pushd $TMP_WORKSPACE + +git clean -x -d -f + +pushd build ../maint/release.pl --branch=master --version=master --git-repo=git://git.mpich.org/mpich.git +popd + +cp -a $TMP_WORKSPACE/build/mpich-master.tar.gz $WORKSPACE/build + +popd + +rm -rf $TMP_WORKSPACE + exit 0 http://git.mpich.org/mpich.git/commitdiff/729dd94332ba2e360bfba75ab9db2a56aa... commit 729dd94332ba2e360bfba75ab9db2a56aae19e76 Author: Yanfei Guo <[email protected]> Date: Tue Oct 6 11:03:27 2015 -0400 maint/jenkins: remove MXM_SHM_KCOPY_MODE=off for testing This option is no longer needed as KMEM in MXM is already working on all nodes. No reviewer. diff --git a/maint/jenkins/test-worker.sh b/maint/jenkins/test-worker.sh index fdec609..8309576 100755 --- a/maint/jenkins/test-worker.sh +++ b/maint/jenkins/test-worker.sh @@ -503,8 +503,6 @@ case "$netmod" in "mxm") MXM_LOG_LEVEL=error export MXM_LOG_LEVEL - MXM_SHM_KCOPY_MODE=off - export MXM_SHM_KCOPY_MODE ;; "ofi" | "portals4") MXM_LOG_LEVEL=error http://git.mpich.org/mpich.git/commitdiff/7f1e00c834d6bf85195b587d64855eaca1... commit 7f1e00c834d6bf85195b587d64855eaca1a48a5e Author: Yanfei Guo <[email protected]> Date: Tue Oct 6 10:23:06 2015 -0400 maint/jenkins: remove killing for hydra_pmi_proxy The killing for remaining hydra_pmi_proxy is no longer needed as the bug has been fixed. No reviewer. diff --git a/maint/jenkins/test-worker.sh b/maint/jenkins/test-worker.sh index fdf3e9a..fdec609 100755 --- a/maint/jenkins/test-worker.sh +++ b/maint/jenkins/test-worker.sh @@ -530,10 +530,6 @@ case "$jenkins_configure" in ;; esac -if killall -9 hydra_pmi_proxy; then - echo "leftover hydra_pmi_proxy processes killed" -fi - ##################################################################### ## Copy Test results and Cleanup ##################################################################### http://git.mpich.org/mpich.git/commitdiff/ba0fdd3595b09f0033c849eb7d0a0167f6... commit ba0fdd3595b09f0033c849eb7d0a0167f6cf9636 Author: Yanfei Guo <[email protected]> Date: Tue Sep 29 11:17:19 2015 -0500 maint/jenkins: add multi-node test scripts Two scripts are added for multi-node tests. They are designed to be submitted as two consecutive jobs. mn-build.sh builds the MPI library and test binaries on local disk. The temporary workspace on local disk is packed as a tarball and saved in the Jenkins workspace. This script should be submitted as a 1-node SLURM job using `srun`. mn-run.sh recreates the temporary workspace and extracts the files from the tarball in the previous step. Then it runs all the tests and collects the results. This script should be submitted as a 2-node SLURM job using `srun`. Notes: 1. These two scripts must use the same temporary workspace location. The location of temporary workspace is specified using "-t" option. 2. Each of these two scripts should only be ran one time. When submitted through `srun`, SLURM will try to run these scripts on every allocated nodes, which is incorrect. These scripts checks the SLURM_NODEID variable and exit if it is not running on the first allocated node. 3. In order to specify interface for tcp jobs. We hack the test/mpi/runtests script to include "-iface ib0" with mpiexec. No reviewer. diff --git a/maint/jenkins/multinode/mn-build.sh b/maint/jenkins/multinode/mn-build.sh new file mode 100755 index 0000000..01cb213 --- /dev/null +++ b/maint/jenkins/multinode/mn-build.sh @@ -0,0 +1,506 @@ +#!/bin/zsh -xe + +# This script only run on the first node. The mpiexec will automatically spawned +if test ! $SLURM_NODEID -eq 0; then + exit 0 +fi + +mount | grep autotest + +hostname + +WORKSPACE="" +TMP_WORKSPACE="" +compiler="gnu" +jenkins_configure="default" +queue="ib64" +netmod="default" +ofi_prov="sockets" +N_MAKE_JOBS=8 +GIT_BRANCH="" +BUILD_MODE="per-commit" + +##################################################################### +## Initialization +##################################################################### + +while getopts ":h:c:o:q:m:n:b:t:" opt; do + case "$opt" in + h) + WORKSPACE=$OPTARG ;; + c) + compiler=$OPTARG ;; + o) + jenkins_configure=$OPTARG ;; + q) + queue=$OPTARG ;; + m) + _netmod=${OPTARG%%,*} + if [[ "$_netmod" == "ofi" ]]; then + netmod=$_netmod + ofi_prov=${OPTARG/$_netmod,} + else + netmod=$_netmod + fi + ;; + n) + N_MAKE_JOBS=$OPTARG ;; + b) + GIT_BRANCH=$OPTARG ;; + t) + TMP_WORKSPACE=$OPTARG ;; + \?) + echo "Invalid option: -$OPTARG" >&2 + exit 1 + esac +done + +cd $WORKSPACE + +if test "$GIT_BRANCH" = "" ; then + BUILD_MODE="nightly" +fi + +case "$BUILD_MODE" in + "nightly") + if [[ -x mpich-master/maint/jenkins/skip_test.sh ]]; then + ./mpich-master/maint/jenkins/skip_test.sh -j $JOB_NAME -c $compiler -o $jenkins_configure -q $queue -m $_netmod \ + -s mpich-master/test/mpi/summary.junit.xml + if [[ -f mpich-master/test/mpi/summary.junit.xml ]]; then + exit 0 + fi + fi + ;; + "per-commit") + if [[ -x maint/jenkins/skip_test.sh ]]; then + ./maint/jenkins/skip_test.sh -j $JOB_NAME -c $compiler -o $jenkins_configure -q $queue -m $_netmod \ + -s test/mpi/summary.junit.xml + if [[ -f test/mpi/summary.junit.xml ]]; then + exit 0 + fi + fi + ;; +esac + + +if test -d "$TMP_WORKSPACE"; then + rm -rf "$TMP_WORKSPACE" +fi +mkdir -p "$TMP_WORKSPACE" +SRC=$WORKSPACE +TMP_SRC=$TMP_WORKSPACE + +# Preparing the source +case "$BUILD_MODE" in + "nightly") + SRC=$WORKSPACE/mpich-master + TMP_SRC=$TMP_WORKSPACE/mpich-master + cp $WORKSPACE/mpich-master.tar.gz $TMP_WORKSPACE/ + pushd "$TMP_WORKSPACE" + tar zxvf mpich-master.tar.gz + popd + ;; + "per-commit") + git clean -x -d -f + cp -a $WORKSPACE/* $TMP_WORKSPACE/ + ;; + *) + echo "Invalid BUILD_MODE $BUILD_MODE. Set by mistake?" + exit 1 +esac + + +##################################################################### +## Functions +##################################################################### + +CollectResults() { + # TODO: copy saved test binaries (for failed cases) + if [[ "$BUILD_MODE" != "per-commit" ]]; then + find . \ + \( -name "filtered-make.txt" \ + -o -name "apply-xfail.sh" \ + -o -name "autogen.log" \ + -o -name "config.log" \ + -o -name "c.txt" \ + -o -name "m.txt" \ + -o -name "mi.txt" \ + -o -name "summary.junit.xml" \) \ + | while read -r line; do + mkdir -p "$SRC/$(dirname $line)" + done + fi + + find . \ + \( -name "filtered-make.txt" -o \ + -name "apply-xfail.sh" -o \ + -name "autogen.log" -o \ + -name "config.log" -o \ + -name "c.txt" -o \ + -name "m.txt" -o \ + -name "mi.txt" -o \ + -name "summary.junit.xml" \) \ + -exec cp {} $SRC/{} \; +} + +##################################################################### +## Logic to generate random configure options +##################################################################### +RandArgs() { + # Chosen *without* replacement. If an option is chosen twice, + # then there will be fewer options + n_choice=$1 + array=(${(P)${2}}) + optname=$3 + negoptname=$4 + chosen=() + args="" + ret_args="" + array_len=$#array + idx=0 + + for i in `seq $array_len`; do + chosen[$i]=0 + done + + for i in `seq $n_choice`; do + let idx=$[RANDOM % $array_len]+1 + if [ $chosen[$idx] -eq 1 ]; then continue; fi + chosen[$idx]=1 + args=("${(s/;/)array[$idx]}") + name=$args[1] + if [ $#args -eq 1 ]; then + # Only the name is provided. Choose one of three + # choices: + # No option (skip this one) + # just --$optname-$name + # just --$negoptname-$name + let idx=$[RANDOM % 3]+1 + if [ $idx -eq 1 ]; then + ret_args="$ret_args --$optname-$name" + elif [ $idx -eq 2 ]; then + ret_args="$ret_args --$negoptname-$name" + fi + else + let idx=$[RANDOM % ($#args-1)]+2 + # Special cases + if [ "$args[$idx]" = "ch3:sock" ]; then + ret_args="$ret_args --disable-ft-tests --disable-comm-overlap-tests" + elif [ "$args[$idx]" = "gforker" ]; then + if [ $chosen[4] -eq 1 ]; then + continue + else + ret_args="$ret_args --with-namepublisher=file" + chosen[4]=1 + fi + elif [ "$name" = "namepublisher" -a "$args[$idx]" = "no" ]; then + if [ $chosen[3] -eq 1 ]; then + continue + fi + elif [ "$args[$idx]" = "ndebug" -a "$CC" = "suncc" -a "$label" = "ubuntu32" ]; then + # On ubuntu32, suncc has a bug whose workaround is to add -O flag (ticket #2105) + CFLAGS="-O1" + export CFLAGS + fi + ret_args="$ret_args --$optname-$name=$args[$idx]" + fi + done + echo $ret_args +} + +RandConfig() { + # WARNING: If moving anything in the two following arrays, check the indices in "Special cases" above + enable_array=( + 'error-checking;no;runtime;all' + 'error-messages;all;generic;class;none' + 'timer-type;linux86_cycle;clock_gettime;gettimeofday' + 'timing;none;all;runtime;log;log_detailed' + 'g;none;all;handle;dbg;log;meminit;handlealloc;instr;mem;mutex;mutexnesting' + 'fast;O0;O1;O2;O3;ndebug;all;yes;none' + 'fortran' + 'cxx' + 'romio' + 'check-compiler-flags' + 'strict;c99;posix' + 'debuginfo' + 'weak-symbols;no;yes' + 'threads;single;multiple;runtime' + 'thread-cs;global' + 'refcount;lock-free;none' + 'mutex-timing' + 'handle-allocation;tls;mutex' + 'multi-aliases' + 'predefined-refcount' + 'alloca' + 'yield;sched_yield;select' + 'runtimevalues' + ) + with_array=( + 'logging;none' + 'pmi;simple' + 'pm;gforker' + 'namepublisher;no;file' + 'device;ch3;ch3:sock' + ) + let n_enable=$#enable_array+1 + let n_with=$#with_array+1 + enable_args=$(RandArgs $n_enable "enable_array" "enable" "disable") + with_args=$(RandArgs $n_with "with_array" "with" "without") + echo "$enable_args $with_args" +} + +PrepareEnv() { + case "$queue" in + "ubuntu32" | "ubuntu64" ) + source /software/common/adm/etc/softenv-aliases.sh + source /software/common/adm/etc/softenv-load.sh + soft add +intel + soft add +pgi + soft add +absoft + soft add +nagfor + soft add +solarisstudio-12.4 + soft add +ekopath + ;; + "freebsd64" | "freebsd32") + PATH=/usr/local/bin:$PATH + LD_LIBRARY_PATH=/usr/local/lib/gcc46:$LD_LIBRARY_PATH + export LD_LIBRARY_PATH + ;; + "solaris") + PATH=/usr/gcc/4.3/bin:/usr/gnu/bin:/usr/sbin:$PATH + CONFIG_SHELL=/usr/gnu/bin/sh + export CONFIG_SHELL + ;; + esac + PATH=$HOME/software/autotools/bin:$PATH + export PATH + echo "$PATH" +} + +SetCompiler() { + case "$compiler" in + "gnu") + CC=gcc + CXX=g++ + F77=gfortran + FC=gfortran + ;; + "clang") + CC=clang + CXX=clang++ + F77=gfortran + FC=gfortran + ;; + "intel") + CC=icc + CXX=icpc + F77=ifort + FC=ifort + ;; + "pgi") + CC=pgcc + CXX=pgcpp + F77=pgf77 + FC=pgfortran + ;; + "absoft") + CC=gcc + CXX=g++ + F77=af77 + FC=af90 + ;; + "nag") + CC=gcc + CXX=g++ + F77=nagfor + FC=nagfor + FFLAGS="-mismatch" + FCFLAGS="-mismatch" + export FFLAGS + export FCFLAGS + ;; + "solstudio") + CC=suncc + CXX=sunCC + F77=sunf77 + FC=sunf90 + ;; + "sunstudio") + CC=cc + CXX=CC + F77=f77 + FC=f90 + ;; + "pathscale") + CC=pathcc + CXX=pathCC + F77=pathf95 + FC=pathf95 + ;; + *) + echo "Unknown compiler suite" + exit 1 + esac + + export CC + export CXX + export F77 + export FC + + which $CC + which $CXX + which $F77 + which $FC +} + +SetNetmod() { + netmod_opt="__NULL__" + case "$netmod" in + "default") # for solaris, may use with sock + netmod_opt= + ;; + "mxm") + netmod_opt="--with-device=ch3:nemesis:mxm --with-mxm=$HOME/software/mellanox/mxm --disable-spawn --disable-ft-tests" + ;; + "ofi") + netmod_opt="--with-device=ch3:nemesis:ofi --with-ofi=$HOME/software/libfabric/$ofi_prov --disable-spawn --disable-ft-tests LD_LIBRARY_PATH=$HOME/software/libfabric/lib" + ;; + "portals4") + netmod_opt="--with-device=ch3:nemesis:portals4 --with-portals4=$HOME/software/portals4 --disable-spawn --disable-ft-tests" + ;; + "sock") + netmod_opt="--with-device=ch3:sock --disable-ft-tests --disable-comm-overlap-tests" + ;; + "tcp") + netmod_opt= + ;; + *) + echo "Unknown netmod type" + exit 1 + esac + export netmod_opt + echo "$netmod_opt" +} + +SetConfigOpt() { + config_opt="__TO_BE_FILLED__" + case "$jenkins_configure" in + "default") + config_opt= + ;; + "strict") + config_opt="--enable-strict" + ;; + "fast") + config_opt="--enable-fast=all" + ;; + "nofast") + config_opt="--disable-fast" + ;; + "noshared") + config_opt="--disable-shared" + ;; + "debug") + config_opt="--enable-g=all" + ;; + "noweak") + config_opt="--disable-weak-symbols" + ;; + "strictnoweak") + config_opt="--enable-strict --disable-weak-symbols" + ;; + "nofortran") + config_opt="--disable-fortran" + ;; + "nocxx") + config_opt="--disable-cxx" + ;; + "multithread") + config_opt="--enable-threads=multiple" + ;; + "debuginfo") + config_opt="--enable-debuginfo" + ;; + "noerrorchecking") + config_opt="--disable-error-checking" + ;; + "sock") # for solaris + sock + config_opt="--with-device=ch3:sock --disable-ft-tests --disable-comm-overlap-tests" + ;; + "mpd") + config_opt="--with-pm=mpd --with-namepublisher=file" + ;; + "gforker") + config_opt="--with-pm=gforker --with-namepublisher=file" + ;; + "shmem") + config_opt= + ;; + "async") + config_opt= + ;; + "random") + config_opt=$(RandArgs) + ;; + *) + echo "Bad configure option: $jenkins_configure" + exit 1 + esac + + if test "$queue" = "osx" -a "$FC" = "ifort"; then + config_opt="$config_opt lv_cv_ld_force_load=no" + fi + + # ROMIO is always disabled on multinode tests + config_opt="$config_opt --disable-romio" + + export config_opt + echo "$config_opt" +} + +##################################################################### +## Main() { Setup Environment and Build +##################################################################### +# determine if this is a nightly job or a per-commit job +PrepareEnv + +SetCompiler "$compiler" + +pushd "$TMP_SRC" + +if test "$BUILD_MODE" = "per-commit" ; then + ./autogen.sh 2>&1 | tee autogen.log +fi + +if [[ -x maint/jenkins/set-xfail.sh ]]; then + ./maint/jenkins/set-xfail.sh -j $JOB_NAME -c $compiler -o $jenkins_configure -q $queue -m $_netmod +fi + +./configure --prefix="$TMP_SRC/_inst" $(SetNetmod $netmod) $(SetConfigOpt $jenkins_configure) \ + --disable-perftest \ + 2>&1 | tee c.txt +make -j$N_MAKE_JOBS 2>&1 | tee m.txt +if test "${pipestatus[-2]}" != "0"; then + CollectResults + exit 1 +fi +make -j$N_MAKE_JOBS install 2>&1 | tee mi.txt +if test "${pipestatus[-2]}" != "0"; then + CollectResults + exit 1 +fi +cat m.txt mi.txt | ./maint/clmake > filtered-make.txt 2>&1 + +# Make the binaries of tests before rsync back to NFS +pushd test/mpi +make -j$N_MAKE_JOBS +popd + +tar cf $TMP_WORKSPACE/mpich-test-pack.tar * +cp $TMP_WORKSPACE/mpich-test-pack.tar $WORKSPACE +rm $TMP_WORKSPACE/mpich-test-pack.tar +popd + +rm -rf $TMP_WORKSPACE +exit 0 + diff --git a/maint/jenkins/multinode/mn-run.sh b/maint/jenkins/multinode/mn-run.sh new file mode 100755 index 0000000..3b6a12b --- /dev/null +++ b/maint/jenkins/multinode/mn-run.sh @@ -0,0 +1,240 @@ +#!/bin/zsh -xe + +# This script only run on the first node. The mpiexec will automatically spawned +if test ! $SLURM_NODEID -eq 0; then + exit 0 +fi + +hostname + +WORKSPACE="" +TMP_WORKSPACE="" +compiler="gnu" +jenkins_configure="default" +queue="ib64" +netmod="default" +ofi_prov="sockets" +N_MAKE_JOBS=8 +GIT_BRANCH="" +BUILD_MODE="per-commit" + +##################################################################### +## Initialization +##################################################################### + +while getopts ":h:c:o:q:m:n:b:t:" opt; do + case "$opt" in + h) + WORKSPACE=$OPTARG ;; + c) + compiler=$OPTARG ;; + o) + jenkins_configure=$OPTARG ;; + q) + queue=$OPTARG ;; + m) + _netmod=${OPTARG%%,*} + if [[ "$_netmod" == "ofi" ]]; then + netmod=$_netmod + ofi_prov=${OPTARG/$_netmod,} + else + netmod=$_netmod + fi + ;; + n) + N_MAKE_JOBS=$OPTARG ;; + b) + GIT_BRANCH=$OPTARG ;; + t) + TMP_WORKSPACE=$OPTARG ;; + \?) + echo "Invalid option: -$OPTARG" >&2 + exit 1 + esac +done + +cd $WORKSPACE + +if test "$GIT_BRANCH" = "" ; then + BUILD_MODE="nightly" +fi + +case "$BUILD_MODE" in + "nightly") + if [[ -x mpich-master/maint/jenkins/skip_test.sh ]]; then + ./mpich-master/maint/jenkins/skip_test.sh -j $JOB_NAME -c $compiler -o $jenkins_configure -q $queue -m $_netmod \ + -s mpich-master/test/mpi/summary.junit.xml + if [[ -f mpich-master/test/mpi/summary.junit.xml ]]; then + exit 0 + fi + fi + ;; + "per-commit") + if [[ -x maint/jenkins/skip_test.sh ]]; then + ./maint/jenkins/skip_test.sh -j $JOB_NAME -c $compiler -o $jenkins_configure -q $queue -m $_netmod \ + -s test/mpi/summary.junit.xml + if [[ -f test/mpi/summary.junit.xml ]]; then + exit 0 + fi + fi + ;; +esac + + +if test -d "$TMP_WORKSPACE"; then + rm -rf "$TMP_WORKSPACE" +fi +SRC=$WORKSPACE +TMP_SRC=$TMP_WORKSPACE + +# Preparing the source +case "$BUILD_MODE" in + "nightly") + SRC=$WORKSPACE/mpich-master + TMP_SRC=$TMP_WORKSPACE/mpich-master + ;; + "per-commit") + ;; + *) + echo "Invalid BUILD_MODE $BUILD_MODE. Set by mistake?" + exit 1 +esac + +srun mkdir -p $TMP_SRC + +# distribute source and binary +pushd $WORKSPACE +srun cp $WORKSPACE/mpich-test-pack.tar $TMP_SRC +srun tar xf $TMP_SRC/mpich-test-pack.tar -C $TMP_SRC +popd + + +##################################################################### +## Functions +##################################################################### + +CollectResults() { + # TODO: copy saved test binaries (for failed cases) + if [[ "$BUILD_MODE" != "per-commit" ]]; then + find . \ + \( -name "filtered-make.txt" \ + -o -name "apply-xfail.sh" \ + -o -name "autogen.log" \ + -o -name "config.log" \ + -o -name "c.txt" \ + -o -name "m.txt" \ + -o -name "mi.txt" \ + -o -name "summary.junit.xml" \) \ + | while read -r line; do + mkdir -p "$SRC/$(dirname $line)" + done + fi + + find . \ + \( -name "filtered-make.txt" -o \ + -name "apply-xfail.sh" -o \ + -name "autogen.log" -o \ + -name "config.log" -o \ + -name "c.txt" -o \ + -name "m.txt" -o \ + -name "mi.txt" -o \ + -name "summary.junit.xml" \) \ + -exec cp {} $SRC/{} \; +} + +PrepareEnv() { + case "$queue" in + "ubuntu32" | "ubuntu64" ) + source /software/common/adm/etc/softenv-aliases.sh + source /software/common/adm/etc/softenv-load.sh + soft add +intel + soft add +pgi + soft add +absoft + soft add +nagfor + soft add +solarisstudio-12.4 + soft add +ekopath + ;; + "freebsd64" | "freebsd32") + PATH=/usr/local/bin:$PATH + LD_LIBRARY_PATH=/usr/local/lib/gcc46:$LD_LIBRARY_PATH + export LD_LIBRARY_PATH + ;; + "solaris") + PATH=/usr/gcc/4.3/bin:/usr/gnu/bin:/usr/sbin:$PATH + CONFIG_SHELL=/usr/gnu/bin/sh + export CONFIG_SHELL + ;; + esac + PATH=$HOME/software/autotools/bin:$PATH + export PATH + echo "$PATH" +} + + +##################################################################### +## Main() { Run tests +## Multi-node tests are running from Jenkins workspace +##################################################################### + +pushd "$TMP_SRC" + +# Preparation +case "$jenkins_configure" in + "mpd") + $TMP_SRC/_inst/bin/mpd & + sleep 1 + ;; + "async") + MPIR_CVAR_ASYNC_PROGRESS=1 + export MPIR_CVAR_ASYNC_PROGRESS + ;; + "multithread") + MPIR_CVAR_DEFAULT_THREAD_LEVEL=MPI_THREAD_MULTIPLE + export MPIR_CVAR_DEFAULT_THREAD_LEVEL + ;; +esac + +case "$netmod" in + "mxm") + MXM_LOG_LEVEL=error + export MXM_LOG_LEVEL + ;; + "ofi" | "portals4") + MXM_LOG_LEVEL=error + export MXM_LOG_LEVEL + ;; +esac + +# run only the minimum level of datatype tests when it is per-commit job +if [[ "$BUILD_MODE" = "per-commit" ]]; then + MPITEST_DATATYPE_TEST_LEVEL=min + export MPITEST_DATATYPE_TEST_LEVEL +fi + +# hack for passing -iface ib0 +if test "$netmod" = "tcp"; then + sed -i 's+/bin/mpiexec+/bin/mpiexec -iface ib0+g' test/mpi/runtests +fi + +make testing + +# Cleanup +case "$jenkins_configure" in + "mpd") + $TMP_SRC/_inst/bin/mpdallexit + ;; + "async") + unset MPIR_CVAR_ASYNC_PROGRESS + ;; +esac + +##################################################################### +## Copy Test results and Cleanup +##################################################################### + +CollectResults + +popd +srun rm -rf $TMP_WORKSPACE +exit 0 + http://git.mpich.org/mpich.git/commitdiff/ecbe04f11e73de7505dac9b46db8849431... commit ecbe04f11e73de7505dac9b46db8849431bb5e31 Author: Yanfei Guo <[email protected]> Date: Mon Sep 28 13:22:04 2015 -0500 maint/jenkins: print hostname in test worker scripts The hostname of the node that runs the test is printed at the beginning of the test job. No reviewer. diff --git a/maint/jenkins/test-worker-abi-prolog.sh b/maint/jenkins/test-worker-abi-prolog.sh index 52dd32f..e9e7146 100755 --- a/maint/jenkins/test-worker-abi-prolog.sh +++ b/maint/jenkins/test-worker-abi-prolog.sh @@ -1,5 +1,7 @@ #!/bin/zsh -xe +hostname + WORKSPACE="" compiler="gnu" jenkins_configure="default" diff --git a/maint/jenkins/test-worker-abi.sh b/maint/jenkins/test-worker-abi.sh index 1722558..4538e90 100755 --- a/maint/jenkins/test-worker-abi.sh +++ b/maint/jenkins/test-worker-abi.sh @@ -1,5 +1,7 @@ #!/bin/zsh -x +hostname + WORKSPACE="" build_mpi="mpich-master" compiler="gnu" diff --git a/maint/jenkins/test-worker-armci.sh b/maint/jenkins/test-worker-armci.sh index d11164c..a073864 100755 --- a/maint/jenkins/test-worker-armci.sh +++ b/maint/jenkins/test-worker-armci.sh @@ -1,5 +1,7 @@ #!/bin/zsh -xe +hostname + WORKSPACE="" GIT_BRANCH="" BUILD_MODE="per-commit" diff --git a/maint/jenkins/test-worker-tarball.sh b/maint/jenkins/test-worker-tarball.sh index ee443a6..300631b 100755 --- a/maint/jenkins/test-worker-tarball.sh +++ b/maint/jenkins/test-worker-tarball.sh @@ -1,5 +1,7 @@ #!/bin/zsh -xe +hostname + export PATH=$HOME/software/autotools/bin:$HOME/software/sowing/bin:$PATH export DOCTEXT_PATH=$HOME/software/sowing/share/doctext export CC=gcc diff --git a/maint/jenkins/test-worker.sh b/maint/jenkins/test-worker.sh index 2a0e115..fdf3e9a 100755 --- a/maint/jenkins/test-worker.sh +++ b/maint/jenkins/test-worker.sh @@ -1,5 +1,7 @@ #!/bin/zsh -xe +hostname + WORKSPACE="" compiler="gnu" jenkins_configure="default" ----------------------------------------------------------------------- Summary of changes: .../{test-worker.sh => multinode/mn-build.sh} | 94 +++------ maint/jenkins/multinode/mn-run.sh | 240 ++++++++++++++++++++ maint/jenkins/multinode/timeout.conf | 23 ++ maint/jenkins/{set-xfail.sh => set-timeout.sh} | 50 ++-- maint/jenkins/test-worker-abi-prolog.sh | 2 + maint/jenkins/test-worker-abi.sh | 2 + maint/jenkins/test-worker-armci.sh | 2 + maint/jenkins/test-worker-tarball.sh | 20 ++- maint/jenkins/test-worker.sh | 8 +- maint/jenkins/xfail.conf | 2 +- 10 files changed, 343 insertions(+), 100 deletions(-) copy maint/jenkins/{test-worker.sh => multinode/mn-build.sh} (88%) create mode 100755 maint/jenkins/multinode/mn-run.sh create mode 100644 maint/jenkins/multinode/timeout.conf copy maint/jenkins/{set-xfail.sh => set-timeout.sh} (66%) hooks/post-receive -- MPICH primary repository