On Aug 8, 2016, at 1:49 PM, Steven C. Pieper <spieper@anl.gov> wrote:
I can run a cobalt script on one node with -ppn 6 just fine. But
if I try two nodes it fails with the following in stderr:
knl08.ftm.alcf.anl.gov.148827Driver initialization failure on /dev/hfi1 (err=23)
[6] DAPL startup: RLIMIT_MEMLOCK too small
knl08.ftm.alcf.anl.gov.148832Driver initialization failure on /dev/hfi1 (err=23)
[11] DAPL startup: RLIMIT_MEMLOCK too small
knl08.ftm.alcf.anl.gov.148831Driver initialization failure on /dev/hfi1 (err=23)
knl08.ftm.alcf.anl.gov.148828Driver initialization failure on /dev/hfi1 (err=23)
[10] DAPL startup: RLIMIT_MEMLOCK too small
knl08.ftm.alcf.anl.gov.148829Driver initialization failure on /dev/hfi1 (err=23)
knl08.ftm.alcf.anl.gov.148830Driver initialization failure on /dev/hfi1 (err=23)
[7] DAPL startup: RLIMIT_MEMLOCK too small
[8] DAPL startup: RLIMIT_MEMLOCK too small
[9] DAPL startup: RLIMIT_MEMLOCK too small
Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(784)........:
MPID_Init(1332)..............: channel initialization failed
MPIDI_CH3_Init(136)..........:
MPIDI_CH3_VC_Init(256).......:
MPID_nem_vc_init(1927).......:
MPID_nem_tmi_vc_init(1107)...:
MPID_nem_tmi_vc_connect(942).:
MPID_nem_tmi_get_from_bc(889): no space for the host description
Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(784)........:
MPID_Init(1332)..............: channel initialization failed
MPIDI_CH3_Init(136)..........:
MPIDI_CH3_VC_Init(256).......:
MPID_nem_vc_init(1927).......:
MPID_nem_tmi_vc_init(1107)...:
MPID_nem_tmi_vc_connect(942).:
MPID_nem_tmi_get_from_bc(889): no space for the host description
Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(784)........:
MPID_Init(1332)..............: channel initialization failed
MPIDI_CH3_Init(136)..........:
MPIDI_CH3_VC_Init(256).......:
MPID_nem_vc_init(1927).......:
MPID_nem_tmi_vc_init(1107)...:
MPID_nem_tmi_vc_connect(942).:
MPID_nem_tmi_get_from_bc(889): no space for the host description
Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(784)........:
MPID_Init(1332)..............: channel initialization failed
MPIDI_CH3_Init(136)..........:
MPIDI_CH3_VC_Init(256).......:
MPID_nem_vc_init(1927).......:
MPID_nem_tmi_vc_init(1107)...:
MPID_nem_tmi_vc_connect(942).:
MPID_nem_tmi_get_from_bc(889): no space for the host description
Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(784)........:
MPID_Init(1332)..............: channel initialization failed
MPIDI_CH3_Init(136)..........:
MPIDI_CH3_VC_Init(256).......:
MPID_nem_vc_init(1927).......:
MPID_nem_tmi_vc_init(1107)...:
MPID_nem_tmi_vc_connect(942).:
MPID_nem_tmi_get_from_bc(889): no space for the host description
Fatal error in PMPI_Init_thread: Other MPI error, error stack:
MPIR_Init_thread(784)........:
MPID_Init(1332)..............: channel initialization failed
MPIDI_CH3_Init(136)..........:
MPIDI_CH3_VC_Init(256).......:
MPID_nem_vc_init(1927).......:
MPID_nem_tmi_vc_init(1107)...:
MPID_nem_tmi_vc_connect(942).:
MPID_nem_tmi_get_from_bc(889): no space for the host description
my script is:
#!/bin/bash
source /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh intel64
source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh
pwd
date
mpirun -f $COBALT_NODEFILE -ppn 6 -env OMP_NUM_THREADS=10 $HOME/gfmc/mc/x.mc.4he-pn he4-test-pn-03.vmc.in
The pwd and date commands are executed.
The qsub command is
qsub -n 2 -t 10 -q testingknl -M spieper@anl.gov ./doit
What is wrong?
Steve
--
Steven C. Pieper, Senior Physicist Emeritus:
spieper@anl.gov
Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439
Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100
http://www.phy.anl.gov/theory/staff/scp.html
_______________________________________________
Intel-nda mailing list
Intel-nda@lists.jlse.anl.gov
https://lists.jlse.anl.gov/mailman/listinfo/intel-nda