OK, that fixed it. Thanks Steve Steven C. Pieper, Senior Physicist Emeritus: [email protected] Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 http://www.phy.anl.gov/theory/staff/scp.html On 08/08/16 14:08, Allen, Benjamin S. wrote:

Should be fixed now.

Ben

...

On Aug 8, 2016, at 2:03 PM, Allen, Benjamin S. <[email protected]> wrote:

ulimit -l is set to the default 64k evidently. Will fix shortly.

Ben

...

On Aug 8, 2016, at 1:49 PM, Steven C. Pieper <[email protected]> wrote:

I can run a cobalt script on one node with -ppn 6 just fine. But if I try two nodes it fails with the following in stderr:

knl08.ftm.alcf.anl.gov.148827Driver initialization failure on /dev/hfi1 (err=23) [6] DAPL startup: RLIMIT_MEMLOCK too small knl08.ftm.alcf.anl.gov.148832Driver initialization failure on /dev/hfi1 (err=23) [11] DAPL startup: RLIMIT_MEMLOCK too small knl08.ftm.alcf.anl.gov.148831Driver initialization failure on /dev/hfi1 (err=23) knl08.ftm.alcf.anl.gov.148828Driver initialization failure on /dev/hfi1 (err=23) [10] DAPL startup: RLIMIT_MEMLOCK too small knl08.ftm.alcf.anl.gov.148829Driver initialization failure on /dev/hfi1 (err=23) knl08.ftm.alcf.anl.gov.148830Driver initialization failure on /dev/hfi1 (err=23) [7] DAPL startup: RLIMIT_MEMLOCK too small [8] DAPL startup: RLIMIT_MEMLOCK too small [9] DAPL startup: RLIMIT_MEMLOCK too small Fatal error in PMPI_Init_thread: Other MPI error, error stack: MPIR_Init_thread(784)........: MPID_Init(1332)..............: channel initialization failed MPIDI_CH3_Init(136)..........: MPIDI_CH3_VC_Init(256).......: MPID_nem_vc_init(1927).......: MPID_nem_tmi_vc_init(1107)...: MPID_nem_tmi_vc_connect(942).: MPID_nem_tmi_get_from_bc(889): no space for the host description Fatal error in PMPI_Init_thread: Other MPI error, error stack: MPIR_Init_thread(784)........: MPID_Init(1332)..............: channel initialization failed MPIDI_CH3_Init(136)..........: MPIDI_CH3_VC_Init(256).......: MPID_nem_vc_init(1927).......: MPID_nem_tmi_vc_init(1107)...: MPID_nem_tmi_vc_connect(942).: MPID_nem_tmi_get_from_bc(889): no space for the host description Fatal error in PMPI_Init_thread: Other MPI error, error stack: MPIR_Init_thread(784)........: MPID_Init(1332)..............: channel initialization failed MPIDI_CH3_Init(136)..........: MPIDI_CH3_VC_Init(256).......: MPID_nem_vc_init(1927).......: MPID_nem_tmi_vc_init(1107)...: MPID_nem_tmi_vc_connect(942).: MPID_nem_tmi_get_from_bc(889): no space for the host description Fatal error in PMPI_Init_thread: Other MPI error, error stack: MPIR_Init_thread(784)........: MPID_Init(1332)..............: channel initialization failed MPIDI_CH3_Init(136)..........: MPIDI_CH3_VC_Init(256).......: MPID_nem_vc_init(1927).......: MPID_nem_tmi_vc_init(1107)...: MPID_nem_tmi_vc_connect(942).: MPID_nem_tmi_get_from_bc(889): no space for the host description Fatal error in PMPI_Init_thread: Other MPI error, error stack: MPIR_Init_thread(784)........: MPID_Init(1332)..............: channel initialization failed MPIDI_CH3_Init(136)..........: MPIDI_CH3_VC_Init(256).......: MPID_nem_vc_init(1927).......: MPID_nem_tmi_vc_init(1107)...: MPID_nem_tmi_vc_connect(942).: MPID_nem_tmi_get_from_bc(889): no space for the host description Fatal error in PMPI_Init_thread: Other MPI error, error stack: MPIR_Init_thread(784)........: MPID_Init(1332)..............: channel initialization failed MPIDI_CH3_Init(136)..........: MPIDI_CH3_VC_Init(256).......: MPID_nem_vc_init(1927).......: MPID_nem_tmi_vc_init(1107)...: MPID_nem_tmi_vc_connect(942).: MPID_nem_tmi_get_from_bc(889): no space for the host description

my script is:

#!/bin/bash

source /soft/compilers/intel_beta/compilers_and_libraries/linux/bin/compilervars.sh intel64 source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh

pwd date

mpirun -f $COBALT_NODEFILE -ppn 6 -env OMP_NUM_THREADS=10 $HOME/gfmc/mc/x.mc.4he-pn he4-test-pn-03.vmc.in

The pwd and date commands are executed.

The qsub command is

qsub -n 2 -t 10 -q testingknl -M [email protected] ./doit

What is wrong?

Steve

-- Steven C. Pieper, Senior Physicist Emeritus: [email protected]

Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100

http://www.phy.anl.gov/theory/staff/scp.html _______________________________________________ Intel-nda mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

---

Intel-nda mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/intel-nda