I submitted a job that changed the memory mode.  After the reboot
the job script started but my mpirun command failed with:

[mpiexec@knl04.ftm.alcf.anl.gov] HYDU_parse_hostfile (../../utils/args/args.c:535): unable to open host file: /var/tmp/cobalt.152
[mpiexec@knl04.ftm.alcf.anl.gov] mfile_fn (../../ui/mpich/utils.c:450): error parsing hostfile
[mpiexec@knl04.ftm.alcf.anl.gov] match_arg (../../utils/args/args.c:243): match handler returned error
[mpiexec@knl04.ftm.alcf.anl.gov] HYDU_parse_array (../../utils/args/args.c:269): argument matching returned error
[mpiexec@knl04.ftm.alcf.anl.gov] parse_args (../../ui/mpich/utils.c:4498): error parsing input array
[mpiexec@knl04.ftm.alcf.anl.gov] HYD_uii_mpx_get_parameters (../../ui/mpich/utils.c:4819): unable to parse user arguments
[mpiexec@knl04.ftm.alcf.anl.gov] HYDU_parse_hostfile (../../utils/args/args.c:535): unable to open host file: /var/tmp/cobalt.152
[mpiexec@knl04.ftm.alcf.anl.gov] mfile_fn (../../ui/mpich/utils.c:450): error parsing hostfile
[mpiexec@knl04.ftm.alcf.anl.gov] match_arg (../../utils/args/args.c:243): match handler returned error
[mpiexec@knl04.ftm.alcf.anl.gov] HYDU_parse_array (../../utils/args/args.c:269): argument matching returned error
[mpiexec@knl04.ftm.alcf.anl.gov] parse_args (../../ui/mpich/utils.c:4498): error parsing input array
[mpiexec@knl04.ftm.alcf.anl.gov] HYD_uii_mpx_get_parameters (../../ui/mpich/utils.c:4819): unable to parse user arguments


Then I tried to start an interactive job (without specifying any memory mode)
and I got the same nodes (knl04,05) and there is no node file:


[jlselogin1 he4] qsub -n 2 -q knl_7210 -t 60 -I
Wait for job 154 to start...
Opening interactive session to knl04
Last login: Tue Aug  9 16:24:36 2016 from jlselogin1.ftm.alcf.anl.gov
DISPLAY = 10.17.6.108:10.0 connected to knl04
ssh connection, ssh display would be:, 10.17.6.11:0.0
 
[knl04 ~] echo $COBALT_NODEFILE

[knl04 ~]


Steve
-- 
Steven C. Pieper, Senior Physicist Emeritus:                spieper@anl.gov 
Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439
Phone:  630-252-4232         Fax -6008        Secretary, Debra Beres, -4100
http://www.phy.anl.gov/theory/staff/scp.html