Ah you just found a bug in how we ordered the KNL reboot logic and adding the node file. I'll have a fix shortly. Thanks for testing this out, Ben

On Aug 9, 2016, at 11:45 AM, Steven C. Pieper <[email protected]> wrote:

OK on the interactive jobs. But the job that switched memory mode was a script and it had no node file apparently. A subsequent job that did not switch modes worked ok. I'm trying again with a mode switch. (Perhaps because the job had "exceeded" its time the node file was removed?)

Steve Steven C. Pieper, Senior Physicist Emeritus: [email protected]

Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100

http://www.phy.anl.gov/theory/staff/scp.html On 08/09/16 11:41, Allen, Benjamin S. wrote:

...

Environment vars aren't passed or setup for you in an interactive job unfortunately. You can find the node file in /var/tmp/cobalt.<job_id> however.

This is on the list of bugs to fix, but unknown timeframe.

Ben

...

On Aug 9, 2016, at 11:33 AM, Steven C. Pieper <[email protected]> wrote:

I submitted a job that changed the memory mode. After the reboot the job script started but my mpirun command failed with:

[ [email protected] ] HYDU_parse_hostfile (../../utils/args/args.c:535): unable to open host file: /var/tmp/cobalt.152 [ [email protected] ] mfile_fn (../../ui/mpich/utils.c:450): error parsing hostfile [ [email protected] ] match_arg (../../utils/args/args.c:243): match handler returned error [ [email protected] ] HYDU_parse_array (../../utils/args/args.c:269): argument matching returned error [ [email protected] ] parse_args (../../ui/mpich/utils.c:4498): error parsing input array [ [email protected] ] HYD_uii_mpx_get_parameters (../../ui/mpich/utils.c:4819): unable to parse user arguments [ [email protected] ] HYDU_parse_hostfile (../../utils/args/args.c:535): unable to open host file: /var/tmp/cobalt.152 [ [email protected] ] mfile_fn (../../ui/mpich/utils.c:450): error parsing hostfile [ [email protected] ] match_arg (../../utils/args/args.c:243): match handler returned error [ [email protected] ] HYDU_parse_array (../../utils/args/args.c:269): argument matching returned error [ [email protected] ] parse_args (../../ui/mpich/utils.c:4498): error parsing input array [ [email protected] ] HYD_uii_mpx_get_parameters (../../ui/mpich/utils.c:4819): unable to parse user arguments

Then I tried to start an interactive job (without specifying any memory mode) and I got the same nodes (knl04,05) and there is no node file:

[jlselogin1 he4] qsub -n 2 -q knl_7210 -t 60 -I Wait for job 154 to start... Opening interactive session to knl04 Last login: Tue Aug 9 16:24:36 2016 from jlselogin1.ftm.alcf.anl.gov DISPLAY = 10.17.6.108:10.0 connected to knl04 ssh connection, ssh display would be:, 10.17.6.11:0.0

[knl04 ~] echo $COBALT_NODEFILE

[knl04 ~]

Steve

-- Steven C. Pieper, Senior Physicist Emeritus:

[email protected]

Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100

http://www.phy.anl.gov/theory/staff/scp.html

_______________________________________________ Intel-nda mailing list

[email protected] https://lists.jlse.anl.gov/mailman/listinfo/intel-nda