Steven C. Pieper, Senior Physicist Emeritus:                spieper@anl.gov 
Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439
Phone:  630-252-4232         Fax -6008        Secretary, Debra Beres, -4100
http://www.phy.anl.gov/theory/staff/scp.html

Just after 12:11:08 CDT so no, it wouldn't have been in place yet.

Fix for COBALT_NODEFILE and COBALT_JOBID in interactive jobs is now in place as well.

Ben

On Aug 9, 2016, at 12:41 PM, Steven C. Pieper <spieper@anl.gov> wrote:

I left for lunch with two jobs rebooting -- one with a 5-minute
job time and one with 60.  Both ran ok (found their node files).
The 5-minute one started running at 11:38.  Had you put in your
fix that quickly?  I'm trying the 5-minute one again.

Steve
 Steven C. Pieper, Senior Physicist Emeritus:                spieper@anl.gov

Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439
Phone:  630-252-4232         Fax -6008        Secretary, Debra Beres, -4100

http://www.phy.anl.gov/theory/staff/scp.html
On 08/09/16 12:26, Allen, Benjamin S. wrote:

This should be fixed now. I'm also rolling out a fix for including COBALT_NODEFILE and COBALT_JOBID env variables in interactive jobs in the next 10 minutes.

Let me know what else you find.

Ben

On Aug 9, 2016, at 11:49 AM, Allen, Benjamin S. <bsallen@alcf.anl.gov>
 wrote:

Ah you just found a bug in how we ordered the KNL reboot logic and adding the node file.

I'll have a fix shortly.

Thanks for testing this out,

Ben

On Aug 9, 2016, at 11:45 AM, Steven C. Pieper <spieper@anl.gov>
 wrote:

OK on the interactive jobs.  But the job that switched memory mode
was a script and it had no node file apparently.  A subsequent
job that did not switch modes worked ok.  I'm trying again with
a mode switch.  (Perhaps because the job had "exceeded"
its time the node file was removed?)

Steve
Steven C. Pieper, Senior Physicist Emeritus:
spieper@anl.gov


Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439
Phone:  630-252-4232         Fax -6008        Secretary, Debra Beres, -4100


http://www.phy.anl.gov/theory/staff/scp.html

On 08/09/16 11:41, Allen, Benjamin S. wrote:

Environment vars aren't passed or setup for you in an interactive job unfortunately. You can find the node file in /var/tmp/cobalt.<job_id> however.

This is on the list of bugs to fix, but unknown timeframe.

Ben

On Aug 9, 2016, at 11:33 AM, Steven C. Pieper <spieper@anl.gov>

wrote:

I submitted a job that changed the memory mode.  After the reboot
the job script started but my mpirun command failed with:

[

mpiexec@knl04.ftm.alcf.anl.gov

] HYDU_parse_hostfile (../../utils/args/args.c:535): unable to open host file: /var/tmp/cobalt.152
[

mpiexec@knl04.ftm.alcf.anl.gov

] mfile_fn (../../ui/mpich/utils.c:450): error parsing hostfile
[

mpiexec@knl04.ftm.alcf.anl.gov

] match_arg (../../utils/args/args.c:243): match handler returned error
[

mpiexec@knl04.ftm.alcf.anl.gov

] HYDU_parse_array (../../utils/args/args.c:269): argument matching returned error
[

mpiexec@knl04.ftm.alcf.anl.gov

] parse_args (../../ui/mpich/utils.c:4498): error parsing input array
[

mpiexec@knl04.ftm.alcf.anl.gov

] HYD_uii_mpx_get_parameters (../../ui/mpich/utils.c:4819): unable to parse user arguments
[

mpiexec@knl04.ftm.alcf.anl.gov

] HYDU_parse_hostfile (../../utils/args/args.c:535): unable to open host file: /var/tmp/cobalt.152
[

mpiexec@knl04.ftm.alcf.anl.gov

] mfile_fn (../../ui/mpich/utils.c:450): error parsing hostfile
[

mpiexec@knl04.ftm.alcf.anl.gov

] match_arg (../../utils/args/args.c:243): match handler returned error
[

mpiexec@knl04.ftm.alcf.anl.gov

] HYDU_parse_array (../../utils/args/args.c:269): argument matching returned error
[

mpiexec@knl04.ftm.alcf.anl.gov

] parse_args (../../ui/mpich/utils.c:4498): error parsing input array
[

mpiexec@knl04.ftm.alcf.anl.gov

] HYD_uii_mpx_get_parameters (../../ui/mpich/utils.c:4819): unable to parse user arguments


Then I tried to start an interactive job (without specifying any memory mode)
and I got the same nodes (knl04,05) and there is no node file:


[jlselogin1 he4] qsub -n 2 -q knl_7210 -t 60 -I
Wait for job 154 to start...
Opening interactive session to knl04
Last login: Tue Aug  9 16:24:36 2016 from jlselogin1.ftm.alcf.anl.gov
DISPLAY = 10.17.6.108:10.0 connected to knl04
ssh connection, ssh display would be:, 10.17.6.11:0.0

[knl04 ~] echo $COBALT_NODEFILE

[knl04 ~]


Steve

--
Steven C. Pieper, Senior Physicist Emeritus:


spieper@anl.gov



Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439
Phone:  630-252-4232         Fax -6008        Secretary, Debra Beres, -4100



http://www.phy.anl.gov/theory/staff/scp.html


_______________________________________________
Intel-nda mailing list


Intel-nda@lists.jlse.anl.gov
https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

_______________________________________________
Intel-nda mailing list

Intel-nda@lists.jlse.anl.gov
https://lists.jlse.anl.gov/mailman/listinfo/intel-nda