August 2016 - Intel-nda - mailman3.cels.anl.gov

Re: [JLSE-Intel-NDA] no nodefile
by Steven C. Pieper 09 Aug '16

09 Aug '16

OK, somethings are meant to be mysteries. It works for me now. Also an interactive job got a nodefile. Thanks, Steve Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 http://www.phy.anl.gov/theory/staff/scp.html On 08/09/16 12:45, Allen, Benjamin S. wrote: > Just after 12:11:08 CDT so no, it wouldn't have been in place yet. > > Fix for COBALT_NODEFILE and COBALT_JOBID in interactive jobs is now in place as well. > > Ben > >> On Aug 9, 2016, at 12:41 PM, Steven C. Pieper <spieper(a)anl.gov> wrote: >> >> I left for lunch with two jobs rebooting -- one with a 5-minute >> job time and one with 60. Both ran ok (found their node files). >> The 5-minute one started running at 11:38. Had you put in your >> fix that quickly? I'm trying the 5-minute one again. >> >> Steve >> Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov >> >> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 >> >> http://www.phy.anl.gov/theory/staff/scp.html >> On 08/09/16 12:26, Allen, Benjamin S. wrote: >>> This should be fixed now. I'm also rolling out a fix for including COBALT_NODEFILE and COBALT_JOBID env variables in interactive jobs in the next 10 minutes. >>> >>> Let me know what else you find. >>> >>> Ben >>> >>> >>>> On Aug 9, 2016, at 11:49 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >>>> wrote: >>>> >>>> Ah you just found a bug in how we ordered the KNL reboot logic and adding the node file. >>>> >>>> I'll have a fix shortly. >>>> >>>> Thanks for testing this out, >>>> >>>> Ben >>>> >>>> >>>> >>>>> On Aug 9, 2016, at 11:45 AM, Steven C. Pieper <spieper(a)anl.gov> >>>>> wrote: >>>>> >>>>> OK on the interactive jobs. But the job that switched memory mode >>>>> was a script and it had no node file apparently. A subsequent >>>>> job that did not switch modes worked ok. I'm trying again with >>>>> a mode switch. (Perhaps because the job had "exceeded" >>>>> its time the node file was removed?) >>>>> >>>>> Steve >>>>> Steven C. Pieper, Senior Physicist Emeritus: >>>>> spieper(a)anl.gov >>>>> >>>>> >>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >>>>> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 >>>>> >>>>> >>>>> http://www.phy.anl.gov/theory/staff/scp.html >>>>> >>>>> On 08/09/16 11:41, Allen, Benjamin S. wrote: >>>>> >>>>>> Environment vars aren't passed or setup for you in an interactive job unfortunately. You can find the node file in /var/tmp/cobalt.<job_id> however. >>>>>> >>>>>> This is on the list of bugs to fix, but unknown timeframe. >>>>>> >>>>>> Ben >>>>>> >>>>>> >>>>>> >>>>>>> On Aug 9, 2016, at 11:33 AM, Steven C. Pieper <spieper(a)anl.gov> >>>>>>> >>>>>>> wrote: >>>>>>> >>>>>>> I submitted a job that changed the memory mode. After the reboot >>>>>>> the job script started but my mpirun command failed with: >>>>>>> >>>>>>> [ >>>>>>> >>>>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>>>> >>>>>>> ] HYDU_parse_hostfile (../../utils/args/args.c:535): unable to open host file: /var/tmp/cobalt.152 >>>>>>> [ >>>>>>> >>>>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>>>> >>>>>>> ] mfile_fn (../../ui/mpich/utils.c:450): error parsing hostfile >>>>>>> [ >>>>>>> >>>>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>>>> >>>>>>> ] match_arg (../../utils/args/args.c:243): match handler returned error >>>>>>> [ >>>>>>> >>>>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>>>> >>>>>>> ] HYDU_parse_array (../../utils/args/args.c:269): argument matching returned error >>>>>>> [ >>>>>>> >>>>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>>>> >>>>>>> ] parse_args (../../ui/mpich/utils.c:4498): error parsing input array >>>>>>> [ >>>>>>> >>>>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>>>> >>>>>>> ] HYD_uii_mpx_get_parameters (../../ui/mpich/utils.c:4819): unable to parse user arguments >>>>>>> [ >>>>>>> >>>>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>>>> >>>>>>> ] HYDU_parse_hostfile (../../utils/args/args.c:535): unable to open host file: /var/tmp/cobalt.152 >>>>>>> [ >>>>>>> >>>>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>>>> >>>>>>> ] mfile_fn (../../ui/mpich/utils.c:450): error parsing hostfile >>>>>>> [ >>>>>>> >>>>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>>>> >>>>>>> ] match_arg (../../utils/args/args.c:243): match handler returned error >>>>>>> [ >>>>>>> >>>>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>>>> >>>>>>> ] HYDU_parse_array (../../utils/args/args.c:269): argument matching returned error >>>>>>> [ >>>>>>> >>>>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>>>> >>>>>>> ] parse_args (../../ui/mpich/utils.c:4498): error parsing input array >>>>>>> [ >>>>>>> >>>>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>>>> >>>>>>> ] HYD_uii_mpx_get_parameters (../../ui/mpich/utils.c:4819): unable to parse user arguments >>>>>>> >>>>>>> >>>>>>> Then I tried to start an interactive job (without specifying any memory mode) >>>>>>> and I got the same nodes (knl04,05) and there is no node file: >>>>>>> >>>>>>> >>>>>>> [jlselogin1 he4] qsub -n 2 -q knl_7210 -t 60 -I >>>>>>> Wait for job 154 to start... >>>>>>> Opening interactive session to knl04 >>>>>>> Last login: Tue Aug 9 16:24:36 2016 from jlselogin1.ftm.alcf.anl.gov >>>>>>> DISPLAY = 10.17.6.108:10.0 connected to knl04 >>>>>>> ssh connection, ssh display would be:, 10.17.6.11:0.0 >>>>>>> >>>>>>> [knl04 ~] echo $COBALT_NODEFILE >>>>>>> >>>>>>> [knl04 ~] >>>>>>> >>>>>>> >>>>>>> Steve >>>>>>> >>>>>>> -- >>>>>>> Steven C. Pieper, Senior Physicist Emeritus: >>>>>>> >>>>>>> >>>>>>> spieper(a)anl.gov >>>>>>> >>>>>>> >>>>>>> >>>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >>>>>>> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 >>>>>>> >>>>>>> >>>>>>> >>>>>>> http://www.phy.anl.gov/theory/staff/scp.html >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Intel-nda mailing list >>>>>>> >>>>>>> >>>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>> _______________________________________________ >>>> Intel-nda mailing list >>>> >>>> Intel-nda(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] no nodefile
by Allen, Benjamin S. 09 Aug '16

09 Aug '16

Just after 12:11:08 CDT so no, it wouldn't have been in place yet. Fix for COBALT_NODEFILE and COBALT_JOBID in interactive jobs is now in place as well. Ben > On Aug 9, 2016, at 12:41 PM, Steven C. Pieper <spieper(a)anl.gov> wrote: > > I left for lunch with two jobs rebooting -- one with a 5-minute > job time and one with 60. Both ran ok (found their node files). > The 5-minute one started running at 11:38. Had you put in your > fix that quickly? I'm trying the 5-minute one again. > > Steve > Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov > > Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 > Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 > > http://www.phy.anl.gov/theory/staff/scp.html > On 08/09/16 12:26, Allen, Benjamin S. wrote: >> This should be fixed now. I'm also rolling out a fix for including COBALT_NODEFILE and COBALT_JOBID env variables in interactive jobs in the next 10 minutes. >> >> Let me know what else you find. >> >> Ben >> >> >>> On Aug 9, 2016, at 11:49 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >>> wrote: >>> >>> Ah you just found a bug in how we ordered the KNL reboot logic and adding the node file. >>> >>> I'll have a fix shortly. >>> >>> Thanks for testing this out, >>> >>> Ben >>> >>> >>> >>>> On Aug 9, 2016, at 11:45 AM, Steven C. Pieper <spieper(a)anl.gov> >>>> wrote: >>>> >>>> OK on the interactive jobs. But the job that switched memory mode >>>> was a script and it had no node file apparently. A subsequent >>>> job that did not switch modes worked ok. I'm trying again with >>>> a mode switch. (Perhaps because the job had "exceeded" >>>> its time the node file was removed?) >>>> >>>> Steve >>>> Steven C. Pieper, Senior Physicist Emeritus: >>>> spieper(a)anl.gov >>>> >>>> >>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >>>> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 >>>> >>>> >>>> http://www.phy.anl.gov/theory/staff/scp.html >>>> >>>> On 08/09/16 11:41, Allen, Benjamin S. wrote: >>>> >>>>> Environment vars aren't passed or setup for you in an interactive job unfortunately. You can find the node file in /var/tmp/cobalt.<job_id> however. >>>>> >>>>> This is on the list of bugs to fix, but unknown timeframe. >>>>> >>>>> Ben >>>>> >>>>> >>>>> >>>>>> On Aug 9, 2016, at 11:33 AM, Steven C. Pieper <spieper(a)anl.gov> >>>>>> >>>>>> wrote: >>>>>> >>>>>> I submitted a job that changed the memory mode. After the reboot >>>>>> the job script started but my mpirun command failed with: >>>>>> >>>>>> [ >>>>>> >>>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>>> >>>>>> ] HYDU_parse_hostfile (../../utils/args/args.c:535): unable to open host file: /var/tmp/cobalt.152 >>>>>> [ >>>>>> >>>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>>> >>>>>> ] mfile_fn (../../ui/mpich/utils.c:450): error parsing hostfile >>>>>> [ >>>>>> >>>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>>> >>>>>> ] match_arg (../../utils/args/args.c:243): match handler returned error >>>>>> [ >>>>>> >>>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>>> >>>>>> ] HYDU_parse_array (../../utils/args/args.c:269): argument matching returned error >>>>>> [ >>>>>> >>>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>>> >>>>>> ] parse_args (../../ui/mpich/utils.c:4498): error parsing input array >>>>>> [ >>>>>> >>>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>>> >>>>>> ] HYD_uii_mpx_get_parameters (../../ui/mpich/utils.c:4819): unable to parse user arguments >>>>>> [ >>>>>> >>>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>>> >>>>>> ] HYDU_parse_hostfile (../../utils/args/args.c:535): unable to open host file: /var/tmp/cobalt.152 >>>>>> [ >>>>>> >>>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>>> >>>>>> ] mfile_fn (../../ui/mpich/utils.c:450): error parsing hostfile >>>>>> [ >>>>>> >>>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>>> >>>>>> ] match_arg (../../utils/args/args.c:243): match handler returned error >>>>>> [ >>>>>> >>>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>>> >>>>>> ] HYDU_parse_array (../../utils/args/args.c:269): argument matching returned error >>>>>> [ >>>>>> >>>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>>> >>>>>> ] parse_args (../../ui/mpich/utils.c:4498): error parsing input array >>>>>> [ >>>>>> >>>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>>> >>>>>> ] HYD_uii_mpx_get_parameters (../../ui/mpich/utils.c:4819): unable to parse user arguments >>>>>> >>>>>> >>>>>> Then I tried to start an interactive job (without specifying any memory mode) >>>>>> and I got the same nodes (knl04,05) and there is no node file: >>>>>> >>>>>> >>>>>> [jlselogin1 he4] qsub -n 2 -q knl_7210 -t 60 -I >>>>>> Wait for job 154 to start... >>>>>> Opening interactive session to knl04 >>>>>> Last login: Tue Aug 9 16:24:36 2016 from jlselogin1.ftm.alcf.anl.gov >>>>>> DISPLAY = 10.17.6.108:10.0 connected to knl04 >>>>>> ssh connection, ssh display would be:, 10.17.6.11:0.0 >>>>>> >>>>>> [knl04 ~] echo $COBALT_NODEFILE >>>>>> >>>>>> [knl04 ~] >>>>>> >>>>>> >>>>>> Steve >>>>>> >>>>>> -- >>>>>> Steven C. Pieper, Senior Physicist Emeritus: >>>>>> >>>>>> >>>>>> spieper(a)anl.gov >>>>>> >>>>>> >>>>>> >>>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >>>>>> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 >>>>>> >>>>>> >>>>>> >>>>>> http://www.phy.anl.gov/theory/staff/scp.html >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Intel-nda mailing list >>>>>> >>>>>> >>>>>> Intel-nda(a)lists.jlse.anl.gov >>>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>> _______________________________________________ >>> Intel-nda mailing list >>> >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] no nodefile
by Steven C. Pieper 09 Aug '16

09 Aug '16

I left for lunch with two jobs rebooting -- one with a 5-minute job time and one with 60. Both ran ok (found their node files). The 5-minute one started running at 11:38. Had you put in your fix that quickly? I'm trying the 5-minute one again. Steve Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 http://www.phy.anl.gov/theory/staff/scp.html On 08/09/16 12:26, Allen, Benjamin S. wrote: > This should be fixed now. I'm also rolling out a fix for including COBALT_NODEFILE and COBALT_JOBID env variables in interactive jobs in the next 10 minutes. > > Let me know what else you find. > > Ben > >> On Aug 9, 2016, at 11:49 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >> >> Ah you just found a bug in how we ordered the KNL reboot logic and adding the node file. >> >> I'll have a fix shortly. >> >> Thanks for testing this out, >> >> Ben >> >> >>> On Aug 9, 2016, at 11:45 AM, Steven C. Pieper <spieper(a)anl.gov> wrote: >>> >>> OK on the interactive jobs. But the job that switched memory mode >>> was a script and it had no node file apparently. A subsequent >>> job that did not switch modes worked ok. I'm trying again with >>> a mode switch. (Perhaps because the job had "exceeded" >>> its time the node file was removed?) >>> >>> Steve >>> Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov >>> >>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >>> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 >>> >>> http://www.phy.anl.gov/theory/staff/scp.html >>> On 08/09/16 11:41, Allen, Benjamin S. wrote: >>>> Environment vars aren't passed or setup for you in an interactive job unfortunately. You can find the node file in /var/tmp/cobalt.<job_id> however. >>>> >>>> This is on the list of bugs to fix, but unknown timeframe. >>>> >>>> Ben >>>> >>>> >>>>> On Aug 9, 2016, at 11:33 AM, Steven C. Pieper <spieper(a)anl.gov> >>>>> wrote: >>>>> >>>>> I submitted a job that changed the memory mode. After the reboot >>>>> the job script started but my mpirun command failed with: >>>>> >>>>> [ >>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>> ] HYDU_parse_hostfile (../../utils/args/args.c:535): unable to open host file: /var/tmp/cobalt.152 >>>>> [ >>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>> ] mfile_fn (../../ui/mpich/utils.c:450): error parsing hostfile >>>>> [ >>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>> ] match_arg (../../utils/args/args.c:243): match handler returned error >>>>> [ >>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>> ] HYDU_parse_array (../../utils/args/args.c:269): argument matching returned error >>>>> [ >>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>> ] parse_args (../../ui/mpich/utils.c:4498): error parsing input array >>>>> [ >>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>> ] HYD_uii_mpx_get_parameters (../../ui/mpich/utils.c:4819): unable to parse user arguments >>>>> [ >>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>> ] HYDU_parse_hostfile (../../utils/args/args.c:535): unable to open host file: /var/tmp/cobalt.152 >>>>> [ >>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>> ] mfile_fn (../../ui/mpich/utils.c:450): error parsing hostfile >>>>> [ >>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>> ] match_arg (../../utils/args/args.c:243): match handler returned error >>>>> [ >>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>> ] HYDU_parse_array (../../utils/args/args.c:269): argument matching returned error >>>>> [ >>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>> ] parse_args (../../ui/mpich/utils.c:4498): error parsing input array >>>>> [ >>>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>>> ] HYD_uii_mpx_get_parameters (../../ui/mpich/utils.c:4819): unable to parse user arguments >>>>> >>>>> >>>>> Then I tried to start an interactive job (without specifying any memory mode) >>>>> and I got the same nodes (knl04,05) and there is no node file: >>>>> >>>>> >>>>> [jlselogin1 he4] qsub -n 2 -q knl_7210 -t 60 -I >>>>> Wait for job 154 to start... >>>>> Opening interactive session to knl04 >>>>> Last login: Tue Aug 9 16:24:36 2016 from jlselogin1.ftm.alcf.anl.gov >>>>> DISPLAY = 10.17.6.108:10.0 connected to knl04 >>>>> ssh connection, ssh display would be:, 10.17.6.11:0.0 >>>>> >>>>> [knl04 ~] echo $COBALT_NODEFILE >>>>> >>>>> [knl04 ~] >>>>> >>>>> >>>>> Steve >>>>> >>>>> -- >>>>> Steven C. Pieper, Senior Physicist Emeritus: >>>>> >>>>> spieper(a)anl.gov >>>>> >>>>> >>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >>>>> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 >>>>> >>>>> >>>>> http://www.phy.anl.gov/theory/staff/scp.html >>>>> >>>>> _______________________________________________ >>>>> Intel-nda mailing list >>>>> >>>>> Intel-nda(a)lists.jlse.anl.gov >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] no nodefile
by Allen, Benjamin S. 09 Aug '16

09 Aug '16

This should be fixed now. I'm also rolling out a fix for including COBALT_NODEFILE and COBALT_JOBID env variables in interactive jobs in the next 10 minutes. Let me know what else you find. Ben > On Aug 9, 2016, at 11:49 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > Ah you just found a bug in how we ordered the KNL reboot logic and adding the node file. > > I'll have a fix shortly. > > Thanks for testing this out, > > Ben > > >> On Aug 9, 2016, at 11:45 AM, Steven C. Pieper <spieper(a)anl.gov> wrote: >> >> OK on the interactive jobs. But the job that switched memory mode >> was a script and it had no node file apparently. A subsequent >> job that did not switch modes worked ok. I'm trying again with >> a mode switch. (Perhaps because the job had "exceeded" >> its time the node file was removed?) >> >> Steve >> Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov >> >> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 >> >> http://www.phy.anl.gov/theory/staff/scp.html >> On 08/09/16 11:41, Allen, Benjamin S. wrote: >>> Environment vars aren't passed or setup for you in an interactive job unfortunately. You can find the node file in /var/tmp/cobalt.<job_id> however. >>> >>> This is on the list of bugs to fix, but unknown timeframe. >>> >>> Ben >>> >>> >>>> On Aug 9, 2016, at 11:33 AM, Steven C. Pieper <spieper(a)anl.gov> >>>> wrote: >>>> >>>> I submitted a job that changed the memory mode. After the reboot >>>> the job script started but my mpirun command failed with: >>>> >>>> [ >>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>> ] HYDU_parse_hostfile (../../utils/args/args.c:535): unable to open host file: /var/tmp/cobalt.152 >>>> [ >>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>> ] mfile_fn (../../ui/mpich/utils.c:450): error parsing hostfile >>>> [ >>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>> ] match_arg (../../utils/args/args.c:243): match handler returned error >>>> [ >>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>> ] HYDU_parse_array (../../utils/args/args.c:269): argument matching returned error >>>> [ >>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>> ] parse_args (../../ui/mpich/utils.c:4498): error parsing input array >>>> [ >>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>> ] HYD_uii_mpx_get_parameters (../../ui/mpich/utils.c:4819): unable to parse user arguments >>>> [ >>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>> ] HYDU_parse_hostfile (../../utils/args/args.c:535): unable to open host file: /var/tmp/cobalt.152 >>>> [ >>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>> ] mfile_fn (../../ui/mpich/utils.c:450): error parsing hostfile >>>> [ >>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>> ] match_arg (../../utils/args/args.c:243): match handler returned error >>>> [ >>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>> ] HYDU_parse_array (../../utils/args/args.c:269): argument matching returned error >>>> [ >>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>> ] parse_args (../../ui/mpich/utils.c:4498): error parsing input array >>>> [ >>>> mpiexec(a)knl04.ftm.alcf.anl.gov >>>> ] HYD_uii_mpx_get_parameters (../../ui/mpich/utils.c:4819): unable to parse user arguments >>>> >>>> >>>> Then I tried to start an interactive job (without specifying any memory mode) >>>> and I got the same nodes (knl04,05) and there is no node file: >>>> >>>> >>>> [jlselogin1 he4] qsub -n 2 -q knl_7210 -t 60 -I >>>> Wait for job 154 to start... >>>> Opening interactive session to knl04 >>>> Last login: Tue Aug 9 16:24:36 2016 from jlselogin1.ftm.alcf.anl.gov >>>> DISPLAY = 10.17.6.108:10.0 connected to knl04 >>>> ssh connection, ssh display would be:, 10.17.6.11:0.0 >>>> >>>> [knl04 ~] echo $COBALT_NODEFILE >>>> >>>> [knl04 ~] >>>> >>>> >>>> Steve >>>> >>>> -- >>>> Steven C. Pieper, Senior Physicist Emeritus: >>>> >>>> spieper(a)anl.gov >>>> >>>> >>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >>>> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 >>>> >>>> >>>> http://www.phy.anl.gov/theory/staff/scp.html >>>> >>>> _______________________________________________ >>>> Intel-nda mailing list >>>> >>>> Intel-nda(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] no nodefile
by Allen, Benjamin S. 09 Aug '16

09 Aug '16

Ah you just found a bug in how we ordered the KNL reboot logic and adding the node file. I'll have a fix shortly. Thanks for testing this out, Ben > On Aug 9, 2016, at 11:45 AM, Steven C. Pieper <spieper(a)anl.gov> wrote: > > OK on the interactive jobs. But the job that switched memory mode > was a script and it had no node file apparently. A subsequent > job that did not switch modes worked ok. I'm trying again with > a mode switch. (Perhaps because the job had "exceeded" > its time the node file was removed?) > > Steve > Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov > > Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 > Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 > > http://www.phy.anl.gov/theory/staff/scp.html > On 08/09/16 11:41, Allen, Benjamin S. wrote: >> Environment vars aren't passed or setup for you in an interactive job unfortunately. You can find the node file in /var/tmp/cobalt.<job_id> however. >> >> This is on the list of bugs to fix, but unknown timeframe. >> >> Ben >> >> >>> On Aug 9, 2016, at 11:33 AM, Steven C. Pieper <spieper(a)anl.gov> >>> wrote: >>> >>> I submitted a job that changed the memory mode. After the reboot >>> the job script started but my mpirun command failed with: >>> >>> [ >>> mpiexec(a)knl04.ftm.alcf.anl.gov >>> ] HYDU_parse_hostfile (../../utils/args/args.c:535): unable to open host file: /var/tmp/cobalt.152 >>> [ >>> mpiexec(a)knl04.ftm.alcf.anl.gov >>> ] mfile_fn (../../ui/mpich/utils.c:450): error parsing hostfile >>> [ >>> mpiexec(a)knl04.ftm.alcf.anl.gov >>> ] match_arg (../../utils/args/args.c:243): match handler returned error >>> [ >>> mpiexec(a)knl04.ftm.alcf.anl.gov >>> ] HYDU_parse_array (../../utils/args/args.c:269): argument matching returned error >>> [ >>> mpiexec(a)knl04.ftm.alcf.anl.gov >>> ] parse_args (../../ui/mpich/utils.c:4498): error parsing input array >>> [ >>> mpiexec(a)knl04.ftm.alcf.anl.gov >>> ] HYD_uii_mpx_get_parameters (../../ui/mpich/utils.c:4819): unable to parse user arguments >>> [ >>> mpiexec(a)knl04.ftm.alcf.anl.gov >>> ] HYDU_parse_hostfile (../../utils/args/args.c:535): unable to open host file: /var/tmp/cobalt.152 >>> [ >>> mpiexec(a)knl04.ftm.alcf.anl.gov >>> ] mfile_fn (../../ui/mpich/utils.c:450): error parsing hostfile >>> [ >>> mpiexec(a)knl04.ftm.alcf.anl.gov >>> ] match_arg (../../utils/args/args.c:243): match handler returned error >>> [ >>> mpiexec(a)knl04.ftm.alcf.anl.gov >>> ] HYDU_parse_array (../../utils/args/args.c:269): argument matching returned error >>> [ >>> mpiexec(a)knl04.ftm.alcf.anl.gov >>> ] parse_args (../../ui/mpich/utils.c:4498): error parsing input array >>> [ >>> mpiexec(a)knl04.ftm.alcf.anl.gov >>> ] HYD_uii_mpx_get_parameters (../../ui/mpich/utils.c:4819): unable to parse user arguments >>> >>> >>> Then I tried to start an interactive job (without specifying any memory mode) >>> and I got the same nodes (knl04,05) and there is no node file: >>> >>> >>> [jlselogin1 he4] qsub -n 2 -q knl_7210 -t 60 -I >>> Wait for job 154 to start... >>> Opening interactive session to knl04 >>> Last login: Tue Aug 9 16:24:36 2016 from jlselogin1.ftm.alcf.anl.gov >>> DISPLAY = 10.17.6.108:10.0 connected to knl04 >>> ssh connection, ssh display would be:, 10.17.6.11:0.0 >>> >>> [knl04 ~] echo $COBALT_NODEFILE >>> >>> [knl04 ~] >>> >>> >>> Steve >>> >>> -- >>> Steven C. Pieper, Senior Physicist Emeritus: >>> >>> spieper(a)anl.gov >>> >>> >>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >>> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 >>> >>> >>> http://www.phy.anl.gov/theory/staff/scp.html >>> >>> _______________________________________________ >>> Intel-nda mailing list >>> >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] no nodefile
by Steven C. Pieper 09 Aug '16

09 Aug '16

OK on the interactive jobs. But the job that switched memory mode was a script and it had no node file apparently. A subsequent job that did not switch modes worked ok. I'm trying again with a mode switch. (Perhaps because the job had "exceeded" its time the node file was removed?) Steve Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 http://www.phy.anl.gov/theory/staff/scp.html On 08/09/16 11:41, Allen, Benjamin S. wrote: > Environment vars aren't passed or setup for you in an interactive job unfortunately. You can find the node file in /var/tmp/cobalt.<job_id> however. > > This is on the list of bugs to fix, but unknown timeframe. > > Ben > >> On Aug 9, 2016, at 11:33 AM, Steven C. Pieper <spieper(a)anl.gov> wrote: >> >> I submitted a job that changed the memory mode. After the reboot >> the job script started but my mpirun command failed with: >> >> [mpiexec(a)knl04.ftm.alcf.anl.gov] HYDU_parse_hostfile (../../utils/args/args.c:535): unable to open host file: /var/tmp/cobalt.152 >> [mpiexec(a)knl04.ftm.alcf.anl.gov] mfile_fn (../../ui/mpich/utils.c:450): error parsing hostfile >> [mpiexec(a)knl04.ftm.alcf.anl.gov] match_arg (../../utils/args/args.c:243): match handler returned error >> [mpiexec(a)knl04.ftm.alcf.anl.gov] HYDU_parse_array (../../utils/args/args.c:269): argument matching returned error >> [mpiexec(a)knl04.ftm.alcf.anl.gov] parse_args (../../ui/mpich/utils.c:4498): error parsing input array >> [mpiexec(a)knl04.ftm.alcf.anl.gov] HYD_uii_mpx_get_parameters (../../ui/mpich/utils.c:4819): unable to parse user arguments >> [mpiexec(a)knl04.ftm.alcf.anl.gov] HYDU_parse_hostfile (../../utils/args/args.c:535): unable to open host file: /var/tmp/cobalt.152 >> [mpiexec(a)knl04.ftm.alcf.anl.gov] mfile_fn (../../ui/mpich/utils.c:450): error parsing hostfile >> [mpiexec(a)knl04.ftm.alcf.anl.gov] match_arg (../../utils/args/args.c:243): match handler returned error >> [mpiexec(a)knl04.ftm.alcf.anl.gov] HYDU_parse_array (../../utils/args/args.c:269): argument matching returned error >> [mpiexec(a)knl04.ftm.alcf.anl.gov] parse_args (../../ui/mpich/utils.c:4498): error parsing input array >> [mpiexec(a)knl04.ftm.alcf.anl.gov] HYD_uii_mpx_get_parameters (../../ui/mpich/utils.c:4819): unable to parse user arguments >> >> >> Then I tried to start an interactive job (without specifying any memory mode) >> and I got the same nodes (knl04,05) and there is no node file: >> >> >> [jlselogin1 he4] qsub -n 2 -q knl_7210 -t 60 -I >> Wait for job 154 to start... >> Opening interactive session to knl04 >> Last login: Tue Aug 9 16:24:36 2016 from jlselogin1.ftm.alcf.anl.gov >> DISPLAY = 10.17.6.108:10.0 connected to knl04 >> ssh connection, ssh display would be:, 10.17.6.11:0.0 >> >> [knl04 ~] echo $COBALT_NODEFILE >> >> [knl04 ~] >> >> >> Steve >> >> -- >> Steven C. Pieper, Senior Physicist Emeritus: >> spieper(a)anl.gov >> >> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 >> >> http://www.phy.anl.gov/theory/staff/scp.html >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] no nodefile
by Allen, Benjamin S. 09 Aug '16

09 Aug '16

Environment vars aren't passed or setup for you in an interactive job unfortunately. You can find the node file in /var/tmp/cobalt.<job_id> however. This is on the list of bugs to fix, but unknown timeframe. Ben > On Aug 9, 2016, at 11:33 AM, Steven C. Pieper <spieper(a)anl.gov> wrote: > > I submitted a job that changed the memory mode. After the reboot > the job script started but my mpirun command failed with: > > [mpiexec(a)knl04.ftm.alcf.anl.gov] HYDU_parse_hostfile (../../utils/args/args.c:535): unable to open host file: /var/tmp/cobalt.152 > [mpiexec(a)knl04.ftm.alcf.anl.gov] mfile_fn (../../ui/mpich/utils.c:450): error parsing hostfile > [mpiexec(a)knl04.ftm.alcf.anl.gov] match_arg (../../utils/args/args.c:243): match handler returned error > [mpiexec(a)knl04.ftm.alcf.anl.gov] HYDU_parse_array (../../utils/args/args.c:269): argument matching returned error > [mpiexec(a)knl04.ftm.alcf.anl.gov] parse_args (../../ui/mpich/utils.c:4498): error parsing input array > [mpiexec(a)knl04.ftm.alcf.anl.gov] HYD_uii_mpx_get_parameters (../../ui/mpich/utils.c:4819): unable to parse user arguments > [mpiexec(a)knl04.ftm.alcf.anl.gov] HYDU_parse_hostfile (../../utils/args/args.c:535): unable to open host file: /var/tmp/cobalt.152 > [mpiexec(a)knl04.ftm.alcf.anl.gov] mfile_fn (../../ui/mpich/utils.c:450): error parsing hostfile > [mpiexec(a)knl04.ftm.alcf.anl.gov] match_arg (../../utils/args/args.c:243): match handler returned error > [mpiexec(a)knl04.ftm.alcf.anl.gov] HYDU_parse_array (../../utils/args/args.c:269): argument matching returned error > [mpiexec(a)knl04.ftm.alcf.anl.gov] parse_args (../../ui/mpich/utils.c:4498): error parsing input array > [mpiexec(a)knl04.ftm.alcf.anl.gov] HYD_uii_mpx_get_parameters (../../ui/mpich/utils.c:4819): unable to parse user arguments > > > Then I tried to start an interactive job (without specifying any memory mode) > and I got the same nodes (knl04,05) and there is no node file: > > > [jlselogin1 he4] qsub -n 2 -q knl_7210 -t 60 -I > Wait for job 154 to start... > Opening interactive session to knl04 > Last login: Tue Aug 9 16:24:36 2016 from jlselogin1.ftm.alcf.anl.gov > DISPLAY = 10.17.6.108:10.0 connected to knl04 > ssh connection, ssh display would be:, 10.17.6.11:0.0 > > [knl04 ~] echo $COBALT_NODEFILE > > [knl04 ~] > > > Steve > > -- > Steven C. Pieper, Senior Physicist Emeritus: > spieper(a)anl.gov > > Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 > Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 > > http://www.phy.anl.gov/theory/staff/scp.html > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] questions about queues
by Allen, Benjamin S. 09 Aug '16

09 Aug '16

It has no way to kill the job and cleanup after it when the nodes are in the middle of a reboot. It should wait until the nodes come back up, and kill the job if needed. I would think the job state would change from starting however if it was being killed. Looking at the logs for job 152, it never attempted to kill the job. I suspect the way we're running the node setup scripts (i.e. job_prescript in cobalt.conf), is acting as intended and the time isn't being counted against the job's walltime, even though its being reported in RunTime and incorrectly in TimeRemaining. You could test this via submitting a short job, which causes a reboot with a sleep 1200 in it or similar. Ben > On Aug 9, 2016, at 11:19 AM, Steven C. Pieper <spieper(a)anl.gov> wrote: > > I just submitted my first job that changes the knl_memory_mode. > qstat is counting the reboot time as runtime, but it doesn't kill > the job when it runs out of time: > > JobID JobName User WallTime QueuedTime RunTime TimeRemaining Nodes State Location Mode Procs Preemptable Queue StartTime Index > ========================================================================================================================================================================================================= > 146 N/A chunduri 01:00:00 00:00:02 00:50:34 00:09:25 1 running knl08 interactive 1 False knl_7210 Tue Aug 09 10:26:41 2016 -0500 (CDT) None > 152 he4-test-pn-10.vmc b21541 00:05:00 00:00:10 00:04:29 00:00:30 2 starting [knl04-05] script 2 False knl_7210 Tue Aug 09 11:12:46 2016 -0500 (CDT) None > > JobID JobName User WallTime QueuedTime RunTime TimeRemaining Nodes State Location Mode Procs Preemptable Queue StartTime Index > ========================================================================================================================================================================================================= > 146 N/A chunduri 01:00:00 00:00:02 00:52:20 00:07:39 1 running knl08 interactive 1 False knl_7210 Tue Aug 09 10:26:41 2016 -0500 (CDT) None > 152 he4-test-pn-10.vmc b21541 00:05:00 00:00:10 00:06:16 00:00:00 2 starting [knl04-05] script 2 False knl_7210 Tue Aug 09 11:12:46 2016 -0500 (CDT) None > > > Steve > > > > Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov > > Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 > Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 > > http://www.phy.anl.gov/theory/staff/scp.html > On 08/09/16 11:15, Allen, Benjamin S. wrote: >> This should be fixed. Time was off on the logins. >> >> Ben >> >> >>> On Aug 9, 2016, at 10:56 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >>> wrote: >>> >>> Also I'll look into the odd time related issues in queue time and run time. >>> >>> Ben >>> >>> >>>> On Aug 9, 2016, at 10:53 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> >>>> wrote: >>>> >>>> Hi Steve, >>>> >>>> As of yesterday we're no longer testing Cobalt, so we've moved to the knl_7210 and knl_7250 queues. These queues are documented in >>>> https://wiki.jlse.anl.gov/display/knl/Intel+KNL >>>> . >>>> >>>> I'll remove the testingknl queue shortly to avoid confusion. >>>> >>>> Ben >>>> >>>> >>>>> On Aug 9, 2016, at 10:16 AM, Steven C. Pieper <spieper(a)anl.gov> >>>>> wrote: >>>>> >>>>> 1) The testingknl queue is stopped. I found that I can use knl_7210. >>>>> Should I not do this? When will testingknl be started. >>>>> >>>>> >>>>> Name Users Groups MinTime MaxTime MaxRunning MaxQueued MaxUserNodes MaxNodeHours TotalNodes State >>>>> ===================================================================================================================== >>>>> knl_7210 None intel None None None None 9 None None running >>>>> knl_7250 None intel None None None None None None None running >>>>> testingknl None intel None None None None None None None stopped >>>>> >>>>> >>>>> 2) The times are messed up: >>>>> >>>>> JobID JobName User WallTime QueuedTime RunTime TimeRemaining Nodes State Location Mode Procs Preemptable Queue StartTime Index >>>>> ===================================================================================================================================================================================================== >>>>> 136 N/A lecompte 01:00:00 00:00:09 01:12:26 00:00:00 1 running knl08 script 1 False knl_7210 Tue Aug 09 09:59:24 2016 -0500 (CDT) None >>>>> 137 N/A lecompte 01:00:00 00:00:03 01:11:45 00:00:00 1 running knl04 script 1 False knl_7210 Tue Aug 09 10:00:05 2016 -0500 (CDT) None >>>>> 138 N/A lecompte 01:00:00 00:00:03 01:11:15 00:00:00 1 running knl05 script 1 False knl_7210 Tue Aug 09 10:00:36 2016 -0500 (CDT) None >>>>> 139 N/A lecompte 01:00:00 00:00:02 01:10:54 00:00:00 1 running knl06 script 1 False knl_7210 Tue Aug 09 10:00:56 2016 -0500 (CDT) None >>>>> 140 N/A lecompte 01:00:00 00:00:09 01:10:23 00:00:00 1 running knl07 script 1 False knl_7210 Tue Aug 09 10:01:27 2016 -0500 (CDT) None >>>>> 143 he4-test-pn-06b.vmc b21541 00:05:00 01:04:31 N/A N/A 1 queued None script 1 False testingknl N/A None >>>>> 144 he4-test-pn-06b.vmc b21541 00:05:00 00:00:08 01:00:44 00:00:00 1 starting knl00 script 1 False knl_7210 Tue Aug 09 10:11:06 2016 -0500 (CDT) None >>>>> >>>>> >>>>> Job 143 was submitted only 12 minutes ago but it shows up as being in the >>>>> queue 1h 4m. Lecompte's jobs appear to have exceeded their time but >>>>> are still running. >>>>> >>>>> Steve >>>>> >>>>> >>>>> >>>>> -- >>>>> Steven C. Pieper, Senior Physicist Emeritus: >>>>> >>>>> spieper(a)anl.gov >>>>> >>>>> >>>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >>>>> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 >>>>> >>>>> >>>>> http://www.phy.anl.gov/theory/staff/scp.html >>>>> >>>>> _______________________________________________ >>>>> Intel-nda mailing list >>>>> >>>>> Intel-nda(a)lists.jlse.anl.gov >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>> _______________________________________________ >>>> Intel-nda mailing list >>>> >>>> Intel-nda(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>> _______________________________________________ >>> Intel-nda mailing list >>> >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

[JLSE-Intel-NDA] no nodefile
by Steven C. Pieper 09 Aug '16

09 Aug '16

I submitted a job that changed the memory mode. After the reboot the job script started but my mpirun command failed with: [mpiexec(a)knl04.ftm.alcf.anl.gov] HYDU_parse_hostfile (../../utils/args/args.c:535): unable to open host file: /var/tmp/cobalt.152 [mpiexec(a)knl04.ftm.alcf.anl.gov] mfile_fn (../../ui/mpich/utils.c:450): error parsing hostfile [mpiexec(a)knl04.ftm.alcf.anl.gov] match_arg (../../utils/args/args.c:243): match handler returned error [mpiexec(a)knl04.ftm.alcf.anl.gov] HYDU_parse_array (../../utils/args/args.c:269): argument matching returned error [mpiexec(a)knl04.ftm.alcf.anl.gov] parse_args (../../ui/mpich/utils.c:4498): error parsing input array [mpiexec(a)knl04.ftm.alcf.anl.gov] HYD_uii_mpx_get_parameters (../../ui/mpich/utils.c:4819): unable to parse user arguments [mpiexec(a)knl04.ftm.alcf.anl.gov] HYDU_parse_hostfile (../../utils/args/args.c:535): unable to open host file: /var/tmp/cobalt.152 [mpiexec(a)knl04.ftm.alcf.anl.gov] mfile_fn (../../ui/mpich/utils.c:450): error parsing hostfile [mpiexec(a)knl04.ftm.alcf.anl.gov] match_arg (../../utils/args/args.c:243): match handler returned error [mpiexec(a)knl04.ftm.alcf.anl.gov] HYDU_parse_array (../../utils/args/args.c:269): argument matching returned error [mpiexec(a)knl04.ftm.alcf.anl.gov] parse_args (../../ui/mpich/utils.c:4498): error parsing input array [mpiexec(a)knl04.ftm.alcf.anl.gov] HYD_uii_mpx_get_parameters (../../ui/mpich/utils.c:4819): unable to parse user arguments Then I tried to start an interactive job (without specifying any memory mode) and I got the same nodes (knl04,05) and there is no node file: [jlselogin1 he4] qsub -n 2 -q knl_7210 -t 60 -I Wait for job 154 to start... Opening interactive session to knl04 Last login: Tue Aug 9 16:24:36 2016 from jlselogin1.ftm.alcf.anl.gov DISPLAY = 10.17.6.108:10.0 connected to knl04 ssh connection, ssh display would be:, 10.17.6.11:0.0 [knl04 ~] echo $COBALT_NODEFILE [knl04 ~] Steve -- Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 http://www.phy.anl.gov/theory/staff/scp.html

1 0

Re: [JLSE-Intel-NDA] questions about queues
by Steven C. Pieper 09 Aug '16

09 Aug '16

I just submitted my first job that changes the knl_memory_mode. qstat is counting the reboot time as runtime, but it doesn't kill the job when it runs out of time: JobID JobName User WallTime QueuedTime RunTime TimeRemaining Nodes State Location Mode Procs Preemptable Queue StartTime Index ========================================================================================================================================================================================================= 146 N/A chunduri 01:00:00 00:00:02 00:50:34 00:09:25 1 running knl08 interactive 1 False knl_7210 Tue Aug 09 10:26:41 2016 -0500 (CDT) None 152 he4-test-pn-10.vmc b21541 00:05:00 00:00:10 00:04:29 00:00:30 2 starting [knl04-05] script 2 False knl_7210 Tue Aug 09 11:12:46 2016 -0500 (CDT) None JobID JobName User WallTime QueuedTime RunTime TimeRemaining Nodes State Location Mode Procs Preemptable Queue StartTime Index ========================================================================================================================================================================================================= 146 N/A chunduri 01:00:00 00:00:02 00:52:20 00:07:39 1 running knl08 interactive 1 False knl_7210 Tue Aug 09 10:26:41 2016 -0500 (CDT) None 152 he4-test-pn-10.vmc b21541 00:05:00 00:00:10 00:06:16 00:00:00 2 starting [knl04-05] script 2 False knl_7210 Tue Aug 09 11:12:46 2016 -0500 (CDT) None Steve Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 http://www.phy.anl.gov/theory/staff/scp.html On 08/09/16 11:15, Allen, Benjamin S. wrote: > This should be fixed. Time was off on the logins. > > Ben > >> On Aug 9, 2016, at 10:56 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >> >> Also I'll look into the odd time related issues in queue time and run time. >> >> Ben >> >>> On Aug 9, 2016, at 10:53 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >>> >>> Hi Steve, >>> >>> As of yesterday we're no longer testing Cobalt, so we've moved to the knl_7210 and knl_7250 queues. These queues are documented in https://wiki.jlse.anl.gov/display/knl/Intel+KNL. >>> >>> I'll remove the testingknl queue shortly to avoid confusion. >>> >>> Ben >>> >>>> On Aug 9, 2016, at 10:16 AM, Steven C. Pieper <spieper(a)anl.gov> wrote: >>>> >>>> 1) The testingknl queue is stopped. I found that I can use knl_7210. >>>> Should I not do this? When will testingknl be started. >>>> >>>> >>>> Name Users Groups MinTime MaxTime MaxRunning MaxQueued MaxUserNodes MaxNodeHours TotalNodes State >>>> ===================================================================================================================== >>>> knl_7210 None intel None None None None 9 None None running >>>> knl_7250 None intel None None None None None None None running >>>> testingknl None intel None None None None None None None stopped >>>> >>>> >>>> 2) The times are messed up: >>>> >>>> JobID JobName User WallTime QueuedTime RunTime TimeRemaining Nodes State Location Mode Procs Preemptable Queue StartTime Index >>>> ===================================================================================================================================================================================================== >>>> 136 N/A lecompte 01:00:00 00:00:09 01:12:26 00:00:00 1 running knl08 script 1 False knl_7210 Tue Aug 09 09:59:24 2016 -0500 (CDT) None >>>> 137 N/A lecompte 01:00:00 00:00:03 01:11:45 00:00:00 1 running knl04 script 1 False knl_7210 Tue Aug 09 10:00:05 2016 -0500 (CDT) None >>>> 138 N/A lecompte 01:00:00 00:00:03 01:11:15 00:00:00 1 running knl05 script 1 False knl_7210 Tue Aug 09 10:00:36 2016 -0500 (CDT) None >>>> 139 N/A lecompte 01:00:00 00:00:02 01:10:54 00:00:00 1 running knl06 script 1 False knl_7210 Tue Aug 09 10:00:56 2016 -0500 (CDT) None >>>> 140 N/A lecompte 01:00:00 00:00:09 01:10:23 00:00:00 1 running knl07 script 1 False knl_7210 Tue Aug 09 10:01:27 2016 -0500 (CDT) None >>>> 143 he4-test-pn-06b.vmc b21541 00:05:00 01:04:31 N/A N/A 1 queued None script 1 False testingknl N/A None >>>> 144 he4-test-pn-06b.vmc b21541 00:05:00 00:00:08 01:00:44 00:00:00 1 starting knl00 script 1 False knl_7210 Tue Aug 09 10:11:06 2016 -0500 (CDT) None >>>> >>>> >>>> Job 143 was submitted only 12 minutes ago but it shows up as being in the >>>> queue 1h 4m. Lecompte's jobs appear to have exceeded their time but >>>> are still running. >>>> >>>> Steve >>>> >>>> >>>> >>>> -- >>>> Steven C. Pieper, Senior Physicist Emeritus: >>>> spieper(a)anl.gov >>>> >>>> Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 >>>> Phone: 630-252-4232 Fax -6008 Secretary, Debra Beres, -4100 >>>> >>>> http://www.phy.anl.gov/theory/staff/scp.html >>>> _______________________________________________ >>>> Intel-nda mailing list >>>> Intel-nda(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0