December 2016 - Intel-nda - mailman3.cels.anl.gov

Re: [JLSE-Intel-NDA] Omni-Path network down?
by Allen, Benjamin S. 13 Dec '16

13 Dec '16

Hi Paul, I had to restart the embedded subnet manager on the OmniPath switch. It appears to be passing traffic again. Thanks for the heads up, Ben > On Dec 12, 2016, at 6:15 PM, Paul Hargrove <phhargrove(a)lbl.gov> wrote: > > As a developer of network middleware, part of my interest in the KNL nodes at JLSE is the Omni-Path network. > > I had been successfully using both pshm2 and librabric (psm2 provider) on the KNLs. > However, sometime between Oct 11 and Dec 3 things ceased to work as they had. > Those are the dates of the latest successful run and earliest fail run. > Sorry that I don't have anything between those dates. > > Both psm2 and libfabric based programs are now failing with messages like: > knl08.ftm.alcf.anl.gov.223177PSM found 0 available contexts on opa device(s). (err=21) > > Additionally, /usr/sbin/opainfo says: > {hargrove@knl04 ~}$ /usr/sbin/opainfo > oib_utils ERROR: [23478] oib_open_port: can't open UMAD port (hfi1_0:1) > opainfo: Unable to open hfi:port 0:1 > (Though, in all honesty, I don't know if opainfo ever worked before). > > -Paul > > -- > Paul H. Hargrove PHHargrove(a)lbl.gov > Computer Languages & Systems Software (CLaSS) Group > Computer Science Department Tel: +1-510-495-2352 > Lawrence Berkeley National Laboratory Fax: +1-510-486-6900 > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] Omni-Path network down?
by Paul Hargrove 12 Dec '16

12 Dec '16

As a developer of network middleware, part of my interest in the KNL nodes at JLSE is the Omni-Path network. I had been successfully using both pshm2 and librabric (psm2 provider) on the KNLs. However, sometime between Oct 11 and Dec 3 things ceased to work as they had. Those are the dates of the latest successful run and earliest fail run. Sorry that I don't have anything between those dates. Both psm2 and libfabric based programs are now failing with messages like: knl08.ftm.alcf.anl.gov.223177PSM found 0 available contexts on opa device(s). (err=21) Additionally, /usr/sbin/opainfo says: {hargrove@knl04 ~}$ /usr/sbin/opainfo oib_utils ERROR: [23478] oib_open_port: can't open UMAD port (hfi1_0:1) opainfo: Unable to open hfi:port 0:1 (Though, in all honesty, I don't know if opainfo ever worked before). -Paul -- Paul H. Hargrove PHHargrove(a)lbl.gov Computer Languages & Systems Software (CLaSS) Group Computer Science Department Tel: +1-510-495-2352 Lawrence Berkeley National Laboratory Fax: +1-510-486-6900

1 0

Re: [JLSE-Intel-NDA] no nodefile
by Paul Romano 07 Dec '16

07 Dec '16

When I run an interactive job, I don't seem to be getting either the COBALT_NODEFILE or COBALT_JOBID environment variables. Has something been changed since that capability was implemented? Paul On Tue, Aug 9, 2016 at 12:54 PM, Pieper, Steven C. <spieper(a)anl.gov> wrote: > OK, somethings are meant to be mysteries. It works for me now. > Also an interactive job got a nodefile. > > Thanks, > Steve > > Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov > Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 > Phone: 630-252-4232 <(630)%20252-4232> Fax -6008 Secretary, Debra Beres, -4100http://www.phy.anl.gov/theory/staff/scp.html > > On 08/09/16 12:45, Allen, Benjamin S. wrote: > > Just after 12:11:08 CDT so no, it wouldn't have been in place yet. > > Fix for COBALT_NODEFILE and COBALT_JOBID in interactive jobs is now in place as well. > > Ben > > > On Aug 9, 2016, at 12:41 PM, Steven C. Pieper <spieper(a)anl.gov> <spieper(a)anl.gov> wrote: > > I left for lunch with two jobs rebooting -- one with a 5-minute > job time and one with 60. Both ran ok (found their node files). > The 5-minute one started running at 11:38. Had you put in your > fix that quickly? I'm trying the 5-minute one again. > > Steve > Steven C. Pieper, Senior Physicist Emeritus: spieper(a)anl.gov > > Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 > Phone: 630-252-4232 <(630)%20252-4232> Fax -6008 Secretary, Debra Beres, -4100 > http://www.phy.anl.gov/theory/staff/scp.html > On 08/09/16 12:26, Allen, Benjamin S. wrote: > > This should be fixed now. I'm also rolling out a fix for including COBALT_NODEFILE and COBALT_JOBID env variables in interactive jobs in the next 10 minutes. > > Let me know what else you find. > > Ben > > > > On Aug 9, 2016, at 11:49 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> <bsallen(a)alcf.anl.gov> > wrote: > > Ah you just found a bug in how we ordered the KNL reboot logic and adding the node file. > > I'll have a fix shortly. > > Thanks for testing this out, > > Ben > > > > > On Aug 9, 2016, at 11:45 AM, Steven C. Pieper <spieper(a)anl.gov> <spieper(a)anl.gov> > wrote: > > OK on the interactive jobs. But the job that switched memory mode > was a script and it had no node file apparently. A subsequent > job that did not switch modes worked ok. I'm trying again with > a mode switch. (Perhaps because the job had "exceeded" > its time the node file was removed?) > > Steve > Steven C. Pieper, Senior Physicist Emeritus:[email protected] > > > Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 > Phone: 630-252-4232 <(630)%20252-4232> Fax -6008 Secretary, Debra Beres, -4100 > > http://www.phy.anl.gov/theory/staff/scp.html > > On 08/09/16 11:41, Allen, Benjamin S. wrote: > > > Environment vars aren't passed or setup for you in an interactive job unfortunately. You can find the node file in /var/tmp/cobalt.<job_id> however. > > This is on the list of bugs to fix, but unknown timeframe. > > Ben > > > > > On Aug 9, 2016, at 11:33 AM, Steven C. Pieper <spieper(a)anl.gov> <spieper(a)anl.gov> > > wrote: > > I submitted a job that changed the memory mode. After the reboot > the job script started but my mpirun command failed with: > > [ > mpiexec(a)knl04.ftm.alcf.anl.gov > > ] HYDU_parse_hostfile (../../utils/args/args.c:535): unable to open host file: /var/tmp/cobalt.152 > [ > mpiexec(a)knl04.ftm.alcf.anl.gov > > ] mfile_fn (../../ui/mpich/utils.c:450): error parsing hostfile > [ > mpiexec(a)knl04.ftm.alcf.anl.gov > > ] match_arg (../../utils/args/args.c:243): match handler returned error > [ > mpiexec(a)knl04.ftm.alcf.anl.gov > > ] HYDU_parse_array (../../utils/args/args.c:269): argument matching returned error > [ > mpiexec(a)knl04.ftm.alcf.anl.gov > > ] parse_args (../../ui/mpich/utils.c:4498): error parsing input array > [ > mpiexec(a)knl04.ftm.alcf.anl.gov > > ] HYD_uii_mpx_get_parameters (../../ui/mpich/utils.c:4819): unable to parse user arguments > [ > mpiexec(a)knl04.ftm.alcf.anl.gov > > ] HYDU_parse_hostfile (../../utils/args/args.c:535): unable to open host file: /var/tmp/cobalt.152 > [ > mpiexec(a)knl04.ftm.alcf.anl.gov > > ] mfile_fn (../../ui/mpich/utils.c:450): error parsing hostfile > [ > mpiexec(a)knl04.ftm.alcf.anl.gov > > ] match_arg (../../utils/args/args.c:243): match handler returned error > [ > mpiexec(a)knl04.ftm.alcf.anl.gov > > ] HYDU_parse_array (../../utils/args/args.c:269): argument matching returned error > [ > mpiexec(a)knl04.ftm.alcf.anl.gov > > ] parse_args (../../ui/mpich/utils.c:4498): error parsing input array > [ > mpiexec(a)knl04.ftm.alcf.anl.gov > > ] HYD_uii_mpx_get_parameters (../../ui/mpich/utils.c:4819): unable to parse user arguments > > > Then I tried to start an interactive job (without specifying any memory mode) > and I got the same nodes (knl04,05) and there is no node file: > > > [jlselogin1 he4] qsub -n 2 -q knl_7210 -t 60 -I > Wait for job 154 to start... > Opening interactive session to knl04 > Last login: Tue Aug 9 16:24:36 2016 from jlselogin1.ftm.alcf.anl.gov > DISPLAY = 10.17.6.108:10.0 connected to knl04 > ssh connection, ssh display would be:, 10.17.6.11:0.0 > > [knl04 ~] echo $COBALT_NODEFILE > > [knl04 ~] > > > Steve > > -- > Steven C. Pieper, Senior Physicist Emeritus: > > spieper(a)anl.gov > > > > Argonne National Laboratory, Physics Division, Bldg. 203, Argonne, IL 60439 > Phone: 630-252-4232 <(630)%20252-4232> Fax -6008 Secretary, Debra Beres, -4100 > > > http://www.phy.anl.gov/theory/staff/scp.html > > > _______________________________________________ > Intel-nda mailing list > > [email protected]://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > _______________________________________________ > Intel-nda mailing list > [email protected]://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > >

1 0