As a developer of network middleware, part of my interest in the KNL nodes at JLSE is the Omni-Path network.

I had been successfully using both pshm2 and librabric (psm2 provider) on the KNLs.
However, sometime between Oct 11 and Dec 3 things ceased to work as they had.
Those are the dates of the latest successful run and earliest fail run.
Sorry that I don't have anything between those dates.

Both psm2 and libfabric based programs are now failing with messages like:
knl08.ftm.alcf.anl.gov.223177PSM found 0 available contexts on opa device(s). (err=21)   

Additionally, /usr/sbin/opainfo says:
{hargrove@knl04 ~}$ /usr/sbin/opainfo
oib_utils ERROR: [23478] oib_open_port: can't open UMAD port (hfi1_0:1)
opainfo: Unable to open hfi:port 0:1
(Though, in all honesty, I don't know if opainfo ever worked before).

-Paul

--
Paul H. Hargrove                          PHHargrove@lbl.gov
Computer Languages & Systems Software (CLaSS) Group
Computer Science Department               Tel: +1-510-495-2352
Lawrence Berkeley National Laboratory     Fax: +1-510-486-6900