November 2017 - mpiwg-ft - mailman3.cels.anl.gov

FTWG Meeting Topics for Tomorrow?
by Bland, Wesley 07 Nov '17

07 Nov '17

Hi folks, I have an all day training tomorrow that I don't get to step out of so I won't be able to attend the con call. Are there topics that we need to discuss? You're of course welcome to do so without me but if there are none, we can cancel. Thanks, Wesley

1 0

ULFM 2.0 release candidate
by Aurelien Bouteiller 03 Nov '17

03 Nov '17

All, This is with great pleasure that the Open MPI ULFM team announces the new release candidate for ULFM 2.0. [https://bitbucket.org/icldistcomp/ulfm2/get/ulfm2.0rc.tar.bz2] The focus for ULFM 2.0 has been toward integration with current Open MPI master, performance, and stability. - ULFM is now based upon Open MPI master branch (#689f1be9). - Fault Tolerance is enabled by default and is controlled with MCA variables. - Added support for multithreaded modes (MPI_THREAD_MULTIPLE, etc.) - Added support for non-blocking collective operations (NBC). - Added support for CMA shared memory transport (Vader). - Added support for advanced failure detection at the MPI level. Implements the algorithm described in "Failure detection and propagation in HPC systems." <https://doi.org/10.1109/SC.2016.26>. - Removed the need for special handling of CID allocation. - Non-usable components are automatically removed from the build during configure - RMA, FILES, and TOPO components are enabled by default, and usage in a fault tolerant execution warns that they may cause undefined behavior after a failure. - Bugfixes, bugfixes, bugfixes As usual, you can find more information on [http://fault-tolerance.org/2017/11/03/ulfm-2-0/] or on the (new*) project repository [https://bitbucket.org/icldistcomp/ulfm2/overview] Happy Hacking, The Open MPI ULFM team. * the ULFM1 repository will go into archival mode.

1 0

Re: [mpiwg-ft] Subjects for Tomorrow's FTWG Call
by Bland, Wesley 03 Nov '17

03 Nov '17

Apparently my attempt to solve the organization problem by creating a calendar invitation has failed spectacularly, so here's the call in information again. I'll go back to just sending an email before the call each week: === Join from PC, Mac, Linux, iOS or Android: https://tennessee.zoom.us/j/632356722?pwd=lI4%2F169CGcewIumekTziMw%3D%3D Password: mpiforum Or iPhone one-tap (US Toll): +14086380968,632356722# or +16465588656,632356722# Or Telephone: Dial: +1 408 638 0968 (US Toll) or +1 646 558 8656 (US Toll) Meeting ID: 632 356 722 International numbers available: https://tennessee.zoom.us/zoomconference?m=GscM59o_Qoig8v4aJl1OrsnXL-7Blrke Or an H.323/SIP room system: H.323: 162.255.37.11 (US West) or 162.255.36.11 (US East) Meeting ID: 632 356 722 Password: 366244 SIP: 632356722(a)zoomcrc.com<mailto:[email protected]> Password: 366244 On Nov 2, 2017, at 11:06 PM, Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> wrote: Wednesday 9 AM Oregon time on is feasible, but I can't remember the bridge info. Can you forward to my gmail?

1 0

Re: [mpiwg-ft] Subjects for Tomorrow's FTWG Call
by Jeff Hammond 02 Nov '17

02 Nov '17

> > > Why don't we go ahead and cancel for this week. Jeff, can you attend next > week? (Wednesdays at 12:00 PM Eastern US) > > Wednesday 9 AM Oregon time on is feasible, but I can't remember the bridge info. Can you forward to my gmail? Jeff -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpiwg-ft] Subjects for Tomorrow's FTWG Call
by Jeff Hammond 02 Nov '17

02 Nov '17

> Jim - Should MPI_ERR_DATA_UNAVAILABLE be usable outside of RMA? Does it apply to point-to-point or collectives? No. RMA is the only context where data movement/access is decoupled from processing at the target. It is impossible to make data available via two-sided communication if the/a target process is dead. > Jim/Aurelien - Is the justification for this work that flush doesn't allow detection of process failure? They're still not convinced that this is true. The issue is not that flush can't detect process failure in some cases. It is that flush cannot detect process failure in all cases, and by cases, I mean implementations. If your RMA implementation utilizes an ASIC, rather than the CPU, then it has no reason to determine if the target process is alive or not, and burdening it with doing so may be a performance disaster. The other scenario where detecting if the target process is alive or not is anything like Casper, where the remote processing is done by an agent other than the target MPI process, although I recall that we discussed a model where any process failure associated with the communicator associated with a shared-memory window would need to take out all of those processes in order to have a sane model. > Jim - On the other hand, it might be true that we can't guarantee any process failure detection in any RMA operation. Maybe we should just not allow process failure errors (as opposed to "upgrading" other types of errors to process failure). There is no reason to disallow detecting of process failure. We know many implementations use an active message implementation where such detection is trivial. > Jim - One place this still makes sense as is is having a process with data corrupted because another process failed during a put. If a third process is reading the bad memory, it could get MPI_ERR_DATA_UNAVAILABLE instead of MPI_ERR_PROC_FAILED. This sounds like the Byzantine problem. MPI should not be responsible for detecting data corruption. > We're still unclear on the failure model expected here. We probably need to get more feedback from Jeff. First, all-or-nothing window invalidation is awful. I think we've moved beyond that, but just to be sure, I'll justify why it's awful. Some PGAS applications - NWChem being the most famous example - have read-only and update-only epochs, which are collective. If a process fails, the application should be able to continue to use data that it knows programmatically could not have been modified during the epoch. Furthermore, the application may know programmatically which processes touched which data and thus can recover some portion of the window(s) touched in an update-only epoch. Second, I do not want to burden RMA implementations with "is target process alive?" active-message traffic in every communication operation, when the fault-intolerant implementation would not interact with the target process, e.g. RDMA networks. I want an error code that RMA can use that recognizes that RMA decouples synchronization and data movement and that process live/dead checking is a form of synchronization. Jeff On Wed, Nov 1, 2017 at 6:24 AM, Bland, Wesley <wesley.bland(a)intel.com> wrote: > > I was waiting until you got back from leave. > > We have some notes from the discussion two weeks ago on the wiki page: https://github.com/mpiwg-ft/ft-issues/wiki/2017-10-18. We'd like to get a little more feedback on the failure model you were envisioning. It'd probably be most productive to do this on a call so we can get some back and forth. > > Why don't we go ahead and cancel for this week. Jeff, can you attend next week? (Wednesdays at 12:00 PM Eastern US) > > Thanks, > Wesley > > On Oct 31, 2017, at 4:46 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > What feedback do you want? I haven’t seen any specific requests for comment. > > Jeff > > On Tue, Oct 31, 2017 at 1:04 PM Bland, Wesley <wesley.bland(a)intel.com> wrote: >> >> Hi all, >> >> I think we're still on hold for the ULFM data resilience discussion until we're able to get some more feedback from Jeff and our non-ULFM topics are ready for another discussion with the rest of the forum. >> >> Are there other topics that people want to discuss tomorrow or should we cancel the call? >> >> Thanks, >> Wesley >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft > > > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Canceled: Fault Tolerance Working Group Con Call
by Bland, Wesley 01 Nov '17

01 Nov '17

When: Wednesday, November 01, 2017 4:00 PM-5:00 PM. UTC Where: Zoom *~*~*~*~*~*~*~*~*~* Weekly con call to discuss Fault Tolerance Topics as they relate to the MPI Standard. I’ll try to update the agenda before the call weekly (as opposed to the old way of sending a reminder email). === Join from PC, Mac, Linux, iOS or Android: https://tennessee.zoom.us/j/632356722?pwd=lI4%2F169CGcewIumekTziMw%3D%3D Password: mpiforum Or iPhone one-tap (US Toll): +14086380968,632356722# or +16465588656,632356722# Or Telephone: Dial: +1 408 638 0968 (US Toll) or +1 646 558 8656 (US Toll) Meeting ID: 632 356 722 International numbers available: https://tennessee.zoom.us/zoomconference?m=GscM59o_Qoig8v4aJl1OrsnXL-7Blrke Or an H.323/SIP room system: H.323: 162.255.37.11 (US West) or 162.255.36.11 (US East) Meeting ID: 632 356 722 Password: 366244 SIP: 632356722(a)zoomcrc.com Password: 366244

1 0

Re: [mpiwg-ft] Subjects for Tomorrow's FTWG Call
by Bland, Wesley 01 Nov '17

01 Nov '17

I was waiting until you got back from leave. We have some notes from the discussion two weeks ago on the wiki page: https://github.com/mpiwg-ft/ft-issues/wiki/2017-10-18. We'd like to get a little more feedback on the failure model you were envisioning. It'd probably be most productive to do this on a call so we can get some back and forth. Why don't we go ahead and cancel for this week. Jeff, can you attend next week? (Wednesdays at 12:00 PM Eastern US) Thanks, Wesley On Oct 31, 2017, at 4:46 PM, Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> wrote: What feedback do you want? I haven’t seen any specific requests for comment. Jeff On Tue, Oct 31, 2017 at 1:04 PM Bland, Wesley <wesley.bland(a)intel.com<mailto:[email protected]>> wrote: Hi all, I think we're still on hold for the ULFM data resilience discussion until we're able to get some more feedback from Jeff and our non-ULFM topics are ready for another discussion with the rest of the forum. Are there other topics that people want to discuss tomorrow or should we cancel the call? Thanks, Wesley _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> http://jeffhammond.github.io/ _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft

1 0