September 2017 - mpiwg-ft - mailman3.cels.anl.gov

Re: [mpiwg-ft] [EXTERNAL] Re: Any agenda for the MPI-Forum September?
by Teranishi, Keita 13 Sep '17

13 Sep '17

Wesley, I saw the agenda on the webpage. I will join remotely. (*) I could not attend the past meetings due to my schedule conflict (2nd and 4th Wednesday lunch time). I will be having this duty for the rest of year, which make me difficult to attend the FTWG meetings, but I follow the meeting notes! Thanks, Keita From: mpiwg-ft <mpiwg-ft-bounces(a)lists.mpi-forum.org> on behalf of "Bland, Wesley" <wesley.bland(a)intel.com> Reply-To: MPI WG Fault Tolerance and Dynamic Process Control working Group <mpiwg-ft(a)lists.mpi-forum.org> Date: Thursday, August 24, 2017 at 11:18 AM To: "mpiwg-ft(a)lists.mpi-forum.org" <mpiwg-ft(a)lists.mpi-forum.org> Subject: [EXTERNAL] Re: [mpiwg-ft] Any agenda for the MPI-Forum September? I don't think Martin has started putting together the agenda yet, but we've asked for time for a reading of the error handler issues and a half day time for WG time to discuss catastrophic errors and continue our ULFM/Reinit/etc. discussions. On Aug 24, 2017, at 1:09 PM, Teranishi, Keita <knteran(a)sandia.gov<mailto:[email protected]>> wrote: All, I am curious the dates and discussion items in FTWG in the MPI Forum. Do you have any info? Thanks, ----------------------------------------------------------------------------- Keita Teranishi Principal Member of Technical Staff Scalable Modeling and Analysis Systems Sandia National Laboratories Livermore, CA 94551 +1 (925) 294-3738 _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft

1 0

2017-09-13 FTWG Con Call Notes
by Bland, Wesley 13 Sep '17

13 Sep '17

The notes for today's call are now posted on the wiki (https://github.com/mpiwg-ft/ft-issues/wiki/2017-09-13) I'm also going to copy them here to make it easier for those who are keeping up only via the mailing lists. For next week's F2F, if there's anything you'd like to add to the agenda, please let me know. Also, there will be a webex for us to use. The information is posted on the meeting website (http://mpi-forum.org/meetings/2017/09/agenda) If you're planning to call in for a particular part and want us to cover something first or last, let me know. We'll do our best to accommodate, but I imagine that most of our time will be spent on process fault tolerance so it might be hard to carve out time for specific pieces. Thanks, Wesley ==== <https://github.com/mpiwg-ft/ft-issues/wiki/2017-09-13#agenda-for-f2f>Agenda for F2F * WG Time * Briefly? discuss catastrophic errors again * Move forward on process failure (ULFM, Reinit, etc.) * Reading * Read error handlers <https://github.com/mpiwg-ft/ft-issues/wiki/2017-09-13#con-call-notes>Con Call Notes <https://github.com/mpiwg-ft/ft-issues/wiki/2017-09-13#error-handlers>Error Handlers * Went over slides and PDF for reading * Want to change one sentence in advice to implementors in Section 8.2. * This should be a small enough change to be acceptable. Will point it out separately. <https://github.com/mpiwg-ft/ft-issues/wiki/2017-09-13#catastrophic-errors>Catastrophic Errors * Discussed current proposal and decided that we're still happy with it. * Global state of MPI_GET_STATE is ok because if any thread is catastrophic, all threads are catastrophic and can't recover anyway. * If you're checking the state, you're probably going to do it in an error handlers so you'll know which error code to look for to find out about the error. * Bill Gropp was asking us to look at things like what POSIX does for errors, but it's difficult to replicate that in MPI because of the much larger amount of state that MPI has to maintain across multiple processes. POSIX is more local and stateless (or the state lives in the user's data). * We might end up needing more error classes so we can give the user specific information about errors. * Might be ready to move forward on a December reading here. <https://github.com/mpiwg-ft/ft-issues/wiki/2017-09-13#process-failure>Process Failure * Aurelien proposed adding a MPI_COMM_REVOKE_ALL function to resolve the deadlock problem with overlapping communicators. * Others were skeptical because you might always have to assume that you need to revoke all communicators any time you have overlapping communication. * Aurelien asserted that having concurrent communication with overlapping communicators is not common and might not be as bad as we think.

1 0

Re: [mpiwg-ft] FTWG Con Call Today
by Anthony Skjellum 13 Sep '17

13 Sep '17

Thank you for this Anthony Skjellum, PhD 205-807-4968 > On Sep 13, 2017, at 1:19 PM, Ignacio Laguna <lagunaperalt1(a)llnl.gov> wrote: > > Hi Wesley, all, > > I would like to revisit the action items that we posted on the wiki for (2017/05/10): > > **** > - Working Group - Try to fix the deadlock problem in ULFM, either via automatic recovery or otherwise. If we do something else, we need to be able to explain why it's better than automatic recovery. > - Working Group - Add function to the FT chapter to pick which FT model we want to use. > - Ignacio - Write text for the backward recovery model to move forward with standardization of a second recovery model. This is probably the best way to make sure that ULFM and Reinit can live in the Standard together. > **** > > I admit I haven't done my homework on the last item, but I would like to continue the discussion in the F2F about how two recovery models for different classes of applications could live together, plus finding ways to solve the ULFM deadlock issues. > > Thanks! > > -- > Ignacio Laguna > Center for Applied Scientific Computing (CASC) > Lawrence Livermore National Laboratory > Phone: 925-422-7308, Fax: 925-422-6287 > >> On 9/13/17 9:08 AM, Bland, Wesley wrote: >> The Fault Tolerance Working Group’s biweekly con call is today at 3:00 PM Eastern. Today's agenda: >> * Go over F2F agenda and slides. >> If there's something else that people would like to discuss, please just send an email to the WG so we can get it on the agenda. >> Thanks, >> Wesley >> ......................................................................................................................................... >> Join from PC, Mac, Linux, iOS or Android: https://tennessee.zoom.us/j/632356722?pwd=lI4%2F169CGcewIumekTziMw%3D%3D >> Password: mpiforum >> Or iPhone one-tap (US Toll): +14086380968,632356722# or +16465588656,632356722# >> Or Telephone: >> Dial: +1 408 638 0968 (US Toll) or +1 646 558 8656 (US Toll) >> Meeting ID: 632 356 722 >> International numbers available: https://tennessee.zoom.us/zoomconference?m=GscM59o_Qoig8v4aJl1OrsnXL-7Blrke >> Or an H.323/SIP room system: >> H.323: 162.255.37.11 (US West) or 162.255.36.11 (US East) >> Meeting ID: 632 356 722 >> Password: 366244 >> SIP: 632356722(a)zoomcrc.com >> Password: 366244 >> ......................................................................................................................................... >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft

1 0

Re: [mpiwg-ft] FTWG Con Call Today
by Van Der Wijngaart, Rob F 13 Sep '17

13 Sep '17

Interesting paper for this forum: https://arxiv.org/abs/1709.03316 What does fault tolerant Deep Learning need from MPI? Vinay Amatya, Abhinav Vishnu, Charles Siegel, Jeff Daily (Submitted on 11 Sep 2017) Deep Learning (DL) algorithms have become the de facto Machine Learning (ML) algorithm for large scale data analysis. DL algorithms are computationally expensive - even distributed DL implementations which use MPI require days of training (model learning) time on commonly studied datasets. Long running DL applications become susceptible to faults - requiring development of a fault tolerant system infrastructure, in addition to fault tolerant DL algorithms. This raises an important question: What is needed from MPI for de- signing fault tolerant DL implementations? In this paper, we address this problem for permanent faults. We motivate the need for a fault tolerant MPI specification by an in-depth consideration of recent innovations in DL algorithms and their properties, which drive the need for specific fault tolerance features. We present an in-depth discussion on the suitability of different parallelism types (model, data and hybrid); a need (or lack thereof) for check-pointing of any critical data structures; and most importantly, consideration for several fault tolerance proposals (user-level fault mitigation (ULFM), Reinit) in MPI and their applicability to fault tolerant DL implementations. We leverage a distributed memory implementation of Caffe, currently available under the Machine Learning Toolkit for Extreme Scale (MaTEx). We implement our approaches by ex- tending MaTEx-Caffe for using ULFM-based implementation. Our evaluation using the ImageNet dataset and AlexNet, and GoogLeNet neural network topologies demonstrates the effectiveness of the proposed fault tolerant DL implementation using OpenMPI based ULFM. -----Original Message----- From: mpiwg-ft [mailto:[email protected]] On Behalf Of Bland, Wesley Sent: Wednesday, September 13, 2017 9:08 AM To: FTWG <mpiwg-ft(a)lists.mpi-forum.org> Subject: [mpiwg-ft] FTWG Con Call Today The Fault Tolerance Working Group’s biweekly con call is today at 3:00 PM Eastern. Today's agenda: * Go over F2F agenda and slides. If there's something else that people would like to discuss, please just send an email to the WG so we can get it on the agenda. Thanks, Wesley ......................................................................................................................................... Join from PC, Mac, Linux, iOS or Android: https://tennessee.zoom.us/j/632356722?pwd=lI4%2F169CGcewIumekTziMw%3D%3D Password: mpiforum Or iPhone one-tap (US Toll): +14086380968,632356722# or +16465588656,632356722# Or Telephone: Dial: +1 408 638 0968 (US Toll) or +1 646 558 8656 (US Toll) Meeting ID: 632 356 722 International numbers available: https://tennessee.zoom.us/zoomconference?m=GscM59o_Qoig8v4aJl1OrsnXL-7Blrke Or an H.323/SIP room system: H.323: 162.255.37.11 (US West) or 162.255.36.11 (US East) Meeting ID: 632 356 722 Password: 366244 SIP: 632356722(a)zoomcrc.com Password: 366244 ......................................................................................................................................... _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft

1 0

Re: [mpiwg-ft] FTWG Con Call Today
by Ignacio Laguna 13 Sep '17

13 Sep '17

Hi Wesley, all, I would like to revisit the action items that we posted on the wiki for (2017/05/10): **** - Working Group - Try to fix the deadlock problem in ULFM, either via automatic recovery or otherwise. If we do something else, we need to be able to explain why it's better than automatic recovery. - Working Group - Add function to the FT chapter to pick which FT model we want to use. - Ignacio - Write text for the backward recovery model to move forward with standardization of a second recovery model. This is probably the best way to make sure that ULFM and Reinit can live in the Standard together. **** I admit I haven't done my homework on the last item, but I would like to continue the discussion in the F2F about how two recovery models for different classes of applications could live together, plus finding ways to solve the ULFM deadlock issues. Thanks! -- Ignacio Laguna Center for Applied Scientific Computing (CASC) Lawrence Livermore National Laboratory Phone: 925-422-7308, Fax: 925-422-6287 On 9/13/17 9:08 AM, Bland, Wesley wrote: > The Fault Tolerance Working Group’s biweekly con call is today at 3:00 PM Eastern. Today's agenda: > > * Go over F2F agenda and slides. > > If there's something else that people would like to discuss, please just send an email to the WG so we can get it on the agenda. > > Thanks, > Wesley > > ......................................................................................................................................... > Join from PC, Mac, Linux, iOS or Android: https://tennessee.zoom.us/j/632356722?pwd=lI4%2F169CGcewIumekTziMw%3D%3D > Password: mpiforum > > Or iPhone one-tap (US Toll): +14086380968,632356722# or +16465588656,632356722# > > Or Telephone: > Dial: +1 408 638 0968 (US Toll) or +1 646 558 8656 (US Toll) > Meeting ID: 632 356 722 > International numbers available: https://tennessee.zoom.us/zoomconference?m=GscM59o_Qoig8v4aJl1OrsnXL-7Blrke > > Or an H.323/SIP room system: > H.323: 162.255.37.11 (US West) or 162.255.36.11 (US East) > Meeting ID: 632 356 722 > Password: 366244 > > SIP: 632356722(a)zoomcrc.com > Password: 366244 > ......................................................................................................................................... > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft >

1 0

FTWG Con Call Today
by Bland, Wesley 13 Sep '17

13 Sep '17

The Fault Tolerance Working Group’s biweekly con call is today at 3:00 PM Eastern. Today's agenda: * Go over F2F agenda and slides. If there's something else that people would like to discuss, please just send an email to the WG so we can get it on the agenda. Thanks, Wesley ......................................................................................................................................... Join from PC, Mac, Linux, iOS or Android: https://tennessee.zoom.us/j/632356722?pwd=lI4%2F169CGcewIumekTziMw%3D%3D Password: mpiforum Or iPhone one-tap (US Toll): +14086380968,632356722# or +16465588656,632356722# Or Telephone: Dial: +1 408 638 0968 (US Toll) or +1 646 558 8656 (US Toll) Meeting ID: 632 356 722 International numbers available: https://tennessee.zoom.us/zoomconference?m=GscM59o_Qoig8v4aJl1OrsnXL-7Blrke Or an H.323/SIP room system: H.323: 162.255.37.11 (US West) or 162.255.36.11 (US East) Meeting ID: 632 356 722 Password: 366244 SIP: 632356722(a)zoomcrc.com Password: 366244 .........................................................................................................................................

1 0