- mpiwg-ft - mailman3.cels.anl.gov

Re: [mpiwg-ft] FTWG Con Call Today?
by Ignacio Laguna 08 Mar '21

08 Mar '21

We are working on releasing a newer version of the Reinit spec. We would appreciate the feedback of the group again. We can go after Tony. Ignacio On 3/8/21 7:48 AM, Wesley Bland via mpiwg-ft wrote: > Hi all, > > Tony wants to wait until next time to talk about FA-MPI and Stages, do > we have anything to discuss from Aurelien or Ignacio? If not, we can > cancel this week. > > Thanks, > Wes > > The Fault Tolerance Working Group’s weekly con call is today at 12:00 > PM Eastern. Today's agenda: > > * ??? > > If there's something else that people would like to discuss, please > just send an email to the WG so we can get it on the agenda. > > Thanks, > Wes > > ......................................................................................................................................... > > Join from PC, Mac, Linux, iOS or Android: > https://tennessee.zoom.us/j/632356722?pwd=lI4_169CGcewIumekTziMw > <https://urldefense.us/v3/__https://tennessee.zoom.us/j/632356722?pwd=lI4_16…> > Password: mpiforum > > Or iPhone one-tap (US Toll): +16468769923,632356722# or > +16699006833,632356722# > > Or Telephone: > Dial: > +1 646 876 9923 (US Toll) > +1 669 900 6833 (US Toll) > Meeting ID: 632 356 722 > International numbers available: https://zoom.us/u/6uINe > <https://urldefense.us/v3/__https://zoom.us/u/6uINe__;!!G2kpM7uM-TzIFchu!kwH…> > > Or an H.323/SIP room system: > H.323: 162.255.37.11 (US West) or 162.255.36.11 (US East) > Meeting ID: 632 356 722 > Password: 364216 > > SIP:[email protected] <mailto:[email protected]> > Password: 364216 > ......................................................................................................................................... > > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > https://urldefense.us/v3/__https://lists.mpi-forum.org/mailman/listinfo/mpi…

1 0

FTWG Con Call Today?
by Wesley Bland 08 Mar '21

08 Mar '21

Hi all, Tony wants to wait until next time to talk about FA-MPI and Stages, do we have anything to discuss from Aurelien or Ignacio? If not, we can cancel this week. Thanks, Wes The Fault Tolerance Working Group’s weekly con call is today at 12:00 PM Eastern. Today's agenda: * ??? If there's something else that people would like to discuss, please just send an email to the WG so we can get it on the agenda. Thanks, Wes ......................................................................................................................................... Join from PC, Mac, Linux, iOS or Android: https://tennessee.zoom.us/j/632356722?pwd=lI4_169CGcewIumekTziMw <https://tennessee.zoom.us/j/632356722?pwd=lI4_169CGcewIumekTziMw> Password: mpiforum Or iPhone one-tap (US Toll): +16468769923,632356722# or +16699006833,632356722# Or Telephone: Dial: +1 646 876 9923 (US Toll) +1 669 900 6833 (US Toll) Meeting ID: 632 356 722 International numbers available: https://zoom.us/u/6uINe <https://zoom.us/u/6uINe> Or an H.323/SIP room system: H.323: 162.255.37.11 (US West) or 162.255.36.11 (US East) Meeting ID: 632 356 722 Password: 364216 SIP: 632356722(a)zoomcrc.com <mailto:[email protected]> Password: 364216 .........................................................................................................................................

1 0

FTWG Con Call
by Wesley Bland 08 Feb '21

08 Feb '21

The Fault Tolerance Working Group’s weekly con call is today at 12:00 PM Eastern. Today's agenda: * FA-MPI (Tony) * Reinit questions (Ignacio) * Other topics? If there's something else that people would like to discuss, please just send an email to the WG so we can get it on the agenda. Thanks, Wes ......................................................................................................................................... Join from PC, Mac, Linux, iOS or Android: https://tennessee.zoom.us/j/632356722?pwd=lI4_169CGcewIumekTziMw Password: mpiforum Or iPhone one-tap (US Toll): +16468769923,632356722# or +16699006833,632356722# Or Telephone: Dial: +1 646 876 9923 (US Toll) +1 669 900 6833 (US Toll) Meeting ID: 632 356 722 International numbers available: https://zoom.us/u/6uINe Or an H.323/SIP room system: H.323: 162.255.37.11 (US West) or 162.255.36.11 (US East) Meeting ID: 632 356 722 Password: 364216 SIP: 632356722(a)zoomcrc.com Password: 364216 .........................................................................................................................................

1 0

Re: [mpiwg-ft] [Mpi-forum] FTWG Con Call Today
by Wesley Bland 02 Feb '21

02 Feb '21

[Replacing full MPI Forum list with FTWG list since I messed up the original email] Thanks Ignacio. I’ve added these to my list. Right now, Tony is scheduled to talk about FA-MPI on Monday. If he doesn’t use all of his time, we can talk about as many of these as we can get through. Otherwise we can put them on the agenda for the next meeting. Thanks, Wes > On Feb 1, 2021, at 4:10 PM, Ignacio Laguna <lagunaperalt1(a)llnl.gov> wrote: > > Hi Wesley, all, > > Wherever we have a chance in the working group, we would like to discuss the following: > > 1. The latest Reinit spec (0.1.1), which you can find in the following URL. This includes changes (in blue) based on the last feedback the group gave us. There is a section at the end with the list of changes in this version. > > https://reinit.github.io/reinit/docs/reinit-0.1.1.pdf > > 2. The behavior of MPI under different combinations of error handlers. See the attached table. Based on my reading of the standard, I still have questions about the expected behavior for some of these combinations. > > 3. The state of MPI when a failure occurs and before we call MPI_test_failure in Reinit is not clearly defined. We would like to discuss if we can re-use the same text that ULFM uses (as far as I understand ULFM has to handle this situation). > > Thanks! > > Ignacio > > > On 1/25/21 6:20 AM, Wesley Bland via mpi-forum wrote: >> Hi all, >> After talking to Tony, we're going to delay this discussion until the next call on Feb 8. Today's call is cancelled. >> Thanks, >> Wes >> On Mon, Jan 25, 2021 at 8:15 AM work(a)wesbland.com <mailto:[email protected]> <work(a)wesbland.com <mailto:[email protected]>> wrote: >> The Fault Tolerance Working Group’s weekly con call is today at >> 12:00 PM Eastern. Today's agenda:____ >> __ __ >> * FA-MPI (Tony)____ >> * Other updates (All)____ >> __ __ >> If there's something else that people would like to discuss, please >> just send an email to the WG so we can get it on the agenda.____ >> __ __ >> Thanks, ____ >> Wes ____ >> __ __ >> ....................................................................................................................................... >> ____ >> Join from PC, Mac, Linux, iOS or Android: >> https://tennessee.zoom.us/j/632356722?pwd=lI4_169CGcewIumekTziMw____ >> Password: mpiforum____ >> __ __ >> Or iPhone one-tap (US Toll): +16468769923,632356722# or >> +16699006833,632356722# ____ >> __ __ >> Or Telephone:____ >> Dial:____ >> +1 646 876 9923 (US Toll)____ >> +1 669 900 6833 (US Toll)____ >> Meeting ID: 632 356 722____ >> International numbers available: https://zoom.us/u/6uINe____ >> __ __ >> Or an H.323/SIP room system:____ >> H.323: 162.255.37.11 (US West) or 162.255.36.11 (US East) ____ >> Meeting ID: 632 356 722____ >> Password: 364216____ >> __ __ >> SIP: 632356722(a)zoomcrc.com <mailto:[email protected]>____ >> Password: 364216____ >> .......................................................................................................................................____ >> _______________________________________________ >> mpi-forum mailing list >> mpi-forum(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpi-forum > <mpi_error_handlers.pptx>

1 0

Re: [mpiwg-ft] [EXTERNAL] Re: [Mpi-forum] FTWG Con Call Today
by Ignacio Laguna 25 Jan '21

25 Jan '21

Hi Keita, Good question. We have two operations mode, asynchronous and synchronous. In the asynchronous case, the application automatically jumps back to the reinit point. In the synchronous case, the user places MPI_test_failure calls to check for failures and jump back. The latter is to deal with OpenMP regions because we don't want to jump back in the middle of them. With GPU/Accelerator model, host code will get an error from a kernel launch and will trigger recovery after that. Th user is responsible for cleaning up state in the GPU as well as calling libraries to clean up their state as well. I hope that answers the question. Ignacio On 1/25/21 10:16 AM, Teranishi, Keita wrote: > All, > > From my experience with Fenix (our ULFM-based recovery model similar to ReInit), we found that this approach works with flat-MPI mode. > However, if the program is using an exotic runtime (such as HCLIB, on-node task parallel runtime from GaTech), setjump/longjump messes up the program. > This required us to either (1) refactor the runtime to be MPI-Aware, or give up setjump/longjump approach for the work presented in the last ExaMPI workshop. > > I am wondering how to handle a program with OpenMP/CUDA (and AMD and Intel accelerator runtime) in the setjump-longjump recovery model. It appears that it is safe to clean-up the runtime before calling longjump. With OpenMP, longjump should happen outside any OpenMP pragama (it is easy to implement. I think). With GPU/Accelerator model, all kernel executions should be completed or cancelled. My question is that who is responsible for this type of clean-up. Is it user's responsibility? > > Regards, > Keita > > On 1/25/21, 9:20 AM, "mpiwg-ft on behalf of Ignacio Laguna via mpiwg-ft" <mpiwg-ft-bounces(a)lists.mpi-forum.org on behalf of mpiwg-ft(a)lists.mpi-forum.org> wrote: > > Michael: > > It's very simple. The high-level idea is to encapsulate the main > function into a resilient_main function, which we can call again when a > failure occurs. In our implementation, we use set-jump/long-jump > semantics. As long the encapsulation is done properly, branches higher > up in the stack won't affect it. > > Ignacio > > On 1/25/21 9:13 AM, Steyer, Michael wrote: > > Thanks Ignacio, I'd be very interested in learning how that approach works. Especially the "goes always back to the resilient_function call 1" part, without adding another branch on the call stack? > > > > /Michael > > > > -----Original Message----- > > From: mpiwg-ft <mpiwg-ft-bounces(a)lists.mpi-forum.org> On Behalf Of Ignacio Laguna via mpiwg-ft > > Sent: Montag, 25. Januar 2021 17:33 > > To: MPI WG Fault Tolerance and Dynamic Process Control working Group <mpiwg-ft(a)lists.mpi-forum.org> > > Cc: Ignacio Laguna <lagunaperalt1(a)llnl.gov> > > Subject: Re: [mpiwg-ft] [Mpi-forum] FTWG Con Call Today > > > > The model is that the app goes always back to the resilient_function call 1 (we cannot call this function twice or more statically in the program). Perhaps we can discuss that again. > > > > Ignacio > > > > > > On 1/25/21 8:25 AM, Wesley Bland via mpiwg-ft wrote: > >> There was another question that came up in internal conversations > >> around here with Reinit: > >> > >> What's going to happen to the call stack. E.g. MPI_Init -> ... -> > >> resilient_function call 1 -> Failure -> ReInit - resilient_function > >> call > >> 2 -> End of Work -> back to resilient_function call 1? > >> > >> On Mon, Jan 25, 2021 at 10:01 AM Ignacio Laguna > >> <lagunaperalt1(a)llnl.gov <mailto:[email protected]>> wrote: > >> > >> That works for me (I couldn't attend today neither). > >> > >> We are almost done with the new Reinit spec but we have a few topics we > >> would like to discuss in the group: (1) using several error handlers > >> and > >> how this is specified in the standard, (2) the state of MPI between a > >> failure and its recovery (how does ULFM does it? Perhaps Reinit can > >> re-use the same text?). > >> > >> Thanks! > >> > >> Ignacio > >> > >> On 1/25/21 6:20 AM, Wesley Bland via mpi-forum wrote: > >> > Hi all, > >> > > >> > After talking to Tony, we're going to delay this discussion until > >> the > >> > next call on Feb 8. Today's call is cancelled. > >> > > >> > Thanks, > >> > Wes > >> > > >> > On Mon, Jan 25, 2021 at 8:15 AM work(a)wesbland.com > >> <mailto:[email protected]> > >> > <mailto:[email protected] <mailto:[email protected]>> > >> <work(a)wesbland.com <mailto:[email protected]> > >> > <mailto:[email protected] <mailto:[email protected]>>> wrote: > >> > > >> > The Fault Tolerance Working Group’s weekly con call is today at > >> > 12:00 PM Eastern. Today's agenda:____ > >> > > >> > __ __ > >> > > >> > * FA-MPI (Tony)____ > >> > > >> > * Other updates (All)____ > >> > > >> > __ __ > >> > > >> > If there's something else that people would like to discuss, > >> please > >> > just send an email to the WG so we can get it on the agenda.____ > >> > > >> > __ __ > >> > > >> > Thanks, ____ > >> > > >> > Wes ____ > >> > > >> > __ __ > >> > > >> > > >> ....................................................................................................................................... > >> > ____ > >> > > >> > Join from PC, Mac, Linux, iOS or Android: > >> > https://tennessee.zoom.us/j/632356722?pwd=lI4_169CGcewIumekTziMw____ > >> > > >> > Password: mpiforum____ > >> > > >> > __ __ > >> > > >> > Or iPhone one-tap (US Toll): +16468769923,632356722# or > >> > +16699006833,632356722# ____ > >> > > >> > __ __ > >> > > >> > Or Telephone:____ > >> > > >> > Dial:____ > >> > > >> > +1 646 876 9923 (US Toll)____ > >> > > >> > +1 669 900 6833 (US Toll)____ > >> > > >> > Meeting ID: 632 356 722____ > >> > > >> > International numbers available: https://zoom.us/u/6uINe____ > >> > > >> > __ __ > >> > > >> > Or an H.323/SIP room system:____ > >> > > >> > H.323: 162.255.37.11 (US West) or 162.255.36.11 (US > >> East) ____ > >> > > >> > Meeting ID: 632 356 722____ > >> > > >> > Password: 364216____ > >> > > >> > __ __ > >> > > >> > SIP: 632356722(a)zoomcrc.com > >> <mailto:[email protected]> <mailto:[email protected] > >> <mailto:[email protected]>>____ > >> > > >> > Password: 364216____ > >> > > >> > > >> .......................................................................................................................................____ > >> > > >> > > >> > _______________________________________________ > >> > mpi-forum mailing list > >> > mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> > >> > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum > >> > > >> > >> > >> _______________________________________________ > >> mpiwg-ft mailing list > >> mpiwg-ft(a)lists.mpi-forum.org > >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft > >> > > _______________________________________________ > > mpiwg-ft mailing list > > mpiwg-ft(a)lists.mpi-forum.org > > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft > > Intel Deutschland GmbH > > Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany > > Tel: +49 89 99 8853-0, www.intel.de > > Managing Directors: Christin Eisenschmid, Gary Kershaw > > Chairperson of the Supervisory Board: Nicole Lau > > Registered Office: Munich > > Commercial Register: Amtsgericht Muenchen HRB 186928 > > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft > >

1 0

Re: [mpiwg-ft] [EXTERNAL] Re: [Mpi-forum] FTWG Con Call Today
by Teranishi, Keita 25 Jan '21

25 Jan '21

All, From my experience with Fenix (our ULFM-based recovery model similar to ReInit), we found that this approach works with flat-MPI mode. However, if the program is using an exotic runtime (such as HCLIB, on-node task parallel runtime from GaTech), setjump/longjump messes up the program. This required us to either (1) refactor the runtime to be MPI-Aware, or give up setjump/longjump approach for the work presented in the last ExaMPI workshop. I am wondering how to handle a program with OpenMP/CUDA (and AMD and Intel accelerator runtime) in the setjump-longjump recovery model. It appears that it is safe to clean-up the runtime before calling longjump. With OpenMP, longjump should happen outside any OpenMP pragama (it is easy to implement. I think). With GPU/Accelerator model, all kernel executions should be completed or cancelled. My question is that who is responsible for this type of clean-up. Is it user's responsibility? Regards, Keita On 1/25/21, 9:20 AM, "mpiwg-ft on behalf of Ignacio Laguna via mpiwg-ft" <mpiwg-ft-bounces(a)lists.mpi-forum.org on behalf of mpiwg-ft(a)lists.mpi-forum.org> wrote: Michael: It's very simple. The high-level idea is to encapsulate the main function into a resilient_main function, which we can call again when a failure occurs. In our implementation, we use set-jump/long-jump semantics. As long the encapsulation is done properly, branches higher up in the stack won't affect it. Ignacio On 1/25/21 9:13 AM, Steyer, Michael wrote: > Thanks Ignacio, I'd be very interested in learning how that approach works. Especially the "goes always back to the resilient_function call 1" part, without adding another branch on the call stack? > > /Michael > > -----Original Message----- > From: mpiwg-ft <mpiwg-ft-bounces(a)lists.mpi-forum.org> On Behalf Of Ignacio Laguna via mpiwg-ft > Sent: Montag, 25. Januar 2021 17:33 > To: MPI WG Fault Tolerance and Dynamic Process Control working Group <mpiwg-ft(a)lists.mpi-forum.org> > Cc: Ignacio Laguna <lagunaperalt1(a)llnl.gov> > Subject: Re: [mpiwg-ft] [Mpi-forum] FTWG Con Call Today > > The model is that the app goes always back to the resilient_function call 1 (we cannot call this function twice or more statically in the program). Perhaps we can discuss that again. > > Ignacio > > > On 1/25/21 8:25 AM, Wesley Bland via mpiwg-ft wrote: >> There was another question that came up in internal conversations >> around here with Reinit: >> >> What's going to happen to the call stack. E.g. MPI_Init -> ... -> >> resilient_function call 1 -> Failure -> ReInit - resilient_function >> call >> 2 -> End of Work -> back to resilient_function call 1? >> >> On Mon, Jan 25, 2021 at 10:01 AM Ignacio Laguna >> <lagunaperalt1(a)llnl.gov <mailto:[email protected]>> wrote: >> >> That works for me (I couldn't attend today neither). >> >> We are almost done with the new Reinit spec but we have a few topics we >> would like to discuss in the group: (1) using several error handlers >> and >> how this is specified in the standard, (2) the state of MPI between a >> failure and its recovery (how does ULFM does it? Perhaps Reinit can >> re-use the same text?). >> >> Thanks! >> >> Ignacio >> >> On 1/25/21 6:20 AM, Wesley Bland via mpi-forum wrote: >> > Hi all, >> > >> > After talking to Tony, we're going to delay this discussion until >> the >> > next call on Feb 8. Today's call is cancelled. >> > >> > Thanks, >> > Wes >> > >> > On Mon, Jan 25, 2021 at 8:15 AM work(a)wesbland.com >> <mailto:[email protected]> >> > <mailto:[email protected] <mailto:[email protected]>> >> <work(a)wesbland.com <mailto:[email protected]> >> > <mailto:[email protected] <mailto:[email protected]>>> wrote: >> > >> > The Fault Tolerance Working Group’s weekly con call is today at >> > 12:00 PM Eastern. Today's agenda:____ >> > >> > __ __ >> > >> > * FA-MPI (Tony)____ >> > >> > * Other updates (All)____ >> > >> > __ __ >> > >> > If there's something else that people would like to discuss, >> please >> > just send an email to the WG so we can get it on the agenda.____ >> > >> > __ __ >> > >> > Thanks, ____ >> > >> > Wes ____ >> > >> > __ __ >> > >> > >> ....................................................................................................................................... >> > ____ >> > >> > Join from PC, Mac, Linux, iOS or Android: >> > https://tennessee.zoom.us/j/632356722?pwd=lI4_169CGcewIumekTziMw____ >> > >> > Password: mpiforum____ >> > >> > __ __ >> > >> > Or iPhone one-tap (US Toll): +16468769923,632356722# or >> > +16699006833,632356722# ____ >> > >> > __ __ >> > >> > Or Telephone:____ >> > >> > Dial:____ >> > >> > +1 646 876 9923 (US Toll)____ >> > >> > +1 669 900 6833 (US Toll)____ >> > >> > Meeting ID: 632 356 722____ >> > >> > International numbers available: https://zoom.us/u/6uINe____ >> > >> > __ __ >> > >> > Or an H.323/SIP room system:____ >> > >> > H.323: 162.255.37.11 (US West) or 162.255.36.11 (US >> East) ____ >> > >> > Meeting ID: 632 356 722____ >> > >> > Password: 364216____ >> > >> > __ __ >> > >> > SIP: 632356722(a)zoomcrc.com >> <mailto:[email protected]> <mailto:[email protected] >> <mailto:[email protected]>>____ >> > >> > Password: 364216____ >> > >> > >> .......................................................................................................................................____ >> > >> > >> > _______________________________________________ >> > mpi-forum mailing list >> > mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> >> > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum >> > >> >> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft >> > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft > Intel Deutschland GmbH > Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany > Tel: +49 89 99 8853-0, www.intel.de > Managing Directors: Christin Eisenschmid, Gary Kershaw > Chairperson of the Supervisory Board: Nicole Lau > Registered Office: Munich > Commercial Register: Amtsgericht Muenchen HRB 186928 > _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft

1 0

Re: [mpiwg-ft] [Mpi-forum] FTWG Con Call Today
by Ignacio Laguna 25 Jan '21

25 Jan '21

Michael: It's very simple. The high-level idea is to encapsulate the main function into a resilient_main function, which we can call again when a failure occurs. In our implementation, we use set-jump/long-jump semantics. As long the encapsulation is done properly, branches higher up in the stack won't affect it. Ignacio On 1/25/21 9:13 AM, Steyer, Michael wrote: > Thanks Ignacio, I'd be very interested in learning how that approach works. Especially the "goes always back to the resilient_function call 1" part, without adding another branch on the call stack? > > /Michael > > -----Original Message----- > From: mpiwg-ft <mpiwg-ft-bounces(a)lists.mpi-forum.org> On Behalf Of Ignacio Laguna via mpiwg-ft > Sent: Montag, 25. Januar 2021 17:33 > To: MPI WG Fault Tolerance and Dynamic Process Control working Group <mpiwg-ft(a)lists.mpi-forum.org> > Cc: Ignacio Laguna <lagunaperalt1(a)llnl.gov> > Subject: Re: [mpiwg-ft] [Mpi-forum] FTWG Con Call Today > > The model is that the app goes always back to the resilient_function call 1 (we cannot call this function twice or more statically in the program). Perhaps we can discuss that again. > > Ignacio > > > On 1/25/21 8:25 AM, Wesley Bland via mpiwg-ft wrote: >> There was another question that came up in internal conversations >> around here with Reinit: >> >> What's going to happen to the call stack. E.g. MPI_Init -> ... -> >> resilient_function call 1 -> Failure -> ReInit - resilient_function >> call >> 2 -> End of Work -> back to resilient_function call 1? >> >> On Mon, Jan 25, 2021 at 10:01 AM Ignacio Laguna >> <lagunaperalt1(a)llnl.gov <mailto:[email protected]>> wrote: >> >> That works for me (I couldn't attend today neither). >> >> We are almost done with the new Reinit spec but we have a few topics we >> would like to discuss in the group: (1) using several error handlers >> and >> how this is specified in the standard, (2) the state of MPI between a >> failure and its recovery (how does ULFM does it? Perhaps Reinit can >> re-use the same text?). >> >> Thanks! >> >> Ignacio >> >> On 1/25/21 6:20 AM, Wesley Bland via mpi-forum wrote: >> > Hi all, >> > >> > After talking to Tony, we're going to delay this discussion until >> the >> > next call on Feb 8. Today's call is cancelled. >> > >> > Thanks, >> > Wes >> > >> > On Mon, Jan 25, 2021 at 8:15 AM work(a)wesbland.com >> <mailto:[email protected]> >> > <mailto:[email protected] <mailto:[email protected]>> >> <work(a)wesbland.com <mailto:[email protected]> >> > <mailto:[email protected] <mailto:[email protected]>>> wrote: >> > >> > The Fault Tolerance Working Group’s weekly con call is today at >> > 12:00 PM Eastern. Today's agenda:____ >> > >> > __ __ >> > >> > * FA-MPI (Tony)____ >> > >> > * Other updates (All)____ >> > >> > __ __ >> > >> > If there's something else that people would like to discuss, >> please >> > just send an email to the WG so we can get it on the agenda.____ >> > >> > __ __ >> > >> > Thanks, ____ >> > >> > Wes ____ >> > >> > __ __ >> > >> > >> ....................................................................................................................................... >> > ____ >> > >> > Join from PC, Mac, Linux, iOS or Android: >> > https://tennessee.zoom.us/j/632356722?pwd=lI4_169CGcewIumekTziMw____ >> > >> > Password: mpiforum____ >> > >> > __ __ >> > >> > Or iPhone one-tap (US Toll): +16468769923,632356722# or >> > +16699006833,632356722# ____ >> > >> > __ __ >> > >> > Or Telephone:____ >> > >> > Dial:____ >> > >> > +1 646 876 9923 (US Toll)____ >> > >> > +1 669 900 6833 (US Toll)____ >> > >> > Meeting ID: 632 356 722____ >> > >> > International numbers available: https://zoom.us/u/6uINe____ >> > >> > __ __ >> > >> > Or an H.323/SIP room system:____ >> > >> > H.323: 162.255.37.11 (US West) or 162.255.36.11 (US >> East) ____ >> > >> > Meeting ID: 632 356 722____ >> > >> > Password: 364216____ >> > >> > __ __ >> > >> > SIP: 632356722(a)zoomcrc.com >> <mailto:[email protected]> <mailto:[email protected] >> <mailto:[email protected]>>____ >> > >> > Password: 364216____ >> > >> > >> .......................................................................................................................................____ >> > >> > >> > _______________________________________________ >> > mpi-forum mailing list >> > mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> >> > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum >> > >> >> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft >> > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft > Intel Deutschland GmbH > Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany > Tel: +49 89 99 8853-0, www.intel.de > Managing Directors: Christin Eisenschmid, Gary Kershaw > Chairperson of the Supervisory Board: Nicole Lau > Registered Office: Munich > Commercial Register: Amtsgericht Muenchen HRB 186928 >

1 0

Re: [mpiwg-ft] [Mpi-forum] FTWG Con Call Today
by Steyer, Michael 25 Jan '21

25 Jan '21

Thanks Ignacio, I'd be very interested in learning how that approach works. Especially the "goes always back to the resilient_function call 1" part, without adding another branch on the call stack? /Michael -----Original Message----- From: mpiwg-ft <mpiwg-ft-bounces(a)lists.mpi-forum.org> On Behalf Of Ignacio Laguna via mpiwg-ft Sent: Montag, 25. Januar 2021 17:33 To: MPI WG Fault Tolerance and Dynamic Process Control working Group <mpiwg-ft(a)lists.mpi-forum.org> Cc: Ignacio Laguna <lagunaperalt1(a)llnl.gov> Subject: Re: [mpiwg-ft] [Mpi-forum] FTWG Con Call Today The model is that the app goes always back to the resilient_function call 1 (we cannot call this function twice or more statically in the program). Perhaps we can discuss that again. Ignacio On 1/25/21 8:25 AM, Wesley Bland via mpiwg-ft wrote: > There was another question that came up in internal conversations > around here with Reinit: > > What's going to happen to the call stack. E.g. MPI_Init -> ... -> > resilient_function call 1 -> Failure -> ReInit - resilient_function > call > 2 -> End of Work -> back to resilient_function call 1? > > On Mon, Jan 25, 2021 at 10:01 AM Ignacio Laguna > <lagunaperalt1(a)llnl.gov <mailto:[email protected]>> wrote: > > That works for me (I couldn't attend today neither). > > We are almost done with the new Reinit spec but we have a few topics we > would like to discuss in the group: (1) using several error handlers > and > how this is specified in the standard, (2) the state of MPI between a > failure and its recovery (how does ULFM does it? Perhaps Reinit can > re-use the same text?). > > Thanks! > > Ignacio > > On 1/25/21 6:20 AM, Wesley Bland via mpi-forum wrote: > > Hi all, > > > > After talking to Tony, we're going to delay this discussion until > the > > next call on Feb 8. Today's call is cancelled. > > > > Thanks, > > Wes > > > > On Mon, Jan 25, 2021 at 8:15 AM work(a)wesbland.com > <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>> > <work(a)wesbland.com <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>>> wrote: > > > > The Fault Tolerance Working Group’s weekly con call is today at > > 12:00 PM Eastern. Today's agenda:____ > > > > __ __ > > > > * FA-MPI (Tony)____ > > > > * Other updates (All)____ > > > > __ __ > > > > If there's something else that people would like to discuss, > please > > just send an email to the WG so we can get it on the agenda.____ > > > > __ __ > > > > Thanks, ____ > > > > Wes ____ > > > > __ __ > > > > > ....................................................................................................................................... > > ____ > > > > Join from PC, Mac, Linux, iOS or Android: > > https://tennessee.zoom.us/j/632356722?pwd=lI4_169CGcewIumekTziMw____ > > > > Password: mpiforum____ > > > > __ __ > > > > Or iPhone one-tap (US Toll): +16468769923,632356722# or > > +16699006833,632356722# ____ > > > > __ __ > > > > Or Telephone:____ > > > > Dial:____ > > > > +1 646 876 9923 (US Toll)____ > > > > +1 669 900 6833 (US Toll)____ > > > > Meeting ID: 632 356 722____ > > > > International numbers available: https://zoom.us/u/6uINe____ > > > > __ __ > > > > Or an H.323/SIP room system:____ > > > > H.323: 162.255.37.11 (US West) or 162.255.36.11 (US > East) ____ > > > > Meeting ID: 632 356 722____ > > > > Password: 364216____ > > > > __ __ > > > > SIP: 632356722(a)zoomcrc.com > <mailto:[email protected]> <mailto:[email protected] > <mailto:[email protected]>>____ > > > > Password: 364216____ > > > > > .......................................................................................................................................____ > > > > > > _______________________________________________ > > mpi-forum mailing list > > mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> > > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum > > > > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft > _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft Intel Deutschland GmbH Registered Address: Am Campeon 10-12, 85579 Neubiberg, Germany Tel: +49 89 99 8853-0, www.intel.de Managing Directors: Christin Eisenschmid, Gary Kershaw Chairperson of the Supervisory Board: Nicole Lau Registered Office: Munich Commercial Register: Amtsgericht Muenchen HRB 186928

1 0

Re: [mpiwg-ft] [Mpi-forum] FTWG Con Call Today
by Ignacio Laguna 25 Jan '21

25 Jan '21

The model is that the app goes always back to the resilient_function call 1 (we cannot call this function twice or more statically in the program). Perhaps we can discuss that again. Ignacio On 1/25/21 8:25 AM, Wesley Bland via mpiwg-ft wrote: > There was another question that came up in internal conversations around > here with Reinit: > > What's going to happen to the call stack. E.g. MPI_Init -> ... -> > resilient_function call 1 -> Failure -> ReInit - resilient_function call > 2 -> End of Work -> back to resilient_function call 1? > > On Mon, Jan 25, 2021 at 10:01 AM Ignacio Laguna <lagunaperalt1(a)llnl.gov > <mailto:[email protected]>> wrote: > > That works for me (I couldn't attend today neither). > > We are almost done with the new Reinit spec but we have a few topics we > would like to discuss in the group: (1) using several error handlers > and > how this is specified in the standard, (2) the state of MPI between a > failure and its recovery (how does ULFM does it? Perhaps Reinit can > re-use the same text?). > > Thanks! > > Ignacio > > On 1/25/21 6:20 AM, Wesley Bland via mpi-forum wrote: > > Hi all, > > > > After talking to Tony, we're going to delay this discussion until > the > > next call on Feb 8. Today's call is cancelled. > > > > Thanks, > > Wes > > > > On Mon, Jan 25, 2021 at 8:15 AM work(a)wesbland.com > <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>> > <work(a)wesbland.com <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>>> wrote: > > > > The Fault Tolerance Working Group’s weekly con call is today at > > 12:00 PM Eastern. Today's agenda:____ > > > > __ __ > > > > * FA-MPI (Tony)____ > > > > * Other updates (All)____ > > > > __ __ > > > > If there's something else that people would like to discuss, > please > > just send an email to the WG so we can get it on the agenda.____ > > > > __ __ > > > > Thanks, ____ > > > > Wes ____ > > > > __ __ > > > > > ....................................................................................................................................... > > ____ > > > > Join from PC, Mac, Linux, iOS or Android: > > https://tennessee.zoom.us/j/632356722?pwd=lI4_169CGcewIumekTziMw____ > > > > Password: mpiforum____ > > > > __ __ > > > > Or iPhone one-tap (US Toll): +16468769923,632356722# or > > +16699006833,632356722# ____ > > > > __ __ > > > > Or Telephone:____ > > > > Dial:____ > > > > +1 646 876 9923 (US Toll)____ > > > > +1 669 900 6833 (US Toll)____ > > > > Meeting ID: 632 356 722____ > > > > International numbers available: https://zoom.us/u/6uINe____ > > > > __ __ > > > > Or an H.323/SIP room system:____ > > > > H.323: 162.255.37.11 (US West) or 162.255.36.11 (US > East) ____ > > > > Meeting ID: 632 356 722____ > > > > Password: 364216____ > > > > __ __ > > > > SIP: 632356722(a)zoomcrc.com > <mailto:[email protected]> <mailto:[email protected] > <mailto:[email protected]>>____ > > > > Password: 364216____ > > > > > .......................................................................................................................................____ > > > > > > _______________________________________________ > > mpi-forum mailing list > > mpi-forum(a)lists.mpi-forum.org <mailto:[email protected]> > > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum > > > > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft >

1 0

Re: [mpiwg-ft] [Mpi-forum] FTWG Con Call Today
by Wesley Bland 25 Jan '21

25 Jan '21

There was another question that came up in internal conversations around here with Reinit: What's going to happen to the call stack. E.g. MPI_Init -> ... -> resilient_function call 1 -> Failure -> ReInit - resilient_function call 2 -> End of Work -> back to resilient_function call 1? On Mon, Jan 25, 2021 at 10:01 AM Ignacio Laguna <lagunaperalt1(a)llnl.gov> wrote: > That works for me (I couldn't attend today neither). > > We are almost done with the new Reinit spec but we have a few topics we > would like to discuss in the group: (1) using several error handlers and > how this is specified in the standard, (2) the state of MPI between a > failure and its recovery (how does ULFM does it? Perhaps Reinit can > re-use the same text?). > > Thanks! > > Ignacio > > On 1/25/21 6:20 AM, Wesley Bland via mpi-forum wrote: > > Hi all, > > > > After talking to Tony, we're going to delay this discussion until the > > next call on Feb 8. Today's call is cancelled. > > > > Thanks, > > Wes > > > > On Mon, Jan 25, 2021 at 8:15 AM work(a)wesbland.com > > <mailto:[email protected]> <work(a)wesbland.com > > <mailto:[email protected]>> wrote: > > > > The Fault Tolerance Working Group’s weekly con call is today at > > 12:00 PM Eastern. Today's agenda:____ > > > > __ __ > > > > * FA-MPI (Tony)____ > > > > * Other updates (All)____ > > > > __ __ > > > > If there's something else that people would like to discuss, please > > just send an email to the WG so we can get it on the agenda.____ > > > > __ __ > > > > Thanks, ____ > > > > Wes ____ > > > > __ __ > > > > > ....................................................................................................................................... > > ____ > > > > Join from PC, Mac, Linux, iOS or Android: > > https://tennessee.zoom.us/j/632356722?pwd=lI4_169CGcewIumekTziMw____ > > > > Password: mpiforum____ > > > > __ __ > > > > Or iPhone one-tap (US Toll): +16468769923,632356722# or > > +16699006833,632356722# ____ > > > > __ __ > > > > Or Telephone:____ > > > > Dial:____ > > > > +1 646 876 9923 (US Toll)____ > > > > +1 669 900 6833 (US Toll)____ > > > > Meeting ID: 632 356 722____ > > > > International numbers available: https://zoom.us/u/6uINe____ > > > > __ __ > > > > Or an H.323/SIP room system:____ > > > > H.323: 162.255.37.11 (US West) or 162.255.36.11 (US East) ____ > > > > Meeting ID: 632 356 722____ > > > > Password: 364216____ > > > > __ __ > > > > SIP: 632356722(a)zoomcrc.com <mailto:[email protected]>____ > > > > Password: 364216____ > > > > > .......................................................................................................................................____ > > > > > > _______________________________________________ > > mpi-forum mailing list > > mpi-forum(a)lists.mpi-forum.org > > https://lists.mpi-forum.org/mailman/listinfo/mpi-forum > > >

1 0