December 2016 - mpiwg-ft - mailman3.cels.anl.gov

Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today
by Van Der Wijngaart, Rob F 21 Dec '16

21 Dec '16

That is a great idea, Keita, and I will be happy to participate to explore the relative areas of strength of these respective models. I think the first steps should be: · define a (small) collection of workloads that represent major MPI usage models, such as: master worker, SPMD, small number of communicators with mostly point-to-point communications, many different communicators with mostly collective operations, strictly iterative (identical iterations) versus semi-iterative (same iterations, but data set sizes change due to Adaptive Mesh Refinement, for example), versus non-iterative (some graph problems, such as betweenness centrality), multi-physics models, hybrid MPI/threading models, one-sided communications, applications that use external libraries. Of course, these models are not mutually exclusive. We shouldn’t go crazy here, but make sure we have a sufficient basis for evaluation, and then parameterize these workloads. · Define various fault modes to be covered. Rob -----Original Message----- From: mpiwg-ft-bounces(a)lists.mpi-forum.org [mailto:[email protected]] On Behalf Of Teranishi, Keita Sent: Tuesday, December 20, 2016 11:27 PM To: ilaguna(a)llnl.gov; MPI WG Fault Tolerance and Dynamic Process Control working Group <mpiwg-ft(a)lists.mpi-forum.org>; Bland, Wesley <wesley.bland(a)intel.com> Subject: Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today Ignacio, Yes, ReInit and Fenix-1.0 have the same recovery model. They use longjump for global rollback and fix MPI communicator at the end of "Init² call. I am very happy to perform the feasibility studies of these three (plus one) models. I think that it will be great if we can explore the feasibility through some empirical (prototyping) studies. As for 4th (ReInit/Fenix-1.0) model, we should have clear definition MPI communicator recovery including subcommunicators. In order to utilize the next generation checkpoint library (ECP¹s multi-level checkpointing project) or accommodate application specific recovery schemes, MPI_Comm should provide some information of its past (history failures or change in the rank, comm_size, etc.) as well as its current state. I am hoping that our experience with Fenix will help to design a new spec. Thanks, --------------------------------------------------------------------------- -- Keita Teranishi Principal Member of Technical Staff Scalable Modeling and Analysis Systems Sandia National Laboratories Livermore, CA 94551 +1 (925) 294-3738 On 12/20/16, 4:55 PM, "Ignacio Laguna" <lagunaperalt1(a)llnl.gov<mailto:[email protected]>> wrote: >Hi Keita, > >I think we all agree that there is no silver bullet solution for the FT >problem and that each recovery model (whether it's ULFM, Reinit, Fenix, >or ULFM+autorecovery) works for some codes but doesn't work for others, >and that one of the solutions to cover all applications is to allow >multiple recovery models. > >In the last telecon we discussed two ways to do that: (a) all models >are compatible with each other; (b) they are not compatible, thus the >application has to select the model to be used (which implies libraries >used by the application have to support that model as well). The ideal >case is (a), but we are not sure if it's possible, thus we are going to >discuss each model in detail to explore that possibility. I believe >case >(b) is always a possibility, in which case you can still run Fenix on >top of ULFM in that situation. > >BTW, correct me if I'm wrong, but Reinit and Fenix share (at a >high-level) the same idea of global backward recovery with longjumps to >reinject execution; thus we should call the 4rth option perhaps >Reinit/Fenix. > >Ignacio > > >On 12/20/16 3:06 PM, Teranishi, Keita wrote: >> All, >> >> Throughout the discussion, I am a bit worried about making MPI bigger >> than message passing interface because I wish MPI to serve a good >> abstraction of user-friendly transport layer. Fenix is intended to >> leverage the minimalist approach of MPI-FT (ULFM today) to cover most >> of online recovery models for parallel programs using MPI. The >> current version is designed to support SPMD (Communicating Sequential >> Process) model, but we wish to support other models including >> Master-Worker, Distributed Asynchronous Many Task (AMT) and Message-Logging. >> >> ·ULFM: We have requested non-blocking communicator recovery as well as >> non-blocking comm_dup and comm_split, etc. ULFM already provides good >> mechanism to serve master-worker type recovery like UQ, model >> reduction and a certain family of eigenvalue solvers. I wish to have >> more fine control for revocation because it is possible to keep the >> certain connection of survived process (for master-worker or >> task-parallel computing), but it might be too difficult. >> >> ·ULFM + Auto recovery: I need clarification from Wesly (as my >> knowledge is wrong most likelyŠ but let me continue based on my assumption). >> Fenix assumes that failure happens at a single or a small number of >> processes. In this model, auto-recovery could serve as >> un-coordinated recovery because no comm_shrink call is used to fix the communicator. >> This could help message reply of uncoordinated recovery model. For >> example, recovery is never manifested as ³Failure² to the survived >> ranks, making particular message passing calls very slow. For SPMD >> model, adaptation is so challenging as the user needs to write how to >> recover the lost state of failed processes. However, I can see a >> great benefit for implementing resilient task parallel programming model. >> >> ·Communicator with hole: Master-Worker type applications will benefit >> from this when making collectives to gather the data available. >> >> ·MPI_ReInit: MPI_ReInit is very close to the current Fenix model. >> We have written the API specification (see attached) to support the >> same type of online recovery (global rollback upon process failure). >> The code is implemented using MPI-ULFM, and we have seen some issues >> with MPI-ULFM that makes multiple communicator recovery convoluted. >> We used PMPI to hide all the details of error handling, garbage >> collection and communicator recovery. The rollback (to Fenix_Init) is >> performed through longjmp. Nice features of Fenix are (1) an idea of >> *resilient >> communicator* that allows the users to specify which communicator >> needs to be automatically fixed and (2) *callback functions* to >> assist application-specific recovery followed by communicator >> recovery. We originally do not intend Fenix to be part of the MPI >> standard because we want the role of MPI confined within ³Message Passing² and do not want >> delay the MPI standardization discussions. My understanding with >> MPI_ReInit is standardizing online-rollback recovery and keeping >> PMPI/QMPI layer clean through a tight binding with the layers >> invisible to typical MPI users (or tool developers) --- Ignacio, >> please correct me if I am wrong. My biggest concern of MPI_ReInit is >> that defining rollback model by Message Passing Library may violate >> the original design philosophy of MPI (again this is the reason why >> we did not propose Fenix as MPI standard). Another concern is that >> it might be difficult to keep other recovery options open, but it >> gets much more flexible with a few knobs in the APIs. I think the >> latter is easy to fix with some switches in APIs. I think we can >> figure out the options as we discuss further. >> >> Thanks, >> >> Keita >> >> *From: *"Bland, Wesley" <wesley.bland(a)intel.com<mailto:[email protected]>> >> *Date: *Tuesday, December 20, 2016 at 1:48 PM >> *To: *MPI WG Fault Tolerance and Dynamic Process Control working >> Group <mpiwg-ft(a)lists.mpi-forum.org<mailto:[email protected]>>, "Teranishi, Keita" >> <knteran(a)sandia.gov<mailto:[email protected]>> >> *Subject: *Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today >> >> Probably here since we don't have an issue for this discussion. If >> you want to open issues in our working group's repository >> (github.com/mpiwg-ft/ft-issues) that's probably fine. >> >> On December 20, 2016 at 3:47:25 PM, Teranishi, Keita >> (knteran(a)sandia.gov<mailto:[email protected]> >> <mailto:[email protected]>) wrote: >> >> Wesley, >> >> Should I do here or github issues? >> >> Thanks, >> >> Keita >> >> *From: *"Bland, Wesley" <wesley.bland(a)intel.com<mailto:[email protected]>> >> *Date: *Tuesday, December 20, 2016 at 1:43 PM >> *To: *MPI WG Fault Tolerance and Dynamic Process Control working >> Group <mpiwg-ft(a)lists.mpi-forum.org<mailto:[email protected]>>, "Teranishi, Keita" >> <knteran(a)sandia.gov<mailto:[email protected]>> >> *Subject: *Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today >> >> You don't have to wait. :) If you have comments/concerns, you can >> raise them here too. >> >> On December 20, 2016 at 3:38:47 PM, Teranishi, Keita >> (knteran(a)sandia.gov<mailto:[email protected]> <mailto:[email protected]>) wrote: >> >> All, >> >> Sorry, I could not make it today. I will definitely join the >> meeting next time to make comments/suggestions on the three >> items (ULFM, ULFM+Auto, and ReInit) from Fenix perspective. >> >> Thanks, >> >> Keita >> >> *From: *<mpiwg-ft-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> on behalf of >> "Bland, Wesley" <wesley.bland(a)intel.com<mailto:[email protected]>> >> *Reply-To: *MPI WG Fault Tolerance and Dynamic Process Control >> working Group <mpiwg-ft(a)lists.mpi-forum.org<mailto:[email protected]>> >> *Date: *Tuesday, December 20, 2016 at 1:29 PM >> *To: *FTWG <mpiwg-ft(a)lists.mpi-forum.org<mailto:[email protected]>> >> *Subject: *[EXTERNAL] Re: [mpiwg-ft] FTWG Con Call Today >> >> The notes from today's call are posted on the wiki: >> >> https://github.com/mpiwg-ft/ft-issues/wiki/2016-12-20 >> >> Those who have specific items, please make progress on those >> between now and our next meeting. We will be cancelling the Jan >> 3 call due to the holiday. The next call will be on Jan 17. >> >> Thanks, >> >> Wesley >> >> On December 20, 2016 at 8:15:06 AM, Bland, Wesley >> (wesley.bland(a)intel.com<mailto:[email protected]> <mailto:[email protected]>) wrote: >> >> The Fault Tolerance Working Group¹s biweekly con call is >> today at 3:00 PM Eastern. Today's agenda: >> >> * Recap of face to face meeting >> >> * Go over existing tickets >> >> * Discuss concerns with ULFM and path forward >> >> Thanks, >> >> Wesley >> >> >>......................................................................... >>................................................................ >> >> Join online meeting >> <https://meet.intel.com/wesley.bland/GHHKQ79Y> >> >> https://meet.intel.com/wesley.bland/GHHKQ79Y >> >> Join by Phone >> >> +1(916)356-2663 (or your local bridge access #) Choose >>bridge 5. >> >> Find a local number <https://dial.intel.com> >> >> Conference ID: 757343533 >> >> Forgot your dial-in PIN? <https://dial.intel.com> | First >> online meeting? >> >><http://r.office.microsoft.com/r/rlidOC10?clid=1033&p1=4&p2=1041&pc=oc >>&ve >>r=4&subver=0&bld=7185&bldver=0> >> >> >>......................................................................... >>................................................................ >> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org<mailto:[email protected]> >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft >> >> >> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org<mailto:[email protected]> >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft >> _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft

1 0

Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today
by Teranishi, Keita 21 Dec '16

21 Dec '16

Ignacio, Yes, ReInit and Fenix-1.0 have the same recovery model. They use longjump for global rollback and fix MPI communicator at the end of "Init² call. I am very happy to perform the feasibility studies of these three (plus one) models. I think that it will be great if we can explore the feasibility through some empirical (prototyping) studies. As for 4th (ReInit/Fenix-1.0) model, we should have clear definition MPI communicator recovery including subcommunicators. In order to utilize the next generation checkpoint library (ECP¹s multi-level checkpointing project) or accommodate application specific recovery schemes, MPI_Comm should provide some information of its past (history failures or change in the rank, comm_size, etc.) as well as its current state. I am hoping that our experience with Fenix will help to design a new spec. Thanks, --------------------------------------------------------------------------- -- Keita Teranishi Principal Member of Technical Staff Scalable Modeling and Analysis Systems Sandia National Laboratories Livermore, CA 94551 +1 (925) 294-3738 On 12/20/16, 4:55 PM, "Ignacio Laguna" <lagunaperalt1(a)llnl.gov> wrote: >Hi Keita, > >I think we all agree that there is no silver bullet solution for the FT >problem and that each recovery model (whether it's ULFM, Reinit, Fenix, >or ULFM+autorecovery) works for some codes but doesn't work for others, >and that one of the solutions to cover all applications is to allow >multiple recovery models. > >In the last telecon we discussed two ways to do that: (a) all models are >compatible with each other; (b) they are not compatible, thus the >application has to select the model to be used (which implies libraries >used by the application have to support that model as well). The ideal >case is (a), but we are not sure if it's possible, thus we are going to >discuss each model in detail to explore that possibility. I believe case >(b) is always a possibility, in which case you can still run Fenix on >top of ULFM in that situation. > >BTW, correct me if I'm wrong, but Reinit and Fenix share (at a >high-level) the same idea of global backward recovery with longjumps to >reinject execution; thus we should call the 4rth option perhaps >Reinit/Fenix. > >Ignacio > > >On 12/20/16 3:06 PM, Teranishi, Keita wrote: >> All, >> >> Throughout the discussion, I am a bit worried about making MPI bigger >> than message passing interface because I wish MPI to serve a good >> abstraction of user-friendly transport layer. Fenix is intended to >> leverage the minimalist approach of MPI-FT (ULFM today) to cover most of >> online recovery models for parallel programs using MPI. The current >> version is designed to support SPMD (Communicating Sequential Process) >> model, but we wish to support other models including Master-Worker, >> Distributed Asynchronous Many Task (AMT) and Message-Logging. >> >> ·ULFM: We have requested non-blocking communicator recovery as well as >> non-blocking comm_dup and comm_split, etc. ULFM already provides good >> mechanism to serve master-worker type recovery like UQ, model reduction >> and a certain family of eigenvalue solvers. I wish to have more fine >> control for revocation because it is possible to keep the certain >> connection of survived process (for master-worker or task-parallel >> computing), but it might be too difficult. >> >> ·ULFM + Auto recovery: I need clarification from Wesly (as my knowledge >> is wrong most likelyŠ but let me continue based on my assumption). >> Fenix assumes that failure happens at a single or a small number of >> processes. In this model, auto-recovery could serve as un-coordinated >> recovery because no comm_shrink call is used to fix the communicator. >> This could help message reply of uncoordinated recovery model. For >> example, recovery is never manifested as ³Failure² to the survived >> ranks, making particular message passing calls very slow. For SPMD >> model, adaptation is so challenging as the user needs to write how to >> recover the lost state of failed processes. However, I can see a great >> benefit for implementing resilient task parallel programming model. >> >> ·Communicator with hole: Master-Worker type applications will benefit >> from this when making collectives to gather the data available. >> >> ·MPI_ReInit: MPI_ReInit is very close to the current Fenix model. We >> have written the API specification (see attached) to support the same >> type of online recovery (global rollback upon process failure). The >> code is implemented using MPI-ULFM, and we have seen some issues with >> MPI-ULFM that makes multiple communicator recovery convoluted. We used >> PMPI to hide all the details of error handling, garbage collection and >> communicator recovery. The rollback (to Fenix_Init) is performed through >> longjmp. Nice features of Fenix are (1) an idea of *resilient >> communicator* that allows the users to specify which communicator needs >> to be automatically fixed and (2) *callback functions* to assist >> application-specific recovery followed by communicator recovery. We >> originally do not intend Fenix to be part of the MPI standard because we >> want the role of MPI confined within ³Message Passing² and do not want >> delay the MPI standardization discussions. My understanding with >> MPI_ReInit is standardizing online-rollback recovery and keeping >> PMPI/QMPI layer clean through a tight binding with the layers invisible >> to typical MPI users (or tool developers) --- Ignacio, please correct me >> if I am wrong. My biggest concern of MPI_ReInit is that defining >> rollback model by Message Passing Library may violate the original >> design philosophy of MPI (again this is the reason why we did not >> propose Fenix as MPI standard). Another concern is that it might be >> difficult to keep other recovery options open, but it gets much more >> flexible with a few knobs in the APIs. I think the latter is easy to >> fix with some switches in APIs. I think we can figure out the options >> as we discuss further. >> >> Thanks, >> >> Keita >> >> *From: *"Bland, Wesley" <wesley.bland(a)intel.com> >> *Date: *Tuesday, December 20, 2016 at 1:48 PM >> *To: *MPI WG Fault Tolerance and Dynamic Process Control working Group >> <mpiwg-ft(a)lists.mpi-forum.org>, "Teranishi, Keita" <knteran(a)sandia.gov> >> *Subject: *Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today >> >> Probably here since we don't have an issue for this discussion. If you >> want to open issues in our working group's repository >> (github.com/mpiwg-ft/ft-issues) that's probably fine. >> >> On December 20, 2016 at 3:47:25 PM, Teranishi, Keita (knteran(a)sandia.gov >> <mailto:[email protected]>) wrote: >> >> Wesley, >> >> Should I do here or github issues? >> >> Thanks, >> >> Keita >> >> *From: *"Bland, Wesley" <wesley.bland(a)intel.com> >> *Date: *Tuesday, December 20, 2016 at 1:43 PM >> *To: *MPI WG Fault Tolerance and Dynamic Process Control working >> Group <mpiwg-ft(a)lists.mpi-forum.org>, "Teranishi, Keita" >> <knteran(a)sandia.gov> >> *Subject: *Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today >> >> You don't have to wait. :) If you have comments/concerns, you can >> raise them here too. >> >> On December 20, 2016 at 3:38:47 PM, Teranishi, Keita >> (knteran(a)sandia.gov <mailto:[email protected]>) wrote: >> >> All, >> >> Sorry, I could not make it today. I will definitely join the >> meeting next time to make comments/suggestions on the three >> items (ULFM, ULFM+Auto, and ReInit) from Fenix perspective. >> >> Thanks, >> >> Keita >> >> *From: *<mpiwg-ft-bounces(a)lists.mpi-forum.org> on behalf of >> "Bland, Wesley" <wesley.bland(a)intel.com> >> *Reply-To: *MPI WG Fault Tolerance and Dynamic Process Control >> working Group <mpiwg-ft(a)lists.mpi-forum.org> >> *Date: *Tuesday, December 20, 2016 at 1:29 PM >> *To: *FTWG <mpiwg-ft(a)lists.mpi-forum.org> >> *Subject: *[EXTERNAL] Re: [mpiwg-ft] FTWG Con Call Today >> >> The notes from today's call are posted on the wiki: >> >> https://github.com/mpiwg-ft/ft-issues/wiki/2016-12-20 >> >> Those who have specific items, please make progress on those >> between now and our next meeting. We will be cancelling the Jan >> 3 call due to the holiday. The next call will be on Jan 17. >> >> Thanks, >> >> Wesley >> >> On December 20, 2016 at 8:15:06 AM, Bland, Wesley >> (wesley.bland(a)intel.com <mailto:[email protected]>) wrote: >> >> The Fault Tolerance Working Group¹s biweekly con call is >> today at 3:00 PM Eastern. Today's agenda: >> >> * Recap of face to face meeting >> >> * Go over existing tickets >> >> * Discuss concerns with ULFM and path forward >> >> Thanks, >> >> Wesley >> >> >>......................................................................... >>................................................................ >> >> Join online meeting >> <https://meet.intel.com/wesley.bland/GHHKQ79Y> >> >> https://meet.intel.com/wesley.bland/GHHKQ79Y >> >> Join by Phone >> >> +1(916)356-2663 (or your local bridge access #) Choose >>bridge 5. >> >> Find a local number <https://dial.intel.com> >> >> Conference ID: 757343533 >> >> Forgot your dial-in PIN? <https://dial.intel.com> | First >> online meeting? >> >><http://r.office.microsoft.com/r/rlidOC10?clid=1033&p1=4&p2=1041&pc=oc&ve >>r=4&subver=0&bld=7185&bldver=0> >> >> >>......................................................................... >>................................................................ >> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft >> >> >> >> _______________________________________________ >> mpiwg-ft mailing list >> mpiwg-ft(a)lists.mpi-forum.org >> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft >>

1 0

Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today
by Teranishi, Keita 20 Dec '16

20 Dec '16

Jeff, I admit I have been mistaken in the definition of runtime, but MPI’s has been serving a thin (but robust) runtime substrate for various programing models. Even for CSP/SPMD models, I think several successful frameworks such as PETSc and Trillions are owing to MPI’s design. Just going back to the original discussion, I hope resilience of CSP/SPMD can be done in the same philosophy and will introduce minimal additions to MPI. Thanks, ----------------------------------------------------------------------------- Keita Teranishi Principal Member of Technical Staff Scalable Modeling and Analysis Systems Sandia National Laboratories Livermore, CA 94551 +1 (925) 294-3738 From: <mpiwg-ft-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> on behalf of Jeff Hammond <jeff.science(a)gmail.com<mailto:[email protected]>> Reply-To: MPI WG Fault Tolerance and Dynamic Process Control working Group <mpiwg-ft(a)lists.mpi-forum.org<mailto:[email protected]>> Date: Tuesday, December 20, 2016 at 3:31 PM To: MPI WG Fault Tolerance and Dynamic Process Control working Group <mpiwg-ft(a)lists.mpi-forum.org<mailto:[email protected]>> Subject: Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today MPI has not been a message passing library since MPI-1. It's a runtime system for HPC that provides interprocess communication of many kinds (not just message passing - see RMA) as well as parallel file I/O. Jeff On Tue, Dec 20, 2016 at 3:06 PM, Teranishi, Keita <knteran(a)sandia.gov<mailto:[email protected]>> wrote: All, Throughout the discussion, I am a bit worried about making MPI bigger than message passing interface because I wish MPI to serve a good abstraction of user-friendly transport layer. Fenix is intended to leverage the minimalist approach of MPI-FT (ULFM today) to cover most of online recovery models for parallel programs using MPI. The current version is designed to support SPMD (Communicating Sequential Process) model, but we wish to support other models including Master-Worker, Distributed Asynchronous Many Task (AMT) and Message-Logging. · ULFM: We have requested non-blocking communicator recovery as well as non-blocking comm_dup and comm_split, etc. ULFM already provides good mechanism to serve master-worker type recovery like UQ, model reduction and a certain family of eigenvalue solvers. I wish to have more fine control for revocation because it is possible to keep the certain connection of survived process (for master-worker or task-parallel computing), but it might be too difficult. · ULFM + Auto recovery: I need clarification from Wesly (as my knowledge is wrong most likely… but let me continue based on my assumption). Fenix assumes that failure happens at a single or a small number of processes. In this model, auto-recovery could serve as un-coordinated recovery because no comm_shrink call is used to fix the communicator. This could help message reply of uncoordinated recovery model. For example, recovery is never manifested as “Failure” to the survived ranks, making particular message passing calls very slow. For SPMD model, adaptation is so challenging as the user needs to write how to recover the lost state of failed processes. However, I can see a great benefit for implementing resilient task parallel programming model. · Communicator with hole: Master-Worker type applications will benefit from this when making collectives to gather the data available. · MPI_ReInit: MPI_ReInit is very close to the current Fenix model. We have written the API specification (see attached) to support the same type of online recovery (global rollback upon process failure). The code is implemented using MPI-ULFM, and we have seen some issues with MPI-ULFM that makes multiple communicator recovery convoluted. We used PMPI to hide all the details of error handling, garbage collection and communicator recovery. The rollback (to Fenix_Init) is performed through longjmp. Nice features of Fenix are (1) an idea of resilient communicator that allows the users to specify which communicator needs to be automatically fixed and (2) callback functions to assist application-specific recovery followed by communicator recovery. We originally do not intend Fenix to be part of the MPI standard because we want the role of MPI confined within “Message Passing” and do not want delay the MPI standardization discussions. My understanding with MPI_ReInit is standardizing online-rollback recovery and keeping PMPI/QMPI layer clean through a tight binding with the layers invisible to typical MPI users (or tool developers) --- Ignacio, please correct me if I am wrong. My biggest concern of MPI_ReInit is that defining rollback model by Message Passing Library may violate the original design philosophy of MPI (again this is the reason why we did not propose Fenix as MPI standard). Another concern is that it might be difficult to keep other recovery options open, but it gets much more flexible with a few knobs in the APIs. I think the latter is easy to fix with some switches in APIs. I think we can figure out the options as we discuss further. Thanks, Keita From: "Bland, Wesley" <wesley.bland(a)intel.com<mailto:[email protected]>> Date: Tuesday, December 20, 2016 at 1:48 PM To: MPI WG Fault Tolerance and Dynamic Process Control working Group <mpiwg-ft(a)lists.mpi-forum.org<mailto:[email protected]>>, "Teranishi, Keita" <knteran(a)sandia.gov<mailto:[email protected]>> Subject: Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today Probably here since we don't have an issue for this discussion. If you want to open issues in our working group's repository (github.com/mpiwg-ft/ft-issues<http://github.com/mpiwg-ft/ft-issues>), that's probably fine. On December 20, 2016 at 3:47:25 PM, Teranishi, Keita (knteran(a)sandia.gov<mailto:[email protected]>) wrote: Wesley, Should I do here or github issues? Thanks, Keita From: "Bland, Wesley" <wesley.bland(a)intel.com<mailto:[email protected]>> Date: Tuesday, December 20, 2016 at 1:43 PM To: MPI WG Fault Tolerance and Dynamic Process Control working Group <mpiwg-ft(a)lists.mpi-forum.org<mailto:[email protected]>>, "Teranishi, Keita" <knteran(a)sandia.gov<mailto:[email protected]>> Subject: Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today You don't have to wait. :) If you have comments/concerns, you can raise them here too. On December 20, 2016 at 3:38:47 PM, Teranishi, Keita (knteran(a)sandia.gov<mailto:[email protected]>) wrote: All, Sorry, I could not make it today. I will definitely join the meeting next time to make comments/suggestions on the three items (ULFM, ULFM+Auto, and ReInit) from Fenix perspective. Thanks, Keita From: <mpiwg-ft-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> on behalf of "Bland, Wesley" <wesley.bland(a)intel.com<mailto:[email protected]>> Reply-To: MPI WG Fault Tolerance and Dynamic Process Control working Group <mpiwg-ft(a)lists.mpi-forum.org<mailto:[email protected]>> Date: Tuesday, December 20, 2016 at 1:29 PM To: FTWG <mpiwg-ft(a)lists.mpi-forum.org<mailto:[email protected]>> Subject: [EXTERNAL] Re: [mpiwg-ft] FTWG Con Call Today The notes from today's call are posted on the wiki: https://github.com/mpiwg-ft/ft-issues/wiki/2016-12-20 Those who have specific items, please make progress on those between now and our next meeting. We will be cancelling the Jan 3 call due to the holiday. The next call will be on Jan 17. Thanks, Wesley On December 20, 2016 at 8:15:06 AM, Bland, Wesley (wesley.bland(a)intel.com<mailto:[email protected]>) wrote: The Fault Tolerance Working Group’s biweekly con call is today at 3:00 PM Eastern. Today's agenda: * Recap of face to face meeting * Go over existing tickets * Discuss concerns with ULFM and path forward Thanks, Wesley ......................................................................................................................................... Join online meeting <https://meet.intel.com/wesley.bland/GHHKQ79Y> https://meet.intel.com/wesley.bland/GHHKQ79Y Join by Phone +1(916)356-2663 (or your local bridge access #) Choose bridge 5. Find a local number <https://dial.intel.com> Conference ID: 757343533 Forgot your dial-in PIN? <https://dial.intel.com> | First online meeting? <http://r.office.microsoft.com/r/rlidOC10?clid=1033&p1=4&p2=1041&pc=oc&ver=4…> ......................................................................................................................................... _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> http://jeffhammond.github.io/

1 0

Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today
by Ignacio Laguna 20 Dec '16

20 Dec '16

Hi Keita, I think we all agree that there is no silver bullet solution for the FT problem and that each recovery model (whether it's ULFM, Reinit, Fenix, or ULFM+autorecovery) works for some codes but doesn't work for others, and that one of the solutions to cover all applications is to allow multiple recovery models. In the last telecon we discussed two ways to do that: (a) all models are compatible with each other; (b) they are not compatible, thus the application has to select the model to be used (which implies libraries used by the application have to support that model as well). The ideal case is (a), but we are not sure if it's possible, thus we are going to discuss each model in detail to explore that possibility. I believe case (b) is always a possibility, in which case you can still run Fenix on top of ULFM in that situation. BTW, correct me if I'm wrong, but Reinit and Fenix share (at a high-level) the same idea of global backward recovery with longjumps to reinject execution; thus we should call the 4rth option perhaps Reinit/Fenix. Ignacio On 12/20/16 3:06 PM, Teranishi, Keita wrote: > All, > > Throughout the discussion, I am a bit worried about making MPI bigger > than message passing interface because I wish MPI to serve a good > abstraction of user-friendly transport layer. Fenix is intended to > leverage the minimalist approach of MPI-FT (ULFM today) to cover most of > online recovery models for parallel programs using MPI. The current > version is designed to support SPMD (Communicating Sequential Process) > model, but we wish to support other models including Master-Worker, > Distributed Asynchronous Many Task (AMT) and Message-Logging. > > ï¿½ULFM: We have requested non-blocking communicator recovery as well as > non-blocking comm_dup and comm_split, etc. ULFM already provides good > mechanism to serve master-worker type recovery like UQ, model reduction > and a certain family of eigenvalue solvers. I wish to have more fine > control for revocation because it is possible to keep the certain > connection of survived process (for master-worker or task-parallel > computing), but it might be too difficult. > > ï¿½ULFM + Auto recovery: I need clarification from Wesly (as my knowledge > is wrong most likelyï¿½ but let me continue based on my assumption). > Fenix assumes that failure happens at a single or a small number of > processes. In this model, auto-recovery could serve as un-coordinated > recovery because no comm_shrink call is used to fix the communicator. > This could help message reply of uncoordinated recovery model. For > example, recovery is never manifested as ï¿½Failureï¿½ to the survived > ranks, making particular message passing calls very slow. For SPMD > model, adaptation is so challenging as the user needs to write how to > recover the lost state of failed processes. However, I can see a great > benefit for implementing resilient task parallel programming model. > > ï¿½Communicator with hole: Master-Worker type applications will benefit > from this when making collectives to gather the data available. > > ï¿½MPI_ReInit: MPI_ReInit is very close to the current Fenix model. We > have written the API specification (see attached) to support the same > type of online recovery (global rollback upon process failure). The > code is implemented using MPI-ULFM, and we have seen some issues with > MPI-ULFM that makes multiple communicator recovery convoluted. We used > PMPI to hide all the details of error handling, garbage collection and > communicator recovery. The rollback (to Fenix_Init) is performed through > longjmp. Nice features of Fenix are (1) an idea of *resilient > communicator* that allows the users to specify which communicator needs > to be automatically fixed and (2) *callback functions* to assist > application-specific recovery followed by communicator recovery. We > originally do not intend Fenix to be part of the MPI standard because we > want the role of MPI confined within ï¿½Message Passingï¿½ and do not want > delay the MPI standardization discussions. My understanding with > MPI_ReInit is standardizing online-rollback recovery and keeping > PMPI/QMPI layer clean through a tight binding with the layers invisible > to typical MPI users (or tool developers) --- Ignacio, please correct me > if I am wrong. My biggest concern of MPI_ReInit is that defining > rollback model by Message Passing Library may violate the original > design philosophy of MPI (again this is the reason why we did not > propose Fenix as MPI standard). Another concern is that it might be > difficult to keep other recovery options open, but it gets much more > flexible with a few knobs in the APIs. I think the latter is easy to > fix with some switches in APIs. I think we can figure out the options > as we discuss further. > > Thanks, > > Keita > > *From: *"Bland, Wesley" <wesley.bland(a)intel.com> > *Date: *Tuesday, December 20, 2016 at 1:48 PM > *To: *MPI WG Fault Tolerance and Dynamic Process Control working Group > <mpiwg-ft(a)lists.mpi-forum.org>, "Teranishi, Keita" <knteran(a)sandia.gov> > *Subject: *Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today > > Probably here since we don't have an issue for this discussion. If you > want to open issues in our working group's repository > (github.com/mpiwg-ft/ft-issues) that's probably fine. > > On December 20, 2016 at 3:47:25 PM, Teranishi, Keita (knteran(a)sandia.gov > <mailto:[email protected]>) wrote: > > Wesley, > > Should I do here or github issues? > > Thanks, > > Keita > > *From: *"Bland, Wesley" <wesley.bland(a)intel.com> > *Date: *Tuesday, December 20, 2016 at 1:43 PM > *To: *MPI WG Fault Tolerance and Dynamic Process Control working > Group <mpiwg-ft(a)lists.mpi-forum.org>, "Teranishi, Keita" > <knteran(a)sandia.gov> > *Subject: *Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today > > You don't have to wait. :) If you have comments/concerns, you can > raise them here too. > > On December 20, 2016 at 3:38:47 PM, Teranishi, Keita > (knteran(a)sandia.gov <mailto:[email protected]>) wrote: > > All, > > Sorry, I could not make it today. I will definitely join the > meeting next time to make comments/suggestions on the three > items (ULFM, ULFM+Auto, and ReInit) from Fenix perspective. > > Thanks, > > Keita > > *From: *<mpiwg-ft-bounces(a)lists.mpi-forum.org> on behalf of > "Bland, Wesley" <wesley.bland(a)intel.com> > *Reply-To: *MPI WG Fault Tolerance and Dynamic Process Control > working Group <mpiwg-ft(a)lists.mpi-forum.org> > *Date: *Tuesday, December 20, 2016 at 1:29 PM > *To: *FTWG <mpiwg-ft(a)lists.mpi-forum.org> > *Subject: *[EXTERNAL] Re: [mpiwg-ft] FTWG Con Call Today > > The notes from today's call are posted on the wiki: > > https://github.com/mpiwg-ft/ft-issues/wiki/2016-12-20 > > Those who have specific items, please make progress on those > between now and our next meeting. We will be cancelling the Jan > 3 call due to the holiday. The next call will be on Jan 17. > > Thanks, > > Wesley > > On December 20, 2016 at 8:15:06 AM, Bland, Wesley > (wesley.bland(a)intel.com <mailto:[email protected]>) wrote: > > The Fault Tolerance Working Groupï¿½s biweekly con call is > today at 3:00 PM Eastern. Today's agenda: > > * Recap of face to face meeting > > * Go over existing tickets > > * Discuss concerns with ULFM and path forward > > Thanks, > > Wesley > > ......................................................................................................................................... > > Join online meeting > <https://meet.intel.com/wesley.bland/GHHKQ79Y> > > https://meet.intel.com/wesley.bland/GHHKQ79Y > > Join by Phone > > +1(916)356-2663 (or your local bridge access #) Choose bridge 5. > > Find a local number <https://dial.intel.com> > > Conference ID: 757343533 > > Forgot your dial-in PIN? <https://dial.intel.com> | First > online meeting? > <http://r.office.microsoft.com/r/rlidOC10?clid=1033&p1=4&p2=1041&pc=oc&ver=4…> > > ......................................................................................................................................... > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft > > > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft >

1 0

Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today
by Jeff Hammond 20 Dec '16

20 Dec '16

MPI has not been a message passing library since MPI-1. It's a runtime system for HPC that provides interprocess communication of many kinds (not just message passing - see RMA) as well as parallel file I/O. Jeff On Tue, Dec 20, 2016 at 3:06 PM, Teranishi, Keita <knteran(a)sandia.gov> wrote: > All, > > > > Throughout the discussion, I am a bit worried about making MPI bigger than > message passing interface because I wish MPI to serve a good abstraction of > user-friendly transport layer. Fenix is intended to leverage the > minimalist approach of MPI-FT (ULFM today) to cover most of online recovery > models for parallel programs using MPI. The current version is designed to > support SPMD (Communicating Sequential Process) model, but we wish to > support other models including Master-Worker, Distributed Asynchronous Many > Task (AMT) and Message-Logging. > > > > · ULFM: We have requested non-blocking communicator recovery as > well as non-blocking comm_dup and comm_split, etc. ULFM already provides > good mechanism to serve master-worker type recovery like UQ, model > reduction and a certain family of eigenvalue solvers. I wish to have more > fine control for revocation because it is possible to keep the certain > connection of survived process (for master-worker or task-parallel > computing), but it might be too difficult. > > > > · ULFM + Auto recovery: I need clarification from Wesly (as my > knowledge is wrong most likely… but let me continue based on my > assumption). Fenix assumes that failure happens at a single or a small > number of processes. In this model, auto-recovery could serve as > un-coordinated recovery because no comm_shrink call is used to fix the > communicator. This could help message reply of uncoordinated recovery > model. For example, recovery is never manifested as “Failure” to the > survived ranks, making particular message passing calls very slow. For > SPMD model, adaptation is so challenging as the user needs to write how to > recover the lost state of failed processes. However, I can see a great > benefit for implementing resilient task parallel programming model. > > > > · Communicator with hole: Master-Worker type applications will > benefit from this when making collectives to gather the data available. > > > > · MPI_ReInit: MPI_ReInit is very close to the current Fenix > model. We have written the API specification (see attached) to support the > same type of online recovery (global rollback upon process failure). The > code is implemented using MPI-ULFM, and we have seen some issues with > MPI-ULFM that makes multiple communicator recovery convoluted. We used > PMPI to hide all the details of error handling, garbage collection and > communicator recovery. The rollback (to Fenix_Init) is performed through > longjmp. Nice features of Fenix are (1) an idea of *resilient > communicator* that allows the users to specify which communicator needs > to be automatically fixed and (2) *callback functions* to assist > application-specific recovery followed by communicator recovery. We > originally do not intend Fenix to be part of the MPI standard because we > want the role of MPI confined within “Message Passing” and do not want > delay the MPI standardization discussions. My understanding with > MPI_ReInit is standardizing online-rollback recovery and keeping PMPI/QMPI > layer clean through a tight binding with the layers invisible to typical > MPI users (or tool developers) --- Ignacio, please correct me if I am > wrong. My biggest concern of MPI_ReInit is that defining rollback model by > Message Passing Library may violate the original design philosophy of MPI > (again this is the reason why we did not propose Fenix as MPI standard). > Another concern is that it might be difficult to keep other recovery > options open, but it gets much more flexible with a few knobs in the APIs. > I think the latter is easy to fix with some switches in APIs. I think we > can figure out the options as we discuss further. > > > > Thanks, > > Keita > > > > > > *From: *"Bland, Wesley" <wesley.bland(a)intel.com> > *Date: *Tuesday, December 20, 2016 at 1:48 PM > > *To: *MPI WG Fault Tolerance and Dynamic Process Control working Group < > mpiwg-ft(a)lists.mpi-forum.org>, "Teranishi, Keita" <knteran(a)sandia.gov> > *Subject: *Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today > > > > Probably here since we don't have an issue for this discussion. If you > want to open issues in our working group's repository ( > github.com/mpiwg-ft/ft-issues) that's probably fine. > > > > On December 20, 2016 at 3:47:25 PM, Teranishi, Keita (knteran(a)sandia.gov) > wrote: > > Wesley, > > > > Should I do here or github issues? > > > > Thanks, > > Keita > > > > > > *From: *"Bland, Wesley" <wesley.bland(a)intel.com> > *Date: *Tuesday, December 20, 2016 at 1:43 PM > *To: *MPI WG Fault Tolerance and Dynamic Process Control working Group < > mpiwg-ft(a)lists.mpi-forum.org>, "Teranishi, Keita" <knteran(a)sandia.gov> > *Subject: *Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today > > > > You don't have to wait. :) If you have comments/concerns, you can raise > them here too. > > > > On December 20, 2016 at 3:38:47 PM, Teranishi, Keita (knteran(a)sandia.gov) > wrote: > > All, > > > > Sorry, I could not make it today. I will definitely join the meeting next > time to make comments/suggestions on the three items (ULFM, ULFM+Auto, and > ReInit) from Fenix perspective. > > > > Thanks, > > Keita > > > > *From: *<mpiwg-ft-bounces(a)lists.mpi-forum.org> on behalf of "Bland, > Wesley" <wesley.bland(a)intel.com> > *Reply-To: *MPI WG Fault Tolerance and Dynamic Process Control working > Group <mpiwg-ft(a)lists.mpi-forum.org> > *Date: *Tuesday, December 20, 2016 at 1:29 PM > *To: *FTWG <mpiwg-ft(a)lists.mpi-forum.org> > *Subject: *[EXTERNAL] Re: [mpiwg-ft] FTWG Con Call Today > > > > The notes from today's call are posted on the wiki: > > > > https://github.com/mpiwg-ft/ft-issues/wiki/2016-12-20 > > > > Those who have specific items, please make progress on those between now > and our next meeting. We will be cancelling the Jan 3 call due to the > holiday. The next call will be on Jan 17. > > > > Thanks, > > Wesley > > > > > > On December 20, 2016 at 8:15:06 AM, Bland, Wesley (wesley.bland(a)intel.com) > wrote: > > > > The Fault Tolerance Working Group’s biweekly con call is today at 3:00 PM > Eastern. Today's agenda: > > > > * Recap of face to face meeting > > * Go over existing tickets > > * Discuss concerns with ULFM and path forward > > > > Thanks, > > Wesley > > > > ............................................................ > ............................................................ > ................. > > Join online meeting <https://meet.intel.com/wesley.bland/GHHKQ79Y> > > https://meet.intel.com/wesley.bland/GHHKQ79Y > > > > Join by Phone > > +1(916)356-2663 (or your local bridge access #) Choose bridge 5. > > Find a local number <https://dial.intel.com> > > > > Conference ID: 757343533 > > > > Forgot your dial-in PIN? <https://dial.intel.com> | First online meeting? > <http://r.office.microsoft.com/r/rlidOC10?clid=1033&p1=4& > p2=1041&pc=oc&ver=4&subver=0&bld=7185&bldver=0> > > ............................................................ > ............................................................ > ................. > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft > > > _______________________________________________ > mpiwg-ft mailing list > mpiwg-ft(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today
by Teranishi, Keita 20 Dec '16

20 Dec '16

All, Throughout the discussion, I am a bit worried about making MPI bigger than message passing interface because I wish MPI to serve a good abstraction of user-friendly transport layer. Fenix is intended to leverage the minimalist approach of MPI-FT (ULFM today) to cover most of online recovery models for parallel programs using MPI. The current version is designed to support SPMD (Communicating Sequential Process) model, but we wish to support other models including Master-Worker, Distributed Asynchronous Many Task (AMT) and Message-Logging. · ULFM: We have requested non-blocking communicator recovery as well as non-blocking comm_dup and comm_split, etc. ULFM already provides good mechanism to serve master-worker type recovery like UQ, model reduction and a certain family of eigenvalue solvers. I wish to have more fine control for revocation because it is possible to keep the certain connection of survived process (for master-worker or task-parallel computing), but it might be too difficult. · ULFM + Auto recovery: I need clarification from Wesly (as my knowledge is wrong most likely… but let me continue based on my assumption). Fenix assumes that failure happens at a single or a small number of processes. In this model, auto-recovery could serve as un-coordinated recovery because no comm_shrink call is used to fix the communicator. This could help message reply of uncoordinated recovery model. For example, recovery is never manifested as “Failure” to the survived ranks, making particular message passing calls very slow. For SPMD model, adaptation is so challenging as the user needs to write how to recover the lost state of failed processes. However, I can see a great benefit for implementing resilient task parallel programming model. · Communicator with hole: Master-Worker type applications will benefit from this when making collectives to gather the data available. · MPI_ReInit: MPI_ReInit is very close to the current Fenix model. We have written the API specification (see attached) to support the same type of online recovery (global rollback upon process failure). The code is implemented using MPI-ULFM, and we have seen some issues with MPI-ULFM that makes multiple communicator recovery convoluted. We used PMPI to hide all the details of error handling, garbage collection and communicator recovery. The rollback (to Fenix_Init) is performed through longjmp. Nice features of Fenix are (1) an idea of resilient communicator that allows the users to specify which communicator needs to be automatically fixed and (2) callback functions to assist application-specific recovery followed by communicator recovery. We originally do not intend Fenix to be part of the MPI standard because we want the role of MPI confined within “Message Passing” and do not want delay the MPI standardization discussions. My understanding with MPI_ReInit is standardizing online-rollback recovery and keeping PMPI/QMPI layer clean through a tight binding with the layers invisible to typical MPI users (or tool developers) --- Ignacio, please correct me if I am wrong. My biggest concern of MPI_ReInit is that defining rollback model by Message Passing Library may violate the original design philosophy of MPI (again this is the reason why we did not propose Fenix as MPI standard). Another concern is that it might be difficult to keep other recovery options open, but it gets much more flexible with a few knobs in the APIs. I think the latter is easy to fix with some switches in APIs. I think we can figure out the options as we discuss further. Thanks, Keita From: "Bland, Wesley" <wesley.bland(a)intel.com> Date: Tuesday, December 20, 2016 at 1:48 PM To: MPI WG Fault Tolerance and Dynamic Process Control working Group <mpiwg-ft(a)lists.mpi-forum.org>, "Teranishi, Keita" <knteran(a)sandia.gov> Subject: Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today Probably here since we don't have an issue for this discussion. If you want to open issues in our working group's repository (github.com/mpiwg-ft/ft-issues) that's probably fine. On December 20, 2016 at 3:47:25 PM, Teranishi, Keita (knteran(a)sandia.gov<mailto:[email protected]>) wrote: Wesley, Should I do here or github issues? Thanks, Keita From: "Bland, Wesley" <wesley.bland(a)intel.com> Date: Tuesday, December 20, 2016 at 1:43 PM To: MPI WG Fault Tolerance and Dynamic Process Control working Group <mpiwg-ft(a)lists.mpi-forum.org>, "Teranishi, Keita" <knteran(a)sandia.gov> Subject: Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today You don't have to wait. :) If you have comments/concerns, you can raise them here too. On December 20, 2016 at 3:38:47 PM, Teranishi, Keita (knteran(a)sandia.gov<mailto:[email protected]>) wrote: All, Sorry, I could not make it today. I will definitely join the meeting next time to make comments/suggestions on the three items (ULFM, ULFM+Auto, and ReInit) from Fenix perspective. Thanks, Keita From: <mpiwg-ft-bounces(a)lists.mpi-forum.org> on behalf of "Bland, Wesley" <wesley.bland(a)intel.com> Reply-To: MPI WG Fault Tolerance and Dynamic Process Control working Group <mpiwg-ft(a)lists.mpi-forum.org> Date: Tuesday, December 20, 2016 at 1:29 PM To: FTWG <mpiwg-ft(a)lists.mpi-forum.org> Subject: [EXTERNAL] Re: [mpiwg-ft] FTWG Con Call Today The notes from today's call are posted on the wiki: https://github.com/mpiwg-ft/ft-issues/wiki/2016-12-20 Those who have specific items, please make progress on those between now and our next meeting. We will be cancelling the Jan 3 call due to the holiday. The next call will be on Jan 17. Thanks, Wesley On December 20, 2016 at 8:15:06 AM, Bland, Wesley (wesley.bland(a)intel.com<mailto:[email protected]>) wrote: The Fault Tolerance Working Group’s biweekly con call is today at 3:00 PM Eastern. Today's agenda: * Recap of face to face meeting * Go over existing tickets * Discuss concerns with ULFM and path forward Thanks, Wesley ......................................................................................................................................... Join online meeting <https://meet.intel.com/wesley.bland/GHHKQ79Y> https://meet.intel.com/wesley.bland/GHHKQ79Y Join by Phone +1(916)356-2663 (or your local bridge access #) Choose bridge 5. Find a local number <https://dial.intel.com> Conference ID: 757343533 Forgot your dial-in PIN? <https://dial.intel.com> | First online meeting? <http://r.office.microsoft.com/r/rlidOC10?clid=1033&p1=4&p2=1041&pc=oc&ver=4…> ......................................................................................................................................... _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft

1 0

Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today
by Bland, Wesley 20 Dec '16

20 Dec '16

Probably here since we don't have an issue for this discussion. If you want to open issues in our working group's repository (github.com/mpiwg-ft/ft-issues) that's probably fine. On December 20, 2016 at 3:47:25 PM, Teranishi, Keita (knteran(a)sandia.gov<mailto:[email protected]>) wrote: Wesley, Should I do here or github issues? Thanks, Keita From: "Bland, Wesley" <wesley.bland(a)intel.com> Date: Tuesday, December 20, 2016 at 1:43 PM To: MPI WG Fault Tolerance and Dynamic Process Control working Group <mpiwg-ft(a)lists.mpi-forum.org>, "Teranishi, Keita" <knteran(a)sandia.gov> Subject: Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today You don't have to wait. :) If you have comments/concerns, you can raise them here too. On December 20, 2016 at 3:38:47 PM, Teranishi, Keita (knteran(a)sandia.gov<mailto:[email protected]>) wrote: All, Sorry, I could not make it today. I will definitely join the meeting next time to make comments/suggestions on the three items (ULFM, ULFM+Auto, and ReInit) from Fenix perspective. Thanks, Keita From: <mpiwg-ft-bounces(a)lists.mpi-forum.org> on behalf of "Bland, Wesley" <wesley.bland(a)intel.com> Reply-To: MPI WG Fault Tolerance and Dynamic Process Control working Group <mpiwg-ft(a)lists.mpi-forum.org> Date: Tuesday, December 20, 2016 at 1:29 PM To: FTWG <mpiwg-ft(a)lists.mpi-forum.org> Subject: [EXTERNAL] Re: [mpiwg-ft] FTWG Con Call Today The notes from today's call are posted on the wiki: https://github.com/mpiwg-ft/ft-issues/wiki/2016-12-20 Those who have specific items, please make progress on those between now and our next meeting. We will be cancelling the Jan 3 call due to the holiday. The next call will be on Jan 17. Thanks, Wesley On December 20, 2016 at 8:15:06 AM, Bland, Wesley (wesley.bland(a)intel.com<mailto:[email protected]>) wrote: The Fault Tolerance Working Group’s biweekly con call is today at 3:00 PM Eastern. Today's agenda: * Recap of face to face meeting * Go over existing tickets * Discuss concerns with ULFM and path forward Thanks, Wesley ......................................................................................................................................... Join online meeting <https://meet.intel.com/wesley.bland/GHHKQ79Y> https://meet.intel.com/wesley.bland/GHHKQ79Y Join by Phone +1(916)356-2663 (or your local bridge access #) Choose bridge 5. Find a local number <https://dial.intel.com> Conference ID: 757343533 Forgot your dial-in PIN? <https://dial.intel.com> | First online meeting? <http://r.office.microsoft.com/r/rlidOC10?clid=1033&p1=4&p2=1041&pc=oc&ver=4…> ......................................................................................................................................... _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft

1 0

Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today
by Teranishi, Keita 20 Dec '16

20 Dec '16

Wesley, Should I do here or github issues? Thanks, Keita From: "Bland, Wesley" <wesley.bland(a)intel.com> Date: Tuesday, December 20, 2016 at 1:43 PM To: MPI WG Fault Tolerance and Dynamic Process Control working Group <mpiwg-ft(a)lists.mpi-forum.org>, "Teranishi, Keita" <knteran(a)sandia.gov> Subject: Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today You don't have to wait. :) If you have comments/concerns, you can raise them here too. On December 20, 2016 at 3:38:47 PM, Teranishi, Keita (knteran(a)sandia.gov<mailto:[email protected]>) wrote: All, Sorry, I could not make it today. I will definitely join the meeting next time to make comments/suggestions on the three items (ULFM, ULFM+Auto, and ReInit) from Fenix perspective. Thanks, Keita From: <mpiwg-ft-bounces(a)lists.mpi-forum.org> on behalf of "Bland, Wesley" <wesley.bland(a)intel.com> Reply-To: MPI WG Fault Tolerance and Dynamic Process Control working Group <mpiwg-ft(a)lists.mpi-forum.org> Date: Tuesday, December 20, 2016 at 1:29 PM To: FTWG <mpiwg-ft(a)lists.mpi-forum.org> Subject: [EXTERNAL] Re: [mpiwg-ft] FTWG Con Call Today The notes from today's call are posted on the wiki: https://github.com/mpiwg-ft/ft-issues/wiki/2016-12-20 Those who have specific items, please make progress on those between now and our next meeting. We will be cancelling the Jan 3 call due to the holiday. The next call will be on Jan 17. Thanks, Wesley On December 20, 2016 at 8:15:06 AM, Bland, Wesley (wesley.bland(a)intel.com<mailto:[email protected]>) wrote: The Fault Tolerance Working Group’s biweekly con call is today at 3:00 PM Eastern. Today's agenda: * Recap of face to face meeting * Go over existing tickets * Discuss concerns with ULFM and path forward Thanks, Wesley ......................................................................................................................................... Join online meeting <https://meet.intel.com/wesley.bland/GHHKQ79Y> https://meet.intel.com/wesley.bland/GHHKQ79Y Join by Phone +1(916)356-2663 (or your local bridge access #) Choose bridge 5. Find a local number <https://dial.intel.com> Conference ID: 757343533 Forgot your dial-in PIN? <https://dial.intel.com> | First online meeting? <http://r.office.microsoft.com/r/rlidOC10?clid=1033&p1=4&p2=1041&pc=oc&ver=4…> ......................................................................................................................................... _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft

1 0

Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today
by Bland, Wesley 20 Dec '16

20 Dec '16

You don't have to wait. :) If you have comments/concerns, you can raise them here too. On December 20, 2016 at 3:38:47 PM, Teranishi, Keita (knteran(a)sandia.gov<mailto:[email protected]>) wrote: All, Sorry, I could not make it today. I will definitely join the meeting next time to make comments/suggestions on the three items (ULFM, ULFM+Auto, and ReInit) from Fenix perspective. Thanks, Keita From: <mpiwg-ft-bounces(a)lists.mpi-forum.org> on behalf of "Bland, Wesley" <wesley.bland(a)intel.com> Reply-To: MPI WG Fault Tolerance and Dynamic Process Control working Group <mpiwg-ft(a)lists.mpi-forum.org> Date: Tuesday, December 20, 2016 at 1:29 PM To: FTWG <mpiwg-ft(a)lists.mpi-forum.org> Subject: [EXTERNAL] Re: [mpiwg-ft] FTWG Con Call Today The notes from today's call are posted on the wiki: https://github.com/mpiwg-ft/ft-issues/wiki/2016-12-20 Those who have specific items, please make progress on those between now and our next meeting. We will be cancelling the Jan 3 call due to the holiday. The next call will be on Jan 17. Thanks, Wesley On December 20, 2016 at 8:15:06 AM, Bland, Wesley (wesley.bland(a)intel.com<mailto:[email protected]>) wrote: The Fault Tolerance Working Group’s biweekly con call is today at 3:00 PM Eastern. Today's agenda: * Recap of face to face meeting * Go over existing tickets * Discuss concerns with ULFM and path forward Thanks, Wesley ......................................................................................................................................... Join online meeting <https://meet.intel.com/wesley.bland/GHHKQ79Y> https://meet.intel.com/wesley.bland/GHHKQ79Y Join by Phone +1(916)356-2663 (or your local bridge access #) Choose bridge 5. Find a local number <https://dial.intel.com> Conference ID: 757343533 Forgot your dial-in PIN? <https://dial.intel.com> | First online meeting? <http://r.office.microsoft.com/r/rlidOC10?clid=1033&p1=4&p2=1041&pc=oc&ver=4…> ......................................................................................................................................... _______________________________________________ mpiwg-ft mailing list mpiwg-ft(a)lists.mpi-forum.org https://lists.mpi-forum.org/mailman/listinfo/mpiwg-ft

1 0

Re: [mpiwg-ft] [EXTERNAL] Re: FTWG Con Call Today
by Teranishi, Keita 20 Dec '16

20 Dec '16

All, Sorry, I could not make it today. I will definitely join the meeting next time to make comments/suggestions on the three items (ULFM, ULFM+Auto, and ReInit) from Fenix perspective. Thanks, Keita From: <mpiwg-ft-bounces(a)lists.mpi-forum.org> on behalf of "Bland, Wesley" <wesley.bland(a)intel.com> Reply-To: MPI WG Fault Tolerance and Dynamic Process Control working Group <mpiwg-ft(a)lists.mpi-forum.org> Date: Tuesday, December 20, 2016 at 1:29 PM To: FTWG <mpiwg-ft(a)lists.mpi-forum.org> Subject: [EXTERNAL] Re: [mpiwg-ft] FTWG Con Call Today The notes from today's call are posted on the wiki: https://github.com/mpiwg-ft/ft-issues/wiki/2016-12-20 Those who have specific items, please make progress on those between now and our next meeting. We will be cancelling the Jan 3 call due to the holiday. The next call will be on Jan 17. Thanks, Wesley On December 20, 2016 at 8:15:06 AM, Bland, Wesley (wesley.bland(a)intel.com<mailto:[email protected]>) wrote: The Fault Tolerance Working Group’s biweekly con call is today at 3:00 PM Eastern. Today's agenda: * Recap of face to face meeting * Go over existing tickets * Discuss concerns with ULFM and path forward Thanks, Wesley ......................................................................................................................................... Join online meeting <https://meet.intel.com/wesley.bland/GHHKQ79Y> https://meet.intel.com/wesley.bland/GHHKQ79Y Join by Phone +1(916)356-2663 (or your local bridge access #) Choose bridge 5. Find a local number <https://dial.intel.com> Conference ID: 757343533 Forgot your dial-in PIN? <https://dial.intel.com> | First online meeting? <http://r.office.microsoft.com/r/rlidOC10?clid=1033&p1=4&p2=1041&pc=oc&ver=4…> .........................................................................................................................................

1 0