March 2022 - mpiwg-hybridpm - mailman3.cels.anl.gov

Cholesky code with "continuation"
by Joachim Protze 30 Mar '22

30 Mar '22

Hi Dan, Benson, HACC-wg, The paper I referred to in the virtual meeting was the Euro-MPI'20 paper about MPI_Detach: https://dl.acm.org/doi/abs/10.1145/3416315.3416323 In the paper we also sketch, how continuations can be used to implement C++ code like: https://github.com/mpi-forum/mpi-issues/issues/288#issuecomment-619053687 The Cholesky code with MPI_Detach aka Continuation: The communication task in https://github.com/RWTH-HPC/cholesky_omptasks/blob/mpi-detach/detach-deps/c… finishes execution immediately after calling MPI_Detach in https://github.com/RWTH-HPC/cholesky_omptasks/blob/mpi-detach/detach-deps/c… Because of the detach clause on the task (OpenMP 5.0 feature), the task only completes, when the callback is called. Releasing the task dependencies depends on task completion. Effectively, this allows to span task dependence graphs across MPI ranks. A copy of my implementation of MPI_Detach as a wrapper (detach.cpp) is next to this file. If you want to try and run the code, make sure to export MPIX_DETACH=progress, so that the wrapper starts the progress thread. The Cholesky code without detach: The communication task explicitly waits on completion of the MPI communication: https://github.com/RWTH-HPC/cholesky_omptasks/blob/mpi-detach/fine-deps/ch_… in this wait function: https://github.com/RWTH-HPC/cholesky_omptasks/blob/mpi-detach/fine-deps/ch_… Implemented in here: https://github.com/RWTH-HPC/cholesky_omptasks/blob/mpi-detach/fine-deps/ch_… The taskyield allows other tasks to get scheduled while waiting the MPI communication to complete. Unfortunately, the OpenMP semantics of taskyield are as weak as the progress guarantees from the MPI side ;) As a result, this code can deadlock, if too many communication tasks are created and only the OMPSS runtime can actually avoid the deadlock. The other versions of the code reduce the possible concurrency more and more, avoiding deadlocks, but increasing the execution time. Just recently, I did an extended performance analysis of the code, which shows that the overhead for the detach version displayed in the paper is purely a result of increasing data transfer with increasing number of nodes, amplified by the reduced amount of work per rank due to strong scaling. I.e., at some point there is not enough work available to overlap communication. Best Joachim -- Dr. rer. nat. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpiwg-hybridpm] HACC Meeting This Week
by Dan Holmes 30 Mar '22

30 Mar '22

Hi Jim, I have an alpha version/preview of my upcoming presentation for next week’s virtual forum meeting (topic: refinement of MPI operations stages), which relates strongly with the discussion on MPI_PREPARE. I’d like to socialise that linkage in the HACC WG and hopefully move the discussion on that topic forward (or at least onward in several complementary directions at once). Cheers, Dan. — Dr Daniel Holmes PhD Executive Director Chief Technology Officer CHI Ltd danholmes(a)chi.scot > On 29 Mar 2022, at 15:54, Jim Dinan via mpiwg-hybridpm <mpiwg-hybridpm(a)lists.mpi-forum.org> wrote: > > Hi All, > > Please let me know if you have any topics for this week. We had initially scheduled Joseph's continuations topic, but there is a virtual meeting on the same topic right afterward so it doesn't make much sense to meet twice in a row on the same topic. > > In the absence of a fixed agenda, I'd like to suggest that we convene for informal updates on all of the active topics in the WG. > > Thanks, > ~Jim. > _______________________________________________ > mpiwg-hybridpm mailing list > mpiwg-hybridpm(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-hybridpm

1 0

HACC Meeting This Week
by Jim Dinan 29 Mar '22

29 Mar '22

Hi All, Please let me know if you have any topics for this week. We had initially scheduled Joseph's continuations topic, but there is a virtual meeting on the same topic right afterward so it doesn't make much sense to meet twice in a row on the same topic. In the absence of a fixed agenda, I'd like to suggest that we convene for informal updates on all of the active topics in the WG. Thanks, ~Jim.

1 0

Re: [mpiwg-hybridpm] No Meeting Today
by Joseph Schuchart 23 Mar '22

23 Mar '22

Next week sounds good, yes. Thanks, Joseph On 3/23/22 10:07, Jim Dinan wrote: > Hi Joseph, > > Sorry that I missed your message. If it's helpful, we can discuss your > topic next Wednesday before the virtual meeting. Please let me know. > > Best, > ~Jim. > > On Wed, Mar 16, 2022 at 10:32 AM Joseph Schuchart via mpiwg-hybridpm > <mpiwg-hybridpm(a)lists.mpi-forum.org> wrote: > > I'd like to discuss persistent continuations. I have a slot at the > virtual meeting in 2 weeks (March 30) and thought it might be good to > talk this through in the WG before taking it back to the wider Forum. > > Cheers > Joseph > > On 3/16/22 10:29, Jim Dinan via mpiwg-hybridpm wrote: > > Hi All, > > > > Apologies, I forgot to set up this week's meeting. Please send > topics > > you'd like to cover at our next meeting. > > > > Cheers, > > ~Jim. > > > > _______________________________________________ > > mpiwg-hybridpm mailing list > > mpiwg-hybridpm(a)lists.mpi-forum.org > > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-hybridpm > > _______________________________________________ > mpiwg-hybridpm mailing list > mpiwg-hybridpm(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-hybridpm >

1 0

Re: [mpiwg-hybridpm] No Meeting This Week
by Jim Dinan 23 Mar '22

23 Mar '22

Hi Benson, This sounds like a great topic. Do you have someone in mind to lead such a discussion? Best, ~Jim. On Tue, Mar 22, 2022 at 9:25 AM Benson Muite via mpiwg-hybridpm < mpiwg-hybridpm(a)lists.mpi-forum.org> wrote: > It may be helpful to have some discussion on SmartNICs/DPUs. > > Some related links: > https://github.com/tangjianback/DPU_MPI > https://github.com/beanfacts/dpu_mpi > > On 3/22/22 3:42 PM, Jim Dinan via mpiwg-hybridpm wrote: > > Hi All, > > The HACC WG is off this week and persistence WG will use the Wednesday > slot. > > Please let me know if you have a topic you'd like to cover next week. > > Best, > ~Jim. > > PS - Hope to see you at NVIDIA GTC, which is free to attend and kicks off > today! > > _______________________________________________ > mpiwg-hybridpm mailing [email protected]://lists.mpi-forum.org/mailman/listinfo/mpiwg-hybridpm > > > _______________________________________________ > mpiwg-hybridpm mailing list > mpiwg-hybridpm(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-hybridpm >

1 0

Re: [mpiwg-hybridpm] No Meeting Today
by Jim Dinan 23 Mar '22

23 Mar '22

Hi Joseph, Sorry that I missed your message. If it's helpful, we can discuss your topic next Wednesday before the virtual meeting. Please let me know. Best, ~Jim. On Wed, Mar 16, 2022 at 10:32 AM Joseph Schuchart via mpiwg-hybridpm < mpiwg-hybridpm(a)lists.mpi-forum.org> wrote: > I'd like to discuss persistent continuations. I have a slot at the > virtual meeting in 2 weeks (March 30) and thought it might be good to > talk this through in the WG before taking it back to the wider Forum. > > Cheers > Joseph > > On 3/16/22 10:29, Jim Dinan via mpiwg-hybridpm wrote: > > Hi All, > > > > Apologies, I forgot to set up this week's meeting. Please send topics > > you'd like to cover at our next meeting. > > > > Cheers, > > ~Jim. > > > > _______________________________________________ > > mpiwg-hybridpm mailing list > > mpiwg-hybridpm(a)lists.mpi-forum.org > > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-hybridpm > > _______________________________________________ > mpiwg-hybridpm mailing list > mpiwg-hybridpm(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-hybridpm >

1 0

Re: [mpiwg-hybridpm] No Meeting This Week
by Benson Muite 22 Mar '22

22 Mar '22

It may be helpful to have some discussion on SmartNICs/DPUs. Some related links: https://github.com/tangjianback/DPU_MPI https://github.com/beanfacts/dpu_mpi On 3/22/22 3:42 PM, Jim Dinan via mpiwg-hybridpm wrote: > Hi All, > > The HACC WG is off this week and persistence WG will use the Wednesday > slot. > > Please let me know if you have a topic you'd like to cover next week. > > Best, > ~Jim. > > PS - Hope to see you at NVIDIA GTC, which is free to attend and kicks > off today! > > _______________________________________________ > mpiwg-hybridpm mailing list > mpiwg-hybridpm(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-hybridpm

1 0

No Meeting This Week
by Jim Dinan 22 Mar '22

22 Mar '22

Hi All, The HACC WG is off this week and persistence WG will use the Wednesday slot. Please let me know if you have a topic you'd like to cover next week. Best, ~Jim. PS - Hope to see you at NVIDIA GTC, which is free to attend and kicks off today!

1 0

Re: [mpiwg-hybridpm] No Meeting Today
by Joseph Schuchart 16 Mar '22

16 Mar '22

I'd like to discuss persistent continuations. I have a slot at the virtual meeting in 2 weeks (March 30) and thought it might be good to talk this through in the WG before taking it back to the wider Forum. Cheers Joseph On 3/16/22 10:29, Jim Dinan via mpiwg-hybridpm wrote: > Hi All, > > Apologies, I forgot to set up this week's meeting. Please send topics > you'd like to cover at our next meeting. > > Cheers, > ~Jim. > > _______________________________________________ > mpiwg-hybridpm mailing list > mpiwg-hybridpm(a)lists.mpi-forum.org > https://lists.mpi-forum.org/mailman/listinfo/mpiwg-hybridpm

1 0

No Meeting Today
by Jim Dinan 16 Mar '22

16 Mar '22

Hi All, Apologies, I forgot to set up this week's meeting. Please send topics you'd like to cover at our next meeting. Cheers, ~Jim.

1 0