February 2021 - mpiwg-hybridpm - mailman3.cels.anl.gov

Re: [mpiwg-hybridpm] Call for Topics
by Jim Dinan 10 Feb '21

10 Feb '21

Hi All, We don't have a set agenda, but let's meet this morning to check in with open topics and collect any new topics of interest. See you at 10am ET. Connection details are on the wiki: https://github.com/mpiwg-hybrid/hybrid-issues/wiki Cheers, ~Jim. On Tue, Feb 9, 2021 at 2:09 PM Jim Dinan <james.dinan(a)gmail.com> wrote: > Hi All, > > The Hybrid & Accelerator WG will meet tomorrow. The agenda so far is > empty. Please let me know if you have any topics that you would like to > discuss. > > Cheers, > ~Jim. >

1 0

Re: [mpiwg-hybridpm] Call for Topics
by Zhang, Junchao 09 Feb '21

09 Feb '21

OK, I agree with that. I have thought you meant MPI_Recv(gpubuf,..); cudaDeviceSynchronize(); Kernel<<<>>>(gpubuf,..); In my code, I assume cudaDeviceSynchronize() is not needed after recv, but is needed before send. Thanks --Junchao Zhang On Feb 9, 2021, at 9:23 PM, Choi, Jaemin <jchoi157(a)illinois.edu<mailto:[email protected]>> wrote: I think an example is when you want to invoke a GPU kernel that depends on the arriving data. Currently the host code has to wait until the MPI_Recv is complete before it is able to invoke the kernel, which can be avoided if CUDA streams are supported in MPI. Jaemin Choi PhD Candidate in Computer Science Research Assistant at the Parallel Programming Laboratory University of Illinois Urbana-Champaign From: Zhang, Junchao <jczhang(a)mcs.anl.gov<mailto:[email protected]>> Sent: Tuesday, February 9, 2021 9:09 PM To: Hybrid working group mailing list <mpiwg-hybridpm(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: Jim Dinan <james.dinan(a)gmail.com<mailto:[email protected]>>; Choi, Jaemin <jchoi157(a)illinois.edu<mailto:[email protected]>> Subject: Re: [mpiwg-hybridpm] Call for Topics Is the host-side synchronization after communication needed? I thought no. Thanks --Junchao Zhang On Feb 9, 2021, at 8:19 PM, Choi, Jaemin via mpiwg-hybridpm <mpiwg-hybridpm(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Thanks Jim, I must have missed the discussions from the last couple of meetings. We’ve recently implemented GPU-aware communication in Charm++ and Adaptive MPI using UCX, and just started to look into how we can avoid the host-side synchronizations before/after communication. UCX supporting CUDA streams could be a solution, but I also wanted to explore other options as well. I wasn’t aware of libmp, I will definitely have a look at that and NCCL to see how streams are integrated there. Best, Jaemin Choi PhD Candidate in Computer Science Research Assistant at the Parallel Programming Laboratory University of Illinois Urbana-Champaign From: Jim Dinan <james.dinan(a)gmail.com<mailto:[email protected]>> Sent: Tuesday, February 9, 2021 6:08 PM To: Choi, Jaemin <jchoi157(a)illinois.edu<mailto:[email protected]>> Cc: Hybrid working group mailing list <mpiwg-hybridpm(a)lists.mpi-forum.org<mailto:[email protected]>> Subject: Re: [mpiwg-hybridpm] Call for Topics We have been discussing this topic in the working group, and would be glad to have your input. If you're looking for something you can use today, NCCL is a good option. There's also a library called libmp that uses GPUDirect Async for stream-based communication [1]. ~Jim. [1] https://github.com/gpudirect/libmp<https://urldefense.com/v3/__https:/github.com/gpudirect/libmp__;!!DZ3fjg!q1…> On Tue, Feb 9, 2021 at 3:09 PM Choi, Jaemin <jchoi157(a)illinois.edu<mailto:[email protected]>> wrote: Hi Jim, Do you happen to know if there has been progress on supporting CUDA streams in the MPI standard? I’d be interested to hear about this or other frameworks (e.g. NCCL) that support asynchronous communication via streams. Best, Jaemin Choi PhD Candidate in Computer Science Research Assistant at the Parallel Programming Laboratory University of Illinois Urbana-Champaign From: mpiwg-hybridpm <mpiwg-hybridpm-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> On Behalf Of Jim Dinan via mpiwg-hybridpm Sent: Tuesday, February 9, 2021 1:10 PM To: Hybrid working group mailing list <mpiwg-hybridpm(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: Jim Dinan <james.dinan(a)gmail.com<mailto:[email protected]>> Subject: [mpiwg-hybridpm] Call for Topics Hi All, The Hybrid & Accelerator WG will meet tomorrow. The agenda so far is empty. Please let me know if you have any topics that you would like to discuss. Cheers, ~Jim. _______________________________________________ mpiwg-hybridpm mailing list mpiwg-hybridpm(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-hybridpm<https://urldefense.com/v3/__https:/lists.mpi-forum.org/mailman/listinfo/mpi…>

1 0

Re: [mpiwg-hybridpm] Call for Topics
by Choi, Jaemin 09 Feb '21

09 Feb '21

I think an example is when you want to invoke a GPU kernel that depends on the arriving data. Currently the host code has to wait until the MPI_Recv is complete before it is able to invoke the kernel, which can be avoided if CUDA streams are supported in MPI. Jaemin Choi PhD Candidate in Computer Science Research Assistant at the Parallel Programming Laboratory University of Illinois Urbana-Champaign From: Zhang, Junchao <jczhang(a)mcs.anl.gov> Sent: Tuesday, February 9, 2021 9:09 PM To: Hybrid working group mailing list <mpiwg-hybridpm(a)lists.mpi-forum.org> Cc: Jim Dinan <james.dinan(a)gmail.com>; Choi, Jaemin <jchoi157(a)illinois.edu> Subject: Re: [mpiwg-hybridpm] Call for Topics Is the host-side synchronization after communication needed? I thought no. Thanks --Junchao Zhang On Feb 9, 2021, at 8:19 PM, Choi, Jaemin via mpiwg-hybridpm <mpiwg-hybridpm(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Thanks Jim, I must have missed the discussions from the last couple of meetings. We’ve recently implemented GPU-aware communication in Charm++ and Adaptive MPI using UCX, and just started to look into how we can avoid the host-side synchronizations before/after communication. UCX supporting CUDA streams could be a solution, but I also wanted to explore other options as well. I wasn’t aware of libmp, I will definitely have a look at that and NCCL to see how streams are integrated there. Best, Jaemin Choi PhD Candidate in Computer Science Research Assistant at the Parallel Programming Laboratory University of Illinois Urbana-Champaign From: Jim Dinan <james.dinan(a)gmail.com<mailto:[email protected]>> Sent: Tuesday, February 9, 2021 6:08 PM To: Choi, Jaemin <jchoi157(a)illinois.edu<mailto:[email protected]>> Cc: Hybrid working group mailing list <mpiwg-hybridpm(a)lists.mpi-forum.org<mailto:[email protected]>> Subject: Re: [mpiwg-hybridpm] Call for Topics We have been discussing this topic in the working group, and would be glad to have your input. If you're looking for something you can use today, NCCL is a good option. There's also a library called libmp that uses GPUDirect Async for stream-based communication [1]. ~Jim. [1] https://github.com/gpudirect/libmp<https://urldefense.com/v3/__https:/github.com/gpudirect/libmp__;!!DZ3fjg!q1…> On Tue, Feb 9, 2021 at 3:09 PM Choi, Jaemin <jchoi157(a)illinois.edu<mailto:[email protected]>> wrote: Hi Jim, Do you happen to know if there has been progress on supporting CUDA streams in the MPI standard? I’d be interested to hear about this or other frameworks (e.g. NCCL) that support asynchronous communication via streams. Best, Jaemin Choi PhD Candidate in Computer Science Research Assistant at the Parallel Programming Laboratory University of Illinois Urbana-Champaign From: mpiwg-hybridpm <mpiwg-hybridpm-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> On Behalf Of Jim Dinan via mpiwg-hybridpm Sent: Tuesday, February 9, 2021 1:10 PM To: Hybrid working group mailing list <mpiwg-hybridpm(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: Jim Dinan <james.dinan(a)gmail.com<mailto:[email protected]>> Subject: [mpiwg-hybridpm] Call for Topics Hi All, The Hybrid & Accelerator WG will meet tomorrow. The agenda so far is empty. Please let me know if you have any topics that you would like to discuss. Cheers, ~Jim. _______________________________________________ mpiwg-hybridpm mailing list mpiwg-hybridpm(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-hybridpm<https://urldefense.com/v3/__https:/lists.mpi-forum.org/mailman/listinfo/mpi…>

1 0

Re: [mpiwg-hybridpm] Call for Topics
by Zhang, Junchao 09 Feb '21

09 Feb '21

Is the host-side synchronization after communication needed? I thought no. Thanks --Junchao Zhang On Feb 9, 2021, at 8:19 PM, Choi, Jaemin via mpiwg-hybridpm <mpiwg-hybridpm(a)lists.mpi-forum.org<mailto:[email protected]>> wrote: Thanks Jim, I must have missed the discussions from the last couple of meetings. We’ve recently implemented GPU-aware communication in Charm++ and Adaptive MPI using UCX, and just started to look into how we can avoid the host-side synchronizations before/after communication. UCX supporting CUDA streams could be a solution, but I also wanted to explore other options as well. I wasn’t aware of libmp, I will definitely have a look at that and NCCL to see how streams are integrated there. Best, Jaemin Choi PhD Candidate in Computer Science Research Assistant at the Parallel Programming Laboratory University of Illinois Urbana-Champaign From: Jim Dinan <james.dinan(a)gmail.com<mailto:[email protected]>> Sent: Tuesday, February 9, 2021 6:08 PM To: Choi, Jaemin <jchoi157(a)illinois.edu<mailto:[email protected]>> Cc: Hybrid working group mailing list <mpiwg-hybridpm(a)lists.mpi-forum.org<mailto:[email protected]>> Subject: Re: [mpiwg-hybridpm] Call for Topics We have been discussing this topic in the working group, and would be glad to have your input. If you're looking for something you can use today, NCCL is a good option. There's also a library called libmp that uses GPUDirect Async for stream-based communication [1]. ~Jim. [1] https://github.com/gpudirect/libmp<https://urldefense.com/v3/__https:/github.com/gpudirect/libmp__;!!DZ3fjg!q1…> On Tue, Feb 9, 2021 at 3:09 PM Choi, Jaemin <jchoi157(a)illinois.edu<mailto:[email protected]>> wrote: Hi Jim, Do you happen to know if there has been progress on supporting CUDA streams in the MPI standard? I’d be interested to hear about this or other frameworks (e.g. NCCL) that support asynchronous communication via streams. Best, Jaemin Choi PhD Candidate in Computer Science Research Assistant at the Parallel Programming Laboratory University of Illinois Urbana-Champaign From: mpiwg-hybridpm <mpiwg-hybridpm-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> On Behalf Of Jim Dinan via mpiwg-hybridpm Sent: Tuesday, February 9, 2021 1:10 PM To: Hybrid working group mailing list <mpiwg-hybridpm(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: Jim Dinan <james.dinan(a)gmail.com<mailto:[email protected]>> Subject: [mpiwg-hybridpm] Call for Topics Hi All, The Hybrid & Accelerator WG will meet tomorrow. The agenda so far is empty. Please let me know if you have any topics that you would like to discuss. Cheers, ~Jim. _______________________________________________ mpiwg-hybridpm mailing list mpiwg-hybridpm(a)lists.mpi-forum.org<mailto:[email protected]> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-hybridpm

1 0

Re: [mpiwg-hybridpm] Call for Topics
by Choi, Jaemin 09 Feb '21

09 Feb '21

Thanks Jim, I must have missed the discussions from the last couple of meetings. We’ve recently implemented GPU-aware communication in Charm++ and Adaptive MPI using UCX, and just started to look into how we can avoid the host-side synchronizations before/after communication. UCX supporting CUDA streams could be a solution, but I also wanted to explore other options as well. I wasn’t aware of libmp, I will definitely have a look at that and NCCL to see how streams are integrated there. Best, Jaemin Choi PhD Candidate in Computer Science Research Assistant at the Parallel Programming Laboratory University of Illinois Urbana-Champaign From: Jim Dinan <james.dinan(a)gmail.com> Sent: Tuesday, February 9, 2021 6:08 PM To: Choi, Jaemin <jchoi157(a)illinois.edu> Cc: Hybrid working group mailing list <mpiwg-hybridpm(a)lists.mpi-forum.org> Subject: Re: [mpiwg-hybridpm] Call for Topics We have been discussing this topic in the working group, and would be glad to have your input. If you're looking for something you can use today, NCCL is a good option. There's also a library called libmp that uses GPUDirect Async for stream-based communication [1]. ~Jim. [1] https://github.com/gpudirect/libmp<https://urldefense.com/v3/__https:/github.com/gpudirect/libmp__;!!DZ3fjg!q1…> On Tue, Feb 9, 2021 at 3:09 PM Choi, Jaemin <jchoi157(a)illinois.edu<mailto:[email protected]>> wrote: Hi Jim, Do you happen to know if there has been progress on supporting CUDA streams in the MPI standard? I’d be interested to hear about this or other frameworks (e.g. NCCL) that support asynchronous communication via streams. Best, Jaemin Choi PhD Candidate in Computer Science Research Assistant at the Parallel Programming Laboratory University of Illinois Urbana-Champaign From: mpiwg-hybridpm <mpiwg-hybridpm-bounces(a)lists.mpi-forum.org<mailto:[email protected]>> On Behalf Of Jim Dinan via mpiwg-hybridpm Sent: Tuesday, February 9, 2021 1:10 PM To: Hybrid working group mailing list <mpiwg-hybridpm(a)lists.mpi-forum.org<mailto:[email protected]>> Cc: Jim Dinan <james.dinan(a)gmail.com<mailto:[email protected]>> Subject: [mpiwg-hybridpm] Call for Topics Hi All, The Hybrid & Accelerator WG will meet tomorrow. The agenda so far is empty. Please let me know if you have any topics that you would like to discuss. Cheers, ~Jim.

1 0

Re: [mpiwg-hybridpm] Call for Topics
by Jim Dinan 09 Feb '21

09 Feb '21

We have been discussing this topic in the working group, and would be glad to have your input. If you're looking for something you can use today, NCCL is a good option. There's also a library called libmp that uses GPUDirect Async for stream-based communication [1]. ~Jim. [1] https://github.com/gpudirect/libmp On Tue, Feb 9, 2021 at 3:09 PM Choi, Jaemin <jchoi157(a)illinois.edu> wrote: > Hi Jim, > > > > Do you happen to know if there has been progress on supporting CUDA > streams in the MPI standard? > > I’d be interested to hear about this or other frameworks (e.g. NCCL) that > support asynchronous communication via streams. > > > > Best, > > > > Jaemin Choi > > PhD Candidate in Computer Science > > Research Assistant at the Parallel Programming Laboratory > > University of Illinois Urbana-Champaign > > > > *From:* mpiwg-hybridpm <mpiwg-hybridpm-bounces(a)lists.mpi-forum.org> *On > Behalf Of *Jim Dinan via mpiwg-hybridpm > *Sent:* Tuesday, February 9, 2021 1:10 PM > *To:* Hybrid working group mailing list < > mpiwg-hybridpm(a)lists.mpi-forum.org> > *Cc:* Jim Dinan <james.dinan(a)gmail.com> > *Subject:* [mpiwg-hybridpm] Call for Topics > > > > Hi All, > > > > The Hybrid & Accelerator WG will meet tomorrow. The agenda so far is > empty. Please let me know if you have any topics that you would like to > discuss. > > > > Cheers, > > ~Jim. >

1 0

Re: [mpiwg-hybridpm] Call for Topics
by Choi, Jaemin 09 Feb '21

09 Feb '21

Hi Jim, Do you happen to know if there has been progress on supporting CUDA streams in the MPI standard? I’d be interested to hear about this or other frameworks (e.g. NCCL) that support asynchronous communication via streams. Best, Jaemin Choi PhD Candidate in Computer Science Research Assistant at the Parallel Programming Laboratory University of Illinois Urbana-Champaign From: mpiwg-hybridpm <mpiwg-hybridpm-bounces(a)lists.mpi-forum.org> On Behalf Of Jim Dinan via mpiwg-hybridpm Sent: Tuesday, February 9, 2021 1:10 PM To: Hybrid working group mailing list <mpiwg-hybridpm(a)lists.mpi-forum.org> Cc: Jim Dinan <james.dinan(a)gmail.com> Subject: [mpiwg-hybridpm] Call for Topics Hi All, The Hybrid & Accelerator WG will meet tomorrow. The agenda so far is empty. Please let me know if you have any topics that you would like to discuss. Cheers, ~Jim.

1 0

Call for Topics
by Jim Dinan 09 Feb '21

09 Feb '21

Hi All, The Hybrid & Accelerator WG will meet tomorrow. The agenda so far is empty. Please let me know if you have any topics that you would like to discuss. Cheers, ~Jim.

1 0