April 2021 - Ecp-aurora-sdk-discuss - mailman3.cels.anl.gov

Re: [Ecp-aurora-sdk-discuss] Upcoming oneapi update
by Steinbrecher, Patrick 12 Apr '21

12 Apr '21

> Kokkos added to oneapi modulefile. The new included Kokkos module supports now hierarchical parallelism and is based one latest Kokkos develop + some more features. All required compiler flags should be inherited if you setup build via CMake. The module has OpenMP Target as default execution space. The host OpenMP and Serial execution space are active as well. Credit to Rahul Gayatri for his work on the OpenMP Target backend. From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Chan-nui, Christopher via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Date: Monday, April 12, 2021 at 15:32 To: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] Upcoming oneapi update At 6pm CDT the oneapi will be updated. Running jobs will not be affected. Default modulefiles will be updated. A number of changes will be made at that time: 1. Update to 2021.2 oneAPI + engineering compiler release. 2. Removal of old individual component modulefiles. (dpcpp, advisor, daal etc…) 3. Removal of old SDK installations. 4. Kokkos added to oneapi modulefile. This will be silently updated without changing the oneapi version number. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com

1 0

Upcoming oneapi update
by Chan-nui, Christopher 12 Apr '21

12 Apr '21

At 6pm CDT the oneapi will be updated. Running jobs will not be affected. Default modulefiles will be updated. A number of changes will be made at that time: 1. Update to 2021.2 oneAPI + engineering compiler release. 2. Removal of old individual component modulefiles. (dpcpp, advisor, daal etc…) 3. Removal of old SDK installations. 4. Kokkos added to oneapi modulefile. This will be silently updated without changing the oneapi version number. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com

1 0

Re: [Ecp-aurora-sdk-discuss] [mkl] fft normalization 1d vs 2d
by Romero, Nichols A. 12 Apr '21

12 Apr '21

Bryce, Thanks for attaching the reproducer. Many of us are at the ECP Annual Meeting. We will take a look soon and get back to you. And file a bug for you as well. From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Bryce Allen via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Reply-To: Bryce Allen <bdallen(a)uchicago.edu> Date: Saturday, April 10, 2021 at 8:17 PM To: "ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] [mkl] fft normalization 1d vs 2d Attached is a reproducer using oneMKL dft, without gtensor, along with an updated original of the working example using plain MKL. Note that this reproducer produces the same value in the position where I expect to see something close to 1 (within machine precision), from where it is one in the input, as my gtensor test. On 4/10/21 15:32, Bryce Allen via Ecp-aurora-sdk-discuss wrote: I found the FORWARD_SCALE and BACKWARD_SCALE config parameters. I checked and the default values, for both 1d and 2d, are 1 and 1, so unless I am misunderstanding, the 2d FFT should also not be doing normalization, so the result should be the same as from cuFFT. I also tried setting them to 1 (double or float depending on type used) and that doesn't change anything. So this looks like a bug to me. I created a plain MKL example, which is basically doing the same thing, but producing the expected (unnormalized) result. See attached. -Bryce On 4/10/21 11:15, Bryce Allen via Ecp-aurora-sdk-discuss wrote: Hi, I am creating test cases for a GPU fft wrapper library, that wraps cuFFT, hipFFT, and oneAPI MKL dft. For 2d and 3d tests, I am using the delta function - zero everywhere except for one point in the center of the input matrix where it has the value 1. The FFT transform of this function is ones everywhere, so it makes it particularly convenient for writing multi-d tests. When I run this test in cuFFT, it produces unnormalized result, e.g. for a 17x17 matrix as input, if I forward transform then backwards transform it, I get a 17x17 real matrix with 289 in the central (8,8) spot, and values very near zero everywhere else. With MKL dft, I get result matrix that is already normalized - the central point is already 1 or very close to 1. For 1d FFT, MKL dft produces unnormalized results when round-tripping like this. So I am confused that this is different for 2d. Is there a reason for this discrepancy? For reference, my 2d test case is here: https://github.com/bd4/gtensor/blob/pr/fft-hip-fix/tests/test_fft.cxx#L357 and the wrapper interface is here: https://github.com/bd4/gtensor/blob/pr/fft-hip-fix/include/gt-fft/backend/s… Thanks, Bryce

1 0

Re: [Ecp-aurora-sdk-discuss] [mkl] fft normalization 1d vs 2d
by Bryce Allen 10 Apr '21

10 Apr '21

Attached is a reproducer using oneMKL dft, without gtensor, along with an updated original of the working example using plain MKL. Note that this reproducer produces the same value in the position where I expect to see something close to 1 (within machine precision), from where it is one in the input, as my gtensor test. On 4/10/21 15:32, Bryce Allen via Ecp-aurora-sdk-discuss wrote: > I found the FORWARD_SCALE and BACKWARD_SCALE config parameters. I > checked and the default values, for both 1d and 2d, are 1 and 1, so > unless I am misunderstanding, the 2d FFT should also not be doing > normalization, so the result should be the same as from cuFFT. I also > tried setting them to 1 (double or float depending on type used) and > that doesn't change anything. So this looks like a bug to me. > > I created a plain MKL example, which is basically doing the same > thing, but producing the expected (unnormalized) result. See attached. > > -Bryce > > On 4/10/21 11:15, Bryce Allen via Ecp-aurora-sdk-discuss wrote: >> Hi, >> >> I am creating test cases for a GPU fft wrapper library, that wraps >> cuFFT, hipFFT, and oneAPI MKL dft. For 2d and 3d tests, I am using >> the delta function - zero everywhere except for one point in the >> center of the input matrix where it has the value 1. The FFT >> transform of this function is ones everywhere, so it makes it >> particularly convenient for writing multi-d tests. >> >> When I run this test in cuFFT, it produces unnormalized result, e.g. >> for a 17x17 matrix as input, if I forward transform then backwards >> transform it, I get a 17x17 real matrix with 289 in the central (8,8) >> spot, and values very near zero everywhere else. With MKL dft, I get >> result matrix that is already normalized - the central point is >> already 1 or very close to 1. >> >> For 1d FFT, MKL dft produces unnormalized results when round-tripping >> like this. So I am confused that this is different for 2d. Is there a >> reason for this discrepancy? >> >> For reference, my 2d test case is here: >> https://github.com/bd4/gtensor/blob/pr/fft-hip-fix/tests/test_fft.cxx#L357 >> >> and the wrapper interface is here: >> https://github.com/bd4/gtensor/blob/pr/fft-hip-fix/include/gt-fft/backend/s… >> >> >> Thanks, >> Bryce >

1 0

Re: [Ecp-aurora-sdk-discuss] [mkl] fft normalization 1d vs 2d
by Bryce Allen 10 Apr '21

10 Apr '21

I found the FORWARD_SCALE and BACKWARD_SCALE config parameters. I checked and the default values, for both 1d and 2d, are 1 and 1, so unless I am misunderstanding, the 2d FFT should also not be doing normalization, so the result should be the same as from cuFFT. I also tried setting them to 1 (double or float depending on type used) and that doesn't change anything. So this looks like a bug to me. I created a plain MKL example, which is basically doing the same thing, but producing the expected (unnormalized) result. See attached. -Bryce On 4/10/21 11:15, Bryce Allen via Ecp-aurora-sdk-discuss wrote: > Hi, > > I am creating test cases for a GPU fft wrapper library, that wraps > cuFFT, hipFFT, and oneAPI MKL dft. For 2d and 3d tests, I am using the > delta function - zero everywhere except for one point in the center of > the input matrix where it has the value 1. The FFT transform of this > function is ones everywhere, so it makes it particularly convenient > for writing multi-d tests. > > When I run this test in cuFFT, it produces unnormalized result, e.g. > for a 17x17 matrix as input, if I forward transform then backwards > transform it, I get a 17x17 real matrix with 289 in the central (8,8) > spot, and values very near zero everywhere else. With MKL dft, I get > result matrix that is already normalized - the central point is > already 1 or very close to 1. > > For 1d FFT, MKL dft produces unnormalized results when round-tripping > like this. So I am confused that this is different for 2d. Is there a > reason for this discrepancy? > > For reference, my 2d test case is here: > https://github.com/bd4/gtensor/blob/pr/fft-hip-fix/tests/test_fft.cxx#L357 > > and the wrapper interface is here: > https://github.com/bd4/gtensor/blob/pr/fft-hip-fix/include/gt-fft/backend/s… > > > Thanks, > Bryce

1 0

[mkl] fft normalization 1d vs 2d
by Bryce Allen 10 Apr '21

10 Apr '21

Hi, I am creating test cases for a GPU fft wrapper library, that wraps cuFFT, hipFFT, and oneAPI MKL dft. For 2d and 3d tests, I am using the delta function - zero everywhere except for one point in the center of the input matrix where it has the value 1. The FFT transform of this function is ones everywhere, so it makes it particularly convenient for writing multi-d tests. When I run this test in cuFFT, it produces unnormalized result, e.g. for a 17x17 matrix as input, if I forward transform then backwards transform it, I get a 17x17 real matrix with 289 in the central (8,8) spot, and values very near zero everywhere else. With MKL dft, I get result matrix that is already normalized - the central point is already 1 or very close to 1. For 1d FFT, MKL dft produces unnormalized results when round-tripping like this. So I am confused that this is different for 2d. Is there a reason for this discrepancy? For reference, my 2d test case is here: https://github.com/bd4/gtensor/blob/pr/fft-hip-fix/tests/test_fft.cxx#L357 and the wrapper interface is here: https://github.com/bd4/gtensor/blob/pr/fft-hip-fix/include/gt-fft/backend/s… Thanks, Bryce

1 0

Re: [Ecp-aurora-sdk-discuss] GPU Debug Support with oneapi
by Muralidharan, Servesh 05 Apr '21

05 Apr '21

Hello folks, This migration is now complete. All of the iris nodes for the jobs created henceforth support debugging on gpu at this point. On behalf of early hardware team. -- Servesh From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Muralidharan, Servesh via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Date: Friday, April 2, 2021 at 4:35 PM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] GPU Debug Support with oneapi Hello folks, This roll out is scheduled to happen early next week and will migrate to all iris nodes. On behalf of early hardware team. -- Servesh From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Muralidharan, Servesh via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Date: Monday, March 15, 2021 at 5:07 PM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] GPU Debug Support with oneapi Hello folks, The queue “iris_debug” has the necessary updates to the kernel config to enable GPU debugging. This is currently in beta testing phase so feel free to test it and give us an update should you run into any particular issue. After a week or two of testing, we will migrate the rest of the nodes in “iris” queue to have this support as well. Loading the latest oneapi module should give you `gdb-oneapi` in your environment that can be used for this purpose. On behalf of early hardware team. -- Servesh

1 0

Re: [Ecp-aurora-sdk-discuss] GPU Debug Support with oneapi
by Muralidharan, Servesh 02 Apr '21

02 Apr '21

Hello folks, This roll out is scheduled to happen early next week and will migrate to all iris nodes. On behalf of early hardware team. -- Servesh From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Muralidharan, Servesh via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Date: Monday, March 15, 2021 at 5:07 PM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] GPU Debug Support with oneapi Hello folks, The queue “iris_debug” has the necessary updates to the kernel config to enable GPU debugging. This is currently in beta testing phase so feel free to test it and give us an update should you run into any particular issue. After a week or two of testing, we will migrate the rest of the nodes in “iris” queue to have this support as well. Loading the latest oneapi module should give you `gdb-oneapi` in your environment that can be used for this purpose. On behalf of early hardware team. -- Servesh

1 0