May 2021 - Ecp-aurora-sdk-discuss - mailman3.cels.anl.gov

Re: [Ecp-aurora-sdk-discuss] which oneapi module for gpu_amd_mi50?
by Vincent R. Pascuzzi 05 May '21

05 May '21

Hi, Charles. You should be able to install MKL from a oneAPI installer in a directory writable to you. You can optionally also install the DPC++ compiler as well, or use intel/llvm. - Vince On 5/4/21 4:35 PM, Charles Leggett via Ecp-aurora-sdk-discuss wrote: > I would like to use some mkl libs on the host gpu_amd_mi50 (with the > functions running on the CPU, not the GPU), but when I try to load the > only available oneapi module, the directories it uses don’t exist: > > |> module load opensuse-oneapi/default setting CMPROOT set CMPROOT to > /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09 > Directory > /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/bin > is not accessible! Directory > /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/bin/intel64 > is not accessible! Directory > /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/vtune/2021.1-beta09/bin64 > is not accessible! Directory > /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/advisor/2021.1-beta09/bin64 > is not accessible! Directory > /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/include > is not accessible! Directory > /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/mkl/2021.1-beta09/include > is not accessible! Directory > /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/tbb/2021.1-beta09/include > is not accessible! Directory > /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/dnnl/2021.1-beta09/cpu_dpcpp_gpu_dpcpp/include > is not accessible! Directory > /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/daal/2021.1-beta09/include > is not accessible! Directory > /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/lib > is not accessible! Directory > /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/mkl/2021.1-beta09/lib/intel64 > is not accessible! Directory > /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/tbb/2021.1-beta09/lib/intel64/gcc4.8 > is not accessible! Directory > /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/dnnl/2021.1-beta09/cpu_dpcpp_gpu_dpcpp/lib > is not accessible! Directory > /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/daal/2021.1-beta09/lib/intel64 > is not accessible! Directory > /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/lib > is not accessible! Directory > /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/lib/x64 > is not accessible! Directory > /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/compiler/lib/intel64_lin > is not accessible! Directory > /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/compiler/lib > is not accessible! Directory > /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/mkl/2021.1-beta09/lib/intel64 > is not accessible! Directory > /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/tbb/2021.1-beta09/lib/intel64/gcc4.8 > is not accessible! Directory > /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/dnnl/2021.1-beta09/cpu_dpcpp_gpu_dpcpp/lib > is not accessible! Directory > /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/daal/2021.1-beta09/lib/intel64 > is not accessible! Loading opensuse-oneapi/default Loading requirement: > gcc-10.1.0-gcc-10.1.0-wygjayf opensuse-.igc/release/master-2020.09.17 > opensuse-intel_compute_runtime/release/2020.09.15.001 | > > Is there another module I should be using? A grep through |module avail| > didn’t reveal anything suggestive. > > thanks, Charles. > > -- > Charles Leggett <CGLeggett(a)lbl.gov <mailto:[email protected]>> > Lawrence Berkeley National Lab > 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 > tel: 510.495.2930 >

1 0

Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target
by Luo, Ye 04 May '21

04 May '21

Call to getOrAllocTgtPtr returned null pointer It is likely that it ran out of GPU memory Ye ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Tuesday, May 4, 2021 6:58 PM To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com> Cc: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target On Tue, May 04, 2021 at 10:37:38PM +0000, Steinbrecher, Patrick wrote: > For OpenMP offload on Intel GPU you should use the flags: > > -fiopenmp -fopenmp-targets=spir64 Yes, the full set of flags I have is: -std=gnu99 -Wall -fiopenmp -fopenmp-targets=spir64 -D__STRICT_ANSI__ -g -pg -O3 -DMPI -fopenmp -D__STRICT_ANSI__ -cc=icx > I don't know about MPI error. Looks like you are using MPI with threading. Maybe that is not supported with that module. Try to just use MPI_Init. My source code does just call MPI_Init. And it's only yarrow that gives me the error in PMPI_Init. On iris, I get a different error, later on: > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Date: Tuesday, May 4, 2021 at 17:29 > To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Subject: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > I have been building and running the openmp-offload source in XSBench: > https://github.com/ANL-CESAR/XSBench > > I can get it to build if in openmp-offload/Makefile I set > CC = /soft/compilers/intel-2020.4/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpicc > and > CFLAGS += -fopenmp -D__STRICT_ANSI__ -cc=icx > > But then I get consistently different errors when I run it on iris or yarrow. > > On yarrow, qsub -n 2 -t 5 -q yarrow runicx.sh > it terminates before producing any output: > Abort(1094799) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: > MPIR_Init_thread(666)......: > MPID_Init(922).............: > MPIDI_NM_mpi_init_hook(719): OFI addrinfo() failed (ofi_init.h:719:MPIDI_NM_mpi_init_hook:No data available) > > On iris, qsub -n 2 -t 5 -q iris runicx.sh > it gets further along in execution, as I see from output, until: > Libomptarget error: Call to getOrAllocTgtPtr returned null pointer (device failure or illegal mapping). > Libomptarget error: Call to targetDataBegin failed, abort target. > Libomptarget error: Failed to process data before launching the kernel. > Libomptarget error: Run with LIBOMPTARGET_DEBUG=4 to dump host-target pointer mappings. > unknown:68:44: Libomptarget fatal error 1: failure of target construct while offloading is mandatory > > This suggests that the problem on iris might be with the OpenMP target > pragma, but the only #pragma omp target statement in the source > code is this, and it works on other non-JLSE platforms: > #pragma omp target teams distribute parallel for\ > map(to: SD.max_num_nucs)\ > map(to: SD.num_nucs[:SD.length_num_nucs])\ > map(to: SD.concs[:SD.length_concs])\ > map(to: SD.mats[:SD.length_mats])\ > map(to: SD.unionized_energy_array[:SD.length_unionized_energy_array])\ > map(to: SD.index_grid[:SD.length_index_grid])\ > map(to: SD.nuclide_grid[:SD.length_nuclide_grid])\ > reduction(+:verification) > for( int i = 0; i < in.lookups; i++ ) > ... > > Peter > -- > Ecp-aurora-sdk-discuss mailing list > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss -- Ecp-aurora-sdk-discuss mailing list Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

Re: [Ecp-aurora-sdk-discuss] which oneapi module for gpu_amd_mi50?
by Luo, Ye 04 May '21

04 May '21

Use intel/2019 module to access MKL on mi50. Correctness and performance will be on your own. Ye ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Charles Leggett via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Tuesday, May 4, 2021 6:35 PM To: ecp-aurora-sdk-discuss(a)jlse.anl.gov <ecp-aurora-sdk-discuss(a)jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] which oneapi module for gpu_amd_mi50? I would like to use some mkl libs on the host gpu_amd_mi50 (with the functions running on the CPU, not the GPU), but when I try to load the only available oneapi module, the directories it uses don’t exist: > module load opensuse-oneapi/default setting CMPROOT set CMPROOT to /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09 Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/bin is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/bin/intel64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/vtune/2021.1-beta09/bin64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/advisor/2021.1-beta09/bin64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/include is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/mkl/2021.1-beta09/include is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/tbb/2021.1-beta09/include is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/dnnl/2021.1-beta09/cpu_dpcpp_gpu_dpcpp/include is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/daal/2021.1-beta09/include is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/lib is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/mkl/2021.1-beta09/lib/intel64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/tbb/2021.1-beta09/lib/intel64/gcc4.8 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/dnnl/2021.1-beta09/cpu_dpcpp_gpu_dpcpp/lib is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/daal/2021.1-beta09/lib/intel64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/lib is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/lib/x64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/compiler/lib/intel64_lin is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/compiler/lib is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/mkl/2021.1-beta09/lib/intel64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/tbb/2021.1-beta09/lib/intel64/gcc4.8 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/dnnl/2021.1-beta09/cpu_dpcpp_gpu_dpcpp/lib is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/daal/2021.1-beta09/lib/intel64 is not accessible! Loading opensuse-oneapi/default Loading requirement: gcc-10.1.0-gcc-10.1.0-wygjayf opensuse-.igc/release/master-2020.09.17 opensuse-intel_compute_runtime/release/2020.09.15.001 Is there another module I should be using? A grep through module avail didn’t reveal anything suggestive. thanks, Charles. -- Charles Leggett <CGLeggett(a)lbl.gov<mailto:[email protected]>> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930

1 0

Re: [Ecp-aurora-sdk-discuss] which oneapi module for gpu_amd_mi50?
by Kumaran, Kalyan 04 May '21

04 May '21

Charles, we have not made oneAPI available on the MI 50s. -Kumar On May 4, 2021, at 6:57 PM, Charles Leggett via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> wrote: hi Servesh - The oneapi module isn’t available on gpu_amd_mi50, as the directory /soft/restricted/CNDA/modulefiles doesn’t exist there. In fact, /soft/restricted is empty. cheers, Charles. On Tue, May 4, 2021 at 4:49 PM Muralidharan, Servesh <servesh(a)anl.gov<mailto:[email protected]>> wrote: Hi Charles, module load opensuse-oneapi/default This is not a maintained module and probably was used by SW dev for his builds. oneapi can be loaded in JLSE as follows, module use /soft/restricted/CNDA/modulefiles module load oneapi Now whether it actually works on gpu_amd_mi50 nodes might be an issue, since the host CPU is based of AMD Rome architecture. -- Servesh From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of Charles Leggett via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Date: Tuesday, May 4, 2021 at 6:36 PM To: ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]> <ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] which oneapi module for gpu_amd_mi50? I would like to use some mkl libs on the host gpu_amd_mi50 (with the functions running on the CPU, not the GPU), but when I try to load the only available oneapi module, the directories it uses don’t exist: > module load opensuse-oneapi/default setting CMPROOT set CMPROOT to /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09 Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/bin is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/bin/intel64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/vtune/2021.1-beta09/bin64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/advisor/2021.1-beta09/bin64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/include is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/mkl/2021.1-beta09/include is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/tbb/2021.1-beta09/include is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/dnnl/2021.1-beta09/cpu_dpcpp_gpu_dpcpp/include is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/daal/2021.1-beta09/include is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/lib is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/mkl/2021.1-beta09/lib/intel64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/tbb/2021.1-beta09/lib/intel64/gcc4.8 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/dnnl/2021.1-beta09/cpu_dpcpp_gpu_dpcpp/lib is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/daal/2021.1-beta09/lib/intel64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/lib is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/lib/x64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/compiler/lib/intel64_lin is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/compiler/lib is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/mkl/2021.1-beta09/lib/intel64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/tbb/2021.1-beta09/lib/intel64/gcc4.8 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/dnnl/2021.1-beta09/cpu_dpcpp_gpu_dpcpp/lib is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/daal/2021.1-beta09/lib/intel64 is not accessible! Loading opensuse-oneapi/default Loading requirement: gcc-10.1.0-gcc-10.1.0-wygjayf opensuse-.igc/release/master-2020.09.17 opensuse-intel_compute_runtime/release/2020.09.15.001 Is there another module I should be using? A grep through module avail didn’t reveal anything suggestive. thanks, Charles. -- Charles Leggett <CGLeggett(a)lbl.gov<mailto:[email protected]>> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930 -- Charles Leggett <CGLeggett(a)lbl.gov<mailto:[email protected]>> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930 -- Ecp-aurora-sdk-discuss mailing list Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target
by Peter McCorquodale 04 May '21

04 May '21

On Tue, May 04, 2021 at 10:37:38PM +0000, Steinbrecher, Patrick wrote: > For OpenMP offload on Intel GPU you should use the flags: > > -fiopenmp -fopenmp-targets=spir64 Yes, the full set of flags I have is: -std=gnu99 -Wall -fiopenmp -fopenmp-targets=spir64 -D__STRICT_ANSI__ -g -pg -O3 -DMPI -fopenmp -D__STRICT_ANSI__ -cc=icx > I don't know about MPI error. Looks like you are using MPI with threading. Maybe that is not supported with that module. Try to just use MPI_Init. My source code does just call MPI_Init. And it's only yarrow that gives me the error in PMPI_Init. On iris, I get a different error, later on: > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Date: Tuesday, May 4, 2021 at 17:29 > To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Subject: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > I have been building and running the openmp-offload source in XSBench: > https://github.com/ANL-CESAR/XSBench > > I can get it to build if in openmp-offload/Makefile I set > CC = /soft/compilers/intel-2020.4/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpicc > and > CFLAGS += -fopenmp -D__STRICT_ANSI__ -cc=icx > > But then I get consistently different errors when I run it on iris or yarrow. > > On yarrow, qsub -n 2 -t 5 -q yarrow runicx.sh > it terminates before producing any output: > Abort(1094799) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: > MPIR_Init_thread(666)......: > MPID_Init(922).............: > MPIDI_NM_mpi_init_hook(719): OFI addrinfo() failed (ofi_init.h:719:MPIDI_NM_mpi_init_hook:No data available) > > On iris, qsub -n 2 -t 5 -q iris runicx.sh > it gets further along in execution, as I see from output, until: > Libomptarget error: Call to getOrAllocTgtPtr returned null pointer (device failure or illegal mapping). > Libomptarget error: Call to targetDataBegin failed, abort target. > Libomptarget error: Failed to process data before launching the kernel. > Libomptarget error: Run with LIBOMPTARGET_DEBUG=4 to dump host-target pointer mappings. > unknown:68:44: Libomptarget fatal error 1: failure of target construct while offloading is mandatory > > This suggests that the problem on iris might be with the OpenMP target > pragma, but the only #pragma omp target statement in the source > code is this, and it works on other non-JLSE platforms: > #pragma omp target teams distribute parallel for\ > map(to: SD.max_num_nucs)\ > map(to: SD.num_nucs[:SD.length_num_nucs])\ > map(to: SD.concs[:SD.length_concs])\ > map(to: SD.mats[:SD.length_mats])\ > map(to: SD.unionized_energy_array[:SD.length_unionized_energy_array])\ > map(to: SD.index_grid[:SD.length_index_grid])\ > map(to: SD.nuclide_grid[:SD.length_nuclide_grid])\ > reduction(+:verification) > for( int i = 0; i < in.lookups; i++ ) > ... > > Peter > -- > Ecp-aurora-sdk-discuss mailing list > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

Re: [Ecp-aurora-sdk-discuss] which oneapi module for gpu_amd_mi50?
by Charles Leggett 04 May '21

04 May '21

hi Servesh - The oneapi module isn’t available on gpu_amd_mi50, as the directory /soft/restricted/CNDA/modulefiles doesn’t exist there. In fact, /soft/restricted is empty. cheers, Charles. On Tue, May 4, 2021 at 4:49 PM Muralidharan, Servesh <servesh(a)anl.gov> wrote: > Hi Charles, > > > > module load opensuse-oneapi/default > > This is not a maintained module and probably was used by SW dev for his > builds. > > > > oneapi can be loaded in JLSE as follows, > > module use /soft/restricted/CNDA/modulefiles > > module load oneapi > > > > Now whether it actually works on gpu_amd_mi50 nodes might be an issue, > since the host CPU is based of AMD Rome architecture. > > > > -- > > Servesh > > > > *From: *Ecp-aurora-sdk-discuss < > ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Charles > Leggett via Ecp-aurora-sdk-discuss < > ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > *Date: *Tuesday, May 4, 2021 at 6:36 PM > *To: *ecp-aurora-sdk-discuss(a)jlse.anl.gov < > ecp-aurora-sdk-discuss(a)jlse.anl.gov> > *Subject: *[Ecp-aurora-sdk-discuss] which oneapi module for gpu_amd_mi50? > > I would like to use some mkl libs on the host gpu_amd_mi50 (with the > functions running on the CPU, not the GPU), but when I try to load the only > available oneapi module, the directories it uses don’t exist: > > > module load opensuse-oneapi/default > > setting CMPROOT > > > > set CMPROOT to /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09 > > > > Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/bin is not accessible! > > Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/bin/intel64 is not accessible! > > Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/vtune/2021.1-beta09/bin64 is not accessible! > > Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/advisor/2021.1-beta09/bin64 is not accessible! > > Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/include is not accessible! > > Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/mkl/2021.1-beta09/include is not accessible! > > Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/tbb/2021.1-beta09/include is not accessible! > > Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/dnnl/2021.1-beta09/cpu_dpcpp_gpu_dpcpp/include is not accessible! > > Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/daal/2021.1-beta09/include is not accessible! > > Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/lib is not accessible! > > Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/mkl/2021.1-beta09/lib/intel64 is not accessible! > > Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/tbb/2021.1-beta09/lib/intel64/gcc4.8 is not accessible! > > Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/dnnl/2021.1-beta09/cpu_dpcpp_gpu_dpcpp/lib is not accessible! > > Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/daal/2021.1-beta09/lib/intel64 is not accessible! > > Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/lib is not accessible! > > Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/lib/x64 is not accessible! > > Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/compiler/lib/intel64_lin is not accessible! > > Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/compiler/lib is not accessible! > > Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/mkl/2021.1-beta09/lib/intel64 is not accessible! > > Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/tbb/2021.1-beta09/lib/intel64/gcc4.8 is not accessible! > > Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/dnnl/2021.1-beta09/cpu_dpcpp_gpu_dpcpp/lib is not accessible! > > Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/daal/2021.1-beta09/lib/intel64 is not accessible! > > > > Loading opensuse-oneapi/default > > Loading requirement: gcc-10.1.0-gcc-10.1.0-wygjayf opensuse-.igc/release/master-2020.09.17 > > opensuse-intel_compute_runtime/release/2020.09.15.001 > > Is there another module I should be using? A grep through module avail > didn’t reveal anything suggestive. > > thanks, Charles. > > -- > > Charles Leggett <CGLeggett(a)lbl.gov> > Lawrence Berkeley National Lab > 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 > > tel: 510.495.2930 > -- Charles Leggett <CGLeggett(a)lbl.gov> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930

1 0

Re: [Ecp-aurora-sdk-discuss] which oneapi module for gpu_amd_mi50?
by Muralidharan, Servesh 04 May '21

04 May '21

Hi Charles, module load opensuse-oneapi/default This is not a maintained module and probably was used by SW dev for his builds. oneapi can be loaded in JLSE as follows, module use /soft/restricted/CNDA/modulefiles module load oneapi Now whether it actually works on gpu_amd_mi50 nodes might be an issue, since the host CPU is based of AMD Rome architecture. -- Servesh From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Charles Leggett via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Date: Tuesday, May 4, 2021 at 6:36 PM To: ecp-aurora-sdk-discuss(a)jlse.anl.gov <ecp-aurora-sdk-discuss(a)jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] which oneapi module for gpu_amd_mi50? I would like to use some mkl libs on the host gpu_amd_mi50 (with the functions running on the CPU, not the GPU), but when I try to load the only available oneapi module, the directories it uses don’t exist: > module load opensuse-oneapi/default setting CMPROOT set CMPROOT to /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09 Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/bin is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/bin/intel64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/vtune/2021.1-beta09/bin64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/advisor/2021.1-beta09/bin64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/include is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/mkl/2021.1-beta09/include is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/tbb/2021.1-beta09/include is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/dnnl/2021.1-beta09/cpu_dpcpp_gpu_dpcpp/include is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/daal/2021.1-beta09/include is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/lib is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/mkl/2021.1-beta09/lib/intel64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/tbb/2021.1-beta09/lib/intel64/gcc4.8 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/dnnl/2021.1-beta09/cpu_dpcpp_gpu_dpcpp/lib is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/daal/2021.1-beta09/lib/intel64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/lib is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/lib/x64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/compiler/lib/intel64_lin is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/compiler/lib is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/mkl/2021.1-beta09/lib/intel64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/tbb/2021.1-beta09/lib/intel64/gcc4.8 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/dnnl/2021.1-beta09/cpu_dpcpp_gpu_dpcpp/lib is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/daal/2021.1-beta09/lib/intel64 is not accessible! Loading opensuse-oneapi/default Loading requirement: gcc-10.1.0-gcc-10.1.0-wygjayf opensuse-.igc/release/master-2020.09.17 opensuse-intel_compute_runtime/release/2020.09.15.001 Is there another module I should be using? A grep through module avail didn’t reveal anything suggestive. thanks, Charles. -- Charles Leggett <CGLeggett(a)lbl.gov<mailto:[email protected]>> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930

1 0

which oneapi module for gpu_amd_mi50?
by Charles Leggett 04 May '21

04 May '21

I would like to use some mkl libs on the host gpu_amd_mi50 (with the functions running on the CPU, not the GPU), but when I try to load the only available oneapi module, the directories it uses don’t exist: > module load opensuse-oneapi/default setting CMPROOT set CMPROOT to /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09 Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/bin is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/bin/intel64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/vtune/2021.1-beta09/bin64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/advisor/2021.1-beta09/bin64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/include is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/mkl/2021.1-beta09/include is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/tbb/2021.1-beta09/include is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/dnnl/2021.1-beta09/cpu_dpcpp_gpu_dpcpp/include is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/daal/2021.1-beta09/include is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/lib is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/mkl/2021.1-beta09/lib/intel64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/tbb/2021.1-beta09/lib/intel64/gcc4.8 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/dnnl/2021.1-beta09/cpu_dpcpp_gpu_dpcpp/lib is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/daal/2021.1-beta09/lib/intel64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/lib is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/lib/x64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/compiler/lib/intel64_lin is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/compiler/2021.1-beta09/linux/compiler/lib is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/mkl/2021.1-beta09/lib/intel64 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/tbb/2021.1-beta09/lib/intel64/gcc4.8 is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/dnnl/2021.1-beta09/cpu_dpcpp_gpu_dpcpp/lib is not accessible! Directory /soft/restricted/CNDA/sdk/2020.9.15.1/oneapi/daal/2021.1-beta09/lib/intel64 is not accessible! Loading opensuse-oneapi/default Loading requirement: gcc-10.1.0-gcc-10.1.0-wygjayf opensuse-.igc/release/master-2020.09.17 opensuse-intel_compute_runtime/release/2020.09.15.001 Is there another module I should be using? A grep through module avail didn’t reveal anything suggestive. thanks, Charles. -- Charles Leggett <CGLeggett(a)lbl.gov> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930

1 0

Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target
by Steinbrecher, Patrick 04 May '21

04 May '21

For OpenMP offload on Intel GPU you should use the flags: -fiopenmp -fopenmp-targets=spir64 I don't know about MPI error. Looks like you are using MPI with threading. Maybe that is not supported with that module. Try to just use MPI_Init. From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Date: Tuesday, May 4, 2021 at 17:29 To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] offloading with OpenMP target I have been building and running the openmp-offload source in XSBench: https://github.com/ANL-CESAR/XSBench I can get it to build if in openmp-offload/Makefile I set CC = /soft/compilers/intel-2020.4/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpicc and CFLAGS += -fopenmp -D__STRICT_ANSI__ -cc=icx But then I get consistently different errors when I run it on iris or yarrow. On yarrow, qsub -n 2 -t 5 -q yarrow runicx.sh it terminates before producing any output: Abort(1094799) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(666)......: MPID_Init(922).............: MPIDI_NM_mpi_init_hook(719): OFI addrinfo() failed (ofi_init.h:719:MPIDI_NM_mpi_init_hook:No data available) On iris, qsub -n 2 -t 5 -q iris runicx.sh it gets further along in execution, as I see from output, until: Libomptarget error: Call to getOrAllocTgtPtr returned null pointer (device failure or illegal mapping). Libomptarget error: Call to targetDataBegin failed, abort target. Libomptarget error: Failed to process data before launching the kernel. Libomptarget error: Run with LIBOMPTARGET_DEBUG=4 to dump host-target pointer mappings. unknown:68:44: Libomptarget fatal error 1: failure of target construct while offloading is mandatory This suggests that the problem on iris might be with the OpenMP target pragma, but the only #pragma omp target statement in the source code is this, and it works on other non-JLSE platforms: #pragma omp target teams distribute parallel for\ map(to: SD.max_num_nucs)\ map(to: SD.num_nucs[:SD.length_num_nucs])\ map(to: SD.concs[:SD.length_concs])\ map(to: SD.mats[:SD.length_mats])\ map(to: SD.unionized_energy_array[:SD.length_unionized_energy_array])\ map(to: SD.index_grid[:SD.length_index_grid])\ map(to: SD.nuclide_grid[:SD.length_nuclide_grid])\ reduction(+:verification) for( int i = 0; i < in.lookups; i++ ) ... Peter -- Ecp-aurora-sdk-discuss mailing list Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

offloading with OpenMP target
by Peter McCorquodale 04 May '21

04 May '21

I have been building and running the openmp-offload source in XSBench: https://github.com/ANL-CESAR/XSBench I can get it to build if in openmp-offload/Makefile I set CC = /soft/compilers/intel-2020.4/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpicc and CFLAGS += -fopenmp -D__STRICT_ANSI__ -cc=icx But then I get consistently different errors when I run it on iris or yarrow. On yarrow, qsub -n 2 -t 5 -q yarrow runicx.sh it terminates before producing any output: Abort(1094799) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(666)......: MPID_Init(922).............: MPIDI_NM_mpi_init_hook(719): OFI addrinfo() failed (ofi_init.h:719:MPIDI_NM_mpi_init_hook:No data available) On iris, qsub -n 2 -t 5 -q iris runicx.sh it gets further along in execution, as I see from output, until: Libomptarget error: Call to getOrAllocTgtPtr returned null pointer (device failure or illegal mapping). Libomptarget error: Call to targetDataBegin failed, abort target. Libomptarget error: Failed to process data before launching the kernel. Libomptarget error: Run with LIBOMPTARGET_DEBUG=4 to dump host-target pointer mappings. unknown:68:44: Libomptarget fatal error 1: failure of target construct while offloading is mandatory This suggests that the problem on iris might be with the OpenMP target pragma, but the only #pragma omp target statement in the source code is this, and it works on other non-JLSE platforms: #pragma omp target teams distribute parallel for\ map(to: SD.max_num_nucs)\ map(to: SD.num_nucs[:SD.length_num_nucs])\ map(to: SD.concs[:SD.length_concs])\ map(to: SD.mats[:SD.length_mats])\ map(to: SD.unionized_energy_array[:SD.length_unionized_energy_array])\ map(to: SD.index_grid[:SD.length_index_grid])\ map(to: SD.nuclide_grid[:SD.length_nuclide_grid])\ reduction(+:verification) for( int i = 0; i < in.lookups; i++ ) ... Peter

1 0