May 2021 - Ecp-aurora-sdk-discuss - mailman3.cels.anl.gov

Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target
by Peter McCorquodale 06 May '21

06 May '21

Colleen: Yes, when I try that on yarrow I get a lot of: error: double type is not supported on this platform error: backend compiler failed build. There are some differences between iris and arcticus. XSBench runs to completion on both, but there are a couple of problems on arcticus: - the printf statements that I put inside the offloaded OpenMP target loop do not produce any output; - I get an invalid verification checksum for the results. Peter On Thu, May 06, 2021 at 03:15:47PM +0000, Bertoni, Colleen wrote: > Thanks! I checked with John and the codes are similar, so it's likely a similar issue. > > Since yarrow doesn't have hardware support for double precision, it just emulates double precision in software. And since yarrow isn't meant to support double precision, the software emulation support isn't perfect, and even when it works, performance can be 10-100X slower. > > To see if the issue is from the double precision emulation, if you turn off the emulator and enable OMP debugging: > ``` > unset IGC_EnableDPEmulation > export LIBOMPTARGET_DEBUG=1 > ``` > the fail should be visible: > ``` > Target LEVEL0 RTL --> Error: module creation failed -- see below for details. > error: double type is not supported on this platform > error: backend compiler failed build. > ``` > > If you really want to run on yarrow, you can switch the types in xsbench to "float" and it should run, but the answers might be wrong. It might be worth it to stick on iris for now, or try arcticus. > > (Thanks to Roy (Intel) for commenting on the issue!) > > Colleen > ________________________________ > From: Peter McCorquodale <PWMcCorquodale(a)lbl.gov> > Sent: Wednesday, May 5, 2021 6:41 PM > To: Bertoni, Colleen <bertoni(a)anl.gov> > Cc: Tramm, John R. <jtramm(a)anl.gov>; Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Luo, Ye <yeluo(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > Colleen, > OK, I put a print statement at the beginning of the loop that should be > offloaded, and it never shows up in the output on yarrow. > Yes, I pulled from https://github.com/ANL-CESAR/XSBench. > > Peter > > On Wed, May 05, 2021 at 11:24:00PM +0000, Bertoni, Colleen wrote: > > Yarrow has no hardware support for double precision numbers, so it could just be very slow. On the other hand, an XSBench test from John Tramm (CC'd) is known to hang on Yarrow, and we have an open bug with Intel about it (CMPLRLLVM-24234). > > > > So 2 things: > > 1) maybe try running for 10 min, or adding a print statement to see if it's making progress. > > 2) John, could you say if the code you put in the test set is the same as what Peter is using (I think just pulled from https://github.com/ANL-CESAR/XSBench, right?) > > > > Colleen > > ________________________________ > > From: Peter McCorquodale <PWMcCorquodale(a)lbl.gov> > > Sent: Wednesday, May 5, 2021 5:30 PM > > To: Bertoni, Colleen <bertoni(a)anl.gov> > > Cc: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Luo, Ye <yeluo(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > > > If I try that with either -n 2 or -n 1, same setting for mpirun and > > qsub, then that runs fine on iris, in less than 1 minute, but on > > yarrow, it hangs somewhere in the middle of its execution, and > > exhausts its 5-minute time limit. > > > > Peter > > > > On Wed, May 05, 2021 at 09:33:39PM +0000, Bertoni, Colleen wrote: > > > Thanks for the response! Could you try > > > > > > mpirun -n ${num_process} ./XSBench-icx -m event -s small > > > > > > instead of "./XSBench-icx -m event -s small" ? Some mpi versions need the `mpirun` to launch the job. > > > > > > Colleen > > > ________________________________ > > > From: Peter McCorquodale <PWMcCorquodale(a)lbl.gov> > > > Sent: Wednesday, May 5, 2021 4:25 PM > > > To: Bertoni, Colleen <bertoni(a)anl.gov> > > > Cc: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Luo, Ye <yeluo(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > > > > > Colleen: Yes, on 2 nodes with a 5-minute time limit. > > > > > > Following Ye Luo's suggestion that I was running out of memory, I > > > tried a smaller problem size, and now it runs to completion on iris, > > > but I still get those MPI errors on yarrow and arcticus. The script is: > > > #!/bin/bash > > > source /usr/share/Modules/init/bash > > > module use /soft/restricted/CNDA/modulefiles > > > module load mpi > > > module load oneapi > > > ./XSBench-icx -m event -s small > > > > > > Peter > > > > > > On Wed, May 05, 2021 at 05:15:41PM +0000, Bertoni, Colleen wrote: > > > > Hi Peter, > > > > > > > > Could you show us `runicx.sh`? The other thing I noticed is that you're executing `qsub -n 2 -t 5 -q iris runicx.sh`, which tries to execute on 2 iris nodes with a time limit of 5 minutes. To be clear, you were trying to run across two iris nodes? > > > > > > > > I know we have a version of XSBench which runs on iris, but I think it's only been on a single iris node. > > > > > > > > Thanks, > > > > Colleen > > > > ________________________________ > > > > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Luo, Ye via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > > Sent: Tuesday, May 4, 2021 7:09 PM > > > > To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Peter McCorquodale <PWMcCorquodale(a)lbl.gov> > > > > Cc: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > > > > > > > Call to getOrAllocTgtPtr returned null pointer > > > > It is likely that it ran out of GPU memory > > > > Ye > > > > ________________________________ > > > > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > > Sent: Tuesday, May 4, 2021 6:58 PM > > > > To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com> > > > > Cc: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > > > > > > > On Tue, May 04, 2021 at 10:37:38PM +0000, Steinbrecher, Patrick wrote: > > > > > For OpenMP offload on Intel GPU you should use the flags: > > > > > > > > > > -fiopenmp -fopenmp-targets=spir64 > > > > > > > > Yes, the full set of flags I have is: > > > > -std=gnu99 -Wall -fiopenmp -fopenmp-targets=spir64 -D__STRICT_ANSI__ -g -pg -O3 -DMPI -fopenmp -D__STRICT_ANSI__ -cc=icx > > > > > > > > > I don't know about MPI error. Looks like you are using MPI with threading. Maybe that is not supported with that module. Try to just use MPI_Init. > > > > > > > > My source code does just call MPI_Init. And it's only yarrow that gives me > > > > the error in PMPI_Init. On iris, I get a different error, later on: > > > > > > > > > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > > > Date: Tuesday, May 4, 2021 at 17:29 > > > > > To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > > > Subject: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > > > > I have been building and running the openmp-offload source in XSBench: > > > > > https://github.com/ANL-CESAR/XSBench > > > > > > > > > > I can get it to build if in openmp-offload/Makefile I set > > > > > CC = /soft/compilers/intel-2020.4/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpicc > > > > > and > > > > > CFLAGS += -fopenmp -D__STRICT_ANSI__ -cc=icx > > > > > > > > > > But then I get consistently different errors when I run it on iris or yarrow. > > > > > > > > > > On yarrow, qsub -n 2 -t 5 -q yarrow runicx.sh > > > > > it terminates before producing any output: > > > > > Abort(1094799) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: > > > > > MPIR_Init_thread(666)......: > > > > > MPID_Init(922).............: > > > > > MPIDI_NM_mpi_init_hook(719): OFI addrinfo() failed (ofi_init.h:719:MPIDI_NM_mpi_init_hook:No data available) > > > > > > > > > > On iris, qsub -n 2 -t 5 -q iris runicx.sh > > > > > it gets further along in execution, as I see from output, until: > > > > > Libomptarget error: Call to getOrAllocTgtPtr returned null pointer (device failure or illegal mapping). > > > > > Libomptarget error: Call to targetDataBegin failed, abort target. > > > > > Libomptarget error: Failed to process data before launching the kernel. > > > > > Libomptarget error: Run with LIBOMPTARGET_DEBUG=4 to dump host-target pointer mappings. > > > > > unknown:68:44: Libomptarget fatal error 1: failure of target construct while offloading is mandatory > > > > > > > > > > This suggests that the problem on iris might be with the OpenMP target > > > > > pragma, but the only #pragma omp target statement in the source > > > > > code is this, and it works on other non-JLSE platforms: > > > > > #pragma omp target teams distribute parallel for\ > > > > > map(to: SD.max_num_nucs)\ > > > > > map(to: SD.num_nucs[:SD.length_num_nucs])\ > > > > > map(to: SD.concs[:SD.length_concs])\ > > > > > map(to: SD.mats[:SD.length_mats])\ > > > > > map(to: SD.unionized_energy_array[:SD.length_unionized_energy_array])\ > > > > > map(to: SD.index_grid[:SD.length_index_grid])\ > > > > > map(to: SD.nuclide_grid[:SD.length_nuclide_grid])\ > > > > > reduction(+:verification) > > > > > for( int i = 0; i < in.lookups; i++ ) > > > > > ... > > > > > > > > > > Peter > > > > > -- > > > > > Ecp-aurora-sdk-discuss mailing list > > > > > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > > > > > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss > > > > -- > > > > Ecp-aurora-sdk-discuss mailing list > > > > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > > > > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target
by Luo, Ye 06 May '21

06 May '21

On the out-of-memory issue. It is intel plugin that didn't properly printout device runtime error. In the upstream clang targeting nvidia GPUs. The error message is clear. yeluo@epyc-server:~$ ./a.out CUDA error: Error returned from cuMemAlloc CUDA error: out of memory CUDA error: Error returned from cuMemAlloc CUDA error: out of memory Libomptarget error: Call to getOrAllocTgtPtr returned null pointer (device failure or illegal mapping). Libomptarget error: Run with LIBOMPTARGET_DEBUG=4 to dump host-target pointer mappings. Libomptarget error: Source location information not present. Compile with -g or -gline-tables-only. Libomptarget fatal error 1: failure of target construct while offloading is mandatory Ye ________________________________ From: Peter McCorquodale <PWMcCorquodale(a)lbl.gov> Sent: Wednesday, May 5, 2021 4:25 PM To: Bertoni, Colleen <bertoni(a)anl.gov> Cc: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Luo, Ye <yeluo(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target Colleen: Yes, on 2 nodes with a 5-minute time limit. Following Ye Luo's suggestion that I was running out of memory, I tried a smaller problem size, and now it runs to completion on iris, but I still get those MPI errors on yarrow and arcticus. The script is: #!/bin/bash source /usr/share/Modules/init/bash module use /soft/restricted/CNDA/modulefiles module load mpi module load oneapi ./XSBench-icx -m event -s small Peter On Wed, May 05, 2021 at 05:15:41PM +0000, Bertoni, Colleen wrote: > Hi Peter, > > Could you show us `runicx.sh`? The other thing I noticed is that you're executing `qsub -n 2 -t 5 -q iris runicx.sh`, which tries to execute on 2 iris nodes with a time limit of 5 minutes. To be clear, you were trying to run across two iris nodes? > > I know we have a version of XSBench which runs on iris, but I think it's only been on a single iris node. > > Thanks, > Colleen > ________________________________ > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Luo, Ye via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Sent: Tuesday, May 4, 2021 7:09 PM > To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Peter McCorquodale <PWMcCorquodale(a)lbl.gov> > Cc: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > Call to getOrAllocTgtPtr returned null pointer > It is likely that it ran out of GPU memory > Ye > ________________________________ > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Sent: Tuesday, May 4, 2021 6:58 PM > To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com> > Cc: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > On Tue, May 04, 2021 at 10:37:38PM +0000, Steinbrecher, Patrick wrote: > > For OpenMP offload on Intel GPU you should use the flags: > > > > -fiopenmp -fopenmp-targets=spir64 > > Yes, the full set of flags I have is: > -std=gnu99 -Wall -fiopenmp -fopenmp-targets=spir64 -D__STRICT_ANSI__ -g -pg -O3 -DMPI -fopenmp -D__STRICT_ANSI__ -cc=icx > > > I don't know about MPI error. Looks like you are using MPI with threading. Maybe that is not supported with that module. Try to just use MPI_Init. > > My source code does just call MPI_Init. And it's only yarrow that gives me > the error in PMPI_Init. On iris, I get a different error, later on: > > > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > Date: Tuesday, May 4, 2021 at 17:29 > > To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > Subject: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > I have been building and running the openmp-offload source in XSBench: > > https://github.com/ANL-CESAR/XSBench > > > > I can get it to build if in openmp-offload/Makefile I set > > CC = /soft/compilers/intel-2020.4/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpicc > > and > > CFLAGS += -fopenmp -D__STRICT_ANSI__ -cc=icx > > > > But then I get consistently different errors when I run it on iris or yarrow. > > > > On yarrow, qsub -n 2 -t 5 -q yarrow runicx.sh > > it terminates before producing any output: > > Abort(1094799) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: > > MPIR_Init_thread(666)......: > > MPID_Init(922).............: > > MPIDI_NM_mpi_init_hook(719): OFI addrinfo() failed (ofi_init.h:719:MPIDI_NM_mpi_init_hook:No data available) > > > > On iris, qsub -n 2 -t 5 -q iris runicx.sh > > it gets further along in execution, as I see from output, until: > > Libomptarget error: Call to getOrAllocTgtPtr returned null pointer (device failure or illegal mapping). > > Libomptarget error: Call to targetDataBegin failed, abort target. > > Libomptarget error: Failed to process data before launching the kernel. > > Libomptarget error: Run with LIBOMPTARGET_DEBUG=4 to dump host-target pointer mappings. > > unknown:68:44: Libomptarget fatal error 1: failure of target construct while offloading is mandatory > > > > This suggests that the problem on iris might be with the OpenMP target > > pragma, but the only #pragma omp target statement in the source > > code is this, and it works on other non-JLSE platforms: > > #pragma omp target teams distribute parallel for\ > > map(to: SD.max_num_nucs)\ > > map(to: SD.num_nucs[:SD.length_num_nucs])\ > > map(to: SD.concs[:SD.length_concs])\ > > map(to: SD.mats[:SD.length_mats])\ > > map(to: SD.unionized_energy_array[:SD.length_unionized_energy_array])\ > > map(to: SD.index_grid[:SD.length_index_grid])\ > > map(to: SD.nuclide_grid[:SD.length_nuclide_grid])\ > > reduction(+:verification) > > for( int i = 0; i < in.lookups; i++ ) > > ... > > > > Peter > > -- > > Ecp-aurora-sdk-discuss mailing list > > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss > -- > Ecp-aurora-sdk-discuss mailing list > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target
by Bertoni, Colleen 06 May '21

06 May '21

Thanks! I checked with John and the codes are similar, so it's likely a similar issue. Since yarrow doesn't have hardware support for double precision, it just emulates double precision in software. And since yarrow isn't meant to support double precision, the software emulation support isn't perfect, and even when it works, performance can be 10-100X slower. To see if the issue is from the double precision emulation, if you turn off the emulator and enable OMP debugging: ``` unset IGC_EnableDPEmulation export LIBOMPTARGET_DEBUG=1 ``` the fail should be visible: ``` Target LEVEL0 RTL --> Error: module creation failed -- see below for details. error: double type is not supported on this platform error: backend compiler failed build. ``` If you really want to run on yarrow, you can switch the types in xsbench to "float" and it should run, but the answers might be wrong. It might be worth it to stick on iris for now, or try arcticus. (Thanks to Roy (Intel) for commenting on the issue!) Colleen ________________________________ From: Peter McCorquodale <PWMcCorquodale(a)lbl.gov> Sent: Wednesday, May 5, 2021 6:41 PM To: Bertoni, Colleen <bertoni(a)anl.gov> Cc: Tramm, John R. <jtramm(a)anl.gov>; Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Luo, Ye <yeluo(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target Colleen, OK, I put a print statement at the beginning of the loop that should be offloaded, and it never shows up in the output on yarrow. Yes, I pulled from https://github.com/ANL-CESAR/XSBench. Peter On Wed, May 05, 2021 at 11:24:00PM +0000, Bertoni, Colleen wrote: > Yarrow has no hardware support for double precision numbers, so it could just be very slow. On the other hand, an XSBench test from John Tramm (CC'd) is known to hang on Yarrow, and we have an open bug with Intel about it (CMPLRLLVM-24234). > > So 2 things: > 1) maybe try running for 10 min, or adding a print statement to see if it's making progress. > 2) John, could you say if the code you put in the test set is the same as what Peter is using (I think just pulled from https://github.com/ANL-CESAR/XSBench, right?) > > Colleen > ________________________________ > From: Peter McCorquodale <PWMcCorquodale(a)lbl.gov> > Sent: Wednesday, May 5, 2021 5:30 PM > To: Bertoni, Colleen <bertoni(a)anl.gov> > Cc: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Luo, Ye <yeluo(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > If I try that with either -n 2 or -n 1, same setting for mpirun and > qsub, then that runs fine on iris, in less than 1 minute, but on > yarrow, it hangs somewhere in the middle of its execution, and > exhausts its 5-minute time limit. > > Peter > > On Wed, May 05, 2021 at 09:33:39PM +0000, Bertoni, Colleen wrote: > > Thanks for the response! Could you try > > > > mpirun -n ${num_process} ./XSBench-icx -m event -s small > > > > instead of "./XSBench-icx -m event -s small" ? Some mpi versions need the `mpirun` to launch the job. > > > > Colleen > > ________________________________ > > From: Peter McCorquodale <PWMcCorquodale(a)lbl.gov> > > Sent: Wednesday, May 5, 2021 4:25 PM > > To: Bertoni, Colleen <bertoni(a)anl.gov> > > Cc: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Luo, Ye <yeluo(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > > > Colleen: Yes, on 2 nodes with a 5-minute time limit. > > > > Following Ye Luo's suggestion that I was running out of memory, I > > tried a smaller problem size, and now it runs to completion on iris, > > but I still get those MPI errors on yarrow and arcticus. The script is: > > #!/bin/bash > > source /usr/share/Modules/init/bash > > module use /soft/restricted/CNDA/modulefiles > > module load mpi > > module load oneapi > > ./XSBench-icx -m event -s small > > > > Peter > > > > On Wed, May 05, 2021 at 05:15:41PM +0000, Bertoni, Colleen wrote: > > > Hi Peter, > > > > > > Could you show us `runicx.sh`? The other thing I noticed is that you're executing `qsub -n 2 -t 5 -q iris runicx.sh`, which tries to execute on 2 iris nodes with a time limit of 5 minutes. To be clear, you were trying to run across two iris nodes? > > > > > > I know we have a version of XSBench which runs on iris, but I think it's only been on a single iris node. > > > > > > Thanks, > > > Colleen > > > ________________________________ > > > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Luo, Ye via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > Sent: Tuesday, May 4, 2021 7:09 PM > > > To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Peter McCorquodale <PWMcCorquodale(a)lbl.gov> > > > Cc: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > > > > > Call to getOrAllocTgtPtr returned null pointer > > > It is likely that it ran out of GPU memory > > > Ye > > > ________________________________ > > > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > Sent: Tuesday, May 4, 2021 6:58 PM > > > To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com> > > > Cc: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > > > > > On Tue, May 04, 2021 at 10:37:38PM +0000, Steinbrecher, Patrick wrote: > > > > For OpenMP offload on Intel GPU you should use the flags: > > > > > > > > -fiopenmp -fopenmp-targets=spir64 > > > > > > Yes, the full set of flags I have is: > > > -std=gnu99 -Wall -fiopenmp -fopenmp-targets=spir64 -D__STRICT_ANSI__ -g -pg -O3 -DMPI -fopenmp -D__STRICT_ANSI__ -cc=icx > > > > > > > I don't know about MPI error. Looks like you are using MPI with threading. Maybe that is not supported with that module. Try to just use MPI_Init. > > > > > > My source code does just call MPI_Init. And it's only yarrow that gives me > > > the error in PMPI_Init. On iris, I get a different error, later on: > > > > > > > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > > Date: Tuesday, May 4, 2021 at 17:29 > > > > To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > > Subject: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > > > I have been building and running the openmp-offload source in XSBench: > > > > https://github.com/ANL-CESAR/XSBench > > > > > > > > I can get it to build if in openmp-offload/Makefile I set > > > > CC = /soft/compilers/intel-2020.4/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpicc > > > > and > > > > CFLAGS += -fopenmp -D__STRICT_ANSI__ -cc=icx > > > > > > > > But then I get consistently different errors when I run it on iris or yarrow. > > > > > > > > On yarrow, qsub -n 2 -t 5 -q yarrow runicx.sh > > > > it terminates before producing any output: > > > > Abort(1094799) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: > > > > MPIR_Init_thread(666)......: > > > > MPID_Init(922).............: > > > > MPIDI_NM_mpi_init_hook(719): OFI addrinfo() failed (ofi_init.h:719:MPIDI_NM_mpi_init_hook:No data available) > > > > > > > > On iris, qsub -n 2 -t 5 -q iris runicx.sh > > > > it gets further along in execution, as I see from output, until: > > > > Libomptarget error: Call to getOrAllocTgtPtr returned null pointer (device failure or illegal mapping). > > > > Libomptarget error: Call to targetDataBegin failed, abort target. > > > > Libomptarget error: Failed to process data before launching the kernel. > > > > Libomptarget error: Run with LIBOMPTARGET_DEBUG=4 to dump host-target pointer mappings. > > > > unknown:68:44: Libomptarget fatal error 1: failure of target construct while offloading is mandatory > > > > > > > > This suggests that the problem on iris might be with the OpenMP target > > > > pragma, but the only #pragma omp target statement in the source > > > > code is this, and it works on other non-JLSE platforms: > > > > #pragma omp target teams distribute parallel for\ > > > > map(to: SD.max_num_nucs)\ > > > > map(to: SD.num_nucs[:SD.length_num_nucs])\ > > > > map(to: SD.concs[:SD.length_concs])\ > > > > map(to: SD.mats[:SD.length_mats])\ > > > > map(to: SD.unionized_energy_array[:SD.length_unionized_energy_array])\ > > > > map(to: SD.index_grid[:SD.length_index_grid])\ > > > > map(to: SD.nuclide_grid[:SD.length_nuclide_grid])\ > > > > reduction(+:verification) > > > > for( int i = 0; i < in.lookups; i++ ) > > > > ... > > > > > > > > Peter > > > > -- > > > > Ecp-aurora-sdk-discuss mailing list > > > > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > > > > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss > > > -- > > > Ecp-aurora-sdk-discuss mailing list > > > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > > > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target
by Peter McCorquodale 05 May '21

05 May '21

Colleen, OK, I put a print statement at the beginning of the loop that should be offloaded, and it never shows up in the output on yarrow. Yes, I pulled from https://github.com/ANL-CESAR/XSBench. Peter On Wed, May 05, 2021 at 11:24:00PM +0000, Bertoni, Colleen wrote: > Yarrow has no hardware support for double precision numbers, so it could just be very slow. On the other hand, an XSBench test from John Tramm (CC'd) is known to hang on Yarrow, and we have an open bug with Intel about it (CMPLRLLVM-24234). > > So 2 things: > 1) maybe try running for 10 min, or adding a print statement to see if it's making progress. > 2) John, could you say if the code you put in the test set is the same as what Peter is using (I think just pulled from https://github.com/ANL-CESAR/XSBench, right?) > > Colleen > ________________________________ > From: Peter McCorquodale <PWMcCorquodale(a)lbl.gov> > Sent: Wednesday, May 5, 2021 5:30 PM > To: Bertoni, Colleen <bertoni(a)anl.gov> > Cc: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Luo, Ye <yeluo(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > If I try that with either -n 2 or -n 1, same setting for mpirun and > qsub, then that runs fine on iris, in less than 1 minute, but on > yarrow, it hangs somewhere in the middle of its execution, and > exhausts its 5-minute time limit. > > Peter > > On Wed, May 05, 2021 at 09:33:39PM +0000, Bertoni, Colleen wrote: > > Thanks for the response! Could you try > > > > mpirun -n ${num_process} ./XSBench-icx -m event -s small > > > > instead of "./XSBench-icx -m event -s small" ? Some mpi versions need the `mpirun` to launch the job. > > > > Colleen > > ________________________________ > > From: Peter McCorquodale <PWMcCorquodale(a)lbl.gov> > > Sent: Wednesday, May 5, 2021 4:25 PM > > To: Bertoni, Colleen <bertoni(a)anl.gov> > > Cc: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Luo, Ye <yeluo(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > > > Colleen: Yes, on 2 nodes with a 5-minute time limit. > > > > Following Ye Luo's suggestion that I was running out of memory, I > > tried a smaller problem size, and now it runs to completion on iris, > > but I still get those MPI errors on yarrow and arcticus. The script is: > > #!/bin/bash > > source /usr/share/Modules/init/bash > > module use /soft/restricted/CNDA/modulefiles > > module load mpi > > module load oneapi > > ./XSBench-icx -m event -s small > > > > Peter > > > > On Wed, May 05, 2021 at 05:15:41PM +0000, Bertoni, Colleen wrote: > > > Hi Peter, > > > > > > Could you show us `runicx.sh`? The other thing I noticed is that you're executing `qsub -n 2 -t 5 -q iris runicx.sh`, which tries to execute on 2 iris nodes with a time limit of 5 minutes. To be clear, you were trying to run across two iris nodes? > > > > > > I know we have a version of XSBench which runs on iris, but I think it's only been on a single iris node. > > > > > > Thanks, > > > Colleen > > > ________________________________ > > > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Luo, Ye via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > Sent: Tuesday, May 4, 2021 7:09 PM > > > To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Peter McCorquodale <PWMcCorquodale(a)lbl.gov> > > > Cc: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > > > > > Call to getOrAllocTgtPtr returned null pointer > > > It is likely that it ran out of GPU memory > > > Ye > > > ________________________________ > > > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > Sent: Tuesday, May 4, 2021 6:58 PM > > > To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com> > > > Cc: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > > > > > On Tue, May 04, 2021 at 10:37:38PM +0000, Steinbrecher, Patrick wrote: > > > > For OpenMP offload on Intel GPU you should use the flags: > > > > > > > > -fiopenmp -fopenmp-targets=spir64 > > > > > > Yes, the full set of flags I have is: > > > -std=gnu99 -Wall -fiopenmp -fopenmp-targets=spir64 -D__STRICT_ANSI__ -g -pg -O3 -DMPI -fopenmp -D__STRICT_ANSI__ -cc=icx > > > > > > > I don't know about MPI error. Looks like you are using MPI with threading. Maybe that is not supported with that module. Try to just use MPI_Init. > > > > > > My source code does just call MPI_Init. And it's only yarrow that gives me > > > the error in PMPI_Init. On iris, I get a different error, later on: > > > > > > > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > > Date: Tuesday, May 4, 2021 at 17:29 > > > > To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > > Subject: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > > > I have been building and running the openmp-offload source in XSBench: > > > > https://github.com/ANL-CESAR/XSBench > > > > > > > > I can get it to build if in openmp-offload/Makefile I set > > > > CC = /soft/compilers/intel-2020.4/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpicc > > > > and > > > > CFLAGS += -fopenmp -D__STRICT_ANSI__ -cc=icx > > > > > > > > But then I get consistently different errors when I run it on iris or yarrow. > > > > > > > > On yarrow, qsub -n 2 -t 5 -q yarrow runicx.sh > > > > it terminates before producing any output: > > > > Abort(1094799) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: > > > > MPIR_Init_thread(666)......: > > > > MPID_Init(922).............: > > > > MPIDI_NM_mpi_init_hook(719): OFI addrinfo() failed (ofi_init.h:719:MPIDI_NM_mpi_init_hook:No data available) > > > > > > > > On iris, qsub -n 2 -t 5 -q iris runicx.sh > > > > it gets further along in execution, as I see from output, until: > > > > Libomptarget error: Call to getOrAllocTgtPtr returned null pointer (device failure or illegal mapping). > > > > Libomptarget error: Call to targetDataBegin failed, abort target. > > > > Libomptarget error: Failed to process data before launching the kernel. > > > > Libomptarget error: Run with LIBOMPTARGET_DEBUG=4 to dump host-target pointer mappings. > > > > unknown:68:44: Libomptarget fatal error 1: failure of target construct while offloading is mandatory > > > > > > > > This suggests that the problem on iris might be with the OpenMP target > > > > pragma, but the only #pragma omp target statement in the source > > > > code is this, and it works on other non-JLSE platforms: > > > > #pragma omp target teams distribute parallel for\ > > > > map(to: SD.max_num_nucs)\ > > > > map(to: SD.num_nucs[:SD.length_num_nucs])\ > > > > map(to: SD.concs[:SD.length_concs])\ > > > > map(to: SD.mats[:SD.length_mats])\ > > > > map(to: SD.unionized_energy_array[:SD.length_unionized_energy_array])\ > > > > map(to: SD.index_grid[:SD.length_index_grid])\ > > > > map(to: SD.nuclide_grid[:SD.length_nuclide_grid])\ > > > > reduction(+:verification) > > > > for( int i = 0; i < in.lookups; i++ ) > > > > ... > > > > > > > > Peter > > > > -- > > > > Ecp-aurora-sdk-discuss mailing list > > > > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > > > > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss > > > -- > > > Ecp-aurora-sdk-discuss mailing list > > > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > > > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target
by Bertoni, Colleen 05 May '21

05 May '21

Yarrow has no hardware support for double precision numbers, so it could just be very slow. On the other hand, an XSBench test from John Tramm (CC'd) is known to hang on Yarrow, and we have an open bug with Intel about it (CMPLRLLVM-24234). So 2 things: 1) maybe try running for 10 min, or adding a print statement to see if it's making progress. 2) John, could you say if the code you put in the test set is the same as what Peter is using (I think just pulled from https://github.com/ANL-CESAR/XSBench, right?) Colleen ________________________________ From: Peter McCorquodale <PWMcCorquodale(a)lbl.gov> Sent: Wednesday, May 5, 2021 5:30 PM To: Bertoni, Colleen <bertoni(a)anl.gov> Cc: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Luo, Ye <yeluo(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target If I try that with either -n 2 or -n 1, same setting for mpirun and qsub, then that runs fine on iris, in less than 1 minute, but on yarrow, it hangs somewhere in the middle of its execution, and exhausts its 5-minute time limit. Peter On Wed, May 05, 2021 at 09:33:39PM +0000, Bertoni, Colleen wrote: > Thanks for the response! Could you try > > mpirun -n ${num_process} ./XSBench-icx -m event -s small > > instead of "./XSBench-icx -m event -s small" ? Some mpi versions need the `mpirun` to launch the job. > > Colleen > ________________________________ > From: Peter McCorquodale <PWMcCorquodale(a)lbl.gov> > Sent: Wednesday, May 5, 2021 4:25 PM > To: Bertoni, Colleen <bertoni(a)anl.gov> > Cc: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Luo, Ye <yeluo(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > Colleen: Yes, on 2 nodes with a 5-minute time limit. > > Following Ye Luo's suggestion that I was running out of memory, I > tried a smaller problem size, and now it runs to completion on iris, > but I still get those MPI errors on yarrow and arcticus. The script is: > #!/bin/bash > source /usr/share/Modules/init/bash > module use /soft/restricted/CNDA/modulefiles > module load mpi > module load oneapi > ./XSBench-icx -m event -s small > > Peter > > On Wed, May 05, 2021 at 05:15:41PM +0000, Bertoni, Colleen wrote: > > Hi Peter, > > > > Could you show us `runicx.sh`? The other thing I noticed is that you're executing `qsub -n 2 -t 5 -q iris runicx.sh`, which tries to execute on 2 iris nodes with a time limit of 5 minutes. To be clear, you were trying to run across two iris nodes? > > > > I know we have a version of XSBench which runs on iris, but I think it's only been on a single iris node. > > > > Thanks, > > Colleen > > ________________________________ > > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Luo, Ye via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > Sent: Tuesday, May 4, 2021 7:09 PM > > To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Peter McCorquodale <PWMcCorquodale(a)lbl.gov> > > Cc: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > > > Call to getOrAllocTgtPtr returned null pointer > > It is likely that it ran out of GPU memory > > Ye > > ________________________________ > > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > Sent: Tuesday, May 4, 2021 6:58 PM > > To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com> > > Cc: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > > > On Tue, May 04, 2021 at 10:37:38PM +0000, Steinbrecher, Patrick wrote: > > > For OpenMP offload on Intel GPU you should use the flags: > > > > > > -fiopenmp -fopenmp-targets=spir64 > > > > Yes, the full set of flags I have is: > > -std=gnu99 -Wall -fiopenmp -fopenmp-targets=spir64 -D__STRICT_ANSI__ -g -pg -O3 -DMPI -fopenmp -D__STRICT_ANSI__ -cc=icx > > > > > I don't know about MPI error. Looks like you are using MPI with threading. Maybe that is not supported with that module. Try to just use MPI_Init. > > > > My source code does just call MPI_Init. And it's only yarrow that gives me > > the error in PMPI_Init. On iris, I get a different error, later on: > > > > > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > Date: Tuesday, May 4, 2021 at 17:29 > > > To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > Subject: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > > I have been building and running the openmp-offload source in XSBench: > > > https://github.com/ANL-CESAR/XSBench > > > > > > I can get it to build if in openmp-offload/Makefile I set > > > CC = /soft/compilers/intel-2020.4/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpicc > > > and > > > CFLAGS += -fopenmp -D__STRICT_ANSI__ -cc=icx > > > > > > But then I get consistently different errors when I run it on iris or yarrow. > > > > > > On yarrow, qsub -n 2 -t 5 -q yarrow runicx.sh > > > it terminates before producing any output: > > > Abort(1094799) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: > > > MPIR_Init_thread(666)......: > > > MPID_Init(922).............: > > > MPIDI_NM_mpi_init_hook(719): OFI addrinfo() failed (ofi_init.h:719:MPIDI_NM_mpi_init_hook:No data available) > > > > > > On iris, qsub -n 2 -t 5 -q iris runicx.sh > > > it gets further along in execution, as I see from output, until: > > > Libomptarget error: Call to getOrAllocTgtPtr returned null pointer (device failure or illegal mapping). > > > Libomptarget error: Call to targetDataBegin failed, abort target. > > > Libomptarget error: Failed to process data before launching the kernel. > > > Libomptarget error: Run with LIBOMPTARGET_DEBUG=4 to dump host-target pointer mappings. > > > unknown:68:44: Libomptarget fatal error 1: failure of target construct while offloading is mandatory > > > > > > This suggests that the problem on iris might be with the OpenMP target > > > pragma, but the only #pragma omp target statement in the source > > > code is this, and it works on other non-JLSE platforms: > > > #pragma omp target teams distribute parallel for\ > > > map(to: SD.max_num_nucs)\ > > > map(to: SD.num_nucs[:SD.length_num_nucs])\ > > > map(to: SD.concs[:SD.length_concs])\ > > > map(to: SD.mats[:SD.length_mats])\ > > > map(to: SD.unionized_energy_array[:SD.length_unionized_energy_array])\ > > > map(to: SD.index_grid[:SD.length_index_grid])\ > > > map(to: SD.nuclide_grid[:SD.length_nuclide_grid])\ > > > reduction(+:verification) > > > for( int i = 0; i < in.lookups; i++ ) > > > ... > > > > > > Peter > > > -- > > > Ecp-aurora-sdk-discuss mailing list > > > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > > > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss > > -- > > Ecp-aurora-sdk-discuss mailing list > > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target
by Peter McCorquodale 05 May '21

05 May '21

If I try that with either -n 2 or -n 1, same setting for mpirun and qsub, then that runs fine on iris, in less than 1 minute, but on yarrow, it hangs somewhere in the middle of its execution, and exhausts its 5-minute time limit. Peter On Wed, May 05, 2021 at 09:33:39PM +0000, Bertoni, Colleen wrote: > Thanks for the response! Could you try > > mpirun -n ${num_process} ./XSBench-icx -m event -s small > > instead of "./XSBench-icx -m event -s small" ? Some mpi versions need the `mpirun` to launch the job. > > Colleen > ________________________________ > From: Peter McCorquodale <PWMcCorquodale(a)lbl.gov> > Sent: Wednesday, May 5, 2021 4:25 PM > To: Bertoni, Colleen <bertoni(a)anl.gov> > Cc: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Luo, Ye <yeluo(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > Colleen: Yes, on 2 nodes with a 5-minute time limit. > > Following Ye Luo's suggestion that I was running out of memory, I > tried a smaller problem size, and now it runs to completion on iris, > but I still get those MPI errors on yarrow and arcticus. The script is: > #!/bin/bash > source /usr/share/Modules/init/bash > module use /soft/restricted/CNDA/modulefiles > module load mpi > module load oneapi > ./XSBench-icx -m event -s small > > Peter > > On Wed, May 05, 2021 at 05:15:41PM +0000, Bertoni, Colleen wrote: > > Hi Peter, > > > > Could you show us `runicx.sh`? The other thing I noticed is that you're executing `qsub -n 2 -t 5 -q iris runicx.sh`, which tries to execute on 2 iris nodes with a time limit of 5 minutes. To be clear, you were trying to run across two iris nodes? > > > > I know we have a version of XSBench which runs on iris, but I think it's only been on a single iris node. > > > > Thanks, > > Colleen > > ________________________________ > > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Luo, Ye via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > Sent: Tuesday, May 4, 2021 7:09 PM > > To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Peter McCorquodale <PWMcCorquodale(a)lbl.gov> > > Cc: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > > > Call to getOrAllocTgtPtr returned null pointer > > It is likely that it ran out of GPU memory > > Ye > > ________________________________ > > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > Sent: Tuesday, May 4, 2021 6:58 PM > > To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com> > > Cc: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > > > On Tue, May 04, 2021 at 10:37:38PM +0000, Steinbrecher, Patrick wrote: > > > For OpenMP offload on Intel GPU you should use the flags: > > > > > > -fiopenmp -fopenmp-targets=spir64 > > > > Yes, the full set of flags I have is: > > -std=gnu99 -Wall -fiopenmp -fopenmp-targets=spir64 -D__STRICT_ANSI__ -g -pg -O3 -DMPI -fopenmp -D__STRICT_ANSI__ -cc=icx > > > > > I don't know about MPI error. Looks like you are using MPI with threading. Maybe that is not supported with that module. Try to just use MPI_Init. > > > > My source code does just call MPI_Init. And it's only yarrow that gives me > > the error in PMPI_Init. On iris, I get a different error, later on: > > > > > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > Date: Tuesday, May 4, 2021 at 17:29 > > > To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > Subject: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > > I have been building and running the openmp-offload source in XSBench: > > > https://github.com/ANL-CESAR/XSBench > > > > > > I can get it to build if in openmp-offload/Makefile I set > > > CC = /soft/compilers/intel-2020.4/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpicc > > > and > > > CFLAGS += -fopenmp -D__STRICT_ANSI__ -cc=icx > > > > > > But then I get consistently different errors when I run it on iris or yarrow. > > > > > > On yarrow, qsub -n 2 -t 5 -q yarrow runicx.sh > > > it terminates before producing any output: > > > Abort(1094799) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: > > > MPIR_Init_thread(666)......: > > > MPID_Init(922).............: > > > MPIDI_NM_mpi_init_hook(719): OFI addrinfo() failed (ofi_init.h:719:MPIDI_NM_mpi_init_hook:No data available) > > > > > > On iris, qsub -n 2 -t 5 -q iris runicx.sh > > > it gets further along in execution, as I see from output, until: > > > Libomptarget error: Call to getOrAllocTgtPtr returned null pointer (device failure or illegal mapping). > > > Libomptarget error: Call to targetDataBegin failed, abort target. > > > Libomptarget error: Failed to process data before launching the kernel. > > > Libomptarget error: Run with LIBOMPTARGET_DEBUG=4 to dump host-target pointer mappings. > > > unknown:68:44: Libomptarget fatal error 1: failure of target construct while offloading is mandatory > > > > > > This suggests that the problem on iris might be with the OpenMP target > > > pragma, but the only #pragma omp target statement in the source > > > code is this, and it works on other non-JLSE platforms: > > > #pragma omp target teams distribute parallel for\ > > > map(to: SD.max_num_nucs)\ > > > map(to: SD.num_nucs[:SD.length_num_nucs])\ > > > map(to: SD.concs[:SD.length_concs])\ > > > map(to: SD.mats[:SD.length_mats])\ > > > map(to: SD.unionized_energy_array[:SD.length_unionized_energy_array])\ > > > map(to: SD.index_grid[:SD.length_index_grid])\ > > > map(to: SD.nuclide_grid[:SD.length_nuclide_grid])\ > > > reduction(+:verification) > > > for( int i = 0; i < in.lookups; i++ ) > > > ... > > > > > > Peter > > > -- > > > Ecp-aurora-sdk-discuss mailing list > > > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > > > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss > > -- > > Ecp-aurora-sdk-discuss mailing list > > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target
by Bertoni, Colleen 05 May '21

05 May '21

Thanks for the response! Could you try mpirun -n ${num_process} ./XSBench-icx -m event -s small instead of "./XSBench-icx -m event -s small" ? Some mpi versions need the `mpirun` to launch the job. Colleen ________________________________ From: Peter McCorquodale <PWMcCorquodale(a)lbl.gov> Sent: Wednesday, May 5, 2021 4:25 PM To: Bertoni, Colleen <bertoni(a)anl.gov> Cc: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Luo, Ye <yeluo(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target Colleen: Yes, on 2 nodes with a 5-minute time limit. Following Ye Luo's suggestion that I was running out of memory, I tried a smaller problem size, and now it runs to completion on iris, but I still get those MPI errors on yarrow and arcticus. The script is: #!/bin/bash source /usr/share/Modules/init/bash module use /soft/restricted/CNDA/modulefiles module load mpi module load oneapi ./XSBench-icx -m event -s small Peter On Wed, May 05, 2021 at 05:15:41PM +0000, Bertoni, Colleen wrote: > Hi Peter, > > Could you show us `runicx.sh`? The other thing I noticed is that you're executing `qsub -n 2 -t 5 -q iris runicx.sh`, which tries to execute on 2 iris nodes with a time limit of 5 minutes. To be clear, you were trying to run across two iris nodes? > > I know we have a version of XSBench which runs on iris, but I think it's only been on a single iris node. > > Thanks, > Colleen > ________________________________ > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Luo, Ye via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Sent: Tuesday, May 4, 2021 7:09 PM > To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Peter McCorquodale <PWMcCorquodale(a)lbl.gov> > Cc: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > Call to getOrAllocTgtPtr returned null pointer > It is likely that it ran out of GPU memory > Ye > ________________________________ > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Sent: Tuesday, May 4, 2021 6:58 PM > To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com> > Cc: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > On Tue, May 04, 2021 at 10:37:38PM +0000, Steinbrecher, Patrick wrote: > > For OpenMP offload on Intel GPU you should use the flags: > > > > -fiopenmp -fopenmp-targets=spir64 > > Yes, the full set of flags I have is: > -std=gnu99 -Wall -fiopenmp -fopenmp-targets=spir64 -D__STRICT_ANSI__ -g -pg -O3 -DMPI -fopenmp -D__STRICT_ANSI__ -cc=icx > > > I don't know about MPI error. Looks like you are using MPI with threading. Maybe that is not supported with that module. Try to just use MPI_Init. > > My source code does just call MPI_Init. And it's only yarrow that gives me > the error in PMPI_Init. On iris, I get a different error, later on: > > > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > Date: Tuesday, May 4, 2021 at 17:29 > > To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > Subject: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > I have been building and running the openmp-offload source in XSBench: > > https://github.com/ANL-CESAR/XSBench > > > > I can get it to build if in openmp-offload/Makefile I set > > CC = /soft/compilers/intel-2020.4/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpicc > > and > > CFLAGS += -fopenmp -D__STRICT_ANSI__ -cc=icx > > > > But then I get consistently different errors when I run it on iris or yarrow. > > > > On yarrow, qsub -n 2 -t 5 -q yarrow runicx.sh > > it terminates before producing any output: > > Abort(1094799) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: > > MPIR_Init_thread(666)......: > > MPID_Init(922).............: > > MPIDI_NM_mpi_init_hook(719): OFI addrinfo() failed (ofi_init.h:719:MPIDI_NM_mpi_init_hook:No data available) > > > > On iris, qsub -n 2 -t 5 -q iris runicx.sh > > it gets further along in execution, as I see from output, until: > > Libomptarget error: Call to getOrAllocTgtPtr returned null pointer (device failure or illegal mapping). > > Libomptarget error: Call to targetDataBegin failed, abort target. > > Libomptarget error: Failed to process data before launching the kernel. > > Libomptarget error: Run with LIBOMPTARGET_DEBUG=4 to dump host-target pointer mappings. > > unknown:68:44: Libomptarget fatal error 1: failure of target construct while offloading is mandatory > > > > This suggests that the problem on iris might be with the OpenMP target > > pragma, but the only #pragma omp target statement in the source > > code is this, and it works on other non-JLSE platforms: > > #pragma omp target teams distribute parallel for\ > > map(to: SD.max_num_nucs)\ > > map(to: SD.num_nucs[:SD.length_num_nucs])\ > > map(to: SD.concs[:SD.length_concs])\ > > map(to: SD.mats[:SD.length_mats])\ > > map(to: SD.unionized_energy_array[:SD.length_unionized_energy_array])\ > > map(to: SD.index_grid[:SD.length_index_grid])\ > > map(to: SD.nuclide_grid[:SD.length_nuclide_grid])\ > > reduction(+:verification) > > for( int i = 0; i < in.lookups; i++ ) > > ... > > > > Peter > > -- > > Ecp-aurora-sdk-discuss mailing list > > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss > -- > Ecp-aurora-sdk-discuss mailing list > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target
by Peter McCorquodale 05 May '21

05 May '21

Colleen: Yes, on 2 nodes with a 5-minute time limit. Following Ye Luo's suggestion that I was running out of memory, I tried a smaller problem size, and now it runs to completion on iris, but I still get those MPI errors on yarrow and arcticus. The script is: #!/bin/bash source /usr/share/Modules/init/bash module use /soft/restricted/CNDA/modulefiles module load mpi module load oneapi ./XSBench-icx -m event -s small Peter On Wed, May 05, 2021 at 05:15:41PM +0000, Bertoni, Colleen wrote: > Hi Peter, > > Could you show us `runicx.sh`? The other thing I noticed is that you're executing `qsub -n 2 -t 5 -q iris runicx.sh`, which tries to execute on 2 iris nodes with a time limit of 5 minutes. To be clear, you were trying to run across two iris nodes? > > I know we have a version of XSBench which runs on iris, but I think it's only been on a single iris node. > > Thanks, > Colleen > ________________________________ > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Luo, Ye via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Sent: Tuesday, May 4, 2021 7:09 PM > To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Peter McCorquodale <PWMcCorquodale(a)lbl.gov> > Cc: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > Call to getOrAllocTgtPtr returned null pointer > It is likely that it ran out of GPU memory > Ye > ________________________________ > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Sent: Tuesday, May 4, 2021 6:58 PM > To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com> > Cc: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > On Tue, May 04, 2021 at 10:37:38PM +0000, Steinbrecher, Patrick wrote: > > For OpenMP offload on Intel GPU you should use the flags: > > > > -fiopenmp -fopenmp-targets=spir64 > > Yes, the full set of flags I have is: > -std=gnu99 -Wall -fiopenmp -fopenmp-targets=spir64 -D__STRICT_ANSI__ -g -pg -O3 -DMPI -fopenmp -D__STRICT_ANSI__ -cc=icx > > > I don't know about MPI error. Looks like you are using MPI with threading. Maybe that is not supported with that module. Try to just use MPI_Init. > > My source code does just call MPI_Init. And it's only yarrow that gives me > the error in PMPI_Init. On iris, I get a different error, later on: > > > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > Date: Tuesday, May 4, 2021 at 17:29 > > To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > Subject: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > I have been building and running the openmp-offload source in XSBench: > > https://github.com/ANL-CESAR/XSBench > > > > I can get it to build if in openmp-offload/Makefile I set > > CC = /soft/compilers/intel-2020.4/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpicc > > and > > CFLAGS += -fopenmp -D__STRICT_ANSI__ -cc=icx > > > > But then I get consistently different errors when I run it on iris or yarrow. > > > > On yarrow, qsub -n 2 -t 5 -q yarrow runicx.sh > > it terminates before producing any output: > > Abort(1094799) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: > > MPIR_Init_thread(666)......: > > MPID_Init(922).............: > > MPIDI_NM_mpi_init_hook(719): OFI addrinfo() failed (ofi_init.h:719:MPIDI_NM_mpi_init_hook:No data available) > > > > On iris, qsub -n 2 -t 5 -q iris runicx.sh > > it gets further along in execution, as I see from output, until: > > Libomptarget error: Call to getOrAllocTgtPtr returned null pointer (device failure or illegal mapping). > > Libomptarget error: Call to targetDataBegin failed, abort target. > > Libomptarget error: Failed to process data before launching the kernel. > > Libomptarget error: Run with LIBOMPTARGET_DEBUG=4 to dump host-target pointer mappings. > > unknown:68:44: Libomptarget fatal error 1: failure of target construct while offloading is mandatory > > > > This suggests that the problem on iris might be with the OpenMP target > > pragma, but the only #pragma omp target statement in the source > > code is this, and it works on other non-JLSE platforms: > > #pragma omp target teams distribute parallel for\ > > map(to: SD.max_num_nucs)\ > > map(to: SD.num_nucs[:SD.length_num_nucs])\ > > map(to: SD.concs[:SD.length_concs])\ > > map(to: SD.mats[:SD.length_mats])\ > > map(to: SD.unionized_energy_array[:SD.length_unionized_energy_array])\ > > map(to: SD.index_grid[:SD.length_index_grid])\ > > map(to: SD.nuclide_grid[:SD.length_nuclide_grid])\ > > reduction(+:verification) > > for( int i = 0; i < in.lookups; i++ ) > > ... > > > > Peter > > -- > > Ecp-aurora-sdk-discuss mailing list > > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss > -- > Ecp-aurora-sdk-discuss mailing list > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

New version of mpich on JLSE (40.2)
by Chan-nui, Christopher 05 May '21

05 May '21

Hello, We are pleased to announce that an update to mpich, drop 40.2, has been deployed on JLSE. module use /soft/restricted/CNDA/modulefiles module avail mpi/aurora_mpich Release Notes · MPICH is now fully GPU-aware, that is, it supports the MPI standard for host/device/shared memory allocated buffers. · MPICH compilers use icx and icpx compilers. Details · Added native reduction support for collectives and RMA on GPUs · Wrappers for “mpicc”, “mpicxx” are no longer needed. Just using “mpicc” means that “icx” will be used and using “mpicxx” will use icpx. “mpifort”, “mpif77” and “mpif90” still use “ifort”. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com

1 0

Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target
by Bertoni, Colleen 05 May '21

05 May '21

Hi Peter, Could you show us `runicx.sh`? The other thing I noticed is that you're executing `qsub -n 2 -t 5 -q iris runicx.sh`, which tries to execute on 2 iris nodes with a time limit of 5 minutes. To be clear, you were trying to run across two iris nodes? I know we have a version of XSBench which runs on iris, but I think it's only been on a single iris node. Thanks, Colleen ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Luo, Ye via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Tuesday, May 4, 2021 7:09 PM To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Peter McCorquodale <PWMcCorquodale(a)lbl.gov> Cc: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target Call to getOrAllocTgtPtr returned null pointer It is likely that it ran out of GPU memory Ye ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Tuesday, May 4, 2021 6:58 PM To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com> Cc: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target On Tue, May 04, 2021 at 10:37:38PM +0000, Steinbrecher, Patrick wrote: > For OpenMP offload on Intel GPU you should use the flags: > > -fiopenmp -fopenmp-targets=spir64 Yes, the full set of flags I have is: -std=gnu99 -Wall -fiopenmp -fopenmp-targets=spir64 -D__STRICT_ANSI__ -g -pg -O3 -DMPI -fopenmp -D__STRICT_ANSI__ -cc=icx > I don't know about MPI error. Looks like you are using MPI with threading. Maybe that is not supported with that module. Try to just use MPI_Init. My source code does just call MPI_Init. And it's only yarrow that gives me the error in PMPI_Init. On iris, I get a different error, later on: > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Date: Tuesday, May 4, 2021 at 17:29 > To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Subject: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > I have been building and running the openmp-offload source in XSBench: > https://github.com/ANL-CESAR/XSBench > > I can get it to build if in openmp-offload/Makefile I set > CC = /soft/compilers/intel-2020.4/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpicc > and > CFLAGS += -fopenmp -D__STRICT_ANSI__ -cc=icx > > But then I get consistently different errors when I run it on iris or yarrow. > > On yarrow, qsub -n 2 -t 5 -q yarrow runicx.sh > it terminates before producing any output: > Abort(1094799) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: > MPIR_Init_thread(666)......: > MPID_Init(922).............: > MPIDI_NM_mpi_init_hook(719): OFI addrinfo() failed (ofi_init.h:719:MPIDI_NM_mpi_init_hook:No data available) > > On iris, qsub -n 2 -t 5 -q iris runicx.sh > it gets further along in execution, as I see from output, until: > Libomptarget error: Call to getOrAllocTgtPtr returned null pointer (device failure or illegal mapping). > Libomptarget error: Call to targetDataBegin failed, abort target. > Libomptarget error: Failed to process data before launching the kernel. > Libomptarget error: Run with LIBOMPTARGET_DEBUG=4 to dump host-target pointer mappings. > unknown:68:44: Libomptarget fatal error 1: failure of target construct while offloading is mandatory > > This suggests that the problem on iris might be with the OpenMP target > pragma, but the only #pragma omp target statement in the source > code is this, and it works on other non-JLSE platforms: > #pragma omp target teams distribute parallel for\ > map(to: SD.max_num_nucs)\ > map(to: SD.num_nucs[:SD.length_num_nucs])\ > map(to: SD.concs[:SD.length_concs])\ > map(to: SD.mats[:SD.length_mats])\ > map(to: SD.unionized_energy_array[:SD.length_unionized_energy_array])\ > map(to: SD.index_grid[:SD.length_index_grid])\ > map(to: SD.nuclide_grid[:SD.length_nuclide_grid])\ > reduction(+:verification) > for( int i = 0; i < in.lookups; i++ ) > ... > > Peter > -- > Ecp-aurora-sdk-discuss mailing list > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss -- Ecp-aurora-sdk-discuss mailing list Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0