On the out-of-memory issue. It is intel plugin that didn't properly printout device runtime error. In the upstream clang targeting nvidia GPUs. The error message is clear. yeluo@epyc-server:~$ ./a.out CUDA error: Error returned from cuMemAlloc CUDA error: out of memory CUDA error: Error returned from cuMemAlloc CUDA error: out of memory Libomptarget error: Call to getOrAllocTgtPtr returned null pointer (device failure or illegal mapping). Libomptarget error: Run with LIBOMPTARGET_DEBUG=4 to dump host-target pointer mappings. Libomptarget error: Source location information not present. Compile with -g or -gline-tables-only. Libomptarget fatal error 1: failure of target construct while offloading is mandatory Ye ________________________________ From: Peter McCorquodale <[email protected]> Sent: Wednesday, May 5, 2021 4:25 PM To: Bertoni, Colleen <[email protected]> Cc: Steinbrecher, Patrick <[email protected]>; Luo, Ye <[email protected]>; [email protected] <[email protected]> Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target Colleen: Yes, on 2 nodes with a 5-minute time limit. Following Ye Luo's suggestion that I was running out of memory, I tried a smaller problem size, and now it runs to completion on iris, but I still get those MPI errors on yarrow and arcticus. The script is: #!/bin/bash source /usr/share/Modules/init/bash module use /soft/restricted/CNDA/modulefiles module load mpi module load oneapi ./XSBench-icx -m event -s small Peter On Wed, May 05, 2021 at 05:15:41PM +0000, Bertoni, Colleen wrote:

Hi Peter,

Could you show us `runicx.sh`? The other thing I noticed is that you're executing `qsub -n 2 -t 5 -q iris runicx.sh`, which tries to execute on 2 iris nodes with a time limit of 5 minutes. To be clear, you were trying to run across two iris nodes?

I know we have a version of XSBench which runs on iris, but I think it's only been on a single iris node.

Thanks, Colleen ________________________________ From: Ecp-aurora-sdk-discuss <[email protected]> on behalf of Luo, Ye via Ecp-aurora-sdk-discuss <[email protected]> Sent: Tuesday, May 4, 2021 7:09 PM To: Steinbrecher, Patrick <[email protected]>; Peter McCorquodale <[email protected]> Cc: [email protected] <[email protected]> Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target

Call to getOrAllocTgtPtr returned null pointer It is likely that it ran out of GPU memory Ye ________________________________ From: Ecp-aurora-sdk-discuss <[email protected]> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <[email protected]> Sent: Tuesday, May 4, 2021 6:58 PM To: Steinbrecher, Patrick <[email protected]> Cc: [email protected] <[email protected]> Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target

On Tue, May 04, 2021 at 10:37:38PM +0000, Steinbrecher, Patrick wrote:

...

For OpenMP offload on Intel GPU you should use the flags:

-fiopenmp -fopenmp-targets=spir64

Yes, the full set of flags I have is: -std=gnu99 -Wall -fiopenmp -fopenmp-targets=spir64 -D__STRICT_ANSI__ -g -pg -O3 -DMPI -fopenmp -D__STRICT_ANSI__ -cc=icx

...

I don't know about MPI error. Looks like you are using MPI with threading. Maybe that is not supported with that module. Try to just use MPI_Init.

My source code does just call MPI_Init. And it's only yarrow that gives me the error in PMPI_Init. On iris, I get a different error, later on:

...

From: Ecp-aurora-sdk-discuss <[email protected]> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <[email protected]> Date: Tuesday, May 4, 2021 at 17:29 To: [email protected] <[email protected]> Subject: [Ecp-aurora-sdk-discuss] offloading with OpenMP target I have been building and running the openmp-offload source in XSBench: https://github.com/ANL-CESAR/XSBench

I can get it to build if in openmp-offload/Makefile I set CC = /soft/compilers/intel-2020.4/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpicc and CFLAGS += -fopenmp -D__STRICT_ANSI__ -cc=icx

But then I get consistently different errors when I run it on iris or yarrow.

On yarrow, qsub -n 2 -t 5 -q yarrow runicx.sh it terminates before producing any output: Abort(1094799) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(666)......: MPID_Init(922).............: MPIDI_NM_mpi_init_hook(719): OFI addrinfo() failed (ofi_init.h:719:MPIDI_NM_mpi_init_hook:No data available)

On iris, qsub -n 2 -t 5 -q iris runicx.sh it gets further along in execution, as I see from output, until: Libomptarget error: Call to getOrAllocTgtPtr returned null pointer (device failure or illegal mapping). Libomptarget error: Call to targetDataBegin failed, abort target. Libomptarget error: Failed to process data before launching the kernel. Libomptarget error: Run with LIBOMPTARGET_DEBUG=4 to dump host-target pointer mappings. unknown:68:44: Libomptarget fatal error 1: failure of target construct while offloading is mandatory

This suggests that the problem on iris might be with the OpenMP target pragma, but the only #pragma omp target statement in the source code is this, and it works on other non-JLSE platforms: #pragma omp target teams distribute parallel for\ map(to: SD.max_num_nucs)\ map(to: SD.num_nucs[:SD.length_num_nucs])\ map(to: SD.concs[:SD.length_concs])\ map(to: SD.mats[:SD.length_mats])\ map(to: SD.unionized_energy_array[:SD.length_unionized_energy_array])\ map(to: SD.index_grid[:SD.length_index_grid])\ map(to: SD.nuclide_grid[:SD.length_nuclide_grid])\ reduction(+:verification) for( int i = 0; i < in.lookups; i++ ) ...

Peter -- Ecp-aurora-sdk-discuss mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss -- Ecp-aurora-sdk-discuss mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss