On the out-of-memory issue. It is intel plugin that didn't properly printout device runtime error.
In the upstream clang targeting nvidia GPUs.
The error message is clear.

yeluo@epyc-server:~$ ./a.out
CUDA error: Error returned from cuMemAlloc
CUDA error: out of memory
CUDA error: Error returned from cuMemAlloc
CUDA error: out of memory
Libomptarget error: Call to getOrAllocTgtPtr returned null pointer (device failure or illegal mapping).
Libomptarget error: Run with LIBOMPTARGET_DEBUG=4 to dump host-target pointer mappings.
Libomptarget error: Source location information not present. Compile with -g or -gline-tables-only.
Libomptarget fatal error 1: failure of target construct while offloading is mandatory

Ye
From: Peter McCorquodale <PWMcCorquodale@lbl.gov>
Sent: Wednesday, May 5, 2021 4:25 PM
To: Bertoni, Colleen <bertoni@anl.gov>
Cc: Steinbrecher, Patrick <patrick.steinbrecher@intel.com>; Luo, Ye <yeluo@anl.gov>; ecp-aurora-sdk-discuss@lists.jlse.anl.gov <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target
 
Colleen:  Yes, on 2 nodes with a 5-minute time limit.

Following Ye Luo's suggestion that I was running out of memory, I
tried a smaller problem size, and now it runs to completion on iris,
but I still get those MPI errors on yarrow and arcticus.  The script is:
#!/bin/bash
source /usr/share/Modules/init/bash
module use /soft/restricted/CNDA/modulefiles
module load mpi
module load oneapi
./XSBench-icx -m event -s small

Peter

On Wed, May 05, 2021 at 05:15:41PM +0000, Bertoni, Colleen wrote:
> Hi Peter,
>
> Could you show us `runicx.sh`? The other thing I noticed is that you're executing `qsub -n 2 -t 5 -q iris runicx.sh`, which tries to execute on 2 iris nodes with a time limit of 5 minutes. To be clear, you were trying to run across two iris nodes?
>
> I know we have a version of XSBench which runs on iris, but I think it's only been on a single iris node.
>
> Thanks,
> Colleen
> ________________________________
> From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces@lists.jlse.anl.gov> on behalf of Luo, Ye via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
> Sent: Tuesday, May 4, 2021 7:09 PM
> To: Steinbrecher, Patrick <patrick.steinbrecher@intel.com>; Peter McCorquodale <PWMcCorquodale@lbl.gov>
> Cc: ecp-aurora-sdk-discuss@lists.jlse.anl.gov <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
> Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target
>
> Call to getOrAllocTgtPtr returned null pointer
> It is likely that it ran out of GPU memory
> Ye
> ________________________________
> From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces@lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
> Sent: Tuesday, May 4, 2021 6:58 PM
> To: Steinbrecher, Patrick <patrick.steinbrecher@intel.com>
> Cc: ecp-aurora-sdk-discuss@lists.jlse.anl.gov <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
> Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target
>
> On Tue, May 04, 2021 at 10:37:38PM +0000, Steinbrecher, Patrick wrote:
> > For OpenMP offload on Intel GPU you should use the flags:
> >
> > -fiopenmp -fopenmp-targets=spir64
>
> Yes, the full set of flags I have is:
> -std=gnu99 -Wall -fiopenmp -fopenmp-targets=spir64 -D__STRICT_ANSI__  -g -pg -O3 -DMPI -fopenmp -D__STRICT_ANSI__ -cc=icx
>
> > I don't know about MPI error. Looks like you are using MPI with threading. Maybe that is not supported with that module. Try to just use MPI_Init.
>
> My source code does just call MPI_Init.  And it's only yarrow that gives me
> the error in PMPI_Init.  On iris, I get a different error, later on:
>
> > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces@lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
> > Date: Tuesday, May 4, 2021 at 17:29
> > To: ecp-aurora-sdk-discuss@lists.jlse.anl.gov <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
> > Subject: [Ecp-aurora-sdk-discuss] offloading with OpenMP target
> > I have been building and running the openmp-offload source in XSBench:
> > https://github.com/ANL-CESAR/XSBench
> >
> > I can get it to build if in openmp-offload/Makefile I set
> > CC = /soft/compilers/intel-2020.4/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpicc
> > and
> > CFLAGS += -fopenmp -D__STRICT_ANSI__ -cc=icx
> >
> > But then I get consistently different errors when I run it on iris or yarrow.
> >
> > On yarrow, qsub -n 2 -t 5 -q yarrow runicx.sh
> > it terminates before producing any output:
> > Abort(1094799) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack:
> > MPIR_Init_thread(666)......:
> > MPID_Init(922).............:
> > MPIDI_NM_mpi_init_hook(719): OFI addrinfo() failed (ofi_init.h:719:MPIDI_NM_mpi_init_hook:No data available)
> >
> > On iris, qsub -n 2 -t 5 -q iris runicx.sh
> > it gets further along in execution, as I see from output, until:
> > Libomptarget error: Call to getOrAllocTgtPtr returned null pointer (device failure or illegal mapping).
> > Libomptarget error: Call to targetDataBegin failed, abort target.
> > Libomptarget error: Failed to process data before launching the kernel.
> > Libomptarget error: Run with LIBOMPTARGET_DEBUG=4 to dump host-target pointer mappings.
> > unknown:68:44: Libomptarget fatal error 1: failure of target construct while offloading is mandatory
> >
> > This suggests that the problem on iris might be with the OpenMP target
> > pragma, but the only #pragma omp target statement in the source
> > code is this, and it works on other non-JLSE platforms:
> >         #pragma omp target teams distribute parallel for\
> >         map(to: SD.max_num_nucs)\
> >         map(to: SD.num_nucs[:SD.length_num_nucs])\
> >         map(to: SD.concs[:SD.length_concs])\
> >         map(to: SD.mats[:SD.length_mats])\
> >         map(to: SD.unionized_energy_array[:SD.length_unionized_energy_array])\
> >         map(to: SD.index_grid[:SD.length_index_grid])\
> >         map(to: SD.nuclide_grid[:SD.length_nuclide_grid])\
> >         reduction(+:verification)
> >         for( int i = 0; i < in.lookups; i++ )
> >         ...
> >
> > Peter
> > --
> > Ecp-aurora-sdk-discuss mailing list
> > Ecp-aurora-sdk-discuss@lists.jlse.anl.gov
> > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss
> --
> Ecp-aurora-sdk-discuss mailing list
> Ecp-aurora-sdk-discuss@lists.jlse.anl.gov
> https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss