Colleen: Yes, when I try that on yarrow I get a lot of: error: double type is not supported on this platform error: backend compiler failed build. There are some differences between iris and arcticus. XSBench runs to completion on both, but there are a couple of problems on arcticus: - the printf statements that I put inside the offloaded OpenMP target loop do not produce any output; - I get an invalid verification checksum for the results. Peter On Thu, May 06, 2021 at 03:15:47PM +0000, Bertoni, Colleen wrote:

Thanks! I checked with John and the codes are similar, so it's likely a similar issue.

Since yarrow doesn't have hardware support for double precision, it just emulates double precision in software. And since yarrow isn't meant to support double precision, the software emulation support isn't perfect, and even when it works, performance can be 10-100X slower.

To see if the issue is from the double precision emulation, if you turn off the emulator and enable OMP debugging: ``` unset IGC_EnableDPEmulation export LIBOMPTARGET_DEBUG=1 ``` the fail should be visible: ``` Target LEVEL0 RTL --> Error: module creation failed -- see below for details. error: double type is not supported on this platform error: backend compiler failed build. ```

If you really want to run on yarrow, you can switch the types in xsbench to "float" and it should run, but the answers might be wrong. It might be worth it to stick on iris for now, or try arcticus.

(Thanks to Roy (Intel) for commenting on the issue!)

Colleen ________________________________ From: Peter McCorquodale <[email protected]> Sent: Wednesday, May 5, 2021 6:41 PM To: Bertoni, Colleen <[email protected]> Cc: Tramm, John R. <[email protected]>; Steinbrecher, Patrick <[email protected]>; Luo, Ye <[email protected]>; [email protected] <[email protected]> Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target

Colleen, OK, I put a print statement at the beginning of the loop that should be offloaded, and it never shows up in the output on yarrow. Yes, I pulled from https://github.com/ANL-CESAR/XSBench.

Peter

On Wed, May 05, 2021 at 11:24:00PM +0000, Bertoni, Colleen wrote:

...

Yarrow has no hardware support for double precision numbers, so it could just be very slow. On the other hand, an XSBench test from John Tramm (CC'd) is known to hang on Yarrow, and we have an open bug with Intel about it (CMPLRLLVM-24234).

So 2 things: 1) maybe try running for 10 min, or adding a print statement to see if it's making progress. 2) John, could you say if the code you put in the test set is the same as what Peter is using (I think just pulled from https://github.com/ANL-CESAR/XSBench, right?)

Colleen ________________________________ From: Peter McCorquodale <[email protected]> Sent: Wednesday, May 5, 2021 5:30 PM To: Bertoni, Colleen <[email protected]> Cc: Steinbrecher, Patrick <[email protected]>; Luo, Ye <[email protected]>; [email protected] <[email protected]> Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target

If I try that with either -n 2 or -n 1, same setting for mpirun and qsub, then that runs fine on iris, in less than 1 minute, but on yarrow, it hangs somewhere in the middle of its execution, and exhausts its 5-minute time limit.

Peter

On Wed, May 05, 2021 at 09:33:39PM +0000, Bertoni, Colleen wrote:

...

Thanks for the response! Could you try

mpirun -n ${num_process} ./XSBench-icx -m event -s small

instead of "./XSBench-icx -m event -s small" ? Some mpi versions need the `mpirun` to launch the job.

Colleen ________________________________ From: Peter McCorquodale <[email protected]> Sent: Wednesday, May 5, 2021 4:25 PM To: Bertoni, Colleen <[email protected]> Cc: Steinbrecher, Patrick <[email protected]>; Luo, Ye <[email protected]>; [email protected] <[email protected]> Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target

Colleen: Yes, on 2 nodes with a 5-minute time limit.

Following Ye Luo's suggestion that I was running out of memory, I tried a smaller problem size, and now it runs to completion on iris, but I still get those MPI errors on yarrow and arcticus. The script is: #!/bin/bash source /usr/share/Modules/init/bash module use /soft/restricted/CNDA/modulefiles module load mpi module load oneapi ./XSBench-icx -m event -s small

Peter

On Wed, May 05, 2021 at 05:15:41PM +0000, Bertoni, Colleen wrote:

...

Hi Peter,

Could you show us `runicx.sh`? The other thing I noticed is that you're executing `qsub -n 2 -t 5 -q iris runicx.sh`, which tries to execute on 2 iris nodes with a time limit of 5 minutes. To be clear, you were trying to run across two iris nodes?

I know we have a version of XSBench which runs on iris, but I think it's only been on a single iris node.

Thanks, Colleen ________________________________ From: Ecp-aurora-sdk-discuss <[email protected]> on behalf of Luo, Ye via Ecp-aurora-sdk-discuss <[email protected]> Sent: Tuesday, May 4, 2021 7:09 PM To: Steinbrecher, Patrick <[email protected]>; Peter McCorquodale <[email protected]> Cc: [email protected] <[email protected]> Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target

Call to getOrAllocTgtPtr returned null pointer It is likely that it ran out of GPU memory Ye ________________________________ From: Ecp-aurora-sdk-discuss <[email protected]> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <[email protected]> Sent: Tuesday, May 4, 2021 6:58 PM To: Steinbrecher, Patrick <[email protected]> Cc: [email protected] <[email protected]> Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target

On Tue, May 04, 2021 at 10:37:38PM +0000, Steinbrecher, Patrick wrote:

...

For OpenMP offload on Intel GPU you should use the flags:

-fiopenmp -fopenmp-targets=spir64

Yes, the full set of flags I have is: -std=gnu99 -Wall -fiopenmp -fopenmp-targets=spir64 -D__STRICT_ANSI__ -g -pg -O3 -DMPI -fopenmp -D__STRICT_ANSI__ -cc=icx

...

I don't know about MPI error. Looks like you are using MPI with threading. Maybe that is not supported with that module. Try to just use MPI_Init.

My source code does just call MPI_Init. And it's only yarrow that gives me the error in PMPI_Init. On iris, I get a different error, later on:

...

From: Ecp-aurora-sdk-discuss <[email protected]> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <[email protected]> Date: Tuesday, May 4, 2021 at 17:29 To: [email protected] <[email protected]> Subject: [Ecp-aurora-sdk-discuss] offloading with OpenMP target I have been building and running the openmp-offload source in XSBench: https://github.com/ANL-CESAR/XSBench

I can get it to build if in openmp-offload/Makefile I set CC = /soft/compilers/intel-2020.4/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpicc and CFLAGS += -fopenmp -D__STRICT_ANSI__ -cc=icx

But then I get consistently different errors when I run it on iris or yarrow.

On yarrow, qsub -n 2 -t 5 -q yarrow runicx.sh it terminates before producing any output: Abort(1094799) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: MPIR_Init_thread(666)......: MPID_Init(922).............: MPIDI_NM_mpi_init_hook(719): OFI addrinfo() failed (ofi_init.h:719:MPIDI_NM_mpi_init_hook:No data available)

On iris, qsub -n 2 -t 5 -q iris runicx.sh it gets further along in execution, as I see from output, until: Libomptarget error: Call to getOrAllocTgtPtr returned null pointer (device failure or illegal mapping). Libomptarget error: Call to targetDataBegin failed, abort target. Libomptarget error: Failed to process data before launching the kernel. Libomptarget error: Run with LIBOMPTARGET_DEBUG=4 to dump host-target pointer mappings. unknown:68:44: Libomptarget fatal error 1: failure of target construct while offloading is mandatory

This suggests that the problem on iris might be with the OpenMP target pragma, but the only #pragma omp target statement in the source code is this, and it works on other non-JLSE platforms: #pragma omp target teams distribute parallel for\ map(to: SD.max_num_nucs)\ map(to: SD.num_nucs[:SD.length_num_nucs])\ map(to: SD.concs[:SD.length_concs])\ map(to: SD.mats[:SD.length_mats])\ map(to: SD.unionized_energy_array[:SD.length_unionized_energy_array])\ map(to: SD.index_grid[:SD.length_index_grid])\ map(to: SD.nuclide_grid[:SD.length_nuclide_grid])\ reduction(+:verification) for( int i = 0; i < in.lookups; i++ ) ...

Peter -- Ecp-aurora-sdk-discuss mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss -- Ecp-aurora-sdk-discuss mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss