Hi Thomas.

This bit should be set. See line 401:

https://github.com/fwinter/qdp-jit/blob/feature/avx/lib/qdp_gpu_l0.cc

I understand your call trace
1st allocation is the 4 bytes for a float that I copy around for correct timing (only really needed for async kernel calls)
2nd allocation is the pool. I see the *pNext structure being non-zero.

The manual wasn't clear to me whether to set this bit inside the *pNext structure or the parent structure. Admitted I have set it in the *pNext structure as I found a version of the Sycl runtime where it's done like that.
Besides, when I did not set this flag (in *pNext) then the allocation failed. So, this assured me that I'm probably doing the right thing.

Since the parent structure also has a 'flags' for the sake of experimenting I set that bit in the parent too and ran again. Same behavior. First run goes well, then a hang. I left the executable on the node:

arcticus09:~/git/package/level0-jit-noquda/build/build_qdpxx/examples> ls -l /tmp/t_dslashm.tgz
-rw-r--r-- 1 ac.fwinter jlse 3111981 Mar  7 20:35 /tmp/t_dslashm.tgz

If you build this yourself, the line to add is:

    if ( size > deviceProperties.maxMemAllocSize )
      {
        std::cout << "L0 performance warning: memory allocation size larger than maximum allowed: " << size << " (" << deviceProperties.maxMemAllocSize << ").\n";
        memAllocDesc.pNext = &relaxed;

        std::cout << "bit set\n";
        memAllocDesc.flags = ZE_RELAXED_ALLOCATION_LIMITS_EXP_FLAG_MAX_SIZE;     // this is extra
      }


I think we still have good leverage to locate this issue.


Frank



From: Applencourt, Thomas <tapplencourt@anl.gov>
Sent: Monday, March 7, 2022 1:40 PM
To: Frank Winter <fwinter@jlab.org>; ecp-aurora-sdk-discuss@lists.jlse.anl.gov <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>; Allen, Benjamin S. <bsallen@alcf.anl.gov>; Bertoni, Colleen <bertoni@anl.gov>
Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: weird hang on Arcticus
 
Hi Frank,

To be able to use larger allocation,  ZE_RELAXED_ALLOCATION_LIMITS_EXP_FLAG_MAX_SIZE need to be seet
(https://spec.oneapi.io/level-zero/latest/core/api.html#_CPPv439ze_relaxed_allocation_limits_exp_flag_t) to when allocating memory


I don't see the flags set when tracing your code,

18:26:34.981501615 - arcticus10 - vpid: 28509, vtid: 28509 - lttng_ust_ze:zeMemAllocDevice_entry: { hContext: 0x0000000002463a20, device_desc: 0x00007ffd782d53e0, size: 4, alignment: 1, hDevice: 0x0000000002440590, pptr: 0x0000000000ca7b50, device_desc_val: { stype: ZE_STRUCTURE_TYPE_DEVICE_MEM_ALLOC_DESC, pNext: 0x0000000000000000, flags: [  ], ordinal: 0 } }
18:26:34.981757487 - arcticus10 - vpid: 28509, vtid: 28509 - lttng_ust_ze:zeMemAllocDevice_exit: { zeResult: ZE_RESULT_SUCCESS, pptr_val: 0xffffc001ffd50000 }
18:26:35.474509495 - arcticus10 - vpid: 28509, vtid: 28509 - lttng_ust_ze:zeMemAllocDevice_entry: { hContext: 0x0000000002463a20, device_desc: 0x00007ffd782d2b20, size: 4294967552, alignment: 1, hDevice: 0x0000000002440590, pptr: 0x0000000002489220, device_desc_val: { stype: ZE_STRUCTURE_TYPE_DEVICE_MEM_ALLOC_DESC, pNext: 0x00007ffd782d2b00, flags: [  ], ordinal: 0 } }
18:26:35.536957811 - arcticus10 - vpid: 28509, vtid: 28509 - lttng_ust_ze:zeMemAllocDevice_exit: { zeResult: ZE_RESULT_SUCCESS, pptr_val: 0xffffd556aaa00000 }

So maybe this code is triggering "undefined behavior". Maybe the first test is to try to set these flags and see.

Saying that why the allocation even works without this flag is another question... I will ask.

Hope this help,
Thomas


Thomas Applencourt
Leadership Computing Facility
Argonne National Laboratory
Building 240 Office - 1.D.18
9700 S Cass Av., Lemont Il, 60439
(630) 252-0036
From: Applencourt, Thomas <tapplencourt@anl.gov>
Sent: Monday, March 7, 2022 12:11
To: Frank Winter <fwinter@jlab.org>; ecp-aurora-sdk-discuss@lists.jlse.anl.gov <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>; Allen, Benjamin S. <bsallen@alcf.anl.gov>; Bertoni, Colleen <bertoni@anl.gov>
Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: weird hang on Arcticus
 
Can reproduce!  Will investigate thanks a lot. First run works well, and then it hangs. Interesting.


> Great not to be the only one seeing that sort of hang.

A lot of people have hang with large problem sizes. One of our main bug.!


Thomas Applencourt
Leadership Computing Facility
Argonne National Laboratory
Building 240 Office - 1.D.18
9700 S Cass Av., Lemont Il, 60439
(630) 252-0036
From: Frank Winter <fwinter@jlab.org>
Sent: Monday, March 7, 2022 11:59
To: Applencourt, Thomas <tapplencourt@anl.gov>; ecp-aurora-sdk-discuss@lists.jlse.anl.gov <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>; Allen, Benjamin S. <bsallen@alcf.anl.gov>; Bertoni, Colleen <bertoni@anl.gov>
Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: weird hang on Arcticus
 
Hi Thomas.

Great not to be the only one seeing that sort of hang.

I zipped the reproducer and left it on /tmp:

ac.fwinter@jlselogin5:~> ls -l /tmp/reproducer.tgz
-rw-r--r-- 1 ac.fwinter jlse 41164393 Mar  4 23:21 /tmp/reproducer.tgz

(mind the dependence on LLVM)

Or, if you want to build this yourself the easiest would be to download a 'meta' package which downloads and builds everything for you. That package is currently configured for Arcticus.

git clone https://github.com/fwinter/package.git
cd package
./download_sources_intel.sh
cd level0-jit-noquda/
./build_all.sh

Then there should be a t_dslashm in:

./build/build_qdpxx/examples/

Sitting in that directory the following shell script would hang on iteration 3 (or so)


#!/bin/bash

EXE=./t_dslashm

for i in `seq 4 2 24`
do
    echo $'\nLattice size = '$i
    $EXE -lat $i -opt-shift -poolsize 24g
done


.. and it won't hang with '-poolsize 3.9g'.


Frank



From: Applencourt, Thomas <tapplencourt@anl.gov>
Sent: Monday, March 7, 2022 12:10 PM
To: ecp-aurora-sdk-discuss@lists.jlse.anl.gov <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>; Allen, Benjamin S. <bsallen@alcf.anl.gov>; Bertoni, Colleen <bertoni@anl.gov>; Frank Winter <fwinter@jlab.org>
Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: weird hang on Arcticus
 
Hi Frank,

Oh, nice. I saw a hang when using lot of memory too. I reported the bug, but you seem different.

I was not able to find the source code in your reproducer in ` ~ac.fwinter/reproducer/. I just have the binary (​t_dslashm1), do you have the source code somewhere? I can try to write a reproducer if I just need to do an alloc in L0 with the relaxed flag if you think.


Regards,
Thomas

Thomas Applencourt
Leadership Computing Facility
Argonne National Laboratory
Building 240 Office - 1.D.18
9700 S Cass Av., Lemont Il, 60439
(630) 252-0036
From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces@lists.jlse.anl.gov> on behalf of Frank Winter via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
Sent: Monday, March 7, 2022 10:50
To: ecp-aurora-sdk-discuss@lists.jlse.anl.gov <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>; Allen, Benjamin S. <bsallen@alcf.anl.gov>; Bertoni, Colleen <bertoni@anl.gov>
Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: weird hang on Arcticus
 
I narrowed it down a little. This only happens when the program allocates a big chunk of memory, like 24 GB (of the 32 GB available on the card).
In that case the program is run two or three times by the shell script where the next run would hang forever. I checked that the program is not put into background; every iteration of the shell script loop waits for the process to exit.

Now, if I limit the memory allocation to something smaller, like 3.9GB, then all is good. In the reproducer you can change '-poolsize 24g' to '-poolsize 3.9g' to do that.

In Level 0, memory allocation that are larger than the 'maximum allocation size' of 4GB must be done with the 'relaxed' attribute. 
There might be a problem with memory cleanup when using the relaxed memory allocation path.

Frank


From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces@lists.jlse.anl.gov> on behalf of Ben Allen via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
Sent: Friday, March 4, 2022 6:33 PM
To: ecp-aurora-sdk-discuss@lists.jlse.anl.gov <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
Subject: [EXTERNAL] Re: [Ecp-aurora-sdk-discuss] weird hang on Arcticus
 
I grabbed this for Colleen. Thanks Frank.

Ben

On 3/4/22 5:51 PM, Bertoni, Colleen via Ecp-aurora-sdk-discuss wrote:
> Thanks for the report!
>
> Could you tar and send the reproducer here, or put it in /tmp/ on a login node of JLSE (and tell us which login node)? We don't have permission to see into your home directory.
>
> Thanks,
> Colleen
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 --
> *From:* Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces@lists.jlse.anl.gov> on behalf of Frank Winter via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
> *Sent:* Friday, March 4, 2022 4:49 PM
> *To:* ecp-aurora-sdk-discuss@lists.jlse.anl.gov <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
> *Subject:* [Ecp-aurora-sdk-discuss] weird hang on Arcticus
> Since today on Arcticus, there's a weird hang in my bash script that wasn't there before. It's a simple script that calls the program to benchmark in a loop. At iteration 3 it hangs. Forever, from what I can see.
>
> I put together a little reproducer in
> ~ac.fwinter/reproducer/
>
> Reproduce with:
> . env_arcticus.sh
> ./go.sh
>
>  > ./go.sh
>
> Lattice size = 6
> LAT 6.50828673473069
>
> Lattice size = 8
> LAT 20.287761797337
>
> Lattice size = 10
> (hangs)
>
> After that I see the process being there and it can't be killed.
>
> Frank
>
>
--
Ecp-aurora-sdk-discuss mailing list
Ecp-aurora-sdk-discuss@lists.jlse.anl.gov
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.jlse.anl.gov_mailman_listinfo_ecp-2Daurora-2Dsdk-2Ddiscuss&d=DwICAg&c=CJqEzB1piLOyyvZjb8YUQw&r=tFpAzszScTWMAFcrGFW5xk9WGxvO1Lei68OSot3V93w&m=cwMsb8tm0c8d-YMeliUBcaGgaAC5kt3GnT4kVRtp2hAFVqWjaDD8k3NGGc2v3uKH&s=4B9WwgHLgwd2DdNePafI-VPzKxwARqIsjl-JDLFDpz0&e=