Yes, you are 100 correct! My bad, I didn't see the pnext​ value. Thanks to you we discovered that a missing functionally with our tracing, we will dump all the pnext value! It will be more obvious. Thanks! I was able to reproduce when using our simple ruby reproducer below: ``` thapi_root = ENV["THAPI_ROOT"] require "/#{thapi_root}/share/ze_bindings" driver = ZE::drivers.first device = driver.devices.first context = driver.context_create size_alloc_device = 2*device.properties[:maxMemAllocSize] size_malloc = 10000 c = FFI::MemoryPointer.new(size_malloc) a = context.mem_alloc_device(size_alloc_device, device, relaxed_allocation_limits_exp_flags: :ZE_RELAXED_ALLOCATION_LIMITS_EXP_FLAG_MAX_SIZE) command_queue = context.command_queue_create(device) command_list = context.command_list_create(device) command_list.append_memory_copy(a,c, size:size_malloc) command_list.close command_queue.execute_command_lists(command_list) ``` If I run this script inside a loop, it will hang the node. Progress! Thanks a lot for the reproducer and bug report. I will let you know when we got any update Thomas Applencourt Leadership Computing Facility Argonne National Laboratory Building 240 Office - 1.D.18 9700 S Cass Av., Lemont Il, 60439 (630) 252-0036 ________________________________ From: Frank Winter <[email protected]> Sent: Monday, March 7, 2022 14:49 To: Applencourt, Thomas <[email protected]>; [email protected] <[email protected]>; Allen, Benjamin S. <[email protected]>; Bertoni, Colleen <[email protected]> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: weird hang on Arcticus Hi Thomas. This bit should be set. See line 401: https://github.com/fwinter/qdp-jit/blob/feature/avx/lib/qdp_gpu_l0.cc I understand your call trace 1st allocation is the 4 bytes for a float that I copy around for correct timing (only really needed for async kernel calls) 2nd allocation is the pool. I see the *pNext structure being non-zero. The manual wasn't clear to me whether to set this bit inside the *pNext structure or the parent structure. Admitted I have set it in the *pNext structure as I found a version of the Sycl runtime where it's done like that. Besides, when I did not set this flag (in *pNext) then the allocation failed. So, this assured me that I'm probably doing the right thing. Since the parent structure also has a 'flags' for the sake of experimenting I set that bit in the parent too and ran again. Same behavior. First run goes well, then a hang. I left the executable on the node: arcticus09:~/git/package/level0-jit-noquda/build/build_qdpxx/examples> ls -l /tmp/t_dslashm.tgz -rw-r--r-- 1 ac.fwinter jlse 3111981 Mar 7 20:35 /tmp/t_dslashm.tgz If you build this yourself, the line to add is: if ( size > deviceProperties.maxMemAllocSize ) { std::cout << "L0 performance warning: memory allocation size larger than maximum allowed: " << size << " (" << deviceProperties.maxMemAllocSize << ").\n"; memAllocDesc.pNext = &relaxed; std::cout << "bit set\n"; memAllocDesc.flags = ZE_RELAXED_ALLOCATION_LIMITS_EXP_FLAG_MAX_SIZE; // this is extra } I think we still have good leverage to locate this issue. Frank ________________________________ From: Applencourt, Thomas <[email protected]> Sent: Monday, March 7, 2022 1:40 PM To: Frank Winter <[email protected]>; [email protected] <[email protected]>; Allen, Benjamin S. <[email protected]>; Bertoni, Colleen <[email protected]> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: weird hang on Arcticus Hi Frank, To be able to use larger allocation, ZE_RELAXED_ALLOCATION_LIMITS_EXP_FLAG_MAX_SIZE need to be seet (https://spec.oneapi.io/level-zero/latest/core/api.html#_CPPv439ze_relaxed_allocation_limits_exp_flag_t<https://urldefense.proofpoint.com/v2/url?u=https-3A__spec.oneapi.io_level-2Dzero_latest_core_api.html-23-5FCPPv439ze-5Frelaxed-5Fallocation-5Flimits-5Fexp-5Fflag-5Ft&d=DwMGaQ&c=CJqEzB1piLOyyvZjb8YUQw&r=tFpAzszScTWMAFcrGFW5xk9WGxvO1Lei68OSot3V93w&m=CwAxVUhdQrRkABpJA6vqYBt1kzWOREpnnRzqlSn16mexQPNtPb4oumk9p3G0VFcJ&s=_TEBr_eRgxpa6kFRWdQhsHOt7tUOdAFdA9Iua_aZ3SQ&e=>) to when allocating memory I don't see the flags set when tracing your code, 18:26:34.981501615 - arcticus10 - vpid: 28509, vtid: 28509 - lttng_ust_ze:zeMemAllocDevice_entry: { hContext: 0x0000000002463a20, device_desc: 0x00007ffd782d53e0, size: 4, alignment: 1, hDevice: 0x0000000002440590, pptr: 0x0000000000ca7b50, device_desc_val: { stype: ZE_STRUCTURE_TYPE_DEVICE_MEM_ALLOC_DESC, pNext: 0x0000000000000000, flags: [ ], ordinal: 0 } } 18:26:34.981757487 - arcticus10 - vpid: 28509, vtid: 28509 - lttng_ust_ze:zeMemAllocDevice_exit: { zeResult: ZE_RESULT_SUCCESS, pptr_val: 0xffffc001ffd50000 } 18:26:35.474509495 - arcticus10 - vpid: 28509, vtid: 28509 - lttng_ust_ze:zeMemAllocDevice_entry: { hContext: 0x0000000002463a20, device_desc: 0x00007ffd782d2b20, size: 4294967552, alignment: 1, hDevice: 0x0000000002440590, pptr: 0x0000000002489220, device_desc_val: { stype: ZE_STRUCTURE_TYPE_DEVICE_MEM_ALLOC_DESC, pNext: 0x00007ffd782d2b00, flags: [ ], ordinal: 0 } } 18:26:35.536957811 - arcticus10 - vpid: 28509, vtid: 28509 - lttng_ust_ze:zeMemAllocDevice_exit: { zeResult: ZE_RESULT_SUCCESS, pptr_val: 0xffffd556aaa00000 } So maybe this code is triggering "undefined behavior". Maybe the first test is to try to set these flags and see. Saying that why the allocation even works without this flag is another question... I will ask. Hope this help, Thomas Thomas Applencourt Leadership Computing Facility Argonne National Laboratory Building 240 Office - 1.D.18 9700 S Cass Av., Lemont Il, 60439 (630) 252-0036 ________________________________ From: Applencourt, Thomas <[email protected]> Sent: Monday, March 7, 2022 12:11 To: Frank Winter <[email protected]>; [email protected] <[email protected]>; Allen, Benjamin S. <[email protected]>; Bertoni, Colleen <[email protected]> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: weird hang on Arcticus Can reproduce! Will investigate thanks a lot. First run works well, and then it hangs. Interesting.

Great not to be the only one seeing that sort of hang.

A lot of people have hang with large problem sizes. One of our main bug.! Thomas Applencourt Leadership Computing Facility Argonne National Laboratory Building 240 Office - 1.D.18 9700 S Cass Av., Lemont Il, 60439 (630) 252-0036 ________________________________ From: Frank Winter <[email protected]> Sent: Monday, March 7, 2022 11:59 To: Applencourt, Thomas <[email protected]>; [email protected] <[email protected]>; Allen, Benjamin S. <[email protected]>; Bertoni, Colleen <[email protected]> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: weird hang on Arcticus Hi Thomas. Great not to be the only one seeing that sort of hang. I zipped the reproducer and left it on /tmp: ac.fwinter@jlselogin5:~> ls -l /tmp/reproducer.tgz -rw-r--r-- 1 ac.fwinter jlse 41164393 Mar 4 23:21 /tmp/reproducer.tgz (mind the dependence on LLVM) Or, if you want to build this yourself the easiest would be to download a 'meta' package which downloads and builds everything for you. That package is currently configured for Arcticus. git clone https://github.com/fwinter/package.git cd package ./download_sources_intel.sh cd level0-jit-noquda/ ./build_all.sh Then there should be a t_dslashm in: ./build/build_qdpxx/examples/ Sitting in that directory the following shell script would hang on iteration 3 (or so) #!/bin/bash EXE=./t_dslashm for i in `seq 4 2 24` do echo $'\nLattice size = '$i $EXE -lat $i -opt-shift -poolsize 24g done .. and it won't hang with '-poolsize 3.9g'. Frank ________________________________ From: Applencourt, Thomas <[email protected]> Sent: Monday, March 7, 2022 12:10 PM To: [email protected] <[email protected]>; Allen, Benjamin S. <[email protected]>; Bertoni, Colleen <[email protected]>; Frank Winter <[email protected]> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: weird hang on Arcticus Hi Frank, Oh, nice. I saw a hang when using lot of memory too. I reported the bug, but you seem different. I was not able to find the source code in your reproducer in ` ~ac.fwinter/reproducer/. I just have the binary (​t_dslashm1), do you have the source code somewhere? I can try to write a reproducer if I just need to do an alloc in L0 with the relaxed flag if you think. Regards, Thomas Thomas Applencourt Leadership Computing Facility Argonne National Laboratory Building 240 Office - 1.D.18 9700 S Cass Av., Lemont Il, 60439 (630) 252-0036 ________________________________ From: Ecp-aurora-sdk-discuss <[email protected]> on behalf of Frank Winter via Ecp-aurora-sdk-discuss <[email protected]> Sent: Monday, March 7, 2022 10:50 To: [email protected] <[email protected]>; Allen, Benjamin S. <[email protected]>; Bertoni, Colleen <[email protected]> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: weird hang on Arcticus I narrowed it down a little. This only happens when the program allocates a big chunk of memory, like 24 GB (of the 32 GB available on the card). In that case the program is run two or three times by the shell script where the next run would hang forever. I checked that the program is not put into background; every iteration of the shell script loop waits for the process to exit. Now, if I limit the memory allocation to something smaller, like 3.9GB, then all is good. In the reproducer you can change '-poolsize 24g' to '-poolsize 3.9g' to do that. In Level 0, memory allocation that are larger than the 'maximum allocation size' of 4GB must be done with the 'relaxed' attribute. There might be a problem with memory cleanup when using the relaxed memory allocation path. Frank ________________________________ From: Ecp-aurora-sdk-discuss <[email protected]> on behalf of Ben Allen via Ecp-aurora-sdk-discuss <[email protected]> Sent: Friday, March 4, 2022 6:33 PM To: [email protected] <[email protected]> Subject: [EXTERNAL] Re: [Ecp-aurora-sdk-discuss] weird hang on Arcticus I grabbed this for Colleen. Thanks Frank. Ben On 3/4/22 5:51 PM, Bertoni, Colleen via Ecp-aurora-sdk-discuss wrote:

Thanks for the report!

Could you tar and send the reproducer here, or put it in /tmp/ on a login node of JLSE (and tell us which login node)? We don't have permission to see into your home directory.

Thanks, Colleen ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -- *From:* Ecp-aurora-sdk-discuss <[email protected]> on behalf of Frank Winter via Ecp-aurora-sdk-discuss <[email protected]> *Sent:* Friday, March 4, 2022 4:49 PM *To:* [email protected] <[email protected]> *Subject:* [Ecp-aurora-sdk-discuss] weird hang on Arcticus Since today on Arcticus, there's a weird hang in my bash script that wasn't there before. It's a simple script that calls the program to benchmark in a loop. At iteration 3 it hangs. Forever, from what I can see.

I put together a little reproducer in ~ac.fwinter/reproducer/

Reproduce with: . env_arcticus.sh ./go.sh

...

./go.sh

Lattice size = 6 LAT 6.50828673473069

Lattice size = 8 LAT 20.287761797337

Lattice size = 10 (hangs)

After that I see the process being there and it can't be killed.

Frank

-- Ecp-aurora-sdk-discuss mailing list [email protected] https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.jlse.anl.gov_mail...