March 2022 - Ecp-aurora-sdk-discuss - mailman3.cels.anl.gov

Re: [Ecp-aurora-sdk-discuss] Large slowdown with latest oneapi compiler
by Bertoni, Colleen 30 Mar '22

30 Mar '22

Thanks for the report! For performance issues normally we want to try to figure out whether the regression is coming from the compiler or the runtime or somewhere else. So you can check a couple of things: --Check if the number of work groups/work items the runtime uses has changed for that kernel. You can do that with `iprof -k ./a.out...` and look at the number of work groups/work items for the kernel. --Try using the current SDK but with the runtime from the last SDK and vice versa (for example, module load oneapi; module switch -f intel_compute_runtime/release/agama-prerelease-316), and maybe we can pinpoint if it's from the compiler or the runtime. Other people on the list might have ideas, but this is where I'd start. If you have time, we can have a call and look at it together. Thanks, Colleen ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Moore, Stan via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Wednesday, March 30, 2022 9:56 AM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] Large slowdown with latest oneapi compiler Hi everyone, I'm seeing a ~6x overall slowdown for the EXAALT KPP benchmark (LAMMPS SNAP) when switching from oneapi/eng-compiler/2022.01.30.002 to oneapi/eng-compiler/2022.01.30.003. This is using the Kokkos SYCL backend, keeping both Kokkos and LAMMPS versions constant. I profiled the kernels and the most expensive kernel (ComputeYi) became ~10x slower due to the compiler update. Most of the other kernels have similar performance between the two compiler versions. I have set SYCL_PROGRAM_COMPILE_OPTIONS="-ze-opt-large-register-file" but it doesn't seem to have any effect. Any ideas? Thank you, Stan

1 0

Large slowdown with latest oneapi compiler
by Moore, Stan 30 Mar '22

30 Mar '22

Hi everyone, I'm seeing a ~6x overall slowdown for the EXAALT KPP benchmark (LAMMPS SNAP) when switching from oneapi/eng-compiler/2022.01.30.002 to oneapi/eng-compiler/2022.01.30.003. This is using the Kokkos SYCL backend, keeping both Kokkos and LAMMPS versions constant. I profiled the kernels and the most expensive kernel (ComputeYi) became ~10x slower due to the compiler update. Most of the other kernels have similar performance between the two compiler versions. I have set SYCL_PROGRAM_COMPILE_OPTIONS="-ze-opt-large-register-file" but it doesn't seem to have any effect. Any ideas? Thank you, Stan

1 0

New Aurora Compiler+MKL available in JLSE
by Chan-nui, Christopher 18 Mar '22

18 Mar '22

Hello, We are pleased to announce that a new Aurora SDK engineering drop with compiler and mkl updates has been deployed in JLSE. This module version is available as onapi or oneapi/eng-compiler/2022.01.30.003 For additional details please refer to the latest release notes in the Wiki: https://wiki.jlse.anl.gov/display/inteldga/DGA+SDK+Release+Notes -- Christopher Chan-Nui christopher.chan-nui(a)intel.com

1 0

Upcoming oneapi update
by Chan-nui, Christopher 18 Mar '22

18 Mar '22

At 5pm CDT the oneapi modulefile will be updated. Running jobs will not be affected. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com

1 0

Additional ATS-P nodes available at DevEP cluster
by Muralidharan, Servesh 17 Mar '22

17 Mar '22

Hello Arcticus node users, Since we have an ongoing ESP dungeon which has a reservation in place, we are seeing several jobs waiting on queue to get a node. We wanted to remind you that there are additional nodes with the same config/oneapi module setup available via DevEP cluster (https://wiki.jlse.anl.gov/display/inteldga/Intel+DevEP) If your work permits, please consider using the additional resource. Access to the DevEP cluster should be available to all users who currently have access to Arcticus. Reach out to me or DevEP.JLSE.Help@<mailto:[email protected]>intel.com<http://intel.com> should you have issues accessing the nodes. On behalf of early hardware team. -- Servesh

1 0

Re: [Ecp-aurora-sdk-discuss] Accessing the GPU on Arcticus nodes in Singularity
by Muralidharan, Servesh 09 Mar '22

09 Mar '22

Hi Sameer, If you can pass the host env to Singularity then what you can do, is remove any UMD driver stack from within the image and then load the appropriate runtime outside of the Singularity image and run it. -- Servesh From: Sameer Shende <sameer(a)cs.uoregon.edu> Date: Wednesday, March 9, 2022 at 12:59 PM To: Muralidharan, Servesh <servesh(a)anl.gov> Cc: Sameer Shende <sameer(a)cs.uoregon.edu>, ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] Accessing the GPU on Arcticus nodes in Singularity Hi Servesh, Yes, the image has no NDA components and uses publicly available Intel drivers and software. That would explain why it can’t support the ATS hardware. Do you know of any other images that can use the GPU built with proprietary NDA drivers and software? Thanks, - Sameer On Mar 9, 2022, at 10:49 AM, Muralidharan, Servesh <servesh(a)anl.gov> wrote: Hi Sameer, Happen to know how this image was built? I suspect that you are using the public user mode drivers and IGC which do not work on these Intel early testebed hw. The Arcticus nodes use drivers that are not publicly available and are restricted to use on the specific nodes with a specific config. The same reason this image may work on Gen9 or Iris nodes is those are using the public user mode drivers and upstream kernel. -- Servesh From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Sameer Shende via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Date: Wednesday, March 9, 2022 at 12:38 PM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] Accessing the GPU on Arcticus nodes in Singularity Hi, I am unable to access the GPU on Arcticus nodes in Singularity but I have no problem accessing the GPUs on Iris nodes using the same image in Singularity. It also works on integrated GPUs (TigerLake Gen12 laptop) and on Intel DG1 systems. Here is the output: ac.sshende@arcticus10:~/scr/images> singularity run ./ecp.simg INFO: Converting SIF file to temporary sandbox... Singularity> source /opt/intel/oneapi/setvars.sh :: initializing oneAPI environment ... bash: BASH_VERSION = 5.0.17(1)-release args: Using "$@" for setvars.sh arguments: ... :: vpl -- latest :: oneAPI environment initialized :: Singularity> clinfo -l Platform #0: Intel(R) FPGA Emulation Platform for OpenCL(TM) `-- Device #0: Intel(R) FPGA Emulation Device Platform #1: Intel(R) OpenCL `-- Device #0: Intel(R) Xeon(R) Gold 6336Y CPU @ 2.40GHz Platform #2: AMD Accelerated Parallel Processing Singularity> and on Iris: ac.sshende@iris09:~/scr/images> singularity run ./ecp.simg INFO: Converting SIF file to temporary sandbox... Singularity> source /opt/intel/oneapi/setvars.sh :: initializing oneAPI environment ... bash: BASH_VERSION = 5.0.17(1)-release args: Using "$@" for setvars.sh arguments: :: ccl -- latest ... :: vpl -- latest :: oneAPI environment initialized :: Singularity> clinfo -l Platform #0: Intel(R) FPGA Emulation Platform for OpenCL(TM) `-- Device #0: Intel(R) FPGA Emulation Device Platform #1: Intel(R) OpenCL `-- Device #0: Intel(R) Xeon(R) CPU E3-1585 v5 @ 3.50GHz Platform #2: AMD Accelerated Parallel Processing Platform #3: Intel(R) OpenCL HD Graphics `-- Device #0: Intel(R) Iris(R) Pro Graphics P580 [0x193a] Singularity> The first output is missing the Platform #3. The image (from E4S.io) is in: /home/ac.sshende/scr/images/ecp.simg. Has anyone accessed the GPU in Singularity on Arcticus? Would you please point me to an image that works? Thanks, - Sameer -- Ecp-aurora-sdk-discuss mailing list Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

Re: [Ecp-aurora-sdk-discuss] Accessing the GPU on Arcticus nodes in Singularity
by Sameer Shende 09 Mar '22

09 Mar '22

Hi Servesh, Yes, the image has no NDA components and uses publicly available Intel drivers and software. That would explain why it can’t support the ATS hardware. Do you know of any other images that can use the GPU built with proprietary NDA drivers and software? Thanks, - Sameer > On Mar 9, 2022, at 10:49 AM, Muralidharan, Servesh <servesh(a)anl.gov> wrote: > > > Hi Sameer, > > Happen to know how this image was built? > > I suspect that you are using the public user mode drivers and IGC which do not work on these Intel early testebed hw. > The Arcticus nodes use drivers that are not publicly available and are restricted to use on the specific nodes with a specific config. > > The same reason this image may work on Gen9 or Iris nodes is those are using the public user mode drivers and upstream kernel. > > -- > Servesh > > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Sameer Shende via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Date: Wednesday, March 9, 2022 at 12:38 PM > To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Subject: [Ecp-aurora-sdk-discuss] Accessing the GPU on Arcticus nodes in Singularity > > Hi, > I am unable to access the GPU on Arcticus nodes in Singularity but I have no problem accessing the GPUs on Iris nodes using the same image in Singularity. It also works on integrated GPUs (TigerLake Gen12 laptop) and on Intel DG1 systems. Here is the output: > ac.sshende@arcticus10:~/scr/images> singularity run ./ecp.simg > INFO: Converting SIF file to temporary sandbox... > Singularity> source /opt/intel/oneapi/setvars.sh > :: initializing oneAPI environment ... > bash: BASH_VERSION = 5.0.17(1)-release > args: Using "$@" for setvars.sh arguments: > ... > :: vpl -- latest > :: oneAPI environment initialized :: > Singularity> clinfo -l > Platform #0: Intel(R) FPGA Emulation Platform for OpenCL(TM) > `-- Device #0: Intel(R) FPGA Emulation Device > Platform #1: Intel(R) OpenCL > `-- Device #0: Intel(R) Xeon(R) Gold 6336Y CPU @ 2.40GHz > Platform #2: AMD Accelerated Parallel Processing > Singularity> > > and on Iris: > ac.sshende@iris09:~/scr/images> singularity run ./ecp.simg > INFO: Converting SIF file to temporary sandbox... > Singularity> source /opt/intel/oneapi/setvars.sh > :: initializing oneAPI environment ... > bash: BASH_VERSION = 5.0.17(1)-release > args: Using "$@" for setvars.sh arguments: > :: ccl -- latest > ... > :: vpl -- latest > :: oneAPI environment initialized :: > Singularity> clinfo -l > Platform #0: Intel(R) FPGA Emulation Platform for OpenCL(TM) > `-- Device #0: Intel(R) FPGA Emulation Device > Platform #1: Intel(R) OpenCL > `-- Device #0: Intel(R) Xeon(R) CPU E3-1585 v5 @ 3.50GHz > Platform #2: AMD Accelerated Parallel Processing > Platform #3: Intel(R) OpenCL HD Graphics > `-- Device #0: Intel(R) Iris(R) Pro Graphics P580 [0x193a] > Singularity> > > The first output is missing the Platform #3. The image (from E4S.io) is in: /home/ac.sshende/scr/images/ecp.simg. > > Has anyone accessed the GPU in Singularity on Arcticus? Would you please point me to an image that works? > Thanks, > - Sameer > > > -- > Ecp-aurora-sdk-discuss mailing list > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

Re: [Ecp-aurora-sdk-discuss] Accessing the GPU on Arcticus nodes in Singularity
by Muralidharan, Servesh 09 Mar '22

09 Mar '22

Hi Sameer, Happen to know how this image was built? I suspect that you are using the public user mode drivers and IGC which do not work on these Intel early testebed hw. The Arcticus nodes use drivers that are not publicly available and are restricted to use on the specific nodes with a specific config. The same reason this image may work on Gen9 or Iris nodes is those are using the public user mode drivers and upstream kernel. -- Servesh From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Sameer Shende via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Date: Wednesday, March 9, 2022 at 12:38 PM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] Accessing the GPU on Arcticus nodes in Singularity Hi, I am unable to access the GPU on Arcticus nodes in Singularity but I have no problem accessing the GPUs on Iris nodes using the same image in Singularity. It also works on integrated GPUs (TigerLake Gen12 laptop) and on Intel DG1 systems. Here is the output: ac.sshende@arcticus10:~/scr/images> singularity run ./ecp.simg INFO: Converting SIF file to temporary sandbox... Singularity> source /opt/intel/oneapi/setvars.sh :: initializing oneAPI environment ... bash: BASH_VERSION = 5.0.17(1)-release args: Using "$@" for setvars.sh arguments: ... :: vpl -- latest :: oneAPI environment initialized :: Singularity> clinfo -l Platform #0: Intel(R) FPGA Emulation Platform for OpenCL(TM) `-- Device #0: Intel(R) FPGA Emulation Device Platform #1: Intel(R) OpenCL `-- Device #0: Intel(R) Xeon(R) Gold 6336Y CPU @ 2.40GHz Platform #2: AMD Accelerated Parallel Processing Singularity> and on Iris: ac.sshende@iris09:~/scr/images> singularity run ./ecp.simg INFO: Converting SIF file to temporary sandbox... Singularity> source /opt/intel/oneapi/setvars.sh :: initializing oneAPI environment ... bash: BASH_VERSION = 5.0.17(1)-release args: Using "$@" for setvars.sh arguments: :: ccl -- latest ... :: vpl -- latest :: oneAPI environment initialized :: Singularity> clinfo -l Platform #0: Intel(R) FPGA Emulation Platform for OpenCL(TM) `-- Device #0: Intel(R) FPGA Emulation Device Platform #1: Intel(R) OpenCL `-- Device #0: Intel(R) Xeon(R) CPU E3-1585 v5 @ 3.50GHz Platform #2: AMD Accelerated Parallel Processing Platform #3: Intel(R) OpenCL HD Graphics `-- Device #0: Intel(R) Iris(R) Pro Graphics P580 [0x193a] Singularity> The first output is missing the Platform #3. The image (from E4S.io) is in: /home/ac.sshende/scr/images/ecp.simg. Has anyone accessed the GPU in Singularity on Arcticus? Would you please point me to an image that works? Thanks, - Sameer -- Ecp-aurora-sdk-discuss mailing list Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

Accessing the GPU on Arcticus nodes in Singularity
by Sameer Shende 09 Mar '22

09 Mar '22

Hi, I am unable to access the GPU on Arcticus nodes in Singularity but I have no problem accessing the GPUs on Iris nodes using the same image in Singularity. It also works on integrated GPUs (TigerLake Gen12 laptop) and on Intel DG1 systems. Here is the output: ac.sshende@arcticus10:~/scr/images> singularity run ./ecp.simg INFO: Converting SIF file to temporary sandbox... Singularity> source /opt/intel/oneapi/setvars.sh :: initializing oneAPI environment ... bash: BASH_VERSION = 5.0.17(1)-release args: Using "$@" for setvars.sh arguments: ... :: vpl -- latest :: oneAPI environment initialized :: Singularity> clinfo -l Platform #0: Intel(R) FPGA Emulation Platform for OpenCL(TM) `-- Device #0: Intel(R) FPGA Emulation Device Platform #1: Intel(R) OpenCL `-- Device #0: Intel(R) Xeon(R) Gold 6336Y CPU @ 2.40GHz Platform #2: AMD Accelerated Parallel Processing Singularity> and on Iris: ac.sshende@iris09:~/scr/images> singularity run ./ecp.simg INFO: Converting SIF file to temporary sandbox... Singularity> source /opt/intel/oneapi/setvars.sh :: initializing oneAPI environment ... bash: BASH_VERSION = 5.0.17(1)-release args: Using "$@" for setvars.sh arguments: :: ccl -- latest ... :: vpl -- latest :: oneAPI environment initialized :: Singularity> clinfo -l Platform #0: Intel(R) FPGA Emulation Platform for OpenCL(TM) `-- Device #0: Intel(R) FPGA Emulation Device Platform #1: Intel(R) OpenCL `-- Device #0: Intel(R) Xeon(R) CPU E3-1585 v5 @ 3.50GHz Platform #2: AMD Accelerated Parallel Processing Platform #3: Intel(R) OpenCL HD Graphics `-- Device #0: Intel(R) Iris(R) Pro Graphics P580 [0x193a] Singularity> The first output is missing the Platform #3. The image (from E4S.io) is in: /home/ac.sshende/scr/images/ecp.simg. Has anyone accessed the GPU in Singularity on Arcticus? Would you please point me to an image that works? Thanks, - Sameer

1 0

Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: weird hang on Arcticus
by Applencourt, Thomas 07 Mar '22

07 Mar '22

Yes, you are 100 correct! My bad, I didn't see the pnext value. Thanks to you we discovered that a missing functionally with our tracing, we will dump all the pnext value! It will be more obvious. Thanks! I was able to reproduce when using our simple ruby reproducer below: ``` thapi_root = ENV["THAPI_ROOT"] require "/#{thapi_root}/share/ze_bindings" driver = ZE::drivers.first device = driver.devices.first context = driver.context_create size_alloc_device = 2*device.properties[:maxMemAllocSize] size_malloc = 10000 c = FFI::MemoryPointer.new(size_malloc) a = context.mem_alloc_device(size_alloc_device, device, relaxed_allocation_limits_exp_flags: :ZE_RELAXED_ALLOCATION_LIMITS_EXP_FLAG_MAX_SIZE) command_queue = context.command_queue_create(device) command_list = context.command_list_create(device) command_list.append_memory_copy(a,c, size:size_malloc) command_list.close command_queue.execute_command_lists(command_list) ``` If I run this script inside a loop, it will hang the node. Progress! Thanks a lot for the reproducer and bug report. I will let you know when we got any update Thomas Applencourt Leadership Computing Facility Argonne National Laboratory Building 240 Office - 1.D.18 9700 S Cass Av., Lemont Il, 60439 (630) 252-0036 ________________________________ From: Frank Winter <fwinter(a)jlab.org> Sent: Monday, March 7, 2022 14:49 To: Applencourt, Thomas <tapplencourt(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov>; Bertoni, Colleen <bertoni(a)anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: weird hang on Arcticus Hi Thomas. This bit should be set. See line 401: https://github.com/fwinter/qdp-jit/blob/feature/avx/lib/qdp_gpu_l0.cc I understand your call trace 1st allocation is the 4 bytes for a float that I copy around for correct timing (only really needed for async kernel calls) 2nd allocation is the pool. I see the *pNext structure being non-zero. The manual wasn't clear to me whether to set this bit inside the *pNext structure or the parent structure. Admitted I have set it in the *pNext structure as I found a version of the Sycl runtime where it's done like that. Besides, when I did not set this flag (in *pNext) then the allocation failed. So, this assured me that I'm probably doing the right thing. Since the parent structure also has a 'flags' for the sake of experimenting I set that bit in the parent too and ran again. Same behavior. First run goes well, then a hang. I left the executable on the node: arcticus09:~/git/package/level0-jit-noquda/build/build_qdpxx/examples> ls -l /tmp/t_dslashm.tgz -rw-r--r-- 1 ac.fwinter jlse 3111981 Mar 7 20:35 /tmp/t_dslashm.tgz If you build this yourself, the line to add is: if ( size > deviceProperties.maxMemAllocSize ) { std::cout << "L0 performance warning: memory allocation size larger than maximum allowed: " << size << " (" << deviceProperties.maxMemAllocSize << ").\n"; memAllocDesc.pNext = &relaxed; std::cout << "bit set\n"; memAllocDesc.flags = ZE_RELAXED_ALLOCATION_LIMITS_EXP_FLAG_MAX_SIZE; // this is extra } I think we still have good leverage to locate this issue. Frank ________________________________ From: Applencourt, Thomas <tapplencourt(a)anl.gov> Sent: Monday, March 7, 2022 1:40 PM To: Frank Winter <fwinter(a)jlab.org>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov>; Bertoni, Colleen <bertoni(a)anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: weird hang on Arcticus Hi Frank, To be able to use larger allocation, ZE_RELAXED_ALLOCATION_LIMITS_EXP_FLAG_MAX_SIZE need to be seet (https://spec.oneapi.io/level-zero/latest/core/api.html#_CPPv439ze_relaxed_a…<https://urldefense.proofpoint.com/v2/url?u=https-3A__spec.oneapi.io_level-2…>) to when allocating memory I don't see the flags set when tracing your code, 18:26:34.981501615 - arcticus10 - vpid: 28509, vtid: 28509 - lttng_ust_ze:zeMemAllocDevice_entry: { hContext: 0x0000000002463a20, device_desc: 0x00007ffd782d53e0, size: 4, alignment: 1, hDevice: 0x0000000002440590, pptr: 0x0000000000ca7b50, device_desc_val: { stype: ZE_STRUCTURE_TYPE_DEVICE_MEM_ALLOC_DESC, pNext: 0x0000000000000000, flags: [ ], ordinal: 0 } } 18:26:34.981757487 - arcticus10 - vpid: 28509, vtid: 28509 - lttng_ust_ze:zeMemAllocDevice_exit: { zeResult: ZE_RESULT_SUCCESS, pptr_val: 0xffffc001ffd50000 } 18:26:35.474509495 - arcticus10 - vpid: 28509, vtid: 28509 - lttng_ust_ze:zeMemAllocDevice_entry: { hContext: 0x0000000002463a20, device_desc: 0x00007ffd782d2b20, size: 4294967552, alignment: 1, hDevice: 0x0000000002440590, pptr: 0x0000000002489220, device_desc_val: { stype: ZE_STRUCTURE_TYPE_DEVICE_MEM_ALLOC_DESC, pNext: 0x00007ffd782d2b00, flags: [ ], ordinal: 0 } } 18:26:35.536957811 - arcticus10 - vpid: 28509, vtid: 28509 - lttng_ust_ze:zeMemAllocDevice_exit: { zeResult: ZE_RESULT_SUCCESS, pptr_val: 0xffffd556aaa00000 } So maybe this code is triggering "undefined behavior". Maybe the first test is to try to set these flags and see. Saying that why the allocation even works without this flag is another question... I will ask. Hope this help, Thomas Thomas Applencourt Leadership Computing Facility Argonne National Laboratory Building 240 Office - 1.D.18 9700 S Cass Av., Lemont Il, 60439 (630) 252-0036 ________________________________ From: Applencourt, Thomas <tapplencourt(a)anl.gov> Sent: Monday, March 7, 2022 12:11 To: Frank Winter <fwinter(a)jlab.org>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov>; Bertoni, Colleen <bertoni(a)anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: weird hang on Arcticus Can reproduce! Will investigate thanks a lot. First run works well, and then it hangs. Interesting. > Great not to be the only one seeing that sort of hang. A lot of people have hang with large problem sizes. One of our main bug.! Thomas Applencourt Leadership Computing Facility Argonne National Laboratory Building 240 Office - 1.D.18 9700 S Cass Av., Lemont Il, 60439 (630) 252-0036 ________________________________ From: Frank Winter <fwinter(a)jlab.org> Sent: Monday, March 7, 2022 11:59 To: Applencourt, Thomas <tapplencourt(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov>; Bertoni, Colleen <bertoni(a)anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: weird hang on Arcticus Hi Thomas. Great not to be the only one seeing that sort of hang. I zipped the reproducer and left it on /tmp: ac.fwinter@jlselogin5:~> ls -l /tmp/reproducer.tgz -rw-r--r-- 1 ac.fwinter jlse 41164393 Mar 4 23:21 /tmp/reproducer.tgz (mind the dependence on LLVM) Or, if you want to build this yourself the easiest would be to download a 'meta' package which downloads and builds everything for you. That package is currently configured for Arcticus. git clone https://github.com/fwinter/package.git cd package ./download_sources_intel.sh cd level0-jit-noquda/ ./build_all.sh Then there should be a t_dslashm in: ./build/build_qdpxx/examples/ Sitting in that directory the following shell script would hang on iteration 3 (or so) #!/bin/bash EXE=./t_dslashm for i in `seq 4 2 24` do echo $'\nLattice size = '$i $EXE -lat $i -opt-shift -poolsize 24g done .. and it won't hang with '-poolsize 3.9g'. Frank ________________________________ From: Applencourt, Thomas <tapplencourt(a)anl.gov> Sent: Monday, March 7, 2022 12:10 PM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov>; Bertoni, Colleen <bertoni(a)anl.gov>; Frank Winter <fwinter(a)jlab.org> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: weird hang on Arcticus Hi Frank, Oh, nice. I saw a hang when using lot of memory too. I reported the bug, but you seem different. I was not able to find the source code in your reproducer in ` ~ac.fwinter/reproducer/. I just have the binary (t_dslashm1), do you have the source code somewhere? I can try to write a reproducer if I just need to do an alloc in L0 with the relaxed flag if you think. Regards, Thomas Thomas Applencourt Leadership Computing Facility Argonne National Laboratory Building 240 Office - 1.D.18 9700 S Cass Av., Lemont Il, 60439 (630) 252-0036 ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Frank Winter via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Monday, March 7, 2022 10:50 To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov>; Bertoni, Colleen <bertoni(a)anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] [EXTERNAL] Re: weird hang on Arcticus I narrowed it down a little. This only happens when the program allocates a big chunk of memory, like 24 GB (of the 32 GB available on the card). In that case the program is run two or three times by the shell script where the next run would hang forever. I checked that the program is not put into background; every iteration of the shell script loop waits for the process to exit. Now, if I limit the memory allocation to something smaller, like 3.9GB, then all is good. In the reproducer you can change '-poolsize 24g' to '-poolsize 3.9g' to do that. In Level 0, memory allocation that are larger than the 'maximum allocation size' of 4GB must be done with the 'relaxed' attribute. There might be a problem with memory cleanup when using the relaxed memory allocation path. Frank ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Ben Allen via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Friday, March 4, 2022 6:33 PM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [EXTERNAL] Re: [Ecp-aurora-sdk-discuss] weird hang on Arcticus I grabbed this for Colleen. Thanks Frank. Ben On 3/4/22 5:51 PM, Bertoni, Colleen via Ecp-aurora-sdk-discuss wrote: > Thanks for the report! > > Could you tar and send the reproducer here, or put it in /tmp/ on a login node of JLSE (and tell us which login node)? We don't have permission to see into your home directory. > > Thanks, > Colleen > ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -- > *From:* Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Frank Winter via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > *Sent:* Friday, March 4, 2022 4:49 PM > *To:* ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > *Subject:* [Ecp-aurora-sdk-discuss] weird hang on Arcticus > Since today on Arcticus, there's a weird hang in my bash script that wasn't there before. It's a simple script that calls the program to benchmark in a loop. At iteration 3 it hangs. Forever, from what I can see. > > I put together a little reproducer in > ~ac.fwinter/reproducer/ > > Reproduce with: > . env_arcticus.sh > ./go.sh > > > ./go.sh > > Lattice size = 6 > LAT 6.50828673473069 > > Lattice size = 8 > LAT 20.287761797337 > > Lattice size = 10 > (hangs) > > After that I see the process being there and it can't be killed. > > Frank > > -- Ecp-aurora-sdk-discuss mailing list Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.jlse.anl.gov_mai…

1 0