December 2019 - Ecp-aurora-sdk-discuss

New A21 SDK available in JLSE
by Rosales-fernandez, Carlos 19 Dec '19

19 Dec '19

Hi all, We are pleased to announce that a new A21 SDK has been deployed in JLSE. This version 2019.12.15.018 includes multiple improvements, including improved Fortran support and the first daal4py with GPU support. For additional details please refer to the latest release notes in the Wiki: https://wiki.jlse.anl.gov/display/inteldga/DGA+SDK+Release+Notes Modules will enable the newer SDK by default, but the older versions remain available for regression testing. As usual, please let us know of any issues you encounter and any improvements we can include. A couple of important notes: 1. This release does not contain a MKL-DNN (oneDNN) update. 2. A new module has been create for daal4py. Please read the “module help daal4py” text before using it, as it requires activating the Intel Distribution for Python prior to loading the module for things to work correctly. 3. The CNDA version of Intel Advisor has not been updated in this release, but it is scheduled for update in January. Regards, Carlos

1 0

A21 SDK update will happen at 9:30 Central
by Rosales-fernandez, Carlos 19 Dec '19

19 Dec '19

Hi all, Just a heads-up that we will update the A21 SDK at 9:30 CST. At that time default modules will switch to the new version. Running jobs will not be affected. A follow-up email will be send with details once the update is complete. Regards, Carlos

1 0

ComputeCpp v1.2.0 available
by Knight, Christopher J. 18 Dec '19

18 Dec '19

Just an FYI that the sycl_codeplay modules have been updated to point to v1.2.0 this morning. This is the version that ‘module load sycl_codeplay’ will make accessible. The README file has been updated with reported improvements and bug fixes. In particular, the version of LLVM was updated to 7. chris ============= Community Edition 1.2.0 Improvement LLVM version updated to 7 Improvements to how the scheduler starts and stops worker threads Bug Fixes Various minor device compiler and runtime fixes.

1 0

vtune information and concurrent gpu kernels
by Charles Leggett 13 Dec '19

13 Dec '19

I'm looking at the output of some trivial kernels, and have some questions: - can we run vtune on the iris nodes with the new SDK from 11/15? When I tried to run just now, I got this error: vtune: Error: The following events cannot be collected: UNC_IMC_DRAM_DATA_READS,UNC_IMC_DRAM_DATA_WRITES. Consider removing the events from the collection, loading the VTune Profiler (beta) sampling driver using the root credentials, or updating the OS kernel. vtune: Collection failed. this was the command: > vtune -collect gpu-offload -knob enable-stack-collection=true -target-duration-type=veryshort -finalization-mode=full -- PROGRAM - running the publicly available beta 0.3 on my desktop, vtune doesn't show the name of the function being executed in the graphs (see image attached), it just shows "func @ memory address". The programs are compiled with -g, and if I click on the relevant line in the "CPU Time" window, it shows the source, so it can indeed find it. Double clicking on the CPU chart bar, produces a page that says "Source / assembly code for the selected object is not available. Possible reasons are:.... (and a bunch of reasons)". The debug info should be available. Are there special opts for the compilation other than "-g" that one should use? - the GPU kernels are very small (parallel_for is only 1024 wide), have no data dependencies between them, and should be able to run concurrently. They are submitted from separate std::threads using a shared sycl::queue. What is preventing them from running concurrently? thanks, Charles. -- Charles Leggett <CGLeggett(a)lbl.gov> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930

1 0

Re: [Ecp-aurora-sdk-discuss] runaway SYCL kernels
by Peterka, Thomas 05 Dec '19

05 Dec '19

Colleen, Thomas, I can confirm that the latest version of the script works for me. I can now reset the GPU anytime I mess it up. This is exactly what I needed. Thanks! Tom ----------------- Tom Peterka Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave. Lemont, IL 60439 Bldg 240, Rm 2148 e-mail: tpeterka(a)mcs.anl.gov<mailto:[email protected]> web: www.mcs.anl.gov/~tpeterka phone: 630-252-7198 On Dec 5, 2019, at 3:26 PM, Bertoni, Colleen <bertoni(a)anl.gov<mailto:[email protected]>> wrote: Thanks, Thomas! We had the order of commands in the script flipped in the CNDA directory. Ben kindly fixed it, and it should work now. : ) Tom, could you verify it works for you now? Colleen ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of Applencourt, Thomas via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Sent: Thursday, December 5, 2019 3:22 PM To: Bertoni, Colleen via Ecp-aurora-sdk-discuss <Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>>; Peterka, Thomas <tpeterka(a)mcs.anl.gov<mailto:[email protected]>> Subject: Re: [Ecp-aurora-sdk-discuss] runaway SYCL kernels > I tried running the script on another iris node that I did not corrupt, and the script also does not return there. I’m not sure how long it should take. Strange. I just checked the script is working for me (aka no hanging, and returning instantly on `iris19` with a clean env). Maybe something strange about your environment? Can you try with clean `.bashrc`? Thomas Applencourt Leadership Computing Facility Argonne National Laboratory Building 240 Office - 1.D.18 9700 S Cass Av., Lemont Il, 60439 (630) 252-0036 ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of Bertoni, Colleen via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Sent: Thursday, December 5, 2019 15:12 To: Peterka, Thomas <tpeterka(a)mcs.anl.gov<mailto:[email protected]>> Cc: Ecp-aurora-sdk-discuss <Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: Re: [Ecp-aurora-sdk-discuss] runaway SYCL kernels It should not take very long. We'll look into it and get back to you. Colleen ________________________________ From: Peterka, Thomas <tpeterka(a)mcs.anl.gov<mailto:[email protected]>> Sent: Thursday, December 5, 2019 3:09 PM To: Bertoni, Colleen <bertoni(a)anl.gov<mailto:[email protected]>> Cc: Ecp-aurora-sdk-discuss <Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: Re: runaway SYCL kernels Hi Colleen, Hmm, the reset gpu script hangs and does not return within a few minutes. I tried running the script on another iris node that I did not corrupt, and the script also does not return there. I’m not sure how long it should take. Tom ----------------- Tom Peterka Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave. Lemont, IL 60439 Bldg 240, Rm 2148 e-mail: tpeterka(a)mcs.anl.gov<mailto:[email protected]> web: www.mcs.anl.gov/~tpeterka<http://www.mcs.anl.gov/~tpeterka> phone: 630-252-7198 On Dec 5, 2019, at 2:56 PM, Bertoni, Colleen <bertoni(a)anl.gov<mailto:[email protected]>> wrote: Hi Tom, If the GPU hangs and gets in a bad state, after killing the run with Ctrl-C, you should be able to reset the gpu by running "/soft/restricted/CNDA/tools/reset_gpu.sh". Then subsequent runs shouldn't all hang the gpu (unless it's an infinite loop or something). (We run this script as part of the job scheduler epilogue.) Could you try running the reset script for your use cases and let us know if there are any problems? Thanks, Colleen ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of Peterka, Thomas via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Sent: Thursday, December 5, 2019 2:50 PM To: Ecp-aurora-sdk-discuss <Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] runaway SYCL kernels Hi, This is a newbie question. If I run a code with an infinite loop in a SYCL kernel (or in general, a kernel that hangs and doesn’t complete in a reasonable time), kill the program with ctrl-c, fix the code, rebuild, and rerun, the kernel will still hang in the same way. I guess this means the original kernel is still running, and ctrl-c did not kill the kernel. I can log into a different iris node and run there, but each time I hang my program, I need to change nodes. In debugging the code, I’m going to generate multiple instances of runaway kernels. What is the right way to kill a kernel or handle this case? Thanks, Tom ----------------- Tom Peterka Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave. Lemont, IL 60439 Bldg 240, Rm 2148 e-mail: tpeterka(a)mcs.anl.gov<mailto:[email protected]> web: www.mcs.anl.gov/~tpeterka<http://www.mcs.anl.gov/~tpeterka> phone: 630-252-7198

1 0

Re: [Ecp-aurora-sdk-discuss] runaway SYCL kernels
by Bertoni, Colleen 05 Dec '19

05 Dec '19

Thanks, Thomas! We had the order of commands in the script flipped in the CNDA directory. Ben kindly fixed it, and it should work now. : ) Tom, could you verify it works for you now? Colleen ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Applencourt, Thomas via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Thursday, December 5, 2019 3:22 PM To: Bertoni, Colleen via Ecp-aurora-sdk-discuss <Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov>; Peterka, Thomas <tpeterka(a)mcs.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] runaway SYCL kernels > I tried running the script on another iris node that I did not corrupt, and the script also does not return there. I’m not sure how long it should take. Strange. I just checked the script is working for me (aka no hanging, and returning instantly on `iris19` with a clean env). Maybe something strange about your environment? Can you try with clean `.bashrc`? Thomas Applencourt Leadership Computing Facility Argonne National Laboratory Building 240 Office - 1.D.18 9700 S Cass Av., Lemont Il, 60439 (630) 252-0036 ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Bertoni, Colleen via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Thursday, December 5, 2019 15:12 To: Peterka, Thomas <tpeterka(a)mcs.anl.gov> Cc: Ecp-aurora-sdk-discuss <Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] runaway SYCL kernels It should not take very long. We'll look into it and get back to you. Colleen ________________________________ From: Peterka, Thomas <tpeterka(a)mcs.anl.gov> Sent: Thursday, December 5, 2019 3:09 PM To: Bertoni, Colleen <bertoni(a)anl.gov> Cc: Ecp-aurora-sdk-discuss <Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: runaway SYCL kernels Hi Colleen, Hmm, the reset gpu script hangs and does not return within a few minutes. I tried running the script on another iris node that I did not corrupt, and the script also does not return there. I’m not sure how long it should take. Tom ----------------- Tom Peterka Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave. Lemont, IL 60439 Bldg 240, Rm 2148 e-mail: tpeterka(a)mcs.anl.gov<mailto:[email protected]> web: www.mcs.anl.gov/~tpeterka phone: 630-252-7198 On Dec 5, 2019, at 2:56 PM, Bertoni, Colleen <bertoni(a)anl.gov<mailto:[email protected]>> wrote: Hi Tom, If the GPU hangs and gets in a bad state, after killing the run with Ctrl-C, you should be able to reset the gpu by running "/soft/restricted/CNDA/tools/reset_gpu.sh". Then subsequent runs shouldn't all hang the gpu (unless it's an infinite loop or something). (We run this script as part of the job scheduler epilogue.) Could you try running the reset script for your use cases and let us know if there are any problems? Thanks, Colleen ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of Peterka, Thomas via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Sent: Thursday, December 5, 2019 2:50 PM To: Ecp-aurora-sdk-discuss <Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] runaway SYCL kernels Hi, This is a newbie question. If I run a code with an infinite loop in a SYCL kernel (or in general, a kernel that hangs and doesn’t complete in a reasonable time), kill the program with ctrl-c, fix the code, rebuild, and rerun, the kernel will still hang in the same way. I guess this means the original kernel is still running, and ctrl-c did not kill the kernel. I can log into a different iris node and run there, but each time I hang my program, I need to change nodes. In debugging the code, I’m going to generate multiple instances of runaway kernels. What is the right way to kill a kernel or handle this case? Thanks, Tom ----------------- Tom Peterka Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave. Lemont, IL 60439 Bldg 240, Rm 2148 e-mail: tpeterka(a)mcs.anl.gov<mailto:[email protected]> web: www.mcs.anl.gov/~tpeterka<http://www.mcs.anl.gov/~tpeterka> phone: 630-252-7198

1 0

Re: [Ecp-aurora-sdk-discuss] runaway SYCL kernels
by Applencourt, Thomas 05 Dec '19

05 Dec '19

> I tried running the script on another iris node that I did not corrupt, and the script also does not return there. I’m not sure how long it should take. Strange. I just checked the script is working for me (aka no hanging, and returning instantly on `iris19` with a clean env). Maybe something strange about your environment? Can you try with clean `.bashrc`? Thomas Applencourt Leadership Computing Facility Argonne National Laboratory Building 240 Office - 1.D.18 9700 S Cass Av., Lemont Il, 60439 (630) 252-0036 ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Bertoni, Colleen via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Thursday, December 5, 2019 15:12 To: Peterka, Thomas <tpeterka(a)mcs.anl.gov> Cc: Ecp-aurora-sdk-discuss <Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] runaway SYCL kernels It should not take very long. We'll look into it and get back to you. Colleen ________________________________ From: Peterka, Thomas <tpeterka(a)mcs.anl.gov> Sent: Thursday, December 5, 2019 3:09 PM To: Bertoni, Colleen <bertoni(a)anl.gov> Cc: Ecp-aurora-sdk-discuss <Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: runaway SYCL kernels Hi Colleen, Hmm, the reset gpu script hangs and does not return within a few minutes. I tried running the script on another iris node that I did not corrupt, and the script also does not return there. I’m not sure how long it should take. Tom ----------------- Tom Peterka Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave. Lemont, IL 60439 Bldg 240, Rm 2148 e-mail: tpeterka(a)mcs.anl.gov<mailto:[email protected]> web: www.mcs.anl.gov/~tpeterka phone: 630-252-7198 On Dec 5, 2019, at 2:56 PM, Bertoni, Colleen <bertoni(a)anl.gov<mailto:[email protected]>> wrote: Hi Tom, If the GPU hangs and gets in a bad state, after killing the run with Ctrl-C, you should be able to reset the gpu by running "/soft/restricted/CNDA/tools/reset_gpu.sh". Then subsequent runs shouldn't all hang the gpu (unless it's an infinite loop or something). (We run this script as part of the job scheduler epilogue.) Could you try running the reset script for your use cases and let us know if there are any problems? Thanks, Colleen ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of Peterka, Thomas via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Sent: Thursday, December 5, 2019 2:50 PM To: Ecp-aurora-sdk-discuss <Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] runaway SYCL kernels Hi, This is a newbie question. If I run a code with an infinite loop in a SYCL kernel (or in general, a kernel that hangs and doesn’t complete in a reasonable time), kill the program with ctrl-c, fix the code, rebuild, and rerun, the kernel will still hang in the same way. I guess this means the original kernel is still running, and ctrl-c did not kill the kernel. I can log into a different iris node and run there, but each time I hang my program, I need to change nodes. In debugging the code, I’m going to generate multiple instances of runaway kernels. What is the right way to kill a kernel or handle this case? Thanks, Tom ----------------- Tom Peterka Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave. Lemont, IL 60439 Bldg 240, Rm 2148 e-mail: tpeterka(a)mcs.anl.gov<mailto:[email protected]> web: www.mcs.anl.gov/~tpeterka<http://www.mcs.anl.gov/~tpeterka> phone: 630-252-7198

1 0

Re: [Ecp-aurora-sdk-discuss] runaway SYCL kernels
by Bertoni, Colleen 05 Dec '19

05 Dec '19

It should not take very long. We'll look into it and get back to you. Colleen ________________________________ From: Peterka, Thomas <tpeterka(a)mcs.anl.gov> Sent: Thursday, December 5, 2019 3:09 PM To: Bertoni, Colleen <bertoni(a)anl.gov> Cc: Ecp-aurora-sdk-discuss <Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: runaway SYCL kernels Hi Colleen, Hmm, the reset gpu script hangs and does not return within a few minutes. I tried running the script on another iris node that I did not corrupt, and the script also does not return there. I’m not sure how long it should take. Tom ----------------- Tom Peterka Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave. Lemont, IL 60439 Bldg 240, Rm 2148 e-mail: tpeterka(a)mcs.anl.gov<mailto:[email protected]> web: www.mcs.anl.gov/~tpeterka phone: 630-252-7198 On Dec 5, 2019, at 2:56 PM, Bertoni, Colleen <bertoni(a)anl.gov<mailto:[email protected]>> wrote: Hi Tom, If the GPU hangs and gets in a bad state, after killing the run with Ctrl-C, you should be able to reset the gpu by running "/soft/restricted/CNDA/tools/reset_gpu.sh". Then subsequent runs shouldn't all hang the gpu (unless it's an infinite loop or something). (We run this script as part of the job scheduler epilogue.) Could you try running the reset script for your use cases and let us know if there are any problems? Thanks, Colleen ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of Peterka, Thomas via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Sent: Thursday, December 5, 2019 2:50 PM To: Ecp-aurora-sdk-discuss <Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] runaway SYCL kernels Hi, This is a newbie question. If I run a code with an infinite loop in a SYCL kernel (or in general, a kernel that hangs and doesn’t complete in a reasonable time), kill the program with ctrl-c, fix the code, rebuild, and rerun, the kernel will still hang in the same way. I guess this means the original kernel is still running, and ctrl-c did not kill the kernel. I can log into a different iris node and run there, but each time I hang my program, I need to change nodes. In debugging the code, I’m going to generate multiple instances of runaway kernels. What is the right way to kill a kernel or handle this case? Thanks, Tom ----------------- Tom Peterka Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave. Lemont, IL 60439 Bldg 240, Rm 2148 e-mail: tpeterka(a)mcs.anl.gov<mailto:[email protected]> web: www.mcs.anl.gov/~tpeterka<http://www.mcs.anl.gov/~tpeterka> phone: 630-252-7198

1 0

Re: [Ecp-aurora-sdk-discuss] runaway SYCL kernels
by Peterka, Thomas 05 Dec '19

05 Dec '19

Hi Colleen, Hmm, the reset gpu script hangs and does not return within a few minutes. I tried running the script on another iris node that I did not corrupt, and the script also does not return there. I’m not sure how long it should take. Tom ----------------- Tom Peterka Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave. Lemont, IL 60439 Bldg 240, Rm 2148 e-mail: tpeterka(a)mcs.anl.gov<mailto:[email protected]> web: www.mcs.anl.gov/~tpeterka phone: 630-252-7198 On Dec 5, 2019, at 2:56 PM, Bertoni, Colleen <bertoni(a)anl.gov<mailto:[email protected]>> wrote: Hi Tom, If the GPU hangs and gets in a bad state, after killing the run with Ctrl-C, you should be able to reset the gpu by running "/soft/restricted/CNDA/tools/reset_gpu.sh". Then subsequent runs shouldn't all hang the gpu (unless it's an infinite loop or something). (We run this script as part of the job scheduler epilogue.) Could you try running the reset script for your use cases and let us know if there are any problems? Thanks, Colleen ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of Peterka, Thomas via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Sent: Thursday, December 5, 2019 2:50 PM To: Ecp-aurora-sdk-discuss <Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] runaway SYCL kernels Hi, This is a newbie question. If I run a code with an infinite loop in a SYCL kernel (or in general, a kernel that hangs and doesn’t complete in a reasonable time), kill the program with ctrl-c, fix the code, rebuild, and rerun, the kernel will still hang in the same way. I guess this means the original kernel is still running, and ctrl-c did not kill the kernel. I can log into a different iris node and run there, but each time I hang my program, I need to change nodes. In debugging the code, I’m going to generate multiple instances of runaway kernels. What is the right way to kill a kernel or handle this case? Thanks, Tom ----------------- Tom Peterka Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave. Lemont, IL 60439 Bldg 240, Rm 2148 e-mail: tpeterka(a)mcs.anl.gov<mailto:[email protected]> web: www.mcs.anl.gov/~tpeterka<http://www.mcs.anl.gov/~tpeterka> phone: 630-252-7198

1 0

Re: [Ecp-aurora-sdk-discuss] runaway SYCL kernels
by Bertoni, Colleen 05 Dec '19

05 Dec '19

Hi Tom, If the GPU hangs and gets in a bad state, after killing the run with Ctrl-C, you should be able to reset the gpu by running "/soft/restricted/CNDA/tools/reset_gpu.sh". Then subsequent runs shouldn't all hang the gpu (unless it's an infinite loop or something). (We run this script as part of the job scheduler epilogue.) Could you try running the reset script for your use cases and let us know if there are any problems? Thanks, Colleen ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peterka, Thomas via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Thursday, December 5, 2019 2:50 PM To: Ecp-aurora-sdk-discuss <Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] runaway SYCL kernels Hi, This is a newbie question. If I run a code with an infinite loop in a SYCL kernel (or in general, a kernel that hangs and doesn’t complete in a reasonable time), kill the program with ctrl-c, fix the code, rebuild, and rerun, the kernel will still hang in the same way. I guess this means the original kernel is still running, and ctrl-c did not kill the kernel. I can log into a different iris node and run there, but each time I hang my program, I need to change nodes. In debugging the code, I’m going to generate multiple instances of runaway kernels. What is the right way to kill a kernel or handle this case? Thanks, Tom ----------------- Tom Peterka Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave. Lemont, IL 60439 Bldg 240, Rm 2148 e-mail: tpeterka(a)mcs.anl.gov<mailto:[email protected]> web: www.mcs.anl.gov/~tpeterka phone: 630-252-7198

1 0