Colleen, Thomas, I can confirm that the latest version of the script works for me. I can now reset the GPU anytime I mess it up. This is exactly what I needed. Thanks! Tom ----------------- Tom Peterka Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave. Lemont, IL 60439 Bldg 240, Rm 2148 e-mail: [email protected]<mailto:[email protected]> web: www.mcs.anl.gov/~tpeterka phone: 630-252-7198 On Dec 5, 2019, at 3:26 PM, Bertoni, Colleen <[email protected]<mailto:[email protected]>> wrote: Thanks, Thomas! We had the order of commands in the script flipped in the CNDA directory. Ben kindly fixed it, and it should work now. : ) Tom, could you verify it works for you now? Colleen ________________________________ From: Ecp-aurora-sdk-discuss <[email protected]<mailto:[email protected]>> on behalf of Applencourt, Thomas via Ecp-aurora-sdk-discuss <[email protected]<mailto:[email protected]>> Sent: Thursday, December 5, 2019 3:22 PM To: Bertoni, Colleen via Ecp-aurora-sdk-discuss <[email protected]<mailto:[email protected]>>; Peterka, Thomas <[email protected]<mailto:[email protected]>> Subject: Re: [Ecp-aurora-sdk-discuss] runaway SYCL kernels

I tried running the script on another iris node that I did not corrupt, and the script also does not return there. I’m not sure how long it should take.

Strange. I just checked the script is working for me (aka no hanging, and returning instantly on `iris19` with a clean env). Maybe something strange about your environment? Can you try with clean `.bashrc`? Thomas Applencourt Leadership Computing Facility Argonne National Laboratory Building 240 Office - 1.D.18 9700 S Cass Av., Lemont Il, 60439 (630) 252-0036 ________________________________ From: Ecp-aurora-sdk-discuss <[email protected]<mailto:[email protected]>> on behalf of Bertoni, Colleen via Ecp-aurora-sdk-discuss <[email protected]<mailto:[email protected]>> Sent: Thursday, December 5, 2019 15:12 To: Peterka, Thomas <[email protected]<mailto:[email protected]>> Cc: Ecp-aurora-sdk-discuss <[email protected]<mailto:[email protected]>> Subject: Re: [Ecp-aurora-sdk-discuss] runaway SYCL kernels It should not take very long. We'll look into it and get back to you. Colleen ________________________________ From: Peterka, Thomas <[email protected]<mailto:[email protected]>> Sent: Thursday, December 5, 2019 3:09 PM To: Bertoni, Colleen <[email protected]<mailto:[email protected]>> Cc: Ecp-aurora-sdk-discuss <[email protected]<mailto:[email protected]>> Subject: Re: runaway SYCL kernels Hi Colleen, Hmm, the reset gpu script hangs and does not return within a few minutes. I tried running the script on another iris node that I did not corrupt, and the script also does not return there. I’m not sure how long it should take. Tom ----------------- Tom Peterka Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave. Lemont, IL 60439 Bldg 240, Rm 2148 e-mail: [email protected]<mailto:[email protected]> web: www.mcs.anl.gov/~tpeterka<http://www.mcs.anl.gov/~tpeterka> phone: 630-252-7198 On Dec 5, 2019, at 2:56 PM, Bertoni, Colleen <[email protected]<mailto:[email protected]>> wrote: Hi Tom, If the GPU hangs and gets in a bad state, after killing the run with Ctrl-C, you should be able to reset the gpu by running "/soft/restricted/CNDA/tools/reset_gpu.sh". Then subsequent runs shouldn't all hang the gpu (unless it's an infinite loop or something). (We run this script as part of the job scheduler epilogue.) Could you try running the reset script for your use cases and let us know if there are any problems? Thanks, Colleen ________________________________ From: Ecp-aurora-sdk-discuss <[email protected]<mailto:[email protected]>> on behalf of Peterka, Thomas via Ecp-aurora-sdk-discuss <[email protected]<mailto:[email protected]>> Sent: Thursday, December 5, 2019 2:50 PM To: Ecp-aurora-sdk-discuss <[email protected]<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] runaway SYCL kernels Hi, This is a newbie question. If I run a code with an infinite loop in a SYCL kernel (or in general, a kernel that hangs and doesn’t complete in a reasonable time), kill the program with ctrl-c, fix the code, rebuild, and rerun, the kernel will still hang in the same way. I guess this means the original kernel is still running, and ctrl-c did not kill the kernel. I can log into a different iris node and run there, but each time I hang my program, I need to change nodes. In debugging the code, I’m going to generate multiple instances of runaway kernels. What is the right way to kill a kernel or handle this case? Thanks, Tom ----------------- Tom Peterka Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave. Lemont, IL 60439 Bldg 240, Rm 2148 e-mail: [email protected]<mailto:[email protected]> web: www.mcs.anl.gov/~tpeterka<http://www.mcs.anl.gov/~tpeterka> phone: 630-252-7198