If the GPU hangs and gets in a bad state, after killing the run with Ctrl-C, you should be able to reset the gpu by running "/soft/restricted/CNDA/tools/reset_gpu.sh". Then subsequent runs shouldn't all hang the gpu (unless it's an infinite loop or something).
(We run this script as part of the job scheduler epilogue.)
Could you try running the reset script for your use cases and let us know if there are any problems?
Hi,
This is a newbie question. If I run a code with an infinite loop in a SYCL kernel (or in general, a kernel that hangs and doesn’t complete in a reasonable time), kill the program with ctrl-c, fix the code, rebuild, and rerun, the kernel will still
hang in the same way.
I guess this means the original kernel is still running, and ctrl-c did not kill the kernel.
I can log into a different iris node and run there, but each time I hang my program, I need to change nodes. In debugging the code, I’m going to generate multiple instances of runaway kernels.
What is the right way to kill a kernel or handle this case?
Thanks,
Tom