Colleen, Thomas,

I can confirm that the latest version of the script works for me. I can now reset the GPU anytime I mess it up. This is exactly what I needed.

Thanks!
Tom
-----------------
Tom Peterka
Mathematics and Computer Science Division
Argonne National Laboratory
9700 S. Cass Ave.
Lemont, IL 60439
Bldg 240, Rm 2148
e-mail: tpeterka@mcs.anl.gov
web: www.mcs.anl.gov/~tpeterka
phone: 630-252-7198

On Dec 5, 2019, at 3:26 PM, Bertoni, Colleen <bertoni@anl.gov> wrote:

Thanks, Thomas!

We had the order of commands in the script flipped in the CNDA directory. Ben kindly fixed it, and it should work now. : )

Tom, could you verify it works for you now?

Colleen

From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces@lists.jlse.anl.gov> on behalf of Applencourt, Thomas via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
Sent: Thursday, December 5, 2019 3:22 PM
To: Bertoni, Colleen via Ecp-aurora-sdk-discuss <Ecp-aurora-sdk-discuss@lists.jlse.anl.gov>; Peterka, Thomas <tpeterka@mcs.anl.gov>
Subject: Re: [Ecp-aurora-sdk-discuss] runaway SYCL kernels
 
> I tried running the script on another iris node that I did not corrupt, and the script also does not return there. I’m not sure how long it should take.

Strange. I just checked the script is working for me  (aka no hanging, and returning instantly on `iris19` with a clean env). Maybe something strange about your environment? Can you try with clean `.bashrc`?


Thomas Applencourt
Leadership Computing Facility
Argonne National Laboratory
Building 240 Office - 1.D.18
9700 S Cass Av., Lemont Il, 60439
(630) 252-0036
From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces@lists.jlse.anl.gov> on behalf of Bertoni, Colleen via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
Sent: Thursday, December 5, 2019 15:12
To: Peterka, Thomas <tpeterka@mcs.anl.gov>
Cc: Ecp-aurora-sdk-discuss <Ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
Subject: Re: [Ecp-aurora-sdk-discuss] runaway SYCL kernels
 
It should not take very long. We'll look into it and get back to you.

Colleen
From: Peterka, Thomas <tpeterka@mcs.anl.gov>
Sent: Thursday, December 5, 2019 3:09 PM
To: Bertoni, Colleen <bertoni@anl.gov>
Cc: Ecp-aurora-sdk-discuss <Ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
Subject: Re: runaway SYCL kernels
 
Hi Colleen,

Hmm, the reset gpu script hangs and does not return within a few minutes. I tried running the script on another iris node that I did not corrupt, and the script also does not return there. I’m not sure how long it should take.

Tom
-----------------
Tom Peterka
Mathematics and Computer Science Division
Argonne National Laboratory
9700 S. Cass Ave.
Lemont, IL 60439
Bldg 240, Rm 2148
e-mail: tpeterka@mcs.anl.gov
web: www.mcs.anl.gov/~tpeterka
phone: 630-252-7198

On Dec 5, 2019, at 2:56 PM, Bertoni, Colleen <bertoni@anl.gov> wrote:

Hi Tom,

If the GPU hangs and gets in a bad state, after killing the run with Ctrl-C, you should be able to reset the gpu by running "/soft/restricted/CNDA/tools/reset_gpu.sh". Then subsequent runs shouldn't all hang the gpu (unless it's an infinite loop or something). (We run this script as part of the job scheduler epilogue.)

Could you try running the reset script for your use cases and let us know if there are any problems?

Thanks,
Colleen
From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces@lists.jlse.anl.gov> on behalf of Peterka, Thomas via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
Sent: Thursday, December 5, 2019 2:50 PM
To: Ecp-aurora-sdk-discuss <Ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
Subject: [Ecp-aurora-sdk-discuss] runaway SYCL kernels
 
Hi,

This is a newbie question. If I run a code with an infinite loop in a SYCL kernel (or in general, a kernel that hangs and doesn’t complete in a reasonable time), kill the program with ctrl-c, fix the code, rebuild, and rerun, the kernel will still hang in the same way.

I guess this means the original kernel is still running, and ctrl-c did not kill the kernel.

I can log into a different iris node and run there, but each time I hang my program, I need to change nodes. In debugging the code, I’m going to generate multiple instances of runaway kernels.

What is the right way to kill a kernel or handle this case?

Thanks,
Tom
-----------------
Tom Peterka
Mathematics and Computer Science Division
Argonne National Laboratory
9700 S. Cass Ave.
Lemont, IL 60439
Bldg 240, Rm 2148
e-mail: tpeterka@mcs.anl.gov
web: www.mcs.anl.gov/~tpeterka
phone: 630-252-7198