How many binaries are you running with task set? If you run multiple binaries, how do you map them to different GPUs? What are the exact core numbers you are using (from sched_getcpu()) ? From: Charles Leggett <[email protected]> Date: Friday, March 24, 2023 at 19:19 To: Steinbrecher, Patrick <[email protected]> Cc: Aurora Discussion List <[email protected]> Subject: Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot It shows the core that taskset has selected. I'm assuming you don't mean doing this from inside the kokkos::parallel_for. On Fri, Mar 24, 2023 at 3:38 PM Steinbrecher, Patrick <[email protected]<mailto:[email protected]>> wrote: In your program that is run under taskset add this: #include <sched.h> std::cout << “I am running on core: ” << sched_getcpu() << std::endl; What is the output you get? From: Charles Leggett <[email protected]<mailto:[email protected]>> Date: Friday, March 24, 2023 at 17:21 To: Steinbrecher, Patrick <[email protected]<mailto:[email protected]>> Cc: Aurora Discussion List <[email protected]<mailto:[email protected]>> Subject: Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot There is no explicit use of MPI in user code. I have no idea what Kokkos is doing under the hood, but I thought it was using a SYCL/Level zero backend. On Fri, Mar 24, 2023 at 3:09 PM Steinbrecher, Patrick <[email protected]<mailto:[email protected]>> wrote: Are you using MPI in this case? From: Ecp-aurora-sdk-discuss <[email protected]<mailto:[email protected]>> on behalf of Charles Leggett via Ecp-aurora-sdk-discuss <[email protected]<mailto:[email protected]>> Date: Friday, March 24, 2023 at 12:50 To: Aurora Discussion List <[email protected]<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot I have a script that launches jobs which employs `taskset -c` to lock an executable onto a specific core (this was done to test concurrent job execution). What's very odd is that using taskset slows down kernel execution on the GPU by about a factor of 600. eg: without taskset: kernel total /s avg launch /us std dev /us sim_clean 0.02091981 486.5 18.7 sim_A 0.10563853 2456.7 703.2 sim_ct 0.03402171 791.2 9.6 sim_cp 0.01612411 375.0 7.0 with taskset: kernel total /s avg launch /us std dev /us sim_clean 0.02155314 501.2 20.5 sim_A 60.56034054 1408380.0 398894.4 sim_ct 1.33204648 30977.8 50526.0 sim_cp 0.99585704 23159.5 13514.5 I dump the environment vars within the job, and they look identical. I tried a few different cores to set the cpu affinity, and it doesn't seem to make a difference (I didn't try all 208) I'm using oneapi/eng-compiler/2022.12.30.003 and kokkos/2022.12.30.003/eng-compiler/sycl_intel_aot I don't think I noticed this behavior with pure SYCL code, but now I'll have to go back and check. This was on node x1921c2s5b0n0 cheers, Charles. -- Charles Leggett <[email protected]<mailto:[email protected]>> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930 -- Charles Leggett <[email protected]<mailto:[email protected]>> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930 -- Charles Leggett <[email protected]<mailto:[email protected]>> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930