according to onetrace, the kernel runtimes look about the same (my original timings were done by putting std::chrono calls around the kernel launches). The total API time for the L0 backend goes from 501942950 to 11021208546 when it’s done with taskset. here are the explicit numbers: no taskset: Function, Calls, Time (ns), Time (%), Average (ns), Min (ns), Max (ns) zeModuleCreate, 2, 169715504, 33.81, 84857752, 8910086, 160805418 zeCommandQueueSynchronize, 9154, 142616861, 28.41, 15579, 53, 1037213 zeCommandListReset, 3976, 86385717, 17.21, 21726, 183, 245071 zeCommandListAppendMemoryCopy, 1700, 44595645, 8.88, 26232, 18077, 172220 zeCommandQueueExecuteCommandLists, 3976, 25818462, 5.14, 6493, 1307, 2848223 zeMemAllocDevice, 67, 15173695, 3.02, 226473, 90346, 2874175 zeCommandListAppendMemoryFill, 503, 5027360, 1.00, 9994, 1520, 1823447 zeKernelCreate, 22, 1828342, 0.36, 83106, 2202, 828879 zeCommandListCreateImmediate, 1, 1803461, 0.36, 1803461, 1803461, 1803461 zeEventQueryStatus, 1381, 1714500, 0.34, 1241, 364, 98024 zeCommandQueueCreate, 2, 1408878, 0.28, 704439, 259941, 1148937 zeMemFree, 6, 1364497, 0.27, 227416, 184775, 293323 zeCommandListCreate, 3, 854943, 0.17, 284981, 168795, 417739 zeCommandListAppendBarrier, 1381, 616387, 0.12, 446, 210, 6665 zeCommandListAppendLaunchKernel, 393, 600600, 0.12, 1528, 667, 31174 zeFenceQueryStatus, 3977, 401428, 0.08, 100, 41, 3480 zeEventHostReset, 3974, 389108, 0.08, 97, 55, 3097 zeMemGetAllocProperties, 1892, 370859, 0.07, 196, 66, 3120 zeCommandListClose, 3976, 347435, 0.07, 87, 39, 1089 zeMemAllocHost, 1, 293942, 0.06, 293942, 293942, 293942 zeFenceReset, 3976, 147342, 0.03, 37, 16, 3051 zeKernelSetArgumentValue, 602, 139952, 0.03, 232, 48, 3794 zeKernelSuggestGroupSize, 393, 108755, 0.02, 276, 97, 3128 zeEventPoolCreate, 1, 104642, 0.02, 104642, 104642, 104642 zeKernelSetGroupSize, 393, 55013, 0.01, 139, 60, 523 zeDeviceGetCommandQueueGroupProperties, 60, 16436, 0.00, 273, 42, 1570 zeDeviceGetProperties, 18, 8115, 0.00, 450, 240, 1604 zeEventCreate, 3, 7557, 0.00, 2519, 735, 5934 zeContextCreate, 1, 5449, 0.00, 5449, 5449, 5449 zeKernelSetIndirectAccess, 22, 4009, 0.00, 182, 27, 1807 zeKernelGetProperties, 22, 3981, 0.00, 180, 48, 479 zeDeviceGetModuleProperties, 6, 2776, 0.00, 462, 121, 1977 zeDriverGetExtensionProperties, 2, 2318, 0.00, 1159, 584, 1734 zeFenceCreate, 3, 1882, 0.00, 627, 220, 996 zeDeviceGetSubDevices, 12, 1722, 0.00, 143, 46, 297 zeDriverGetProperties, 1, 1192, 0.00, 1192, 1192, 1192 zeModuleGetProperties, 2, 945, 0.00, 472, 461, 484 zeDeviceGetComputeProperties, 1, 775, 0.00, 775, 775, 775 zeInit, 1, 759, 0.00, 759, 759, 759 zeDeviceGet, 2, 731, 0.00, 365, 263, 468 zeDriverGet, 2, 371, 0.00, 185, 119, 252 zeDeviceGetImageProperties, 6, 317, 0.00, 52, 30, 150 zeDriverGetApiVersion, 1, 287, 0.00, 287, 287, 287 with taskset: Function, Calls, Time (ns), Time (%), Average (ns), Min (ns), Max (ns) zeCommandQueueSynchronize, 9154, 7637427277, 69.30, 834326, 43, 171775424 zeEventQueryStatus, 1381, 2438612599, 22.13, 1765830, 294, 190701852 zeCommandListReset, 3976, 261265117, 2.37, 65710, 182, 2516131 zeCommandListCreateImmediate, 1, 189954019, 1.72, 189954019, 189954019, 189954019 zeModuleCreate, 2, 162170189, 1.47, 81085094, 8081706, 154088483 zeMemFree, 6, 130338449, 1.18, 21723074, 213200, 129109118 zeCommandQueueExecuteCommandLists, 3976, 92877003, 0.84, 23359, 1211, 3024913 zeCommandListAppendMemoryCopy, 1700, 47362492, 0.43, 27860, 15042, 946050 zeMemAllocDevice, 67, 43488320, 0.39, 649079, 97185, 15510137 zeKernelCreate, 22, 7986520, 0.07, 363023, 2088, 4065025 zeCommandListAppendMemoryFill, 503, 4381709, 0.04, 8711, 1004, 1694162 zeCommandQueueCreate, 2, 1350202, 0.01, 675101, 225169, 1125033 zeCommandListCreate, 3, 815519, 0.01, 271839, 142204, 360245 zeCommandListAppendLaunchKernel, 393, 729006, 0.01, 1854, 573, 19302 zeCommandListAppendBarrier, 1381, 419554, 0.00, 303, 193, 3665 zeMemGetAllocProperties, 1892, 378477, 0.00, 200, 62, 2390 zeEventHostReset, 3974, 340554, 0.00, 85, 44, 6242 zeCommandListClose, 3976, 281788, 0.00, 70, 30, 759 zeFenceQueryStatus, 3977, 280667, 0.00, 70, 40, 1319 zeMemAllocHost, 1, 236196, 0.00, 236196, 236196, 236196 zeKernelSetArgumentValue, 602, 137640, 0.00, 228, 40, 2542 zeKernelSuggestGroupSize, 393, 125753, 0.00, 319, 91, 2101 zeEventPoolCreate, 1, 78056, 0.00, 78056, 78056, 78056 zeFenceReset, 3976, 73421, 0.00, 18, 16, 110 zeKernelSetGroupSize, 393, 54161, 0.00, 137, 38, 357 zeDeviceGetCommandQueueGroupProperties, 60, 13418, 0.00, 223, 20, 1281 zeDeviceGetProperties, 18, 6354, 0.00, 353, 189, 1069 zeEventCreate, 3, 4290, 0.00, 1430, 385, 3395 zeKernelGetProperties, 22, 3189, 0.00, 144, 43, 286 zeContextCreate, 1, 3134, 0.00, 3134, 3134, 3134 zeKernelSetIndirectAccess, 22, 3084, 0.00, 140, 21, 1170 zeDeviceGetModuleProperties, 6, 1817, 0.00, 302, 84, 1118 zeDeviceGetSubDevices, 12, 1816, 0.00, 151, 28, 329 zeDriverGetExtensionProperties, 2, 1593, 0.00, 796, 316, 1277 zeFenceCreate, 3, 1033, 0.00, 344, 115, 524 zeDriverGetProperties, 1, 998, 0.00, 998, 998, 998 zeModuleGetProperties, 2, 876, 0.00, 438, 422, 454 zeDeviceGet, 2, 657, 0.00, 328, 252, 405 zeInit, 1, 562, 0.00, 562, 562, 562 zeDeviceGetComputeProperties, 1, 322, 0.00, 322, 322, 322 zeDriverGet, 2, 290, 0.00, 145, 107, 183 zeDriverGetApiVersion, 1, 223, 0.00, 223, 223, 223 zeDeviceGetImageProperties, 6, 202, 0.00, 33, 17, 102 On Mon, Mar 27, 2023 at 6:44 AM Steinbrecher, Patrick < [email protected]> wrote:

Can you run onetrace profiler to see if the kernel execution times goes up compared to your run without taskset?

module load tools/pti-gpu

onetrace -d ./a.out

if that does not show a difference look at host API overhead

onterace -h ./a.out

*From: *Charles Leggett <[email protected]> *Date: *Friday, March 24, 2023 at 19:59 *To: *Steinbrecher, Patrick <[email protected]> *Cc: *Aurora Discussion List <[email protected]> *Subject: *Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot

It's only running a single binary on a single core. The script is setup to be able to launch multiple concurrent jobs on different cores, but I'm only using it on a single one right now.

The cores I tried are 0, 1, 8, 10, 18, 51 and 200. All performed the same.

On Fri, Mar 24, 2023 at 5:44 PM Steinbrecher, Patrick < [email protected]> wrote:

How many binaries are you running with task set? If you run multiple binaries, how do you map them to different GPUs?

What are the exact core numbers you are using (from sched_getcpu()) ?

*From: *Charles Leggett <[email protected]> *Date: *Friday, March 24, 2023 at 19:19 *To: *Steinbrecher, Patrick <[email protected]> *Cc: *Aurora Discussion List <[email protected]> *Subject: *Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot

It shows the core that taskset has selected.

I'm assuming you don't mean doing this from inside the kokkos::parallel_for.

On Fri, Mar 24, 2023 at 3:38 PM Steinbrecher, Patrick < [email protected]> wrote:

In your program that is run under taskset add this:

#include <sched.h>

std::cout << “I am running on core: ” << sched_getcpu() << std::endl;

What is the output you get?

*From: *Charles Leggett <[email protected]> *Date: *Friday, March 24, 2023 at 17:21 *To: *Steinbrecher, Patrick <[email protected]> *Cc: *Aurora Discussion List <[email protected]> *Subject: *Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot

There is no explicit use of MPI in user code. I have no idea what Kokkos is doing under the hood, but I thought it was using a SYCL/Level zero backend.

On Fri, Mar 24, 2023 at 3:09 PM Steinbrecher, Patrick < [email protected]> wrote:

Are you using MPI in this case?

*From: *Ecp-aurora-sdk-discuss < [email protected]> on behalf of Charles Leggett via Ecp-aurora-sdk-discuss < [email protected]> *Date: *Friday, March 24, 2023 at 12:50 *To: *Aurora Discussion List <[email protected]> *Subject: *[Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot

I have a script that launches jobs which employs `taskset -c` to lock an executable onto a specific core (this was done to test concurrent job execution). What's very odd is that using taskset slows down kernel execution on the GPU by about a factor of 600. eg:

without taskset:

kernel total /s avg launch /us std dev /us

sim_clean 0.02091981 486.5 18.7

sim_A 0.10563853 2456.7 703.2

sim_ct 0.03402171 791.2 9.6

sim_cp 0.01612411 375.0 7.0

with taskset:

kernel total /s avg launch /us std dev /us

sim_clean 0.02155314 501.2 20.5

sim_A 60.56034054 1408380.0 398894.4

sim_ct 1.33204648 30977.8 50526.0

sim_cp 0.99585704 23159.5 13514.5

I dump the environment vars within the job, and they look identical.

I tried a few different cores to set the cpu affinity, and it doesn't seem to make a difference (I didn't try all 208)

I'm using oneapi/eng-compiler/2022.12.30.003 and kokkos/2022.12.30.003/eng-compiler/sycl_intel_aot

I don't think I noticed this behavior with pure SYCL code, but now I'll have to go back and check.

This was on node x1921c2s5b0n0

cheers, Charles.

--

Charles Leggett <[email protected]> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720

tel: 510.495.2930

--

Charles Leggett <[email protected]> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720

tel: 510.495.2930

--

Charles Leggett <[email protected]> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720

tel: 510.495.2930

--

Charles Leggett <[email protected]> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720

tel: 510.495.2930

-- Charles Leggett <[email protected]> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930