according to onetrace, the kernel runtimes look about the same (my original timings were done by putting std::chrono calls around the kernel launches). The total API time for the L0 backend goes from 501942950 to 11021208546 when it’s done with taskset. here are the explicit numbers:
no taskset:
Function, Calls, Time (ns), Time (%), Average (ns), Min (ns), Max (ns)
zeModuleCreate, 2, 169715504, 33.81, 84857752, 8910086, 160805418
zeCommandQueueSynchronize, 9154, 142616861, 28.41, 15579, 53, 1037213
zeCommandListReset, 3976, 86385717, 17.21, 21726, 183, 245071
zeCommandListAppendMemoryCopy, 1700, 44595645, 8.88, 26232, 18077, 172220
zeCommandQueueExecuteCommandLists, 3976, 25818462, 5.14, 6493, 1307, 2848223
zeMemAllocDevice, 67, 15173695, 3.02, 226473, 90346, 2874175
zeCommandListAppendMemoryFill, 503, 5027360, 1.00, 9994, 1520, 1823447
zeKernelCreate, 22, 1828342, 0.36, 83106, 2202, 828879
zeCommandListCreateImmediate, 1, 1803461, 0.36, 1803461, 1803461, 1803461
zeEventQueryStatus, 1381, 1714500, 0.34, 1241, 364, 98024
zeCommandQueueCreate, 2, 1408878, 0.28, 704439, 259941, 1148937
zeMemFree, 6, 1364497, 0.27, 227416, 184775, 293323
zeCommandListCreate, 3, 854943, 0.17, 284981, 168795, 417739
zeCommandListAppendBarrier, 1381, 616387, 0.12, 446, 210, 6665
zeCommandListAppendLaunchKernel, 393, 600600, 0.12, 1528, 667, 31174
zeFenceQueryStatus, 3977, 401428, 0.08, 100, 41, 3480
zeEventHostReset, 3974, 389108, 0.08, 97, 55, 3097
zeMemGetAllocProperties, 1892, 370859, 0.07, 196, 66, 3120
zeCommandListClose, 3976, 347435, 0.07, 87, 39, 1089
zeMemAllocHost, 1, 293942, 0.06, 293942, 293942, 293942
zeFenceReset, 3976, 147342, 0.03, 37, 16, 3051
zeKernelSetArgumentValue, 602, 139952, 0.03, 232, 48, 3794
zeKernelSuggestGroupSize, 393, 108755, 0.02, 276, 97, 3128
zeEventPoolCreate, 1, 104642, 0.02, 104642, 104642, 104642
zeKernelSetGroupSize, 393, 55013, 0.01, 139, 60, 523
zeDeviceGetCommandQueueGroupProperties, 60, 16436, 0.00, 273, 42, 1570
zeDeviceGetProperties, 18, 8115, 0.00, 450, 240, 1604
zeEventCreate, 3, 7557, 0.00, 2519, 735, 5934
zeContextCreate, 1, 5449, 0.00, 5449, 5449, 5449
zeKernelSetIndirectAccess, 22, 4009, 0.00, 182, 27, 1807
zeKernelGetProperties, 22, 3981, 0.00, 180, 48, 479
zeDeviceGetModuleProperties, 6, 2776, 0.00, 462, 121, 1977
zeDriverGetExtensionProperties, 2, 2318, 0.00, 1159, 584, 1734
zeFenceCreate, 3, 1882, 0.00, 627, 220, 996
zeDeviceGetSubDevices, 12, 1722, 0.00, 143, 46, 297
zeDriverGetProperties, 1, 1192, 0.00, 1192, 1192, 1192
zeModuleGetProperties, 2, 945, 0.00, 472, 461, 484
zeDeviceGetComputeProperties, 1, 775, 0.00, 775, 775, 775
zeInit, 1, 759, 0.00, 759, 759, 759
zeDeviceGet, 2, 731, 0.00, 365, 263, 468
zeDriverGet, 2, 371, 0.00, 185, 119, 252
zeDeviceGetImageProperties, 6, 317, 0.00, 52, 30, 150
zeDriverGetApiVersion, 1, 287, 0.00, 287, 287, 287
with taskset:
Function, Calls, Time (ns), Time (%), Average (ns), Min (ns), Max (ns)
zeCommandQueueSynchronize, 9154, 7637427277, 69.30, 834326, 43, 171775424
zeEventQueryStatus, 1381, 2438612599, 22.13, 1765830, 294, 190701852
zeCommandListReset, 3976, 261265117, 2.37, 65710, 182, 2516131
zeCommandListCreateImmediate, 1, 189954019, 1.72, 189954019, 189954019, 189954019
zeModuleCreate, 2, 162170189, 1.47, 81085094, 8081706, 154088483
zeMemFree, 6, 130338449, 1.18, 21723074, 213200, 129109118
zeCommandQueueExecuteCommandLists, 3976, 92877003, 0.84, 23359, 1211, 3024913
zeCommandListAppendMemoryCopy, 1700, 47362492, 0.43, 27860, 15042, 946050
zeMemAllocDevice, 67, 43488320, 0.39, 649079, 97185, 15510137
zeKernelCreate, 22, 7986520, 0.07, 363023, 2088, 4065025
zeCommandListAppendMemoryFill, 503, 4381709, 0.04, 8711, 1004, 1694162
zeCommandQueueCreate, 2, 1350202, 0.01, 675101, 225169, 1125033
zeCommandListCreate, 3, 815519, 0.01, 271839, 142204, 360245
zeCommandListAppendLaunchKernel, 393, 729006, 0.01, 1854, 573, 19302
zeCommandListAppendBarrier, 1381, 419554, 0.00, 303, 193, 3665
zeMemGetAllocProperties, 1892, 378477, 0.00, 200, 62, 2390
zeEventHostReset, 3974, 340554, 0.00, 85, 44, 6242
zeCommandListClose, 3976, 281788, 0.00, 70, 30, 759
zeFenceQueryStatus, 3977, 280667, 0.00, 70, 40, 1319
zeMemAllocHost, 1, 236196, 0.00, 236196, 236196, 236196
zeKernelSetArgumentValue, 602, 137640, 0.00, 228, 40, 2542
zeKernelSuggestGroupSize, 393, 125753, 0.00, 319, 91, 2101
zeEventPoolCreate, 1, 78056, 0.00, 78056, 78056, 78056
zeFenceReset, 3976, 73421, 0.00, 18, 16, 110
zeKernelSetGroupSize, 393, 54161, 0.00, 137, 38, 357
zeDeviceGetCommandQueueGroupProperties, 60, 13418, 0.00, 223, 20, 1281
zeDeviceGetProperties, 18, 6354, 0.00, 353, 189, 1069
zeEventCreate, 3, 4290, 0.00, 1430, 385, 3395
zeKernelGetProperties, 22, 3189, 0.00, 144, 43, 286
zeContextCreate, 1, 3134, 0.00, 3134, 3134, 3134
zeKernelSetIndirectAccess, 22, 3084, 0.00, 140, 21, 1170
zeDeviceGetModuleProperties, 6, 1817, 0.00, 302, 84, 1118
zeDeviceGetSubDevices, 12, 1816, 0.00, 151, 28, 329
zeDriverGetExtensionProperties, 2, 1593, 0.00, 796, 316, 1277
zeFenceCreate, 3, 1033, 0.00, 344, 115, 524
zeDriverGetProperties, 1, 998, 0.00, 998, 998, 998
zeModuleGetProperties, 2, 876, 0.00, 438, 422, 454
zeDeviceGet, 2, 657, 0.00, 328, 252, 405
zeInit, 1, 562, 0.00, 562, 562, 562
zeDeviceGetComputeProperties, 1, 322, 0.00, 322, 322, 322
zeDriverGet, 2, 290, 0.00, 145, 107, 183
zeDriverGetApiVersion, 1, 223, 0.00, 223, 223, 223
zeDeviceGetImageProperties, 6, 202, 0.00, 33, 17, 102
Can you run onetrace profiler to see if the kernel execution times goes up compared to your run without taskset?
module load tools/pti-gpu
onetrace -d ./a.out
if that does not show a difference look at host API overhead
onterace -h ./a.out
From: Charles Leggett <cgleggett@lbl.gov>
Date: Friday, March 24, 2023 at 19:59
To: Steinbrecher, Patrick <patrick.steinbrecher@intel.com>
Cc: Aurora Discussion List <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
Subject: Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot
It's only running a single binary on a single core. The script is setup to be able to launch multiple concurrent jobs on different cores, but I'm only using it on a single one right now.
The cores I tried are 0, 1, 8, 10, 18, 51 and 200. All performed the same.
On Fri, Mar 24, 2023 at 5:44 PM Steinbrecher, Patrick <patrick.steinbrecher@intel.com> wrote:
How many binaries are you running with task set? If you run multiple binaries, how do you map them to different GPUs?
What are the exact core numbers you are using (from sched_getcpu()) ?
From: Charles Leggett <cgleggett@lbl.gov>
Date: Friday, March 24, 2023 at 19:19
To: Steinbrecher, Patrick <patrick.steinbrecher@intel.com>
Cc: Aurora Discussion List <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
Subject: Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot
It shows the core that taskset has selected.
I'm assuming you don't mean doing this from inside the kokkos::parallel_for.
On Fri, Mar 24, 2023 at 3:38 PM Steinbrecher, Patrick <patrick.steinbrecher@intel.com> wrote:
In your program that is run under taskset add this:
#include <sched.h>
std::cout << “I am running on core: ” << sched_getcpu() << std::endl;
What is the output you get?
From: Charles Leggett <cgleggett@lbl.gov>
Date: Friday, March 24, 2023 at 17:21
To: Steinbrecher, Patrick <patrick.steinbrecher@intel.com>
Cc: Aurora Discussion List <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
Subject: Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot
There is no explicit use of MPI in user code. I have no idea what Kokkos is doing under the hood, but I thought it was using a SYCL/Level zero backend.
On Fri, Mar 24, 2023 at 3:09 PM Steinbrecher, Patrick <patrick.steinbrecher@intel.com> wrote:
Are you using MPI in this case?
From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces@lists.jlse.anl.gov> on behalf of Charles Leggett via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
Date: Friday, March 24, 2023 at 12:50
To: Aurora Discussion List <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
Subject: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot
I have a script that launches jobs which employs `taskset -c` to lock an executable onto a specific core (this was done to test concurrent job execution). What's very odd is that using taskset slows down kernel execution on the GPU by about a factor of 600. eg:
without taskset:
kernel total /s avg launch /us std dev /ussim_clean 0.02091981 486.5 18.7sim_A 0.10563853 2456.7 703.2sim_ct 0.03402171 791.2 9.6sim_cp 0.01612411 375.0 7.0
with taskset:
kernel total /s avg launch /us std dev /ussim_clean 0.02155314 501.2 20.5sim_A 60.56034054 1408380.0 398894.4sim_ct 1.33204648 30977.8 50526.0sim_cp 0.99585704 23159.5 13514.5
I dump the environment vars within the job, and they look identical.
I tried a few different cores to set the cpu affinity, and it doesn't seem to make a difference (I didn't try all 208)
I'm using oneapi/eng-compiler/2022.12.30.003 and kokkos/2022.12.30.003/eng-compiler/sycl_intel_aot
I don't think I noticed this behavior with pure SYCL code, but now I'll have to go back and check.
This was on node x1921c2s5b0n0
cheers, Charles.
--Charles Leggett <CGLeggett@lbl.gov>
Lawrence Berkeley National Lab
1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720tel: 510.495.2930
--Charles Leggett <CGLeggett@lbl.gov>
Lawrence Berkeley National Lab
1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720tel: 510.495.2930
--Charles Leggett <CGLeggett@lbl.gov>
Lawrence Berkeley National Lab
1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720tel: 510.495.2930
--Charles Leggett <CGLeggett@lbl.gov>
Lawrence Berkeley National Lab
1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720tel: 510.495.2930