Interesting, looks like zeCommandQueueSynchronize takes way longer.
Is this SYCL or OpenMP? Just wondering if one of the helper threads keeps competing with your app.
Can you try giving two cores (ideally not hyper threads, e.g. 2 and 4) to the binary via taskset?
From: Charles Leggett <cgleggett@lbl.gov>
Date: Monday, March 27, 2023 at 10:07
To: Steinbrecher, Patrick <patrick.steinbrecher@intel.com>
Cc: Aurora Discussion List <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
Subject: Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspotaccording to onetrace, the kernel runtimes look about the same (my original timings were done by putting std::chrono calls around the kernel launches). The total API time for the L0 backend goes from 501942950 to 11021208546 when it’s done with taskset. here are the explicit numbers:
no taskset:
Function, Calls, Time (ns), Time (%), Average (ns), Min (ns), Max (ns)zeModuleCreate, 2, 169715504, 33.81, 84857752, 8910086, 160805418zeCommandQueueSynchronize, 9154, 142616861, 28.41, 15579, 53, 1037213zeCommandListReset, 3976, 86385717, 17.21, 21726, 183, 245071zeCommandListAppendMemoryCopy, 1700, 44595645, 8.88, 26232, 18077, 172220zeCommandQueueExecuteCommandLists, 3976, 25818462, 5.14, 6493, 1307, 2848223zeMemAllocDevice, 67, 15173695, 3.02, 226473, 90346, 2874175zeCommandListAppendMemoryFill, 503, 5027360, 1.00, 9994, 1520, 1823447zeKernelCreate, 22, 1828342, 0.36, 83106, 2202, 828879zeCommandListCreateImmediate, 1, 1803461, 0.36, 1803461, 1803461, 1803461zeEventQueryStatus, 1381, 1714500, 0.34, 1241, 364, 98024zeCommandQueueCreate, 2, 1408878, 0.28, 704439, 259941, 1148937zeMemFree, 6, 1364497, 0.27, 227416, 184775, 293323zeCommandListCreate, 3, 854943, 0.17, 284981, 168795, 417739zeCommandListAppendBarrier, 1381, 616387, 0.12, 446, 210, 6665zeCommandListAppendLaunchKernel, 393, 600600, 0.12, 1528, 667, 31174zeFenceQueryStatus, 3977, 401428, 0.08, 100, 41, 3480zeEventHostReset, 3974, 389108, 0.08, 97, 55, 3097zeMemGetAllocProperties, 1892, 370859, 0.07, 196, 66, 3120zeCommandListClose, 3976, 347435, 0.07, 87, 39, 1089zeMemAllocHost, 1, 293942, 0.06, 293942, 293942, 293942zeFenceReset, 3976, 147342, 0.03, 37, 16, 3051zeKernelSetArgumentValue, 602, 139952, 0.03, 232, 48, 3794zeKernelSuggestGroupSize, 393, 108755, 0.02, 276, 97, 3128zeEventPoolCreate, 1, 104642, 0.02, 104642, 104642, 104642zeKernelSetGroupSize, 393, 55013, 0.01, 139, 60, 523zeDeviceGetCommandQueueGroupProperties, 60, 16436, 0.00, 273, 42, 1570zeDeviceGetProperties, 18, 8115, 0.00, 450, 240, 1604zeEventCreate, 3, 7557, 0.00, 2519, 735, 5934zeContextCreate, 1, 5449, 0.00, 5449, 5449, 5449zeKernelSetIndirectAccess, 22, 4009, 0.00, 182, 27, 1807zeKernelGetProperties, 22, 3981, 0.00, 180, 48, 479zeDeviceGetModuleProperties, 6, 2776, 0.00, 462, 121, 1977zeDriverGetExtensionProperties, 2, 2318, 0.00, 1159, 584, 1734zeFenceCreate, 3, 1882, 0.00, 627, 220, 996zeDeviceGetSubDevices, 12, 1722, 0.00, 143, 46, 297zeDriverGetProperties, 1, 1192, 0.00, 1192, 1192, 1192zeModuleGetProperties, 2, 945, 0.00, 472, 461, 484zeDeviceGetComputeProperties, 1, 775, 0.00, 775, 775, 775zeInit, 1, 759, 0.00, 759, 759, 759zeDeviceGet, 2, 731, 0.00, 365, 263, 468zeDriverGet, 2, 371, 0.00, 185, 119, 252zeDeviceGetImageProperties, 6, 317, 0.00, 52, 30, 150zeDriverGetApiVersion, 1, 287, 0.00, 287, 287, 287with taskset:
Function, Calls, Time (ns), Time (%), Average (ns), Min (ns), Max (ns)zeCommandQueueSynchronize, 9154, 7637427277, 69.30, 834326, 43, 171775424zeEventQueryStatus, 1381, 2438612599, 22.13, 1765830, 294, 190701852zeCommandListReset, 3976, 261265117, 2.37, 65710, 182, 2516131zeCommandListCreateImmediate, 1, 189954019, 1.72, 189954019, 189954019, 189954019zeModuleCreate, 2, 162170189, 1.47, 81085094, 8081706, 154088483zeMemFree, 6, 130338449, 1.18, 21723074, 213200, 129109118zeCommandQueueExecuteCommandLists, 3976, 92877003, 0.84, 23359, 1211, 3024913zeCommandListAppendMemoryCopy, 1700, 47362492, 0.43, 27860, 15042, 946050zeMemAllocDevice, 67, 43488320, 0.39, 649079, 97185, 15510137zeKernelCreate, 22, 7986520, 0.07, 363023, 2088, 4065025zeCommandListAppendMemoryFill, 503, 4381709, 0.04, 8711, 1004, 1694162zeCommandQueueCreate, 2, 1350202, 0.01, 675101, 225169, 1125033zeCommandListCreate, 3, 815519, 0.01, 271839, 142204, 360245zeCommandListAppendLaunchKernel, 393, 729006, 0.01, 1854, 573, 19302zeCommandListAppendBarrier, 1381, 419554, 0.00, 303, 193, 3665zeMemGetAllocProperties, 1892, 378477, 0.00, 200, 62, 2390zeEventHostReset, 3974, 340554, 0.00, 85, 44, 6242zeCommandListClose, 3976, 281788, 0.00, 70, 30, 759zeFenceQueryStatus, 3977, 280667, 0.00, 70, 40, 1319zeMemAllocHost, 1, 236196, 0.00, 236196, 236196, 236196zeKernelSetArgumentValue, 602, 137640, 0.00, 228, 40, 2542zeKernelSuggestGroupSize, 393, 125753, 0.00, 319, 91, 2101zeEventPoolCreate, 1, 78056, 0.00, 78056, 78056, 78056zeFenceReset, 3976, 73421, 0.00, 18, 16, 110zeKernelSetGroupSize, 393, 54161, 0.00, 137, 38, 357zeDeviceGetCommandQueueGroupProperties, 60, 13418, 0.00, 223, 20, 1281zeDeviceGetProperties, 18, 6354, 0.00, 353, 189, 1069zeEventCreate, 3, 4290, 0.00, 1430, 385, 3395zeKernelGetProperties, 22, 3189, 0.00, 144, 43, 286zeContextCreate, 1, 3134, 0.00, 3134, 3134, 3134zeKernelSetIndirectAccess, 22, 3084, 0.00, 140, 21, 1170zeDeviceGetModuleProperties, 6, 1817, 0.00, 302, 84, 1118zeDeviceGetSubDevices, 12, 1816, 0.00, 151, 28, 329zeDriverGetExtensionProperties, 2, 1593, 0.00, 796, 316, 1277zeFenceCreate, 3, 1033, 0.00, 344, 115, 524zeDriverGetProperties, 1, 998, 0.00, 998, 998, 998zeModuleGetProperties, 2, 876, 0.00, 438, 422, 454zeDeviceGet, 2, 657, 0.00, 328, 252, 405zeInit, 1, 562, 0.00, 562, 562, 562zeDeviceGetComputeProperties, 1, 322, 0.00, 322, 322, 322zeDriverGet, 2, 290, 0.00, 145, 107, 183zeDriverGetApiVersion, 1, 223, 0.00, 223, 223, 223zeDeviceGetImageProperties, 6, 202, 0.00, 33, 17, 102
On Mon, Mar 27, 2023 at 6:44 AM Steinbrecher, Patrick <patrick.steinbrecher@intel.com> wrote:
Can you run onetrace profiler to see if the kernel execution times goes up compared to your run without taskset?
module load tools/pti-gpu
onetrace -d ./a.out
if that does not show a difference look at host API overhead
onterace -h ./a.out
From: Charles Leggett <cgleggett@lbl.gov>
Date: Friday, March 24, 2023 at 19:59
To: Steinbrecher, Patrick <patrick.steinbrecher@intel.com>
Cc: Aurora Discussion List <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
Subject: Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot
It's only running a single binary on a single core. The script is setup to be able to launch multiple concurrent jobs on different cores, but I'm only using it on a single one right now.
The cores I tried are 0, 1, 8, 10, 18, 51 and 200. All performed the same.
On Fri, Mar 24, 2023 at 5:44 PM Steinbrecher, Patrick <patrick.steinbrecher@intel.com> wrote:
How many binaries are you running with task set? If you run multiple binaries, how do you map them to different GPUs?
What are the exact core numbers you are using (from sched_getcpu()) ?
From: Charles Leggett <cgleggett@lbl.gov>
Date: Friday, March 24, 2023 at 19:19
To: Steinbrecher, Patrick <patrick.steinbrecher@intel.com>
Cc: Aurora Discussion List <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
Subject: Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot
It shows the core that taskset has selected.
I'm assuming you don't mean doing this from inside the kokkos::parallel_for.
On Fri, Mar 24, 2023 at 3:38 PM Steinbrecher, Patrick <patrick.steinbrecher@intel.com> wrote:
In your program that is run under taskset add this:
#include <sched.h>
std::cout << “I am running on core: ” << sched_getcpu() << std::endl;
What is the output you get?
From: Charles Leggett <cgleggett@lbl.gov>
Date: Friday, March 24, 2023 at 17:21
To: Steinbrecher, Patrick <patrick.steinbrecher@intel.com>
Cc: Aurora Discussion List <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
Subject: Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot
There is no explicit use of MPI in user code. I have no idea what Kokkos is doing under the hood, but I thought it was using a SYCL/Level zero backend.
On Fri, Mar 24, 2023 at 3:09 PM Steinbrecher, Patrick <patrick.steinbrecher@intel.com> wrote:
Are you using MPI in this case?
From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces@lists.jlse.anl.gov> on behalf of Charles Leggett via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
Date: Friday, March 24, 2023 at 12:50
To: Aurora Discussion List <ecp-aurora-sdk-discuss@lists.jlse.anl.gov>
Subject: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot
I have a script that launches jobs which employs `taskset -c` to lock an executable onto a specific core (this was done to test concurrent job execution). What's very odd is that using taskset slows down kernel execution on the GPU by about a factor of 600. eg:
without taskset:
kernel total /s avg launch /us std dev /ussim_clean 0.02091981 486.5 18.7sim_A 0.10563853 2456.7 703.2sim_ct 0.03402171 791.2 9.6sim_cp 0.01612411 375.0 7.0
with taskset:
kernel total /s avg launch /us std dev /ussim_clean 0.02155314 501.2 20.5sim_A 60.56034054 1408380.0 398894.4sim_ct 1.33204648 30977.8 50526.0sim_cp 0.99585704 23159.5 13514.5
I dump the environment vars within the job, and they look identical.
I tried a few different cores to set the cpu affinity, and it doesn't seem to make a difference (I didn't try all 208)
I'm using oneapi/eng-compiler/2022.12.30.003 and kokkos/2022.12.30.003/eng-compiler/sycl_intel_aot
I don't think I noticed this behavior with pure SYCL code, but now I'll have to go back and check.
This was on node x1921c2s5b0n0
cheers, Charles.
--Charles Leggett <CGLeggett@lbl.gov>
Lawrence Berkeley National Lab
1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720tel: 510.495.2930
--Charles Leggett <CGLeggett@lbl.gov>
Lawrence Berkeley National Lab
1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720tel: 510.495.2930
--Charles Leggett <CGLeggett@lbl.gov>
Lawrence Berkeley National Lab
1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720tel: 510.495.2930
--Charles Leggett <CGLeggett@lbl.gov>
Lawrence Berkeley National Lab
1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720tel: 510.495.2930
--Charles Leggett <CGLeggett@lbl.gov>
Lawrence Berkeley National Lab
1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720tel: 510.495.2930