March 2023 - Ecp-aurora-sdk-discuss - mailman3.cels.anl.gov

teams using buffers and accessors in SYCL?
by Bertoni, Colleen 29 Mar '23

29 Mar '23

Hi everyone, This is an informal survey to see how many teams are using the buffers and accessors in SYCL to do data management (as opposed to SYCL USM and pointers). If you or your team is, please respond directly to me. Thanks, Colleen

1 0

Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot
by Steinbrecher, Patrick 27 Mar '23

27 Mar '23

Can you give me steps how to compile and run? From: Charles Leggett <cgleggett(a)lbl.gov> Date: Monday, March 27, 2023 at 10:18 To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com> Cc: Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot this is using Kokkos. I tried running with 2,3 and 4 cores, but it didn't improve the situation. On Mon, Mar 27, 2023 at 8:12 AM Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com<mailto:[email protected]>> wrote: Interesting, looks like zeCommandQueueSynchronize takes way longer. Is this SYCL or OpenMP? Just wondering if one of the helper threads keeps competing with your app. Can you try giving two cores (ideally not hyper threads, e.g. 2 and 4) to the binary via taskset? From: Charles Leggett <cgleggett(a)lbl.gov<mailto:[email protected]>> Date: Monday, March 27, 2023 at 10:07 To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com<mailto:[email protected]>> Cc: Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot according to onetrace, the kernel runtimes look about the same (my original timings were done by putting std::chrono calls around the kernel launches). The total API time for the L0 backend goes from 501942950 to 11021208546 when it’s done with taskset. here are the explicit numbers: no taskset: Function, Calls, Time (ns), Time (%), Average (ns), Min (ns), Max (ns) zeModuleCreate, 2, 169715504, 33.81, 84857752, 8910086, 160805418 zeCommandQueueSynchronize, 9154, 142616861, 28.41, 15579, 53, 1037213 zeCommandListReset, 3976, 86385717, 17.21, 21726, 183, 245071 zeCommandListAppendMemoryCopy, 1700, 44595645, 8.88, 26232, 18077, 172220 zeCommandQueueExecuteCommandLists, 3976, 25818462, 5.14, 6493, 1307, 2848223 zeMemAllocDevice, 67, 15173695, 3.02, 226473, 90346, 2874175 zeCommandListAppendMemoryFill, 503, 5027360, 1.00, 9994, 1520, 1823447 zeKernelCreate, 22, 1828342, 0.36, 83106, 2202, 828879 zeCommandListCreateImmediate, 1, 1803461, 0.36, 1803461, 1803461, 1803461 zeEventQueryStatus, 1381, 1714500, 0.34, 1241, 364, 98024 zeCommandQueueCreate, 2, 1408878, 0.28, 704439, 259941, 1148937 zeMemFree, 6, 1364497, 0.27, 227416, 184775, 293323 zeCommandListCreate, 3, 854943, 0.17, 284981, 168795, 417739 zeCommandListAppendBarrier, 1381, 616387, 0.12, 446, 210, 6665 zeCommandListAppendLaunchKernel, 393, 600600, 0.12, 1528, 667, 31174 zeFenceQueryStatus, 3977, 401428, 0.08, 100, 41, 3480 zeEventHostReset, 3974, 389108, 0.08, 97, 55, 3097 zeMemGetAllocProperties, 1892, 370859, 0.07, 196, 66, 3120 zeCommandListClose, 3976, 347435, 0.07, 87, 39, 1089 zeMemAllocHost, 1, 293942, 0.06, 293942, 293942, 293942 zeFenceReset, 3976, 147342, 0.03, 37, 16, 3051 zeKernelSetArgumentValue, 602, 139952, 0.03, 232, 48, 3794 zeKernelSuggestGroupSize, 393, 108755, 0.02, 276, 97, 3128 zeEventPoolCreate, 1, 104642, 0.02, 104642, 104642, 104642 zeKernelSetGroupSize, 393, 55013, 0.01, 139, 60, 523 zeDeviceGetCommandQueueGroupProperties, 60, 16436, 0.00, 273, 42, 1570 zeDeviceGetProperties, 18, 8115, 0.00, 450, 240, 1604 zeEventCreate, 3, 7557, 0.00, 2519, 735, 5934 zeContextCreate, 1, 5449, 0.00, 5449, 5449, 5449 zeKernelSetIndirectAccess, 22, 4009, 0.00, 182, 27, 1807 zeKernelGetProperties, 22, 3981, 0.00, 180, 48, 479 zeDeviceGetModuleProperties, 6, 2776, 0.00, 462, 121, 1977 zeDriverGetExtensionProperties, 2, 2318, 0.00, 1159, 584, 1734 zeFenceCreate, 3, 1882, 0.00, 627, 220, 996 zeDeviceGetSubDevices, 12, 1722, 0.00, 143, 46, 297 zeDriverGetProperties, 1, 1192, 0.00, 1192, 1192, 1192 zeModuleGetProperties, 2, 945, 0.00, 472, 461, 484 zeDeviceGetComputeProperties, 1, 775, 0.00, 775, 775, 775 zeInit, 1, 759, 0.00, 759, 759, 759 zeDeviceGet, 2, 731, 0.00, 365, 263, 468 zeDriverGet, 2, 371, 0.00, 185, 119, 252 zeDeviceGetImageProperties, 6, 317, 0.00, 52, 30, 150 zeDriverGetApiVersion, 1, 287, 0.00, 287, 287, 287 with taskset: Function, Calls, Time (ns), Time (%), Average (ns), Min (ns), Max (ns) zeCommandQueueSynchronize, 9154, 7637427277, 69.30, 834326, 43, 171775424 zeEventQueryStatus, 1381, 2438612599, 22.13, 1765830, 294, 190701852 zeCommandListReset, 3976, 261265117, 2.37, 65710, 182, 2516131 zeCommandListCreateImmediate, 1, 189954019, 1.72, 189954019, 189954019, 189954019 zeModuleCreate, 2, 162170189, 1.47, 81085094, 8081706, 154088483 zeMemFree, 6, 130338449, 1.18, 21723074, 213200, 129109118 zeCommandQueueExecuteCommandLists, 3976, 92877003, 0.84, 23359, 1211, 3024913 zeCommandListAppendMemoryCopy, 1700, 47362492, 0.43, 27860, 15042, 946050 zeMemAllocDevice, 67, 43488320, 0.39, 649079, 97185, 15510137 zeKernelCreate, 22, 7986520, 0.07, 363023, 2088, 4065025 zeCommandListAppendMemoryFill, 503, 4381709, 0.04, 8711, 1004, 1694162 zeCommandQueueCreate, 2, 1350202, 0.01, 675101, 225169, 1125033 zeCommandListCreate, 3, 815519, 0.01, 271839, 142204, 360245 zeCommandListAppendLaunchKernel, 393, 729006, 0.01, 1854, 573, 19302 zeCommandListAppendBarrier, 1381, 419554, 0.00, 303, 193, 3665 zeMemGetAllocProperties, 1892, 378477, 0.00, 200, 62, 2390 zeEventHostReset, 3974, 340554, 0.00, 85, 44, 6242 zeCommandListClose, 3976, 281788, 0.00, 70, 30, 759 zeFenceQueryStatus, 3977, 280667, 0.00, 70, 40, 1319 zeMemAllocHost, 1, 236196, 0.00, 236196, 236196, 236196 zeKernelSetArgumentValue, 602, 137640, 0.00, 228, 40, 2542 zeKernelSuggestGroupSize, 393, 125753, 0.00, 319, 91, 2101 zeEventPoolCreate, 1, 78056, 0.00, 78056, 78056, 78056 zeFenceReset, 3976, 73421, 0.00, 18, 16, 110 zeKernelSetGroupSize, 393, 54161, 0.00, 137, 38, 357 zeDeviceGetCommandQueueGroupProperties, 60, 13418, 0.00, 223, 20, 1281 zeDeviceGetProperties, 18, 6354, 0.00, 353, 189, 1069 zeEventCreate, 3, 4290, 0.00, 1430, 385, 3395 zeKernelGetProperties, 22, 3189, 0.00, 144, 43, 286 zeContextCreate, 1, 3134, 0.00, 3134, 3134, 3134 zeKernelSetIndirectAccess, 22, 3084, 0.00, 140, 21, 1170 zeDeviceGetModuleProperties, 6, 1817, 0.00, 302, 84, 1118 zeDeviceGetSubDevices, 12, 1816, 0.00, 151, 28, 329 zeDriverGetExtensionProperties, 2, 1593, 0.00, 796, 316, 1277 zeFenceCreate, 3, 1033, 0.00, 344, 115, 524 zeDriverGetProperties, 1, 998, 0.00, 998, 998, 998 zeModuleGetProperties, 2, 876, 0.00, 438, 422, 454 zeDeviceGet, 2, 657, 0.00, 328, 252, 405 zeInit, 1, 562, 0.00, 562, 562, 562 zeDeviceGetComputeProperties, 1, 322, 0.00, 322, 322, 322 zeDriverGet, 2, 290, 0.00, 145, 107, 183 zeDriverGetApiVersion, 1, 223, 0.00, 223, 223, 223 zeDeviceGetImageProperties, 6, 202, 0.00, 33, 17, 102 On Mon, Mar 27, 2023 at 6:44 AM Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com<mailto:[email protected]>> wrote: Can you run onetrace profiler to see if the kernel execution times goes up compared to your run without taskset? module load tools/pti-gpu onetrace -d ./a.out if that does not show a difference look at host API overhead onterace -h ./a.out From: Charles Leggett <cgleggett(a)lbl.gov<mailto:[email protected]>> Date: Friday, March 24, 2023 at 19:59 To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com<mailto:[email protected]>> Cc: Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot It's only running a single binary on a single core. The script is setup to be able to launch multiple concurrent jobs on different cores, but I'm only using it on a single one right now. The cores I tried are 0, 1, 8, 10, 18, 51 and 200. All performed the same. On Fri, Mar 24, 2023 at 5:44 PM Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com<mailto:[email protected]>> wrote: How many binaries are you running with task set? If you run multiple binaries, how do you map them to different GPUs? What are the exact core numbers you are using (from sched_getcpu()) ? From: Charles Leggett <cgleggett(a)lbl.gov<mailto:[email protected]>> Date: Friday, March 24, 2023 at 19:19 To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com<mailto:[email protected]>> Cc: Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot It shows the core that taskset has selected. I'm assuming you don't mean doing this from inside the kokkos::parallel_for. On Fri, Mar 24, 2023 at 3:38 PM Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com<mailto:[email protected]>> wrote: In your program that is run under taskset add this: #include <sched.h> std::cout << “I am running on core: ” << sched_getcpu() << std::endl; What is the output you get? From: Charles Leggett <cgleggett(a)lbl.gov<mailto:[email protected]>> Date: Friday, March 24, 2023 at 17:21 To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com<mailto:[email protected]>> Cc: Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot There is no explicit use of MPI in user code. I have no idea what Kokkos is doing under the hood, but I thought it was using a SYCL/Level zero backend. On Fri, Mar 24, 2023 at 3:09 PM Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com<mailto:[email protected]>> wrote: Are you using MPI in this case? From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of Charles Leggett via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Date: Friday, March 24, 2023 at 12:50 To: Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot I have a script that launches jobs which employs `taskset -c` to lock an executable onto a specific core (this was done to test concurrent job execution). What's very odd is that using taskset slows down kernel execution on the GPU by about a factor of 600. eg: without taskset: kernel total /s avg launch /us std dev /us sim_clean 0.02091981 486.5 18.7 sim_A 0.10563853 2456.7 703.2 sim_ct 0.03402171 791.2 9.6 sim_cp 0.01612411 375.0 7.0 with taskset: kernel total /s avg launch /us std dev /us sim_clean 0.02155314 501.2 20.5 sim_A 60.56034054 1408380.0 398894.4 sim_ct 1.33204648 30977.8 50526.0 sim_cp 0.99585704 23159.5 13514.5 I dump the environment vars within the job, and they look identical. I tried a few different cores to set the cpu affinity, and it doesn't seem to make a difference (I didn't try all 208) I'm using oneapi/eng-compiler/2022.12.30.003 and kokkos/2022.12.30.003/eng-compiler/sycl_intel_aot I don't think I noticed this behavior with pure SYCL code, but now I'll have to go back and check. This was on node x1921c2s5b0n0 cheers, Charles. -- Charles Leggett <CGLeggett(a)lbl.gov<mailto:[email protected]>> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930 -- Charles Leggett <CGLeggett(a)lbl.gov<mailto:[email protected]>> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930 -- Charles Leggett <CGLeggett(a)lbl.gov<mailto:[email protected]>> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930 -- Charles Leggett <CGLeggett(a)lbl.gov<mailto:[email protected]>> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930 -- Charles Leggett <CGLeggett(a)lbl.gov<mailto:[email protected]>> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930 -- Charles Leggett <CGLeggett(a)lbl.gov<mailto:[email protected]>> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930

1 0

Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot
by Charles Leggett 27 Mar '23

27 Mar '23

this is using Kokkos. I tried running with 2,3 and 4 cores, but it didn't improve the situation. On Mon, Mar 27, 2023 at 8:12 AM Steinbrecher, Patrick < patrick.steinbrecher(a)intel.com> wrote: > Interesting, looks like zeCommandQueueSynchronize takes way longer. > > > > Is this SYCL or OpenMP? Just wondering if one of the helper threads keeps > competing with your app. > > > > Can you try giving two cores (ideally not hyper threads, e.g. 2 and 4) to > the binary via taskset? > > > > *From: *Charles Leggett <cgleggett(a)lbl.gov> > *Date: *Monday, March 27, 2023 at 10:07 > *To: *Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com> > *Cc: *Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > *Subject: *Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on > sunspot > > according to onetrace, the kernel runtimes look about the same (my > original timings were done by putting std::chrono calls around the kernel > launches). The total API time for the L0 backend goes from 501942950 to > 11021208546 when it’s done with taskset. here are the explicit numbers: > > no taskset: > > Function, Calls, Time (ns), Time (%), Average (ns), Min (ns), Max (ns) > > zeModuleCreate, 2, 169715504, 33.81, 84857752, 8910086, 160805418 > > zeCommandQueueSynchronize, 9154, 142616861, 28.41, 15579, 53, 1037213 > > zeCommandListReset, 3976, 86385717, 17.21, 21726, 183, 245071 > > zeCommandListAppendMemoryCopy, 1700, 44595645, 8.88, 26232, 18077, 172220 > > zeCommandQueueExecuteCommandLists, 3976, 25818462, 5.14, 6493, 1307, 2848223 > > zeMemAllocDevice, 67, 15173695, 3.02, 226473, 90346, 2874175 > > zeCommandListAppendMemoryFill, 503, 5027360, 1.00, 9994, 1520, 1823447 > > zeKernelCreate, 22, 1828342, 0.36, 83106, 2202, 828879 > > zeCommandListCreateImmediate, 1, 1803461, 0.36, 1803461, 1803461, 1803461 > > zeEventQueryStatus, 1381, 1714500, 0.34, 1241, 364, 98024 > > zeCommandQueueCreate, 2, 1408878, 0.28, 704439, 259941, 1148937 > > zeMemFree, 6, 1364497, 0.27, 227416, 184775, 293323 > > zeCommandListCreate, 3, 854943, 0.17, 284981, 168795, 417739 > > zeCommandListAppendBarrier, 1381, 616387, 0.12, 446, 210, 6665 > > zeCommandListAppendLaunchKernel, 393, 600600, 0.12, 1528, 667, 31174 > > zeFenceQueryStatus, 3977, 401428, 0.08, 100, 41, 3480 > > zeEventHostReset, 3974, 389108, 0.08, 97, 55, 3097 > > zeMemGetAllocProperties, 1892, 370859, 0.07, 196, 66, 3120 > > zeCommandListClose, 3976, 347435, 0.07, 87, 39, 1089 > > zeMemAllocHost, 1, 293942, 0.06, 293942, 293942, 293942 > > zeFenceReset, 3976, 147342, 0.03, 37, 16, 3051 > > zeKernelSetArgumentValue, 602, 139952, 0.03, 232, 48, 3794 > > zeKernelSuggestGroupSize, 393, 108755, 0.02, 276, 97, 3128 > > zeEventPoolCreate, 1, 104642, 0.02, 104642, 104642, 104642 > > zeKernelSetGroupSize, 393, 55013, 0.01, 139, 60, 523 > > zeDeviceGetCommandQueueGroupProperties, 60, 16436, 0.00, 273, 42, 1570 > > zeDeviceGetProperties, 18, 8115, 0.00, 450, 240, 1604 > > zeEventCreate, 3, 7557, 0.00, 2519, 735, 5934 > > zeContextCreate, 1, 5449, 0.00, 5449, 5449, 5449 > > zeKernelSetIndirectAccess, 22, 4009, 0.00, 182, 27, 1807 > > zeKernelGetProperties, 22, 3981, 0.00, 180, 48, 479 > > zeDeviceGetModuleProperties, 6, 2776, 0.00, 462, 121, 1977 > > zeDriverGetExtensionProperties, 2, 2318, 0.00, 1159, 584, 1734 > > zeFenceCreate, 3, 1882, 0.00, 627, 220, 996 > > zeDeviceGetSubDevices, 12, 1722, 0.00, 143, 46, 297 > > zeDriverGetProperties, 1, 1192, 0.00, 1192, 1192, 1192 > > zeModuleGetProperties, 2, 945, 0.00, 472, 461, 484 > > zeDeviceGetComputeProperties, 1, 775, 0.00, 775, 775, 775 > > zeInit, 1, 759, 0.00, 759, 759, 759 > > zeDeviceGet, 2, 731, 0.00, 365, 263, 468 > > zeDriverGet, 2, 371, 0.00, 185, 119, 252 > > zeDeviceGetImageProperties, 6, 317, 0.00, 52, 30, 150 > > zeDriverGetApiVersion, 1, 287, 0.00, 287, 287, 287 > > with taskset: > > Function, Calls, Time (ns), Time (%), Average (ns), Min (ns), Max (ns) > > zeCommandQueueSynchronize, 9154, 7637427277, 69.30, 834326, 43, 171775424 > > zeEventQueryStatus, 1381, 2438612599, 22.13, 1765830, 294, 190701852 > > zeCommandListReset, 3976, 261265117, 2.37, 65710, 182, 2516131 > > zeCommandListCreateImmediate, 1, 189954019, 1.72, 189954019, 189954019, 189954019 > > zeModuleCreate, 2, 162170189, 1.47, 81085094, 8081706, 154088483 > > zeMemFree, 6, 130338449, 1.18, 21723074, 213200, 129109118 > > zeCommandQueueExecuteCommandLists, 3976, 92877003, 0.84, 23359, 1211, 3024913 > > zeCommandListAppendMemoryCopy, 1700, 47362492, 0.43, 27860, 15042, 946050 > > zeMemAllocDevice, 67, 43488320, 0.39, 649079, 97185, 15510137 > > zeKernelCreate, 22, 7986520, 0.07, 363023, 2088, 4065025 > > zeCommandListAppendMemoryFill, 503, 4381709, 0.04, 8711, 1004, 1694162 > > zeCommandQueueCreate, 2, 1350202, 0.01, 675101, 225169, 1125033 > > zeCommandListCreate, 3, 815519, 0.01, 271839, 142204, 360245 > > zeCommandListAppendLaunchKernel, 393, 729006, 0.01, 1854, 573, 19302 > > zeCommandListAppendBarrier, 1381, 419554, 0.00, 303, 193, 3665 > > zeMemGetAllocProperties, 1892, 378477, 0.00, 200, 62, 2390 > > zeEventHostReset, 3974, 340554, 0.00, 85, 44, 6242 > > zeCommandListClose, 3976, 281788, 0.00, 70, 30, 759 > > zeFenceQueryStatus, 3977, 280667, 0.00, 70, 40, 1319 > > zeMemAllocHost, 1, 236196, 0.00, 236196, 236196, 236196 > > zeKernelSetArgumentValue, 602, 137640, 0.00, 228, 40, 2542 > > zeKernelSuggestGroupSize, 393, 125753, 0.00, 319, 91, 2101 > > zeEventPoolCreate, 1, 78056, 0.00, 78056, 78056, 78056 > > zeFenceReset, 3976, 73421, 0.00, 18, 16, 110 > > zeKernelSetGroupSize, 393, 54161, 0.00, 137, 38, 357 > > zeDeviceGetCommandQueueGroupProperties, 60, 13418, 0.00, 223, 20, 1281 > > zeDeviceGetProperties, 18, 6354, 0.00, 353, 189, 1069 > > zeEventCreate, 3, 4290, 0.00, 1430, 385, 3395 > > zeKernelGetProperties, 22, 3189, 0.00, 144, 43, 286 > > zeContextCreate, 1, 3134, 0.00, 3134, 3134, 3134 > > zeKernelSetIndirectAccess, 22, 3084, 0.00, 140, 21, 1170 > > zeDeviceGetModuleProperties, 6, 1817, 0.00, 302, 84, 1118 > > zeDeviceGetSubDevices, 12, 1816, 0.00, 151, 28, 329 > > zeDriverGetExtensionProperties, 2, 1593, 0.00, 796, 316, 1277 > > zeFenceCreate, 3, 1033, 0.00, 344, 115, 524 > > zeDriverGetProperties, 1, 998, 0.00, 998, 998, 998 > > zeModuleGetProperties, 2, 876, 0.00, 438, 422, 454 > > zeDeviceGet, 2, 657, 0.00, 328, 252, 405 > > zeInit, 1, 562, 0.00, 562, 562, 562 > > zeDeviceGetComputeProperties, 1, 322, 0.00, 322, 322, 322 > > zeDriverGet, 2, 290, 0.00, 145, 107, 183 > > zeDriverGetApiVersion, 1, 223, 0.00, 223, 223, 223 > > zeDeviceGetImageProperties, 6, 202, 0.00, 33, 17, 102 > > > > On Mon, Mar 27, 2023 at 6:44 AM Steinbrecher, Patrick < > patrick.steinbrecher(a)intel.com> wrote: > > Can you run onetrace profiler to see if the kernel execution times goes up > compared to your run without taskset? > > > > module load tools/pti-gpu > > > > onetrace -d ./a.out > > > > if that does not show a difference look at host API overhead > > > > onterace -h ./a.out > > > > *From: *Charles Leggett <cgleggett(a)lbl.gov> > *Date: *Friday, March 24, 2023 at 19:59 > *To: *Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com> > *Cc: *Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > *Subject: *Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on > sunspot > > > > It's only running a single binary on a single core. The script is setup to > be able to launch multiple concurrent jobs on different cores, but I'm only > using it on a single one right now. > > > > The cores I tried are 0, 1, 8, 10, 18, 51 and 200. All performed the same. > > > > > > > > On Fri, Mar 24, 2023 at 5:44 PM Steinbrecher, Patrick < > patrick.steinbrecher(a)intel.com> wrote: > > How many binaries are you running with task set? If you run multiple > binaries, how do you map them to different GPUs? > > What are the exact core numbers you are using (from sched_getcpu()) ? > > > > *From: *Charles Leggett <cgleggett(a)lbl.gov> > *Date: *Friday, March 24, 2023 at 19:19 > *To: *Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com> > *Cc: *Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > *Subject: *Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on > sunspot > > > > It shows the core that taskset has selected. > > > > I'm assuming you don't mean doing this from inside the > kokkos::parallel_for. > > > > On Fri, Mar 24, 2023 at 3:38 PM Steinbrecher, Patrick < > patrick.steinbrecher(a)intel.com> wrote: > > In your program that is run under taskset add this: > > > > #include <sched.h> > > > > std::cout << “I am running on core: ” << sched_getcpu() << std::endl; > > > > > > What is the output you get? > > > > *From: *Charles Leggett <cgleggett(a)lbl.gov> > *Date: *Friday, March 24, 2023 at 17:21 > *To: *Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com> > *Cc: *Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > *Subject: *Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on > sunspot > > > > There is no explicit use of MPI in user code. I have no idea what Kokkos > is doing under the hood, but I thought it was using a SYCL/Level zero > backend. > > > > On Fri, Mar 24, 2023 at 3:09 PM Steinbrecher, Patrick < > patrick.steinbrecher(a)intel.com> wrote: > > Are you using MPI in this case? > > > > *From: *Ecp-aurora-sdk-discuss < > ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Charles > Leggett via Ecp-aurora-sdk-discuss < > ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > *Date: *Friday, March 24, 2023 at 12:50 > *To: *Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > *Subject: *[Ecp-aurora-sdk-discuss] using taskset slows down GPU on > sunspot > > > > I have a script that launches jobs which employs `taskset -c` to lock an > executable onto a specific core (this was done to test concurrent job > execution). What's very odd is that using taskset slows down kernel > execution on the GPU by about a factor of 600. eg: > > > > without taskset: > > kernel total /s avg launch /us std dev /us > > sim_clean 0.02091981 486.5 18.7 > > sim_A 0.10563853 2456.7 703.2 > > sim_ct 0.03402171 791.2 9.6 > > sim_cp 0.01612411 375.0 7.0 > > > > with taskset: > > kernel total /s avg launch /us std dev /us > > sim_clean 0.02155314 501.2 20.5 > > sim_A 60.56034054 1408380.0 398894.4 > > sim_ct 1.33204648 30977.8 50526.0 > > sim_cp 0.99585704 23159.5 13514.5 > > > > I dump the environment vars within the job, and they look identical. > > > > I tried a few different cores to set the cpu affinity, and it doesn't seem > to make a difference (I didn't try all 208) > > > > I'm using oneapi/eng-compiler/2022.12.30.003 and > kokkos/2022.12.30.003/eng-compiler/sycl_intel_aot > > > > I don't think I noticed this behavior with pure SYCL code, but now I'll > have to go back and check. > > > > This was on node x1921c2s5b0n0 > > > > > > cheers, Charles. > > > -- > > Charles Leggett <CGLeggett(a)lbl.gov> > Lawrence Berkeley National Lab > 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 > > tel: 510.495.2930 > > > > -- > > Charles Leggett <CGLeggett(a)lbl.gov> > Lawrence Berkeley National Lab > 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 > > tel: 510.495.2930 > > > > -- > > Charles Leggett <CGLeggett(a)lbl.gov> > Lawrence Berkeley National Lab > 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 > > tel: 510.495.2930 > > > > -- > > Charles Leggett <CGLeggett(a)lbl.gov> > Lawrence Berkeley National Lab > 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 > > tel: 510.495.2930 > > > > -- > > Charles Leggett <CGLeggett(a)lbl.gov> > Lawrence Berkeley National Lab > 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 > > tel: 510.495.2930 > -- Charles Leggett <CGLeggett(a)lbl.gov> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930

1 0

Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot
by Steinbrecher, Patrick 27 Mar '23

27 Mar '23

Interesting, looks like zeCommandQueueSynchronize takes way longer. Is this SYCL or OpenMP? Just wondering if one of the helper threads keeps competing with your app. Can you try giving two cores (ideally not hyper threads, e.g. 2 and 4) to the binary via taskset? From: Charles Leggett <cgleggett(a)lbl.gov> Date: Monday, March 27, 2023 at 10:07 To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com> Cc: Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot according to onetrace, the kernel runtimes look about the same (my original timings were done by putting std::chrono calls around the kernel launches). The total API time for the L0 backend goes from 501942950 to 11021208546 when it’s done with taskset. here are the explicit numbers: no taskset: Function, Calls, Time (ns), Time (%), Average (ns), Min (ns), Max (ns) zeModuleCreate, 2, 169715504, 33.81, 84857752, 8910086, 160805418 zeCommandQueueSynchronize, 9154, 142616861, 28.41, 15579, 53, 1037213 zeCommandListReset, 3976, 86385717, 17.21, 21726, 183, 245071 zeCommandListAppendMemoryCopy, 1700, 44595645, 8.88, 26232, 18077, 172220 zeCommandQueueExecuteCommandLists, 3976, 25818462, 5.14, 6493, 1307, 2848223 zeMemAllocDevice, 67, 15173695, 3.02, 226473, 90346, 2874175 zeCommandListAppendMemoryFill, 503, 5027360, 1.00, 9994, 1520, 1823447 zeKernelCreate, 22, 1828342, 0.36, 83106, 2202, 828879 zeCommandListCreateImmediate, 1, 1803461, 0.36, 1803461, 1803461, 1803461 zeEventQueryStatus, 1381, 1714500, 0.34, 1241, 364, 98024 zeCommandQueueCreate, 2, 1408878, 0.28, 704439, 259941, 1148937 zeMemFree, 6, 1364497, 0.27, 227416, 184775, 293323 zeCommandListCreate, 3, 854943, 0.17, 284981, 168795, 417739 zeCommandListAppendBarrier, 1381, 616387, 0.12, 446, 210, 6665 zeCommandListAppendLaunchKernel, 393, 600600, 0.12, 1528, 667, 31174 zeFenceQueryStatus, 3977, 401428, 0.08, 100, 41, 3480 zeEventHostReset, 3974, 389108, 0.08, 97, 55, 3097 zeMemGetAllocProperties, 1892, 370859, 0.07, 196, 66, 3120 zeCommandListClose, 3976, 347435, 0.07, 87, 39, 1089 zeMemAllocHost, 1, 293942, 0.06, 293942, 293942, 293942 zeFenceReset, 3976, 147342, 0.03, 37, 16, 3051 zeKernelSetArgumentValue, 602, 139952, 0.03, 232, 48, 3794 zeKernelSuggestGroupSize, 393, 108755, 0.02, 276, 97, 3128 zeEventPoolCreate, 1, 104642, 0.02, 104642, 104642, 104642 zeKernelSetGroupSize, 393, 55013, 0.01, 139, 60, 523 zeDeviceGetCommandQueueGroupProperties, 60, 16436, 0.00, 273, 42, 1570 zeDeviceGetProperties, 18, 8115, 0.00, 450, 240, 1604 zeEventCreate, 3, 7557, 0.00, 2519, 735, 5934 zeContextCreate, 1, 5449, 0.00, 5449, 5449, 5449 zeKernelSetIndirectAccess, 22, 4009, 0.00, 182, 27, 1807 zeKernelGetProperties, 22, 3981, 0.00, 180, 48, 479 zeDeviceGetModuleProperties, 6, 2776, 0.00, 462, 121, 1977 zeDriverGetExtensionProperties, 2, 2318, 0.00, 1159, 584, 1734 zeFenceCreate, 3, 1882, 0.00, 627, 220, 996 zeDeviceGetSubDevices, 12, 1722, 0.00, 143, 46, 297 zeDriverGetProperties, 1, 1192, 0.00, 1192, 1192, 1192 zeModuleGetProperties, 2, 945, 0.00, 472, 461, 484 zeDeviceGetComputeProperties, 1, 775, 0.00, 775, 775, 775 zeInit, 1, 759, 0.00, 759, 759, 759 zeDeviceGet, 2, 731, 0.00, 365, 263, 468 zeDriverGet, 2, 371, 0.00, 185, 119, 252 zeDeviceGetImageProperties, 6, 317, 0.00, 52, 30, 150 zeDriverGetApiVersion, 1, 287, 0.00, 287, 287, 287 with taskset: Function, Calls, Time (ns), Time (%), Average (ns), Min (ns), Max (ns) zeCommandQueueSynchronize, 9154, 7637427277, 69.30, 834326, 43, 171775424 zeEventQueryStatus, 1381, 2438612599, 22.13, 1765830, 294, 190701852 zeCommandListReset, 3976, 261265117, 2.37, 65710, 182, 2516131 zeCommandListCreateImmediate, 1, 189954019, 1.72, 189954019, 189954019, 189954019 zeModuleCreate, 2, 162170189, 1.47, 81085094, 8081706, 154088483 zeMemFree, 6, 130338449, 1.18, 21723074, 213200, 129109118 zeCommandQueueExecuteCommandLists, 3976, 92877003, 0.84, 23359, 1211, 3024913 zeCommandListAppendMemoryCopy, 1700, 47362492, 0.43, 27860, 15042, 946050 zeMemAllocDevice, 67, 43488320, 0.39, 649079, 97185, 15510137 zeKernelCreate, 22, 7986520, 0.07, 363023, 2088, 4065025 zeCommandListAppendMemoryFill, 503, 4381709, 0.04, 8711, 1004, 1694162 zeCommandQueueCreate, 2, 1350202, 0.01, 675101, 225169, 1125033 zeCommandListCreate, 3, 815519, 0.01, 271839, 142204, 360245 zeCommandListAppendLaunchKernel, 393, 729006, 0.01, 1854, 573, 19302 zeCommandListAppendBarrier, 1381, 419554, 0.00, 303, 193, 3665 zeMemGetAllocProperties, 1892, 378477, 0.00, 200, 62, 2390 zeEventHostReset, 3974, 340554, 0.00, 85, 44, 6242 zeCommandListClose, 3976, 281788, 0.00, 70, 30, 759 zeFenceQueryStatus, 3977, 280667, 0.00, 70, 40, 1319 zeMemAllocHost, 1, 236196, 0.00, 236196, 236196, 236196 zeKernelSetArgumentValue, 602, 137640, 0.00, 228, 40, 2542 zeKernelSuggestGroupSize, 393, 125753, 0.00, 319, 91, 2101 zeEventPoolCreate, 1, 78056, 0.00, 78056, 78056, 78056 zeFenceReset, 3976, 73421, 0.00, 18, 16, 110 zeKernelSetGroupSize, 393, 54161, 0.00, 137, 38, 357 zeDeviceGetCommandQueueGroupProperties, 60, 13418, 0.00, 223, 20, 1281 zeDeviceGetProperties, 18, 6354, 0.00, 353, 189, 1069 zeEventCreate, 3, 4290, 0.00, 1430, 385, 3395 zeKernelGetProperties, 22, 3189, 0.00, 144, 43, 286 zeContextCreate, 1, 3134, 0.00, 3134, 3134, 3134 zeKernelSetIndirectAccess, 22, 3084, 0.00, 140, 21, 1170 zeDeviceGetModuleProperties, 6, 1817, 0.00, 302, 84, 1118 zeDeviceGetSubDevices, 12, 1816, 0.00, 151, 28, 329 zeDriverGetExtensionProperties, 2, 1593, 0.00, 796, 316, 1277 zeFenceCreate, 3, 1033, 0.00, 344, 115, 524 zeDriverGetProperties, 1, 998, 0.00, 998, 998, 998 zeModuleGetProperties, 2, 876, 0.00, 438, 422, 454 zeDeviceGet, 2, 657, 0.00, 328, 252, 405 zeInit, 1, 562, 0.00, 562, 562, 562 zeDeviceGetComputeProperties, 1, 322, 0.00, 322, 322, 322 zeDriverGet, 2, 290, 0.00, 145, 107, 183 zeDriverGetApiVersion, 1, 223, 0.00, 223, 223, 223 zeDeviceGetImageProperties, 6, 202, 0.00, 33, 17, 102 On Mon, Mar 27, 2023 at 6:44 AM Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com<mailto:[email protected]>> wrote: Can you run onetrace profiler to see if the kernel execution times goes up compared to your run without taskset? module load tools/pti-gpu onetrace -d ./a.out if that does not show a difference look at host API overhead onterace -h ./a.out From: Charles Leggett <cgleggett(a)lbl.gov<mailto:[email protected]>> Date: Friday, March 24, 2023 at 19:59 To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com<mailto:[email protected]>> Cc: Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot It's only running a single binary on a single core. The script is setup to be able to launch multiple concurrent jobs on different cores, but I'm only using it on a single one right now. The cores I tried are 0, 1, 8, 10, 18, 51 and 200. All performed the same. On Fri, Mar 24, 2023 at 5:44 PM Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com<mailto:[email protected]>> wrote: How many binaries are you running with task set? If you run multiple binaries, how do you map them to different GPUs? What are the exact core numbers you are using (from sched_getcpu()) ? From: Charles Leggett <cgleggett(a)lbl.gov<mailto:[email protected]>> Date: Friday, March 24, 2023 at 19:19 To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com<mailto:[email protected]>> Cc: Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot It shows the core that taskset has selected. I'm assuming you don't mean doing this from inside the kokkos::parallel_for. On Fri, Mar 24, 2023 at 3:38 PM Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com<mailto:[email protected]>> wrote: In your program that is run under taskset add this: #include <sched.h> std::cout << “I am running on core: ” << sched_getcpu() << std::endl; What is the output you get? From: Charles Leggett <cgleggett(a)lbl.gov<mailto:[email protected]>> Date: Friday, March 24, 2023 at 17:21 To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com<mailto:[email protected]>> Cc: Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot There is no explicit use of MPI in user code. I have no idea what Kokkos is doing under the hood, but I thought it was using a SYCL/Level zero backend. On Fri, Mar 24, 2023 at 3:09 PM Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com<mailto:[email protected]>> wrote: Are you using MPI in this case? From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of Charles Leggett via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Date: Friday, March 24, 2023 at 12:50 To: Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot I have a script that launches jobs which employs `taskset -c` to lock an executable onto a specific core (this was done to test concurrent job execution). What's very odd is that using taskset slows down kernel execution on the GPU by about a factor of 600. eg: without taskset: kernel total /s avg launch /us std dev /us sim_clean 0.02091981 486.5 18.7 sim_A 0.10563853 2456.7 703.2 sim_ct 0.03402171 791.2 9.6 sim_cp 0.01612411 375.0 7.0 with taskset: kernel total /s avg launch /us std dev /us sim_clean 0.02155314 501.2 20.5 sim_A 60.56034054 1408380.0 398894.4 sim_ct 1.33204648 30977.8 50526.0 sim_cp 0.99585704 23159.5 13514.5 I dump the environment vars within the job, and they look identical. I tried a few different cores to set the cpu affinity, and it doesn't seem to make a difference (I didn't try all 208) I'm using oneapi/eng-compiler/2022.12.30.003 and kokkos/2022.12.30.003/eng-compiler/sycl_intel_aot I don't think I noticed this behavior with pure SYCL code, but now I'll have to go back and check. This was on node x1921c2s5b0n0 cheers, Charles. -- Charles Leggett <CGLeggett(a)lbl.gov<mailto:[email protected]>> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930 -- Charles Leggett <CGLeggett(a)lbl.gov<mailto:[email protected]>> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930 -- Charles Leggett <CGLeggett(a)lbl.gov<mailto:[email protected]>> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930 -- Charles Leggett <CGLeggett(a)lbl.gov<mailto:[email protected]>> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930 -- Charles Leggett <CGLeggett(a)lbl.gov<mailto:[email protected]>> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930

1 0

Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot
by Charles Leggett 27 Mar '23

27 Mar '23

according to onetrace, the kernel runtimes look about the same (my original timings were done by putting std::chrono calls around the kernel launches). The total API time for the L0 backend goes from 501942950 to 11021208546 when it’s done with taskset. here are the explicit numbers: no taskset: Function, Calls, Time (ns), Time (%), Average (ns), Min (ns), Max (ns) zeModuleCreate, 2, 169715504, 33.81, 84857752, 8910086, 160805418 zeCommandQueueSynchronize, 9154, 142616861, 28.41, 15579, 53, 1037213 zeCommandListReset, 3976, 86385717, 17.21, 21726, 183, 245071 zeCommandListAppendMemoryCopy, 1700, 44595645, 8.88, 26232, 18077, 172220 zeCommandQueueExecuteCommandLists, 3976, 25818462, 5.14, 6493, 1307, 2848223 zeMemAllocDevice, 67, 15173695, 3.02, 226473, 90346, 2874175 zeCommandListAppendMemoryFill, 503, 5027360, 1.00, 9994, 1520, 1823447 zeKernelCreate, 22, 1828342, 0.36, 83106, 2202, 828879 zeCommandListCreateImmediate, 1, 1803461, 0.36, 1803461, 1803461, 1803461 zeEventQueryStatus, 1381, 1714500, 0.34, 1241, 364, 98024 zeCommandQueueCreate, 2, 1408878, 0.28, 704439, 259941, 1148937 zeMemFree, 6, 1364497, 0.27, 227416, 184775, 293323 zeCommandListCreate, 3, 854943, 0.17, 284981, 168795, 417739 zeCommandListAppendBarrier, 1381, 616387, 0.12, 446, 210, 6665 zeCommandListAppendLaunchKernel, 393, 600600, 0.12, 1528, 667, 31174 zeFenceQueryStatus, 3977, 401428, 0.08, 100, 41, 3480 zeEventHostReset, 3974, 389108, 0.08, 97, 55, 3097 zeMemGetAllocProperties, 1892, 370859, 0.07, 196, 66, 3120 zeCommandListClose, 3976, 347435, 0.07, 87, 39, 1089 zeMemAllocHost, 1, 293942, 0.06, 293942, 293942, 293942 zeFenceReset, 3976, 147342, 0.03, 37, 16, 3051 zeKernelSetArgumentValue, 602, 139952, 0.03, 232, 48, 3794 zeKernelSuggestGroupSize, 393, 108755, 0.02, 276, 97, 3128 zeEventPoolCreate, 1, 104642, 0.02, 104642, 104642, 104642 zeKernelSetGroupSize, 393, 55013, 0.01, 139, 60, 523 zeDeviceGetCommandQueueGroupProperties, 60, 16436, 0.00, 273, 42, 1570 zeDeviceGetProperties, 18, 8115, 0.00, 450, 240, 1604 zeEventCreate, 3, 7557, 0.00, 2519, 735, 5934 zeContextCreate, 1, 5449, 0.00, 5449, 5449, 5449 zeKernelSetIndirectAccess, 22, 4009, 0.00, 182, 27, 1807 zeKernelGetProperties, 22, 3981, 0.00, 180, 48, 479 zeDeviceGetModuleProperties, 6, 2776, 0.00, 462, 121, 1977 zeDriverGetExtensionProperties, 2, 2318, 0.00, 1159, 584, 1734 zeFenceCreate, 3, 1882, 0.00, 627, 220, 996 zeDeviceGetSubDevices, 12, 1722, 0.00, 143, 46, 297 zeDriverGetProperties, 1, 1192, 0.00, 1192, 1192, 1192 zeModuleGetProperties, 2, 945, 0.00, 472, 461, 484 zeDeviceGetComputeProperties, 1, 775, 0.00, 775, 775, 775 zeInit, 1, 759, 0.00, 759, 759, 759 zeDeviceGet, 2, 731, 0.00, 365, 263, 468 zeDriverGet, 2, 371, 0.00, 185, 119, 252 zeDeviceGetImageProperties, 6, 317, 0.00, 52, 30, 150 zeDriverGetApiVersion, 1, 287, 0.00, 287, 287, 287 with taskset: Function, Calls, Time (ns), Time (%), Average (ns), Min (ns), Max (ns) zeCommandQueueSynchronize, 9154, 7637427277, 69.30, 834326, 43, 171775424 zeEventQueryStatus, 1381, 2438612599, 22.13, 1765830, 294, 190701852 zeCommandListReset, 3976, 261265117, 2.37, 65710, 182, 2516131 zeCommandListCreateImmediate, 1, 189954019, 1.72, 189954019, 189954019, 189954019 zeModuleCreate, 2, 162170189, 1.47, 81085094, 8081706, 154088483 zeMemFree, 6, 130338449, 1.18, 21723074, 213200, 129109118 zeCommandQueueExecuteCommandLists, 3976, 92877003, 0.84, 23359, 1211, 3024913 zeCommandListAppendMemoryCopy, 1700, 47362492, 0.43, 27860, 15042, 946050 zeMemAllocDevice, 67, 43488320, 0.39, 649079, 97185, 15510137 zeKernelCreate, 22, 7986520, 0.07, 363023, 2088, 4065025 zeCommandListAppendMemoryFill, 503, 4381709, 0.04, 8711, 1004, 1694162 zeCommandQueueCreate, 2, 1350202, 0.01, 675101, 225169, 1125033 zeCommandListCreate, 3, 815519, 0.01, 271839, 142204, 360245 zeCommandListAppendLaunchKernel, 393, 729006, 0.01, 1854, 573, 19302 zeCommandListAppendBarrier, 1381, 419554, 0.00, 303, 193, 3665 zeMemGetAllocProperties, 1892, 378477, 0.00, 200, 62, 2390 zeEventHostReset, 3974, 340554, 0.00, 85, 44, 6242 zeCommandListClose, 3976, 281788, 0.00, 70, 30, 759 zeFenceQueryStatus, 3977, 280667, 0.00, 70, 40, 1319 zeMemAllocHost, 1, 236196, 0.00, 236196, 236196, 236196 zeKernelSetArgumentValue, 602, 137640, 0.00, 228, 40, 2542 zeKernelSuggestGroupSize, 393, 125753, 0.00, 319, 91, 2101 zeEventPoolCreate, 1, 78056, 0.00, 78056, 78056, 78056 zeFenceReset, 3976, 73421, 0.00, 18, 16, 110 zeKernelSetGroupSize, 393, 54161, 0.00, 137, 38, 357 zeDeviceGetCommandQueueGroupProperties, 60, 13418, 0.00, 223, 20, 1281 zeDeviceGetProperties, 18, 6354, 0.00, 353, 189, 1069 zeEventCreate, 3, 4290, 0.00, 1430, 385, 3395 zeKernelGetProperties, 22, 3189, 0.00, 144, 43, 286 zeContextCreate, 1, 3134, 0.00, 3134, 3134, 3134 zeKernelSetIndirectAccess, 22, 3084, 0.00, 140, 21, 1170 zeDeviceGetModuleProperties, 6, 1817, 0.00, 302, 84, 1118 zeDeviceGetSubDevices, 12, 1816, 0.00, 151, 28, 329 zeDriverGetExtensionProperties, 2, 1593, 0.00, 796, 316, 1277 zeFenceCreate, 3, 1033, 0.00, 344, 115, 524 zeDriverGetProperties, 1, 998, 0.00, 998, 998, 998 zeModuleGetProperties, 2, 876, 0.00, 438, 422, 454 zeDeviceGet, 2, 657, 0.00, 328, 252, 405 zeInit, 1, 562, 0.00, 562, 562, 562 zeDeviceGetComputeProperties, 1, 322, 0.00, 322, 322, 322 zeDriverGet, 2, 290, 0.00, 145, 107, 183 zeDriverGetApiVersion, 1, 223, 0.00, 223, 223, 223 zeDeviceGetImageProperties, 6, 202, 0.00, 33, 17, 102 On Mon, Mar 27, 2023 at 6:44 AM Steinbrecher, Patrick < patrick.steinbrecher(a)intel.com> wrote: > Can you run onetrace profiler to see if the kernel execution times goes up > compared to your run without taskset? > > > > module load tools/pti-gpu > > > > onetrace -d ./a.out > > > > if that does not show a difference look at host API overhead > > > > onterace -h ./a.out > > > > *From: *Charles Leggett <cgleggett(a)lbl.gov> > *Date: *Friday, March 24, 2023 at 19:59 > *To: *Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com> > *Cc: *Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > *Subject: *Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on > sunspot > > > > It's only running a single binary on a single core. The script is setup to > be able to launch multiple concurrent jobs on different cores, but I'm only > using it on a single one right now. > > > > The cores I tried are 0, 1, 8, 10, 18, 51 and 200. All performed the same. > > > > > > > > On Fri, Mar 24, 2023 at 5:44 PM Steinbrecher, Patrick < > patrick.steinbrecher(a)intel.com> wrote: > > How many binaries are you running with task set? If you run multiple > binaries, how do you map them to different GPUs? > > What are the exact core numbers you are using (from sched_getcpu()) ? > > > > *From: *Charles Leggett <cgleggett(a)lbl.gov> > *Date: *Friday, March 24, 2023 at 19:19 > *To: *Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com> > *Cc: *Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > *Subject: *Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on > sunspot > > > > It shows the core that taskset has selected. > > > > I'm assuming you don't mean doing this from inside the > kokkos::parallel_for. > > > > On Fri, Mar 24, 2023 at 3:38 PM Steinbrecher, Patrick < > patrick.steinbrecher(a)intel.com> wrote: > > In your program that is run under taskset add this: > > > > #include <sched.h> > > > > std::cout << “I am running on core: ” << sched_getcpu() << std::endl; > > > > > > What is the output you get? > > > > *From: *Charles Leggett <cgleggett(a)lbl.gov> > *Date: *Friday, March 24, 2023 at 17:21 > *To: *Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com> > *Cc: *Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > *Subject: *Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on > sunspot > > > > There is no explicit use of MPI in user code. I have no idea what Kokkos > is doing under the hood, but I thought it was using a SYCL/Level zero > backend. > > > > On Fri, Mar 24, 2023 at 3:09 PM Steinbrecher, Patrick < > patrick.steinbrecher(a)intel.com> wrote: > > Are you using MPI in this case? > > > > *From: *Ecp-aurora-sdk-discuss < > ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Charles > Leggett via Ecp-aurora-sdk-discuss < > ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > *Date: *Friday, March 24, 2023 at 12:50 > *To: *Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > *Subject: *[Ecp-aurora-sdk-discuss] using taskset slows down GPU on > sunspot > > > > I have a script that launches jobs which employs `taskset -c` to lock an > executable onto a specific core (this was done to test concurrent job > execution). What's very odd is that using taskset slows down kernel > execution on the GPU by about a factor of 600. eg: > > > > without taskset: > > kernel total /s avg launch /us std dev /us > > sim_clean 0.02091981 486.5 18.7 > > sim_A 0.10563853 2456.7 703.2 > > sim_ct 0.03402171 791.2 9.6 > > sim_cp 0.01612411 375.0 7.0 > > > > with taskset: > > kernel total /s avg launch /us std dev /us > > sim_clean 0.02155314 501.2 20.5 > > sim_A 60.56034054 1408380.0 398894.4 > > sim_ct 1.33204648 30977.8 50526.0 > > sim_cp 0.99585704 23159.5 13514.5 > > > > I dump the environment vars within the job, and they look identical. > > > > I tried a few different cores to set the cpu affinity, and it doesn't seem > to make a difference (I didn't try all 208) > > > > I'm using oneapi/eng-compiler/2022.12.30.003 and > kokkos/2022.12.30.003/eng-compiler/sycl_intel_aot > > > > I don't think I noticed this behavior with pure SYCL code, but now I'll > have to go back and check. > > > > This was on node x1921c2s5b0n0 > > > > > > cheers, Charles. > > > -- > > Charles Leggett <CGLeggett(a)lbl.gov> > Lawrence Berkeley National Lab > 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 > > tel: 510.495.2930 > > > > -- > > Charles Leggett <CGLeggett(a)lbl.gov> > Lawrence Berkeley National Lab > 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 > > tel: 510.495.2930 > > > > -- > > Charles Leggett <CGLeggett(a)lbl.gov> > Lawrence Berkeley National Lab > 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 > > tel: 510.495.2930 > > > > -- > > Charles Leggett <CGLeggett(a)lbl.gov> > Lawrence Berkeley National Lab > 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 > > tel: 510.495.2930 > -- Charles Leggett <CGLeggett(a)lbl.gov> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930

1 0

Re: [Ecp-aurora-sdk-discuss] openjdk /soft/packaging/spack/settings/packages.yaml not installed
by Koyama, Sean 27 Mar '23

27 Mar '23

Hi Tim, Thank you for catching that - it appears to be an erroneous entry from Spack's external detection which we didn't catch. I've removed it from /soft/packaging/spack/settings/packages.yaml. Best, Sean ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Tim Haines via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Saturday, March 25, 2023 3:08 PM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] openjdk /soft/packaging/spack/settings/packages.yaml not installed On Sunspot, there is an entry in /soft/packaging/spack/settings/packages.yaml for openjdk, but it doesn't appear to be installed in the location specified. Is this an errant entry or is there supposed to be a system-provided jdk? openjdk: externals: - spec: openjdk(a)11.0.13_8-suse-3.65.1-x8664 prefix: /usr Thanks. - Tim -- Ecp-aurora-sdk-discuss mailing list Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot
by Steinbrecher, Patrick 27 Mar '23

27 Mar '23

Can you run onetrace profiler to see if the kernel execution times goes up compared to your run without taskset? module load tools/pti-gpu onetrace -d ./a.out if that does not show a difference look at host API overhead onterace -h ./a.out From: Charles Leggett <cgleggett(a)lbl.gov> Date: Friday, March 24, 2023 at 19:59 To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com> Cc: Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot It's only running a single binary on a single core. The script is setup to be able to launch multiple concurrent jobs on different cores, but I'm only using it on a single one right now. The cores I tried are 0, 1, 8, 10, 18, 51 and 200. All performed the same. On Fri, Mar 24, 2023 at 5:44 PM Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com<mailto:[email protected]>> wrote: How many binaries are you running with task set? If you run multiple binaries, how do you map them to different GPUs? What are the exact core numbers you are using (from sched_getcpu()) ? From: Charles Leggett <cgleggett(a)lbl.gov<mailto:[email protected]>> Date: Friday, March 24, 2023 at 19:19 To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com<mailto:[email protected]>> Cc: Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot It shows the core that taskset has selected. I'm assuming you don't mean doing this from inside the kokkos::parallel_for. On Fri, Mar 24, 2023 at 3:38 PM Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com<mailto:[email protected]>> wrote: In your program that is run under taskset add this: #include <sched.h> std::cout << “I am running on core: ” << sched_getcpu() << std::endl; What is the output you get? From: Charles Leggett <cgleggett(a)lbl.gov<mailto:[email protected]>> Date: Friday, March 24, 2023 at 17:21 To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com<mailto:[email protected]>> Cc: Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot There is no explicit use of MPI in user code. I have no idea what Kokkos is doing under the hood, but I thought it was using a SYCL/Level zero backend. On Fri, Mar 24, 2023 at 3:09 PM Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com<mailto:[email protected]>> wrote: Are you using MPI in this case? From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of Charles Leggett via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Date: Friday, March 24, 2023 at 12:50 To: Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot I have a script that launches jobs which employs `taskset -c` to lock an executable onto a specific core (this was done to test concurrent job execution). What's very odd is that using taskset slows down kernel execution on the GPU by about a factor of 600. eg: without taskset: kernel total /s avg launch /us std dev /us sim_clean 0.02091981 486.5 18.7 sim_A 0.10563853 2456.7 703.2 sim_ct 0.03402171 791.2 9.6 sim_cp 0.01612411 375.0 7.0 with taskset: kernel total /s avg launch /us std dev /us sim_clean 0.02155314 501.2 20.5 sim_A 60.56034054 1408380.0 398894.4 sim_ct 1.33204648 30977.8 50526.0 sim_cp 0.99585704 23159.5 13514.5 I dump the environment vars within the job, and they look identical. I tried a few different cores to set the cpu affinity, and it doesn't seem to make a difference (I didn't try all 208) I'm using oneapi/eng-compiler/2022.12.30.003 and kokkos/2022.12.30.003/eng-compiler/sycl_intel_aot I don't think I noticed this behavior with pure SYCL code, but now I'll have to go back and check. This was on node x1921c2s5b0n0 cheers, Charles. -- Charles Leggett <CGLeggett(a)lbl.gov<mailto:[email protected]>> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930 -- Charles Leggett <CGLeggett(a)lbl.gov<mailto:[email protected]>> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930 -- Charles Leggett <CGLeggett(a)lbl.gov<mailto:[email protected]>> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930 -- Charles Leggett <CGLeggett(a)lbl.gov<mailto:[email protected]>> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930

1 0

openjdk /soft/packaging/spack/settings/packages.yaml not installed
by Tim Haines 25 Mar '23

25 Mar '23

On Sunspot, there is an entry in /soft/packaging/spack/settings/packages.yaml for openjdk, but it doesn't appear to be installed in the location specified. Is this an errant entry or is there supposed to be a system-provided jdk? openjdk: externals: - spec: openjdk(a)11.0.13_8-suse-3.65.1-x8664 prefix: /usr Thanks. - Tim

1 0

Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot
by Charles Leggett 24 Mar '23

24 Mar '23

It's only running a single binary on a single core. The script is setup to be able to launch multiple concurrent jobs on different cores, but I'm only using it on a single one right now. The cores I tried are 0, 1, 8, 10, 18, 51 and 200. All performed the same. On Fri, Mar 24, 2023 at 5:44 PM Steinbrecher, Patrick < patrick.steinbrecher(a)intel.com> wrote: > How many binaries are you running with task set? If you run multiple > binaries, how do you map them to different GPUs? > > What are the exact core numbers you are using (from sched_getcpu()) ? > > > > *From: *Charles Leggett <cgleggett(a)lbl.gov> > *Date: *Friday, March 24, 2023 at 19:19 > *To: *Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com> > *Cc: *Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > *Subject: *Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on > sunspot > > > > It shows the core that taskset has selected. > > > > I'm assuming you don't mean doing this from inside the > kokkos::parallel_for. > > > > On Fri, Mar 24, 2023 at 3:38 PM Steinbrecher, Patrick < > patrick.steinbrecher(a)intel.com> wrote: > > In your program that is run under taskset add this: > > > > #include <sched.h> > > > > std::cout << “I am running on core: ” << sched_getcpu() << std::endl; > > > > > > What is the output you get? > > > > *From: *Charles Leggett <cgleggett(a)lbl.gov> > *Date: *Friday, March 24, 2023 at 17:21 > *To: *Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com> > *Cc: *Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > *Subject: *Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on > sunspot > > > > There is no explicit use of MPI in user code. I have no idea what Kokkos > is doing under the hood, but I thought it was using a SYCL/Level zero > backend. > > > > On Fri, Mar 24, 2023 at 3:09 PM Steinbrecher, Patrick < > patrick.steinbrecher(a)intel.com> wrote: > > Are you using MPI in this case? > > > > *From: *Ecp-aurora-sdk-discuss < > ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Charles > Leggett via Ecp-aurora-sdk-discuss < > ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > *Date: *Friday, March 24, 2023 at 12:50 > *To: *Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > *Subject: *[Ecp-aurora-sdk-discuss] using taskset slows down GPU on > sunspot > > > > I have a script that launches jobs which employs `taskset -c` to lock an > executable onto a specific core (this was done to test concurrent job > execution). What's very odd is that using taskset slows down kernel > execution on the GPU by about a factor of 600. eg: > > > > without taskset: > > kernel total /s avg launch /us std dev /us > > sim_clean 0.02091981 486.5 18.7 > > sim_A 0.10563853 2456.7 703.2 > > sim_ct 0.03402171 791.2 9.6 > > sim_cp 0.01612411 375.0 7.0 > > > > with taskset: > > kernel total /s avg launch /us std dev /us > > sim_clean 0.02155314 501.2 20.5 > > sim_A 60.56034054 1408380.0 398894.4 > > sim_ct 1.33204648 30977.8 50526.0 > > sim_cp 0.99585704 23159.5 13514.5 > > > > I dump the environment vars within the job, and they look identical. > > > > I tried a few different cores to set the cpu affinity, and it doesn't seem > to make a difference (I didn't try all 208) > > > > I'm using oneapi/eng-compiler/2022.12.30.003 and > kokkos/2022.12.30.003/eng-compiler/sycl_intel_aot > > > > I don't think I noticed this behavior with pure SYCL code, but now I'll > have to go back and check. > > > > This was on node x1921c2s5b0n0 > > > > > > cheers, Charles. > > > -- > > Charles Leggett <CGLeggett(a)lbl.gov> > Lawrence Berkeley National Lab > 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 > > tel: 510.495.2930 > > > > -- > > Charles Leggett <CGLeggett(a)lbl.gov> > Lawrence Berkeley National Lab > 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 > > tel: 510.495.2930 > > > > -- > > Charles Leggett <CGLeggett(a)lbl.gov> > Lawrence Berkeley National Lab > 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 > > tel: 510.495.2930 > -- Charles Leggett <CGLeggett(a)lbl.gov> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930

1 0

Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot
by Steinbrecher, Patrick 24 Mar '23

24 Mar '23

How many binaries are you running with task set? If you run multiple binaries, how do you map them to different GPUs? What are the exact core numbers you are using (from sched_getcpu()) ? From: Charles Leggett <cgleggett(a)lbl.gov> Date: Friday, March 24, 2023 at 19:19 To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com> Cc: Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot It shows the core that taskset has selected. I'm assuming you don't mean doing this from inside the kokkos::parallel_for. On Fri, Mar 24, 2023 at 3:38 PM Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com<mailto:[email protected]>> wrote: In your program that is run under taskset add this: #include <sched.h> std::cout << “I am running on core: ” << sched_getcpu() << std::endl; What is the output you get? From: Charles Leggett <cgleggett(a)lbl.gov<mailto:[email protected]>> Date: Friday, March 24, 2023 at 17:21 To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com<mailto:[email protected]>> Cc: Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: Re: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot There is no explicit use of MPI in user code. I have no idea what Kokkos is doing under the hood, but I thought it was using a SYCL/Level zero backend. On Fri, Mar 24, 2023 at 3:09 PM Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com<mailto:[email protected]>> wrote: Are you using MPI in this case? From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of Charles Leggett via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Date: Friday, March 24, 2023 at 12:50 To: Aurora Discussion List <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] using taskset slows down GPU on sunspot I have a script that launches jobs which employs `taskset -c` to lock an executable onto a specific core (this was done to test concurrent job execution). What's very odd is that using taskset slows down kernel execution on the GPU by about a factor of 600. eg: without taskset: kernel total /s avg launch /us std dev /us sim_clean 0.02091981 486.5 18.7 sim_A 0.10563853 2456.7 703.2 sim_ct 0.03402171 791.2 9.6 sim_cp 0.01612411 375.0 7.0 with taskset: kernel total /s avg launch /us std dev /us sim_clean 0.02155314 501.2 20.5 sim_A 60.56034054 1408380.0 398894.4 sim_ct 1.33204648 30977.8 50526.0 sim_cp 0.99585704 23159.5 13514.5 I dump the environment vars within the job, and they look identical. I tried a few different cores to set the cpu affinity, and it doesn't seem to make a difference (I didn't try all 208) I'm using oneapi/eng-compiler/2022.12.30.003 and kokkos/2022.12.30.003/eng-compiler/sycl_intel_aot I don't think I noticed this behavior with pure SYCL code, but now I'll have to go back and check. This was on node x1921c2s5b0n0 cheers, Charles. -- Charles Leggett <CGLeggett(a)lbl.gov<mailto:[email protected]>> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930 -- Charles Leggett <CGLeggett(a)lbl.gov<mailto:[email protected]>> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930 -- Charles Leggett <CGLeggett(a)lbl.gov<mailto:[email protected]>> Lawrence Berkeley National Lab 1 Cyclotron Road, MS 50F-1650, Berkeley, CA 94720 tel: 510.495.2930

1 0