February 2017 - Intel-nda - mailman3.cels.anl.gov

Re: [JLSE-Intel-NDA] Measuring throughput on KNL
by Stevens, Rick L. 24 Feb '17

24 Feb '17

On Feb 24, 2017, at 12:59, Fischer, Paul F. <fischer(a)mcs.anl.gov<mailto:[email protected]>> wrote: Thanks Ramesh. (Are there any applications where time-to-solution is not important ?) those where throughput might be more important — total volume of number of jobs rather than latency on a single job :) design parameter searching for example Paul ________________________________ From: Balakrishnan, Ramesh Sent: Friday, February 24, 2017 12:32 PM To: Fischer, Paul F. Cc: Morozov, Vitali A.; Hammond, Jeff R; intel-nda(a)jlse.anl.gov<mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] Measuring throughput on KNL Its an interesting question, Paul. For strong scaling problems with engineering applications, where time to solution is important, the n_1/2 criterion would be an important measure. Ramesh On Feb 24, 2017, at 11:31 AM, Fischer, Paul F. <fischer(a)mcs.anl.gov<mailto:[email protected]>> wrote: Does anyone know what the n_1/2 is for DGEMM on KNL ? ________________________________ From: intel-nda-bounces(a)lists.jlse.anl.gov<mailto:[email protected]> [intel-nda-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>] on behalf of Vitali A. Morozov [morozov(a)anl.gov<mailto:[email protected]>] Sent: Friday, February 24, 2017 11:27 AM To: Hammond, Jeff R Cc: intel-nda(a)jlse.anl.gov<mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] Measuring throughput on KNL This is correct - the original code is written with asm, 4 bugs were reported to an Intel compiler team, they all fixed in version 17. Right now, the version 17 must be compiling the HACCmk to an equivalent object code as an asm version. If not, there must be another bug reported. :) Vitali p.s. Calling DGEMM will probably be not a compiler-generated code path as well. On 02/24/2017 11:03 AM, Hammond, Jeff R wrote: Vitali: HACCmk uses asm or intrinsics to get 82%, does it not? If neither, that is great. Aaron: Call DGEMM :-) Jeff Sent from my iPhone On Feb 24, 2017, at 7:43 AM, Vitali A. Morozov <morozov(a)anl.gov<mailto:[email protected]>> wrote: Yes, it is possible. I was doing this for HACCmk and reached 82% of peak. Vitali On 02/24/2017 09:37 AM, Aaron Knoll wrote: Along these lines: Is it possible to write some very simple compute-only code (in either C++ or Fortran) that when compiled would achieve peak flops (or close to it) on KNL, and measure actual throughput? @Jeff what would you recommend for this? -Aaron On Thu, Feb 23, 2017 at 5:12 PM, Scott Parker <sparker(a)anl.gov<mailto:[email protected]>> wrote: You should use /soft/compilers/intel/advisor -Scott On 2/23/17 5:59 PM, Banerjee,Tania wrote: Hi All, Thanks for pitching in. Scott: Which advisor should we use in /soft/compilers/intel ? [tania@jlselogin1 intel]$ ls advisor advisor/ advisor_xe/ advisor_2017/ advisor_xe_2016/ advisor_2017.1.0.477503/ advisor_xe_2016.1.30.450722/ advisor_2017.1.1.486553/ advisor_xe_2016.1.40.463413/ Also about Intel VTunes, I could get only the following summary (using both hpc-performance and memory-access collection modes) , which did not feature the events talked about in this thread: Event summary ------------- Hardware Event Type Hardware Event Count:Self Hardware Event Sample Count:Self Events Per Sample --------------------------------- ------------------------- -------------------------------- ----------------- INST_RETIRED.ANY 1080573000000 83121 13000000 CPU_CLK_UNHALTED.THREAD 2283515000000 175655 13000000 CPU_CLK_UNHALTED.REF_TSC 2120508000000 163116 13000000 MEM_UOPS_RETIRED.L2_MISS_LOADS_PS 396011880 132 1000030 UOPS_RETIRED.ALL 1479002218500 24650 20000030 MEM_UOPS_RETIRED.L2_HIT_LOADS_PS 2079062370<tel:%28207%29%20906-2370> 693 1000030 NO_ALLOC_CYCLES.MISPREDICTS 11220016830 187 20000030 NO_ALLOC_CYCLES.NOT_DELIVERED 720601080900 12010 20000030 Uncore Event summary -------------------- Uncore Event Type Uncore Event Count:Self ... But it could be because I did not enable -avx during compilation, hence no there's no vectorization, whatsoever. I will check this aspect as the other part of my project on performance tuning. Thanks a lot, Tania ________________________________ From: Vitali A. Morozov <morozov(a)anl.gov><mailto:[email protected]> Sent: Wednesday, February 22, 2017 6:08 PM To: Hammond, Jeff R; Scott Parker; Banerjee,Tania; intel-nda(a)jlse.anl.gov<mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] Measuring throughput on KNL In general, scalar code generation may differ from vector code generation - conditionals area perfect example of it. Also, how to account an FMA versus a non-FMA instruction? Vitali On 02/22/2017 04:54 PM, Hammond, Jeff R wrote: I agree that PACKED_SIMD and SCALAR_SIMD include non-float operations. My point was that if the problem is SIMD float ops overcount, then forcing the compiler to turn them all into scalar float ops should prevent that. I assume there is a way to count scalar float ops, but if not, then I guess this doesn't help. At least for SSE*/AVX/AVX2 code, another way to count flops precisely is to use an implementation where the hardware supports precise counters. https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX2 Jeff From: Scott Parker <sparker(a)anl.gov><mailto:[email protected]> Date: Wednesday, February 22, 2017 at 12:32 PM To: "Hammond, Jeff R" <jeff.r.hammond(a)intel.com><mailto:[email protected]>, "Banerjee,Tania" <tmishra(a)ufl.edu><mailto:[email protected]>, "intel-nda(a)jlse.anl.gov"<mailto:[email protected]> <intel-nda(a)jlse.anl.gov><mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] Measuring throughput on KNL I don't think that will get you an accurate flop count either on KNL, though it may be closer. I'm not sure of the full set of instructions that map into the PACKED_SIMD and SCALAR_SIMD counters but I'm pretty sure there are some that are not arithmatic or floating point. From past discussions with Dmitry Prohorov from VTune the Intel Advisor approach is the only way on KNL -Scott On 2/22/17 11:54 AM, Hammond, Jeff R wrote: If one only wants to count flops, I assume disabling SIMD will obviate the overcounting due to masking and such, but I have not experimented with this. -[no-]simd enables(DEFAULT)/disables vectorization using simd pragma Jeff From: <intel-nda-bounces(a)lists.jlse.anl.gov><mailto:[email protected]> on behalf of Scott Parker <sparker(a)anl.gov><mailto:[email protected]> Date: Wednesday, February 22, 2017 at 9:22 AM To: "Banerjee,Tania" <tmishra(a)ufl.edu><mailto:[email protected]>, "intel-nda(a)jlse.anl.gov"<mailto:[email protected]> <intel-nda(a)jlse.anl.gov><mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] Measuring throughput on KNL KNL doesn't have hardware counters for flops, which is why PAPI doesn't provide PAPI_FP_OPS. This is a problem on a number of current Intel platforms. From Vtune you can get a upper bound flops estimate based on the UOPS_RETIRED.PACKED_SIMD and UOPS_RETIRED.SCALAR_SIMD counters. This assumes things like the vector length of instructions is always 8, that FMA are used, and there is no masking, which is never going to be always true and therefore this can only be an upper bound. From Intel Advisor you can get an accurate FLOP count, but this is done via analysis and instrumentation of the binary so it will slow the code execution down noticeably. Further description is given here https://software.intel.com/en-us/articles/intel-advisor-flops -Scott On 2/21/17 8:33 PM, Banerjee,Tania wrote: Hi, How to measure throughput (GFLOPs/second) of a program on KNL? Normally I use PAPI to count the floating point operations (PAPI_FP_OPS / PAPI_FP_INS) and divide that by the time taken. But papi_avail reveals that these events are not supported on KNL. The output of /soft/perftools/tau/papi-knl/bin/papi_avail is given below. Thanks much, Tania ============================================================================= Available PAPI preset and user defined events plus hardware information. -------------------------------------------------------------------------------- PAPI Version : 5.4.3.0 Vendor string and code : GenuineIntel (1) Model string and code : Intel(R) Xeon Phi(TM) CPU 7210 @ 1.30GHz (87) CPU Revision : 1.000000 CPUID Info : Family: 6 Model: 87 Stepping: 1 CPU Max Megahertz : 1500 CPU Min Megahertz : 1000 Hdw Threads per core : 4 Cores per Socket : 64 Sockets : 1 NUMA Nodes : 1 CPUs per Node : 256 Total CPUs : 256 Running in a VM : no Number Hardware Counters : 5 Max Multiplex Counters : 192 -------------------------------------------------------------------------------- ================================================================================ PAPI Preset Events ================================================================================ Name Code Avail Deriv Description (Note) PAPI_L1_DCM 0x80000000 Yes No Level 1 data cache misses PAPI_L1_ICM 0x80000001 Yes No Level 1 instruction cache misses PAPI_L2_DCM 0x80000002 No No Level 2 data cache misses PAPI_L2_ICM 0x80000003 No No Level 2 instruction cache misses PAPI_L3_DCM 0x80000004 No No Level 3 data cache misses PAPI_L3_ICM 0x80000005 No No Level 3 instruction cache misses PAPI_L1_TCM 0x80000006 Yes Yes Level 1 cache misses PAPI_L2_TCM 0x80000007 Yes No Level 2 cache misses PAPI_L3_TCM 0x80000008 No No Level 3 cache misses PAPI_CA_SNP 0x80000009 No No Requests for a snoop PAPI_CA_SHR 0x8000000a No No Requests for exclusive access to shared cache line PAPI_CA_CLN 0x8000000b No No Requests for exclusive access to clean cache line PAPI_CA_INV 0x8000000c No No Requests for cache line invalidation PAPI_CA_ITV 0x8000000d No No Requests for cache line intervention PAPI_L3_LDM 0x8000000e No No Level 3 load misses PAPI_L3_STM 0x8000000f No No Level 3 store misses PAPI_BRU_IDL 0x80000010 No No Cycles branch units are idle PAPI_FXU_IDL 0x80000011 No No Cycles integer units are idle PAPI_FPU_IDL 0x80000012 No No Cycles floating point units are idle PAPI_LSU_IDL 0x80000013 No No Cycles load/store units are idle PAPI_TLB_DM 0x80000014 Yes No Data translation lookaside buffer misses PAPI_TLB_IM 0x80000015 No No Instruction translation lookaside buffer misses PAPI_TLB_TL 0x80000016 No No Total translation lookaside buffer misses PAPI_L1_LDM 0x80000017 Yes No Level 1 load misses PAPI_L1_STM 0x80000018 No No Level 1 store misses PAPI_L2_LDM 0x80000019 Yes No Level 2 load misses PAPI_L2_STM 0x8000001a No No Level 2 store misses PAPI_BTAC_M 0x8000001b No No Branch target address cache misses PAPI_PRF_DM 0x8000001c No No Data prefetch cache misses PAPI_L3_DCH 0x8000001d No No Level 3 data cache hits PAPI_TLB_SD 0x8000001e No No Translation lookaside buffer shootdowns PAPI_CSR_FAL 0x8000001f No No Failed store conditional instructions PAPI_CSR_SUC 0x80000020 No No Successful store conditional instructions PAPI_CSR_TOT 0x80000021 No No Total store conditional instructions PAPI_MEM_SCY 0x80000022 No No Cycles Stalled Waiting for memory accesses PAPI_MEM_RCY 0x80000023 No No Cycles Stalled Waiting for memory Reads PAPI_MEM_WCY 0x80000024 No No Cycles Stalled Waiting for memory writes PAPI_STL_ICY 0x80000025 Yes No Cycles with no instruction issue PAPI_FUL_ICY 0x80000026 No No Cycles with maximum instruction issue PAPI_STL_CCY 0x80000027 No No Cycles with no instructions completed PAPI_FUL_CCY 0x80000028 No No Cycles with maximum instructions completed PAPI_HW_INT 0x80000029 No No Hardware interrupts PAPI_BR_UCN 0x8000002a Yes Yes Unconditional branch instructions PAPI_BR_CN 0x8000002b Yes No Conditional branch instructions PAPI_BR_TKN 0x8000002c Yes No Conditional branch instructions taken PAPI_BR_NTK 0x8000002d Yes Yes Conditional branch instructions not taken PAPI_BR_MSP 0x8000002e Yes No Conditional branch instructions mispredicted PAPI_BR_PRC 0x8000002f No No Conditional branch instructions correctly predicted PAPI_FMA_INS 0x80000030 No No FMA instructions completed PAPI_TOT_IIS 0x80000031 No No Instructions issued PAPI_TOT_INS 0x80000032 Yes No Instructions completed PAPI_INT_INS 0x80000033 No No Integer instructions PAPI_FP_INS 0x80000034 No No Floating point instructions PAPI_LD_INS 0x80000035 Yes No Load instructions PAPI_SR_INS 0x80000036 No No Store instructions PAPI_BR_INS 0x80000037 Yes No Branch instructions PAPI_VEC_INS 0x80000038 No No Vector/SIMD instructions (could include integer) PAPI_RES_STL 0x80000039 Yes No Cycles stalled on any resource PAPI_FP_STAL 0x8000003a No No Cycles the FP unit(s) are stalled PAPI_TOT_CYC 0x8000003b Yes No Total cycles PAPI_LST_INS 0x8000003c Yes Yes Load/store instructions completed PAPI_SYC_INS 0x8000003d No No Synchronization instructions completed PAPI_L1_DCH 0x8000003e No No Level 1 data cache hits PAPI_L2_DCH 0x8000003f No No Level 2 data cache hits PAPI_L1_DCA 0x80000040 Yes Yes Level 1 data cache accesses PAPI_L2_DCA 0x80000041 No No Level 2 data cache accesses PAPI_L3_DCA 0x80000042 No No Level 3 data cache accesses PAPI_L1_DCR 0x80000043 No No Level 1 data cache reads PAPI_L2_DCR 0x80000044 No No Level 2 data cache reads PAPI_L3_DCR 0x80000045 No No Level 3 data cache reads PAPI_L1_DCW 0x80000046 No No Level 1 data cache writes PAPI_L2_DCW 0x80000047 No No Level 2 data cache writes PAPI_L3_DCW 0x80000048 No No Level 3 data cache writes PAPI_L1_ICH 0x80000049 Yes No Level 1 instruction cache hits PAPI_L2_ICH 0x8000004a No No Level 2 instruction cache hits PAPI_L3_ICH 0x8000004b No No Level 3 instruction cache hits PAPI_L1_ICA 0x8000004c Yes No Level 1 instruction cache accesses PAPI_L2_ICA 0x8000004d No No Level 2 instruction cache accesses PAPI_L3_ICA 0x8000004e No No Level 3 instruction cache accesses PAPI_L1_ICR 0x8000004f No No Level 1 instruction cache reads PAPI_L2_ICR 0x80000050 No No Level 2 instruction cache reads PAPI_L3_ICR 0x80000051 No No Level 3 instruction cache reads PAPI_L1_ICW 0x80000052 No No Level 1 instruction cache writes PAPI_L2_ICW 0x80000053 No No Level 2 instruction cache writes PAPI_L3_ICW 0x80000054 No No Level 3 instruction cache writes PAPI_L1_TCH 0x80000055 No No Level 1 total cache hits PAPI_L2_TCH 0x80000056 Yes Yes Level 2 total cache hits PAPI_L3_TCH 0x80000057 No No Level 3 total cache hits PAPI_L1_TCA 0x80000058 No No Level 1 total cache accesses PAPI_L2_TCA 0x80000059 Yes No Level 2 total cache accesses PAPI_L3_TCA 0x8000005a No No Level 3 total cache accesses PAPI_L1_TCR 0x8000005b No No Level 1 total cache reads PAPI_L2_TCR 0x8000005c No No Level 2 total cache reads PAPI_L3_TCR 0x8000005d No No Level 3 total cache reads PAPI_L1_TCW 0x8000005e No No Level 1 total cache writes PAPI_L2_TCW 0x8000005f No No Level 2 total cache writes PAPI_L3_TCW 0x80000060 No No Level 3 total cache writes PAPI_FML_INS 0x80000061 No No Floating point multiply instructions PAPI_FAD_INS 0x80000062 No No Floating point add instructions PAPI_FDV_INS 0x80000063 No No Floating point divide instructions PAPI_FSQ_INS 0x80000064 No No Floating point square root instructions PAPI_FNV_INS 0x80000065 No No Floating point inverse instructions PAPI_FP_OPS 0x80000066 No No Floating point operations PAPI_SP_OPS 0x80000067 No No Floating point operations; optimized to count scaled single precision vector operations PAPI_DP_OPS 0x80000068 No No Floating point operations; optimized to count scaled double precision vector operations PAPI_VEC_SP 0x80000069 No No Single precision vector/SIMD instructions PAPI_VEC_DP 0x8000006a No No Double precision vector/SIMD instructions PAPI_REF_CYC 0x8000006b Yes No Reference clock cycles -------------------------------------------------------------------------------- Of 108 possible events, 25 are available, of which 6 are derived. avail.c PASSED _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Measuring throughput on KNL
by Fischer, Paul F. 24 Feb '17

24 Feb '17

Thanks Ramesh. (Are there any applications where time-to-solution is not important ?) Paul ________________________________ From: Balakrishnan, Ramesh Sent: Friday, February 24, 2017 12:32 PM To: Fischer, Paul F. Cc: Morozov, Vitali A.; Hammond, Jeff R; intel-nda(a)jlse.anl.gov Subject: Re: [JLSE-Intel-NDA] Measuring throughput on KNL Its an interesting question, Paul. For strong scaling problems with engineering applications, where time to solution is important, the n_1/2 criterion would be an important measure. Ramesh On Feb 24, 2017, at 11:31 AM, Fischer, Paul F. <fischer(a)mcs.anl.gov<mailto:[email protected]>> wrote: Does anyone know what the n_1/2 is for DGEMM on KNL ? ________________________________ From: intel-nda-bounces(a)lists.jlse.anl.gov<mailto:[email protected]> [intel-nda-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>] on behalf of Vitali A. Morozov [morozov(a)anl.gov<mailto:[email protected]>] Sent: Friday, February 24, 2017 11:27 AM To: Hammond, Jeff R Cc: intel-nda(a)jlse.anl.gov<mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] Measuring throughput on KNL This is correct - the original code is written with asm, 4 bugs were reported to an Intel compiler team, they all fixed in version 17. Right now, the version 17 must be compiling the HACCmk to an equivalent object code as an asm version. If not, there must be another bug reported. :) Vitali p.s. Calling DGEMM will probably be not a compiler-generated code path as well. On 02/24/2017 11:03 AM, Hammond, Jeff R wrote: Vitali: HACCmk uses asm or intrinsics to get 82%, does it not? If neither, that is great. Aaron: Call DGEMM :-) Jeff Sent from my iPhone On Feb 24, 2017, at 7:43 AM, Vitali A. Morozov <morozov(a)anl.gov<mailto:[email protected]>> wrote: Yes, it is possible. I was doing this for HACCmk and reached 82% of peak. Vitali On 02/24/2017 09:37 AM, Aaron Knoll wrote: Along these lines: Is it possible to write some very simple compute-only code (in either C++ or Fortran) that when compiled would achieve peak flops (or close to it) on KNL, and measure actual throughput? @Jeff what would you recommend for this? -Aaron On Thu, Feb 23, 2017 at 5:12 PM, Scott Parker <sparker(a)anl.gov<mailto:[email protected]>> wrote: You should use /soft/compilers/intel/advisor -Scott On 2/23/17 5:59 PM, Banerjee,Tania wrote: Hi All, Thanks for pitching in. Scott: Which advisor should we use in /soft/compilers/intel ? [tania@jlselogin1 intel]$ ls advisor advisor/ advisor_xe/ advisor_2017/ advisor_xe_2016/ advisor_2017.1.0.477503/ advisor_xe_2016.1.30.450722/ advisor_2017.1.1.486553/ advisor_xe_2016.1.40.463413/ Also about Intel VTunes, I could get only the following summary (using both hpc-performance and memory-access collection modes) , which did not feature the events talked about in this thread: Event summary ------------- Hardware Event Type Hardware Event Count:Self Hardware Event Sample Count:Self Events Per Sample --------------------------------- ------------------------- -------------------------------- ----------------- INST_RETIRED.ANY 1080573000000 83121 13000000 CPU_CLK_UNHALTED.THREAD 2283515000000 175655 13000000 CPU_CLK_UNHALTED.REF_TSC 2120508000000 163116 13000000 MEM_UOPS_RETIRED.L2_MISS_LOADS_PS 396011880 132 1000030 UOPS_RETIRED.ALL 1479002218500 24650 20000030 MEM_UOPS_RETIRED.L2_HIT_LOADS_PS 2079062370<tel:%28207%29%20906-2370> 693 1000030 NO_ALLOC_CYCLES.MISPREDICTS 11220016830 187 20000030 NO_ALLOC_CYCLES.NOT_DELIVERED 720601080900 12010 20000030 Uncore Event summary -------------------- Uncore Event Type Uncore Event Count:Self ... But it could be because I did not enable -avx during compilation, hence no there's no vectorization, whatsoever. I will check this aspect as the other part of my project on performance tuning. Thanks a lot, Tania ________________________________ From: Vitali A. Morozov <morozov(a)anl.gov><mailto:[email protected]> Sent: Wednesday, February 22, 2017 6:08 PM To: Hammond, Jeff R; Scott Parker; Banerjee,Tania; intel-nda(a)jlse.anl.gov<mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] Measuring throughput on KNL In general, scalar code generation may differ from vector code generation - conditionals area perfect example of it. Also, how to account an FMA versus a non-FMA instruction? Vitali On 02/22/2017 04:54 PM, Hammond, Jeff R wrote: I agree that PACKED_SIMD and SCALAR_SIMD include non-float operations. My point was that if the problem is SIMD float ops overcount, then forcing the compiler to turn them all into scalar float ops should prevent that. I assume there is a way to count scalar float ops, but if not, then I guess this doesn't help. At least for SSE*/AVX/AVX2 code, another way to count flops precisely is to use an implementation where the hardware supports precise counters. https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX2 Jeff From: Scott Parker <sparker(a)anl.gov><mailto:[email protected]> Date: Wednesday, February 22, 2017 at 12:32 PM To: "Hammond, Jeff R" <jeff.r.hammond(a)intel.com><mailto:[email protected]>, "Banerjee,Tania" <tmishra(a)ufl.edu><mailto:[email protected]>, "intel-nda(a)jlse.anl.gov"<mailto:[email protected]> <intel-nda(a)jlse.anl.gov><mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] Measuring throughput on KNL I don't think that will get you an accurate flop count either on KNL, though it may be closer. I'm not sure of the full set of instructions that map into the PACKED_SIMD and SCALAR_SIMD counters but I'm pretty sure there are some that are not arithmatic or floating point. From past discussions with Dmitry Prohorov from VTune the Intel Advisor approach is the only way on KNL -Scott On 2/22/17 11:54 AM, Hammond, Jeff R wrote: If one only wants to count flops, I assume disabling SIMD will obviate the overcounting due to masking and such, but I have not experimented with this. -[no-]simd enables(DEFAULT)/disables vectorization using simd pragma Jeff From: <intel-nda-bounces(a)lists.jlse.anl.gov><mailto:[email protected]> on behalf of Scott Parker <sparker(a)anl.gov><mailto:[email protected]> Date: Wednesday, February 22, 2017 at 9:22 AM To: "Banerjee,Tania" <tmishra(a)ufl.edu><mailto:[email protected]>, "intel-nda(a)jlse.anl.gov"<mailto:[email protected]> <intel-nda(a)jlse.anl.gov><mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] Measuring throughput on KNL KNL doesn't have hardware counters for flops, which is why PAPI doesn't provide PAPI_FP_OPS. This is a problem on a number of current Intel platforms. From Vtune you can get a upper bound flops estimate based on the UOPS_RETIRED.PACKED_SIMD and UOPS_RETIRED.SCALAR_SIMD counters. This assumes things like the vector length of instructions is always 8, that FMA are used, and there is no masking, which is never going to be always true and therefore this can only be an upper bound. From Intel Advisor you can get an accurate FLOP count, but this is done via analysis and instrumentation of the binary so it will slow the code execution down noticeably. Further description is given here https://software.intel.com/en-us/articles/intel-advisor-flops -Scott On 2/21/17 8:33 PM, Banerjee,Tania wrote: Hi, How to measure throughput (GFLOPs/second) of a program on KNL? Normally I use PAPI to count the floating point operations (PAPI_FP_OPS / PAPI_FP_INS) and divide that by the time taken. But papi_avail reveals that these events are not supported on KNL. The output of /soft/perftools/tau/papi-knl/bin/papi_avail is given below. Thanks much, Tania ============================================================================= Available PAPI preset and user defined events plus hardware information. -------------------------------------------------------------------------------- PAPI Version : 5.4.3.0 Vendor string and code : GenuineIntel (1) Model string and code : Intel(R) Xeon Phi(TM) CPU 7210 @ 1.30GHz (87) CPU Revision : 1.000000 CPUID Info : Family: 6 Model: 87 Stepping: 1 CPU Max Megahertz : 1500 CPU Min Megahertz : 1000 Hdw Threads per core : 4 Cores per Socket : 64 Sockets : 1 NUMA Nodes : 1 CPUs per Node : 256 Total CPUs : 256 Running in a VM : no Number Hardware Counters : 5 Max Multiplex Counters : 192 -------------------------------------------------------------------------------- ================================================================================ PAPI Preset Events ================================================================================ Name Code Avail Deriv Description (Note) PAPI_L1_DCM 0x80000000 Yes No Level 1 data cache misses PAPI_L1_ICM 0x80000001 Yes No Level 1 instruction cache misses PAPI_L2_DCM 0x80000002 No No Level 2 data cache misses PAPI_L2_ICM 0x80000003 No No Level 2 instruction cache misses PAPI_L3_DCM 0x80000004 No No Level 3 data cache misses PAPI_L3_ICM 0x80000005 No No Level 3 instruction cache misses PAPI_L1_TCM 0x80000006 Yes Yes Level 1 cache misses PAPI_L2_TCM 0x80000007 Yes No Level 2 cache misses PAPI_L3_TCM 0x80000008 No No Level 3 cache misses PAPI_CA_SNP 0x80000009 No No Requests for a snoop PAPI_CA_SHR 0x8000000a No No Requests for exclusive access to shared cache line PAPI_CA_CLN 0x8000000b No No Requests for exclusive access to clean cache line PAPI_CA_INV 0x8000000c No No Requests for cache line invalidation PAPI_CA_ITV 0x8000000d No No Requests for cache line intervention PAPI_L3_LDM 0x8000000e No No Level 3 load misses PAPI_L3_STM 0x8000000f No No Level 3 store misses PAPI_BRU_IDL 0x80000010 No No Cycles branch units are idle PAPI_FXU_IDL 0x80000011 No No Cycles integer units are idle PAPI_FPU_IDL 0x80000012 No No Cycles floating point units are idle PAPI_LSU_IDL 0x80000013 No No Cycles load/store units are idle PAPI_TLB_DM 0x80000014 Yes No Data translation lookaside buffer misses PAPI_TLB_IM 0x80000015 No No Instruction translation lookaside buffer misses PAPI_TLB_TL 0x80000016 No No Total translation lookaside buffer misses PAPI_L1_LDM 0x80000017 Yes No Level 1 load misses PAPI_L1_STM 0x80000018 No No Level 1 store misses PAPI_L2_LDM 0x80000019 Yes No Level 2 load misses PAPI_L2_STM 0x8000001a No No Level 2 store misses PAPI_BTAC_M 0x8000001b No No Branch target address cache misses PAPI_PRF_DM 0x8000001c No No Data prefetch cache misses PAPI_L3_DCH 0x8000001d No No Level 3 data cache hits PAPI_TLB_SD 0x8000001e No No Translation lookaside buffer shootdowns PAPI_CSR_FAL 0x8000001f No No Failed store conditional instructions PAPI_CSR_SUC 0x80000020 No No Successful store conditional instructions PAPI_CSR_TOT 0x80000021 No No Total store conditional instructions PAPI_MEM_SCY 0x80000022 No No Cycles Stalled Waiting for memory accesses PAPI_MEM_RCY 0x80000023 No No Cycles Stalled Waiting for memory Reads PAPI_MEM_WCY 0x80000024 No No Cycles Stalled Waiting for memory writes PAPI_STL_ICY 0x80000025 Yes No Cycles with no instruction issue PAPI_FUL_ICY 0x80000026 No No Cycles with maximum instruction issue PAPI_STL_CCY 0x80000027 No No Cycles with no instructions completed PAPI_FUL_CCY 0x80000028 No No Cycles with maximum instructions completed PAPI_HW_INT 0x80000029 No No Hardware interrupts PAPI_BR_UCN 0x8000002a Yes Yes Unconditional branch instructions PAPI_BR_CN 0x8000002b Yes No Conditional branch instructions PAPI_BR_TKN 0x8000002c Yes No Conditional branch instructions taken PAPI_BR_NTK 0x8000002d Yes Yes Conditional branch instructions not taken PAPI_BR_MSP 0x8000002e Yes No Conditional branch instructions mispredicted PAPI_BR_PRC 0x8000002f No No Conditional branch instructions correctly predicted PAPI_FMA_INS 0x80000030 No No FMA instructions completed PAPI_TOT_IIS 0x80000031 No No Instructions issued PAPI_TOT_INS 0x80000032 Yes No Instructions completed PAPI_INT_INS 0x80000033 No No Integer instructions PAPI_FP_INS 0x80000034 No No Floating point instructions PAPI_LD_INS 0x80000035 Yes No Load instructions PAPI_SR_INS 0x80000036 No No Store instructions PAPI_BR_INS 0x80000037 Yes No Branch instructions PAPI_VEC_INS 0x80000038 No No Vector/SIMD instructions (could include integer) PAPI_RES_STL 0x80000039 Yes No Cycles stalled on any resource PAPI_FP_STAL 0x8000003a No No Cycles the FP unit(s) are stalled PAPI_TOT_CYC 0x8000003b Yes No Total cycles PAPI_LST_INS 0x8000003c Yes Yes Load/store instructions completed PAPI_SYC_INS 0x8000003d No No Synchronization instructions completed PAPI_L1_DCH 0x8000003e No No Level 1 data cache hits PAPI_L2_DCH 0x8000003f No No Level 2 data cache hits PAPI_L1_DCA 0x80000040 Yes Yes Level 1 data cache accesses PAPI_L2_DCA 0x80000041 No No Level 2 data cache accesses PAPI_L3_DCA 0x80000042 No No Level 3 data cache accesses PAPI_L1_DCR 0x80000043 No No Level 1 data cache reads PAPI_L2_DCR 0x80000044 No No Level 2 data cache reads PAPI_L3_DCR 0x80000045 No No Level 3 data cache reads PAPI_L1_DCW 0x80000046 No No Level 1 data cache writes PAPI_L2_DCW 0x80000047 No No Level 2 data cache writes PAPI_L3_DCW 0x80000048 No No Level 3 data cache writes PAPI_L1_ICH 0x80000049 Yes No Level 1 instruction cache hits PAPI_L2_ICH 0x8000004a No No Level 2 instruction cache hits PAPI_L3_ICH 0x8000004b No No Level 3 instruction cache hits PAPI_L1_ICA 0x8000004c Yes No Level 1 instruction cache accesses PAPI_L2_ICA 0x8000004d No No Level 2 instruction cache accesses PAPI_L3_ICA 0x8000004e No No Level 3 instruction cache accesses PAPI_L1_ICR 0x8000004f No No Level 1 instruction cache reads PAPI_L2_ICR 0x80000050 No No Level 2 instruction cache reads PAPI_L3_ICR 0x80000051 No No Level 3 instruction cache reads PAPI_L1_ICW 0x80000052 No No Level 1 instruction cache writes PAPI_L2_ICW 0x80000053 No No Level 2 instruction cache writes PAPI_L3_ICW 0x80000054 No No Level 3 instruction cache writes PAPI_L1_TCH 0x80000055 No No Level 1 total cache hits PAPI_L2_TCH 0x80000056 Yes Yes Level 2 total cache hits PAPI_L3_TCH 0x80000057 No No Level 3 total cache hits PAPI_L1_TCA 0x80000058 No No Level 1 total cache accesses PAPI_L2_TCA 0x80000059 Yes No Level 2 total cache accesses PAPI_L3_TCA 0x8000005a No No Level 3 total cache accesses PAPI_L1_TCR 0x8000005b No No Level 1 total cache reads PAPI_L2_TCR 0x8000005c No No Level 2 total cache reads PAPI_L3_TCR 0x8000005d No No Level 3 total cache reads PAPI_L1_TCW 0x8000005e No No Level 1 total cache writes PAPI_L2_TCW 0x8000005f No No Level 2 total cache writes PAPI_L3_TCW 0x80000060 No No Level 3 total cache writes PAPI_FML_INS 0x80000061 No No Floating point multiply instructions PAPI_FAD_INS 0x80000062 No No Floating point add instructions PAPI_FDV_INS 0x80000063 No No Floating point divide instructions PAPI_FSQ_INS 0x80000064 No No Floating point square root instructions PAPI_FNV_INS 0x80000065 No No Floating point inverse instructions PAPI_FP_OPS 0x80000066 No No Floating point operations PAPI_SP_OPS 0x80000067 No No Floating point operations; optimized to count scaled single precision vector operations PAPI_DP_OPS 0x80000068 No No Floating point operations; optimized to count scaled double precision vector operations PAPI_VEC_SP 0x80000069 No No Single precision vector/SIMD instructions PAPI_VEC_DP 0x8000006a No No Double precision vector/SIMD instructions PAPI_REF_CYC 0x8000006b Yes No Reference clock cycles -------------------------------------------------------------------------------- Of 108 possible events, 25 are available, of which 6 are derived. avail.c PASSED _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Measuring throughput on KNL
by Balakrishnan, Ramesh 24 Feb '17

24 Feb '17

Its an interesting question, Paul. For strong scaling problems with engineering applications, where time to solution is important, the n_1/2 criterion would be an important measure. Ramesh On Feb 24, 2017, at 11:31 AM, Fischer, Paul F. <fischer(a)mcs.anl.gov<mailto:[email protected]>> wrote: Does anyone know what the n_1/2 is for DGEMM on KNL ? ________________________________ From: intel-nda-bounces(a)lists.jlse.anl.gov<mailto:[email protected]> [intel-nda-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>] on behalf of Vitali A. Morozov [morozov(a)anl.gov<mailto:[email protected]>] Sent: Friday, February 24, 2017 11:27 AM To: Hammond, Jeff R Cc: intel-nda(a)jlse.anl.gov<mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] Measuring throughput on KNL This is correct - the original code is written with asm, 4 bugs were reported to an Intel compiler team, they all fixed in version 17. Right now, the version 17 must be compiling the HACCmk to an equivalent object code as an asm version. If not, there must be another bug reported. :) Vitali p.s. Calling DGEMM will probably be not a compiler-generated code path as well. On 02/24/2017 11:03 AM, Hammond, Jeff R wrote: Vitali: HACCmk uses asm or intrinsics to get 82%, does it not? If neither, that is great. Aaron: Call DGEMM :-) Jeff Sent from my iPhone On Feb 24, 2017, at 7:43 AM, Vitali A. Morozov <morozov(a)anl.gov<mailto:[email protected]>> wrote: Yes, it is possible. I was doing this for HACCmk and reached 82% of peak. Vitali On 02/24/2017 09:37 AM, Aaron Knoll wrote: Along these lines: Is it possible to write some very simple compute-only code (in either C++ or Fortran) that when compiled would achieve peak flops (or close to it) on KNL, and measure actual throughput? @Jeff what would you recommend for this? -Aaron On Thu, Feb 23, 2017 at 5:12 PM, Scott Parker <sparker(a)anl.gov<mailto:[email protected]>> wrote: You should use /soft/compilers/intel/advisor -Scott On 2/23/17 5:59 PM, Banerjee,Tania wrote: Hi All, Thanks for pitching in. Scott: Which advisor should we use in /soft/compilers/intel ? [tania@jlselogin1 intel]$ ls advisor advisor/ advisor_xe/ advisor_2017/ advisor_xe_2016/ advisor_2017.1.0.477503/ advisor_xe_2016.1.30.450722/ advisor_2017.1.1.486553/ advisor_xe_2016.1.40.463413/ Also about Intel VTunes, I could get only the following summary (using both hpc-performance and memory-access collection modes) , which did not feature the events talked about in this thread: Event summary ------------- Hardware Event Type Hardware Event Count:Self Hardware Event Sample Count:Self Events Per Sample --------------------------------- ------------------------- -------------------------------- ----------------- INST_RETIRED.ANY 1080573000000 83121 13000000 CPU_CLK_UNHALTED.THREAD 2283515000000 175655 13000000 CPU_CLK_UNHALTED.REF_TSC 2120508000000 163116 13000000 MEM_UOPS_RETIRED.L2_MISS_LOADS_PS 396011880 132 1000030 UOPS_RETIRED.ALL 1479002218500 24650 20000030 MEM_UOPS_RETIRED.L2_HIT_LOADS_PS 2079062370<tel:%28207%29%20906-2370> 693 1000030 NO_ALLOC_CYCLES.MISPREDICTS 11220016830 187 20000030 NO_ALLOC_CYCLES.NOT_DELIVERED 720601080900 12010 20000030 Uncore Event summary -------------------- Uncore Event Type Uncore Event Count:Self ... But it could be because I did not enable -avx during compilation, hence no there's no vectorization, whatsoever. I will check this aspect as the other part of my project on performance tuning. Thanks a lot, Tania ________________________________ From: Vitali A. Morozov <morozov(a)anl.gov><mailto:[email protected]> Sent: Wednesday, February 22, 2017 6:08 PM To: Hammond, Jeff R; Scott Parker; Banerjee,Tania; intel-nda(a)jlse.anl.gov<mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] Measuring throughput on KNL In general, scalar code generation may differ from vector code generation - conditionals area perfect example of it. Also, how to account an FMA versus a non-FMA instruction? Vitali On 02/22/2017 04:54 PM, Hammond, Jeff R wrote: I agree that PACKED_SIMD and SCALAR_SIMD include non-float operations. My point was that if the problem is SIMD float ops overcount, then forcing the compiler to turn them all into scalar float ops should prevent that. I assume there is a way to count scalar float ops, but if not, then I guess this doesn't help. At least for SSE*/AVX/AVX2 code, another way to count flops precisely is to use an implementation where the hardware supports precise counters. https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX2 Jeff From: Scott Parker <sparker(a)anl.gov><mailto:[email protected]> Date: Wednesday, February 22, 2017 at 12:32 PM To: "Hammond, Jeff R" <jeff.r.hammond(a)intel.com><mailto:[email protected]>, "Banerjee,Tania" <tmishra(a)ufl.edu><mailto:[email protected]>, "intel-nda(a)jlse.anl.gov"<mailto:[email protected]> <intel-nda(a)jlse.anl.gov><mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] Measuring throughput on KNL I don't think that will get you an accurate flop count either on KNL, though it may be closer. I'm not sure of the full set of instructions that map into the PACKED_SIMD and SCALAR_SIMD counters but I'm pretty sure there are some that are not arithmatic or floating point. From past discussions with Dmitry Prohorov from VTune the Intel Advisor approach is the only way on KNL -Scott On 2/22/17 11:54 AM, Hammond, Jeff R wrote: If one only wants to count flops, I assume disabling SIMD will obviate the overcounting due to masking and such, but I have not experimented with this. -[no-]simd enables(DEFAULT)/disables vectorization using simd pragma Jeff From: <intel-nda-bounces(a)lists.jlse.anl.gov><mailto:[email protected]> on behalf of Scott Parker <sparker(a)anl.gov><mailto:[email protected]> Date: Wednesday, February 22, 2017 at 9:22 AM To: "Banerjee,Tania" <tmishra(a)ufl.edu><mailto:[email protected]>, "intel-nda(a)jlse.anl.gov"<mailto:[email protected]> <intel-nda(a)jlse.anl.gov><mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] Measuring throughput on KNL KNL doesn't have hardware counters for flops, which is why PAPI doesn't provide PAPI_FP_OPS. This is a problem on a number of current Intel platforms. From Vtune you can get a upper bound flops estimate based on the UOPS_RETIRED.PACKED_SIMD and UOPS_RETIRED.SCALAR_SIMD counters. This assumes things like the vector length of instructions is always 8, that FMA are used, and there is no masking, which is never going to be always true and therefore this can only be an upper bound. From Intel Advisor you can get an accurate FLOP count, but this is done via analysis and instrumentation of the binary so it will slow the code execution down noticeably. Further description is given here https://software.intel.com/en-us/articles/intel-advisor-flops -Scott On 2/21/17 8:33 PM, Banerjee,Tania wrote: Hi, How to measure throughput (GFLOPs/second) of a program on KNL? Normally I use PAPI to count the floating point operations (PAPI_FP_OPS / PAPI_FP_INS) and divide that by the time taken. But papi_avail reveals that these events are not supported on KNL. The output of /soft/perftools/tau/papi-knl/bin/papi_avail is given below. Thanks much, Tania ============================================================================= Available PAPI preset and user defined events plus hardware information. -------------------------------------------------------------------------------- PAPI Version : 5.4.3.0 Vendor string and code : GenuineIntel (1) Model string and code : Intel(R) Xeon Phi(TM) CPU 7210 @ 1.30GHz (87) CPU Revision : 1.000000 CPUID Info : Family: 6 Model: 87 Stepping: 1 CPU Max Megahertz : 1500 CPU Min Megahertz : 1000 Hdw Threads per core : 4 Cores per Socket : 64 Sockets : 1 NUMA Nodes : 1 CPUs per Node : 256 Total CPUs : 256 Running in a VM : no Number Hardware Counters : 5 Max Multiplex Counters : 192 -------------------------------------------------------------------------------- ================================================================================ PAPI Preset Events ================================================================================ Name Code Avail Deriv Description (Note) PAPI_L1_DCM 0x80000000 Yes No Level 1 data cache misses PAPI_L1_ICM 0x80000001 Yes No Level 1 instruction cache misses PAPI_L2_DCM 0x80000002 No No Level 2 data cache misses PAPI_L2_ICM 0x80000003 No No Level 2 instruction cache misses PAPI_L3_DCM 0x80000004 No No Level 3 data cache misses PAPI_L3_ICM 0x80000005 No No Level 3 instruction cache misses PAPI_L1_TCM 0x80000006 Yes Yes Level 1 cache misses PAPI_L2_TCM 0x80000007 Yes No Level 2 cache misses PAPI_L3_TCM 0x80000008 No No Level 3 cache misses PAPI_CA_SNP 0x80000009 No No Requests for a snoop PAPI_CA_SHR 0x8000000a No No Requests for exclusive access to shared cache line PAPI_CA_CLN 0x8000000b No No Requests for exclusive access to clean cache line PAPI_CA_INV 0x8000000c No No Requests for cache line invalidation PAPI_CA_ITV 0x8000000d No No Requests for cache line intervention PAPI_L3_LDM 0x8000000e No No Level 3 load misses PAPI_L3_STM 0x8000000f No No Level 3 store misses PAPI_BRU_IDL 0x80000010 No No Cycles branch units are idle PAPI_FXU_IDL 0x80000011 No No Cycles integer units are idle PAPI_FPU_IDL 0x80000012 No No Cycles floating point units are idle PAPI_LSU_IDL 0x80000013 No No Cycles load/store units are idle PAPI_TLB_DM 0x80000014 Yes No Data translation lookaside buffer misses PAPI_TLB_IM 0x80000015 No No Instruction translation lookaside buffer misses PAPI_TLB_TL 0x80000016 No No Total translation lookaside buffer misses PAPI_L1_LDM 0x80000017 Yes No Level 1 load misses PAPI_L1_STM 0x80000018 No No Level 1 store misses PAPI_L2_LDM 0x80000019 Yes No Level 2 load misses PAPI_L2_STM 0x8000001a No No Level 2 store misses PAPI_BTAC_M 0x8000001b No No Branch target address cache misses PAPI_PRF_DM 0x8000001c No No Data prefetch cache misses PAPI_L3_DCH 0x8000001d No No Level 3 data cache hits PAPI_TLB_SD 0x8000001e No No Translation lookaside buffer shootdowns PAPI_CSR_FAL 0x8000001f No No Failed store conditional instructions PAPI_CSR_SUC 0x80000020 No No Successful store conditional instructions PAPI_CSR_TOT 0x80000021 No No Total store conditional instructions PAPI_MEM_SCY 0x80000022 No No Cycles Stalled Waiting for memory accesses PAPI_MEM_RCY 0x80000023 No No Cycles Stalled Waiting for memory Reads PAPI_MEM_WCY 0x80000024 No No Cycles Stalled Waiting for memory writes PAPI_STL_ICY 0x80000025 Yes No Cycles with no instruction issue PAPI_FUL_ICY 0x80000026 No No Cycles with maximum instruction issue PAPI_STL_CCY 0x80000027 No No Cycles with no instructions completed PAPI_FUL_CCY 0x80000028 No No Cycles with maximum instructions completed PAPI_HW_INT 0x80000029 No No Hardware interrupts PAPI_BR_UCN 0x8000002a Yes Yes Unconditional branch instructions PAPI_BR_CN 0x8000002b Yes No Conditional branch instructions PAPI_BR_TKN 0x8000002c Yes No Conditional branch instructions taken PAPI_BR_NTK 0x8000002d Yes Yes Conditional branch instructions not taken PAPI_BR_MSP 0x8000002e Yes No Conditional branch instructions mispredicted PAPI_BR_PRC 0x8000002f No No Conditional branch instructions correctly predicted PAPI_FMA_INS 0x80000030 No No FMA instructions completed PAPI_TOT_IIS 0x80000031 No No Instructions issued PAPI_TOT_INS 0x80000032 Yes No Instructions completed PAPI_INT_INS 0x80000033 No No Integer instructions PAPI_FP_INS 0x80000034 No No Floating point instructions PAPI_LD_INS 0x80000035 Yes No Load instructions PAPI_SR_INS 0x80000036 No No Store instructions PAPI_BR_INS 0x80000037 Yes No Branch instructions PAPI_VEC_INS 0x80000038 No No Vector/SIMD instructions (could include integer) PAPI_RES_STL 0x80000039 Yes No Cycles stalled on any resource PAPI_FP_STAL 0x8000003a No No Cycles the FP unit(s) are stalled PAPI_TOT_CYC 0x8000003b Yes No Total cycles PAPI_LST_INS 0x8000003c Yes Yes Load/store instructions completed PAPI_SYC_INS 0x8000003d No No Synchronization instructions completed PAPI_L1_DCH 0x8000003e No No Level 1 data cache hits PAPI_L2_DCH 0x8000003f No No Level 2 data cache hits PAPI_L1_DCA 0x80000040 Yes Yes Level 1 data cache accesses PAPI_L2_DCA 0x80000041 No No Level 2 data cache accesses PAPI_L3_DCA 0x80000042 No No Level 3 data cache accesses PAPI_L1_DCR 0x80000043 No No Level 1 data cache reads PAPI_L2_DCR 0x80000044 No No Level 2 data cache reads PAPI_L3_DCR 0x80000045 No No Level 3 data cache reads PAPI_L1_DCW 0x80000046 No No Level 1 data cache writes PAPI_L2_DCW 0x80000047 No No Level 2 data cache writes PAPI_L3_DCW 0x80000048 No No Level 3 data cache writes PAPI_L1_ICH 0x80000049 Yes No Level 1 instruction cache hits PAPI_L2_ICH 0x8000004a No No Level 2 instruction cache hits PAPI_L3_ICH 0x8000004b No No Level 3 instruction cache hits PAPI_L1_ICA 0x8000004c Yes No Level 1 instruction cache accesses PAPI_L2_ICA 0x8000004d No No Level 2 instruction cache accesses PAPI_L3_ICA 0x8000004e No No Level 3 instruction cache accesses PAPI_L1_ICR 0x8000004f No No Level 1 instruction cache reads PAPI_L2_ICR 0x80000050 No No Level 2 instruction cache reads PAPI_L3_ICR 0x80000051 No No Level 3 instruction cache reads PAPI_L1_ICW 0x80000052 No No Level 1 instruction cache writes PAPI_L2_ICW 0x80000053 No No Level 2 instruction cache writes PAPI_L3_ICW 0x80000054 No No Level 3 instruction cache writes PAPI_L1_TCH 0x80000055 No No Level 1 total cache hits PAPI_L2_TCH 0x80000056 Yes Yes Level 2 total cache hits PAPI_L3_TCH 0x80000057 No No Level 3 total cache hits PAPI_L1_TCA 0x80000058 No No Level 1 total cache accesses PAPI_L2_TCA 0x80000059 Yes No Level 2 total cache accesses PAPI_L3_TCA 0x8000005a No No Level 3 total cache accesses PAPI_L1_TCR 0x8000005b No No Level 1 total cache reads PAPI_L2_TCR 0x8000005c No No Level 2 total cache reads PAPI_L3_TCR 0x8000005d No No Level 3 total cache reads PAPI_L1_TCW 0x8000005e No No Level 1 total cache writes PAPI_L2_TCW 0x8000005f No No Level 2 total cache writes PAPI_L3_TCW 0x80000060 No No Level 3 total cache writes PAPI_FML_INS 0x80000061 No No Floating point multiply instructions PAPI_FAD_INS 0x80000062 No No Floating point add instructions PAPI_FDV_INS 0x80000063 No No Floating point divide instructions PAPI_FSQ_INS 0x80000064 No No Floating point square root instructions PAPI_FNV_INS 0x80000065 No No Floating point inverse instructions PAPI_FP_OPS 0x80000066 No No Floating point operations PAPI_SP_OPS 0x80000067 No No Floating point operations; optimized to count scaled single precision vector operations PAPI_DP_OPS 0x80000068 No No Floating point operations; optimized to count scaled double precision vector operations PAPI_VEC_SP 0x80000069 No No Single precision vector/SIMD instructions PAPI_VEC_DP 0x8000006a No No Double precision vector/SIMD instructions PAPI_REF_CYC 0x8000006b Yes No Reference clock cycles -------------------------------------------------------------------------------- Of 108 possible events, 25 are available, of which 6 are derived. avail.c PASSED _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Measuring throughput on KNL
by Fischer, Paul F. 24 Feb '17

24 Feb '17

Does anyone know what the n_1/2 is for DGEMM on KNL ? ________________________________ From: intel-nda-bounces(a)lists.jlse.anl.gov [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Vitali A. Morozov [morozov(a)anl.gov] Sent: Friday, February 24, 2017 11:27 AM To: Hammond, Jeff R Cc: intel-nda(a)jlse.anl.gov Subject: Re: [JLSE-Intel-NDA] Measuring throughput on KNL This is correct - the original code is written with asm, 4 bugs were reported to an Intel compiler team, they all fixed in version 17. Right now, the version 17 must be compiling the HACCmk to an equivalent object code as an asm version. If not, there must be another bug reported. :) Vitali p.s. Calling DGEMM will probably be not a compiler-generated code path as well. On 02/24/2017 11:03 AM, Hammond, Jeff R wrote: Vitali: HACCmk uses asm or intrinsics to get 82%, does it not? If neither, that is great. Aaron: Call DGEMM :-) Jeff Sent from my iPhone On Feb 24, 2017, at 7:43 AM, Vitali A. Morozov <morozov(a)anl.gov<mailto:[email protected]>> wrote: Yes, it is possible. I was doing this for HACCmk and reached 82% of peak. Vitali On 02/24/2017 09:37 AM, Aaron Knoll wrote: Along these lines: Is it possible to write some very simple compute-only code (in either C++ or Fortran) that when compiled would achieve peak flops (or close to it) on KNL, and measure actual throughput? @Jeff what would you recommend for this? -Aaron On Thu, Feb 23, 2017 at 5:12 PM, Scott Parker <sparker(a)anl.gov<mailto:[email protected]>> wrote: You should use /soft/compilers/intel/advisor -Scott On 2/23/17 5:59 PM, Banerjee,Tania wrote: Hi All, Thanks for pitching in. Scott: Which advisor should we use in /soft/compilers/intel ? [tania@jlselogin1 intel]$ ls advisor advisor/ advisor_xe/ advisor_2017/ advisor_xe_2016/ advisor_2017.1.0.477503/ advisor_xe_2016.1.30.450722/ advisor_2017.1.1.486553/ advisor_xe_2016.1.40.463413/ Also about Intel VTunes, I could get only the following summary (using both hpc-performance and memory-access collection modes) , which did not feature the events talked about in this thread: Event summary ------------- Hardware Event Type Hardware Event Count:Self Hardware Event Sample Count:Self Events Per Sample --------------------------------- ------------------------- -------------------------------- ----------------- INST_RETIRED.ANY 1080573000000 83121 13000000 CPU_CLK_UNHALTED.THREAD 2283515000000 175655 13000000 CPU_CLK_UNHALTED.REF_TSC 2120508000000 163116 13000000 MEM_UOPS_RETIRED.L2_MISS_LOADS_PS 396011880 132 1000030 UOPS_RETIRED.ALL 1479002218500 24650 20000030 MEM_UOPS_RETIRED.L2_HIT_LOADS_PS 2079062370<tel:%28207%29%20906-2370> 693 1000030 NO_ALLOC_CYCLES.MISPREDICTS 11220016830 187 20000030 NO_ALLOC_CYCLES.NOT_DELIVERED 720601080900 12010 20000030 Uncore Event summary -------------------- Uncore Event Type Uncore Event Count:Self ... But it could be because I did not enable -avx during compilation, hence no there's no vectorization, whatsoever. I will check this aspect as the other part of my project on performance tuning. Thanks a lot, Tania ________________________________ From: Vitali A. Morozov <morozov(a)anl.gov><mailto:[email protected]> Sent: Wednesday, February 22, 2017 6:08 PM To: Hammond, Jeff R; Scott Parker; Banerjee,Tania; intel-nda(a)jlse.anl.gov<mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] Measuring throughput on KNL In general, scalar code generation may differ from vector code generation - conditionals area perfect example of it. Also, how to account an FMA versus a non-FMA instruction? Vitali On 02/22/2017 04:54 PM, Hammond, Jeff R wrote: I agree that PACKED_SIMD and SCALAR_SIMD include non-float operations. My point was that if the problem is SIMD float ops overcount, then forcing the compiler to turn them all into scalar float ops should prevent that. I assume there is a way to count scalar float ops, but if not, then I guess this doesn't help. At least for SSE*/AVX/AVX2 code, another way to count flops precisely is to use an implementation where the hardware supports precise counters. https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX2 Jeff From: Scott Parker <sparker(a)anl.gov><mailto:[email protected]> Date: Wednesday, February 22, 2017 at 12:32 PM To: "Hammond, Jeff R" <jeff.r.hammond(a)intel.com><mailto:[email protected]>, "Banerjee,Tania" <tmishra(a)ufl.edu><mailto:[email protected]>, "intel-nda(a)jlse.anl.gov"<mailto:[email protected]> <intel-nda(a)jlse.anl.gov><mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] Measuring throughput on KNL I don't think that will get you an accurate flop count either on KNL, though it may be closer. I'm not sure of the full set of instructions that map into the PACKED_SIMD and SCALAR_SIMD counters but I'm pretty sure there are some that are not arithmatic or floating point. From past discussions with Dmitry Prohorov from VTune the Intel Advisor approach is the only way on KNL -Scott On 2/22/17 11:54 AM, Hammond, Jeff R wrote: If one only wants to count flops, I assume disabling SIMD will obviate the overcounting due to masking and such, but I have not experimented with this. -[no-]simd enables(DEFAULT)/disables vectorization using simd pragma Jeff From: <intel-nda-bounces(a)lists.jlse.anl.gov><mailto:[email protected]> on behalf of Scott Parker <sparker(a)anl.gov><mailto:[email protected]> Date: Wednesday, February 22, 2017 at 9:22 AM To: "Banerjee,Tania" <tmishra(a)ufl.edu><mailto:[email protected]>, "intel-nda(a)jlse.anl.gov"<mailto:[email protected]> <intel-nda(a)jlse.anl.gov><mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] Measuring throughput on KNL KNL doesn't have hardware counters for flops, which is why PAPI doesn't provide PAPI_FP_OPS. This is a problem on a number of current Intel platforms. From Vtune you can get a upper bound flops estimate based on the UOPS_RETIRED.PACKED_SIMD and UOPS_RETIRED.SCALAR_SIMD counters. This assumes things like the vector length of instructions is always 8, that FMA are used, and there is no masking, which is never going to be always true and therefore this can only be an upper bound. From Intel Advisor you can get an accurate FLOP count, but this is done via analysis and instrumentation of the binary so it will slow the code execution down noticeably. Further description is given here https://software.intel.com/en-us/articles/intel-advisor-flops -Scott On 2/21/17 8:33 PM, Banerjee,Tania wrote: Hi, How to measure throughput (GFLOPs/second) of a program on KNL? Normally I use PAPI to count the floating point operations (PAPI_FP_OPS / PAPI_FP_INS) and divide that by the time taken. But papi_avail reveals that these events are not supported on KNL. The output of /soft/perftools/tau/papi-knl/bin/papi_avail is given below. Thanks much, Tania ============================================================================= Available PAPI preset and user defined events plus hardware information. -------------------------------------------------------------------------------- PAPI Version : 5.4.3.0 Vendor string and code : GenuineIntel (1) Model string and code : Intel(R) Xeon Phi(TM) CPU 7210 @ 1.30GHz (87) CPU Revision : 1.000000 CPUID Info : Family: 6 Model: 87 Stepping: 1 CPU Max Megahertz : 1500 CPU Min Megahertz : 1000 Hdw Threads per core : 4 Cores per Socket : 64 Sockets : 1 NUMA Nodes : 1 CPUs per Node : 256 Total CPUs : 256 Running in a VM : no Number Hardware Counters : 5 Max Multiplex Counters : 192 -------------------------------------------------------------------------------- ================================================================================ PAPI Preset Events ================================================================================ Name Code Avail Deriv Description (Note) PAPI_L1_DCM 0x80000000 Yes No Level 1 data cache misses PAPI_L1_ICM 0x80000001 Yes No Level 1 instruction cache misses PAPI_L2_DCM 0x80000002 No No Level 2 data cache misses PAPI_L2_ICM 0x80000003 No No Level 2 instruction cache misses PAPI_L3_DCM 0x80000004 No No Level 3 data cache misses PAPI_L3_ICM 0x80000005 No No Level 3 instruction cache misses PAPI_L1_TCM 0x80000006 Yes Yes Level 1 cache misses PAPI_L2_TCM 0x80000007 Yes No Level 2 cache misses PAPI_L3_TCM 0x80000008 No No Level 3 cache misses PAPI_CA_SNP 0x80000009 No No Requests for a snoop PAPI_CA_SHR 0x8000000a No No Requests for exclusive access to shared cache line PAPI_CA_CLN 0x8000000b No No Requests for exclusive access to clean cache line PAPI_CA_INV 0x8000000c No No Requests for cache line invalidation PAPI_CA_ITV 0x8000000d No No Requests for cache line intervention PAPI_L3_LDM 0x8000000e No No Level 3 load misses PAPI_L3_STM 0x8000000f No No Level 3 store misses PAPI_BRU_IDL 0x80000010 No No Cycles branch units are idle PAPI_FXU_IDL 0x80000011 No No Cycles integer units are idle PAPI_FPU_IDL 0x80000012 No No Cycles floating point units are idle PAPI_LSU_IDL 0x80000013 No No Cycles load/store units are idle PAPI_TLB_DM 0x80000014 Yes No Data translation lookaside buffer misses PAPI_TLB_IM 0x80000015 No No Instruction translation lookaside buffer misses PAPI_TLB_TL 0x80000016 No No Total translation lookaside buffer misses PAPI_L1_LDM 0x80000017 Yes No Level 1 load misses PAPI_L1_STM 0x80000018 No No Level 1 store misses PAPI_L2_LDM 0x80000019 Yes No Level 2 load misses PAPI_L2_STM 0x8000001a No No Level 2 store misses PAPI_BTAC_M 0x8000001b No No Branch target address cache misses PAPI_PRF_DM 0x8000001c No No Data prefetch cache misses PAPI_L3_DCH 0x8000001d No No Level 3 data cache hits PAPI_TLB_SD 0x8000001e No No Translation lookaside buffer shootdowns PAPI_CSR_FAL 0x8000001f No No Failed store conditional instructions PAPI_CSR_SUC 0x80000020 No No Successful store conditional instructions PAPI_CSR_TOT 0x80000021 No No Total store conditional instructions PAPI_MEM_SCY 0x80000022 No No Cycles Stalled Waiting for memory accesses PAPI_MEM_RCY 0x80000023 No No Cycles Stalled Waiting for memory Reads PAPI_MEM_WCY 0x80000024 No No Cycles Stalled Waiting for memory writes PAPI_STL_ICY 0x80000025 Yes No Cycles with no instruction issue PAPI_FUL_ICY 0x80000026 No No Cycles with maximum instruction issue PAPI_STL_CCY 0x80000027 No No Cycles with no instructions completed PAPI_FUL_CCY 0x80000028 No No Cycles with maximum instructions completed PAPI_HW_INT 0x80000029 No No Hardware interrupts PAPI_BR_UCN 0x8000002a Yes Yes Unconditional branch instructions PAPI_BR_CN 0x8000002b Yes No Conditional branch instructions PAPI_BR_TKN 0x8000002c Yes No Conditional branch instructions taken PAPI_BR_NTK 0x8000002d Yes Yes Conditional branch instructions not taken PAPI_BR_MSP 0x8000002e Yes No Conditional branch instructions mispredicted PAPI_BR_PRC 0x8000002f No No Conditional branch instructions correctly predicted PAPI_FMA_INS 0x80000030 No No FMA instructions completed PAPI_TOT_IIS 0x80000031 No No Instructions issued PAPI_TOT_INS 0x80000032 Yes No Instructions completed PAPI_INT_INS 0x80000033 No No Integer instructions PAPI_FP_INS 0x80000034 No No Floating point instructions PAPI_LD_INS 0x80000035 Yes No Load instructions PAPI_SR_INS 0x80000036 No No Store instructions PAPI_BR_INS 0x80000037 Yes No Branch instructions PAPI_VEC_INS 0x80000038 No No Vector/SIMD instructions (could include integer) PAPI_RES_STL 0x80000039 Yes No Cycles stalled on any resource PAPI_FP_STAL 0x8000003a No No Cycles the FP unit(s) are stalled PAPI_TOT_CYC 0x8000003b Yes No Total cycles PAPI_LST_INS 0x8000003c Yes Yes Load/store instructions completed PAPI_SYC_INS 0x8000003d No No Synchronization instructions completed PAPI_L1_DCH 0x8000003e No No Level 1 data cache hits PAPI_L2_DCH 0x8000003f No No Level 2 data cache hits PAPI_L1_DCA 0x80000040 Yes Yes Level 1 data cache accesses PAPI_L2_DCA 0x80000041 No No Level 2 data cache accesses PAPI_L3_DCA 0x80000042 No No Level 3 data cache accesses PAPI_L1_DCR 0x80000043 No No Level 1 data cache reads PAPI_L2_DCR 0x80000044 No No Level 2 data cache reads PAPI_L3_DCR 0x80000045 No No Level 3 data cache reads PAPI_L1_DCW 0x80000046 No No Level 1 data cache writes PAPI_L2_DCW 0x80000047 No No Level 2 data cache writes PAPI_L3_DCW 0x80000048 No No Level 3 data cache writes PAPI_L1_ICH 0x80000049 Yes No Level 1 instruction cache hits PAPI_L2_ICH 0x8000004a No No Level 2 instruction cache hits PAPI_L3_ICH 0x8000004b No No Level 3 instruction cache hits PAPI_L1_ICA 0x8000004c Yes No Level 1 instruction cache accesses PAPI_L2_ICA 0x8000004d No No Level 2 instruction cache accesses PAPI_L3_ICA 0x8000004e No No Level 3 instruction cache accesses PAPI_L1_ICR 0x8000004f No No Level 1 instruction cache reads PAPI_L2_ICR 0x80000050 No No Level 2 instruction cache reads PAPI_L3_ICR 0x80000051 No No Level 3 instruction cache reads PAPI_L1_ICW 0x80000052 No No Level 1 instruction cache writes PAPI_L2_ICW 0x80000053 No No Level 2 instruction cache writes PAPI_L3_ICW 0x80000054 No No Level 3 instruction cache writes PAPI_L1_TCH 0x80000055 No No Level 1 total cache hits PAPI_L2_TCH 0x80000056 Yes Yes Level 2 total cache hits PAPI_L3_TCH 0x80000057 No No Level 3 total cache hits PAPI_L1_TCA 0x80000058 No No Level 1 total cache accesses PAPI_L2_TCA 0x80000059 Yes No Level 2 total cache accesses PAPI_L3_TCA 0x8000005a No No Level 3 total cache accesses PAPI_L1_TCR 0x8000005b No No Level 1 total cache reads PAPI_L2_TCR 0x8000005c No No Level 2 total cache reads PAPI_L3_TCR 0x8000005d No No Level 3 total cache reads PAPI_L1_TCW 0x8000005e No No Level 1 total cache writes PAPI_L2_TCW 0x8000005f No No Level 2 total cache writes PAPI_L3_TCW 0x80000060 No No Level 3 total cache writes PAPI_FML_INS 0x80000061 No No Floating point multiply instructions PAPI_FAD_INS 0x80000062 No No Floating point add instructions PAPI_FDV_INS 0x80000063 No No Floating point divide instructions PAPI_FSQ_INS 0x80000064 No No Floating point square root instructions PAPI_FNV_INS 0x80000065 No No Floating point inverse instructions PAPI_FP_OPS 0x80000066 No No Floating point operations PAPI_SP_OPS 0x80000067 No No Floating point operations; optimized to count scaled single precision vector operations PAPI_DP_OPS 0x80000068 No No Floating point operations; optimized to count scaled double precision vector operations PAPI_VEC_SP 0x80000069 No No Single precision vector/SIMD instructions PAPI_VEC_DP 0x8000006a No No Double precision vector/SIMD instructions PAPI_REF_CYC 0x8000006b Yes No Reference clock cycles -------------------------------------------------------------------------------- Of 108 possible events, 25 are available, of which 6 are derived. avail.c PASSED _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Measuring throughput on KNL
by Vitali A. Morozov 24 Feb '17

24 Feb '17

This is correct - the original code is written with asm, 4 bugs were reported to an Intel compiler team, they all fixed in version 17. Right now, the version 17 must be compiling the HACCmk to an equivalent object code as an asm version. If not, there must be another bug reported. :) Vitali p.s. Calling DGEMM will probably be not a compiler-generated code path as well. On 02/24/2017 11:03 AM, Hammond, Jeff R wrote: > Vitali: > > HACCmk uses asm or intrinsics to get 82%, does it not? If neither, > that is great. > > Aaron: > > Call DGEMM :-) > > Jeff > > Sent from my iPhone > > On Feb 24, 2017, at 7:43 AM, Vitali A. Morozov <morozov(a)anl.gov > <mailto:[email protected]>> wrote: > >> >> Yes, it is possible. I was doing this for HACCmk and reached 82% of peak. >> >> Vitali >> >> >> On 02/24/2017 09:37 AM, Aaron Knoll wrote: >>> Along these lines: >>> >>> Is it possible to write some very simple compute-only code (in >>> either C++ or Fortran) that when compiled would achieve peak flops >>> (or close to it) on KNL, and measure actual throughput? >>> >>> @Jeff what would you recommend for this? >>> >>> -Aaron >>> >>> On Thu, Feb 23, 2017 at 5:12 PM, Scott Parker <sparker(a)anl.gov >>> <mailto:[email protected]>> wrote: >>> >>> >>> You should use /soft/compilers/intel/advisor >>> >>> >>> -Scott >>> >>> >>> On 2/23/17 5:59 PM, Banerjee,Tania wrote: >>>> >>>> Hi All, >>>> >>>> >>>> Thanks for pitching in. >>>> >>>> >>>> Scott: >>>> >>>> Which advisor should we use in /soft/compilers/intel ? >>>> >>>> >>>> [tania@jlselogin1 intel]$ ls advisor >>>> advisor/ advisor_xe/ >>>> advisor_2017/ advisor_xe_2016/ >>>> advisor_2017.1.0.477503/ advisor_xe_2016.1.30.450722/ >>>> advisor_2017.1.1.486553/ advisor_xe_2016.1.40.463413/ >>>> >>>> Also about Intel VTunes, I could get only the following >>>> summary (using both hpc-performance and memory-access >>>> collection modes) , which did not feature the events talked >>>> about in this thread: >>>> >>>> Event summary >>>> ------------- >>>> Hardware Event Type Hardware Event Count:Self >>>> Hardware Event Sample Count:Self Events Per Sample >>>> --------------------------------- ------------------------- >>>> -------------------------------- ----------------- >>>> >>>> INST_RETIRED.ANY 1080573000000 >>>> 83121 13000000 >>>> CPU_CLK_UNHALTED.THREAD 2283515000000 >>>> 175655 13000000 >>>> CPU_CLK_UNHALTED.REF_TSC 2120508000000 >>>> 163116 13000000 >>>> MEM_UOPS_RETIRED.L2_MISS_LOADS_PS 396011880 >>>> 132 1000030 >>>> UOPS_RETIRED.ALL 1479002218500 >>>> 24650 20000030 >>>> MEM_UOPS_RETIRED.L2_HIT_LOADS_PS 2079062370 >>>> <tel:%28207%29%20906-2370> 693 >>>> 1000030 >>>> NO_ALLOC_CYCLES.MISPREDICTS 11220016830 >>>> 187 20000030 >>>> NO_ALLOC_CYCLES.NOT_DELIVERED 720601080900 >>>> 12010 20000030 >>>> >>>> Uncore Event summary >>>> -------------------- >>>> Uncore Event Type Uncore Event Count:Self >>>> ... >>>> >>>> >>>> But it could be because I did not enable -avx during >>>> compilation, hence no there's no vectorization, whatsoever. I >>>> will check this aspect as the other part of my project on >>>> performance tuning. >>>> >>>> Thanks a lot, >>>> Tania >>>> >>>> ------------------------------------------------------------------------ >>>> *From:* Vitali A. Morozov <morozov(a)anl.gov> >>>> <mailto:[email protected]> >>>> *Sent:* Wednesday, February 22, 2017 6:08 PM >>>> *To:* Hammond, Jeff R; Scott Parker; Banerjee,Tania; >>>> intel-nda(a)jlse.anl.gov <mailto:[email protected]> >>>> *Subject:* Re: [JLSE-Intel-NDA] Measuring throughput on KNL >>>> >>>> In general, scalar code generation may differ from vector code >>>> generation - conditionals area perfect example of it. Also, how >>>> to account an FMA versus a non-FMA instruction? >>>> >>>> Vitali >>>> >>>> >>>> On 02/22/2017 04:54 PM, Hammond, Jeff R wrote: >>>>> >>>>> I agree that PACKED_SIMD and SCALAR_SIMD include non-float >>>>> operations. My point was that if the problem is SIMD float >>>>> ops overcount, then forcing the compiler to turn them all into >>>>> scalar float ops should prevent that. I assume there is a way >>>>> to count scalar float ops, but if not, then I guess this >>>>> doesn't help. >>>>> >>>>> At least for SSE*/AVX/AVX2 code, another way to count flops >>>>> precisely is to use an implementation where the hardware >>>>> supports precise counters. >>>>> >>>>> https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX >>>>> <https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX> >>>>> >>>>> https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX2 >>>>> <https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX2> >>>>> >>>>> Jeff >>>>> >>>>> *From: *Scott Parker <sparker(a)anl.gov> <mailto:[email protected]> >>>>> *Date: *Wednesday, February 22, 2017 at 12:32 PM >>>>> *To: *"Hammond, Jeff R" <jeff.r.hammond(a)intel.com> >>>>> <mailto:[email protected]>, "Banerjee,Tania" >>>>> <tmishra(a)ufl.edu> <mailto:[email protected]>, >>>>> "intel-nda(a)jlse.anl.gov" <mailto:[email protected]> >>>>> <intel-nda(a)jlse.anl.gov> <mailto:[email protected]> >>>>> *Subject: *Re: [JLSE-Intel-NDA] Measuring throughput on KNL >>>>> >>>>> >>>>> I don't think that will get you an accurate flop count either >>>>> on KNL, though it may be closer. >>>>> I'm not sure of the full set of instructions that map into the >>>>> PACKED_SIMD and SCALAR_SIMD >>>>> counters but I'm pretty sure there are some that are not >>>>> arithmatic or floating point. >>>>> >>>>> From past discussions with Dmitry Prohorov from VTune the >>>>> Intel Advisor approach is the only way on KNL >>>>> >>>>> -Scott >>>>> >>>>> On 2/22/17 11:54 AM, Hammond, Jeff R wrote: >>>>> >>>>> If one only wants to count flops, I assume disabling SIMD >>>>> will obviate the overcounting due to masking and such, but >>>>> I have not experimented with this. >>>>> >>>>> -[no-]simd >>>>> >>>>> enables(DEFAULT)/disables vectorization using simd >>>>> pragma >>>>> >>>>> Jeff >>>>> >>>>> *From: *<intel-nda-bounces(a)lists.jlse.anl.gov> >>>>> <mailto:[email protected]> on behalf of >>>>> Scott Parker <sparker(a)anl.gov> <mailto:[email protected]> >>>>> *Date: *Wednesday, February 22, 2017 at 9:22 AM >>>>> *To: *"Banerjee,Tania" <tmishra(a)ufl.edu> >>>>> <mailto:[email protected]>, "intel-nda(a)jlse.anl.gov" >>>>> <mailto:[email protected]> <intel-nda(a)jlse.anl.gov> >>>>> <mailto:[email protected]> >>>>> *Subject: *Re: [JLSE-Intel-NDA] Measuring throughput on KNL >>>>> >>>>> >>>>> KNL doesn't have hardware counters for flops, which is why >>>>> PAPI doesn't provide PAPI_FP_OPS. >>>>> This is a problem on a number of current Intel platforms. >>>>> >>>>> From Vtune you can get a upper bound flops estimate based >>>>> on the UOPS_RETIRED.PACKED_SIMD >>>>> >>>>> and UOPS_RETIRED.SCALAR_SIMD counters. This assumes things >>>>> like the vector length of instructions >>>>> is always 8, that FMA are used, and there is no masking, >>>>> which is never going to be always true and >>>>> therefore this can only be an upper bound. >>>>> >>>>> From Intel Advisor you can get an accurate FLOP count, but >>>>> this is done via analysis and instrumentation >>>>> of the binary so it will slow the code execution down >>>>> noticeably. Further description is given here >>>>> https://software.intel.com/en-us/articles/intel-advisor-flops >>>>> <https://software.intel.com/en-us/articles/intel-advisor-flops> >>>>> >>>>> -Scott >>>>> >>>>> On 2/21/17 8:33 PM, Banerjee,Tania wrote: >>>>> >>>>> Hi, >>>>> >>>>> How to measure throughput (GFLOPs/second) of a program >>>>> on KNL? >>>>> >>>>> Normally I use PAPI to count the floating point >>>>> operations (PAPI_FP_OPS / PAPI_FP_INS) and divide that >>>>> by the time taken. >>>>> >>>>> But papi_avail reveals that these events are not >>>>> supported on KNL. >>>>> >>>>> The output >>>>> of /soft/perftools/tau/papi-knl/bin/papi_avail is >>>>> given below. >>>>> >>>>> Thanks much, >>>>> >>>>> Tania >>>>> >>>>> ============================================================================= >>>>> >>>>> Available PAPI preset and user defined events plus >>>>> hardware information. >>>>> >>>>> -------------------------------------------------------------------------------- >>>>> >>>>> PAPI Version : 5.4.3.0 >>>>> >>>>> Vendor string and code : GenuineIntel (1) >>>>> >>>>> Model string and code : Intel(R) Xeon Phi(TM) CPU >>>>> 7210 @ 1.30GHz (87) >>>>> >>>>> CPU Revision : 1.000000 >>>>> >>>>> CPUID Info : Family: 6 Model: 87 >>>>> Stepping: 1 >>>>> >>>>> CPU Max Megahertz : 1500 >>>>> >>>>> CPU Min Megahertz : 1000 >>>>> >>>>> Hdw Threads per core : 4 >>>>> >>>>> Cores per Socket : 64 >>>>> >>>>> Sockets : 1 >>>>> >>>>> NUMA Nodes : 1 >>>>> >>>>> CPUs per Node : 256 >>>>> >>>>> Total CPUs : 256 >>>>> >>>>> Running in a VM : no >>>>> >>>>> Number Hardware Counters : 5 >>>>> >>>>> Max Multiplex Counters : 192 >>>>> >>>>> -------------------------------------------------------------------------------- >>>>> >>>>> ================================================================================ >>>>> >>>>> PAPI Preset Events >>>>> >>>>> ================================================================================ >>>>> >>>>> Name Code Avail Deriv Description (Note) >>>>> >>>>> PAPI_L1_DCM 0x80000000 Yes No Level 1 data cache >>>>> misses >>>>> >>>>> PAPI_L1_ICM 0x80000001 Yes No Level 1 >>>>> instruction cache misses >>>>> >>>>> PAPI_L2_DCM 0x80000002 No No Level 2 data cache >>>>> misses >>>>> >>>>> PAPI_L2_ICM 0x80000003 No No Level 2 >>>>> instruction cache misses >>>>> >>>>> PAPI_L3_DCM 0x80000004 No No Level 3 data cache >>>>> misses >>>>> >>>>> PAPI_L3_ICM 0x80000005 No No Level 3 >>>>> instruction cache misses >>>>> >>>>> PAPI_L1_TCM 0x80000006 Yes Yes Level 1 cache misses >>>>> >>>>> PAPI_L2_TCM 0x80000007 Yes No Level 2 cache misses >>>>> >>>>> PAPI_L3_TCM 0x80000008 No No Level 3 cache misses >>>>> >>>>> PAPI_CA_SNP 0x80000009 No No Requests for a snoop >>>>> >>>>> PAPI_CA_SHR 0x8000000a No No Requests for >>>>> exclusive access to shared cache line >>>>> >>>>> PAPI_CA_CLN 0x8000000b No No Requests for >>>>> exclusive access to clean cache line >>>>> >>>>> PAPI_CA_INV 0x8000000c No No Requests for cache >>>>> line invalidation >>>>> >>>>> PAPI_CA_ITV 0x8000000d No No Requests for cache >>>>> line intervention >>>>> >>>>> PAPI_L3_LDM 0x8000000e No No Level 3 load misses >>>>> >>>>> PAPI_L3_STM 0x8000000f No No Level 3 store misses >>>>> >>>>> PAPI_BRU_IDL 0x80000010 No No Cycles branch >>>>> units are idle >>>>> >>>>> PAPI_FXU_IDL 0x80000011 No No Cycles integer >>>>> units are idle >>>>> >>>>> PAPI_FPU_IDL 0x80000012 No No Cycles floating >>>>> point units are idle >>>>> >>>>> PAPI_LSU_IDL 0x80000013 No No Cycles load/store >>>>> units are idle >>>>> >>>>> PAPI_TLB_DM 0x80000014 Yes No Data translation >>>>> lookaside buffer misses >>>>> >>>>> PAPI_TLB_IM 0x80000015 No No Instruction >>>>> translation lookaside buffer misses >>>>> >>>>> PAPI_TLB_TL 0x80000016 No No Total translation >>>>> lookaside buffer misses >>>>> >>>>> PAPI_L1_LDM 0x80000017 Yes No Level 1 load misses >>>>> >>>>> PAPI_L1_STM 0x80000018 No No Level 1 store misses >>>>> >>>>> PAPI_L2_LDM 0x80000019 Yes No Level 2 load misses >>>>> >>>>> PAPI_L2_STM 0x8000001a No No Level 2 store misses >>>>> >>>>> PAPI_BTAC_M 0x8000001b No No Branch target >>>>> address cache misses >>>>> >>>>> PAPI_PRF_DM 0x8000001c No No Data prefetch >>>>> cache misses >>>>> >>>>> PAPI_L3_DCH 0x8000001d No No Level 3 data cache >>>>> hits >>>>> >>>>> PAPI_TLB_SD 0x8000001e No No Translation >>>>> lookaside buffer shootdowns >>>>> >>>>> PAPI_CSR_FAL 0x8000001f No No Failed store >>>>> conditional instructions >>>>> >>>>> PAPI_CSR_SUC 0x80000020 No No Successful store >>>>> conditional instructions >>>>> >>>>> PAPI_CSR_TOT 0x80000021 No No Total store >>>>> conditional instructions >>>>> >>>>> PAPI_MEM_SCY 0x80000022 No No Cycles Stalled >>>>> Waiting for memory accesses >>>>> >>>>> PAPI_MEM_RCY 0x80000023 No No Cycles Stalled >>>>> Waiting for memory Reads >>>>> >>>>> PAPI_MEM_WCY 0x80000024 No No Cycles Stalled >>>>> Waiting for memory writes >>>>> >>>>> PAPI_STL_ICY 0x80000025 Yes No Cycles with no >>>>> instruction issue >>>>> >>>>> PAPI_FUL_ICY 0x80000026 No No Cycles with >>>>> maximum instruction issue >>>>> >>>>> PAPI_STL_CCY 0x80000027 No No Cycles with no >>>>> instructions completed >>>>> >>>>> PAPI_FUL_CCY 0x80000028 No No Cycles with >>>>> maximum instructions completed >>>>> >>>>> PAPI_HW_INT 0x80000029 No No Hardware interrupts >>>>> >>>>> PAPI_BR_UCN 0x8000002a Yes Yes Unconditional >>>>> branch instructions >>>>> >>>>> PAPI_BR_CN 0x8000002b Yes No Conditional branch >>>>> instructions >>>>> >>>>> PAPI_BR_TKN 0x8000002c Yes No Conditional branch >>>>> instructions taken >>>>> >>>>> PAPI_BR_NTK 0x8000002d Yes Yes Conditional branch >>>>> instructions not taken >>>>> >>>>> PAPI_BR_MSP 0x8000002e Yes No Conditional branch >>>>> instructions mispredicted >>>>> >>>>> PAPI_BR_PRC 0x8000002f No No Conditional branch >>>>> instructions correctly predicted >>>>> >>>>> PAPI_FMA_INS 0x80000030 No No FMA instructions >>>>> completed >>>>> >>>>> PAPI_TOT_IIS 0x80000031 No No Instructions issued >>>>> >>>>> PAPI_TOT_INS 0x80000032 Yes No Instructions completed >>>>> >>>>> PAPI_INT_INS 0x80000033 No No Integer instructions >>>>> >>>>> PAPI_FP_INS 0x80000034 No No Floating point >>>>> instructions >>>>> >>>>> PAPI_LD_INS 0x80000035 Yes No Load instructions >>>>> >>>>> PAPI_SR_INS 0x80000036 No No Store instructions >>>>> >>>>> PAPI_BR_INS 0x80000037 Yes No Branch instructions >>>>> >>>>> PAPI_VEC_INS 0x80000038 No No Vector/SIMD >>>>> instructions (could include integer) >>>>> >>>>> PAPI_RES_STL 0x80000039 Yes No Cycles stalled on >>>>> any resource >>>>> >>>>> PAPI_FP_STAL 0x8000003a No No Cycles the FP >>>>> unit(s) are stalled >>>>> >>>>> PAPI_TOT_CYC 0x8000003b Yes No Total cycles >>>>> >>>>> PAPI_LST_INS 0x8000003c Yes Yes Load/store >>>>> instructions completed >>>>> >>>>> PAPI_SYC_INS 0x8000003d No No Synchronization >>>>> instructions completed >>>>> >>>>> PAPI_L1_DCH 0x8000003e No No Level 1 data cache >>>>> hits >>>>> >>>>> PAPI_L2_DCH 0x8000003f No No Level 2 data cache >>>>> hits >>>>> >>>>> PAPI_L1_DCA 0x80000040 Yes Yes Level 1 data cache >>>>> accesses >>>>> >>>>> PAPI_L2_DCA 0x80000041 No No Level 2 data cache >>>>> accesses >>>>> >>>>> PAPI_L3_DCA 0x80000042 No No Level 3 data cache >>>>> accesses >>>>> >>>>> PAPI_L1_DCR 0x80000043 No No Level 1 data cache >>>>> reads >>>>> >>>>> PAPI_L2_DCR 0x80000044 No No Level 2 data cache >>>>> reads >>>>> >>>>> PAPI_L3_DCR 0x80000045 No No Level 3 data cache >>>>> reads >>>>> >>>>> PAPI_L1_DCW 0x80000046 No No Level 1 data cache >>>>> writes >>>>> >>>>> PAPI_L2_DCW 0x80000047 No No Level 2 data cache >>>>> writes >>>>> >>>>> PAPI_L3_DCW 0x80000048 No No Level 3 data cache >>>>> writes >>>>> >>>>> PAPI_L1_ICH 0x80000049 Yes No Level 1 >>>>> instruction cache hits >>>>> >>>>> PAPI_L2_ICH 0x8000004a No No Level 2 >>>>> instruction cache hits >>>>> >>>>> PAPI_L3_ICH 0x8000004b No No Level 3 >>>>> instruction cache hits >>>>> >>>>> PAPI_L1_ICA 0x8000004c Yes No Level 1 >>>>> instruction cache accesses >>>>> >>>>> PAPI_L2_ICA 0x8000004d No No Level 2 >>>>> instruction cache accesses >>>>> >>>>> PAPI_L3_ICA 0x8000004e No No Level 3 >>>>> instruction cache accesses >>>>> >>>>> PAPI_L1_ICR 0x8000004f No No Level 1 >>>>> instruction cache reads >>>>> >>>>> PAPI_L2_ICR 0x80000050 No No Level 2 >>>>> instruction cache reads >>>>> >>>>> PAPI_L3_ICR 0x80000051 No No Level 3 >>>>> instruction cache reads >>>>> >>>>> PAPI_L1_ICW 0x80000052 No No Level 1 >>>>> instruction cache writes >>>>> >>>>> PAPI_L2_ICW 0x80000053 No No Level 2 >>>>> instruction cache writes >>>>> >>>>> PAPI_L3_ICW 0x80000054 No No Level 3 >>>>> instruction cache writes >>>>> >>>>> PAPI_L1_TCH 0x80000055 No No Level 1 total >>>>> cache hits >>>>> >>>>> PAPI_L2_TCH 0x80000056 Yes Yes Level 2 total >>>>> cache hits >>>>> >>>>> PAPI_L3_TCH 0x80000057 No No Level 3 total >>>>> cache hits >>>>> >>>>> PAPI_L1_TCA 0x80000058 No No Level 1 total >>>>> cache accesses >>>>> >>>>> PAPI_L2_TCA 0x80000059 Yes No Level 2 total >>>>> cache accesses >>>>> >>>>> PAPI_L3_TCA 0x8000005a No No Level 3 total >>>>> cache accesses >>>>> >>>>> PAPI_L1_TCR 0x8000005b No No Level 1 total >>>>> cache reads >>>>> >>>>> PAPI_L2_TCR 0x8000005c No No Level 2 total >>>>> cache reads >>>>> >>>>> PAPI_L3_TCR 0x8000005d No No Level 3 total >>>>> cache reads >>>>> >>>>> PAPI_L1_TCW 0x8000005e No No Level 1 total >>>>> cache writes >>>>> >>>>> PAPI_L2_TCW 0x8000005f No No Level 2 total >>>>> cache writes >>>>> >>>>> PAPI_L3_TCW 0x80000060 No No Level 3 total >>>>> cache writes >>>>> >>>>> PAPI_FML_INS 0x80000061 No No Floating point >>>>> multiply instructions >>>>> >>>>> PAPI_FAD_INS 0x80000062 No No Floating point add >>>>> instructions >>>>> >>>>> PAPI_FDV_INS 0x80000063 No No Floating point >>>>> divide instructions >>>>> >>>>> PAPI_FSQ_INS 0x80000064 No No Floating point >>>>> square root instructions >>>>> >>>>> PAPI_FNV_INS 0x80000065 No No Floating point >>>>> inverse instructions >>>>> >>>>> PAPI_FP_OPS 0x80000066 No No Floating point >>>>> operations >>>>> >>>>> PAPI_SP_OPS 0x80000067 No No Floating point >>>>> operations; optimized to count scaled single precision >>>>> vector operations >>>>> >>>>> PAPI_DP_OPS 0x80000068 No No Floating point >>>>> operations; optimized to count scaled double precision >>>>> vector operations >>>>> >>>>> PAPI_VEC_SP 0x80000069 No No Single precision >>>>> vector/SIMD instructions >>>>> >>>>> PAPI_VEC_DP 0x8000006a No No Double precision >>>>> vector/SIMD instructions >>>>> >>>>> PAPI_REF_CYC 0x8000006b Yes No Reference clock cycles >>>>> >>>>> -------------------------------------------------------------------------------- >>>>> >>>>> Of 108 possible events, 25 are available, of which 6 >>>>> are derived. >>>>> >>>>> avail.c PASSED >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> >>>>> Intel-nda mailing list >>>>> >>>>> Intel-nda(a)lists.jlse.anl.gov >>>>> <mailto:[email protected]> >>>>> >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>> <https://lists.jlse.anl.gov/mailman/listinfo/intel-nda> >>>>> >>>>> _______________________________________________ >>>>> Intel-nda mailing list >>>>> Intel-nda(a)lists.jlse.anl.gov <mailto:[email protected]> >>>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>>>> <https://lists.jlse.anl.gov/mailman/listinfo/intel-nda> >>> _______________________________________________ Intel-nda >>> mailing list Intel-nda(a)lists.jlse.anl.gov >>> <mailto:[email protected]> >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>> <https://lists.jlse.anl.gov/mailman/listinfo/intel-nda> >>> >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Measuring throughput on KNL
by Hammond, Jeff R 24 Feb '17

24 Feb '17

Vitali: HACCmk uses asm or intrinsics to get 82%, does it not? If neither, that is great. Aaron: Call DGEMM :-) Jeff Sent from my iPhone On Feb 24, 2017, at 7:43 AM, Vitali A. Morozov <morozov(a)anl.gov<mailto:[email protected]>> wrote: Yes, it is possible. I was doing this for HACCmk and reached 82% of peak. Vitali On 02/24/2017 09:37 AM, Aaron Knoll wrote: Along these lines: Is it possible to write some very simple compute-only code (in either C++ or Fortran) that when compiled would achieve peak flops (or close to it) on KNL, and measure actual throughput? @Jeff what would you recommend for this? -Aaron On Thu, Feb 23, 2017 at 5:12 PM, Scott Parker <sparker(a)anl.gov<mailto:[email protected]>> wrote: You should use /soft/compilers/intel/advisor -Scott On 2/23/17 5:59 PM, Banerjee,Tania wrote: Hi All, Thanks for pitching in. Scott: Which advisor should we use in /soft/compilers/intel ? [tania@jlselogin1 intel]$ ls advisor advisor/ advisor_xe/ advisor_2017/ advisor_xe_2016/ advisor_2017.1.0.477503/ advisor_xe_2016.1.30.450722/ advisor_2017.1.1.486553/ advisor_xe_2016.1.40.463413/ Also about Intel VTunes, I could get only the following summary (using both hpc-performance and memory-access collection modes) , which did not feature the events talked about in this thread: Event summary ------------- Hardware Event Type Hardware Event Count:Self Hardware Event Sample Count:Self Events Per Sample --------------------------------- ------------------------- -------------------------------- ----------------- INST_RETIRED.ANY 1080573000000 83121 13000000 CPU_CLK_UNHALTED.THREAD 2283515000000 175655 13000000 CPU_CLK_UNHALTED.REF_TSC 2120508000000 163116 13000000 MEM_UOPS_RETIRED.L2_MISS_LOADS_PS 396011880 132 1000030 UOPS_RETIRED.ALL 1479002218500 24650 20000030 MEM_UOPS_RETIRED.L2_HIT_LOADS_PS 2079062370<tel:%28207%29%20906-2370> 693 1000030 NO_ALLOC_CYCLES.MISPREDICTS 11220016830 187 20000030 NO_ALLOC_CYCLES.NOT_DELIVERED 720601080900 12010 20000030 Uncore Event summary -------------------- Uncore Event Type Uncore Event Count:Self ... But it could be because I did not enable -avx during compilation, hence no there's no vectorization, whatsoever. I will check this aspect as the other part of my project on performance tuning. Thanks a lot, Tania ________________________________ From: Vitali A. Morozov <morozov(a)anl.gov><mailto:[email protected]> Sent: Wednesday, February 22, 2017 6:08 PM To: Hammond, Jeff R; Scott Parker; Banerjee,Tania; intel-nda(a)jlse.anl.gov<mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] Measuring throughput on KNL In general, scalar code generation may differ from vector code generation - conditionals area perfect example of it. Also, how to account an FMA versus a non-FMA instruction? Vitali On 02/22/2017 04:54 PM, Hammond, Jeff R wrote: I agree that PACKED_SIMD and SCALAR_SIMD include non-float operations. My point was that if the problem is SIMD float ops overcount, then forcing the compiler to turn them all into scalar float ops should prevent that. I assume there is a way to count scalar float ops, but if not, then I guess this doesn't help. At least for SSE*/AVX/AVX2 code, another way to count flops precisely is to use an implementation where the hardware supports precise counters. https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX2 Jeff From: Scott Parker <sparker(a)anl.gov><mailto:[email protected]> Date: Wednesday, February 22, 2017 at 12:32 PM To: "Hammond, Jeff R" <jeff.r.hammond(a)intel.com><mailto:[email protected]>, "Banerjee,Tania" <tmishra(a)ufl.edu><mailto:[email protected]>, "intel-nda(a)jlse.anl.gov"<mailto:[email protected]> <intel-nda(a)jlse.anl.gov><mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] Measuring throughput on KNL I don't think that will get you an accurate flop count either on KNL, though it may be closer. I'm not sure of the full set of instructions that map into the PACKED_SIMD and SCALAR_SIMD counters but I'm pretty sure there are some that are not arithmatic or floating point. From past discussions with Dmitry Prohorov from VTune the Intel Advisor approach is the only way on KNL -Scott On 2/22/17 11:54 AM, Hammond, Jeff R wrote: If one only wants to count flops, I assume disabling SIMD will obviate the overcounting due to masking and such, but I have not experimented with this. -[no-]simd enables(DEFAULT)/disables vectorization using simd pragma Jeff From: <intel-nda-bounces(a)lists.jlse.anl.gov><mailto:[email protected]> on behalf of Scott Parker <sparker(a)anl.gov><mailto:[email protected]> Date: Wednesday, February 22, 2017 at 9:22 AM To: "Banerjee,Tania" <tmishra(a)ufl.edu><mailto:[email protected]>, "intel-nda(a)jlse.anl.gov"<mailto:[email protected]> <intel-nda(a)jlse.anl.gov><mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] Measuring throughput on KNL KNL doesn't have hardware counters for flops, which is why PAPI doesn't provide PAPI_FP_OPS. This is a problem on a number of current Intel platforms. From Vtune you can get a upper bound flops estimate based on the UOPS_RETIRED.PACKED_SIMD and UOPS_RETIRED.SCALAR_SIMD counters. This assumes things like the vector length of instructions is always 8, that FMA are used, and there is no masking, which is never going to be always true and therefore this can only be an upper bound. From Intel Advisor you can get an accurate FLOP count, but this is done via analysis and instrumentation of the binary so it will slow the code execution down noticeably. Further description is given here https://software.intel.com/en-us/articles/intel-advisor-flops -Scott On 2/21/17 8:33 PM, Banerjee,Tania wrote: Hi, How to measure throughput (GFLOPs/second) of a program on KNL? Normally I use PAPI to count the floating point operations (PAPI_FP_OPS / PAPI_FP_INS) and divide that by the time taken. But papi_avail reveals that these events are not supported on KNL. The output of /soft/perftools/tau/papi-knl/bin/papi_avail is given below. Thanks much, Tania ============================================================================= Available PAPI preset and user defined events plus hardware information. -------------------------------------------------------------------------------- PAPI Version : 5.4.3.0 Vendor string and code : GenuineIntel (1) Model string and code : Intel(R) Xeon Phi(TM) CPU 7210 @ 1.30GHz (87) CPU Revision : 1.000000 CPUID Info : Family: 6 Model: 87 Stepping: 1 CPU Max Megahertz : 1500 CPU Min Megahertz : 1000 Hdw Threads per core : 4 Cores per Socket : 64 Sockets : 1 NUMA Nodes : 1 CPUs per Node : 256 Total CPUs : 256 Running in a VM : no Number Hardware Counters : 5 Max Multiplex Counters : 192 -------------------------------------------------------------------------------- ================================================================================ PAPI Preset Events ================================================================================ Name Code Avail Deriv Description (Note) PAPI_L1_DCM 0x80000000 Yes No Level 1 data cache misses PAPI_L1_ICM 0x80000001 Yes No Level 1 instruction cache misses PAPI_L2_DCM 0x80000002 No No Level 2 data cache misses PAPI_L2_ICM 0x80000003 No No Level 2 instruction cache misses PAPI_L3_DCM 0x80000004 No No Level 3 data cache misses PAPI_L3_ICM 0x80000005 No No Level 3 instruction cache misses PAPI_L1_TCM 0x80000006 Yes Yes Level 1 cache misses PAPI_L2_TCM 0x80000007 Yes No Level 2 cache misses PAPI_L3_TCM 0x80000008 No No Level 3 cache misses PAPI_CA_SNP 0x80000009 No No Requests for a snoop PAPI_CA_SHR 0x8000000a No No Requests for exclusive access to shared cache line PAPI_CA_CLN 0x8000000b No No Requests for exclusive access to clean cache line PAPI_CA_INV 0x8000000c No No Requests for cache line invalidation PAPI_CA_ITV 0x8000000d No No Requests for cache line intervention PAPI_L3_LDM 0x8000000e No No Level 3 load misses PAPI_L3_STM 0x8000000f No No Level 3 store misses PAPI_BRU_IDL 0x80000010 No No Cycles branch units are idle PAPI_FXU_IDL 0x80000011 No No Cycles integer units are idle PAPI_FPU_IDL 0x80000012 No No Cycles floating point units are idle PAPI_LSU_IDL 0x80000013 No No Cycles load/store units are idle PAPI_TLB_DM 0x80000014 Yes No Data translation lookaside buffer misses PAPI_TLB_IM 0x80000015 No No Instruction translation lookaside buffer misses PAPI_TLB_TL 0x80000016 No No Total translation lookaside buffer misses PAPI_L1_LDM 0x80000017 Yes No Level 1 load misses PAPI_L1_STM 0x80000018 No No Level 1 store misses PAPI_L2_LDM 0x80000019 Yes No Level 2 load misses PAPI_L2_STM 0x8000001a No No Level 2 store misses PAPI_BTAC_M 0x8000001b No No Branch target address cache misses PAPI_PRF_DM 0x8000001c No No Data prefetch cache misses PAPI_L3_DCH 0x8000001d No No Level 3 data cache hits PAPI_TLB_SD 0x8000001e No No Translation lookaside buffer shootdowns PAPI_CSR_FAL 0x8000001f No No Failed store conditional instructions PAPI_CSR_SUC 0x80000020 No No Successful store conditional instructions PAPI_CSR_TOT 0x80000021 No No Total store conditional instructions PAPI_MEM_SCY 0x80000022 No No Cycles Stalled Waiting for memory accesses PAPI_MEM_RCY 0x80000023 No No Cycles Stalled Waiting for memory Reads PAPI_MEM_WCY 0x80000024 No No Cycles Stalled Waiting for memory writes PAPI_STL_ICY 0x80000025 Yes No Cycles with no instruction issue PAPI_FUL_ICY 0x80000026 No No Cycles with maximum instruction issue PAPI_STL_CCY 0x80000027 No No Cycles with no instructions completed PAPI_FUL_CCY 0x80000028 No No Cycles with maximum instructions completed PAPI_HW_INT 0x80000029 No No Hardware interrupts PAPI_BR_UCN 0x8000002a Yes Yes Unconditional branch instructions PAPI_BR_CN 0x8000002b Yes No Conditional branch instructions PAPI_BR_TKN 0x8000002c Yes No Conditional branch instructions taken PAPI_BR_NTK 0x8000002d Yes Yes Conditional branch instructions not taken PAPI_BR_MSP 0x8000002e Yes No Conditional branch instructions mispredicted PAPI_BR_PRC 0x8000002f No No Conditional branch instructions correctly predicted PAPI_FMA_INS 0x80000030 No No FMA instructions completed PAPI_TOT_IIS 0x80000031 No No Instructions issued PAPI_TOT_INS 0x80000032 Yes No Instructions completed PAPI_INT_INS 0x80000033 No No Integer instructions PAPI_FP_INS 0x80000034 No No Floating point instructions PAPI_LD_INS 0x80000035 Yes No Load instructions PAPI_SR_INS 0x80000036 No No Store instructions PAPI_BR_INS 0x80000037 Yes No Branch instructions PAPI_VEC_INS 0x80000038 No No Vector/SIMD instructions (could include integer) PAPI_RES_STL 0x80000039 Yes No Cycles stalled on any resource PAPI_FP_STAL 0x8000003a No No Cycles the FP unit(s) are stalled PAPI_TOT_CYC 0x8000003b Yes No Total cycles PAPI_LST_INS 0x8000003c Yes Yes Load/store instructions completed PAPI_SYC_INS 0x8000003d No No Synchronization instructions completed PAPI_L1_DCH 0x8000003e No No Level 1 data cache hits PAPI_L2_DCH 0x8000003f No No Level 2 data cache hits PAPI_L1_DCA 0x80000040 Yes Yes Level 1 data cache accesses PAPI_L2_DCA 0x80000041 No No Level 2 data cache accesses PAPI_L3_DCA 0x80000042 No No Level 3 data cache accesses PAPI_L1_DCR 0x80000043 No No Level 1 data cache reads PAPI_L2_DCR 0x80000044 No No Level 2 data cache reads PAPI_L3_DCR 0x80000045 No No Level 3 data cache reads PAPI_L1_DCW 0x80000046 No No Level 1 data cache writes PAPI_L2_DCW 0x80000047 No No Level 2 data cache writes PAPI_L3_DCW 0x80000048 No No Level 3 data cache writes PAPI_L1_ICH 0x80000049 Yes No Level 1 instruction cache hits PAPI_L2_ICH 0x8000004a No No Level 2 instruction cache hits PAPI_L3_ICH 0x8000004b No No Level 3 instruction cache hits PAPI_L1_ICA 0x8000004c Yes No Level 1 instruction cache accesses PAPI_L2_ICA 0x8000004d No No Level 2 instruction cache accesses PAPI_L3_ICA 0x8000004e No No Level 3 instruction cache accesses PAPI_L1_ICR 0x8000004f No No Level 1 instruction cache reads PAPI_L2_ICR 0x80000050 No No Level 2 instruction cache reads PAPI_L3_ICR 0x80000051 No No Level 3 instruction cache reads PAPI_L1_ICW 0x80000052 No No Level 1 instruction cache writes PAPI_L2_ICW 0x80000053 No No Level 2 instruction cache writes PAPI_L3_ICW 0x80000054 No No Level 3 instruction cache writes PAPI_L1_TCH 0x80000055 No No Level 1 total cache hits PAPI_L2_TCH 0x80000056 Yes Yes Level 2 total cache hits PAPI_L3_TCH 0x80000057 No No Level 3 total cache hits PAPI_L1_TCA 0x80000058 No No Level 1 total cache accesses PAPI_L2_TCA 0x80000059 Yes No Level 2 total cache accesses PAPI_L3_TCA 0x8000005a No No Level 3 total cache accesses PAPI_L1_TCR 0x8000005b No No Level 1 total cache reads PAPI_L2_TCR 0x8000005c No No Level 2 total cache reads PAPI_L3_TCR 0x8000005d No No Level 3 total cache reads PAPI_L1_TCW 0x8000005e No No Level 1 total cache writes PAPI_L2_TCW 0x8000005f No No Level 2 total cache writes PAPI_L3_TCW 0x80000060 No No Level 3 total cache writes PAPI_FML_INS 0x80000061 No No Floating point multiply instructions PAPI_FAD_INS 0x80000062 No No Floating point add instructions PAPI_FDV_INS 0x80000063 No No Floating point divide instructions PAPI_FSQ_INS 0x80000064 No No Floating point square root instructions PAPI_FNV_INS 0x80000065 No No Floating point inverse instructions PAPI_FP_OPS 0x80000066 No No Floating point operations PAPI_SP_OPS 0x80000067 No No Floating point operations; optimized to count scaled single precision vector operations PAPI_DP_OPS 0x80000068 No No Floating point operations; optimized to count scaled double precision vector operations PAPI_VEC_SP 0x80000069 No No Single precision vector/SIMD instructions PAPI_VEC_DP 0x8000006a No No Double precision vector/SIMD instructions PAPI_REF_CYC 0x8000006b Yes No Reference clock cycles -------------------------------------------------------------------------------- Of 108 possible events, 25 are available, of which 6 are derived. avail.c PASSED _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Measuring throughput on KNL
by Vitali A. Morozov 24 Feb '17

24 Feb '17

Yes, it is possible. I was doing this for HACCmk and reached 82% of peak. Vitali On 02/24/2017 09:37 AM, Aaron Knoll wrote: > Along these lines: > > Is it possible to write some very simple compute-only code (in either > C++ or Fortran) that when compiled would achieve peak flops (or close > to it) on KNL, and measure actual throughput? > > @Jeff what would you recommend for this? > > -Aaron > > On Thu, Feb 23, 2017 at 5:12 PM, Scott Parker <sparker(a)anl.gov > <mailto:[email protected]>> wrote: > > > You should use /soft/compilers/intel/advisor > > > -Scott > > > On 2/23/17 5:59 PM, Banerjee,Tania wrote: >> >> Hi All, >> >> >> Thanks for pitching in. >> >> >> Scott: >> >> Which advisor should we use in /soft/compilers/intel ? >> >> >> [tania@jlselogin1 intel]$ ls advisor >> advisor/ advisor_xe/ >> advisor_2017/ advisor_xe_2016/ >> advisor_2017.1.0.477503/ advisor_xe_2016.1.30.450722/ >> advisor_2017.1.1.486553/ advisor_xe_2016.1.40.463413/ >> >> Also about Intel VTunes, I could get only the following summary >> (using both hpc-performance and memory-access collection modes) , >> which did not feature the events talked about in this thread: >> >> Event summary >> ------------- >> Hardware Event Type Hardware Event Count:Self Hardware >> Event Sample Count:Self Events Per Sample >> --------------------------------- ------------------------- >> -------------------------------- ----------------- >> >> INST_RETIRED.ANY 1080573000000 83121 >> 13000000 >> CPU_CLK_UNHALTED.THREAD 2283515000000 >> 175655 13000000 >> CPU_CLK_UNHALTED.REF_TSC 2120508000000 >> 163116 13000000 >> MEM_UOPS_RETIRED.L2_MISS_LOADS_PS 396011880 >> 132 1000030 >> UOPS_RETIRED.ALL 1479002218500 24650 >> 20000030 >> MEM_UOPS_RETIRED.L2_HIT_LOADS_PS 2079062370 >> <tel:%28207%29%20906-2370> 693 1000030 >> NO_ALLOC_CYCLES.MISPREDICTS 11220016830 >> 187 20000030 >> NO_ALLOC_CYCLES.NOT_DELIVERED 720601080900 >> 12010 20000030 >> >> Uncore Event summary >> -------------------- >> Uncore Event Type Uncore Event Count:Self >> ... >> >> >> But it could be because I did not enable -avx during compilation, >> hence no there's no vectorization, whatsoever. I will check this >> aspect as the other part of my project on performance tuning. >> >> Thanks a lot, >> Tania >> >> ------------------------------------------------------------------------ >> *From:* Vitali A. Morozov <morozov(a)anl.gov> <mailto:[email protected]> >> *Sent:* Wednesday, February 22, 2017 6:08 PM >> *To:* Hammond, Jeff R; Scott Parker; Banerjee,Tania; >> intel-nda(a)jlse.anl.gov <mailto:[email protected]> >> *Subject:* Re: [JLSE-Intel-NDA] Measuring throughput on KNL >> >> In general, scalar code generation may differ from vector code >> generation - conditionals area perfect example of it. Also, how >> to account an FMA versus a non-FMA instruction? >> >> Vitali >> >> >> On 02/22/2017 04:54 PM, Hammond, Jeff R wrote: >>> >>> I agree that PACKED_SIMD and SCALAR_SIMD include non-float >>> operations. My point was that if the problem is SIMD float ops >>> overcount, then forcing the compiler to turn them all into >>> scalar float ops should prevent that. I assume there is a way >>> to count scalar float ops, but if not, then I guess this doesn't >>> help. >>> >>> At least for SSE*/AVX/AVX2 code, another way to count flops >>> precisely is to use an implementation where the hardware >>> supports precise counters. >>> >>> https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX >>> <https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX> >>> >>> https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX2 >>> <https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX2> >>> >>> Jeff >>> >>> *From: *Scott Parker <sparker(a)anl.gov> <mailto:[email protected]> >>> *Date: *Wednesday, February 22, 2017 at 12:32 PM >>> *To: *"Hammond, Jeff R" <jeff.r.hammond(a)intel.com> >>> <mailto:[email protected]>, "Banerjee,Tania" >>> <tmishra(a)ufl.edu> <mailto:[email protected]>, >>> "intel-nda(a)jlse.anl.gov" <mailto:[email protected]> >>> <intel-nda(a)jlse.anl.gov> <mailto:[email protected]> >>> *Subject: *Re: [JLSE-Intel-NDA] Measuring throughput on KNL >>> >>> >>> I don't think that will get you an accurate flop count either on >>> KNL, though it may be closer. >>> I'm not sure of the full set of instructions that map into the >>> PACKED_SIMD and SCALAR_SIMD >>> counters but I'm pretty sure there are some that are not >>> arithmatic or floating point. >>> >>> From past discussions with Dmitry Prohorov from VTune the Intel >>> Advisor approach is the only way on KNL >>> >>> -Scott >>> >>> On 2/22/17 11:54 AM, Hammond, Jeff R wrote: >>> >>> If one only wants to count flops, I assume disabling SIMD >>> will obviate the overcounting due to masking and such, but I >>> have not experimented with this. >>> >>> -[no-]simd >>> >>> enables(DEFAULT)/disables vectorization using simd pragma >>> >>> Jeff >>> >>> *From: *<intel-nda-bounces(a)lists.jlse.anl.gov> >>> <mailto:[email protected]> on behalf of >>> Scott Parker <sparker(a)anl.gov> <mailto:[email protected]> >>> *Date: *Wednesday, February 22, 2017 at 9:22 AM >>> *To: *"Banerjee,Tania" <tmishra(a)ufl.edu> >>> <mailto:[email protected]>, "intel-nda(a)jlse.anl.gov" >>> <mailto:[email protected]> <intel-nda(a)jlse.anl.gov> >>> <mailto:[email protected]> >>> *Subject: *Re: [JLSE-Intel-NDA] Measuring throughput on KNL >>> >>> >>> KNL doesn't have hardware counters for flops, which is why >>> PAPI doesn't provide PAPI_FP_OPS. >>> This is a problem on a number of current Intel platforms. >>> >>> From Vtune you can get a upper bound flops estimate based on >>> the UOPS_RETIRED.PACKED_SIMD >>> >>> and UOPS_RETIRED.SCALAR_SIMD counters. This assumes things >>> like the vector length of instructions >>> is always 8, that FMA are used, and there is no masking, >>> which is never going to be always true and >>> therefore this can only be an upper bound. >>> >>> From Intel Advisor you can get an accurate FLOP count, but >>> this is done via analysis and instrumentation >>> of the binary so it will slow the code execution down >>> noticeably. Further description is given here >>> https://software.intel.com/en-us/articles/intel-advisor-flops >>> <https://software.intel.com/en-us/articles/intel-advisor-flops> >>> >>> -Scott >>> >>> On 2/21/17 8:33 PM, Banerjee,Tania wrote: >>> >>> Hi, >>> >>> How to measure throughput (GFLOPs/second) of a program >>> on KNL? >>> >>> Normally I use PAPI to count the floating point >>> operations (PAPI_FP_OPS / PAPI_FP_INS) and divide that >>> by the time taken. >>> >>> But papi_avail reveals that these events are not >>> supported on KNL. >>> >>> The output >>> of /soft/perftools/tau/papi-knl/bin/papi_avail is given >>> below. >>> >>> Thanks much, >>> >>> Tania >>> >>> ============================================================================= >>> >>> Available PAPI preset and user defined events plus >>> hardware information. >>> >>> -------------------------------------------------------------------------------- >>> >>> PAPI Version : 5.4.3.0 >>> >>> Vendor string and code : GenuineIntel (1) >>> >>> Model string and code : Intel(R) Xeon Phi(TM) CPU >>> 7210 @ 1.30GHz (87) >>> >>> CPU Revision : 1.000000 >>> >>> CPUID Info : Family: 6 Model: 87 Stepping: 1 >>> >>> CPU Max Megahertz : 1500 >>> >>> CPU Min Megahertz : 1000 >>> >>> Hdw Threads per core : 4 >>> >>> Cores per Socket : 64 >>> >>> Sockets : 1 >>> >>> NUMA Nodes : 1 >>> >>> CPUs per Node : 256 >>> >>> Total CPUs : 256 >>> >>> Running in a VM : no >>> >>> Number Hardware Counters : 5 >>> >>> Max Multiplex Counters : 192 >>> >>> -------------------------------------------------------------------------------- >>> >>> ================================================================================ >>> >>> PAPI Preset Events >>> >>> ================================================================================ >>> >>> Name Code Avail Deriv Description (Note) >>> >>> PAPI_L1_DCM 0x80000000 Yes No Level 1 data cache >>> misses >>> >>> PAPI_L1_ICM 0x80000001 Yes No Level 1 instruction >>> cache misses >>> >>> PAPI_L2_DCM 0x80000002 No No Level 2 data cache >>> misses >>> >>> PAPI_L2_ICM 0x80000003 No No Level 2 instruction >>> cache misses >>> >>> PAPI_L3_DCM 0x80000004 No No Level 3 data cache >>> misses >>> >>> PAPI_L3_ICM 0x80000005 No No Level 3 instruction >>> cache misses >>> >>> PAPI_L1_TCM 0x80000006 Yes Yes Level 1 cache misses >>> >>> PAPI_L2_TCM 0x80000007 Yes No Level 2 cache misses >>> >>> PAPI_L3_TCM 0x80000008 No No Level 3 cache misses >>> >>> PAPI_CA_SNP 0x80000009 No No Requests for a snoop >>> >>> PAPI_CA_SHR 0x8000000a No No Requests for >>> exclusive access to shared cache line >>> >>> PAPI_CA_CLN 0x8000000b No No Requests for >>> exclusive access to clean cache line >>> >>> PAPI_CA_INV 0x8000000c No No Requests for cache >>> line invalidation >>> >>> PAPI_CA_ITV 0x8000000d No No Requests for cache >>> line intervention >>> >>> PAPI_L3_LDM 0x8000000e No No Level 3 load misses >>> >>> PAPI_L3_STM 0x8000000f No No Level 3 store misses >>> >>> PAPI_BRU_IDL 0x80000010 No No Cycles branch units >>> are idle >>> >>> PAPI_FXU_IDL 0x80000011 No No Cycles integer units >>> are idle >>> >>> PAPI_FPU_IDL 0x80000012 No No Cycles floating >>> point units are idle >>> >>> PAPI_LSU_IDL 0x80000013 No No Cycles load/store >>> units are idle >>> >>> PAPI_TLB_DM 0x80000014 Yes No Data translation >>> lookaside buffer misses >>> >>> PAPI_TLB_IM 0x80000015 No No Instruction >>> translation lookaside buffer misses >>> >>> PAPI_TLB_TL 0x80000016 No No Total translation >>> lookaside buffer misses >>> >>> PAPI_L1_LDM 0x80000017 Yes No Level 1 load misses >>> >>> PAPI_L1_STM 0x80000018 No No Level 1 store misses >>> >>> PAPI_L2_LDM 0x80000019 Yes No Level 2 load misses >>> >>> PAPI_L2_STM 0x8000001a No No Level 2 store misses >>> >>> PAPI_BTAC_M 0x8000001b No No Branch target >>> address cache misses >>> >>> PAPI_PRF_DM 0x8000001c No No Data prefetch cache >>> misses >>> >>> PAPI_L3_DCH 0x8000001d No No Level 3 data cache hits >>> >>> PAPI_TLB_SD 0x8000001e No No Translation >>> lookaside buffer shootdowns >>> >>> PAPI_CSR_FAL 0x8000001f No No Failed store >>> conditional instructions >>> >>> PAPI_CSR_SUC 0x80000020 No No Successful store >>> conditional instructions >>> >>> PAPI_CSR_TOT 0x80000021 No No Total store >>> conditional instructions >>> >>> PAPI_MEM_SCY 0x80000022 No No Cycles Stalled >>> Waiting for memory accesses >>> >>> PAPI_MEM_RCY 0x80000023 No No Cycles Stalled >>> Waiting for memory Reads >>> >>> PAPI_MEM_WCY 0x80000024 No No Cycles Stalled >>> Waiting for memory writes >>> >>> PAPI_STL_ICY 0x80000025 Yes No Cycles with no >>> instruction issue >>> >>> PAPI_FUL_ICY 0x80000026 No No Cycles with maximum >>> instruction issue >>> >>> PAPI_STL_CCY 0x80000027 No No Cycles with no >>> instructions completed >>> >>> PAPI_FUL_CCY 0x80000028 No No Cycles with maximum >>> instructions completed >>> >>> PAPI_HW_INT 0x80000029 No No Hardware interrupts >>> >>> PAPI_BR_UCN 0x8000002a Yes Yes Unconditional branch >>> instructions >>> >>> PAPI_BR_CN 0x8000002b Yes No Conditional branch >>> instructions >>> >>> PAPI_BR_TKN 0x8000002c Yes No Conditional branch >>> instructions taken >>> >>> PAPI_BR_NTK 0x8000002d Yes Yes Conditional branch >>> instructions not taken >>> >>> PAPI_BR_MSP 0x8000002e Yes No Conditional branch >>> instructions mispredicted >>> >>> PAPI_BR_PRC 0x8000002f No No Conditional branch >>> instructions correctly predicted >>> >>> PAPI_FMA_INS 0x80000030 No No FMA instructions >>> completed >>> >>> PAPI_TOT_IIS 0x80000031 No No Instructions issued >>> >>> PAPI_TOT_INS 0x80000032 Yes No Instructions completed >>> >>> PAPI_INT_INS 0x80000033 No No Integer instructions >>> >>> PAPI_FP_INS 0x80000034 No No Floating point >>> instructions >>> >>> PAPI_LD_INS 0x80000035 Yes No Load instructions >>> >>> PAPI_SR_INS 0x80000036 No No Store instructions >>> >>> PAPI_BR_INS 0x80000037 Yes No Branch instructions >>> >>> PAPI_VEC_INS 0x80000038 No No Vector/SIMD >>> instructions (could include integer) >>> >>> PAPI_RES_STL 0x80000039 Yes No Cycles stalled on >>> any resource >>> >>> PAPI_FP_STAL 0x8000003a No No Cycles the FP >>> unit(s) are stalled >>> >>> PAPI_TOT_CYC 0x8000003b Yes No Total cycles >>> >>> PAPI_LST_INS 0x8000003c Yes Yes Load/store >>> instructions completed >>> >>> PAPI_SYC_INS 0x8000003d No No Synchronization >>> instructions completed >>> >>> PAPI_L1_DCH 0x8000003e No No Level 1 data cache hits >>> >>> PAPI_L2_DCH 0x8000003f No No Level 2 data cache hits >>> >>> PAPI_L1_DCA 0x80000040 Yes Yes Level 1 data cache >>> accesses >>> >>> PAPI_L2_DCA 0x80000041 No No Level 2 data cache >>> accesses >>> >>> PAPI_L3_DCA 0x80000042 No No Level 3 data cache >>> accesses >>> >>> PAPI_L1_DCR 0x80000043 No No Level 1 data cache reads >>> >>> PAPI_L2_DCR 0x80000044 No No Level 2 data cache reads >>> >>> PAPI_L3_DCR 0x80000045 No No Level 3 data cache reads >>> >>> PAPI_L1_DCW 0x80000046 No No Level 1 data cache >>> writes >>> >>> PAPI_L2_DCW 0x80000047 No No Level 2 data cache >>> writes >>> >>> PAPI_L3_DCW 0x80000048 No No Level 3 data cache >>> writes >>> >>> PAPI_L1_ICH 0x80000049 Yes No Level 1 instruction >>> cache hits >>> >>> PAPI_L2_ICH 0x8000004a No No Level 2 instruction >>> cache hits >>> >>> PAPI_L3_ICH 0x8000004b No No Level 3 instruction >>> cache hits >>> >>> PAPI_L1_ICA 0x8000004c Yes No Level 1 instruction >>> cache accesses >>> >>> PAPI_L2_ICA 0x8000004d No No Level 2 instruction >>> cache accesses >>> >>> PAPI_L3_ICA 0x8000004e No No Level 3 instruction >>> cache accesses >>> >>> PAPI_L1_ICR 0x8000004f No No Level 1 instruction >>> cache reads >>> >>> PAPI_L2_ICR 0x80000050 No No Level 2 instruction >>> cache reads >>> >>> PAPI_L3_ICR 0x80000051 No No Level 3 instruction >>> cache reads >>> >>> PAPI_L1_ICW 0x80000052 No No Level 1 instruction >>> cache writes >>> >>> PAPI_L2_ICW 0x80000053 No No Level 2 instruction >>> cache writes >>> >>> PAPI_L3_ICW 0x80000054 No No Level 3 instruction >>> cache writes >>> >>> PAPI_L1_TCH 0x80000055 No No Level 1 total cache hits >>> >>> PAPI_L2_TCH 0x80000056 Yes Yes Level 2 total cache hits >>> >>> PAPI_L3_TCH 0x80000057 No No Level 3 total cache hits >>> >>> PAPI_L1_TCA 0x80000058 No No Level 1 total cache >>> accesses >>> >>> PAPI_L2_TCA 0x80000059 Yes No Level 2 total cache >>> accesses >>> >>> PAPI_L3_TCA 0x8000005a No No Level 3 total cache >>> accesses >>> >>> PAPI_L1_TCR 0x8000005b No No Level 1 total cache >>> reads >>> >>> PAPI_L2_TCR 0x8000005c No No Level 2 total cache >>> reads >>> >>> PAPI_L3_TCR 0x8000005d No No Level 3 total cache >>> reads >>> >>> PAPI_L1_TCW 0x8000005e No No Level 1 total cache >>> writes >>> >>> PAPI_L2_TCW 0x8000005f No No Level 2 total cache >>> writes >>> >>> PAPI_L3_TCW 0x80000060 No No Level 3 total cache >>> writes >>> >>> PAPI_FML_INS 0x80000061 No No Floating point >>> multiply instructions >>> >>> PAPI_FAD_INS 0x80000062 No No Floating point add >>> instructions >>> >>> PAPI_FDV_INS 0x80000063 No No Floating point >>> divide instructions >>> >>> PAPI_FSQ_INS 0x80000064 No No Floating point >>> square root instructions >>> >>> PAPI_FNV_INS 0x80000065 No No Floating point >>> inverse instructions >>> >>> PAPI_FP_OPS 0x80000066 No No Floating point >>> operations >>> >>> PAPI_SP_OPS 0x80000067 No No Floating point >>> operations; optimized to count scaled single precision >>> vector operations >>> >>> PAPI_DP_OPS 0x80000068 No No Floating point >>> operations; optimized to count scaled double precision >>> vector operations >>> >>> PAPI_VEC_SP 0x80000069 No No Single precision >>> vector/SIMD instructions >>> >>> PAPI_VEC_DP 0x8000006a No No Double precision >>> vector/SIMD instructions >>> >>> PAPI_REF_CYC 0x8000006b Yes No Reference clock cycles >>> >>> -------------------------------------------------------------------------------- >>> >>> Of 108 possible events, 25 are available, of which 6 are >>> derived. >>> >>> avail.c PASSED >>> >>> >>> >>> >>> >>> _______________________________________________ >>> >>> Intel-nda mailing list >>> >>> Intel-nda(a)lists.jlse.anl.gov >>> <mailto:[email protected]> >>> >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>> <https://lists.jlse.anl.gov/mailman/listinfo/intel-nda> >>> >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov <mailto:[email protected]> >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>> <https://lists.jlse.anl.gov/mailman/listinfo/intel-nda> >> > _______________________________________________ Intel-nda mailing > list Intel-nda(a)lists.jlse.anl.gov > <mailto:[email protected]> > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > <https://lists.jlse.anl.gov/mailman/listinfo/intel-nda> > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Measuring throughput on KNL
by Aaron Knoll 24 Feb '17

24 Feb '17

Along these lines: Is it possible to write some very simple compute-only code (in either C++ or Fortran) that when compiled would achieve peak flops (or close to it) on KNL, and measure actual throughput? @Jeff what would you recommend for this? -Aaron On Thu, Feb 23, 2017 at 5:12 PM, Scott Parker <sparker(a)anl.gov> wrote: > > You should use /soft/compilers/intel/advisor > > > -Scott > > On 2/23/17 5:59 PM, Banerjee,Tania wrote: > > Hi All, > > > Thanks for pitching in. > > > Scott: > > Which advisor should we use in /soft/compilers/intel ? > > > [tania@jlselogin1 intel]$ ls advisor > advisor/ advisor_xe/ > advisor_2017/ advisor_xe_2016/ > advisor_2017.1.0.477503/ advisor_xe_2016.1.30.450722/ > advisor_2017.1.1.486553/ advisor_xe_2016.1.40.463413/ > > Also about Intel VTunes, I could get only the following summary (using > both hpc-performance and memory-access collection modes) , which did not > feature the events talked about in this thread: > > Event summary > ------------- > Hardware Event Type Hardware Event Count:Self Hardware > Event Sample Count:Self Events Per Sample > --------------------------------- ------------------------- > -------------------------------- ----------------- > > INST_RETIRED.ANY 1080573000000 > 83121 13000000 > CPU_CLK_UNHALTED.THREAD 2283515000000 > 175655 13000000 > CPU_CLK_UNHALTED.REF_TSC 2120508000000 > 163116 13000000 > MEM_UOPS_RETIRED.L2_MISS_LOADS_PS 396011880 > 132 1000030 > UOPS_RETIRED.ALL 1479002218500 > 24650 20000030 > MEM_UOPS_RETIRED.L2_HIT_LOADS_PS 2079062370 > <(207)%20906-2370> 693 1000030 > NO_ALLOC_CYCLES.MISPREDICTS 11220016830 > 187 20000030 > NO_ALLOC_CYCLES.NOT_DELIVERED 720601080900 > 12010 20000030 > > Uncore Event summary > -------------------- > Uncore Event Type Uncore Event Count:Self > ... > > > But it could be because I did not enable -avx during compilation, hence no > there's no vectorization, whatsoever. I will check this aspect as the other > part of my project on performance tuning. > > Thanks a lot, > Tania > > ------------------------------ > *From:* Vitali A. Morozov <morozov(a)anl.gov> <morozov(a)anl.gov> > *Sent:* Wednesday, February 22, 2017 6:08 PM > *To:* Hammond, Jeff R; Scott Parker; Banerjee,Tania; > intel-nda(a)jlse.anl.gov > *Subject:* Re: [JLSE-Intel-NDA] Measuring throughput on KNL > > > In general, scalar code generation may differ from vector code generation > - conditionals area perfect example of it. Also, how to account an FMA > versus a non-FMA instruction? > > Vitali > > > On 02/22/2017 04:54 PM, Hammond, Jeff R wrote: > > I agree that PACKED_SIMD and SCALAR_SIMD include non-float operations. My > point was that if the problem is SIMD float ops overcount, then forcing the > compiler to turn them all into scalar float ops should prevent that. I > assume there is a way to count scalar float ops, but if not, then I guess > this doesn't help. > > > > At least for SSE*/AVX/AVX2 code, another way to count flops precisely is > to use an implementation where the hardware supports precise counters. > > > > https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX > > https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX2 > > > > Jeff > > > > *From: *Scott Parker <sparker(a)anl.gov> <sparker(a)anl.gov> > *Date: *Wednesday, February 22, 2017 at 12:32 PM > *To: *"Hammond, Jeff R" <jeff.r.hammond(a)intel.com> > <jeff.r.hammond(a)intel.com>, "Banerjee,Tania" <tmishra(a)ufl.edu> > <tmishra(a)ufl.edu>, "intel-nda(a)jlse.anl.gov" <intel-nda(a)jlse.anl.gov> > <intel-nda(a)jlse.anl.gov> <intel-nda(a)jlse.anl.gov> > *Subject: *Re: [JLSE-Intel-NDA] Measuring throughput on KNL > > > > > I don't think that will get you an accurate flop count either on KNL, > though it may be closer. > I'm not sure of the full set of instructions that map into the PACKED_SIMD > and SCALAR_SIMD > counters but I'm pretty sure there are some that are not arithmatic or > floating point. > > From past discussions with Dmitry Prohorov from VTune the Intel Advisor > approach is the only way on KNL > > > > -Scott > > > > On 2/22/17 11:54 AM, Hammond, Jeff R wrote: > > If one only wants to count flops, I assume disabling SIMD will obviate the > overcounting due to masking and such, but I have not experimented with this. > > > > -[no-]simd > > enables(DEFAULT)/disables vectorization using simd pragma > > > > Jeff > > > > *From: *<intel-nda-bounces(a)lists.jlse.anl.gov> > <intel-nda-bounces(a)lists.jlse.anl.gov> on behalf of Scott Parker > <sparker(a)anl.gov> <sparker(a)anl.gov> > *Date: *Wednesday, February 22, 2017 at 9:22 AM > *To: *"Banerjee,Tania" <tmishra(a)ufl.edu> <tmishra(a)ufl.edu>, > "intel-nda(a)jlse.anl.gov" <intel-nda(a)jlse.anl.gov> <intel-nda(a)jlse.anl.gov> > <intel-nda(a)jlse.anl.gov> > *Subject: *Re: [JLSE-Intel-NDA] Measuring throughput on KNL > > > > > KNL doesn't have hardware counters for flops, which is why PAPI doesn't > provide PAPI_FP_OPS. > This is a problem on a number of current Intel platforms. > > From Vtune you can get a upper bound flops estimate based on the > UOPS_RETIRED.PACKED_SIMD > > and UOPS_RETIRED.SCALAR_SIMD counters. This assumes things like the vector > length of instructions > is always 8, that FMA are used, and there is no masking, which is never > going to be always true and > therefore this can only be an upper bound. > > > > From Intel Advisor you can get an accurate FLOP count, but this is done > via analysis and instrumentation > of the binary so it will slow the code execution down noticeably. Further > description is given here > https://software.intel.com/en-us/articles/intel-advisor-flops > > > > -Scott > > > > > > On 2/21/17 8:33 PM, Banerjee,Tania wrote: > > Hi, > > > > How to measure throughput (GFLOPs/second) of a program on KNL? > > > > Normally I use PAPI to count the floating point operations (PAPI_FP_OPS > / PAPI_FP_INS) and divide that by the time taken. > > > > But papi_avail reveals that these events are not supported on KNL. > > > > The output of /soft/perftools/tau/papi-knl/bin/papi_avail is given below. > > > > Thanks much, > > Tania > > > > ============================================================ > ================= > > Available PAPI preset and user defined events plus hardware information. > > ------------------------------------------------------------ > -------------------- > > PAPI Version : 5.4.3.0 > > Vendor string and code : GenuineIntel (1) > > Model string and code : Intel(R) Xeon Phi(TM) CPU 7210 @ 1.30GHz (87) > > CPU Revision : 1.000000 > > CPUID Info : Family: 6 Model: 87 Stepping: 1 > > CPU Max Megahertz : 1500 > > CPU Min Megahertz : 1000 > > Hdw Threads per core : 4 > > Cores per Socket : 64 > > Sockets : 1 > > NUMA Nodes : 1 > > CPUs per Node : 256 > > Total CPUs : 256 > > Running in a VM : no > > Number Hardware Counters : 5 > > Max Multiplex Counters : 192 > > ------------------------------------------------------------ > -------------------- > > > > ============================================================ > ==================== > > PAPI Preset Events > > ============================================================ > ==================== > > Name Code Avail Deriv Description (Note) > > PAPI_L1_DCM 0x80000000 Yes No Level 1 data cache misses > > PAPI_L1_ICM 0x80000001 Yes No Level 1 instruction cache misses > > PAPI_L2_DCM 0x80000002 No No Level 2 data cache misses > > PAPI_L2_ICM 0x80000003 No No Level 2 instruction cache misses > > PAPI_L3_DCM 0x80000004 No No Level 3 data cache misses > > PAPI_L3_ICM 0x80000005 No No Level 3 instruction cache misses > > PAPI_L1_TCM 0x80000006 Yes Yes Level 1 cache misses > > PAPI_L2_TCM 0x80000007 Yes No Level 2 cache misses > > PAPI_L3_TCM 0x80000008 No No Level 3 cache misses > > PAPI_CA_SNP 0x80000009 No No Requests for a snoop > > PAPI_CA_SHR 0x8000000a No No Requests for exclusive access to > shared cache line > > PAPI_CA_CLN 0x8000000b No No Requests for exclusive access to clean > cache line > > PAPI_CA_INV 0x8000000c No No Requests for cache line invalidation > > PAPI_CA_ITV 0x8000000d No No Requests for cache line intervention > > PAPI_L3_LDM 0x8000000e No No Level 3 load misses > > PAPI_L3_STM 0x8000000f No No Level 3 store misses > > PAPI_BRU_IDL 0x80000010 No No Cycles branch units are idle > > PAPI_FXU_IDL 0x80000011 No No Cycles integer units are idle > > PAPI_FPU_IDL 0x80000012 No No Cycles floating point units are idle > > PAPI_LSU_IDL 0x80000013 No No Cycles load/store units are idle > > PAPI_TLB_DM 0x80000014 Yes No Data translation lookaside buffer > misses > > PAPI_TLB_IM 0x80000015 No No Instruction translation lookaside > buffer misses > > PAPI_TLB_TL 0x80000016 No No Total translation lookaside buffer > misses > > PAPI_L1_LDM 0x80000017 Yes No Level 1 load misses > > PAPI_L1_STM 0x80000018 No No Level 1 store misses > > PAPI_L2_LDM 0x80000019 Yes No Level 2 load misses > > PAPI_L2_STM 0x8000001a No No Level 2 store misses > > PAPI_BTAC_M 0x8000001b No No Branch target address cache misses > > PAPI_PRF_DM 0x8000001c No No Data prefetch cache misses > > PAPI_L3_DCH 0x8000001d No No Level 3 data cache hits > > PAPI_TLB_SD 0x8000001e No No Translation lookaside buffer shootdowns > > PAPI_CSR_FAL 0x8000001f No No Failed store conditional instructions > > PAPI_CSR_SUC 0x80000020 No No Successful store conditional > instructions > > PAPI_CSR_TOT 0x80000021 No No Total store conditional instructions > > PAPI_MEM_SCY 0x80000022 No No Cycles Stalled Waiting for memory > accesses > > PAPI_MEM_RCY 0x80000023 No No Cycles Stalled Waiting for memory Reads > > PAPI_MEM_WCY 0x80000024 No No Cycles Stalled Waiting for memory > writes > > PAPI_STL_ICY 0x80000025 Yes No Cycles with no instruction issue > > PAPI_FUL_ICY 0x80000026 No No Cycles with maximum instruction issue > > PAPI_STL_CCY 0x80000027 No No Cycles with no instructions completed > > PAPI_FUL_CCY 0x80000028 No No Cycles with maximum instructions > completed > > PAPI_HW_INT 0x80000029 No No Hardware interrupts > > PAPI_BR_UCN 0x8000002a Yes Yes Unconditional branch instructions > > PAPI_BR_CN 0x8000002b Yes No Conditional branch instructions > > PAPI_BR_TKN 0x8000002c Yes No Conditional branch instructions taken > > PAPI_BR_NTK 0x8000002d Yes Yes Conditional branch instructions not > taken > > PAPI_BR_MSP 0x8000002e Yes No Conditional branch instructions > mispredicted > > PAPI_BR_PRC 0x8000002f No No Conditional branch instructions > correctly predicted > > PAPI_FMA_INS 0x80000030 No No FMA instructions completed > > PAPI_TOT_IIS 0x80000031 No No Instructions issued > > PAPI_TOT_INS 0x80000032 Yes No Instructions completed > > PAPI_INT_INS 0x80000033 No No Integer instructions > > PAPI_FP_INS 0x80000034 No No Floating point instructions > > PAPI_LD_INS 0x80000035 Yes No Load instructions > > PAPI_SR_INS 0x80000036 No No Store instructions > > PAPI_BR_INS 0x80000037 Yes No Branch instructions > > PAPI_VEC_INS 0x80000038 No No Vector/SIMD instructions (could > include integer) > > PAPI_RES_STL 0x80000039 Yes No Cycles stalled on any resource > > PAPI_FP_STAL 0x8000003a No No Cycles the FP unit(s) are stalled > > PAPI_TOT_CYC 0x8000003b Yes No Total cycles > > PAPI_LST_INS 0x8000003c Yes Yes Load/store instructions completed > > PAPI_SYC_INS 0x8000003d No No Synchronization instructions completed > > PAPI_L1_DCH 0x8000003e No No Level 1 data cache hits > > PAPI_L2_DCH 0x8000003f No No Level 2 data cache hits > > PAPI_L1_DCA 0x80000040 Yes Yes Level 1 data cache accesses > > PAPI_L2_DCA 0x80000041 No No Level 2 data cache accesses > > PAPI_L3_DCA 0x80000042 No No Level 3 data cache accesses > > PAPI_L1_DCR 0x80000043 No No Level 1 data cache reads > > PAPI_L2_DCR 0x80000044 No No Level 2 data cache reads > > PAPI_L3_DCR 0x80000045 No No Level 3 data cache reads > > PAPI_L1_DCW 0x80000046 No No Level 1 data cache writes > > PAPI_L2_DCW 0x80000047 No No Level 2 data cache writes > > PAPI_L3_DCW 0x80000048 No No Level 3 data cache writes > > PAPI_L1_ICH 0x80000049 Yes No Level 1 instruction cache hits > > PAPI_L2_ICH 0x8000004a No No Level 2 instruction cache hits > > PAPI_L3_ICH 0x8000004b No No Level 3 instruction cache hits > > PAPI_L1_ICA 0x8000004c Yes No Level 1 instruction cache accesses > > PAPI_L2_ICA 0x8000004d No No Level 2 instruction cache accesses > > PAPI_L3_ICA 0x8000004e No No Level 3 instruction cache accesses > > PAPI_L1_ICR 0x8000004f No No Level 1 instruction cache reads > > PAPI_L2_ICR 0x80000050 No No Level 2 instruction cache reads > > PAPI_L3_ICR 0x80000051 No No Level 3 instruction cache reads > > PAPI_L1_ICW 0x80000052 No No Level 1 instruction cache writes > > PAPI_L2_ICW 0x80000053 No No Level 2 instruction cache writes > > PAPI_L3_ICW 0x80000054 No No Level 3 instruction cache writes > > PAPI_L1_TCH 0x80000055 No No Level 1 total cache hits > > PAPI_L2_TCH 0x80000056 Yes Yes Level 2 total cache hits > > PAPI_L3_TCH 0x80000057 No No Level 3 total cache hits > > PAPI_L1_TCA 0x80000058 No No Level 1 total cache accesses > > PAPI_L2_TCA 0x80000059 Yes No Level 2 total cache accesses > > PAPI_L3_TCA 0x8000005a No No Level 3 total cache accesses > > PAPI_L1_TCR 0x8000005b No No Level 1 total cache reads > > PAPI_L2_TCR 0x8000005c No No Level 2 total cache reads > > PAPI_L3_TCR 0x8000005d No No Level 3 total cache reads > > PAPI_L1_TCW 0x8000005e No No Level 1 total cache writes > > PAPI_L2_TCW 0x8000005f No No Level 2 total cache writes > > PAPI_L3_TCW 0x80000060 No No Level 3 total cache writes > > PAPI_FML_INS 0x80000061 No No Floating point multiply instructions > > PAPI_FAD_INS 0x80000062 No No Floating point add instructions > > PAPI_FDV_INS 0x80000063 No No Floating point divide instructions > > PAPI_FSQ_INS 0x80000064 No No Floating point square root instructions > > PAPI_FNV_INS 0x80000065 No No Floating point inverse instructions > > PAPI_FP_OPS 0x80000066 No No Floating point operations > > PAPI_SP_OPS 0x80000067 No No Floating point operations; optimized > to count scaled single precision vector operations > > PAPI_DP_OPS 0x80000068 No No Floating point operations; optimized > to count scaled double precision vector operations > > PAPI_VEC_SP 0x80000069 No No Single precision vector/SIMD > instructions > > PAPI_VEC_DP 0x8000006a No No Double precision vector/SIMD > instructions > > PAPI_REF_CYC 0x8000006b Yes No Reference clock cycles > > ------------------------------------------------------------ > -------------------- > > Of 108 possible events, 25 are available, of which 6 are derived. > > > > avail.c PASSED > > > > > > > > > > > > > > _______________________________________________ > > Intel-nda mailing list > > Intel-nda(a)lists.jlse.anl.gov > > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > > > > > > > > _______________________________________________ > Intel-nda mailing [email protected]://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > >

1 0

Re: [JLSE-Intel-NDA] Measuring throughput on KNL
by Scott Parker 23 Feb '17

23 Feb '17

You should use /soft/compilers/intel/advisor -Scott On 2/23/17 5:59 PM, Banerjee,Tania wrote: > > Hi All, > > > Thanks for pitching in. > > > Scott: > > Which advisor should we use in /soft/compilers/intel ? > > > [tania@jlselogin1 intel]$ ls advisor > advisor/ advisor_xe/ > advisor_2017/ advisor_xe_2016/ > advisor_2017.1.0.477503/ advisor_xe_2016.1.30.450722/ > advisor_2017.1.1.486553/ advisor_xe_2016.1.40.463413/ > > Also about Intel VTunes, I could get only the following summary > (using both hpc-performance and memory-access collection modes) , > which did not feature the events talked about in this thread: > > Event summary > ------------- > Hardware Event Type Hardware Event Count:Self Hardware > Event Sample Count:Self Events Per Sample > --------------------------------- ------------------------- > -------------------------------- ----------------- > > INST_RETIRED.ANY 1080573000000 83121 13000000 > CPU_CLK_UNHALTED.THREAD 2283515000000 > 175655 13000000 > CPU_CLK_UNHALTED.REF_TSC 2120508000000 > 163116 13000000 > MEM_UOPS_RETIRED.L2_MISS_LOADS_PS 396011880 > 132 1000030 > UOPS_RETIRED.ALL 1479002218500 24650 20000030 > MEM_UOPS_RETIRED.L2_HIT_LOADS_PS 2079062370 > 693 1000030 > NO_ALLOC_CYCLES.MISPREDICTS 11220016830 > 187 20000030 > NO_ALLOC_CYCLES.NOT_DELIVERED 720601080900 > 12010 20000030 > > Uncore Event summary > -------------------- > Uncore Event Type Uncore Event Count:Self > ... > > > But it could be because I did not enable -avx during compilation, > hence no there's no vectorization, whatsoever. I will check this > aspect as the other part of my project on performance tuning. > > Thanks a lot, > Tania > > ------------------------------------------------------------------------ > *From:* Vitali A. Morozov <morozov(a)anl.gov> > *Sent:* Wednesday, February 22, 2017 6:08 PM > *To:* Hammond, Jeff R; Scott Parker; Banerjee,Tania; > intel-nda(a)jlse.anl.gov > *Subject:* Re: [JLSE-Intel-NDA] Measuring throughput on KNL > > In general, scalar code generation may differ from vector code > generation - conditionals area perfect example of it. Also, how to > account an FMA versus a non-FMA instruction? > > Vitali > > > On 02/22/2017 04:54 PM, Hammond, Jeff R wrote: >> >> I agree that PACKED_SIMD and SCALAR_SIMD include non-float >> operations. My point was that if the problem is SIMD float ops >> overcount, then forcing the compiler to turn them all into scalar >> float ops should prevent that. I assume there is a way to count >> scalar float ops, but if not, then I guess this doesn't help. >> >> At least for SSE*/AVX/AVX2 code, another way to count flops precisely >> is to use an implementation where the hardware supports precise counters. >> >> https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX >> >> https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX2 >> >> Jeff >> >> *From: *Scott Parker <sparker(a)anl.gov> >> *Date: *Wednesday, February 22, 2017 at 12:32 PM >> *To: *"Hammond, Jeff R" <jeff.r.hammond(a)intel.com>, "Banerjee,Tania" >> <tmishra(a)ufl.edu>, "intel-nda(a)jlse.anl.gov" <intel-nda(a)jlse.anl.gov> >> *Subject: *Re: [JLSE-Intel-NDA] Measuring throughput on KNL >> >> >> I don't think that will get you an accurate flop count either on KNL, >> though it may be closer. >> I'm not sure of the full set of instructions that map into the >> PACKED_SIMD and SCALAR_SIMD >> counters but I'm pretty sure there are some that are not arithmatic >> or floating point. >> >> From past discussions with Dmitry Prohorov from VTune the Intel >> Advisor approach is the only way on KNL >> >> -Scott >> >> On 2/22/17 11:54 AM, Hammond, Jeff R wrote: >> >> If one only wants to count flops, I assume disabling SIMD will >> obviate the overcounting due to masking and such, but I have not >> experimented with this. >> >> -[no-]simd >> >> enables(DEFAULT)/disables vectorization using simd pragma >> >> Jeff >> >> *From: *<intel-nda-bounces(a)lists.jlse.anl.gov> >> <mailto:[email protected]> on behalf of Scott >> Parker <sparker(a)anl.gov> <mailto:[email protected]> >> *Date: *Wednesday, February 22, 2017 at 9:22 AM >> *To: *"Banerjee,Tania" <tmishra(a)ufl.edu> >> <mailto:[email protected]>, "intel-nda(a)jlse.anl.gov" >> <mailto:[email protected]> <intel-nda(a)jlse.anl.gov> >> <mailto:[email protected]> >> *Subject: *Re: [JLSE-Intel-NDA] Measuring throughput on KNL >> >> >> KNL doesn't have hardware counters for flops, which is why PAPI >> doesn't provide PAPI_FP_OPS. >> This is a problem on a number of current Intel platforms. >> >> From Vtune you can get a upper bound flops estimate based on the >> UOPS_RETIRED.PACKED_SIMD >> >> and UOPS_RETIRED.SCALAR_SIMD counters. This assumes things like >> the vector length of instructions >> is always 8, that FMA are used, and there is no masking, which is >> never going to be always true and >> therefore this can only be an upper bound. >> >> From Intel Advisor you can get an accurate FLOP count, but this >> is done via analysis and instrumentation >> of the binary so it will slow the code execution down noticeably. >> Further description is given here >> https://software.intel.com/en-us/articles/intel-advisor-flops >> >> -Scott >> >> On 2/21/17 8:33 PM, Banerjee,Tania wrote: >> >> Hi, >> >> How to measure throughput (GFLOPs/second) of a program on KNL? >> >> Normally I use PAPI to count the floating point operations >> (PAPI_FP_OPS / PAPI_FP_INS) and divide that by the time taken. >> >> But papi_avail reveals that these events are not supported on >> KNL. >> >> The output of /soft/perftools/tau/papi-knl/bin/papi_avail is >> given below. >> >> Thanks much, >> >> Tania >> >> ============================================================================= >> >> Available PAPI preset and user defined events plus hardware >> information. >> >> -------------------------------------------------------------------------------- >> >> PAPI Version : 5.4.3.0 >> >> Vendor string and code : GenuineIntel (1) >> >> Model string and code : Intel(R) Xeon Phi(TM) CPU 7210 @ >> 1.30GHz (87) >> >> CPU Revision : 1.000000 >> >> CPUID Info : Family: 6 Model: 87 Stepping: 1 >> >> CPU Max Megahertz : 1500 >> >> CPU Min Megahertz : 1000 >> >> Hdw Threads per core : 4 >> >> Cores per Socket : 64 >> >> Sockets : 1 >> >> NUMA Nodes : 1 >> >> CPUs per Node : 256 >> >> Total CPUs : 256 >> >> Running in a VM : no >> >> Number Hardware Counters : 5 >> >> Max Multiplex Counters : 192 >> >> -------------------------------------------------------------------------------- >> >> ================================================================================ >> >> PAPI Preset Events >> >> ================================================================================ >> >> Name Code Avail Deriv Description (Note) >> >> PAPI_L1_DCM 0x80000000 Yes No Level 1 data cache misses >> >> PAPI_L1_ICM 0x80000001 Yes No Level 1 instruction cache >> misses >> >> PAPI_L2_DCM 0x80000002 No No Level 2 data cache misses >> >> PAPI_L2_ICM 0x80000003 No No Level 2 instruction cache >> misses >> >> PAPI_L3_DCM 0x80000004 No No Level 3 data cache misses >> >> PAPI_L3_ICM 0x80000005 No No Level 3 instruction cache >> misses >> >> PAPI_L1_TCM 0x80000006 Yes Yes Level 1 cache misses >> >> PAPI_L2_TCM 0x80000007 Yes No Level 2 cache misses >> >> PAPI_L3_TCM 0x80000008 No No Level 3 cache misses >> >> PAPI_CA_SNP 0x80000009 No No Requests for a snoop >> >> PAPI_CA_SHR 0x8000000a No No Requests for exclusive >> access to shared cache line >> >> PAPI_CA_CLN 0x8000000b No No Requests for exclusive >> access to clean cache line >> >> PAPI_CA_INV 0x8000000c No No Requests for cache line >> invalidation >> >> PAPI_CA_ITV 0x8000000d No No Requests for cache line >> intervention >> >> PAPI_L3_LDM 0x8000000e No No Level 3 load misses >> >> PAPI_L3_STM 0x8000000f No No Level 3 store misses >> >> PAPI_BRU_IDL 0x80000010 No No Cycles branch units are idle >> >> PAPI_FXU_IDL 0x80000011 No No Cycles integer units are idle >> >> PAPI_FPU_IDL 0x80000012 No No Cycles floating point >> units are idle >> >> PAPI_LSU_IDL 0x80000013 No No Cycles load/store units >> are idle >> >> PAPI_TLB_DM 0x80000014 Yes No Data translation >> lookaside buffer misses >> >> PAPI_TLB_IM 0x80000015 No No Instruction translation >> lookaside buffer misses >> >> PAPI_TLB_TL 0x80000016 No No Total translation >> lookaside buffer misses >> >> PAPI_L1_LDM 0x80000017 Yes No Level 1 load misses >> >> PAPI_L1_STM 0x80000018 No No Level 1 store misses >> >> PAPI_L2_LDM 0x80000019 Yes No Level 2 load misses >> >> PAPI_L2_STM 0x8000001a No No Level 2 store misses >> >> PAPI_BTAC_M 0x8000001b No No Branch target address >> cache misses >> >> PAPI_PRF_DM 0x8000001c No No Data prefetch cache misses >> >> PAPI_L3_DCH 0x8000001d No No Level 3 data cache hits >> >> PAPI_TLB_SD 0x8000001e No No Translation lookaside >> buffer shootdowns >> >> PAPI_CSR_FAL 0x8000001f No No Failed store conditional >> instructions >> >> PAPI_CSR_SUC 0x80000020 No No Successful store >> conditional instructions >> >> PAPI_CSR_TOT 0x80000021 No No Total store conditional >> instructions >> >> PAPI_MEM_SCY 0x80000022 No No Cycles Stalled Waiting >> for memory accesses >> >> PAPI_MEM_RCY 0x80000023 No No Cycles Stalled Waiting >> for memory Reads >> >> PAPI_MEM_WCY 0x80000024 No No Cycles Stalled Waiting >> for memory writes >> >> PAPI_STL_ICY 0x80000025 Yes No Cycles with no >> instruction issue >> >> PAPI_FUL_ICY 0x80000026 No No Cycles with maximum >> instruction issue >> >> PAPI_STL_CCY 0x80000027 No No Cycles with no >> instructions completed >> >> PAPI_FUL_CCY 0x80000028 No No Cycles with maximum >> instructions completed >> >> PAPI_HW_INT 0x80000029 No No Hardware interrupts >> >> PAPI_BR_UCN 0x8000002a Yes Yes Unconditional branch >> instructions >> >> PAPI_BR_CN 0x8000002b Yes No Conditional branch >> instructions >> >> PAPI_BR_TKN 0x8000002c Yes No Conditional branch >> instructions taken >> >> PAPI_BR_NTK 0x8000002d Yes Yes Conditional branch >> instructions not taken >> >> PAPI_BR_MSP 0x8000002e Yes No Conditional branch >> instructions mispredicted >> >> PAPI_BR_PRC 0x8000002f No No Conditional branch >> instructions correctly predicted >> >> PAPI_FMA_INS 0x80000030 No No FMA instructions completed >> >> PAPI_TOT_IIS 0x80000031 No No Instructions issued >> >> PAPI_TOT_INS 0x80000032 Yes No Instructions completed >> >> PAPI_INT_INS 0x80000033 No No Integer instructions >> >> PAPI_FP_INS 0x80000034 No No Floating point instructions >> >> PAPI_LD_INS 0x80000035 Yes No Load instructions >> >> PAPI_SR_INS 0x80000036 No No Store instructions >> >> PAPI_BR_INS 0x80000037 Yes No Branch instructions >> >> PAPI_VEC_INS 0x80000038 No No Vector/SIMD instructions >> (could include integer) >> >> PAPI_RES_STL 0x80000039 Yes No Cycles stalled on any >> resource >> >> PAPI_FP_STAL 0x8000003a No No Cycles the FP unit(s) are >> stalled >> >> PAPI_TOT_CYC 0x8000003b Yes No Total cycles >> >> PAPI_LST_INS 0x8000003c Yes Yes Load/store instructions >> completed >> >> PAPI_SYC_INS 0x8000003d No No Synchronization >> instructions completed >> >> PAPI_L1_DCH 0x8000003e No No Level 1 data cache hits >> >> PAPI_L2_DCH 0x8000003f No No Level 2 data cache hits >> >> PAPI_L1_DCA 0x80000040 Yes Yes Level 1 data cache accesses >> >> PAPI_L2_DCA 0x80000041 No No Level 2 data cache accesses >> >> PAPI_L3_DCA 0x80000042 No No Level 3 data cache accesses >> >> PAPI_L1_DCR 0x80000043 No No Level 1 data cache reads >> >> PAPI_L2_DCR 0x80000044 No No Level 2 data cache reads >> >> PAPI_L3_DCR 0x80000045 No No Level 3 data cache reads >> >> PAPI_L1_DCW 0x80000046 No No Level 1 data cache writes >> >> PAPI_L2_DCW 0x80000047 No No Level 2 data cache writes >> >> PAPI_L3_DCW 0x80000048 No No Level 3 data cache writes >> >> PAPI_L1_ICH 0x80000049 Yes No Level 1 instruction cache >> hits >> >> PAPI_L2_ICH 0x8000004a No No Level 2 instruction cache >> hits >> >> PAPI_L3_ICH 0x8000004b No No Level 3 instruction cache >> hits >> >> PAPI_L1_ICA 0x8000004c Yes No Level 1 instruction cache >> accesses >> >> PAPI_L2_ICA 0x8000004d No No Level 2 instruction cache >> accesses >> >> PAPI_L3_ICA 0x8000004e No No Level 3 instruction cache >> accesses >> >> PAPI_L1_ICR 0x8000004f No No Level 1 instruction cache >> reads >> >> PAPI_L2_ICR 0x80000050 No No Level 2 instruction cache >> reads >> >> PAPI_L3_ICR 0x80000051 No No Level 3 instruction cache >> reads >> >> PAPI_L1_ICW 0x80000052 No No Level 1 instruction cache >> writes >> >> PAPI_L2_ICW 0x80000053 No No Level 2 instruction cache >> writes >> >> PAPI_L3_ICW 0x80000054 No No Level 3 instruction cache >> writes >> >> PAPI_L1_TCH 0x80000055 No No Level 1 total cache hits >> >> PAPI_L2_TCH 0x80000056 Yes Yes Level 2 total cache hits >> >> PAPI_L3_TCH 0x80000057 No No Level 3 total cache hits >> >> PAPI_L1_TCA 0x80000058 No No Level 1 total cache accesses >> >> PAPI_L2_TCA 0x80000059 Yes No Level 2 total cache accesses >> >> PAPI_L3_TCA 0x8000005a No No Level 3 total cache accesses >> >> PAPI_L1_TCR 0x8000005b No No Level 1 total cache reads >> >> PAPI_L2_TCR 0x8000005c No No Level 2 total cache reads >> >> PAPI_L3_TCR 0x8000005d No No Level 3 total cache reads >> >> PAPI_L1_TCW 0x8000005e No No Level 1 total cache writes >> >> PAPI_L2_TCW 0x8000005f No No Level 2 total cache writes >> >> PAPI_L3_TCW 0x80000060 No No Level 3 total cache writes >> >> PAPI_FML_INS 0x80000061 No No Floating point multiply >> instructions >> >> PAPI_FAD_INS 0x80000062 No No Floating point add >> instructions >> >> PAPI_FDV_INS 0x80000063 No No Floating point divide >> instructions >> >> PAPI_FSQ_INS 0x80000064 No No Floating point square >> root instructions >> >> PAPI_FNV_INS 0x80000065 No No Floating point inverse >> instructions >> >> PAPI_FP_OPS 0x80000066 No No Floating point operations >> >> PAPI_SP_OPS 0x80000067 No No Floating point >> operations; optimized to count scaled single precision vector >> operations >> >> PAPI_DP_OPS 0x80000068 No No Floating point >> operations; optimized to count scaled double precision vector >> operations >> >> PAPI_VEC_SP 0x80000069 No No Single precision >> vector/SIMD instructions >> >> PAPI_VEC_DP 0x8000006a No No Double precision >> vector/SIMD instructions >> >> PAPI_REF_CYC 0x8000006b Yes No Reference clock cycles >> >> -------------------------------------------------------------------------------- >> >> Of 108 possible events, 25 are available, of which 6 are derived. >> >> avail.c PASSED >> >> >> >> >> >> _______________________________________________ >> >> Intel-nda mailing list >> >> Intel-nda(a)lists.jlse.anl.gov >> <mailto:[email protected]> >> >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> >> >> >> >> >> >> >> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > >

1 0

Re: [JLSE-Intel-NDA] Measuring throughput on KNL
by Banerjee,Tania 23 Feb '17

23 Feb '17

Hi All, Thanks for pitching in. Scott: Which advisor should we use in /soft/compilers/intel ? [tania@jlselogin1 intel]$ ls advisor advisor/ advisor_xe/ advisor_2017/ advisor_xe_2016/ advisor_2017.1.0.477503/ advisor_xe_2016.1.30.450722/ advisor_2017.1.1.486553/ advisor_xe_2016.1.40.463413/ Also about Intel VTunes, I could get only the following summary (using both hpc-performance and memory-access collection modes) , which did not feature the events talked about in this thread: Event summary ------------- Hardware Event Type Hardware Event Count:Self Hardware Event Sample Count:Self Events Per Sample --------------------------------- ------------------------- -------------------------------- ----------------- INST_RETIRED.ANY 1080573000000 83121 13000000 CPU_CLK_UNHALTED.THREAD 2283515000000 175655 13000000 CPU_CLK_UNHALTED.REF_TSC 2120508000000 163116 13000000 MEM_UOPS_RETIRED.L2_MISS_LOADS_PS 396011880 132 1000030 UOPS_RETIRED.ALL 1479002218500 24650 20000030 MEM_UOPS_RETIRED.L2_HIT_LOADS_PS 2079062370 693 1000030 NO_ALLOC_CYCLES.MISPREDICTS 11220016830 187 20000030 NO_ALLOC_CYCLES.NOT_DELIVERED 720601080900 12010 20000030 Uncore Event summary -------------------- Uncore Event Type Uncore Event Count:Self ... But it could be because I did not enable -avx during compilation, hence no there's no vectorization, whatsoever. I will check this aspect as the other part of my project on performance tuning. Thanks a lot, Tania ________________________________ From: Vitali A. Morozov <morozov(a)anl.gov> Sent: Wednesday, February 22, 2017 6:08 PM To: Hammond, Jeff R; Scott Parker; Banerjee,Tania; intel-nda(a)jlse.anl.gov Subject: Re: [JLSE-Intel-NDA] Measuring throughput on KNL In general, scalar code generation may differ from vector code generation - conditionals area perfect example of it. Also, how to account an FMA versus a non-FMA instruction? Vitali On 02/22/2017 04:54 PM, Hammond, Jeff R wrote: I agree that PACKED_SIMD and SCALAR_SIMD include non-float operations. My point was that if the problem is SIMD float ops overcount, then forcing the compiler to turn them all into scalar float ops should prevent that. I assume there is a way to count scalar float ops, but if not, then I guess this doesn't help. At least for SSE*/AVX/AVX2 code, another way to count flops precisely is to use an implementation where the hardware supports precise counters. https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX https://en.wikipedia.org/wiki/Advanced_Vector_Extensions#CPUs_with_AVX2 Jeff From: Scott Parker <sparker(a)anl.gov><mailto:[email protected]> Date: Wednesday, February 22, 2017 at 12:32 PM To: "Hammond, Jeff R" <jeff.r.hammond(a)intel.com><mailto:[email protected]>, "Banerjee,Tania" <tmishra(a)ufl.edu><mailto:[email protected]>, "intel-nda(a)jlse.anl.gov"<mailto:[email protected]> <intel-nda(a)jlse.anl.gov><mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] Measuring throughput on KNL I don't think that will get you an accurate flop count either on KNL, though it may be closer. I'm not sure of the full set of instructions that map into the PACKED_SIMD and SCALAR_SIMD counters but I'm pretty sure there are some that are not arithmatic or floating point. From past discussions with Dmitry Prohorov from VTune the Intel Advisor approach is the only way on KNL -Scott On 2/22/17 11:54 AM, Hammond, Jeff R wrote: If one only wants to count flops, I assume disabling SIMD will obviate the overcounting due to masking and such, but I have not experimented with this. -[no-]simd enables(DEFAULT)/disables vectorization using simd pragma Jeff From: <intel-nda-bounces(a)lists.jlse.anl.gov><mailto:[email protected]> on behalf of Scott Parker <sparker(a)anl.gov><mailto:[email protected]> Date: Wednesday, February 22, 2017 at 9:22 AM To: "Banerjee,Tania" <tmishra(a)ufl.edu><mailto:[email protected]>, "intel-nda(a)jlse.anl.gov"<mailto:[email protected]> <intel-nda(a)jlse.anl.gov><mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] Measuring throughput on KNL KNL doesn't have hardware counters for flops, which is why PAPI doesn't provide PAPI_FP_OPS. This is a problem on a number of current Intel platforms. From Vtune you can get a upper bound flops estimate based on the UOPS_RETIRED.PACKED_SIMD and UOPS_RETIRED.SCALAR_SIMD counters. This assumes things like the vector length of instructions is always 8, that FMA are used, and there is no masking, which is never going to be always true and therefore this can only be an upper bound. From Intel Advisor you can get an accurate FLOP count, but this is done via analysis and instrumentation of the binary so it will slow the code execution down noticeably. Further description is given here https://software.intel.com/en-us/articles/intel-advisor-flops -Scott On 2/21/17 8:33 PM, Banerjee,Tania wrote: Hi, How to measure throughput (GFLOPs/second) of a program on KNL? Normally I use PAPI to count the floating point operations (PAPI_FP_OPS / PAPI_FP_INS) and divide that by the time taken. But papi_avail reveals that these events are not supported on KNL. The output of /soft/perftools/tau/papi-knl/bin/papi_avail is given below. Thanks much, Tania ============================================================================= Available PAPI preset and user defined events plus hardware information. -------------------------------------------------------------------------------- PAPI Version : 5.4.3.0 Vendor string and code : GenuineIntel (1) Model string and code : Intel(R) Xeon Phi(TM) CPU 7210 @ 1.30GHz (87) CPU Revision : 1.000000 CPUID Info : Family: 6 Model: 87 Stepping: 1 CPU Max Megahertz : 1500 CPU Min Megahertz : 1000 Hdw Threads per core : 4 Cores per Socket : 64 Sockets : 1 NUMA Nodes : 1 CPUs per Node : 256 Total CPUs : 256 Running in a VM : no Number Hardware Counters : 5 Max Multiplex Counters : 192 -------------------------------------------------------------------------------- ================================================================================ PAPI Preset Events ================================================================================ Name Code Avail Deriv Description (Note) PAPI_L1_DCM 0x80000000 Yes No Level 1 data cache misses PAPI_L1_ICM 0x80000001 Yes No Level 1 instruction cache misses PAPI_L2_DCM 0x80000002 No No Level 2 data cache misses PAPI_L2_ICM 0x80000003 No No Level 2 instruction cache misses PAPI_L3_DCM 0x80000004 No No Level 3 data cache misses PAPI_L3_ICM 0x80000005 No No Level 3 instruction cache misses PAPI_L1_TCM 0x80000006 Yes Yes Level 1 cache misses PAPI_L2_TCM 0x80000007 Yes No Level 2 cache misses PAPI_L3_TCM 0x80000008 No No Level 3 cache misses PAPI_CA_SNP 0x80000009 No No Requests for a snoop PAPI_CA_SHR 0x8000000a No No Requests for exclusive access to shared cache line PAPI_CA_CLN 0x8000000b No No Requests for exclusive access to clean cache line PAPI_CA_INV 0x8000000c No No Requests for cache line invalidation PAPI_CA_ITV 0x8000000d No No Requests for cache line intervention PAPI_L3_LDM 0x8000000e No No Level 3 load misses PAPI_L3_STM 0x8000000f No No Level 3 store misses PAPI_BRU_IDL 0x80000010 No No Cycles branch units are idle PAPI_FXU_IDL 0x80000011 No No Cycles integer units are idle PAPI_FPU_IDL 0x80000012 No No Cycles floating point units are idle PAPI_LSU_IDL 0x80000013 No No Cycles load/store units are idle PAPI_TLB_DM 0x80000014 Yes No Data translation lookaside buffer misses PAPI_TLB_IM 0x80000015 No No Instruction translation lookaside buffer misses PAPI_TLB_TL 0x80000016 No No Total translation lookaside buffer misses PAPI_L1_LDM 0x80000017 Yes No Level 1 load misses PAPI_L1_STM 0x80000018 No No Level 1 store misses PAPI_L2_LDM 0x80000019 Yes No Level 2 load misses PAPI_L2_STM 0x8000001a No No Level 2 store misses PAPI_BTAC_M 0x8000001b No No Branch target address cache misses PAPI_PRF_DM 0x8000001c No No Data prefetch cache misses PAPI_L3_DCH 0x8000001d No No Level 3 data cache hits PAPI_TLB_SD 0x8000001e No No Translation lookaside buffer shootdowns PAPI_CSR_FAL 0x8000001f No No Failed store conditional instructions PAPI_CSR_SUC 0x80000020 No No Successful store conditional instructions PAPI_CSR_TOT 0x80000021 No No Total store conditional instructions PAPI_MEM_SCY 0x80000022 No No Cycles Stalled Waiting for memory accesses PAPI_MEM_RCY 0x80000023 No No Cycles Stalled Waiting for memory Reads PAPI_MEM_WCY 0x80000024 No No Cycles Stalled Waiting for memory writes PAPI_STL_ICY 0x80000025 Yes No Cycles with no instruction issue PAPI_FUL_ICY 0x80000026 No No Cycles with maximum instruction issue PAPI_STL_CCY 0x80000027 No No Cycles with no instructions completed PAPI_FUL_CCY 0x80000028 No No Cycles with maximum instructions completed PAPI_HW_INT 0x80000029 No No Hardware interrupts PAPI_BR_UCN 0x8000002a Yes Yes Unconditional branch instructions PAPI_BR_CN 0x8000002b Yes No Conditional branch instructions PAPI_BR_TKN 0x8000002c Yes No Conditional branch instructions taken PAPI_BR_NTK 0x8000002d Yes Yes Conditional branch instructions not taken PAPI_BR_MSP 0x8000002e Yes No Conditional branch instructions mispredicted PAPI_BR_PRC 0x8000002f No No Conditional branch instructions correctly predicted PAPI_FMA_INS 0x80000030 No No FMA instructions completed PAPI_TOT_IIS 0x80000031 No No Instructions issued PAPI_TOT_INS 0x80000032 Yes No Instructions completed PAPI_INT_INS 0x80000033 No No Integer instructions PAPI_FP_INS 0x80000034 No No Floating point instructions PAPI_LD_INS 0x80000035 Yes No Load instructions PAPI_SR_INS 0x80000036 No No Store instructions PAPI_BR_INS 0x80000037 Yes No Branch instructions PAPI_VEC_INS 0x80000038 No No Vector/SIMD instructions (could include integer) PAPI_RES_STL 0x80000039 Yes No Cycles stalled on any resource PAPI_FP_STAL 0x8000003a No No Cycles the FP unit(s) are stalled PAPI_TOT_CYC 0x8000003b Yes No Total cycles PAPI_LST_INS 0x8000003c Yes Yes Load/store instructions completed PAPI_SYC_INS 0x8000003d No No Synchronization instructions completed PAPI_L1_DCH 0x8000003e No No Level 1 data cache hits PAPI_L2_DCH 0x8000003f No No Level 2 data cache hits PAPI_L1_DCA 0x80000040 Yes Yes Level 1 data cache accesses PAPI_L2_DCA 0x80000041 No No Level 2 data cache accesses PAPI_L3_DCA 0x80000042 No No Level 3 data cache accesses PAPI_L1_DCR 0x80000043 No No Level 1 data cache reads PAPI_L2_DCR 0x80000044 No No Level 2 data cache reads PAPI_L3_DCR 0x80000045 No No Level 3 data cache reads PAPI_L1_DCW 0x80000046 No No Level 1 data cache writes PAPI_L2_DCW 0x80000047 No No Level 2 data cache writes PAPI_L3_DCW 0x80000048 No No Level 3 data cache writes PAPI_L1_ICH 0x80000049 Yes No Level 1 instruction cache hits PAPI_L2_ICH 0x8000004a No No Level 2 instruction cache hits PAPI_L3_ICH 0x8000004b No No Level 3 instruction cache hits PAPI_L1_ICA 0x8000004c Yes No Level 1 instruction cache accesses PAPI_L2_ICA 0x8000004d No No Level 2 instruction cache accesses PAPI_L3_ICA 0x8000004e No No Level 3 instruction cache accesses PAPI_L1_ICR 0x8000004f No No Level 1 instruction cache reads PAPI_L2_ICR 0x80000050 No No Level 2 instruction cache reads PAPI_L3_ICR 0x80000051 No No Level 3 instruction cache reads PAPI_L1_ICW 0x80000052 No No Level 1 instruction cache writes PAPI_L2_ICW 0x80000053 No No Level 2 instruction cache writes PAPI_L3_ICW 0x80000054 No No Level 3 instruction cache writes PAPI_L1_TCH 0x80000055 No No Level 1 total cache hits PAPI_L2_TCH 0x80000056 Yes Yes Level 2 total cache hits PAPI_L3_TCH 0x80000057 No No Level 3 total cache hits PAPI_L1_TCA 0x80000058 No No Level 1 total cache accesses PAPI_L2_TCA 0x80000059 Yes No Level 2 total cache accesses PAPI_L3_TCA 0x8000005a No No Level 3 total cache accesses PAPI_L1_TCR 0x8000005b No No Level 1 total cache reads PAPI_L2_TCR 0x8000005c No No Level 2 total cache reads PAPI_L3_TCR 0x8000005d No No Level 3 total cache reads PAPI_L1_TCW 0x8000005e No No Level 1 total cache writes PAPI_L2_TCW 0x8000005f No No Level 2 total cache writes PAPI_L3_TCW 0x80000060 No No Level 3 total cache writes PAPI_FML_INS 0x80000061 No No Floating point multiply instructions PAPI_FAD_INS 0x80000062 No No Floating point add instructions PAPI_FDV_INS 0x80000063 No No Floating point divide instructions PAPI_FSQ_INS 0x80000064 No No Floating point square root instructions PAPI_FNV_INS 0x80000065 No No Floating point inverse instructions PAPI_FP_OPS 0x80000066 No No Floating point operations PAPI_SP_OPS 0x80000067 No No Floating point operations; optimized to count scaled single precision vector operations PAPI_DP_OPS 0x80000068 No No Floating point operations; optimized to count scaled double precision vector operations PAPI_VEC_SP 0x80000069 No No Single precision vector/SIMD instructions PAPI_VEC_DP 0x8000006a No No Double precision vector/SIMD instructions PAPI_REF_CYC 0x8000006b Yes No Reference clock cycles -------------------------------------------------------------------------------- Of 108 possible events, 25 are available, of which 6 are derived. avail.c PASSED _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0