Great. For convenience, you may want to script this, but here is the line for hpc performance: amplxe-cl -collect-with runsa -knob event-config=CPU_CLK_UNHALTED.THREAD:sa=2300000,CPU_CLK_UNHALTED.REF_TSC:sa=2300000,INST_RETIRED.ANY:sa=2300000,CYCLE_ACTIVITY.STALLS_L1D_MISS:sa=2000003,CYCLE_ACTIVITY.STALLS_L2_MISS:sa=2000003,CYCLE_ACTIVITY.STALLS_MEM_ANY:sa=2000003,CYCLE_ACTIVITY.STALLS_TOTAL:sa=2000003,IDQ_UOPS_NOT_DELIVERED.CORE:sa=2000003,INT_MISC.RECOVERY_CYCLES:sa=2000003,MEM_LOAD_UOPS_RETIRED.L3_HIT_PS:sa=50003,MEM_LOAD_UOPS_RETIRED.L3_MISS_PS:sa=100003,OFFCORE_REQUESTS_OUTSTANDING.CYCLES_WITH_DEMAND_DATA_RD:sa=2000003,OFFCORE_REQUESTS_OUTSTANDING.DEMAND_DATA_RD_GE_6:sa=2000003,RESOURCE_STALLS.SB:sa=2000003,RS_EVENTS.EMPTY_CYCLES:sa=2000003,UOPS_EXECUTED.CYCLES_GE_1_UOP_EXEC:sa=2000003,UOPS_EXECUTED.CYCLES_GE_2_UOPS_EXEC:sa=2000003,UOPS_EXECUTED.CYCLES_GE_3_UOPS_EXEC:sa=2000003,UOPS_ISSUED.ANY:sa=2000003,UOPS_RETIRED.RETIRE_SLOTS:sa=2000003,IDQ_UOPS_NOT_DELIVERED.CYCLES_0_UOPS_DELIV.CORE:sa=2000003,OFFCORE_REQUESTS_OUTSTANDING.ALL_DATA_RD:cmask=4:sa=2000003,OFFCORE_REQUESTS_OUTSTANDING.CYCLES_WITH_DATA_RD:sa=2000003,FP_ARITH_INST_RETIRED.SCALAR_SINGLE:sa=2000003,FP_ARITH_INST_RETIRED.128B_PACKED_SINGLE:sa=2000003,FP_ARITH_INST_RETIRED.256B_PACKED_SINGLE:sa=2000003,FP_ARITH_INST_RETIRED.SCALAR_DOUBLE:sa=2000003,FP_ARITH_INST_RETIRED.128B_PACKED_DOUBLE:sa=2000003,FP_ARITH_INST_RETIRED.256B_PACKED_DOUBLE:sa=2000003,INST_RETIRED.X87:sa=2000003 -knob collectMemBandwidth=true -knob dram-bandwidth-limits=true -knob analyze-loops=true -knob analyze-openmp=true ./a.out Also, feel free to remove the ‘:sa=xxxxx’ parameter specifications. You may find this (slightly dated) video useful: https://software.intel.com/en-us/videos/intel-vtune-amplifier-xe-video-tutor... More shortly. -michael From: Will Boyd [mailto:[email protected]] Sent: Friday, January 06, 2017 9:36 AM To: D'mello, Michael <[email protected]>; Hammond, Jeff R <[email protected]>; [email protected] Subject: Re: [JLSE-Intel-NDA] Using VTune remotely on the Phi Michael, The output for that command was the following: ]$ amplxe-cl -collect-with runsa -knob event-config=CPU_CLK_UNHALTED.THREAD:sa=2300000,CPU_CLK_UNHALTED.REF_TSC:sa=2300000,INST_RETIRED.ANY:sa=2300000 openmc amplxe: Warning: To profile kernel modules during the session, make sure they are available in the /lib/modules/kernel_version/ location. amplxe: Collection started. To stop the collection, either press CTRL-C or enter from another console window: amplxe-cl -r /home/wboyd/ecp-benchmarks/pin-cell/r001runsa -command stop. ... amplxe: Collection stopped. amplxe: Using result path `/home/wboyd/ecp-benchmarks/pin-cell/r001runsa' amplxe: Executing actions 0 % amplxe: Warning: The result contains a lot of raw data. Finalization may take much time. Use the -finalization-mode=deferred option to finalize the result on a faster single-thread performance machine. amplxe: Executing actions 12 % Loading '91917.perf' file amplxe: Executing actions 19 % Resolving information for `libhdf5_fortran.so.10 amplxe: Warning: Cannot locate file `/lib/modules/3.10.0-514.2.2.el7.x86_64/kernel/fs/jbd2/jbd2.ko'. amplxe: Warning: Cannot locate file `/lib/modules/3.10.0-514.2.2.el7.x86_64/kernel/fs/nfs/nfs.ko'. amplxe: Warning: Cannot locate file `/lib/modules/3.10.0-514.2.2.el7.x86_64/kernel/fs/ext4/ext4.ko'. amplxe: Executing actions 19 % Resolving information for `libhdf5_hl.so.100.0.0 amplxe: Warning: Cannot locate file `/lib/modules/3.10.0-514.2.2.el7.x86_64/kernel/drivers/ata/libahci.ko'. amplxe: Executing actions 19 % Resolving information for `libahci.ko' amplxe: Warning: Cannot locate debugging symbols for file `/usr/lib64/libc-2.17.so<http://libc-2.17.so>'. amplxe: Executing actions 19 % Resolving information for `libpthread-2.17.so<http://libpthread-2.17.so>' amplxe: Warning: Cannot locate debugging symbols for file `/usr/lib64/libpthread-2.17.so<http://libpthread-2.17.so>'. amplxe: Executing actions 20 % Resolving information for `ld-2.17.so<http://ld-2.17.so>' amplxe: Warning: Cannot locate debugging symbols for file `/usr/lib64/ld-2.17.so<http://ld-2.17.so>'. amplxe: Executing actions 20 % Resolving information for `libm-2.17.so<http://libm-2.17.so>' amplxe: Warning: Cannot locate file `/lib/modules/3.10.0-514.2.2.el7.x86_64/kernel/net/sunrpc/sunrpc.ko'. amplxe: Executing actions 20 % Resolving information for `sunrpc.ko' amplxe: Warning: Cannot locate debugging symbols for file `/usr/lib64/libm-2.17.so<http://libm-2.17.so>'. amplxe: Warning: Cannot locate file `/lib/modules/3.10.0-514.2.2.el7.x86_64/kernel/fs/lockd/lockd.ko'. amplxe: Executing actions 20 % Resolving information for `libgomp.so.1.0.0' amplxe: Warning: %CantOpenFile amplxe: Warning: Cannot locate file `/lib/modules/3.10.0-514.2.2.el7.x86_64/kernel/drivers/ata/libata.ko'. amplxe: Executing actions 20 % Resolving information for `libata.ko' amplxe: Warning: Cannot locate debugging symbols for file `/usr/lib64/libgomp.so.1.0.0'. amplxe: Executing actions 21 % Resolving information for `libgfortran.so.3.0.0' amplxe: Warning: Cannot locate debugging symbols for file `/usr/lib64/libgfortran.so.3.0.0'. amplxe: Warning: Cannot locate file `/lib/modules/3.10.0-514.2.2.el7.x86_64/kernel/drivers/net/ethernet/intel/igb/igb.ko'. amplxe: Executing actions 21 % Resolving information for `libmpi.so.0.0.0' amplxe: Warning: Cannot locate file `/lib/modules/3.10.0-514.2.2.el7.x86_64/kernel/fs/nfs/nfsv3.ko'. amplxe: Executing actions 21 % Resolving information for `libgcc_s-4.8.5-201507 amplxe: Warning: Cannot locate debugging symbols for file `/usr/lib64/libgcc_s-4.8.5-20150702.so.1'. amplxe: Executing actions 75 % Generating a report Collection and Platform Info ---------------------------- Parameter r001runsa ------------------------ ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Application Command Line openmc Operating System 3.10.0-514.2.2.el7.x86_64 NAME="Red Hat Enterprise Linux Server" VERSION="7.3 (Maipo)" ID="rhel" ID_LIKE="fedora" VERSION_ID="7.3" PRETTY_NAME="Red Hat Enterprise Linux Server 7.3 (Maipo)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:redhat:enterprise_linux:7.3:GA:server" HOME_URL="https://www.redhat.com/" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" REDHAT_BUGZILLA_PRODUCT_VERSION=7.3 REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" REDHAT_SUPPORT_PRODUCT_VERSION="7.3" Computer Name knl05.ftm.alcf.anl.gov<http://knl05.ftm.alcf.anl.gov> Result Size 1794777221 Collection start time 15:00:58 06/01/2017 UTC Collection stop time 15:03:03 06/01/2017 UTC CPU --- Parameter r001runsa ----------------- --------------------------------------------- Name Intel(R) Processor code named Knights Landing Frequency 1296995999 Logical CPU Count 256 Summary ------- Elapsed Time: 124.992 CPU Time: 6677.394 Average CPU Usage: 53.360 CPI Rate: 3.738 Event summary ------------- Hardware Event Type Hardware Event Count:Self Hardware Event Sample Count:Self Events Per Sample ------------------------ ------------------------- -------------------------------- ----------------- CPU_CLK_UNHALTED.THREAD 6661480800000 2896296 2300000 CPU_CLK_UNHALTED.REF_TSC 8660553400000 3765458 2300000 INST_RETIRED.ANY 1781975600000 774772 2300000 amplxe: Executing actions 100 % done On Fri, Jan 6, 2017 at 9:57 AM D'mello, Michael <[email protected]<mailto:[email protected]>> wrote: Will, This looks good. One more test: amplxe-cl -collect-with runsa -knob event-config=CPU_CLK_UNHALTED.THREAD:sa=2300000,CPU_CLK_UNHALTED.REF_TSC:sa=2300000,INST_RETIRED.ANY:sa=2300000 ./a.out If this works, we can do a similar thing with hpc-performance & memory access. I will send you those command lines. Still checking on a better solution. Regards -michael From: Will Boyd [mailto:[email protected]<mailto:[email protected]>] Sent: Friday, January 06, 2017 8:38 AM To: D'mello, Michael <[email protected]<mailto:[email protected]>>; Hammond, Jeff R <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] Using VTune remotely on the Phi Michael, I am running on the machine named "KNL" - in particular, I most recently ran on node "knl05". I ran the command you suggested and the following is the output: $ amplxe-cl -collect-with runsa –analyze-system –duration 5 amplxe: Collection started. To stop the collection, either press CTRL-C or enter from another console window: amplxe-cl -r /home/wboyd/r000runsa -command stop. amplxe: Collection detached. amplxe: Collection stopped. amplxe: Using result path `/home/wboyd/r000runsa' amplxe: Executing actions 100 % done amplxe: Error: 0x40000025 (Inapplicable report) -- The report 'summary' is not applicable to the result /home/wboyd/r000runsa/r000runsa.amplxe. Let me know if you have any insights based on the concluding error message. Best, Will On Thu, Jan 5, 2017 at 8:28 PM D'mello, Michael <[email protected]<mailto:[email protected]>> wrote: Hi Will, For some reason, I am not able to log into my account on the JLSE login node. My progress on the problem has therefore been delayed. While I sort out my login problems, could you try:

amplxe-cl -collect-with runsa –analyze-system –duration 5

Send me the output of this command. Also which JLSE KNL are you using? Regards -michael From: Will Boyd [mailto:[email protected]<mailto:[email protected]>] Sent: Thursday, January 05, 2017 10:55 AM To: D'mello, Michael <[email protected]<mailto:[email protected]>>; Hammond, Jeff R <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] Using VTune remotely on the Phi For some reason, "hpc-performance" is not provided as an "Analysis Type" in the GUI when the Intel Xeon Phi is selected as the "Analysis Target." Nevertheless, I selected "memory-access" as an "Analysis Type" and produced the following command line: amplxe-cl -target-system mic-native:mic0 -collect memory-access -knob analyze-openmp=true -app-working-dir /home/wboyd/ecp-benchmarks/pin-cell -- /home/wboyd/openmc/build/bin/openmc When launched from a KNL node on JLSE, this results in the following error: amplxe: Using target: mic-host-launch:mic0 amplxe: Error: Amplifier cannot detect Intel Xeon Phi coprocessor configuration. amplxe: Error: Amplifier cannot detect Intel Xeon Phi coprocessor configuration. Thanks in advance for any advice on how to sidestep this issue. On Thu, Jan 5, 2017 at 8:13 AM Will Boyd <[email protected]<mailto:[email protected]>> wrote: Thank you Michael. I don't have root privileges on JLSE, but I will try to extract the command line from the GUI as a workaround. Thanks for looking into this, I really appreciate the help. On Thu, Jan 5, 2017, 1:23 AM D'mello, Michael <[email protected]<mailto:[email protected]>> wrote: Will, From what I can tell, the system you are on seems to be configured to use the Linux “perf” collector by default. For “hpc-performance” to run with this collector, you will have to configure “perf” in system wide mode. The error you were seeing seems to be related to this. The following page provides some details: https://software.intel.com/en-us/perf_collection_from_lin (I have listed the relevant section below). Unfortunately, the information provided will not help directly (unless you have root privelages on jlse). One workaround is to configure the analysis in the gui & output the command line (click on the “Command Line” button). You will have to check the “Use –collect with action” box to get the command line you need. This approach should work, but as you will see the command line is complicated. You can certainly try this, but I believe there is a simpler workaround. More on this shortly. Regards -michael PS: The relevant section on the page is: NOTE • For uncore events collection, configure Perf to run in the system-wide mode by setting the /proc/sys/kernel/perf_event_paranoid value to 0 or less (root privileges are required). • For the kernel modules resolution<https://software.intel.com/node/84ddf2a4-d4a6-4027-92e3-dce7b0fd7044#GUID-8C1ADD41-432E-4FEA-A5B3-99022E32C258>, make sure you have enough permissions to read kernel symbols information from the /proc/kallsyms file. Perf-based driverless collection can be used for running Advanced Hotspots, General Exploration and Custom event-based sampling analysis types on Linux and Android OS. If the uncore events support is available on the system, the VTune Amplifier may also use the Perf collection for Memory Access, HPC Performance Characterization and General Exploration analysis types with the Analyze memory bandwidth option enabled. From: Will Boyd [mailto:[email protected]<mailto:[email protected]>] Sent: Wednesday, January 04, 2017 8:36 PM To: D'mello, Michael <[email protected]<mailto:[email protected]>>; Hammond, Jeff R <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] Using VTune remotely on the Phi Michael, Yes, that command works, along with "concurrency" and "locksandwaits". Only the "hpc-performance" and "memory-access" fail on KNL. Will On Wed, Jan 4, 2017, 8:09 PM D'mello, Michael <[email protected]<mailto:[email protected]>> wrote: Hi Will, Does this command work?

amplxe-cl –collect advanced-hotspots openmc

Regards -michael From: Will Boyd [mailto:[email protected]<mailto:[email protected]>] Sent: Wednesday, January 04, 2017 4:25 PM To: Hammond, Jeff R <[email protected]<mailto:[email protected]>>; D'mello, Michael <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Subject: Re: [JLSE-Intel-NDA] Using VTune remotely on the Phi I am trying to run VTune on KNL but am running into the following error: $ amplxe-cl -collect hpc-performance openmc amplxe: Error: both cgroup and no-aggregation modes only available in system-wide mode usage: perf stat [<options>] [<command>] -G, --cgroup <name> monitor event in cgroup name only -A, --no-aggr disable CPU count aggregation -a, --all-cpus system-wide collection from all CPUs This command works fine on a Haswell node, but breaks when I run it on KNL. I am able to run VTune with the "hotspots" collection argument without on KNL. Does anyone know how to resolve this issue? Thanks, Will On Tue, Dec 13, 2016 at 6:22 PM Hammond, Jeff R <[email protected]<mailto:[email protected]>> wrote: FWIW based upon my experience trying unsuccessfully to debug why NWChem won’t run in the Vtune GUI, I recommend always using Vtune CLI. It’s a bad habit to get used to fancy features like GUIs and dynamic memory allocation if you want to be a serious HPC person ;-) Jeff From: <[email protected]<mailto:[email protected]>> on behalf of Will Boyd <[email protected]<mailto:[email protected]>> Date: Tuesday, December 13, 2016 at 3:00 PM To: "D'mello, Michael" <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: [JLSE-Intel-NDA] Using VTune remotely on the Phi Thank you Michael and Jeff. Downloading the results from the VTune CLI and importing them into the VTune GUI locally works great. On Tue, Dec 13, 2016 at 5:35 PM D'mello, Michael <[email protected]<mailto:[email protected]>> wrote: Hi Will, You can use the command line interface to collect the profiling information. For example:

amplxe_cl –collect hotspots ./a.out

You can then move the results (entire directory) over to the environment (Windows or Linux) in which you would like to view the results. regards -michael From: [email protected]<mailto:[email protected]> [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Will Boyd Sent: Tuesday, December 13, 2016 4:30 PM To: [email protected]<mailto:[email protected]> Subject: [JLSE-Intel-NDA] Using VTune remotely on the Phi I am interested in profiling an application on the Phi using Intel VTune. I would like to use the graphical interface for VTune, but have been unable to launch VTune remotely using my local instance of VTune. Does anyone have any recommendations for how to remotely profile my application on the Phi but view the results locally with the VTune GUI on my workstation? Thanks, Will