November 2018 - Anl-intel-lrz-knr - mailman3.cels.anl.gov

Re: [anl-intel-lrz-knr] SDK release notes
by Knight, Christopher J. 29 Nov '18

29 Nov '18

Hi Tim, OK, thanks for the info. Thomas and I were thinking the same and this is related to his recent post on the bug issue tracking confluence page. chris On Nov 29, 2018, at 2:51 PM, Creech, Timothy M <timothy.m.creech(a)intel.com<mailto:[email protected]>> wrote: Hi Chris, In its current preview form, much of the buffer insertion output goes to stdout. (About 3.2 MB, in this case!) The relevant error message is in the last 40 or so lines: node lmp_jlse_csa0.fountain1.501 not reachable At this point the buffer insertion tool reports failure to the simulator. This looks like a buffer insertion tool bug. I will report it so that it can be fixed. Thanks, Tim From: Knight, Christopher J. [mailto:[email protected]] Sent: Tuesday, November 27, 2018 3:38 PM To: Creech, Timothy M <timothy.m.creech(a)intel.com<mailto:[email protected]>> Cc: anl-intel-knr(a)lists.jlse.anl.gov<mailto:[email protected]>; Glossop, Kent <kent.glossop(a)intel.com<mailto:[email protected]>> Subject: Re: [anl-intel-lrz-knr] SDK release notes Hi Tim, I was able to run the simple workshop example without issue using the buffer insertion tool. It seemed to be taking too long with 16 workers for the simple loop, but I ran 1- and 8-worker tests that completed successfully. I’m not having as good of luck with the LAMMPS test though. Within the large amount of output now generated, I can find messages like the following. 'lmp_jls???sim error: buffer insertion tool failed - see lmp_jlse_???0.graphml_log Could not get implementation graph from ???sim. Could not process config It doesn’t seem like the lmp_jlse_???0.graphml_log file has very helpful debugging information (below). Output from the test is in the following apps directory: /soft/restricted/intel_???/apps/lammps/knight/work/lj_full . Any suggestions for collecting helpful debugging info? chris ============ [knightc@jlselogin1 lj_full]$ cat lmp_jlse_???0.graphml_log ???_asm2graphml: info: Allocate a ??? tile. ???_asm2graphml: info: Load the module. ???_asm2graphml: info: Create the graphml file. ???_asm2graphml: info: Report statistics. ???_asm2graphml: info: Elapsed time: 0.044939 secs. ???_asm2graphml: info: graphML file is lmp_jlse_???0.graphml assembly dump file is lmp_jlse_???0-???*.dump log file is lmp_jlse_???0.graphml_log On Nov 20, 2018, at 2:13 PM, Creech, Timothy M <timothy.m.creech(a)intel.com<mailto:[email protected]>> wrote: We have installed the missing python packages required to run the buffer insertion tool, and it should now be working on JLSE. Please do let us know if other issues are encountered. Thanks, Tim From: anl-intel-lrz-knr [mailto:[email protected]] On Behalf Of Glossop, Kent via anl-intel-lrz-knr Sent: Thursday, November 15, 2018 8:11 PM To: Knight, Christopher J. <knightc(a)anl.gov<mailto:[email protected]>> Cc: anl-intel-knr(a)lists.jlse.anl.gov<mailto:[email protected]> Subject: Re: [anl-intel-lrz-knr] SDK release notes The short answer is there is a list of packages that must be installed for the SDK to work. That set increased with this release, and apparently includes package(s) not installed on JLSE. I was afraid we were going to get into this situation. We’ll have some discussions tomorrow and get back to you. Kent From: Knight, Christopher J. [mailto:[email protected]] Sent: Thursday, November 15, 2018 6:22 PM To: Glossop, Kent <kent.glossop(a)intel.com<mailto:[email protected]>> Cc: anl-intel-knr(a)lists.jlse.anl.gov<mailto:[email protected]> Subject: Re: [anl-intel-lrz-knr] SDK release notes Hi Kent, Was the buffer insertion tool tested in JLSE’s environment? I see error messages (copied below) when testing LAMMPS and one of the workshop examples. chris ========= [LAMMPS] ... Run 0: Running __omp_offloading_27_6d5bf36__ZN9LAMMPS_NS19PairLJCutFull???OMP4evalILi1EEEvii_l239 on the ??? .. Traceback (most recent call last): File "/soft/restricted/intel_???/comp/base/2018-11-06-release/bin/???_buffer_insertion", line 5, in <module> import networkx ImportError: No module named networkx ???sim error: buffer insertion tool failed - see lmp_jlse_???0.graphml_log Could not get implementation graph from ???sim. Could not process config [Workshop Example: derived from /soft/restricted/intel_???/apps/workshop2_examples/jit-c-3] [knightc@thing03 compiler-c-3]$ env ???SIM_BUFFER_INSERT=1 ./exe N= 100000 invoke on host... results= 704982704 704982704 invoke on ???... Using simulator from /soft/restricted/intel_???/comp/base/2018-11-06-release/lib/lib???sim.so ???sim warning: Address Space Layout Randomization (ASLR) is enabled. This can change the way memory accesses cross cache lines, which can change the elapsed cycle count. ASLR can be disabled for a run using "setarch x86_64 -R ./myapp". Run 0: Running __omp_offloading_27_60a6ea5_loop_???_l8 on the ??? .. ???sim error: ???_asm2graphml tool failed - see .graphml_log Could not get implementation graph from ???sim. Could not process config ???sim error: ???_asm2graphml tool failed - see .graphml_log Could not get implementation graph from ???sim. Could not process config Run 0: __omp_offloading_27_60a6ea5_loop_???_l8 ran on the ??? in 0 cycles ???_asm2graphml: Unrecognized filename /tmp/tmpfile_p37DaP???_asm2graphml: Unrecognized filename /tmp/tmpfile_p37DaPresults= 0 0 On Nov 15, 2018, at 9:44 AM, Glossop, Kent via anl-intel-lrz-knr <anl-intel-lrz-knr(a)lists.jlse.anl.gov<mailto:[email protected]>> wrote: A new release of ??? SDK components has been installed, with some notable new items including Fortran. Please see $???_ROOT/doc/???-Release-Notes-2018-11-12.txt for details. _______________________________________________ anl-intel-lrz-knr mailing list anl-intel-lrz-knr(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/anl-intel-lrz-knr

1 0

Re: [anl-intel-lrz-knr] SDK release notes
by Creech, Timothy M 29 Nov '18

29 Nov '18

Hi Chris, In its current preview form, much of the buffer insertion output goes to stdout. (About 3.2 MB, in this case!) The relevant error message is in the last 40 or so lines: node lmp_jlse_csa0.fountain1.501 not reachable At this point the buffer insertion tool reports failure to the simulator. This looks like a buffer insertion tool bug. I will report it so that it can be fixed. Thanks, Tim From: Knight, Christopher J. [mailto:[email protected]] Sent: Tuesday, November 27, 2018 3:38 PM To: Creech, Timothy M <timothy.m.creech(a)intel.com> Cc: anl-intel-knr(a)lists.jlse.anl.gov; Glossop, Kent <kent.glossop(a)intel.com> Subject: Re: [anl-intel-lrz-knr] SDK release notes Hi Tim, I was able to run the simple workshop example without issue using the buffer insertion tool. It seemed to be taking too long with 16 workers for the simple loop, but I ran 1- and 8-worker tests that completed successfully. I’m not having as good of luck with the LAMMPS test though. Within the large amount of output now generated, I can find messages like the following. 'lmp_jls???sim error: buffer insertion tool failed - see lmp_jlse_???0.graphml_log Could not get implementation graph from ???sim. Could not process config It doesn’t seem like the lmp_jlse_???0.graphml_log file has very helpful debugging information (below). Output from the test is in the following apps directory: /soft/restricted/intel_???/apps/lammps/knight/work/lj_full . Any suggestions for collecting helpful debugging info? chris ============ [knightc@jlselogin1 lj_full]$ cat lmp_jlse_???0.graphml_log ???_asm2graphml: info: Allocate a ??? tile. ???_asm2graphml: info: Load the module. ???_asm2graphml: info: Create the graphml file. ???_asm2graphml: info: Report statistics. ???_asm2graphml: info: Elapsed time: 0.044939 secs. ???_asm2graphml: info: graphML file is lmp_jlse_???0.graphml assembly dump file is lmp_jlse_???0-???*.dump log file is lmp_jlse_???0.graphml_log On Nov 20, 2018, at 2:13 PM, Creech, Timothy M <timothy.m.creech(a)intel.com<mailto:[email protected]>> wrote: We have installed the missing python packages required to run the buffer insertion tool, and it should now be working on JLSE. Please do let us know if other issues are encountered. Thanks, Tim From: anl-intel-lrz-knr [mailto:[email protected]] On Behalf Of Glossop, Kent via anl-intel-lrz-knr Sent: Thursday, November 15, 2018 8:11 PM To: Knight, Christopher J. <knightc(a)anl.gov<mailto:[email protected]>> Cc: anl-intel-knr(a)lists.jlse.anl.gov<mailto:[email protected]> Subject: Re: [anl-intel-lrz-knr] SDK release notes The short answer is there is a list of packages that must be installed for the SDK to work. That set increased with this release, and apparently includes package(s) not installed on JLSE. I was afraid we were going to get into this situation. We’ll have some discussions tomorrow and get back to you. Kent From: Knight, Christopher J. [mailto:[email protected]] Sent: Thursday, November 15, 2018 6:22 PM To: Glossop, Kent <kent.glossop(a)intel.com<mailto:[email protected]>> Cc: anl-intel-knr(a)lists.jlse.anl.gov<mailto:[email protected]> Subject: Re: [anl-intel-lrz-knr] SDK release notes Hi Kent, Was the buffer insertion tool tested in JLSE’s environment? I see error messages (copied below) when testing LAMMPS and one of the workshop examples. chris ========= [LAMMPS] ... Run 0: Running __omp_offloading_27_6d5bf36__ZN9LAMMPS_NS19PairLJCutFull???OMP4evalILi1EEEvii_l239 on the ??? .. Traceback (most recent call last): File "/soft/restricted/intel_???/comp/base/2018-11-06-release/bin/???_buffer_insertion", line 5, in <module> import networkx ImportError: No module named networkx ???sim error: buffer insertion tool failed - see lmp_jlse_???0.graphml_log Could not get implementation graph from ???sim. Could not process config [Workshop Example: derived from /soft/restricted/intel_???/apps/workshop2_examples/jit-c-3] [knightc@thing03 compiler-c-3]$ env ???SIM_BUFFER_INSERT=1 ./exe N= 100000 invoke on host... results= 704982704 704982704 invoke on ???... Using simulator from /soft/restricted/intel_???/comp/base/2018-11-06-release/lib/lib???sim.so ???sim warning: Address Space Layout Randomization (ASLR) is enabled. This can change the way memory accesses cross cache lines, which can change the elapsed cycle count. ASLR can be disabled for a run using "setarch x86_64 -R ./myapp". Run 0: Running __omp_offloading_27_60a6ea5_loop_???_l8 on the ??? .. ???sim error: ???_asm2graphml tool failed - see .graphml_log Could not get implementation graph from ???sim. Could not process config ???sim error: ???_asm2graphml tool failed - see .graphml_log Could not get implementation graph from ???sim. Could not process config Run 0: __omp_offloading_27_60a6ea5_loop_???_l8 ran on the ??? in 0 cycles ???_asm2graphml: Unrecognized filename /tmp/tmpfile_p37DaP???_asm2graphml: Unrecognized filename /tmp/tmpfile_p37DaPresults= 0 0 On Nov 15, 2018, at 9:44 AM, Glossop, Kent via anl-intel-lrz-knr <anl-intel-lrz-knr(a)lists.jlse.anl.gov<mailto:[email protected]>> wrote: A new release of ??? SDK components has been installed, with some notable new items including Fortran. Please see $???_ROOT/doc/???-Release-Notes-2018-11-12.txt for details. _______________________________________________ anl-intel-lrz-knr mailing list anl-intel-lrz-knr(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/anl-intel-lrz-knr

1 0

Re: [anl-intel-lrz-knr] preview Fortran compiler on JLSE
by Applencourt, Thomas 28 Nov '18

28 Nov '18

Errata: The documentation is on page 14 not 17. (https://anl.app.box.com/file/358151835663) Regards, Thomas PS: Thanks Alvaro Thomas Applencourt Leadership Computing Facility Argonne National Laboratory Building 240 Office - 1.D.18 9700 S Cass Av., Argonne Il, 60439 (630) 252-0036 ________________________________ From: Vazquez Mayagoitia, Alvaro Sent: Wednesday, November 28, 2018 15:03 To: Applencourt, Thomas Subject: Re: [anl-intel-lrz-knr] preview Fortran compiler on JLSE Thomas, I only see 15 pages, could you confirm? Thanks -Álvaro Vázquez-Mayagoitia, PhD Theoretical and computational chemistry Argonne National Laboratory https://web.alcf.anl.gov/~vama/ Álvaro Vázquez-Mayagoitia, PhD - Argonne National Laboratory<https://web.alcf.anl.gov/~vama/> web.alcf.anl.gov Electronic Structure Infrastructure. Open-source solvers interface for massive parallel electronic structure simulations of solids and clusters. On 11/28/18 2:29 PM, Applencourt, Thomas via anl-intel-lrz-knr wrote: Please note that only fortran77 is supported right now, and the kernel you want to offload need to be extracted in a standalone file. For more information see page 17 of https://anl.app.box.com/file/358151835663 Also, please don't hesitate to contact me if you have any question regarding *** and/or Fortran. Thomas Thomas Applencourt Leadership Computing Facility Argonne National Laboratory Building 240 Office - 1.D.18 9700 S Cass Av., Argonne Il, 60439 (630) 252-0036 ________________________________ From: anl-intel-lrz-knr <anl-intel-lrz-knr-bounces(a)lists.jlse.anl.gov><mailto:[email protected]> on behalf of Bertoni, Colleen via anl-intel-lrz-knr <anl-intel-lrz-knr(a)lists.jlse.anl.gov><mailto:[email protected]> Sent: Wednesday, November 28, 2018 14:23 To: anl-intel-lrz-knr(a)lists.jlse.anl.gov<mailto:[email protected]> Subject: [anl-intel-lrz-knr] preview Fortran compiler on JLSE Hi everyone, This is a note to mention that there's a preview Fortran compiler on JLSE now. So if you have a Fortran application that you want to try porting, now is a good time. See Section 3.2 of the Getting Started Guide (in /soft/restricted/intel_***/doc/ on JLSE) for details. Feedback or bug reports would be appreciated. Thanks, Colleen

1 0

Re: [anl-intel-lrz-knr] preview Fortran compiler on JLSE
by Applencourt, Thomas 28 Nov '18

28 Nov '18

Please note that only fortran77 is supported right now, and the kernel you want to offload need to be extracted in a standalone file. For more information see page 17 of https://anl.app.box.com/file/358151835663 Also, please don't hesitate to contact me if you have any question regarding *** and/or Fortran. Thomas Thomas Applencourt Leadership Computing Facility Argonne National Laboratory Building 240 Office - 1.D.18 9700 S Cass Av., Argonne Il, 60439 (630) 252-0036 ________________________________ From: anl-intel-lrz-knr <anl-intel-lrz-knr-bounces(a)lists.jlse.anl.gov> on behalf of Bertoni, Colleen via anl-intel-lrz-knr <anl-intel-lrz-knr(a)lists.jlse.anl.gov> Sent: Wednesday, November 28, 2018 14:23 To: anl-intel-lrz-knr(a)lists.jlse.anl.gov Subject: [anl-intel-lrz-knr] preview Fortran compiler on JLSE Hi everyone, This is a note to mention that there's a preview Fortran compiler on JLSE now. So if you have a Fortran application that you want to try porting, now is a good time. See Section 3.2 of the Getting Started Guide (in /soft/restricted/intel_***/doc/ on JLSE) for details. Feedback or bug reports would be appreciated. Thanks, Colleen

1 0

preview Fortran compiler on JLSE
by Bertoni, Colleen 28 Nov '18

28 Nov '18

Hi everyone, This is a note to mention that there's a preview Fortran compiler on JLSE now. So if you have a Fortran application that you want to try porting, now is a good time. See Section 3.2 of the Getting Started Guide (in /soft/restricted/intel_***/doc/ on JLSE) for details. Feedback or bug reports would be appreciated. Thanks, Colleen

1 0

Re: [anl-intel-lrz-knr] SDK release notes
by Knight, Christopher J. 27 Nov '18

27 Nov '18

Hi Tim, I was able to run the simple workshop example without issue using the buffer insertion tool. It seemed to be taking too long with 16 workers for the simple loop, but I ran 1- and 8-worker tests that completed successfully. I’m not having as good of luck with the LAMMPS test though. Within the large amount of output now generated, I can find messages like the following. 'lmp_jls???sim error: buffer insertion tool failed - see lmp_jlse_???0.graphml_log Could not get implementation graph from ???sim. Could not process config It doesn’t seem like the lmp_jlse_???0.graphml_log file has very helpful debugging information (below). Output from the test is in the following apps directory: /soft/restricted/intel_???/apps/lammps/knight/work/lj_full . Any suggestions for collecting helpful debugging info? chris ============ [knightc@jlselogin1 lj_full]$ cat lmp_jlse_???0.graphml_log ???_asm2graphml: info: Allocate a ??? tile. ???_asm2graphml: info: Load the module. ???_asm2graphml: info: Create the graphml file. ???_asm2graphml: info: Report statistics. ???_asm2graphml: info: Elapsed time: 0.044939 secs. ???_asm2graphml: info: graphML file is lmp_jlse_???0.graphml assembly dump file is lmp_jlse_???0-???*.dump log file is lmp_jlse_???0.graphml_log On Nov 20, 2018, at 2:13 PM, Creech, Timothy M <timothy.m.creech(a)intel.com<mailto:[email protected]>> wrote: We have installed the missing python packages required to run the buffer insertion tool, and it should now be working on JLSE. Please do let us know if other issues are encountered. Thanks, Tim From: anl-intel-lrz-knr [mailto:[email protected]] On Behalf Of Glossop, Kent via anl-intel-lrz-knr Sent: Thursday, November 15, 2018 8:11 PM To: Knight, Christopher J. <knightc(a)anl.gov<mailto:[email protected]>> Cc: anl-intel-knr(a)lists.jlse.anl.gov<mailto:[email protected]> Subject: Re: [anl-intel-lrz-knr] SDK release notes The short answer is there is a list of packages that must be installed for the SDK to work. That set increased with this release, and apparently includes package(s) not installed on JLSE. I was afraid we were going to get into this situation. We’ll have some discussions tomorrow and get back to you. Kent From: Knight, Christopher J. [mailto:[email protected]] Sent: Thursday, November 15, 2018 6:22 PM To: Glossop, Kent <kent.glossop(a)intel.com<mailto:[email protected]>> Cc: anl-intel-knr(a)lists.jlse.anl.gov<mailto:[email protected]> Subject: Re: [anl-intel-lrz-knr] SDK release notes Hi Kent, Was the buffer insertion tool tested in JLSE’s environment? I see error messages (copied below) when testing LAMMPS and one of the workshop examples. chris ========= [LAMMPS] ... Run 0: Running __omp_offloading_27_6d5bf36__ZN9LAMMPS_NS19PairLJCutFull???OMP4evalILi1EEEvii_l239 on the ??? .. Traceback (most recent call last): File "/soft/restricted/intel_???/comp/base/2018-11-06-release/bin/???_buffer_insertion", line 5, in <module> import networkx ImportError: No module named networkx ???sim error: buffer insertion tool failed - see lmp_jlse_???0.graphml_log Could not get implementation graph from ???sim. Could not process config [Workshop Example: derived from /soft/restricted/intel_???/apps/workshop2_examples/jit-c-3] [knightc@thing03 compiler-c-3]$ env ???SIM_BUFFER_INSERT=1 ./exe N= 100000 invoke on host... results= 704982704 704982704 invoke on ???... Using simulator from /soft/restricted/intel_???/comp/base/2018-11-06-release/lib/lib???sim.so ???sim warning: Address Space Layout Randomization (ASLR) is enabled. This can change the way memory accesses cross cache lines, which can change the elapsed cycle count. ASLR can be disabled for a run using "setarch x86_64 -R ./myapp". Run 0: Running __omp_offloading_27_60a6ea5_loop_???_l8 on the ??? .. ???sim error: ???_asm2graphml tool failed - see .graphml_log Could not get implementation graph from ???sim. Could not process config ???sim error: ???_asm2graphml tool failed - see .graphml_log Could not get implementation graph from ???sim. Could not process config Run 0: __omp_offloading_27_60a6ea5_loop_???_l8 ran on the ??? in 0 cycles ???_asm2graphml: Unrecognized filename /tmp/tmpfile_p37DaP???_asm2graphml: Unrecognized filename /tmp/tmpfile_p37DaPresults= 0 0 On Nov 15, 2018, at 9:44 AM, Glossop, Kent via anl-intel-lrz-knr <anl-intel-lrz-knr(a)lists.jlse.anl.gov<mailto:[email protected]>> wrote: A new release of ??? SDK components has been installed, with some notable new items including Fortran. Please see $???_ROOT/doc/???-Release-Notes-2018-11-12.txt for details. _______________________________________________ anl-intel-lrz-knr mailing list anl-intel-lrz-knr(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/anl-intel-lrz-knr

1 0

Re: [anl-intel-lrz-knr] atomic issues
by Jin, Zheming 26 Nov '18

26 Nov '18

Thank you again for your explanations! The optimization report number ranges from 0 to 3. I think ICX will save an optimization report (filename.optrpt) like ICC in the future release. ________________________________ From: Woodworth, Daniel <daniel.woodworth(a)intel.com> Sent: Monday, November 26, 2018 5:50:04 PM To: Jin, Zheming Cc: anl-intel-lrz-knr(a)lists.jlse.anl.gov Subject: RE: atomic issues One of the things that makes this discussion confusing is that atomic loads (or at least ones that are <= 64 bits and properly aligned) are actually just lowered to normal load ops because (properly aligned) scalar loads and stores are already atomic on our architecture. So the lack of “atm” anywhere in the assembly doesn’t mean that the #pragma omp atomic read was ignored. There are some types of atomics that need extra ordering relative to other operations to make sure that the results of those operations are visible on other tiles/cores, but in your example the atomic loads are monotonic so they don’t fall in that category. In your case the main thing that the #pragma omp atomic read is doing is preventing the compiler from splitting up the load into a few smaller loads. Even if the load wasn’t atomic the compiler probably wasn’t that likely to do that anyway, though until recently there was one corner case where it might have inadvertently broken up (non-atomic) stores. In any case, when I tried your example using a development build of the compiler it was tracking the atomicity of the load throughout, so the #pragma omp atomic read is working as intended. That’s not quite what I meant: my point was that the semantics of num_threads applied to #pragma omp parallel refer to real OpenMP threads (of which there is only one per tile) while the semantics of num_threads applied to #pragma omp parallel for actually refer to workers instead (and there can be many workers instantiated per tile). This makes the meaning of the clause inconsistent between different constructs on the target side despite the spelling being the same. How this relates to the host is an entirely different discussion, but we are making progress in formalizing things so that the way that the target-side constructs behave is more consistent with the way things work on the host. The option for disabling all of the extra functions is “-fno-<insert acronym here>-libs”. I’m not sure if there’s a dedicated mode for storing more verbose optimization reports to a file, but you can always redirect the output from the command in the shell. From: Jin, Zheming [mailto:[email protected]] Sent: Monday, November 26, 2018 5:06 PM To: Woodworth, Daniel <daniel.woodworth(a)intel.com> Cc: anl-intel-lrz-knr(a)lists.jlse.anl.gov Subject: Re: atomic issues HI Daniel, Thanks for your reply. I have some questions after reading your reply. For the 2rd question, I am not sure if the atomic instruction should be generated by the compiler. The compiler flag is used. I listed the function at the bottom. For the 3rd question, syntactically consistent means that we can use the same OpenMP syntax (num_threads) for both CPU and KNR. Semantically consistent means that we need to use different syntax. Is that right ? I tried -qopt-report=1,2,3. The optimization message for the irrelevant functions may be disabled as they are not called in the C function. It seems that the optimization report is not saved in the current directory. LOOP BEGIN at atomic_read.c (16, 13) LOOP END ================================================================= Global loop optimization report for : ceil ================================================================= Global loop optimization report for : ceilf ================================================================= Global loop optimization report for : dp_sqrt_RD ================================================================= Global loop optimization report for : dp_sqrt_RN ================================================================= Global loop optimization report for : dp_sqrt_RU ================================================================= Global loop optimization report for : dp_sqrt_RZ ================================================================= Global loop optimization report for : exp2f ================================================================= .... .... filename: atomic_read.c #include <stdio.h> #include <stdatomic.h> #define N 256 int main(void) { int temp[N]; int x[N]; for (int i = 0; i < N; i++) x[i] = i; #pragma omp target map(from: temp[0:N]) map(to: x[0:N]) { #pragma omp parallel for for(int i = 0; i < N; i++) { #pragma omp atomic read temp[i] = x[i%N]; } } int error = 0; for (int i = 0; i < N; i++) if (temp[i] != i) { error = 1; break; } if (error) printf("Fail\n"); else printf("Success\n"); return 0; } ________________________________ From: Woodworth, Daniel <daniel.woodworth(a)intel.com<mailto:[email protected]>> Sent: Monday, November 26, 2018 2:54:46 PM To: Jin, Zheming Cc: anl-intel-lrz-knr(a)lists.jlse.anl.gov<mailto:[email protected]> Subject: RE: atomic issues 1. The compiler doesn’t support fences yet, but we are currently working on an overhaul of how memory operations are handled by the compiler and fence support is one of the things that will be added as part of that. There are also several cases where the lowering for some OpenMP constructs uses fences where they aren’t really needed and this is also something that we are working on fixing. Either way, there are fixes for this planned for the next release. 2. ‘#pragma omp atomic read’ should work in the latest compiler, though for types larger than 64 bits it falls back to runtime support that is not implemented on our target yet. Where are you seeing the warning? One other thing to keep in mind is that all of the OpenMP pragmas are ignored if -fiopenmp is not passed on the target side, so it might be worth double-checking that that compiler flag is present. 3. The tile is modeled as a single OpenMP thread, so the #pragma omp parallel will have a single thread allocated to it and the block will run once, giving a result of 2.0. This is slightly at odds with our use of num_threads to specify a worker count in #pragma omp parallel for; we’re eventually planning on retiring that and replacing it with new syntax that is more semantically consistent, but we don’t have that ready yet so num_threads is still being used for worker count there for the time being. From: anl-intel-lrz-knr [mailto:[email protected]] On Behalf Of Jin, Zheming via anl-intel-lrz-knr Sent: Monday, November 19, 2018 6:21 PM To: anl-intel-lrz-knr(a)lists.jlse.anl.gov<mailto:[email protected]> Subject: [anl-intel-lrz-knr] atomic issues 1 The latest compiler can give an error message before producing the message: fatal error: error in backend: Cannot select: 0x60cca98: ch = AtomicFence 2 If '#pragma omp atomic read' is ignored by the latest compiler, a warning may be produced. 3 What is the expected value for the following program on the latest simulator ? double r = 0; double i = 1; #pragma omp target map(tofrom: r) map(to:i) { #pragma omp parallel num_threads(4) { double tmp = i+1; #pragma omp atomic r += tmp; } } printf("r = %lf\n", r); Thanks

1 0

Re: [anl-intel-lrz-knr] atomic issues
by Woodworth, Daniel 26 Nov '18

26 Nov '18

One of the things that makes this discussion confusing is that atomic loads (or at least ones that are <= 64 bits and properly aligned) are actually just lowered to normal load ops because (properly aligned) scalar loads and stores are already atomic on our architecture. So the lack of “atm” anywhere in the assembly doesn’t mean that the #pragma omp atomic read was ignored. There are some types of atomics that need extra ordering relative to other operations to make sure that the results of those operations are visible on other tiles/cores, but in your example the atomic loads are monotonic so they don’t fall in that category. In your case the main thing that the #pragma omp atomic read is doing is preventing the compiler from splitting up the load into a few smaller loads. Even if the load wasn’t atomic the compiler probably wasn’t that likely to do that anyway, though until recently there was one corner case where it might have inadvertently broken up (non-atomic) stores. In any case, when I tried your example using a development build of the compiler it was tracking the atomicity of the load throughout, so the #pragma omp atomic read is working as intended. That’s not quite what I meant: my point was that the semantics of num_threads applied to #pragma omp parallel refer to real OpenMP threads (of which there is only one per tile) while the semantics of num_threads applied to #pragma omp parallel for actually refer to workers instead (and there can be many workers instantiated per tile). This makes the meaning of the clause inconsistent between different constructs on the target side despite the spelling being the same. How this relates to the host is an entirely different discussion, but we are making progress in formalizing things so that the way that the target-side constructs behave is more consistent with the way things work on the host. The option for disabling all of the extra functions is “-fno-<insert acronym here>-libs”. I’m not sure if there’s a dedicated mode for storing more verbose optimization reports to a file, but you can always redirect the output from the command in the shell. From: Jin, Zheming [mailto:[email protected]] Sent: Monday, November 26, 2018 5:06 PM To: Woodworth, Daniel <daniel.woodworth(a)intel.com> Cc: anl-intel-lrz-knr(a)lists.jlse.anl.gov Subject: Re: atomic issues HI Daniel, Thanks for your reply. I have some questions after reading your reply. For the 2rd question, I am not sure if the atomic instruction should be generated by the compiler. The compiler flag is used. I listed the function at the bottom. For the 3rd question, syntactically consistent means that we can use the same OpenMP syntax (num_threads) for both CPU and KNR. Semantically consistent means that we need to use different syntax. Is that right ? I tried -qopt-report=1,2,3. The optimization message for the irrelevant functions may be disabled as they are not called in the C function. It seems that the optimization report is not saved in the current directory. LOOP BEGIN at atomic_read.c (16, 13) LOOP END ================================================================= Global loop optimization report for : ceil ================================================================= Global loop optimization report for : ceilf ================================================================= Global loop optimization report for : dp_sqrt_RD ================================================================= Global loop optimization report for : dp_sqrt_RN ================================================================= Global loop optimization report for : dp_sqrt_RU ================================================================= Global loop optimization report for : dp_sqrt_RZ ================================================================= Global loop optimization report for : exp2f ================================================================= .... .... filename: atomic_read.c #include <stdio.h> #include <stdatomic.h> #define N 256 int main(void) { int temp[N]; int x[N]; for (int i = 0; i < N; i++) x[i] = i; #pragma omp target map(from: temp[0:N]) map(to: x[0:N]) { #pragma omp parallel for for(int i = 0; i < N; i++) { #pragma omp atomic read temp[i] = x[i%N]; } } int error = 0; for (int i = 0; i < N; i++) if (temp[i] != i) { error = 1; break; } if (error) printf("Fail\n"); else printf("Success\n"); return 0; } ________________________________ From: Woodworth, Daniel <daniel.woodworth(a)intel.com<mailto:[email protected]>> Sent: Monday, November 26, 2018 2:54:46 PM To: Jin, Zheming Cc: anl-intel-lrz-knr(a)lists.jlse.anl.gov<mailto:[email protected]> Subject: RE: atomic issues 1. The compiler doesn’t support fences yet, but we are currently working on an overhaul of how memory operations are handled by the compiler and fence support is one of the things that will be added as part of that. There are also several cases where the lowering for some OpenMP constructs uses fences where they aren’t really needed and this is also something that we are working on fixing. Either way, there are fixes for this planned for the next release. 2. ‘#pragma omp atomic read’ should work in the latest compiler, though for types larger than 64 bits it falls back to runtime support that is not implemented on our target yet. Where are you seeing the warning? One other thing to keep in mind is that all of the OpenMP pragmas are ignored if -fiopenmp is not passed on the target side, so it might be worth double-checking that that compiler flag is present. 3. The tile is modeled as a single OpenMP thread, so the #pragma omp parallel will have a single thread allocated to it and the block will run once, giving a result of 2.0. This is slightly at odds with our use of num_threads to specify a worker count in #pragma omp parallel for; we’re eventually planning on retiring that and replacing it with new syntax that is more semantically consistent, but we don’t have that ready yet so num_threads is still being used for worker count there for the time being. From: anl-intel-lrz-knr [mailto:[email protected]] On Behalf Of Jin, Zheming via anl-intel-lrz-knr Sent: Monday, November 19, 2018 6:21 PM To: anl-intel-lrz-knr(a)lists.jlse.anl.gov<mailto:[email protected]> Subject: [anl-intel-lrz-knr] atomic issues 1 The latest compiler can give an error message before producing the message: fatal error: error in backend: Cannot select: 0x60cca98: ch = AtomicFence 2 If '#pragma omp atomic read' is ignored by the latest compiler, a warning may be produced. 3 What is the expected value for the following program on the latest simulator ? double r = 0; double i = 1; #pragma omp target map(tofrom: r) map(to:i) { #pragma omp parallel num_threads(4) { double tmp = i+1; #pragma omp atomic r += tmp; } } printf("r = %lf\n", r); Thanks

1 0

Re: [anl-intel-lrz-knr] atomic issues
by Jin, Zheming 26 Nov '18

26 Nov '18

HI Daniel, Thanks for your reply. I have some questions after reading your reply. For the 2rd question, I am not sure if the atomic instruction should be generated by the compiler. The compiler flag is used. I listed the function at the bottom. For the 3rd question, syntactically consistent means that we can use the same OpenMP syntax (num_threads) for both CPU and KNR. Semantically consistent means that we need to use different syntax. Is that right ? I tried -qopt-report=1,2,3. The optimization message for the irrelevant functions may be disabled as they are not called in the C function. It seems that the optimization report is not saved in the current directory. LOOP BEGIN at atomic_read.c (16, 13) LOOP END ================================================================= Global loop optimization report for : ceil ================================================================= Global loop optimization report for : ceilf ================================================================= Global loop optimization report for : dp_sqrt_RD ================================================================= Global loop optimization report for : dp_sqrt_RN ================================================================= Global loop optimization report for : dp_sqrt_RU ================================================================= Global loop optimization report for : dp_sqrt_RZ ================================================================= Global loop optimization report for : exp2f ================================================================= .... .... filename: atomic_read.c #include <stdio.h> #include <stdatomic.h> #define N 256 int main(void) { int temp[N]; int x[N]; for (int i = 0; i < N; i++) x[i] = i; #pragma omp target map(from: temp[0:N]) map(to: x[0:N]) { #pragma omp parallel for for(int i = 0; i < N; i++) { #pragma omp atomic read temp[i] = x[i%N]; } } int error = 0; for (int i = 0; i < N; i++) if (temp[i] != i) { error = 1; break; } if (error) printf("Fail\n"); else printf("Success\n"); return 0; } ________________________________ From: Woodworth, Daniel <daniel.woodworth(a)intel.com> Sent: Monday, November 26, 2018 2:54:46 PM To: Jin, Zheming Cc: anl-intel-lrz-knr(a)lists.jlse.anl.gov Subject: RE: atomic issues 1. The compiler doesn’t support fences yet, but we are currently working on an overhaul of how memory operations are handled by the compiler and fence support is one of the things that will be added as part of that. There are also several cases where the lowering for some OpenMP constructs uses fences where they aren’t really needed and this is also something that we are working on fixing. Either way, there are fixes for this planned for the next release. 2. ‘#pragma omp atomic read’ should work in the latest compiler, though for types larger than 64 bits it falls back to runtime support that is not implemented on our target yet. Where are you seeing the warning? One other thing to keep in mind is that all of the OpenMP pragmas are ignored if -fiopenmp is not passed on the target side, so it might be worth double-checking that that compiler flag is present. 3. The tile is modeled as a single OpenMP thread, so the #pragma omp parallel will have a single thread allocated to it and the block will run once, giving a result of 2.0. This is slightly at odds with our use of num_threads to specify a worker count in #pragma omp parallel for; we’re eventually planning on retiring that and replacing it with new syntax that is more semantically consistent, but we don’t have that ready yet so num_threads is still being used for worker count there for the time being. From: anl-intel-lrz-knr [mailto:[email protected]] On Behalf Of Jin, Zheming via anl-intel-lrz-knr Sent: Monday, November 19, 2018 6:21 PM To: anl-intel-lrz-knr(a)lists.jlse.anl.gov Subject: [anl-intel-lrz-knr] atomic issues 1 The latest compiler can give an error message before producing the message: fatal error: error in backend: Cannot select: 0x60cca98: ch = AtomicFence 2 If '#pragma omp atomic read' is ignored by the latest compiler, a warning may be produced. 3 What is the expected value for the following program on the latest simulator ? double r = 0; double i = 1; #pragma omp target map(tofrom: r) map(to:i) { #pragma omp parallel num_threads(4) { double tmp = i+1; #pragma omp atomic r += tmp; } } printf("r = %lf\n", r); Thanks

1 0

Re: [anl-intel-lrz-knr] atomic issues
by Woodworth, Daniel 26 Nov '18

26 Nov '18

1. The compiler doesn’t support fences yet, but we are currently working on an overhaul of how memory operations are handled by the compiler and fence support is one of the things that will be added as part of that. There are also several cases where the lowering for some OpenMP constructs uses fences where they aren’t really needed and this is also something that we are working on fixing. Either way, there are fixes for this planned for the next release. 2. ‘#pragma omp atomic read’ should work in the latest compiler, though for types larger than 64 bits it falls back to runtime support that is not implemented on our target yet. Where are you seeing the warning? One other thing to keep in mind is that all of the OpenMP pragmas are ignored if -fiopenmp is not passed on the target side, so it might be worth double-checking that that compiler flag is present. 3. The tile is modeled as a single OpenMP thread, so the #pragma omp parallel will have a single thread allocated to it and the block will run once, giving a result of 2.0. This is slightly at odds with our use of num_threads to specify a worker count in #pragma omp parallel for; we’re eventually planning on retiring that and replacing it with new syntax that is more semantically consistent, but we don’t have that ready yet so num_threads is still being used for worker count there for the time being. From: anl-intel-lrz-knr [mailto:[email protected]] On Behalf Of Jin, Zheming via anl-intel-lrz-knr Sent: Monday, November 19, 2018 6:21 PM To: anl-intel-lrz-knr(a)lists.jlse.anl.gov Subject: [anl-intel-lrz-knr] atomic issues 1 The latest compiler can give an error message before producing the message: fatal error: error in backend: Cannot select: 0x60cca98: ch = AtomicFence 2 If '#pragma omp atomic read' is ignored by the latest compiler, a warning may be produced. 3 What is the expected value for the following program on the latest simulator ? double r = 0; double i = 1; #pragma omp target map(tofrom: r) map(to:i) { #pragma omp parallel num_threads(4) { double tmp = i+1; #pragma omp atomic r += tmp; } } printf("r = %lf\n", r); Thanks

1 0