November 2017 - Anl-intel-lrz-knr - mailman3.cels.anl.gov

[Anl-intel-csa] Canceled: ANL/Intel CSA Workload Characterization
by Petrini, Fabrizio 30 Nov '17

30 Nov '17

Placeholder for weekly ANL/Intel CSA Exascale workload characterization meeting ......................................................................................................................................... --> Join Skype Meeting<https://meet.intel.com/fabrizio.petrini/7K7FM32F> Trouble Joining? Try Skype Web App<https://meet.intel.com/fabrizio.petrini/7K7FM32F?sl=1> Join by phone +1(916)356-2663 (or your local bridge access #) Choose bridge 5.<tel:+1(916)356-2663%20(or%20your%20local%20bridge%20access%20#)%20Choose%20bridge%205.> (Global) English (United States) Find a local number<https://dial.intel.com> Conference ID: 791574018 Forgot your dial-in PIN?<https://dial.intel.com> |Help<https://o15.officeredir.microsoft.com/r/rlidLync15?clid=1033&p1=5&p2=2009> [!OC([1033])!] .........................................................................................................................................

1 0

[Anl-intel-csa] No call tomorrow
by Kumaran, Kalyan 30 Nov '17

30 Nov '17

I’m canceling the call tomorrow as we do not have enough topics to discuss. From what I understand Intel is working on PD 6.0. We will have a call on Monday morning with a smaller team to go through application projections for the SOW based on PD 6.0. Thanks, Kumar

1 0

Re: [Anl-intel-csa] Friday call agenda items?
by Christopher Knight 29 Nov '17

29 Nov '17

Some discussion of the memory model and, for example, how loads are modeled would be helpful. Ideas on if or how buffering can help to hide latencies due to memory accesses would also be helpful. Some context is below. Just a thought... chris ================== I’ve been looking at performance of a simple loop as proxy for modeling random memory access patterns in larger LAMMPS loops. void expected_loop(int N, double * array, int * neigh, double & result) { double sum = 0; for (int i = 0; i < N; ++i) sum += array[ neigh[i] ]; result = sum; } Based on visualizations from csadebug and assuming a lic depth of 32, I see something like the following for the loads. *) ld lic fills up 100% in ~32 cycles *) ~100 cycles no activity in ld PE (modeling cache miss) *) ld writes 32 FIFO values in ~32 cycles *) repeat Seems like all addresses in the ld lic are being fetched from memory in pipelined fashion (?) such that values can be written ~100 cycles after they enter the lic. In this case, I see periodic bubbles of ~100 cycles flowing through the graph hurting efficiency. I see something similar in the LAMMPS kernels. Is it the case that once an address arrives in the ld lic that a request to memory is immediately made irrespective of position in FIFO? For the loop above, I’m able to essentially hide all that memory latency by coupling four load PEs together (quad-load-engine?) to continuously write values ~1 per cycle in round robin fashion to the next PE(s) in graph. And after some unrolling, I see performance approaching that of CSASIM_MODEL_MEMORY=0 (everything ~12 cycles?) and the kernel is now memory bandwidth limited. So, now I’m trying to decide if I should update my LAMMPS graphs with these quad-load-engines to improve performance, likely turning them into memory bandwidth limited kernels, or if some amount of buffering can be expected to hide the ~100 cycle bubbles from loads. A better understanding of what is happening with load PEs in the simulator (and in hardware) would inform this decision. On Nov 29, 2017, at 12:50 PM, Kumaran, Kalyan <kumaran(a)anl.gov> wrote: Are there any topics that you would like to discuss on our Friday call this week? Looking for suggestions. Thanks, Kumar _______________________________________________ anl-intel-csa mailing list anl-intel-csa(a)lists.jlse.anl.gov https://lists.jlse.anl.gov/mailman/listinfo/anl-intel-csa

1 0

[Anl-intel-csa] Friday call agenda items?
by Kumaran, Kalyan 29 Nov '17

29 Nov '17

Are there any topics that you would like to discuss on our Friday call this week? Looking for suggestions. Thanks, Kumar

1 0

[Anl-intel-csa] CSA SDK Release Notes for 2017/11/21 version
by Glossop, Kent 21 Nov '17

21 Nov '17

CSA SDK Release Notes for 2017/11/21 version The default release has been changed to 2017-11-21-release. Relevant changes since the 2017-10-29-release: Compiler / Runtime * The compiler now generates streaming memory accesses for loads and stores in loops. This shouldn't break any code, but it can be disabled with -mllvm -csa-disable-streammem if desired. * Support of SPMDization: another way to create workers o The introduction of a new loop intrinsic, namely __builtin_csa_spmdization(k) that triggers, when the optimization level specified in the compiler's command line is not 0, the SPMDization loop transformation. SPMDization creates k workers from an independent loop to be executed on different units of the CSA tile. There is no need to add the CSA parallel loop intrinsic if the SPMDization intrinsic is used. o SPMDization is expected to provide better performance than using #pragma unroll <k> along with __builtin_csa_parallel_loop(), especially when there is reductions in the loop. Unlike unrolling, there is support for reductions between different workers in SPMDization. If SPMDization is slower, it means the current memory ordering pass introduced unnecessary ordering chains that serialize the workers generated by SPMDization. While this will be fixed by an on-going development of a new memory ordering pass, switching back to the previous memory ordering pass by adding "-mllvm -csa-order-memops-type=wavefront" to the compiler's command-line arguments can help in some cases. o If the loop contains a floating reduction, a floating point arithmetic slight difference should be expected with SPMDization due to the support of parallel reductions. Also, SPMDization relies on some LLVM static analyses such as reduction analysis and induction variables detection that might fail in some complex forms of loops. If this happens, a warning will be generated and SPMDization will return the original loop. Consider falling back to the use of unrolling along with the parallel loop intrinsic in this case. o SPMDization is also available in Fortran by adding "call builtin_csa_spmdization(k)" immediately before the desired loop. o Warning: There is a known bug where the workers do not have appropriate zero-trip testing. As a result, problems may occur for cases where the total trip count is small - in the range 1..n-1 for n workers. We expect this issue to be fixed in the next update, but encourage you to experiment with this feature where this is not a blocking issue. * A preview implementation of inner loop pipelining has been added. Inner Loop PipeLining (or "ILPL") is disabled by default. If enabled with "-mllvm -csa-ilpl", the compiler will generate serial inner loops such that their implementation can admit and pipeline multiple instances due to overlapping iterations of a parallel outer loop. This can significantly improve the utilization of the inner loop's units. No modification to the program is necessary. Caveats include: o No analysis is done to determine minimum LIC depths or maximum degree of pipelining allowed. However, the simulator currently assumes that LICs have a default depth of 128, which is sufficient for good performance in many cases. The default maximum degree of pipelining is 32; this can be changed with "-mllvm -csa-ilpl-tokens=N". o Only relatively simple loop nests such as those in $CSA_TOP/samples/fibonnaci, $CSA_TOP/samples/gemv, and $CSASIM_TOP/example/060-binary-search have been tested so far. * Fixed a bug where calls inside of loops were causing hangs. * Moved the loop storage check to run after optimizations, which should eliminate most false positives. * An IR-level code transformation has been added to generate a sequence of operations that can be lowered to a reduction operation by the CSA DF optimization pass. This pass is useful when the candidate for reduction operation is inside an if-then conditional flow (without an explicit 'else'). This pass can be disabled with -mllvm -csa-disable-IR-reduction-opt if desired. Simulator * Experimental CSA debugger "deadlock" command to help identify operations that may have contributed to a deadlock. * Fixed bug loading memory model multiple times - bug resulted in running with memory model for only the first invocation. * Assembler has been enhanced to support 0b notation for binary encoding * New operations repeatc1 and fountain1 (descriptions at end) Advisor/SAPPC * Combined profiling and analysis: SAPPC can now take application binary as input and do profile/analysis with a single command. The two step process of first profile and then SAPPC analysis has been combined into one. Run it with -h for relevant help on how to pass application binary and application options. * No need to copy and paste SAPPC files to a specific location for html report generation. --html_report works from wherever you run your analysis from. * Reduction loop detection support from Spatial Advisor is integrated with SAPPC. * Loop id mismatch between SAPPC reported output and SA GUI reported data is fixed. * Minor bug fixes related to correct total cpu for multi-threaded benchmarks and command line parameters change for flops with stacks collection. ---- New instruction descriptions Operation: repeatc1 res.CRd.i1, count.CRLu.u64, v.CRLu.i1 Semantics: tmp = count; while (tmp--) { och=*v; }; v where * is non-destructive read of v Description: Given a count, generate that number of copies of v to res. While this operation is conceptually similar to repeat, it is more expensive. (only fountain1 is new) Operation: fountain{1,8,16,32,64} res.CRd.iN, addr.CRLu.a64, len.CRLu.u64 Description: Given the sequence of values stored at the specified addr, of length len elements, generate the values out successively to the output channel. The values are stored densely for the type - i.e. for 1b, the values are in adjacent bits. addr is intended to be a scratchpad address. For addressable types, this operation is equivalent to: sldN chan, addr, len, 1, %ign, %ign As with any operation, if the operands are all literals, the operation will re-trigger continuously. Note that this may be implemented in different ways from a microarchitecture perspective Examples: // sequence of 64b values .section .csa.sp.1 pat1: .quad 10,20,30,40 .text fountain64 res, pat1, 4 // sequence of 1b values .section .csa.sp.2 pat2: .quad 0b001111111100 .text fountain1 res2, pat2, 12

1 0

[Anl-intel-csa] CSA SDK Release Notes for 2017/11/21 version
by Glossop, Kent 21 Nov '17

21 Nov '17

CSA SDK Release Notes for 2017/11/21 version The default release has been changed to 2017-11-21-release. Relevant changes since the 2017-10-29-release: Compiler / Runtime * The compiler now generates streaming memory accesses for loads and stores in loops. This shouldn't break any code, but it can be disabled with -mllvm -csa-disable-streammem if desired. * Support of SPMDization: another way to create workers o The introduction of a new loop intrinsic, namely __builtin_csa_spmdization(k) that triggers, when the optimization level specified in the compiler's command line is not 0, the SPMDization loop transformation. SPMDization creates k workers from an independent loop to be executed on different units of the CSA tile. There is no need to add the CSA parallel loop intrinsic if the SPMDization intrinsic is used. o SPMDization is expected to provide better performance than using #pragma unroll <k> along with __builtin_csa_parallel_loop(), especially when there is reductions in the loop. Unlike unrolling, there is support for reductions between different workers in SPMDization. If SPMDization is slower, it means the current memory ordering pass introduced unnecessary ordering chains that serialize the workers generated by SPMDization. While this will be fixed by an on-going development of a new memory ordering pass, switching back to the previous memory ordering pass by adding "-mllvm -csa-order-memops-type=wavefront" to the compiler's command-line arguments can help in some cases. o If the loop contains a floating reduction, a floating point arithmetic slight difference should be expected with SPMDization due to the support of parallel reductions. Also, SPMDization relies on some LLVM static analyses such as reduction analysis and induction variables detection that might fail in some complex forms of loops. If this happens, a warning will be generated and SPMDization will return the original loop. Consider falling back to the use of unrolling along with the parallel loop intrinsic in this case. o SPMDization is also available in Fortran by adding "call builtin_csa_spmdization(k)" immediately before the desired loop. o Warning: There is a known bug where the workers do not have appropriate zero-trip testing. As a result, problems may occur for cases where the total trip count is small - in the range 1..n-1 for n workers. We expect this issue to be fixed in the next update, but encourage you to experiment with this feature where this is not a blocking issue. * A preview implementation of inner loop pipelining has been added. Inner Loop PipeLining (or "ILPL") is disabled by default. If enabled with "-mllvm -csa-ilpl", the compiler will generate serial inner loops such that their implementation can admit and pipeline multiple instances due to overlapping iterations of a parallel outer loop. This can significantly improve the utilization of the inner loop's units. No modification to the program is necessary. Caveats include: o No analysis is done to determine minimum LIC depths or maximum degree of pipelining allowed. However, the simulator currently assumes that LICs have a default depth of 128, which is sufficient for good performance in many cases. The default maximum degree of pipelining is 32; this can be changed with "-mllvm -csa-ilpl-tokens=N". o Only relatively simple loop nests such as those in $CSA_TOP/samples/fibonnaci, $CSA_TOP/samples/gemv, and $CSASIM_TOP/example/060-binary-search have been tested so far. * Fixed a bug where calls inside of loops were causing hangs. * Moved the loop storage check to run after optimizations, which should eliminate most false positives. * An IR-level code transformation has been added to generate a sequence of operations that can be lowered to a reduction operation by the CSA DF optimization pass. This pass is useful when the candidate for reduction operation is inside an if-then conditional flow (without an explicit 'else'). This pass can be disabled with -mllvm -csa-disable-IR-reduction-opt if desired. Simulator * Experimental CSA debugger "deadlock" command to help identify operations that may have contributed to a deadlock. * Fixed bug loading memory model multiple times - bug resulted in running with memory model for only the first invocation. * Assembler has been enhanced to support 0b notation for binary encoding * New operations repeatc1 and fountain1 (descriptions at end) Advisor/SAPPC * Combined profiling and analysis: SAPPC can now take application binary as input and do profile/analysis with a single command. The two step process of first profile and then SAPPC analysis has been combined into one. Run it with -h for relevant help on how to pass application binary and application options. * No need to copy and paste SAPPC files to a specific location for html report generation. --html_report works from wherever you run your analysis from. * Reduction loop detection support from Spatial Advisor is integrated with SAPPC. * Loop id mismatch between SAPPC reported output and SA GUI reported data is fixed. * Minor bug fixes related to correct total cpu for multi-threaded benchmarks and command line parameters change for flops with stacks collection. ---- New instruction descriptions Operation: repeatc1 res.CRd.i1, count.CRLu.u64, v.CRLu.i1 Semantics: tmp = count; while (tmp--) { och=*v; }; v where * is non-destructive read of v Description: Given a count, generate that number of copies of v to res. While this operation is conceptually similar to repeat, it is more expensive. (only fountain1 is new) Operation: fountain{1,8,16,32,64} res.CRd.iN, addr.CRLu.a64, len.CRLu.u64 Description: Given the sequence of values stored at the specified addr, of length len elements, generate the values out successively to the output channel. The values are stored densely for the type - i.e. for 1b, the values are in adjacent bits. addr is intended to be a scratchpad address. For addressable types, this operation is equivalent to: sldN chan, addr, len, 1, %ign, %ign As with any operation, if the operands are all literals, the operation will re-trigger continuously. Note that this may be implemented in different ways from a microarchitecture perspective Examples: // sequence of 64b values .section .csa.sp.1 pat1: .quad 10,20,30,40 .text fountain64 res, pat1, 4 // sequence of 1b values .section .csa.sp.2 pat2: .quad 0b001111111100 .text fountain1 res2, pat2, 12

1 0

[Anl-intel-csa] Canceled: ANL/Intel CSA Workload Characterization
by Petrini, Fabrizio 21 Nov '17

21 Nov '17

Thanksgiving break Placeholder for weekly ANL/Intel CSA Exascale workload characterization meeting ......................................................................................................................................... --> Join Skype Meeting<https://meet.intel.com/fabrizio.petrini/7K7FM32F> Trouble Joining? Try Skype Web App<https://meet.intel.com/fabrizio.petrini/7K7FM32F?sl=1> Join by phone +1(916)356-2663 (or your local bridge access #) Choose bridge 5.<tel:+1(916)356-2663%20(or%20your%20local%20bridge%20access%20#)%20Choose%20bridge%205.> (Global) English (United States) Find a local number<https://dial.intel.com> Conference ID: 791574018 Forgot your dial-in PIN?<https://dial.intel.com> |Help<https://o15.officeredir.microsoft.com/r/rlidLync15?clid=1033&p1=5&p2=2009> [!OC([1033])!] .........................................................................................................................................

1 0

[Anl-intel-csa] Copy unit sometime behave like a two-cycles-latency unit?
by Applencourt, Thomas 17 Nov '17

17 Nov '17

Hi, We find a strange behavior in the copy unit. On some rare occasions, copy unit behaves like a two cycles latency units. We have two equivalents codes. In those two codes, we copy an 32 bits integer from channel a to channel b. The first one (copy.s) us a copy unit (copy b, %ign, %ign, %ign, a) and the second one (add.s) use an add32 unit (add32 b, a, 1). We expected that the total execution time of the two codes to be equal. However, it’s not the case. The add version takes one cycle less than the copy version: Diff {copy,add}.s: 21c21 < copy32 k11, %ign, %ign, %ign, k02 --- > add32 k11, k02, 1 copy.s: cycles 10007 add.s: cycles 10006 We haven’t found an explanation to this behavior, and we supposed that a bug in the functional simulator caused it. But maybe are we missing a subtlety in the logarithm of the FSIM. Do you have a different explanation? If you want to reproduce the result, I attached a zip file with all the required files (source the csa_env variables and launch ./run.sh). Have a nice weekend, Regards, Thomas PS: This is the full copy.s file if you don’t want to open the zip file. .text .globl loop .entry loop, hybriddataflow .result .lic .nil out_ctl .param .lic .nil in_ctl .param .lic .i32 n_gathed .lic .i1 pred .lic .i32 n .lic .i32 k .lic .i32 k01 .lic .i32 k02 .lic .i32 k11 .lic .i32 out pick32 n, in_ctl, n_gathed, %ign seqlts32 k, pred, %ign, %ign, 0, n, 1 copy32 k01, k02, %ign, %ign, k copy32 k11, %ign, %ign, %ign, k02 add32 out, k01, k11 onend out_ctl, pred, out PPS: Note that you need to disable the fusion tool in order to avoid optimization that hide the apparition of the two 2 cycles copy behavior. Thomas Applencourt Leadership Computing Facility Argonne National Laboratory Building 240 Office - 1.D.18 9700 S Cass Av., Argonne Il, 60439 (630) 252-0036

1 0

[Anl-intel-csa] Possible sub-optimal performance with the current Functional Simulator (2017-10-29)?
by Applencourt, Thomas 17 Nov '17

17 Nov '17

Hi, Colleen and I discover that the current simulator (‘2017-10-29-release’) can achieve suboptimal performance in rare case. We narrow down the problem when repeat units are used. We propose a fix. All the code used are in the zip file attached. Please find the minimal non-working example below (and in Unbalenced/accu.s). Is a simple accumulation of a division: a = 0 for k in 0,n: a += k/n The asm code is: .text .globl loop2 .entry loop2, hybriddataflow .set implicit .result .lic .nil out_ctl .param .lic .nil in_ctl .param .lic .i32 n_gathed .lic .i1 pred .lic .i1 pred2 .lic .i1 pred1 .lic .i32 n .lic .i32 n_r .lic .i32 k .lic .i32 out pick32 n, in_ctl, n_gathed, %ign seqlts32 k, pred, %ign, %ign, 0, n, 1 copy1 pred1, pred2, %ign, %ign, pred repeat32 n_r, pred1, n divs32 out, k, n_r redadd32 out_ctl, 0, out, pred2 One can notice that this kernel should take, 37 + (N-1) cycles to execute. For N = 10000, this should lead to 10036 cycles. However, if we run this kernel with the current FSIM, it runs for 10348 cycles. Stat file show (grep -r 'Full' Unbalenced/accu-csa0.stat), that two channels (k, pred2) become full during the simulation. This lead to units not being able to produce values, and hence degradation performances. This can be cured with a correct insertion of buffers. As shown in my presentation, we develop a script in order to determined where and how many buffers should be inserted. The script is located here: /soft/restricted/intel_csa/csatools/csa_equilibration. We can use it to balance this graph: $ /soft/restricted/intel_csa/csatools/csa_equilibration/v0.25/csa_equilibration.py Unbalenced/accu.s >> [('pred2', 33), ('k', 4)] Using those depths for those lics (Balenced/accu.s) lead to the expected number of cycles (10036 cycles). Note: That with the current FSIM, I was not able to change the size of the LIC ( @ and .set licdepth seem to be ignored, see the UnbalencedExtrem folder ), please use an older version. bash # Open a new shell export CSA_VERSION=2017-09-06-release source /soft/restricted/intel_csa/csasetup.bash ./test Balensed/accu.s loop2 Regards, Thomas and Colleen Thomas Applencourt Leadership Computing Facility Argonne National Laboratory Building 240 Office - 1.D.18 9700 S Cass Av., Argonne Il, 60439 (630) 252-0036

1 0

[Anl-intel-csa] csadebug: dump performance trace?
by Christopher Knight 16 Nov '17

16 Nov '17

Hello, Is there a way to dump a representation of the performance trace from csadebug? I’m thinking something similar to the simTrace.csv file that lists values corresponding to the red and black colors rendered for each sample window for variables over span of trace. chris

1 0