September 2019 - Ecp-aurora-sdk-discuss

Re: [Ecp-aurora-sdk-discuss] New A21 SDK available in JLSE
by Dean Howarth 30 Sep '19

30 Sep '19

Carlos, The new module is in the `intel_dga` folder: ~/dpct_examples$ module show dpct ------------------------------------------------------------------- /soft/restricted/CNDA/modulefiles/dpct/2019.9.30.013: setenv A21_SDK_ROOT /soft/restricted/intel_dga/sdk/2019.9.30.013 prepend-path PATH /soft/restricted/intel_dga/sdk/2019.9.30.013/dpcpp-ct/latest/bin prepend-path CPATH /soft/restricted/intel_dga/sdk/2019.9.30.013/dpcpp-ct/latest/include prepend-path C_INCLUDE_PATH /soft/restricted/intel_dga/sdk/2019.9.30.013/dpcpp-ct/latest/include prepend-path CPLUS_INCLUDE_PATH /soft/restricted/intel_dga/sdk/2019.9.30.013/dpcpp-ct/latest/include prepend-path LD_LIBRARY_PATH /soft/compilers/gcc/7.4.0/linux-rhel7-x86_64/lib64 setenv DPCTROOT /soft/restricted/intel_dga/sdk/2019.9.30.013/dpcpp-ct/latest ------------------------------------------------------------------- so I do not have access to it. Can you make the module work for paths under /soft/restricted/CNDA Dean On Mon, Sep 30, 2019 at 5:15 PM Rosales-fernandez, Carlos via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> wrote: > Hi all, > > > > We are pleased to announce that a new A21 SDK has been deployed in JLSE. > This version 2019.9.30.013 includes multiple improvements, including USM > support. For details please see the release notes in the wiki: > > > > > https://wiki.jlse.anl.gov/download/attachments/26214592/SDK-Release-Notes-2… > > > > Modules will enable the newer SDK by default, but the older versions > remain available for regression testing. As usual, please let us know of > any issues you encounter and any improvements we can include. > > > > Regards, > > Carlos > > > -- > Ecp-aurora-sdk-discuss mailing list > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss >

1 0

New A21 SDK available in JLSE
by Rosales-fernandez, Carlos 30 Sep '19

30 Sep '19

Hi all, We are pleased to announce that a new A21 SDK has been deployed in JLSE. This version 2019.9.30.013 includes multiple improvements, including USM support. For details please see the release notes in the wiki: https://wiki.jlse.anl.gov/download/attachments/26214592/SDK-Release-Notes-2… Modules will enable the newer SDK by default, but the older versions remain available for regression testing. As usual, please let us know of any issues you encounter and any improvements we can include. Regards, Carlos

1 0

A212 SDk Update coming later today
by Rosales-fernandez, Carlos 30 Sep '19

30 Sep '19

Hi all, a new A21 SDK version will be released today after 4pm CST. This is just a heads-up so you do not get caught unawares byt the default module changes. I will follow-up with a note at 4pm to confirm changes have been made. Carlos

1 0

Re: [Ecp-aurora-sdk-discuss] OMP target offload and C++ containers
by Peterka, Thomas 20 Sep '19

20 Sep '19

I’m using the clang++ compiler. My cmake configuration is this: -- The C compiler identification is Clang 9.0.0 -- The CXX compiler identification is Clang 9.0.0 -- Check for working C compiler: /home/bhomerding/install/sycl-usmapi/bin/clang -- Check for working C compiler: /home/bhomerding/install/sycl-usmapi/bin/clang -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working CXX compiler: /home/bhomerding/install/sycl-usmapi/bin/clang++ -- Check for working CXX compiler: /home/bhomerding/install/sycl-usmapi/bin/clang++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done I tried adding the —gcc-toolchain=… to the compile line: [100%] Building CXX object examples/fixed/CMakeFiles/fixed.dir/fixed.cpp.o cd /home/tpeterka/software/mfa/build/examples/fixed && /home/bhomerding/install/sycl-usmapi/bin/clang++ -DMFA_NO_TBB -DMFA_NO_WEIGHTS -DMFA_SYCL -std=c++11 -fsycl --gcc-toolchain=/soft/compilers/gcc/8.2.0/linux-rhel7-x86_64 -O3 -DNDEBUG -I/home/tpeterka/software/mfa/include -I/home/tpeterka/software/diy/include -I/home/tpeterka/software/eigen3 -isystem/home/tpeterka/software/mpich-3.3.1/build/include -I/home/tpeterka/software/mfa/examples -o CMakeFiles/fixed.dir/fixed.cpp.o -c /home/tpeterka/software/mfa/examples/fixed/fixed.cpp But I still get the same error messages, and they point to the same gcc7.4 directory (no change). Anyway, I can wait for the SDK. This is not urgent. Tom ----------------- Tom Peterka Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave. Lemont, IL 60439 Bldg 240, Rm 2148 e-mail: tpeterka(a)mcs.anl.gov<mailto:[email protected]> web: www.mcs.anl.gov/~tpeterka phone: 630-252-7198 On Sep 20, 2019, at 3:59 PM, Harms, Kevin <harms(a)alcf.anl.gov<mailto:[email protected]>> wrote: hm... what does your Makefile look like? I think Brian built against gcc8 and not gcc7.4. You need to add this to the CXXFLAGS: --gcc-toolchain=/soft/compilers/gcc/8.2.0/linux-rhel7-x86_64 kevin ________________________________________ From: Peterka, Thomas <tpeterka(a)mcs.anl.gov<mailto:[email protected]>> Sent: Friday, September 20, 2019 3:51 PM To: Harms, Kevin Cc: Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]> Subject: Re: [Ecp-aurora-sdk-discuss] OMP target offload and C++ containers Kevin, I did see that in your slides and tried it, but I wasn’t able to recompile my code with the new module. I’m seeing errors like this: [100%] Building CXX object examples/fixed/CMakeFiles/fixed.dir/fixed.cpp.o In file included from /home/tpeterka/software/mfa/examples/fixed/fixed.cpp:10: In file included from /home/tpeterka/software/mfa/include/mfa/mfa.hpp:31: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl.hpp:11: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl/accessor.hpp:13: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl/buffer.hpp:10: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl/detail/buffer_impl.hpp:19: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl/detail/scheduler/scheduler.hpp:11: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl/detail/cg.hpp:19: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl/nd_item.hpp:16: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl/intel/sub_group.hpp:16: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl/types.hpp:55: In file included from /soft/compilers/gcc/7.4.0/linux-rhel7-x86_64/include/c++/7.4.0/cfenv:41: /soft/compilers/gcc/7.4.0/linux-rhel7-x86_64/include/c++/7.4.0/fenv.h:58:11: error: no member named 'fenv_t' in the global namespace; did you mean 'dev_t'? using ::fenv_t; ~~^ Maybe is something is incompatible with that module and my build setup, which worked before with the first three modules below. 1) dpcpp/2019.6.30.011 2) parallel-studio-2019 3) iprof 4) sycl-usmapi Also, your example was for a malloc’ing a C-style array, but I didn’t see how to use that for a std::vector. Anyway, I appreciate the suggestions. I can wait until the next SDK drop. How will I know when that happens? Thanks again, Tom ----------------- Tom Peterka Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave. Lemont, IL 60439 Bldg 240, Rm 2148 e-mail: tpeterka(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]> web: www.mcs.anl.gov/~tpeterka<http://www.mcs.anl.gov/~tpeterka> phone: 630-252-7198 On Sep 20, 2019, at 3:13 PM, Harms, Kevin <harms(a)alcf.anl.gov<mailto:[email protected]><mailto:[email protected]>> wrote: Tom, we expect the next SDK drop to have USM, but until then, we have a temporary install that can be used like this: module use /home/bhomerding/install/modulefiles module load sycl-usmapi Only the CPU device is supported currently. You still need to run on the iris nodes to test. There is a simple example here: https://github.com/kevin-harms/sycltrain/tree/master/usm kevin ________________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]><mailto:[email protected]>> on behalf of Peterka, Thomas via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]><mailto:[email protected]>> Sent: Friday, September 20, 2019 2:53 PM To: Hammond, Jeff R; Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]><mailto:[email protected]>; Romano, Paul K. Subject: Re: [Ecp-aurora-sdk-discuss] OMP target offload and C++ containers Jeff, Paul, others, The link Jeff pointed to: https://github.com/intel/llvm/blob/sycl/sycl/test/usm/allocator_vector.cpp would be be a good workaround, except that the clang++ compiler reports that there is no member named ‘usm' in namespace 'cl::sycl'. Is this because USM is not implemented yet in the SDK, as was said in the workshop, or did I miss something? I’ll reiterate what Paul said about the need to have STL in the kernel. I wouldn’t be opposed to constructing STL objects outside the kernel on the host, but I do need to use them in the kernel, perhaps restricted to not reallocating their memory (eg,, not push_back or resize). I’m looking at SYCL, not OpenMP. I thought this was the idea of SYCL—that I could write C++ kernels. C++ w/o STL would not work for me. Any suggestions appreciated. This seems to be quite important to deciding what programming model to use for C++ code. Thanks, Tom ----------------- Tom Peterka Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave. Lemont, IL 60439 Bldg 240, Rm 2148 e-mail: tpeterka(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]><mailto:[email protected]> web: www.mcs.anl.gov/~tpeterka<http://www.mcs.anl.gov/~tpeterka><http://www.mcs.anl.gov/~tpeterka> phone: 630-252-7198 On Sep 19, 2019, at 11:57 AM, Hammond, Jeff R via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]><mailto:[email protected]><mailto:[email protected]>> wrote: In general, allocating memory on devices is a bad idea. It can be implemented but only by pre-allocating a pool prior to kernel launch. James Brodman and I were discussing polymorphic memory resources and USM yesterday and he pointed me tohttps://github.com/intel/llvm/blob/sycl/sycl/test/usm/allocator_vector.cpp. That isn’t everything you want by any means (and it is SYCL rather than OpenMP) but maybe it’s a start. Intel is actively working on supporting a subset of the C++ STL in device code, but the focus is SYCL applications. I don’t know what docs we’ve shared so far but if you are at the workshop, find one of the Intel folks and ask them to go through https://software.intel.com/en-us/oneapi-dpcpp-library-guide with you. This website should be public in the near future, but perhaps there is an offline HTML version in the SDK already. If not, hopefully Carlos et al. can help. Jeff From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]><mailto:[email protected]><mailto:[email protected]>> on behalf of "Romano, Paul K. via Ecp-aurora-sdk-discuss" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]><mailto:[email protected]><mailto:[email protected]>> Reply-To: "Romano, Paul K." <promano(a)anl.gov<mailto:[email protected]><mailto:[email protected]><mailto:[email protected]>> Date: Thursday, September 19, 2019 at 9:48 AM To: "Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]><mailto:[email protected]><mailto:[email protected]>" <Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]><mailto:[email protected]><mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] OMP target offload and C++ containers I work on an application that makes heavy use of C++ containers (vectors, unordered_map, etc.) in areas/data structures of the code (good example here<https://github.com/openmc-dev/openmc/blob/develop/include/openmc/nuclide.h#…>) that will need to be offloaded and have a few general questions: - Is it / will it be possible to use these containers on Gen9 / future devices using OpenMP target offload? - If so, are methods that potentially allocate new memory (vector::push_back) permissible? If permissible, should I still avoid them? - Will unified shared memory eventually be able to help? Any suggestions for porting strategies for codes with complex data structures like this would be appreciated. Thanks, Paul -- Ecp-aurora-sdk-discuss mailing list Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]><mailto:[email protected]><mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

Re: [Ecp-aurora-sdk-discuss] OMP target offload and C++ containers
by Hammond, Jeff R 20 Sep '19

20 Sep '19

>From colleague: Could somebody please try the suggested instructions: yum install yum-plugin-copr yum copr enable jdanecki/intel-opencl yum install intel-opencl This is CentOS recipe. Jeff Sent from my iPhone On Sep 20, 2019, at 1:02 PM, Hammond, Jeff R via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> wrote: You probably need SYCL compiler from GitHub to build that example. We are working CentOS OpenCL issue related to USM in other threads. Sent from my iPhone On Sep 20, 2019, at 12:54 PM, Peterka, Thomas <tpeterka(a)mcs.anl.gov<mailto:[email protected]>> wrote: Jeff, Paul, others, The link Jeff pointed to: https://github.com/intel/llvm/blob/sycl/sycl/test/usm/allocator_vector.cpp would be be a good workaround, except that the clang++ compiler reports that there is no member named ‘usm' in namespace 'cl::sycl'. Is this because USM is not implemented yet in the SDK, as was said in the workshop, or did I miss something? I’ll reiterate what Paul said about the need to have STL in the kernel. I wouldn’t be opposed to constructing STL objects outside the kernel on the host, but I do need to use them in the kernel, perhaps restricted to not reallocating their memory (eg,, not push_back or resize). I’m looking at SYCL, not OpenMP. I thought this was the idea of SYCL—that I could write C++ kernels. C++ w/o STL would not work for me. Any suggestions appreciated. This seems to be quite important to deciding what programming model to use for C++ code. Thanks, Tom ----------------- Tom Peterka Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave. Lemont, IL 60439 Bldg 240, Rm 2148 e-mail: tpeterka(a)mcs.anl.gov<mailto:[email protected]> web: www.mcs.anl.gov/~tpeterka<http://www.mcs.anl.gov/~tpeterka> phone: 630-252-7198 On Sep 19, 2019, at 11:57 AM, Hammond, Jeff R via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> wrote: In general, allocating memory on devices is a bad idea. It can be implemented but only by pre-allocating a pool prior to kernel launch. James Brodman and I were discussing polymorphic memory resources and USM yesterday and he pointed me tohttps://github.com/intel/llvm/blob/sycl/sycl/test/usm/allocator_vector.cpp. That isn’t everything you want by any means (and it is SYCL rather than OpenMP) but maybe it’s a start. Intel is actively working on supporting a subset of the C++ STL in device code, but the focus is SYCL applications. I don’t know what docs we’ve shared so far but if you are at the workshop, find one of the Intel folks and ask them to go through https://software.intel.com/en-us/oneapi-dpcpp-library-guide with you. This website should be public in the near future, but perhaps there is an offline HTML version in the SDK already. If not, hopefully Carlos et al. can help. Jeff From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of "Romano, Paul K. via Ecp-aurora-sdk-discuss" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Reply-To: "Romano, Paul K." <promano(a)anl.gov<mailto:[email protected]>> Date: Thursday, September 19, 2019 at 9:48 AM To: "Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>" <Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] OMP target offload and C++ containers I work on an application that makes heavy use of C++ containers (vectors, unordered_map, etc.) in areas/data structures of the code (good example here<https://github.com/openmc-dev/openmc/blob/develop/include/openmc/nuclide.h#…>) that will need to be offloaded and have a few general questions: - Is it / will it be possible to use these containers on Gen9 / future devices using OpenMP target offload? - If so, are methods that potentially allocate new memory (vector::push_back) permissible? If permissible, should I still avoid them? - Will unified shared memory eventually be able to help? Any suggestions for porting strategies for codes with complex data structures like this would be appreciated. Thanks, Paul -- Ecp-aurora-sdk-discuss mailing list Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss -- Ecp-aurora-sdk-discuss mailing list Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

Re: [Ecp-aurora-sdk-discuss] OMP target offload and C++ containers
by Harms, Kevin 20 Sep '19

20 Sep '19

hm... what does your Makefile look like? I think Brian built against gcc8 and not gcc7.4. You need to add this to the CXXFLAGS: --gcc-toolchain=/soft/compilers/gcc/8.2.0/linux-rhel7-x86_64 kevin ________________________________________ From: Peterka, Thomas <tpeterka(a)mcs.anl.gov> Sent: Friday, September 20, 2019 3:51 PM To: Harms, Kevin Cc: Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov Subject: Re: [Ecp-aurora-sdk-discuss] OMP target offload and C++ containers Kevin, I did see that in your slides and tried it, but I wasn’t able to recompile my code with the new module. I’m seeing errors like this: [100%] Building CXX object examples/fixed/CMakeFiles/fixed.dir/fixed.cpp.o In file included from /home/tpeterka/software/mfa/examples/fixed/fixed.cpp:10: In file included from /home/tpeterka/software/mfa/include/mfa/mfa.hpp:31: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl.hpp:11: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl/accessor.hpp:13: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl/buffer.hpp:10: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl/detail/buffer_impl.hpp:19: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl/detail/scheduler/scheduler.hpp:11: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl/detail/cg.hpp:19: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl/nd_item.hpp:16: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl/intel/sub_group.hpp:16: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl/types.hpp:55: In file included from /soft/compilers/gcc/7.4.0/linux-rhel7-x86_64/include/c++/7.4.0/cfenv:41: /soft/compilers/gcc/7.4.0/linux-rhel7-x86_64/include/c++/7.4.0/fenv.h:58:11: error: no member named 'fenv_t' in the global namespace; did you mean 'dev_t'? using ::fenv_t; ~~^ Maybe is something is incompatible with that module and my build setup, which worked before with the first three modules below. 1) dpcpp/2019.6.30.011 2) parallel-studio-2019 3) iprof 4) sycl-usmapi Also, your example was for a malloc’ing a C-style array, but I didn’t see how to use that for a std::vector. Anyway, I appreciate the suggestions. I can wait until the next SDK drop. How will I know when that happens? Thanks again, Tom ----------------- Tom Peterka Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave. Lemont, IL 60439 Bldg 240, Rm 2148 e-mail: tpeterka(a)mcs.anl.gov<mailto:[email protected]> web: www.mcs.anl.gov/~tpeterka phone: 630-252-7198 On Sep 20, 2019, at 3:13 PM, Harms, Kevin <harms(a)alcf.anl.gov<mailto:[email protected]>> wrote: Tom, we expect the next SDK drop to have USM, but until then, we have a temporary install that can be used like this: module use /home/bhomerding/install/modulefiles module load sycl-usmapi Only the CPU device is supported currently. You still need to run on the iris nodes to test. There is a simple example here: https://github.com/kevin-harms/sycltrain/tree/master/usm kevin ________________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of Peterka, Thomas via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Sent: Friday, September 20, 2019 2:53 PM To: Hammond, Jeff R; Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>; Romano, Paul K. Subject: Re: [Ecp-aurora-sdk-discuss] OMP target offload and C++ containers Jeff, Paul, others, The link Jeff pointed to: https://github.com/intel/llvm/blob/sycl/sycl/test/usm/allocator_vector.cpp would be be a good workaround, except that the clang++ compiler reports that there is no member named ‘usm' in namespace 'cl::sycl'. Is this because USM is not implemented yet in the SDK, as was said in the workshop, or did I miss something? I’ll reiterate what Paul said about the need to have STL in the kernel. I wouldn’t be opposed to constructing STL objects outside the kernel on the host, but I do need to use them in the kernel, perhaps restricted to not reallocating their memory (eg,, not push_back or resize). I’m looking at SYCL, not OpenMP. I thought this was the idea of SYCL—that I could write C++ kernels. C++ w/o STL would not work for me. Any suggestions appreciated. This seems to be quite important to deciding what programming model to use for C++ code. Thanks, Tom ----------------- Tom Peterka Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave. Lemont, IL 60439 Bldg 240, Rm 2148 e-mail: tpeterka(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]> web: www.mcs.anl.gov/~tpeterka<http://www.mcs.anl.gov/~tpeterka> phone: 630-252-7198 On Sep 19, 2019, at 11:57 AM, Hammond, Jeff R via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]><mailto:[email protected]>> wrote: In general, allocating memory on devices is a bad idea. It can be implemented but only by pre-allocating a pool prior to kernel launch. James Brodman and I were discussing polymorphic memory resources and USM yesterday and he pointed me tohttps://github.com/intel/llvm/blob/sycl/sycl/test/usm/allocator_vector.cpp. That isn’t everything you want by any means (and it is SYCL rather than OpenMP) but maybe it’s a start. Intel is actively working on supporting a subset of the C++ STL in device code, but the focus is SYCL applications. I don’t know what docs we’ve shared so far but if you are at the workshop, find one of the Intel folks and ask them to go through https://software.intel.com/en-us/oneapi-dpcpp-library-guide with you. This website should be public in the near future, but perhaps there is an offline HTML version in the SDK already. If not, hopefully Carlos et al. can help. Jeff From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]><mailto:[email protected]>> on behalf of "Romano, Paul K. via Ecp-aurora-sdk-discuss" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]><mailto:[email protected]>> Reply-To: "Romano, Paul K." <promano(a)anl.gov<mailto:[email protected]><mailto:[email protected]>> Date: Thursday, September 19, 2019 at 9:48 AM To: "Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]><mailto:[email protected]>" <Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]><mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] OMP target offload and C++ containers I work on an application that makes heavy use of C++ containers (vectors, unordered_map, etc.) in areas/data structures of the code (good example here<https://github.com/openmc-dev/openmc/blob/develop/include/openmc/nuclide.h#…>) that will need to be offloaded and have a few general questions: - Is it / will it be possible to use these containers on Gen9 / future devices using OpenMP target offload? - If so, are methods that potentially allocate new memory (vector::push_back) permissible? If permissible, should I still avoid them? - Will unified shared memory eventually be able to help? Any suggestions for porting strategies for codes with complex data structures like this would be appreciated. Thanks, Paul -- Ecp-aurora-sdk-discuss mailing list Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]><mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

Re: [Ecp-aurora-sdk-discuss] OMP target offload and C++ containers
by Peterka, Thomas 20 Sep '19

20 Sep '19

Kevin, I did see that in your slides and tried it, but I wasn’t able to recompile my code with the new module. I’m seeing errors like this: [100%] Building CXX object examples/fixed/CMakeFiles/fixed.dir/fixed.cpp.o In file included from /home/tpeterka/software/mfa/examples/fixed/fixed.cpp:10: In file included from /home/tpeterka/software/mfa/include/mfa/mfa.hpp:31: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl.hpp:11: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl/accessor.hpp:13: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl/buffer.hpp:10: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl/detail/buffer_impl.hpp:19: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl/detail/scheduler/scheduler.hpp:11: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl/detail/cg.hpp:19: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl/nd_item.hpp:16: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl/intel/sub_group.hpp:16: In file included from /home/bhomerding/install/sycl-usmapi/include/CL/sycl/types.hpp:55: In file included from /soft/compilers/gcc/7.4.0/linux-rhel7-x86_64/include/c++/7.4.0/cfenv:41: /soft/compilers/gcc/7.4.0/linux-rhel7-x86_64/include/c++/7.4.0/fenv.h:58:11: error: no member named 'fenv_t' in the global namespace; did you mean 'dev_t'? using ::fenv_t; ~~^ Maybe is something is incompatible with that module and my build setup, which worked before with the first three modules below. 1) dpcpp/2019.6.30.011 2) parallel-studio-2019 3) iprof 4) sycl-usmapi Also, your example was for a malloc’ing a C-style array, but I didn’t see how to use that for a std::vector. Anyway, I appreciate the suggestions. I can wait until the next SDK drop. How will I know when that happens? Thanks again, Tom ----------------- Tom Peterka Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave. Lemont, IL 60439 Bldg 240, Rm 2148 e-mail: tpeterka(a)mcs.anl.gov<mailto:[email protected]> web: www.mcs.anl.gov/~tpeterka phone: 630-252-7198 On Sep 20, 2019, at 3:13 PM, Harms, Kevin <harms(a)alcf.anl.gov<mailto:[email protected]>> wrote: Tom, we expect the next SDK drop to have USM, but until then, we have a temporary install that can be used like this: module use /home/bhomerding/install/modulefiles module load sycl-usmapi Only the CPU device is supported currently. You still need to run on the iris nodes to test. There is a simple example here: https://github.com/kevin-harms/sycltrain/tree/master/usm kevin ________________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of Peterka, Thomas via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Sent: Friday, September 20, 2019 2:53 PM To: Hammond, Jeff R; Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>; Romano, Paul K. Subject: Re: [Ecp-aurora-sdk-discuss] OMP target offload and C++ containers Jeff, Paul, others, The link Jeff pointed to: https://github.com/intel/llvm/blob/sycl/sycl/test/usm/allocator_vector.cpp would be be a good workaround, except that the clang++ compiler reports that there is no member named ‘usm' in namespace 'cl::sycl'. Is this because USM is not implemented yet in the SDK, as was said in the workshop, or did I miss something? I’ll reiterate what Paul said about the need to have STL in the kernel. I wouldn’t be opposed to constructing STL objects outside the kernel on the host, but I do need to use them in the kernel, perhaps restricted to not reallocating their memory (eg,, not push_back or resize). I’m looking at SYCL, not OpenMP. I thought this was the idea of SYCL—that I could write C++ kernels. C++ w/o STL would not work for me. Any suggestions appreciated. This seems to be quite important to deciding what programming model to use for C++ code. Thanks, Tom ----------------- Tom Peterka Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave. Lemont, IL 60439 Bldg 240, Rm 2148 e-mail: tpeterka(a)mcs.anl.gov<mailto:[email protected]><mailto:[email protected]> web: www.mcs.anl.gov/~tpeterka<http://www.mcs.anl.gov/~tpeterka> phone: 630-252-7198 On Sep 19, 2019, at 11:57 AM, Hammond, Jeff R via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]><mailto:[email protected]>> wrote: In general, allocating memory on devices is a bad idea. It can be implemented but only by pre-allocating a pool prior to kernel launch. James Brodman and I were discussing polymorphic memory resources and USM yesterday and he pointed me tohttps://github.com/intel/llvm/blob/sycl/sycl/test/usm/allocator_vector.cpp. That isn’t everything you want by any means (and it is SYCL rather than OpenMP) but maybe it’s a start. Intel is actively working on supporting a subset of the C++ STL in device code, but the focus is SYCL applications. I don’t know what docs we’ve shared so far but if you are at the workshop, find one of the Intel folks and ask them to go through https://software.intel.com/en-us/oneapi-dpcpp-library-guide with you. This website should be public in the near future, but perhaps there is an offline HTML version in the SDK already. If not, hopefully Carlos et al. can help. Jeff From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]><mailto:[email protected]>> on behalf of "Romano, Paul K. via Ecp-aurora-sdk-discuss" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]><mailto:[email protected]>> Reply-To: "Romano, Paul K." <promano(a)anl.gov<mailto:[email protected]><mailto:[email protected]>> Date: Thursday, September 19, 2019 at 9:48 AM To: "Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]><mailto:[email protected]>" <Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]><mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] OMP target offload and C++ containers I work on an application that makes heavy use of C++ containers (vectors, unordered_map, etc.) in areas/data structures of the code (good example here<https://github.com/openmc-dev/openmc/blob/develop/include/openmc/nuclide.h#…>) that will need to be offloaded and have a few general questions: - Is it / will it be possible to use these containers on Gen9 / future devices using OpenMP target offload? - If so, are methods that potentially allocate new memory (vector::push_back) permissible? If permissible, should I still avoid them? - Will unified shared memory eventually be able to help? Any suggestions for porting strategies for codes with complex data structures like this would be appreciated. Thanks, Paul -- Ecp-aurora-sdk-discuss mailing list Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]><mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

Re: [Ecp-aurora-sdk-discuss] OMP target offload and C++ containers
by Harms, Kevin 20 Sep '19

20 Sep '19

Tom, we expect the next SDK drop to have USM, but until then, we have a temporary install that can be used like this: module use /home/bhomerding/install/modulefiles module load sycl-usmapi Only the CPU device is supported currently. You still need to run on the iris nodes to test. There is a simple example here: https://github.com/kevin-harms/sycltrain/tree/master/usm kevin ________________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peterka, Thomas via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Friday, September 20, 2019 2:53 PM To: Hammond, Jeff R; Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov; Romano, Paul K. Subject: Re: [Ecp-aurora-sdk-discuss] OMP target offload and C++ containers Jeff, Paul, others, The link Jeff pointed to: https://github.com/intel/llvm/blob/sycl/sycl/test/usm/allocator_vector.cpp would be be a good workaround, except that the clang++ compiler reports that there is no member named ‘usm' in namespace 'cl::sycl'. Is this because USM is not implemented yet in the SDK, as was said in the workshop, or did I miss something? I’ll reiterate what Paul said about the need to have STL in the kernel. I wouldn’t be opposed to constructing STL objects outside the kernel on the host, but I do need to use them in the kernel, perhaps restricted to not reallocating their memory (eg,, not push_back or resize). I’m looking at SYCL, not OpenMP. I thought this was the idea of SYCL—that I could write C++ kernels. C++ w/o STL would not work for me. Any suggestions appreciated. This seems to be quite important to deciding what programming model to use for C++ code. Thanks, Tom ----------------- Tom Peterka Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave. Lemont, IL 60439 Bldg 240, Rm 2148 e-mail: tpeterka(a)mcs.anl.gov<mailto:[email protected]> web: www.mcs.anl.gov/~tpeterka phone: 630-252-7198 On Sep 19, 2019, at 11:57 AM, Hammond, Jeff R via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> wrote: In general, allocating memory on devices is a bad idea. It can be implemented but only by pre-allocating a pool prior to kernel launch. James Brodman and I were discussing polymorphic memory resources and USM yesterday and he pointed me tohttps://github.com/intel/llvm/blob/sycl/sycl/test/usm/allocator_vector.cpp. That isn’t everything you want by any means (and it is SYCL rather than OpenMP) but maybe it’s a start. Intel is actively working on supporting a subset of the C++ STL in device code, but the focus is SYCL applications. I don’t know what docs we’ve shared so far but if you are at the workshop, find one of the Intel folks and ask them to go through https://software.intel.com/en-us/oneapi-dpcpp-library-guide with you. This website should be public in the near future, but perhaps there is an offline HTML version in the SDK already. If not, hopefully Carlos et al. can help. Jeff From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of "Romano, Paul K. via Ecp-aurora-sdk-discuss" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Reply-To: "Romano, Paul K." <promano(a)anl.gov<mailto:[email protected]>> Date: Thursday, September 19, 2019 at 9:48 AM To: "Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>" <Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] OMP target offload and C++ containers I work on an application that makes heavy use of C++ containers (vectors, unordered_map, etc.) in areas/data structures of the code (good example here<https://github.com/openmc-dev/openmc/blob/develop/include/openmc/nuclide.h#…>) that will need to be offloaded and have a few general questions: - Is it / will it be possible to use these containers on Gen9 / future devices using OpenMP target offload? - If so, are methods that potentially allocate new memory (vector::push_back) permissible? If permissible, should I still avoid them? - Will unified shared memory eventually be able to help? Any suggestions for porting strategies for codes with complex data structures like this would be appreciated. Thanks, Paul -- Ecp-aurora-sdk-discuss mailing list Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

Re: [Ecp-aurora-sdk-discuss] OMP target offload and C++ containers
by Applencourt, Thomas 20 Sep '19

20 Sep '19

> Why not consider using Kokkos? >From https://github.com/kokkos/kokkos/wiki/Interoperability > One of the most common data objects in C++ codes is std::vector. Its semantics are unfortunately not compatible with Kokkos requirements and it is thus not well supported. Thomas Applencourt Leadership Computing Facility Argonne National Laboratory Building 240 Office - 1.D.18 9700 S Cass Av., Lemont Il, 60439 (630) 252-0036 ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Balakrishnan, Ramesh via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Friday, September 20, 2019 14:56 To: Peterka, Thomas <tpeterka(a)mcs.anl.gov> Cc: Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] OMP target offload and C++ containers On Sep 20, 2019, at 2:53 PM, Peterka, Thomas via Ecp-aurora-sdk-discuss <Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> wrote: This seems to be quite important to deciding what programming model to use for C++ Why not consider using Kokkos? RB

1 0

Re: [Ecp-aurora-sdk-discuss] OMP target offload and C++ containers
by Hammond, Jeff R 20 Sep '19

20 Sep '19

You probably need SYCL compiler from GitHub to build that example. We are working CentOS OpenCL issue related to USM in other threads. Sent from my iPhone On Sep 20, 2019, at 12:54 PM, Peterka, Thomas <tpeterka(a)mcs.anl.gov<mailto:[email protected]>> wrote: Jeff, Paul, others, The link Jeff pointed to: https://github.com/intel/llvm/blob/sycl/sycl/test/usm/allocator_vector.cpp would be be a good workaround, except that the clang++ compiler reports that there is no member named ‘usm' in namespace 'cl::sycl'. Is this because USM is not implemented yet in the SDK, as was said in the workshop, or did I miss something? I’ll reiterate what Paul said about the need to have STL in the kernel. I wouldn’t be opposed to constructing STL objects outside the kernel on the host, but I do need to use them in the kernel, perhaps restricted to not reallocating their memory (eg,, not push_back or resize). I’m looking at SYCL, not OpenMP. I thought this was the idea of SYCL—that I could write C++ kernels. C++ w/o STL would not work for me. Any suggestions appreciated. This seems to be quite important to deciding what programming model to use for C++ code. Thanks, Tom ----------------- Tom Peterka Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Ave. Lemont, IL 60439 Bldg 240, Rm 2148 e-mail: tpeterka(a)mcs.anl.gov<mailto:[email protected]> web: www.mcs.anl.gov/~tpeterka<http://www.mcs.anl.gov/~tpeterka> phone: 630-252-7198 On Sep 19, 2019, at 11:57 AM, Hammond, Jeff R via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> wrote: In general, allocating memory on devices is a bad idea. It can be implemented but only by pre-allocating a pool prior to kernel launch. James Brodman and I were discussing polymorphic memory resources and USM yesterday and he pointed me tohttps://github.com/intel/llvm/blob/sycl/sycl/test/usm/allocator_vector.cpp. That isn’t everything you want by any means (and it is SYCL rather than OpenMP) but maybe it’s a start. Intel is actively working on supporting a subset of the C++ STL in device code, but the focus is SYCL applications. I don’t know what docs we’ve shared so far but if you are at the workshop, find one of the Intel folks and ask them to go through https://software.intel.com/en-us/oneapi-dpcpp-library-guide with you. This website should be public in the near future, but perhaps there is an offline HTML version in the SDK already. If not, hopefully Carlos et al. can help. Jeff From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of "Romano, Paul K. via Ecp-aurora-sdk-discuss" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Reply-To: "Romano, Paul K." <promano(a)anl.gov<mailto:[email protected]>> Date: Thursday, September 19, 2019 at 9:48 AM To: "Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>" <Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] OMP target offload and C++ containers I work on an application that makes heavy use of C++ containers (vectors, unordered_map, etc.) in areas/data structures of the code (good example here<https://github.com/openmc-dev/openmc/blob/develop/include/openmc/nuclide.h#…>) that will need to be offloaded and have a few general questions: - Is it / will it be possible to use these containers on Gen9 / future devices using OpenMP target offload? - If so, are methods that potentially allocate new memory (vector::push_back) permissible? If permissible, should I still avoid them? - Will unified shared memory eventually be able to help? Any suggestions for porting strategies for codes with complex data structures like this would be appreciated. Thanks, Paul -- Ecp-aurora-sdk-discuss mailing list Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0