June 2021 - Ecp-aurora-sdk-discuss - mailman3.cels.anl.gov

Multi GPU update to `arcticus_debug` queue
by Muralidharan, Servesh 29 Jun '21

29 Jun '21

Hello Arcticus Node Users, Please take note that the nodes present in `arcticus_debug` queue has been updated to 2x ATS GPUs. The GPUs are the same silicon rev as present in the other arcticus nodes. In addition, the CPUs have been also updated to a newer silicon rev. Exact details on the hardware and the queue can be found here: https://wiki.jlse.anl.gov/display/inteldga/Arcticus No changes to the software environment are needed and everything should continue to work. Should there be any issues kindly let us know. On behalf of Early Hardware Team. -- Servesh

1 0

Re: [Ecp-aurora-sdk-discuss] OneAPI modules compatibility for jlse
by Denoyelle, Nicolas 22 Jun '21

22 Jun '21

Servesh, Thank you for answering my request. I understand your answer and the motivation behind. My coworkers and I came to the conclusion that we are going to make an alternate test as you suggested. I'll keep you posted if we need come across other issues. Nicolas ________________________________ From: Muralidharan, Servesh <servesh(a)anl.gov> Sent: Tuesday, 22 June 2021 11:41 To: Denoyelle, Nicolas <ndenoyelle(a)anl.gov>; ecp-aurora-sdk-discuss(a)jlse.anl.gov <ecp-aurora-sdk-discuss(a)jlse.anl.gov> Subject: Re: OneAPI modules compatibility for jlse Hi Nicolas, The oneapi modulefile in JLSE does several things that are unique to the environment. One of those is setting a specific driver stack depending on the node/Intel GPU being used. The specific library you mention is part of this driver stack and there are several versions of them. In addition, due to NDA nature of some versions we must move the library after installation thereby rendering the pkgconfig generated being incorrect. So therefore, its intentional from our side to not set the pkgconfig path env. However, you will notice that we set INCLUDE and LIBRARY PATHs to the relevant locations in the module file so that should be sufficient to build anything depending on this module. Could you try using AC_CHECK_LIB/AC_SEARCH_LIBS for detecting the library instead? -- Servesh From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Denoyelle, Nicolas via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Date: Monday, June 21, 2021 at 5:59 PM To: ecp-aurora-sdk-discuss(a)jlse.anl.gov <ecp-aurora-sdk-discuss(a)jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] OneAPI modules compatibility for jlse Good Evening, I am looking to port our library to early access systems. This work includes integration with ECP applications. I have few problems that need fixing to report: 1. When loading oneapi modules, the modules don't export PKG_CONFIG_PATH to the libze_loader.pc. This prevents the libze_loader detection when building our library. 2. Available libze_loader.pc files in "/soft/libraries/intel-level-zero/" do not match oneapi modules version which leads to 3. when manually exporting PKG_CONFIG_PATH with the .pc file and a oneapi module loaded. 3. When running with one of these, there is version mismatch. I attached a minimal test for detection of the library and version check. To reproduce issues, unpack the archive, then in the unpacked directory do: ``` module load oneapi ./configure ``` Configure fails if libze_loader was not detected due to PKG_CONFIG_PATH not pointing to a directory containing libze_loader.pc file. ``` export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/soft/libraries/intel-level-zero/api_+_loader/0d30b1f-Release-2021.03.29/lib64/pkgconfig ./configure make ./version ``` ./version will fail if the library and header version do not match. Thank you ahead for taking a look. Nicolas Denoyelle

1 0

Re: [Ecp-aurora-sdk-discuss] OneAPI modules compatibility for jlse
by Muralidharan, Servesh 22 Jun '21

22 Jun '21

Hi Nicolas, The oneapi modulefile in JLSE does several things that are unique to the environment. One of those is setting a specific driver stack depending on the node/Intel GPU being used. The specific library you mention is part of this driver stack and there are several versions of them. In addition, due to NDA nature of some versions we must move the library after installation thereby rendering the pkgconfig generated being incorrect. So therefore, its intentional from our side to not set the pkgconfig path env. However, you will notice that we set INCLUDE and LIBRARY PATHs to the relevant locations in the module file so that should be sufficient to build anything depending on this module. Could you try using AC_CHECK_LIB/AC_SEARCH_LIBS for detecting the library instead? -- Servesh From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Denoyelle, Nicolas via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Date: Monday, June 21, 2021 at 5:59 PM To: ecp-aurora-sdk-discuss(a)jlse.anl.gov <ecp-aurora-sdk-discuss(a)jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] OneAPI modules compatibility for jlse Good Evening, I am looking to port our library to early access systems. This work includes integration with ECP applications. I have few problems that need fixing to report: 1. When loading oneapi modules, the modules don't export PKG_CONFIG_PATH to the libze_loader.pc. This prevents the libze_loader detection when building our library. 2. Available libze_loader.pc files in "/soft/libraries/intel-level-zero/" do not match oneapi modules version which leads to 3. when manually exporting PKG_CONFIG_PATH with the .pc file and a oneapi module loaded. 3. When running with one of these, there is version mismatch. I attached a minimal test for detection of the library and version check. To reproduce issues, unpack the archive, then in the unpacked directory do: ``` module load oneapi ./configure ``` Configure fails if libze_loader was not detected due to PKG_CONFIG_PATH not pointing to a directory containing libze_loader.pc file. ``` export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/soft/libraries/intel-level-zero/api_+_loader/0d30b1f-Release-2021.03.29/lib64/pkgconfig ./configure make ./version ``` ./version will fail if the library and header version do not match. Thank you ahead for taking a look. Nicolas Denoyelle

1 0

OneAPI modules compatibility for jlse
by Denoyelle, Nicolas 21 Jun '21

21 Jun '21

Good Evening, I am looking to port our library to early access systems. This work includes integration with ECP applications. I have few problems that need fixing to report: 1. When loading oneapi modules, the modules don't export PKG_CONFIG_PATH to the libze_loader.pc. This prevents the libze_loader detection when building our library. 2. Available libze_loader.pc files in "/soft/libraries/intel-level-zero/" do not match oneapi modules version which leads to 3. when manually exporting PKG_CONFIG_PATH with the .pc file and a oneapi module loaded. 3. When running with one of these, there is version mismatch. I attached a minimal test for detection of the library and version check. To reproduce issues, unpack the archive, then in the unpacked directory do: ``` module load oneapi ./configure ``` Configure fails if libze_loader was not detected due to PKG_CONFIG_PATH not pointing to a directory containing libze_loader.pc file. ``` export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/soft/libraries/intel-level-zero/api_+_loader/0d30b1f-Release-2021.03.29/lib64/pkgconfig ./configure make ./version ``` ./version will fail if the library and header version do not match. Thank you ahead for taking a look. Nicolas Denoyelle

1 0

DL FW Tensorflow post-alpha2 module for distributed training using horovod + CCL + MPI
by Ramer, Rebecca 11 Jun '21

11 Jun '21

Hi, An additional Deep Learning Frameworks module which enables distributed training using the Tensoflow framework on 2-tile ATS nodes is now available. This module enables the oneapi CCL and MPI components with horovod along with setting environment variables to enable 2-tile ATS operation. The module can be loaded by: module use /soft/restricted/CNDA/modulefiles module load frameworks/tensorflow/multi-tile/2021.04.30.001 The python environment is activated by: source $FWPROOT/bin/activate conda activate $AURORA_MODELS_BASE_ENV The wiki pages have been updated, a section has been added to: https://wiki.jlse.anl.gov/display/inteldga/Alpha+2+-+Tensorflow and there is a link on the wiki page that provides more details: https://wiki.jlse.anl.gov/display/inteldga/Alpha2++TF+-+Distributed+Training Any feedback is welcome, Rebecca Ramer rebecca.ramer(a)intel.com<mailto:[email protected]>

1 0

Re: [Ecp-aurora-sdk-discuss] New Aurora SDK available in JLSE
by Bertoni, Colleen 09 Jun '21

09 Jun '21

It sounds like the "module use /soft/restricted/CNDA/modulefiles" is not working for you. Could you try: `> ls -ltr /soft/restricted/CNDA/modulefiles` and tell us if you can see that directory? Also, if you type "groups" what groups do you see listed? (I'm asking in case this is a permission issue) Thanks, Colleen ________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Wednesday, June 9, 2021 6:40 PM To: Chan-nui, Christopher <christopher.chan-nui(a)intel.com> Cc: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] New Aurora SDK available in JLSE If I try > module use /soft/restricted/CNDA/modulefiles > module add oneapi then I get: ERROR: Unable to locate a modulefile for 'oneapi' When I do "module avail", I get: ------------------------------ /usr/share/modules ------------------------------ dot module-git module-info modules null use.own On Wed, Jun 09, 2021 at 11:00:49PM +0000, Chan-nui, Christopher via Ecp-aurora-sdk-discuss wrote: > Hello, > > We are pleased to announce that a new Aurora SDK has been deployed in JLSE. This version 2021.04.30.003 includes engineering drops of the compilers, MKL, Advisor and VTune. > > Use the oneapi module to load the appropriate drivers for your system. > > > module use /soft/restricted/CNDA/modulefiles > > module add oneapi > > Additionally there is now a oneapi/2021.04.30.003-beta modulefile which will use Level Zero instead of OpenCL and the latest UMD driver. Please note that there is currently an issue with this configuration that will cause MKL programs to fail with a SEGV at the end of their run. > > > module add oneapi/2021.04.30.003-beta > > MPICH is now loaded as the default. Specify the following environment before loading oneapi to skip loading MPICH. > export ONEAPI_MPICH_OVERRIDE=SKIP > > More information about MPI/MPICH can be found at https://wiki.jlse.anl.gov/display/inteldga/MPI. > > New version of the advisor clients are also available, links available at https://wiki.jlse.anl.gov/display/inteldga/oneAPI+client+software. > > For additional details please refer to the latest release notes in the Wiki: > > https://wiki.jlse.anl.gov/display/inteldga/DGA+SDK+Release+Notes > > Public Documentation > ==================== > * ALCF oneAPI overview - https://www.alcf.anl.gov/support-center/aurora/oneapi-overview > * Intel GPGPU Software Information - https://dgpu-docs.intel.com/ > * Intel oneAPI Documentation - https://software.intel.com/content/www/us/en/develop/tools/oneapi/documenta… > > Please let us know of any issues with the release or the environment settings. > > > > > -- > Christopher Chan-Nui > christopher.chan-nui(a)intel.com > -- > Ecp-aurora-sdk-discuss mailing list > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss -- Ecp-aurora-sdk-discuss mailing list Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

Re: [Ecp-aurora-sdk-discuss] New Aurora SDK available in JLSE
by Peter McCorquodale 09 Jun '21

09 Jun '21

If I try > module use /soft/restricted/CNDA/modulefiles > module add oneapi then I get: ERROR: Unable to locate a modulefile for 'oneapi' When I do "module avail", I get: ------------------------------ /usr/share/modules ------------------------------ dot module-git module-info modules null use.own On Wed, Jun 09, 2021 at 11:00:49PM +0000, Chan-nui, Christopher via Ecp-aurora-sdk-discuss wrote: > Hello, > > We are pleased to announce that a new Aurora SDK has been deployed in JLSE. This version 2021.04.30.003 includes engineering drops of the compilers, MKL, Advisor and VTune. > > Use the oneapi module to load the appropriate drivers for your system. > > > module use /soft/restricted/CNDA/modulefiles > > module add oneapi > > Additionally there is now a oneapi/2021.04.30.003-beta modulefile which will use Level Zero instead of OpenCL and the latest UMD driver. Please note that there is currently an issue with this configuration that will cause MKL programs to fail with a SEGV at the end of their run. > > > module add oneapi/2021.04.30.003-beta > > MPICH is now loaded as the default. Specify the following environment before loading oneapi to skip loading MPICH. > export ONEAPI_MPICH_OVERRIDE=SKIP > > More information about MPI/MPICH can be found at https://wiki.jlse.anl.gov/display/inteldga/MPI. > > New version of the advisor clients are also available, links available at https://wiki.jlse.anl.gov/display/inteldga/oneAPI+client+software. > > For additional details please refer to the latest release notes in the Wiki: > > https://wiki.jlse.anl.gov/display/inteldga/DGA+SDK+Release+Notes > > Public Documentation > ==================== > * ALCF oneAPI overview - https://www.alcf.anl.gov/support-center/aurora/oneapi-overview > * Intel GPGPU Software Information - https://dgpu-docs.intel.com/ > * Intel oneAPI Documentation - https://software.intel.com/content/www/us/en/develop/tools/oneapi/documenta… > > Please let us know of any issues with the release or the environment settings. > > > > > -- > Christopher Chan-Nui > christopher.chan-nui(a)intel.com > -- > Ecp-aurora-sdk-discuss mailing list > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

New Aurora SDK available in JLSE
by Chan-nui, Christopher 09 Jun '21

09 Jun '21

Hello, We are pleased to announce that a new Aurora SDK has been deployed in JLSE. This version 2021.04.30.003 includes engineering drops of the compilers, MKL, Advisor and VTune. Use the oneapi module to load the appropriate drivers for your system. module use /soft/restricted/CNDA/modulefiles module add oneapi Additionally there is now a oneapi/2021.04.30.003-beta modulefile which will use Level Zero instead of OpenCL and the latest UMD driver. Please note that there is currently an issue with this configuration that will cause MKL programs to fail with a SEGV at the end of their run. module add oneapi/2021.04.30.003-beta MPICH is now loaded as the default. Specify the following environment before loading oneapi to skip loading MPICH. export ONEAPI_MPICH_OVERRIDE=SKIP More information about MPI/MPICH can be found at https://wiki.jlse.anl.gov/display/inteldga/MPI. New version of the advisor clients are also available, links available at https://wiki.jlse.anl.gov/display/inteldga/oneAPI+client+software. For additional details please refer to the latest release notes in the Wiki: https://wiki.jlse.anl.gov/display/inteldga/DGA+SDK+Release+Notes Public Documentation ==================== * ALCF oneAPI overview - https://www.alcf.anl.gov/support-center/aurora/oneapi-overview * Intel GPGPU Software Information - https://dgpu-docs.intel.com/ * Intel oneAPI Documentation - https://software.intel.com/content/www/us/en/develop/tools/oneapi/documenta… Please let us know of any issues with the release or the environment settings. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com

1 0

Upcoming oneapi update
by Chan-nui, Christopher 09 Jun '21

09 Jun '21

At 6pm CDT the oneapi installation will be updated. Running jobs will not be affected. Default modulefiles will be updated. This update contains: Updated Engineering Compilers/MKL/Advisor/VTune MPICH will be the default MPI -- Christopher Chan-Nui christopher.chan-nui(a)intel.com

1 0