May 2021 - Ecp-aurora-sdk-discuss - mailman3.cels.anl.gov

DL FW Pytorch alpha2 module for distributed training using horovod + CCL + MPI
by Ramer, Rebecca 27 May '21

27 May '21

Hi, An additional Deep Learning Frameworks module which enables distributed training using the Pytorch framework on 2-tile ATS nodes is now available. This module enables the oneapi CCL and MPI components with horovod along with setting environment variables to enable 2-tile ATS operation. The module can be loaded by: module use /soft/restricted/CNDA/modulefiles module load frameworks/pytorch/multi-tile/2021.04.30.001 The python environment is activated by: source $FWPROOT/bin/activate conda activate $AURORA_MODELS_BASE_ENV The wiki pages have been updated, a section has been added to: https://wiki.jlse.anl.gov/display/inteldga/Alpha+2+-+Pytorch and there is a link on the wiki page that provides more details: https://wiki.jlse.anl.gov/display/inteldga/Alpha2+PT+-+Distributed+Training A module for distributed training using the Tensorflow framework on 2-tile ATS nodes is forthcoming. Any feedback is welcome, Rebecca Ramer rebecca.ramer(a)intel.com<mailto:[email protected]>

1 0

GPU reset process on Arcticus nodes
by Muralidharan, Servesh 14 May '21

14 May '21

Hello folks, We have enabled the ability to reset ATS A1 GPU used in Arcticus nodes. The procedure, FAQs and details are maintained here: https://wiki.jlse.anl.gov/display/inteldga/GPU+Reset+-+ATS To reset the ATS GPU or ensure if it’s behaving normally, please run, $ /soft/restricted/CNDA/tools/flr.sh -r On behalf of early hardware team. -- Servesh

1 0

Re: [Ecp-aurora-sdk-discuss] Need help setting up conda environment
by Muralidharan, Servesh 10 May '21

10 May '21

This is oneapi SDK released Q3’2020, hence its removal. There was notice and deprecation given for this for several months and finally removed since April. Email informing users, From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of Chan-nui, Christopher via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Date: Monday, April 12, 2021 at 15:32 To: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] Upcoming oneapi update At 6pm CDT the oneapi will be updated. Running jobs will not be affected. Default modulefiles will be updated. A number of changes will be made at that time: Update to 2021.2 oneAPI + engineering compiler release. Removal of old individual component modulefiles. (dpcpp, advisor, daal etc…) Removal of old SDK installations. Kokkos added to oneapi modulefile. This will be silently updated without changing the oneapi version number. -- Christopher Chan-Nui christopher.chan-nui(a)intel.com<mailto:[email protected]> As I see there are two options, 1. Use the Python from oneapi SDK, again this will only be maintained for 2 update cycles before removal. * To use it, `module use /soft/restricted/CNDA/modulefiles && module load frameworks/2021.04.30.001 && source $IDPROOT/bin/activate` For more info ` module use /soft/restricted/CNDA/modulefiles && module help frameworks/2021.04.30.001` 2. Use system Python, we are running OpenSUSE 15.2, and we have recent versions of both v3 and v2 installed and you can configure your own local Conda, which will be good until the next major update to the OS happens at JLSE, not foreseen anytime soon. -- Servesh From: Shahzeb Siddiqui <shahzebsiddiqui(a)lbl.gov> Date: Monday, May 10, 2021 at 3:43 PM To: Muralidharan, Servesh <servesh(a)anl.gov> Cc: ecp-aurora-sdk-discuss(a)jlse.anl.gov <ecp-aurora-sdk-discuss(a)jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] Need help setting up conda environment The module was the following: python/2020.09.15.001 The oneapi module is supposed to provide python3 not sure if it provides as conda. If so ill see if that works. On May 10, 2021, at 3:10 PM, Muralidharan, Servesh <servesh(a)anl.gov<mailto:[email protected]>> wrote: Hi Shahzeb, I don’t have access to those files. I don’t think there was a generic version of Conda maintained via module file at JLSE, so I’m not sure where you picked this up from. JLSE is not a production system, it’s an evolving environment that is focused on improving and supporting early testbed platforms, so there will be deprecation/updates on a regular basis. That said we don’t generally remove module file unless there was a very specific reason, i.e. The old versions of the SDK are no longer supported, or the specific platform was migrated to a new version. Could you clarify which module file you were using and from where and I can point you to the reason and when it was removed. Also, which platform/project are you focusing on? If its related Intel testbeds users get regular updates on this mailing list related to these changes. -- Servesh From: Shahzeb Siddiqui <shahzebsiddiqui(a)lbl.gov<mailto:[email protected]>> Date: Monday, May 10, 2021 at 1:49 PM To: Muralidharan, Servesh <servesh(a)anl.gov<mailto:[email protected]>> Cc: ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]> <ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]>> Subject: Re: [Ecp-aurora-sdk-discuss] Need help setting up conda environment Hi Servesh, I had a CI setup for running regression test for buildtest https://gitlab.jlse.anl.gov/e4s/shahzebsiddiqui/buildtest that relied on python module see gitlab pipeline https://gitlab.jlse.anl.gov/e4s/shahzebsiddiqui/buildtest/-/blob/devel/.git…. Once this python module was removed the CI job failed now i have to retool my CI job to fix this. It was not cool, removing a modulefile if someone else depends on it and break their workflow. I have been deploying software stack for past 5 yearss and I never removed modulefile because of this exact reason. You don’t know who relies on the modulefile and their workflows are broken and now i have to spend time figuring this out. I will work through the link perhaps in oneapi https://wiki.jlse.anl.gov/display/inteldga/oneAPI+Software and see if that works. As a end-user it is confusing which sub-page i should reference https://wiki.jlse.anl.gov/display/inteldga/Frameworks+and+Libraries, i am just interested in conda. Most of the pages are geared towards setting up an environment for a specific tool pytorch, numpy, oneDNN, etc… I just need conda. Perhaps there should be a main page just to outline which module to load to get conda. It is not clear to me which module i should be loading to get conda as there are too several modules that provide conda tool but modulefile are tailored to the tool. Regards, Shahzeb On May 10, 2021, at 2:35 PM, Muralidharan, Servesh <servesh(a)anl.gov<mailto:[email protected]>> wrote: Hi Shahzeb, What is your specific requirement? If you are looking for standard Conda install you are welcome to install this to your home directory of GPFS filesystem. JLSE has too many different architectures and platforms to maintain a conda version for each environment. If your focus is more towards Intel GPUs then the recommendation is to use the Conda env or follow the instructions specific to the frameworks section of the wiki,https://wiki.jlse.anl.gov/display/inteldga/Frameworks+and+Libraries JLSE doesn’t have a dedicated software support staff maintaining the packages and most that are present are through user driven collaboration and specific Intel/Argonne POCs. -- Servesh From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of Shahzeb Siddiqui via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Date: Monday, May 10, 2021 at 12:23 PM To: ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]> <ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] Need help setting up conda environment Hi Ben, This is in response to AREQ0437155 <https://servicenow.anl.gov/nav_to.do?uri=u_ad_hoc_request.do%3Fsys_id=b5caf…> I tried using module load pytorch/2020.12.31.001-conda to setup a blank conda environment and i am running into issue. shahzebsiddiqui@jlselogin6:~> conda create env -n buildtest -y Solving environment: failed PackagesNotFoundError: The following packages are not available from current channels: - env Current channels: - https://repo.anaconda.com/pkgs/main/linux-64 - https://repo.anaconda.com/pkgs/main/noarch - https://repo.anaconda.com/pkgs/free/linux-64 - https://repo.anaconda.com/pkgs/free/noarch - https://repo.anaconda.com/pkgs/r/linux-64 - https://repo.anaconda.com/pkgs/r/noarch - https://repo.anaconda.com/pkgs/pro/linux-64 - https://repo.anaconda.com/pkgs/pro/noarch To search for alternate channels that may provide the conda package you're looking for, navigate to https://anaconda.org<https://anaconda.org/> and use the search bar at the top of the page. Can you send me instructions on how to setup a blank conda environment, the instructions are bit unclear. There should be a conda or python module doesn’t make sense i have to load pytorch to access conda binaries. It already setups up an active conda environment named “preview3_pt” this doesn’t work well in a CI workflow if i already have something activated Shown below is the output of “conda env list” it shows many environment but this one is active preview3_pt * /soft/restricted/CNDA/pytorchPOC/install/anaconda3/envs/preview3_pt I tried to deactivate environment but and rerun same command it didn’t work. It would be helpful to get some instructions in wiki for loading conda at JLSE without using pytorch since i dont need this tool. What i expect ismodule load conda and it should give me binaries to conda package and drop me into an empty environment then we can just create environment in our user space. Regards, Shahzeb

1 0

Re: [Ecp-aurora-sdk-discuss] Need help setting up conda environment
by Shahzeb Siddiqui 10 May '21

10 May '21

The module was the following: python/2020.09.15.001 The oneapi module is supposed to provide python3 not sure if it provides as conda. If so ill see if that works. > On May 10, 2021, at 3:10 PM, Muralidharan, Servesh <servesh(a)anl.gov> wrote: > > Hi Shahzeb, > > I don’t have access to those files. I don’t think there was a generic version of Conda maintained via module file at JLSE, so I’m not sure where you picked this up from. > > JLSE is not a production system, it’s an evolving environment that is focused on improving and supporting early testbed platforms, so there will be deprecation/updates on a regular basis. > That said we don’t generally remove module file unless there was a very specific reason, i.e. The old versions of the SDK are no longer supported, or the specific platform was migrated to a new version. > > Could you clarify which module file you were using and from where and I can point you to the reason and when it was removed. > Also, which platform/project are you focusing on? If its related Intel testbeds users get regular updates on this mailing list related to these changes. > > -- > Servesh > > From: Shahzeb Siddiqui <shahzebsiddiqui(a)lbl.gov> > Date: Monday, May 10, 2021 at 1:49 PM > To: Muralidharan, Servesh <servesh(a)anl.gov> > Cc: ecp-aurora-sdk-discuss(a)jlse.anl.gov <ecp-aurora-sdk-discuss(a)jlse.anl.gov> > Subject: Re: [Ecp-aurora-sdk-discuss] Need help setting up conda environment > > Hi Servesh, > > I had a CI setup for running regression test for buildtest https://gitlab.jlse.anl.gov/e4s/shahzebsiddiqui/buildtest <https://gitlab.jlse.anl.gov/e4s/shahzebsiddiqui/buildtest> that relied on python module see gitlab pipeline https://gitlab.jlse.anl.gov/e4s/shahzebsiddiqui/buildtest/-/blob/devel/.git… <https://gitlab.jlse.anl.gov/e4s/shahzebsiddiqui/buildtest/-/blob/devel/.git…>. Once this python module was removed the CI job failed now i have to retool my CI job to fix this. It was not cool, removing a modulefile if someone else depends on it and break their workflow. I have been deploying software stack for past 5 yearss and I never removed modulefile because of this exact reason. You don’t know who relies on the modulefile and their workflows are broken and now i have to spend time figuring this out. > > I will work through the link perhaps in oneapi https://wiki.jlse.anl.gov/display/inteldga/oneAPI+Software <https://wiki.jlse.anl.gov/display/inteldga/oneAPI+Software> and see if that works. As a end-user it is confusing which sub-page i should reference https://wiki.jlse.anl.gov/display/inteldga/Frameworks+and+Libraries <https://wiki.jlse.anl.gov/display/inteldga/Frameworks+and+Libraries>, i am just interested in conda. Most of the pages are geared towards setting up an environment for a specific tool pytorch, numpy, oneDNN, etc… I just need conda. Perhaps there should be a main page just to outline which module to load to get conda. > > It is not clear to me which module i should be loading to get conda as there are too several modules that provide conda tool but modulefile are tailored to the tool. > > > Regards, > Shahzeb > > > On May 10, 2021, at 2:35 PM, Muralidharan, Servesh <servesh(a)anl.gov <mailto:[email protected]>> wrote: > > Hi Shahzeb, > > What is your specific requirement? > > If you are looking for standard Conda install you are welcome to install this to your home directory of GPFS filesystem. JLSE has too many different architectures and platforms to maintain a conda version for each environment. > If your focus is more towards Intel GPUs then the recommendation is to use the Conda env or follow the instructions specific to the frameworks section of the wiki,https://wiki.jlse.anl.gov/display/inteldga/Frameworks+and+Libraries <https://wiki.jlse.anl.gov/display/inteldga/Frameworks+and+Libraries> > > JLSE doesn’t have a dedicated software support staff maintaining the packages and most that are present are through user driven collaboration and specific Intel/Argonne POCs. > > -- > Servesh > > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov <mailto:[email protected]>> on behalf of Shahzeb Siddiqui via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <mailto:[email protected]>> > Date: Monday, May 10, 2021 at 12:23 PM > To: ecp-aurora-sdk-discuss(a)jlse.anl.gov <mailto:[email protected]> <ecp-aurora-sdk-discuss(a)jlse.anl.gov <mailto:[email protected]>> > Subject: [Ecp-aurora-sdk-discuss] Need help setting up conda environment > > Hi Ben, > > This is in response to AREQ0437155 > <https://servicenow.anl.gov/nav_to.do?uri=u_ad_hoc_request.do%3Fsys_id=b5caf…> > I tried using module load pytorch/2020.12.31.001-conda to setup a blank conda environment and i am running into issue. > > > > shahzebsiddiqui@jlselogin6:~> conda create env -n buildtest -y > Solving environment: failed > > PackagesNotFoundError: The following packages are not available from current channels: > > - env > > Current channels: > > - https://repo.anaconda.com/pkgs/main/linux-64 <https://repo.anaconda.com/pkgs/main/linux-64> > - https://repo.anaconda.com/pkgs/main/noarch <https://repo.anaconda.com/pkgs/main/noarch> > - https://repo.anaconda.com/pkgs/free/linux-64 <https://repo.anaconda.com/pkgs/free/linux-64> > - https://repo.anaconda.com/pkgs/free/noarch <https://repo.anaconda.com/pkgs/free/noarch> > - https://repo.anaconda.com/pkgs/r/linux-64 <https://repo.anaconda.com/pkgs/r/linux-64> > - https://repo.anaconda.com/pkgs/r/noarch <https://repo.anaconda.com/pkgs/r/noarch> > - https://repo.anaconda.com/pkgs/pro/linux-64 <https://repo.anaconda.com/pkgs/pro/linux-64> > - https://repo.anaconda.com/pkgs/pro/noarch <https://repo.anaconda.com/pkgs/pro/noarch> > > To search for alternate channels that may provide the conda package you're > looking for, navigate to > > https://anaconda.org <https://anaconda.org/> > > and use the search bar at the top of the page. > > Can you send me instructions on how to setup a blank conda environment, the instructions are bit unclear. There should be a conda or python module doesn’t make sense i have to load pytorch to access conda binaries. It already setups up an active conda environment named “preview3_pt” this doesn’t work well in a CI workflow if i already have something activated > Shown below is the output of “conda env list” it shows many environment but this one is active > > preview3_pt * /soft/restricted/CNDA/pytorchPOC/install/anaconda3/envs/preview3_pt > > I tried to deactivate environment but and rerun same command it didn’t work. It would be helpful to get some instructions in wiki for loading conda at JLSE without using pytorch since i dont need this tool. What i expect ismodule load conda and it should give me binaries to conda package and drop me into an empty environment then we can just create environment in our user space. > > > > Regards, > Shahzeb

1 0

Re: [Ecp-aurora-sdk-discuss] Need help setting up conda environment
by Muralidharan, Servesh 10 May '21

10 May '21

Hi Shahzeb, I don’t have access to those files. I don’t think there was a generic version of Conda maintained via module file at JLSE, so I’m not sure where you picked this up from. JLSE is not a production system, it’s an evolving environment that is focused on improving and supporting early testbed platforms, so there will be deprecation/updates on a regular basis. That said we don’t generally remove module file unless there was a very specific reason, i.e. The old versions of the SDK are no longer supported, or the specific platform was migrated to a new version. Could you clarify which module file you were using and from where and I can point you to the reason and when it was removed. Also, which platform/project are you focusing on? If its related Intel testbeds users get regular updates on this mailing list related to these changes. -- Servesh From: Shahzeb Siddiqui <shahzebsiddiqui(a)lbl.gov> Date: Monday, May 10, 2021 at 1:49 PM To: Muralidharan, Servesh <servesh(a)anl.gov> Cc: ecp-aurora-sdk-discuss(a)jlse.anl.gov <ecp-aurora-sdk-discuss(a)jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] Need help setting up conda environment Hi Servesh, I had a CI setup for running regression test for buildtest https://gitlab.jlse.anl.gov/e4s/shahzebsiddiqui/buildtest that relied on python module see gitlab pipeline https://gitlab.jlse.anl.gov/e4s/shahzebsiddiqui/buildtest/-/blob/devel/.git…. Once this python module was removed the CI job failed now i have to retool my CI job to fix this. It was not cool, removing a modulefile if someone else depends on it and break their workflow. I have been deploying software stack for past 5 yearss and I never removed modulefile because of this exact reason. You don’t know who relies on the modulefile and their workflows are broken and now i have to spend time figuring this out. I will work through the link perhaps in oneapi https://wiki.jlse.anl.gov/display/inteldga/oneAPI+Software and see if that works. As a end-user it is confusing which sub-page i should reference https://wiki.jlse.anl.gov/display/inteldga/Frameworks+and+Libraries, i am just interested in conda. Most of the pages are geared towards setting up an environment for a specific tool pytorch, numpy, oneDNN, etc… I just need conda. Perhaps there should be a main page just to outline which module to load to get conda. It is not clear to me which module i should be loading to get conda as there are too several modules that provide conda tool but modulefile are tailored to the tool. Regards, Shahzeb On May 10, 2021, at 2:35 PM, Muralidharan, Servesh <servesh(a)anl.gov<mailto:[email protected]>> wrote: Hi Shahzeb, What is your specific requirement? If you are looking for standard Conda install you are welcome to install this to your home directory of GPFS filesystem. JLSE has too many different architectures and platforms to maintain a conda version for each environment. If your focus is more towards Intel GPUs then the recommendation is to use the Conda env or follow the instructions specific to the frameworks section of the wiki,https://wiki.jlse.anl.gov/display/inteldga/Frameworks+and+Libraries JLSE doesn’t have a dedicated software support staff maintaining the packages and most that are present are through user driven collaboration and specific Intel/Argonne POCs. -- Servesh From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov<mailto:[email protected]>> on behalf of Shahzeb Siddiqui via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov<mailto:[email protected]>> Date: Monday, May 10, 2021 at 12:23 PM To: ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]> <ecp-aurora-sdk-discuss(a)jlse.anl.gov<mailto:[email protected]>> Subject: [Ecp-aurora-sdk-discuss] Need help setting up conda environment Hi Ben, This is in response to AREQ0437155 <https://servicenow.anl.gov/nav_to.do?uri=u_ad_hoc_request.do%3Fsys_id=b5caf…> I tried using module load pytorch/2020.12.31.001-conda to setup a blank conda environment and i am running into issue. shahzebsiddiqui@jlselogin6:~> conda create env -n buildtest -y Solving environment: failed PackagesNotFoundError: The following packages are not available from current channels: - env Current channels: - https://repo.anaconda.com/pkgs/main/linux-64 - https://repo.anaconda.com/pkgs/main/noarch - https://repo.anaconda.com/pkgs/free/linux-64 - https://repo.anaconda.com/pkgs/free/noarch - https://repo.anaconda.com/pkgs/r/linux-64 - https://repo.anaconda.com/pkgs/r/noarch - https://repo.anaconda.com/pkgs/pro/linux-64 - https://repo.anaconda.com/pkgs/pro/noarch To search for alternate channels that may provide the conda package you're looking for, navigate to https://anaconda.org<https://anaconda.org/> and use the search bar at the top of the page. Can you send me instructions on how to setup a blank conda environment, the instructions are bit unclear. There should be a conda or python module doesn’t make sense i have to load pytorch to access conda binaries. It already setups up an active conda environment named “preview3_pt” this doesn’t work well in a CI workflow if i already have something activated Shown below is the output of “conda env list” it shows many environment but this one is active preview3_pt * /soft/restricted/CNDA/pytorchPOC/install/anaconda3/envs/preview3_pt I tried to deactivate environment but and rerun same command it didn’t work. It would be helpful to get some instructions in wiki for loading conda at JLSE without using pytorch since i dont need this tool. What i expect ismodule load conda and it should give me binaries to conda package and drop me into an empty environment then we can just create environment in our user space. Regards, Shahzeb

1 0

Re: [Ecp-aurora-sdk-discuss] Need help setting up conda environment
by Shahzeb Siddiqui 10 May '21

10 May '21

Hi Servesh, I had a CI setup for running regression test for buildtest https://gitlab.jlse.anl.gov/e4s/shahzebsiddiqui/buildtest <https://gitlab.jlse.anl.gov/e4s/shahzebsiddiqui/buildtest> that relied on python module see gitlab pipeline https://gitlab.jlse.anl.gov/e4s/shahzebsiddiqui/buildtest/-/blob/devel/.git… <https://gitlab.jlse.anl.gov/e4s/shahzebsiddiqui/buildtest/-/blob/devel/.git…>. Once this python module was removed the CI job failed now i have to retool my CI job to fix this. It was not cool, removing a modulefile if someone else depends on it and break their workflow. I have been deploying software stack for past 5 yearss and I never removed modulefile because of this exact reason. You don’t know who relies on the modulefile and their workflows are broken and now i have to spend time figuring this out. I will work through the link perhaps in oneapi https://wiki.jlse.anl.gov/display/inteldga/oneAPI+Software <https://wiki.jlse.anl.gov/display/inteldga/oneAPI+Software> and see if that works. As a end-user it is confusing which sub-page i should reference https://wiki.jlse.anl.gov/display/inteldga/Frameworks+and+Libraries <https://wiki.jlse.anl.gov/display/inteldga/Frameworks+and+Libraries>, i am just interested in conda. Most of the pages are geared towards setting up an environment for a specific tool pytorch, numpy, oneDNN, etc… I just need conda. Perhaps there should be a main page just to outline which module to load to get conda. It is not clear to me which module i should be loading to get conda as there are too several modules that provide conda tool but modulefile are tailored to the tool. Regards, Shahzeb > On May 10, 2021, at 2:35 PM, Muralidharan, Servesh <servesh(a)anl.gov> wrote: > > Hi Shahzeb, > > What is your specific requirement? > > If you are looking for standard Conda install you are welcome to install this to your home directory of GPFS filesystem. JLSE has too many different architectures and platforms to maintain a conda version for each environment. > If your focus is more towards Intel GPUs then the recommendation is to use the Conda env or follow the instructions specific to the frameworks section of the wiki,https://wiki.jlse.anl.gov/display/inteldga/Frameworks+and+Libraries <https://wiki.jlse.anl.gov/display/inteldga/Frameworks+and+Libraries> > > JLSE doesn’t have a dedicated software support staff maintaining the packages and most that are present are through user driven collaboration and specific Intel/Argonne POCs. > > -- > Servesh > > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Shahzeb Siddiqui via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Date: Monday, May 10, 2021 at 12:23 PM > To: ecp-aurora-sdk-discuss(a)jlse.anl.gov <ecp-aurora-sdk-discuss(a)jlse.anl.gov> > Subject: [Ecp-aurora-sdk-discuss] Need help setting up conda environment > > Hi Ben, > > This is in response to AREQ0437155 > <https://servicenow.anl.gov/nav_to.do?uri=u_ad_hoc_request.do%3Fsys_id=b5caf…> > I tried using module load pytorch/2020.12.31.001-conda to setup a blank conda environment and i am running into issue. > > > > shahzebsiddiqui@jlselogin6:~> conda create env -n buildtest -y > Solving environment: failed > > PackagesNotFoundError: The following packages are not available from current channels: > > - env > > Current channels: > > - https://repo.anaconda.com/pkgs/main/linux-64 <https://repo.anaconda.com/pkgs/main/linux-64> > - https://repo.anaconda.com/pkgs/main/noarch <https://repo.anaconda.com/pkgs/main/noarch> > - https://repo.anaconda.com/pkgs/free/linux-64 <https://repo.anaconda.com/pkgs/free/linux-64> > - https://repo.anaconda.com/pkgs/free/noarch <https://repo.anaconda.com/pkgs/free/noarch> > - https://repo.anaconda.com/pkgs/r/linux-64 <https://repo.anaconda.com/pkgs/r/linux-64> > - https://repo.anaconda.com/pkgs/r/noarch <https://repo.anaconda.com/pkgs/r/noarch> > - https://repo.anaconda.com/pkgs/pro/linux-64 <https://repo.anaconda.com/pkgs/pro/linux-64> > - https://repo.anaconda.com/pkgs/pro/noarch <https://repo.anaconda.com/pkgs/pro/noarch> > > To search for alternate channels that may provide the conda package you're > looking for, navigate to > > https://anaconda.org <https://anaconda.org/> > > and use the search bar at the top of the page. > > Can you send me instructions on how to setup a blank conda environment, the instructions are bit unclear. There should be a conda or python module doesn’t make sense i have to load pytorch to access conda binaries. It already setups up an active conda environment named “preview3_pt” this doesn’t work well in a CI workflow if i already have something activated > Shown below is the output of “conda env list” it shows many environment but this one is active > > preview3_pt * /soft/restricted/CNDA/pytorchPOC/install/anaconda3/envs/preview3_pt > > I tried to deactivate environment but and rerun same command it didn’t work. It would be helpful to get some instructions in wiki for loading conda at JLSE without using pytorch since i dont need this tool. What i expect ismodule load conda and it should give me binaries to conda package and drop me into an empty environment then we can just create environment in our user space. > > > > Regards, > Shahzeb

1 0

Re: [Ecp-aurora-sdk-discuss] Need help setting up conda environment
by Muralidharan, Servesh 10 May '21

10 May '21

Hi Shahzeb, What is your specific requirement? If you are looking for standard Conda install you are welcome to install this to your home directory of GPFS filesystem. JLSE has too many different architectures and platforms to maintain a conda version for each environment. If your focus is more towards Intel GPUs then the recommendation is to use the Conda env or follow the instructions specific to the frameworks section of the wiki, https://wiki.jlse.anl.gov/display/inteldga/Frameworks+and+Libraries JLSE doesn’t have a dedicated software support staff maintaining the packages and most that are present are through user driven collaboration and specific Intel/Argonne POCs. -- Servesh From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Shahzeb Siddiqui via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Date: Monday, May 10, 2021 at 12:23 PM To: ecp-aurora-sdk-discuss(a)jlse.anl.gov <ecp-aurora-sdk-discuss(a)jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] Need help setting up conda environment Hi Ben, This is in response to AREQ0437155 <https://servicenow.anl.gov/nav_to.do?uri=u_ad_hoc_request.do%3Fsys_id=b5caf…> I tried using module load pytorch/2020.12.31.001-conda to setup a blank conda environment and i am running into issue. shahzebsiddiqui@jlselogin6:~> conda create env -n buildtest -y Solving environment: failed PackagesNotFoundError: The following packages are not available from current channels: - env Current channels: - https://repo.anaconda.com/pkgs/main/linux-64 - https://repo.anaconda.com/pkgs/main/noarch - https://repo.anaconda.com/pkgs/free/linux-64 - https://repo.anaconda.com/pkgs/free/noarch - https://repo.anaconda.com/pkgs/r/linux-64 - https://repo.anaconda.com/pkgs/r/noarch - https://repo.anaconda.com/pkgs/pro/linux-64 - https://repo.anaconda.com/pkgs/pro/noarch To search for alternate channels that may provide the conda package you're looking for, navigate to https://anaconda.org and use the search bar at the top of the page. Can you send me instructions on how to setup a blank conda environment, the instructions are bit unclear. There should be a conda or python module doesn’t make sense i have to load pytorch to access conda binaries. It already setups up an active conda environment named “preview3_pt” this doesn’t work well in a CI workflow if i already have something activated Shown below is the output of “conda env list” it shows many environment but this one is active preview3_pt * /soft/restricted/CNDA/pytorchPOC/install/anaconda3/envs/preview3_pt I tried to deactivate environment but and rerun same command it didn’t work. It would be helpful to get some instructions in wiki for loading conda at JLSE without using pytorch since i dont need this tool. What i expect is module load conda and it should give me binaries to conda package and drop me into an empty environment then we can just create environment in our user space. Regards, Shahzeb

1 0

Need help setting up conda environment
by Shahzeb Siddiqui 10 May '21

10 May '21

Hi Ben, This is in response to AREQ0437155 <https://servicenow.anl.gov/nav_to.do?uri=u_ad_hoc_request.do%3Fsys_id=b5caf…> I tried using module load pytorch/2020.12.31.001-conda to setup a blank conda environment and i am running into issue. shahzebsiddiqui@jlselogin6:~> conda create env -n buildtest -y Solving environment: failed PackagesNotFoundError: The following packages are not available from current channels: - env Current channels: - https://repo.anaconda.com/pkgs/main/linux-64 <https://repo.anaconda.com/pkgs/main/linux-64>- https://repo.anaconda.com/pkgs/main/noarch <https://repo.anaconda.com/pkgs/main/noarch>- https://repo.anaconda.com/pkgs/free/linux-64 <https://repo.anaconda.com/pkgs/free/linux-64>- https://repo.anaconda.com/pkgs/free/noarch <https://repo.anaconda.com/pkgs/free/noarch>- https://repo.anaconda.com/pkgs/r/linux-64 <https://repo.anaconda.com/pkgs/r/linux-64>- https://repo.anaconda.com/pkgs/r/noarch <https://repo.anaconda.com/pkgs/r/noarch>- https://repo.anaconda.com/pkgs/pro/linux-64 <https://repo.anaconda.com/pkgs/pro/linux-64>- https://repo.anaconda.com/pkgs/pro/noarch <https://repo.anaconda.com/pkgs/pro/noarch> To search for alternate channels that may provide the conda package you're looking for, navigate to https://anaconda.org <https://anaconda.org/> and use the search bar at the top of the page. Can you send me instructions on how to setup a blank conda environment, the instructions are bit unclear. There should be a conda or python module doesn’t make sense i have to load pytorch to access conda binaries. It already setups up an active conda environment named “preview3_pt” this doesn’t work well in a CI workflow if i already have something activated Shown below is the output of “conda env list” it shows many environment but this one is active preview3_pt * /soft/restricted/CNDA/pytorchPOC/install/anaconda3/envs/preview3_pt I tried to deactivate environment but and rerun same command it didn’t work. It would be helpful to get some instructions in wiki for loading conda at JLSE without using pytorch since i dont need this tool. What i expect is module load conda and it should give me binaries to conda package and drop me into an empty environment then we can just create environment in our user space. Regards, Shahzeb

1 0

DL FW Framework module revised to support the FFN model
by Ramer, Rebecca 07 May '21

07 May '21

Hi, A new version of the DL FW module is available in the JLSE. This version of the module provides a python environment specifically for the FFN model, it can be loaded by: module use /soft/restricted/CNDA/modulefiles module load framework/2021.04.30.002 The python environment which support the ffn model is activated by: source $FWPROOT/bin/activate conda activate $AURORA_MODEL_FFN_ENV The wiki pages have been updated, see: https://wiki.jlse.anl.gov/display/inteldga/Alpha+2+-+Tensorflow The run details links are now available, see the Validation Details column in the Aurora Model Status table at the end of the wiki page, eg. ffn run details : https://wiki.jlse.anl.gov/display/inteldga/Alpha2+TF+-+FFN-Run-Details Any feedback is welcome, Rebecca Ramer rebecca.ramer(a)intel.com<mailto:[email protected]>

1 0

Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target
by Bertoni, Colleen 06 May '21

06 May '21

Thanks for checking! Brian reminded me that we have an open bug for the XSBench incorrect value (CMPLRLLVM-25528). I'm also seeing print statements not work on ATS. Brian/Roy mentioned that it's open as `CMPLRLIBS-33357<https://jira.devtools.intel.com/browse/CMPLRLIBS-33357>` , and that if you run with the Level0 backend (export LIBOMPTARGET_PLUGIN=LEVEL0) printf statements should at least work (Thanks Brian and Roy!) Colleen ________________________________ From: Peter McCorquodale <PWMcCorquodale(a)lbl.gov> Sent: Thursday, May 6, 2021 1:24 PM To: Bertoni, Colleen <bertoni(a)anl.gov> Cc: Tramm, John R. <jtramm(a)anl.gov>; Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Luo, Ye <yeluo(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target Colleen: Yes, when I try that on yarrow I get a lot of: error: double type is not supported on this platform error: backend compiler failed build. There are some differences between iris and arcticus. XSBench runs to completion on both, but there are a couple of problems on arcticus: - the printf statements that I put inside the offloaded OpenMP target loop do not produce any output; - I get an invalid verification checksum for the results. Peter On Thu, May 06, 2021 at 03:15:47PM +0000, Bertoni, Colleen wrote: > Thanks! I checked with John and the codes are similar, so it's likely a similar issue. > > Since yarrow doesn't have hardware support for double precision, it just emulates double precision in software. And since yarrow isn't meant to support double precision, the software emulation support isn't perfect, and even when it works, performance can be 10-100X slower. > > To see if the issue is from the double precision emulation, if you turn off the emulator and enable OMP debugging: > ``` > unset IGC_EnableDPEmulation > export LIBOMPTARGET_DEBUG=1 > ``` > the fail should be visible: > ``` > Target LEVEL0 RTL --> Error: module creation failed -- see below for details. > error: double type is not supported on this platform > error: backend compiler failed build. > ``` > > If you really want to run on yarrow, you can switch the types in xsbench to "float" and it should run, but the answers might be wrong. It might be worth it to stick on iris for now, or try arcticus. > > (Thanks to Roy (Intel) for commenting on the issue!) > > Colleen > ________________________________ > From: Peter McCorquodale <PWMcCorquodale(a)lbl.gov> > Sent: Wednesday, May 5, 2021 6:41 PM > To: Bertoni, Colleen <bertoni(a)anl.gov> > Cc: Tramm, John R. <jtramm(a)anl.gov>; Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Luo, Ye <yeluo(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > Colleen, > OK, I put a print statement at the beginning of the loop that should be > offloaded, and it never shows up in the output on yarrow. > Yes, I pulled from https://github.com/ANL-CESAR/XSBench. > > Peter > > On Wed, May 05, 2021 at 11:24:00PM +0000, Bertoni, Colleen wrote: > > Yarrow has no hardware support for double precision numbers, so it could just be very slow. On the other hand, an XSBench test from John Tramm (CC'd) is known to hang on Yarrow, and we have an open bug with Intel about it (CMPLRLLVM-24234). > > > > So 2 things: > > 1) maybe try running for 10 min, or adding a print statement to see if it's making progress. > > 2) John, could you say if the code you put in the test set is the same as what Peter is using (I think just pulled from https://github.com/ANL-CESAR/XSBench, right?) > > > > Colleen > > ________________________________ > > From: Peter McCorquodale <PWMcCorquodale(a)lbl.gov> > > Sent: Wednesday, May 5, 2021 5:30 PM > > To: Bertoni, Colleen <bertoni(a)anl.gov> > > Cc: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Luo, Ye <yeluo(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > > > If I try that with either -n 2 or -n 1, same setting for mpirun and > > qsub, then that runs fine on iris, in less than 1 minute, but on > > yarrow, it hangs somewhere in the middle of its execution, and > > exhausts its 5-minute time limit. > > > > Peter > > > > On Wed, May 05, 2021 at 09:33:39PM +0000, Bertoni, Colleen wrote: > > > Thanks for the response! Could you try > > > > > > mpirun -n ${num_process} ./XSBench-icx -m event -s small > > > > > > instead of "./XSBench-icx -m event -s small" ? Some mpi versions need the `mpirun` to launch the job. > > > > > > Colleen > > > ________________________________ > > > From: Peter McCorquodale <PWMcCorquodale(a)lbl.gov> > > > Sent: Wednesday, May 5, 2021 4:25 PM > > > To: Bertoni, Colleen <bertoni(a)anl.gov> > > > Cc: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Luo, Ye <yeluo(a)anl.gov>; ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > > > > > Colleen: Yes, on 2 nodes with a 5-minute time limit. > > > > > > Following Ye Luo's suggestion that I was running out of memory, I > > > tried a smaller problem size, and now it runs to completion on iris, > > > but I still get those MPI errors on yarrow and arcticus. The script is: > > > #!/bin/bash > > > source /usr/share/Modules/init/bash > > > module use /soft/restricted/CNDA/modulefiles > > > module load mpi > > > module load oneapi > > > ./XSBench-icx -m event -s small > > > > > > Peter > > > > > > On Wed, May 05, 2021 at 05:15:41PM +0000, Bertoni, Colleen wrote: > > > > Hi Peter, > > > > > > > > Could you show us `runicx.sh`? The other thing I noticed is that you're executing `qsub -n 2 -t 5 -q iris runicx.sh`, which tries to execute on 2 iris nodes with a time limit of 5 minutes. To be clear, you were trying to run across two iris nodes? > > > > > > > > I know we have a version of XSBench which runs on iris, but I think it's only been on a single iris node. > > > > > > > > Thanks, > > > > Colleen > > > > ________________________________ > > > > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Luo, Ye via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > > Sent: Tuesday, May 4, 2021 7:09 PM > > > > To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com>; Peter McCorquodale <PWMcCorquodale(a)lbl.gov> > > > > Cc: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > > > > > > > Call to getOrAllocTgtPtr returned null pointer > > > > It is likely that it ran out of GPU memory > > > > Ye > > > > ________________________________ > > > > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > > Sent: Tuesday, May 4, 2021 6:58 PM > > > > To: Steinbrecher, Patrick <patrick.steinbrecher(a)intel.com> > > > > Cc: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > > Subject: Re: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > > > > > > > On Tue, May 04, 2021 at 10:37:38PM +0000, Steinbrecher, Patrick wrote: > > > > > For OpenMP offload on Intel GPU you should use the flags: > > > > > > > > > > -fiopenmp -fopenmp-targets=spir64 > > > > > > > > Yes, the full set of flags I have is: > > > > -std=gnu99 -Wall -fiopenmp -fopenmp-targets=spir64 -D__STRICT_ANSI__ -g -pg -O3 -DMPI -fopenmp -D__STRICT_ANSI__ -cc=icx > > > > > > > > > I don't know about MPI error. Looks like you are using MPI with threading. Maybe that is not supported with that module. Try to just use MPI_Init. > > > > > > > > My source code does just call MPI_Init. And it's only yarrow that gives me > > > > the error in PMPI_Init. On iris, I get a different error, later on: > > > > > > > > > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Peter McCorquodale via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > > > Date: Tuesday, May 4, 2021 at 17:29 > > > > > To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > > > > > Subject: [Ecp-aurora-sdk-discuss] offloading with OpenMP target > > > > > I have been building and running the openmp-offload source in XSBench: > > > > > https://github.com/ANL-CESAR/XSBench > > > > > > > > > > I can get it to build if in openmp-offload/Makefile I set > > > > > CC = /soft/compilers/intel-2020.4/compilers_and_libraries_2020.4.304/linux/mpi/intel64/bin/mpicc > > > > > and > > > > > CFLAGS += -fopenmp -D__STRICT_ANSI__ -cc=icx > > > > > > > > > > But then I get consistently different errors when I run it on iris or yarrow. > > > > > > > > > > On yarrow, qsub -n 2 -t 5 -q yarrow runicx.sh > > > > > it terminates before producing any output: > > > > > Abort(1094799) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Other MPI error, error stack: > > > > > MPIR_Init_thread(666)......: > > > > > MPID_Init(922).............: > > > > > MPIDI_NM_mpi_init_hook(719): OFI addrinfo() failed (ofi_init.h:719:MPIDI_NM_mpi_init_hook:No data available) > > > > > > > > > > On iris, qsub -n 2 -t 5 -q iris runicx.sh > > > > > it gets further along in execution, as I see from output, until: > > > > > Libomptarget error: Call to getOrAllocTgtPtr returned null pointer (device failure or illegal mapping). > > > > > Libomptarget error: Call to targetDataBegin failed, abort target. > > > > > Libomptarget error: Failed to process data before launching the kernel. > > > > > Libomptarget error: Run with LIBOMPTARGET_DEBUG=4 to dump host-target pointer mappings. > > > > > unknown:68:44: Libomptarget fatal error 1: failure of target construct while offloading is mandatory > > > > > > > > > > This suggests that the problem on iris might be with the OpenMP target > > > > > pragma, but the only #pragma omp target statement in the source > > > > > code is this, and it works on other non-JLSE platforms: > > > > > #pragma omp target teams distribute parallel for\ > > > > > map(to: SD.max_num_nucs)\ > > > > > map(to: SD.num_nucs[:SD.length_num_nucs])\ > > > > > map(to: SD.concs[:SD.length_concs])\ > > > > > map(to: SD.mats[:SD.length_mats])\ > > > > > map(to: SD.unionized_energy_array[:SD.length_unionized_energy_array])\ > > > > > map(to: SD.index_grid[:SD.length_index_grid])\ > > > > > map(to: SD.nuclide_grid[:SD.length_nuclide_grid])\ > > > > > reduction(+:verification) > > > > > for( int i = 0; i < in.lookups; i++ ) > > > > > ... > > > > > > > > > > Peter > > > > > -- > > > > > Ecp-aurora-sdk-discuss mailing list > > > > > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > > > > > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss > > > > -- > > > > Ecp-aurora-sdk-discuss mailing list > > > > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > > > > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0