January 2021 - Ecp-aurora-sdk-discuss

Re: [Ecp-aurora-sdk-discuss] DL Framework modules available in the JLSE
by Adams, Corey James 14 Jan '21

14 Jan '21

Hi Becky, Thanks for this announcement. I am pleased ATS is finally supported. I have some questions that are useful to have answered publicly: 1. Is there any reason to use Preview-3? We had to wait so long for it that we get Preview-5 at the same time, which I assume is everything in Preview-3 + updates. 2. If we run on the ATS nodes on JLSE, what tools (similar to nvidia-smi) can be used to monitor gpu memory usage? 3. I tested CosmicTagger right away with Tensorflow, Preview-5. It has been hanging, before the first iteration, for more than 30 minutes. The page below indicates it runs successfully. How can we debug this? 4. What is the status of horovod in Preview-5? Thanks, and happy new year, Corey From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of "Ramer, Rebecca via Ecp-aurora-sdk-discuss" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Reply-To: "Ramer, Rebecca" <rebecca.ramer(a)intel.com> Date: Thursday, January 14, 2021 at 10:53 AM To: "ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] DL Framework modules available in the JLSE Hi, We would like to announce availability of pytorch and tensorflow deep learning framework modules for use in the JLSE. The pytorch and tensorflow alpha 1 update ATS preview 3 modules provide python 3.6.10 python environments and use the oneapi beta09 version of the dpcpp compiler on iris, yarrow, and arcticus nodes. More details, including how to setup and use the python environments can be viewed on the wiki, see: https://wiki.jlse.anl.gov/display/inteldga/Alpha1+update+-+ATS+Preview+3+Py… and https://wiki.jlse.anl.gov/display/inteldga/Alpha1+update+-+ATS+Preview+3+Te… The alpha 1 update ATS preview 5 tensorflow and pytorch modules provide python 3.6.10 python environments and use the oneapi gold nda version of the dpcpp compiler on iris, yarrow, and arcticus nodes. Details, including how to setup and use the python environments can be viewed on the wiki, see: https://wiki.jlse.anl.gov/display/inteldga/Alpha1+update+-+ATS+Preview+5+Py… and https://wiki.jlse.anl.gov/display/inteldga/Alpha1+update+-+ATS+Preview+5+Te… Any feedback is welcome. Rebecca Ramer rebecca.ramer(a)intel.com

1 0

DL Framework modules available in the JLSE
by Ramer, Rebecca 14 Jan '21

14 Jan '21

Hi, We would like to announce availability of pytorch and tensorflow deep learning framework modules for use in the JLSE. The pytorch and tensorflow alpha 1 update ATS preview 3 modules provide python 3.6.10 python environments and use the oneapi beta09 version of the dpcpp compiler on iris, yarrow, and arcticus nodes. More details, including how to setup and use the python environments can be viewed on the wiki, see: https://wiki.jlse.anl.gov/display/inteldga/Alpha1+update+-+ATS+Preview+3+Py… and https://wiki.jlse.anl.gov/display/inteldga/Alpha1+update+-+ATS+Preview+3+Te… The alpha 1 update ATS preview 5 tensorflow and pytorch modules provide python 3.6.10 python environments and use the oneapi gold nda version of the dpcpp compiler on iris, yarrow, and arcticus nodes. Details, including how to setup and use the python environments can be viewed on the wiki, see: https://wiki.jlse.anl.gov/display/inteldga/Alpha1+update+-+ATS+Preview+5+Py… and https://wiki.jlse.anl.gov/display/inteldga/Alpha1+update+-+ATS+Preview+5+Te… Any feedback is welcome. Rebecca Ramer rebecca.ramer(a)intel.com

1 0

New version of mpich on JLSE
by Chan-nui, Christopher 08 Jan '21

08 Jan '21

Hello, We are pleased to announce that an update to mpich drop 38.1 has been deployed on JLSE. module use /soft/restricted/CNDA/modulefiles module avail mpi/aurora_mpich This should work on iris and yarrow clusters. Release Notes · Added support for Argonne DG1 based systems · Added documentation on different features and optimizations · The user guide is available in `$MPICH_INSTALL_DIR/share/doc/mpich/` Details · Support for switch-based collectives · Support for pipelined inter-node communication with GPU based buffers · Added performance profiling counters (PVARS) for striping and multiplexing optimizations and one-sided communication using multiple NICs · Fixed bugs for multi-threaded tests in Yaksa · Fixed bugs for supporting managed (USM shared) memory in Yaksa · Other bug fixes -- Christopher Chan-Nui christopher.chan-nui(a)intel.com

1 0

Re: [Ecp-aurora-sdk-discuss] process per node flag for qsub?
by Harms, Kevin 07 Jan '21

07 Jan '21

Greg, for mpirun, use mpirun -f $COBALT_NODEFILE. $COBALT_NODEFILE should have a list of all the nodes you requested. kevin ________________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Kosinovsky, Greg via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Thursday, January 7, 2021 8:46 AM To: ecp-aurora-sdk-discuss(a)jlse.anl.gov Subject: [Ecp-aurora-sdk-discuss] process per node flag for qsub? I need to run a program on Iris (or another Aurora machine) where there are 3 processes on 3 nodes (1 process per node). I am using qsub in interactive mode – and I am new to qsub. I have experimented with using proccount flag, so I either run “qsub -q iris -t 0:30:00 -n 3 --proccount 3 -I” or “qsub -q iris -t 0:30:00 -n 3 --proccount 1 -I” and then start my program with 3 processes within the session. In either case, all 3 processes are launched on a single node – so proccount is clearly not the right flag to regulate this. IS there a qsub flag that would allow me to distribute the processes

1 0

Re: [Ecp-aurora-sdk-discuss] process per node flag for qsub?
by Anisimov, Victor 07 Jan '21

07 Jan '21

Hi Greg, The qsub command performs resource (number of nodes) allocation. It does not assign processes to nodes. The latter is done by mpirun (mpiexec) command. Once you start an interactive session, do the following export MODULEPATH=/soft/restricted/CNDA/modulefiles module load oneapi module load mpi/aurora_mpich/drop34 mpirun --help Check -n and -ppn options. The first one declares the total number of MPI processes to be started, and the second option tells how many of those will be running on each node. This explanation presumes that your application supports MPI parallel programming model. This is what we are using for distribute parallelism. If you use a different parallel programming model, please let us know. Thank you, Victor -- Victor Anisimov, PhD Application Performance Engineer Argonne Leadership Computing Facility 9700 S. Cass Ave, Bldg 240, Room 1121 Lemont, IL 60439 > On Jan 7, 2021, at 8:46 AM, Kosinovsky, Greg via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> wrote: > > I need to run a program on Iris (or another Aurora machine) where there are 3 processes on 3 nodes (1 process per node). I am using qsub in interactive mode – and I am new to qsub. I have experimented with using proccount flag, so I either run “qsub -q iris -t 0:30:00 -n 3 --proccount 3 -I” or “qsub -q iris -t 0:30:00 -n 3 --proccount 1 -I” and then start my program with 3 processes within the session. > In either case, all 3 processes are launched on a single node – so proccount is clearly not the right flag to regulate this. IS there a qsub flag that would allow me to distribute the processes > -- > Ecp-aurora-sdk-discuss mailing list > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

process per node flag for qsub?
by Kosinovsky, Greg 07 Jan '21

07 Jan '21

I need to run a program on Iris (or another Aurora machine) where there are 3 processes on 3 nodes (1 process per node). I am using qsub in interactive mode - and I am new to qsub. I have experimented with using proccount flag, so I either run "qsub -q iris -t 0:30:00 -n 3 --proccount 3 -I" or "qsub -q iris -t 0:30:00 -n 3 --proccount 1 -I" and then start my program with 3 processes within the session. In either case, all 3 processes are launched on a single node - so proccount is clearly not the right flag to regulate this. IS there a qsub flag that would allow me to distribute the processes

1 0

Using oneDPL
by Arndt, Daniel 06 Jan '21

06 Jan '21

I am currently trying to use oneDPL but apparently oneapi/dpl/algorithm cannot be found in any standard include paths when loading module oneapi. Does an oneDPL implementation exist and if so do I need to load extra modules? Best, Daniel

1 0

Re: [Ecp-aurora-sdk-discuss] /dev/shm equivalent for Iris nodes
by Hammond, Jeff R 05 Jan '21

05 Jan '21

Can you describe what you are seeing that indicates /dev/shm is not present on Iris nodes? I just grabbed an Iris node and /dev/shm is present for me. Jeff [jhammond@jlselogin2 ~]$ qsub -I -t 10 -n 1 -q iris_debug Wait for job 309817 to start... Opening interactive session to iris03 … jhammond@iris03:~> ll /dev/shm/ … jhammond@iris03:~> du -hs /dev/shm/ 360K /dev/shm/ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of "Hammond, Jeff R via Ecp-aurora-sdk-discuss" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Reply-To: "Hammond, Jeff R" <jeff.r.hammond(a)intel.com> Date: Tuesday, January 5, 2021 at 1:27 PM To: "Kosinovsky, Greg" <kosinovsky1(a)llnl.gov>, "ecp-aurora-sdk-discuss(a)jlse.anl.gov" <ecp-aurora-sdk-discuss(a)jlse.anl.gov> Subject: Re: [Ecp-aurora-sdk-discuss] /dev/shm equivalent for Iris nodes The features of /dev/shm should be accessible via https://man7.org/linux/man-pages/man7/shm_overview.7.html, e.g. shm_open. Do you find that these are inadequate, or is POSIX shared-memory not functional on the Iris nodes? Jeff From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of "Kosinovsky, Greg via Ecp-aurora-sdk-discuss" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Reply-To: "Kosinovsky, Greg" <kosinovsky1(a)llnl.gov> Date: Tuesday, January 5, 2021 at 1:24 PM To: "ecp-aurora-sdk-discuss(a)jlse.anl.gov" <ecp-aurora-sdk-discuss(a)jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] /dev/shm equivalent for Iris nodes I just realized that /dev/shm/ is visible from login nodes but not from iris nodes. Is there a shared partition that is visible from Iris nodes? Thank you, Greg Kosinovsky

1 0

Re: [Ecp-aurora-sdk-discuss] /dev/shm equivalent for Iris nodes
by Hammond, Jeff R 05 Jan '21

05 Jan '21

The features of /dev/shm should be accessible via https://man7.org/linux/man-pages/man7/shm_overview.7.html, e.g. shm_open. Do you find that these are inadequate, or is POSIX shared-memory not functional on the Iris nodes? Jeff From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of "Kosinovsky, Greg via Ecp-aurora-sdk-discuss" <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Reply-To: "Kosinovsky, Greg" <kosinovsky1(a)llnl.gov> Date: Tuesday, January 5, 2021 at 1:24 PM To: "ecp-aurora-sdk-discuss(a)jlse.anl.gov" <ecp-aurora-sdk-discuss(a)jlse.anl.gov> Subject: [Ecp-aurora-sdk-discuss] /dev/shm equivalent for Iris nodes I just realized that /dev/shm/ is visible from login nodes but not from iris nodes. Is there a shared partition that is visible from Iris nodes? Thank you, Greg Kosinovsky

1 0

/dev/shm equivalent for Iris nodes
by Kosinovsky, Greg 05 Jan '21

05 Jan '21

I just realized that /dev/shm/ is visible from login nodes but not from iris nodes. Is there a shared partition that is visible from Iris nodes? Thank you, Greg Kosinovsky

1 0