August 2020 - Ecp-aurora-sdk-discuss

New A21 SDK compiler update completed
by Rosales-fernandez, Carlos 28 Aug '20

28 Aug '20

Hi all, We are pleased to announce that a new A21 SDK C/C++/Fortran compiler has been deployed in JLSE. This version 2020.07.30.002 includes multiple ibug fixes, particularly in Fortran. A newer build of MKL is also included in this release in order to maintain compatibility with the new compiler. This version will NOT be default in the system since it contains several important changes that have the potential to affect your build scripts. For additional details please refer to the latest release notes in the Wiki: https://wiki.jlse.anl.gov/display/inteldga/DGA+SDK+Release+Notes Important notes =============== Compiler flag "-qnextgen" has been deprecated, please use "icx/icpx/ifx" directly instead of "icc -qnextgen/icpc -qnextgen/ifor -qnextgen" Behavior of icx and icpx has changed. C++ codes that previously compiled with icx may need to be compiled wtih icpx now. MKL namespace has changed from "mkl" to "oneapi::mkl" When using dynamic linking for threaded MKL calls, please include the tbb location in teh link line. An old tbb.so library is being picked up by the linker from /lib64 and this is a workaround to avoid issues, so substitute your typicall "-ltbb" in the MKL build by (prepending location to LD_LIBRARY_PATH will not work): $ -L$TBBROOT/lib/intel64/gcc4.8 -ltbb Public Documentation ==================== * ALCF oneAPI overview - https://www.alcf.anl.gov/support-center/aurora/oneapi-overview * Intel GPGPU Software Information - https://dgpu-docs.intel.com/ * Intel oneAPI Beta Documentation - https://software.intel.com/content/www/us/en/develop/tools/oneapi/documenta… Please let us know of any issues with the release or the environment settings. Regards, Carlos

1 0

New A21 SDK compiler update coming to JLSE today
by Rosales-fernandez, Carlos 28 Aug '20

28 Aug '20

Hi all, A new A21 SDK compiler will be installed today at 2:30 PM CST. Running jobs will not be affected. A notice will be sent after the install is complete. Regards, Carlos

1 0

Re: [Ecp-aurora-sdk-discuss] libpcreposix missing since openSUSE migration
by Vincent R. Pascuzzi 25 Aug '20

25 Aug '20

Hi, Servesh. > This problem likely comes from differences in pcre between RHEL and SUSE images that run in different nodes. > > My recommendation to build one that is suitable to your local install by following the instructions here: http://www.linuxfromscratch.org/blfs/view/svn/general/pcre.html > This way system level libraries across nodes will not cause compatibility issues. Thanks for the reply and suggestion — I’ll give it a shot. Vince

1 0

Re: [Ecp-aurora-sdk-discuss] libpcreposix missing since openSUSE migration
by Muralidharan, Servesh 25 Aug '20

25 Aug '20

Hi Vincent, Please refrain from sending mail simultaneously to two different mailing lists i.e. intel-dga-discuss(a)lists.jlse.anl.gov <intel-dga-discuss(a)lists.jlse.anl.gov> and ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> The mailing lists are designed to handle information at different levels of NDA. Sending them to both doesn’t really give you a quicker response nor does it get a higher priority. This problem likely comes from differences in pcre between RHEL and SUSE images that run in different nodes. My recommendation to build one that is suitable to your local install by following the instructions here: http://www.linuxfromscratch.org/blfs/view/svn/general/pcre.html This way system level libraries across nodes will not cause compatibility issues. -- Servesh On 8/25/20, 8:13 AM, "Ecp-aurora-sdk-discuss on behalf of Vincent R. Pascuzzi via Ecp-aurora-sdk-discuss" <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov on behalf of ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> wrote: On iris nodes, I have a compilation issue: /home/vrpascuzzi/sw/install/root/6.14.08-gcc-9.3.0/bin/rootcling: error while loading shared libraries: libpcreposix.so.0: cannot open shared object file: No such file or directory Any idea how to fix this? Thanks, Vince -- Ecp-aurora-sdk-discuss mailing list Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

libpcreposix missing since openSUSE migration
by Vincent R. Pascuzzi 25 Aug '20

25 Aug '20

On iris nodes, I have a compilation issue: /home/vrpascuzzi/sw/install/root/6.14.08-gcc-9.3.0/bin/rootcling: error while loading shared libraries: libpcreposix.so.0: cannot open shared object file: No such file or directory Any idea how to fix this? Thanks, Vince

1 0

Re: [Ecp-aurora-sdk-discuss] Iris Nodes Migration to OpenSuse Leap 15.2
by Ben Allen 21 Aug '20

21 Aug '20

All, Remainder of the Iris nodes will be migrated at 4:30pm CDT today. There's a reservation in place for this time. The work will take no more than 2 hours. Ben On 8/14/20 11:46 AM, Ben Allen via Ecp-aurora-sdk-discuss wrote: > All, > > We've started the migration of the Iris nodes from their current RHEL7 > based OS image to OpenSuse Leap 15.2. We're doing this to match the > environment on the upcoming Yarrow (DG1) nodes, which matches the > environment Intel is developing against. > > To start with, we've migrated the 4 nodes iris[00-03] in the > iris_debug queue. These are available now. We'll migrate the remainder > of the nodes by the end of next week as long as we don't hear from you > about any critical issues. > > With this change, we've also been able to mount our GPFS jlse-fs0. > Note that jlse-fs0 will be mounted over 1GbE on the Iris nodes, so > performance will not be as good as nodes with IB. As a reminder, all > users have a directory under /gpfs/jlse-fs0/users/, and we don't > currently enforce quotas on this filesystem. > > Best, > > Ben >

1 0

Iris Nodes Migration to OpenSuse Leap 15.2
by Ben Allen 14 Aug '20

14 Aug '20

All, We've started the migration of the Iris nodes from their current RHEL7 based OS image to OpenSuse Leap 15.2. We're doing this to match the environment on the upcoming Yarrow (DG1) nodes, which matches the environment Intel is developing against. To start with, we've migrated the 4 nodes iris[00-03] in the iris_debug queue. These are available now. We'll migrate the remainder of the nodes by the end of next week as long as we don't hear from you about any critical issues. With this change, we've also been able to mount our GPFS jlse-fs0. Note that jlse-fs0 will be mounted over 1GbE on the Iris nodes, so performance will not be as good as nodes with IB. As a reminder, all users have a directory under /gpfs/jlse-fs0/users/, and we don't currently enforce quotas on this filesystem. Best, Ben

1 0

Re: [Ecp-aurora-sdk-discuss] SYCL contexts with iGPU
by Vincent R. Pascuzzi 12 Aug '20

12 Aug '20

FYI: https://www.khronos.org/registry/SYCL/specs/sycl-1.2.1.pdf Page 33, paragraph 4 gives a good explanation of contexts and queues. Vince > On Aug 12, 2020, at 3:40 PM, Vincent R. Pascuzzi <vrpascuzzi(a)lbl.gov> wrote: > > Hi, Kevin. > > Thanks a lot for the quick reply! > > Coincidentally, I used a shared context before as the same problem popped up earlier in my development, but it turned out to be due to one queue being associated to the ‘host’ device and the other to the ‘GPU’ device. Having both use the same device fixed the problem, so I ignored the context sharing since I assumed "same device, same context.” > > Guess this isn’t quite right. I will go back to the shared context and see what happens. > > Food for thought: Since CUDA doesn’t isn’t context-oriented like OpenCL, I wonder how this business will work with the CUDA PI…I should try. > > > Best, > Vince > >> On Aug 12, 2020, at 3:12 PM, Harms, Kevin <harms(a)alcf.anl.gov> wrote: >> >> Vince, >> >> I think the Intel dpc++ creates a new context for each queue even if the queues use the same device. When you allocate queues, you can pass the context it should use in the constructor. If you try that, you can see if it fixes the issue. You can get the context from the first queue you create. >> >> kevin >> >> ________________________________________ >> From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Vincent R. Pascuzzi via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> >> Sent: Wednesday, August 12, 2020 5:01 PM >> To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov >> Subject: [Ecp-aurora-sdk-discuss] SYCL contexts with iGPU >> >> My understanding of the cl::sycl::context is that they are used among the different devices on a system. For example, I can only access memory allocated on a given cl::sycl::device using a cl::sycl::queue that was instantiated with the cl::sycl::context associated to said device. >> >> Is this correct? >> >> A summary of my problem: I generate random numbers in one class and store put them into the memory of an Iris node’s Neo device using a cl::sycl::queue member variable in this class. In another class, I allocate a bunch of memory on the ‘same’(?) Neo device using a cl::sycl::queue member variable in this other class. I get the associated device in both classes with the same common static function, i.e. the same Neo device is used. There are 3-4 classes that are implemented similarly. Now, it seems in some cases that I can use a single cl::sycl::queue from an arbitrary class to access data allocated in other classes, but in other cases I can’t; I need to use the queue/context that was used to allocate the memory. Of course I can’t operate this way since I need to access data stored among multiple classes using a single queue. >> >> Is there a preferred/common/proper way to achieve what I want to do? Should I have a global “queue manager” class of sorts that uses a single queue/device/context to take care of all device-side tasks? Should I be explicitly using the same context is shared among all classes? (I wouldn’t think this mattered if my understanding of contexts is correct, and there is only a single iGPU on each machine. Unless there is different memory that is not shareable?) >> >> Thanks in advance. >> >> >> Vince >> >> -- >> Ecp-aurora-sdk-discuss mailing list >> Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss >

1 0

Re: [Ecp-aurora-sdk-discuss] SYCL contexts with iGPU
by Vincent R. Pascuzzi 12 Aug '20

12 Aug '20

Hi, Kevin. Thanks a lot for the quick reply! Coincidentally, I used a shared context before as the same problem popped up earlier in my development, but it turned out to be due to one queue being associated to the ‘host’ device and the other to the ‘GPU’ device. Having both use the same device fixed the problem, so I ignored the context sharing since I assumed "same device, same context.” Guess this isn’t quite right. I will go back to the shared context and see what happens. Food for thought: Since CUDA doesn’t isn’t context-oriented like OpenCL, I wonder how this business will work with the CUDA PI…I should try. Best, Vince > On Aug 12, 2020, at 3:12 PM, Harms, Kevin <harms(a)alcf.anl.gov> wrote: > > Vince, > > I think the Intel dpc++ creates a new context for each queue even if the queues use the same device. When you allocate queues, you can pass the context it should use in the constructor. If you try that, you can see if it fixes the issue. You can get the context from the first queue you create. > > kevin > > ________________________________________ > From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Vincent R. Pascuzzi via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> > Sent: Wednesday, August 12, 2020 5:01 PM > To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > Subject: [Ecp-aurora-sdk-discuss] SYCL contexts with iGPU > > My understanding of the cl::sycl::context is that they are used among the different devices on a system. For example, I can only access memory allocated on a given cl::sycl::device using a cl::sycl::queue that was instantiated with the cl::sycl::context associated to said device. > > Is this correct? > > A summary of my problem: I generate random numbers in one class and store put them into the memory of an Iris node’s Neo device using a cl::sycl::queue member variable in this class. In another class, I allocate a bunch of memory on the ‘same’(?) Neo device using a cl::sycl::queue member variable in this other class. I get the associated device in both classes with the same common static function, i.e. the same Neo device is used. There are 3-4 classes that are implemented similarly. Now, it seems in some cases that I can use a single cl::sycl::queue from an arbitrary class to access data allocated in other classes, but in other cases I can’t; I need to use the queue/context that was used to allocate the memory. Of course I can’t operate this way since I need to access data stored among multiple classes using a single queue. > > Is there a preferred/common/proper way to achieve what I want to do? Should I have a global “queue manager” class of sorts that uses a single queue/device/context to take care of all device-side tasks? Should I be explicitly using the same context is shared among all classes? (I wouldn’t think this mattered if my understanding of contexts is correct, and there is only a single iGPU on each machine. Unless there is different memory that is not shareable?) > > Thanks in advance. > > > Vince > > -- > Ecp-aurora-sdk-discuss mailing list > Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0

Re: [Ecp-aurora-sdk-discuss] SYCL contexts with iGPU
by Harms, Kevin 12 Aug '20

12 Aug '20

Vince, I think the Intel dpc++ creates a new context for each queue even if the queues use the same device. When you allocate queues, you can pass the context it should use in the constructor. If you try that, you can see if it fixes the issue. You can get the context from the first queue you create. kevin ________________________________________ From: Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss-bounces(a)lists.jlse.anl.gov> on behalf of Vincent R. Pascuzzi via Ecp-aurora-sdk-discuss <ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov> Sent: Wednesday, August 12, 2020 5:01 PM To: ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov Subject: [Ecp-aurora-sdk-discuss] SYCL contexts with iGPU My understanding of the cl::sycl::context is that they are used among the different devices on a system. For example, I can only access memory allocated on a given cl::sycl::device using a cl::sycl::queue that was instantiated with the cl::sycl::context associated to said device. Is this correct? A summary of my problem: I generate random numbers in one class and store put them into the memory of an Iris node’s Neo device using a cl::sycl::queue member variable in this class. In another class, I allocate a bunch of memory on the ‘same’(?) Neo device using a cl::sycl::queue member variable in this other class. I get the associated device in both classes with the same common static function, i.e. the same Neo device is used. There are 3-4 classes that are implemented similarly. Now, it seems in some cases that I can use a single cl::sycl::queue from an arbitrary class to access data allocated in other classes, but in other cases I can’t; I need to use the queue/context that was used to allocate the memory. Of course I can’t operate this way since I need to access data stored among multiple classes using a single queue. Is there a preferred/common/proper way to achieve what I want to do? Should I have a global “queue manager” class of sorts that uses a single queue/device/context to take care of all device-side tasks? Should I be explicitly using the same context is shared among all classes? (I wouldn’t think this mattered if my understanding of contexts is correct, and there is only a single iGPU on each machine. Unless there is different memory that is not shareable?) Thanks in advance. Vince -- Ecp-aurora-sdk-discuss mailing list Ecp-aurora-sdk-discuss(a)lists.jlse.anl.gov https://lists.jlse.anl.gov/mailman/listinfo/ecp-aurora-sdk-discuss

1 0