February 2017 - Intel-nda - mailman3.cels.anl.gov

Re: [JLSE-Intel-NDA] ScaLAPACK on KNL?
by Hammond, Jeff R 20 Feb '17

20 Feb '17

Yes. It is in $MKLROOT if you source the Intel toolchain script (e.g. /soft/compilers/intel/bin/compilervars.sh). You can link single-threaded ScaLAPACK with -mkl=cluster or link it manually as follows (example is ScaLAPACK with threaded BLAS/LAPACK): MKL_STUFF = ${MKLROOT}/lib/intel64/libmkl_scalapack_lp64.a \ -Wl,--start-group \ ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a \ ${MKLROOT}/lib/intel64/libmkl_core.a \ ${MKLROOT}/lib/intel64/libmkl_intel_thread.a \ ${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_lp64.a \ -Wl,--end-group FFTW_INCL = -I${MKLROOT}/include/fftw/ FFTW_LIBS = ${MKL_STUFF} BLAS_LIBS = ${MKL_STUFF} SCALAPACK_LIBS = ${MKL_STUFF} Change lp64 to ilp64 as appropriate. Jeff From: <intel-nda-bounces(a)lists.jlse.anl.gov> on behalf of "Banerjee,Tania" <tmishra(a)ufl.edu> Date: Monday, February 20, 2017 at 2:12 PM To: "intel-nda(a)jlse.anl.gov" <intel-nda(a)jlse.anl.gov> Subject: [JLSE-Intel-NDA] ScaLAPACK on KNL? Hi, Is MKL ScaLAPACK installed on JLSE for use on KNL? Thanks, Tania

1 0

[JLSE-Intel-NDA] ScaLAPACK on KNL?
by Banerjee,Tania 20 Feb '17

20 Feb '17

Hi, Is MKL ScaLAPACK installed on JLSE for use on KNL? Thanks, Tania

1 0

Re: [JLSE-Intel-NDA] Update XPPSL (MPSP) to 1.4.1
by Allen, Benjamin S. 16 Feb '17

16 Feb '17

Assuming the DAP CentOS install is using the normal CentOS repos you should be able todo a normal "yum update" to get to 7.3. 7.3 kernel is around 3.10.0-514.6.1.el7.x86_64 currently. Ensure the kernel isn't excluded in /etc/yum.conf. Grab XPPSL 1.5 from Intel and install it per its instructions. Ben > On Feb 16, 2017, at 9:16 AM, Romero, Nichols A. <naromero(a)anl.gov> wrote: > > Mike, > > I am free from now until 11 am, and then after lunch 1 pm. > > Jeff, > > Good question. I have two Linux kernels on the DAP box. For some of the QMCPACK benchmarks, I seem to get a lot of Vtune over head with the MPSP kernel. When I used the default CentOS kernel, I don’t see this problem. I have managed by rebooted between kernels, but its kind of a nuisance. > > > > On 2/16/17, 9:01 AM, "D'mello, Michael" <michael.dmello(a)intel.com> wrote: > > Hi Nick, > > Let me know when you want to get this done. Except for a few meetings, I will be in 240 all day. > > -michael > > -----Original Message----- > From: intel-nda-bounces(a)lists.jlse.anl.gov [mailto:[email protected]] On Behalf Of Hammond, Jeff R > Sent: Wednesday, February 15, 2017 11:48 PM > To: Romero, Nichols A. <naromero(a)anl.gov>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov> > Cc: intel-nda(a)lists.jlse.anl.gov > Subject: Re: [JLSE-Intel-NDA] Update XPPSL (MPSP) to 1.4.1 > > Stupid question: Why do you want to do this? > > There are probably many things you want to watch out for. Mike D'mello should be able to get you details in any case. > > Jeff > > On 2/15/17, 9:27 PM, "Romero, Nichols A." <naromero(a)anl.gov> wrote: > > I would like to update my DAP box with the latest kernel. (DAP box is running CentOS 7.2). > https://software.intel.com/en-us/articles/xeon-phi-software > > I am assuming this is relatively straight forward install? Or is there something that I should watch out for? > > On 8/25/16, 1:18 PM, "intel-nda-bounces(a)lists.jlse.anl.gov on behalf of Hammond, Jeff R" <intel-nda-bounces(a)lists.jlse.anl.gov on behalf of jeff.r.hammond(a)intel.com> wrote: > > The free list sort patch for Linux is really important. Please test it like crazy and let us know if there any major regressions. > > Jeff > > Sent from my iPhone > >> On Aug 25, 2016, at 10:56 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: >> >> Intel's XPPSL (MPSP) has been updated to 1.4.1. The KNL nodes will get the updates on their next reboot. >> >> Intel included a new kernel module to help deal with the cached memory mode performance degradation over time issue named zonesort_module. See page 21 in the attached doc. >> >> Note, I plan on loading this module and running "echo 0 > /sys/kernel/zone_sort_free_pages/nodeid" on job start where the node is not rebooted, and the mode is cached. If someone is interested in testing this out before I get to implementing it, let me know. >> >> Interesting known issues: >> >> - The hbmalloc library does not guarantee to allocate memory from the closest MCDRAM node when SNC-2 or SNC-4 Memory Mode is used. >> >> Ben >> >> >> <xppsl_user_guide.pdf> >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > > > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > >

1 0

Re: [JLSE-Intel-NDA] Update XPPSL (MPSP) to 1.4.1
by D'mello, Michael 16 Feb '17

16 Feb '17

Hi Nick, Let me know when you want to get this done. Except for a few meetings, I will be in 240 all day. -michael -----Original Message----- From: intel-nda-bounces(a)lists.jlse.anl.gov [mailto:[email protected]] On Behalf Of Hammond, Jeff R Sent: Wednesday, February 15, 2017 11:48 PM To: Romero, Nichols A. <naromero(a)anl.gov>; Allen, Benjamin S. <bsallen(a)alcf.anl.gov> Cc: intel-nda(a)lists.jlse.anl.gov Subject: Re: [JLSE-Intel-NDA] Update XPPSL (MPSP) to 1.4.1 Stupid question: Why do you want to do this? There are probably many things you want to watch out for. Mike D'mello should be able to get you details in any case. Jeff On 2/15/17, 9:27 PM, "Romero, Nichols A." <naromero(a)anl.gov> wrote: I would like to update my DAP box with the latest kernel. (DAP box is running CentOS 7.2). https://software.intel.com/en-us/articles/xeon-phi-software I am assuming this is relatively straight forward install? Or is there something that I should watch out for? On 8/25/16, 1:18 PM, "intel-nda-bounces(a)lists.jlse.anl.gov on behalf of Hammond, Jeff R" <intel-nda-bounces(a)lists.jlse.anl.gov on behalf of jeff.r.hammond(a)intel.com> wrote: The free list sort patch for Linux is really important. Please test it like crazy and let us know if there any major regressions. Jeff Sent from my iPhone > On Aug 25, 2016, at 10:56 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > Intel's XPPSL (MPSP) has been updated to 1.4.1. The KNL nodes will get the updates on their next reboot. > > Intel included a new kernel module to help deal with the cached memory mode performance degradation over time issue named zonesort_module. See page 21 in the attached doc. > > Note, I plan on loading this module and running "echo 0 > /sys/kernel/zone_sort_free_pages/nodeid" on job start where the node is not rebooted, and the mode is cached. If someone is interested in testing this out before I get to implementing it, let me know. > > Interesting known issues: > > - The hbmalloc library does not guarantee to allocate memory from the closest MCDRAM node when SNC-2 or SNC-4 Memory Mode is used. > > Ben > > > <xppsl_user_guide.pdf> > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Update XPPSL (MPSP) to 1.4.1
by Hammond, Jeff R 15 Feb '17

15 Feb '17

Stupid question: Why do you want to do this? There are probably many things you want to watch out for. Mike D'mello should be able to get you details in any case. Jeff On 2/15/17, 9:27 PM, "Romero, Nichols A." <naromero(a)anl.gov> wrote: I would like to update my DAP box with the latest kernel. (DAP box is running CentOS 7.2). https://software.intel.com/en-us/articles/xeon-phi-software I am assuming this is relatively straight forward install? Or is there something that I should watch out for? On 8/25/16, 1:18 PM, "intel-nda-bounces(a)lists.jlse.anl.gov on behalf of Hammond, Jeff R" <intel-nda-bounces(a)lists.jlse.anl.gov on behalf of jeff.r.hammond(a)intel.com> wrote: The free list sort patch for Linux is really important. Please test it like crazy and let us know if there any major regressions. Jeff Sent from my iPhone > On Aug 25, 2016, at 10:56 AM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > Intel's XPPSL (MPSP) has been updated to 1.4.1. The KNL nodes will get the updates on their next reboot. > > Intel included a new kernel module to help deal with the cached memory mode performance degradation over time issue named zonesort_module. See page 21 in the attached doc. > > Note, I plan on loading this module and running "echo 0 > /sys/kernel/zone_sort_free_pages/nodeid" on job start where the node is not rebooted, and the mode is cached. If someone is interested in testing this out before I get to implementing it, let me know. > > Interesting known issues: > > - The hbmalloc library does not guarantee to allocate memory from the closest MCDRAM node when SNC-2 or SNC-4 Memory Mode is used. > > Ben > > > <xppsl_user_guide.pdf> > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] how to enable omni-path for mpirun
by Li, Ying 07 Feb '17

07 Feb '17

Is there any difference between the knl_7210 nodes on jlse? When I grabbed knl00-02, 08. I didn’t get the ideal performance using omni-path, the peak performance of my code was only 8.5 TFLOP/s. But when I grabbed knl03-07 using omni-path, the peak performance of my code was around 11.8 TFLOP/s. Just wondering is it caused by node difference. So when I grabbed different nodes, I can get different performance. And how to make sure to grab certain nodes on jlse? Thanks Intel MPI compiler. export I_MPI_FABRIC=tmi export I_MPI_FALLBACK=1 export KMP_AFFINITY=compact,granularity=fine mpirun -np 4 ./a.out > On Feb 7, 2017, at 12:09 AM, Hammond, Jeff R <jeff.r.hammond(a)intel.com> wrote: > > Try OSU MPI benchmarks. Those are what I use. Not perfect but easy to understand. > > Jeff > > Sent from my iPhone > >> On Feb 6, 2017, at 9:45 PM, Li, Ying <yingli(a)anl.gov> wrote: >> >> yes, it is intel MPI. >> after export I_MPI_FABRIC=tmi and I_MPI_FALLBACK=1. >> Is it supposed to faster? better performance? However, my code doesn’t show much improvement. >> Is there any sample benchmark that I can look into? IMB-MPI1 is really confusing. >> >>> On Feb 6, 2017, at 10:45 PM, Hammond, Jeff R <jeff.r.hammond(a)intel.com> wrote: >>> >>> Intel MPI? Look up I_MPI_FABRIC docs. >>> >>> Sent from my iPhone >>> >>>> On Feb 6, 2017, at 8:44 PM, Li, Ying <yingli(a)anl.gov> wrote: >>>> >>>> Hello, >>>> >>>> How to enable the tmi fabric on knl nodes on jlse? >>>> I tried adding “-iface ib0 -PSM” to my mpirun. But seems doesn’t work. >>>> >>>> Best, >>>> Amy Ying Li >>>> _______________________________________________ >>>> Intel-nda mailing list >>>> Intel-nda(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >>

1 0

Re: [JLSE-Intel-NDA] how to enable omni-path for mpirun
by Hammond, Jeff R 07 Feb '17

07 Feb '17

Try OSU MPI benchmarks. Those are what I use. Not perfect but easy to understand. Jeff Sent from my iPhone > On Feb 6, 2017, at 9:45 PM, Li, Ying <yingli(a)anl.gov> wrote: > > yes, it is intel MPI. > after export I_MPI_FABRIC=tmi and I_MPI_FALLBACK=1. > Is it supposed to faster? better performance? However, my code doesn’t show much improvement. > Is there any sample benchmark that I can look into? IMB-MPI1 is really confusing. > >> On Feb 6, 2017, at 10:45 PM, Hammond, Jeff R <jeff.r.hammond(a)intel.com> wrote: >> >> Intel MPI? Look up I_MPI_FABRIC docs. >> >> Sent from my iPhone >> >>> On Feb 6, 2017, at 8:44 PM, Li, Ying <yingli(a)anl.gov> wrote: >>> >>> Hello, >>> >>> How to enable the tmi fabric on knl nodes on jlse? >>> I tried adding “-iface ib0 -PSM” to my mpirun. But seems doesn’t work. >>> >>> Best, >>> Amy Ying Li >>> _______________________________________________ >>> Intel-nda mailing list >>> Intel-nda(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] how to enable omni-path for mpirun
by Li, Ying 06 Feb '17

06 Feb '17

yes, it is intel MPI. after export I_MPI_FABRIC=tmi and I_MPI_FALLBACK=1. Is it supposed to faster? better performance? However, my code doesn’t show much improvement. Is there any sample benchmark that I can look into? IMB-MPI1 is really confusing. > On Feb 6, 2017, at 10:45 PM, Hammond, Jeff R <jeff.r.hammond(a)intel.com> wrote: > > Intel MPI? Look up I_MPI_FABRIC docs. > > Sent from my iPhone > >> On Feb 6, 2017, at 8:44 PM, Li, Ying <yingli(a)anl.gov> wrote: >> >> Hello, >> >> How to enable the tmi fabric on knl nodes on jlse? >> I tried adding “-iface ib0 -PSM” to my mpirun. But seems doesn’t work. >> >> Best, >> Amy Ying Li >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] how to enable omni-path for mpirun
by Hammond, Jeff R 06 Feb '17

06 Feb '17

Intel MPI? Look up I_MPI_FABRIC docs. Sent from my iPhone > On Feb 6, 2017, at 8:44 PM, Li, Ying <yingli(a)anl.gov> wrote: > > Hello, > > How to enable the tmi fabric on knl nodes on jlse? > I tried adding “-iface ib0 -PSM” to my mpirun. But seems doesn’t work. > > Best, > Amy Ying Li > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

[JLSE-Intel-NDA] how to enable omni-path for mpirun
by Li, Ying 06 Feb '17

06 Feb '17

Hello, How to enable the tmi fabric on knl nodes on jlse? I tried adding “-iface ib0 -PSM” to my mpirun. But seems doesn’t work. Best, Amy Ying Li

1 0