January 2015 - Admins - mailman3.cels.anl.gov

Re: [JLSE-Admins] JLSE MIC home directory
by Allen, Benjamin S. 20 Jan '15

20 Jan '15

Hi John, It appears our license expired on the 18th. Sorry I wasn't on top of that. I'll let you know when it's updated. Ben > On Jan 20, 2015, at 10:36 AM, John Tramm <jtramm(a)mcs.anl.gov> wrote: > > Hey Ben, > > I'm trying to compile my code, but I seem to be getting license errors when trying to use the intel compiler (I'm using the same environment that had been working previously, I believe): > > jtramm@lucie:~/SimpleMOC-kernel/src/cpu$ make > icc -std=gnu99 -mmic -O3 -ansi-alias -no-prec-div -DINTEL -fp-model fast=2 -openmp -DOPENMP -c main.c -o main.o > > Error: A license for CCompL is not available now (-97,121). > > A connection to the license server could not be made. You should > make sure that your license daemon process is running: both an > lmgrd process and an INTEL process should be running > if your license limits you to a specified number of licenses in use > at a time. Also, check to see if the wrong port@host or the wrong > license file is being used, or if the port or hostname in the license > file has changed. > > License file(s) used were (in this order): > 1. Trusted Storage > ** 2. 28518@ftsn2 > ** 3. /soft/compilers/intel/composer_xe_2013_sp1.1.106/licenses > ** 4. /opt/intel/licenses > ** 5. /home/jtramm/intel/licenses > ** 6. /soft/compilers/intel/composer_xe_2013_sp1.1.106/Licenses > ** 7. /Users/Shared/Library/Application Support/Intel/Licenses > ** 8. /soft/compilers/intel/composer_xe_2013_sp1.1.106/bin/intel64/*.lic > > Please visit http://software.intel.com/sites/support/ if you require technical assistance. > > icc: error #10052: could not checkout FLEXlm license > make: *** [main.o] Error 1 > > On Fri, Jan 16, 2015 at 10:33 AM, John Tramm <jtramm(a)mcs.anl.gov> wrote: > Ben, > > That sounds like a good plan. Thanks for the help! > > -John > > On Thu, Jan 15, 2015 at 3:05 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > Hi John, > > Assuming /home/jtramm/SimpleMOC-kernel/src/cpu/SimpleMOC-kernel is the actual app you're trying to profile, I can start with the packages referenced in the "Cannot locate debugging symbols" below. > > Ben > > > On Jan 14, 2015, at 8:15 PM, John Tramm <jtramm(a)mcs.anl.gov> wrote: > > > > Hey Ben, > > > > Thanks for doing this! I would ideally like all of the packages installed, as I'm not sure which ones I need and which I don't. My app doesn't have any outside dependencies, and it really just uses pretty normal C and OpenMP includes, so I'm guessing if it has issues with just that I will probably need them all (unless you have reason to believe otherwise). > > > > Thanks again! > > > > -John > > > > On Wed, Jan 14, 2015 at 6:58 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > Hi John, > > > > Yea I can do this. I should be able to get to it sometime this week, at worst early next week. Do you want all of the dbg packages installed, or do you have a specific list you want? > > > > Ben > > > > > On Jan 14, 2015, at 11:21 AM, John Tramm <jtramm(a)mcs.anl.gov> wrote: > > > > > > Ben, > > > > > > I tried running through the CLI mode, and had the same problems. It would be really great if you could install the required packages, as Vtune offers a lot in terms of performance analysis that can't be done any other way on the MIC. I understand the desire to keep the install bloat to a minimum, but Vtune is pretty essential to working with the MIC. I'm really in the dark as to what's going on with my application without being able to use Vtune, as even simple things like grpof don't work on there. > > > > > > Here's what I'm getting: > > > > > > jtramm@lucie:~/SimpleMOC-kernel/src/cpu$ export INTEL_LICENSE_FILE=28518@ftsn2 > > > jtramm@lucie:~/SimpleMOC-kernel/src/cpu$ source /soft/compilers/intel/vtune_amplifier_xe/amplxe-vars.sh > > > Copyright (C) 2009-2014 Intel Corporation. All rights reserved. > > > Intel(R) VTune(TM) Amplifier XE 2015 (build 380310) > > > jtramm@lucie:~/SimpleMOC-kernel/src/cpu$ amplxe-cl -collect advanced-hotspots -target-system=mic-native:0 -- /home/jtramm/SimpleMOC-kernel/src/cpu/SimpleMOC-kernel > > > amplxe: Using target: mic-native:0 > > > amplxe: Collection started. To stop the collection, either press CTRL-C or enter from another console window: amplxe-cl -r /home/jtramm/SimpleMOC-kernel/src/cpu/r001ah -command stop. > > > amplxe: Collection stopped. > > > amplxe: Using result path `/home/jtramm/SimpleMOC-kernel/src/cpu/r001ah' > > > amplxe: Executing actions 16 % Resolving module symbols > > > amplxe: Locating file `/usr/lib/debug/lib/modules/2.6.38.8+mpss3.4.2/vmlinux' on the remote system > > > amplxe: Locating file `/boot/vmlinuz-2.6.38.8+mpss3.4.2' on the remote system > > > amplxe: Executing actions 16 % Resolving information for `libc-2.14.90.so' > > > amplxe: Warning: Cannot locate debugging symbols for file `/tmp/amplxe-tmp-jtramm/modules.mic-native_0/libc-2.14.90.so/45397051311f95599054edd6ac8d616d/libc-2.14.90.so'. > > > amplxe: Locating file `vmlinux-2.6.38.8+mpss3.4.2' on the remote system > > > amplxe: Locating file `/boot/vmlinuz' on the remote system > > > amplxe: Executing actions 16 % Resolving information for `ld-2.14.90.so' > > > amplxe: Warning: Cannot locate debugging symbols for file `/tmp/amplxe-tmp-jtramm/modules.mic-native_0/ld-2.14.90.so/5152c3600c8161c8d9c8d152b5ad591e/ld-2.14.90.so'. > > > amplxe: Executing actions 17 % Resolving information for `ld-2.14.90.so' > > > amplxe: Locating file `intel_micveth.ko' on the remote system > > > amplxe: Warning: Cannot locate file `/usr/lib/debug/lib/modules/2.6.38.8+mpss3.4.2/vmlinux'. > > > amplxe: Executing actions 18 % Resolving information for dangling locations > > > amplxe: Warning: Cannot locate file `intel_micveth.ko'. > > > amplxe: Executing actions 18 % Resolving information for `libgnutls.so.26.22.0' > > > amplxe: Warning: Cannot locate debugging symbols for file `/tmp/amplxe-tmp-jtramm/modules.mic-native_0/libgnutls.so.26.22.0/9d99eae1cacffaa3f20e5a2312f69282/libgnutls.so.26.22.0'. > > > amplxe: Executing actions 18 % Resolving information for `libgcrypt.so.11.7.0' > > > amplxe: Warning: Cannot locate debugging symbols for file `/tmp/amplxe-tmp-jtramm/modules.mic-native_0/libgcrypt.so.11.7.0/587958b33eac1b807e852f640262da62/libgcrypt.so.11.7.0'. > > > amplxe: Executing actions 19 % Resolving information for `libnss_ldap-2.12.2.so > > > amplxe: Warning: Cannot locate debugging symbols for file `/tmp/amplxe-tmp-jtramm/modules.mic-native_0/libnss_ldap-2.12.2.so/0d6aa4bfd689f60ef5b0b94edf8d3ed1/libnss_ldap-2.12.2.so'. > > > amplxe: Executing actions 19 % Resolving information for `sshd' > > > amplxe: Warning: Cannot locate debugging symbols for file `/tmp/amplxe-tmp-jtramm/modules.mic-native_0/sshd/4354ab1d5cd53de91c85d8abe4000899/sshd'. > > > amplxe: Executing actions 19 % Resolving information for `libcrypto.so.1.0.0' > > > amplxe: Warning: Cannot locate debugging symbols for file `/tmp/amplxe-tmp-jtramm/modules.mic-native_0/libcrypto.so.1.0.0/2fc14f872d9ca6c49501676046029554/libcrypto.so.1.0.0'. > > > amplxe: Executing actions 20 % Resolving information for `libtasn1.so.3.1.13' > > > amplxe: Warning: Cannot locate debugging symbols for file `/tmp/amplxe-tmp-jtramm/modules.mic-native_0/libtasn1.so.3.1.13/0ba82903ca399644d103aed874e3ba7c/libtasn1.so.3.1.13'. > > > amplxe: Executing actions 50 % Generating a report > > > > > > General Exploration Metrics > > > --------------------------- > > > Parameter r001ah > > > -------------------- --------- > > > CPU Time 0.300 > > > Clockticks 372000558 > > > Instructions Retired 95333238 > > > CPI Rate 3.902 > > > Cache Usage 0.0 > > > Vectorization Usage 0.0 > > > TLB Usage 0.0 > > > > > > Collection and Platform Info > > > ---------------------------- > > > Parameter r001ah > > > ------------------------ ----------------------------------------------------------------- > > > Application Command Line /home/jtramm/SimpleMOC-kernel/src/cpu/SimpleMOC-kernel > > > User Name jtramm > > > Operating System Intel MIC Platform Software Stack (Built by Poky 7.0) 3.4.2 \n \l > > > Computer Name lucie-mic0.ftm.alcf.anl.gov > > > Result Size 2272316 > > > > > > CPU > > > --- > > > Parameter r001ah > > > ----------------- -------------------------- > > > Name Intel Xeon Phi coprocessor > > > Frequency 1238000000 > > > Logical CPU Count 244 > > > > > > Summary > > > ------- > > > Elapsed Time: 0.392 > > > CPU Time: 0.300 > > > Average CPU Usage: 0.548 > > > CPI Rate: 3.902 > > > > > > Event summary > > > ------------- > > > Hardware Event Type Hardware Event Count:Self Hardware Event Sample Count:Self Events Per Sample > > > --------------------- ------------------------- -------------------------------- ----------------- > > > CPU_CLK_UNHALTED 372000558 186 2000003 > > > INSTRUCTIONS_EXECUTED 95333238 77 1238094 > > > amplxe: Executing actions 100 % done > > > > > > > > > Thanks for your help! > > > > > > -John > > > > > > On Wed, Jan 7, 2015 at 2:44 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > > I have little to no experience with VTune. However looking at the screen cap the kernel modules referenced can be found on the MIC at: > > > > > > /lib/modules/2.6.38.8+mpss3.4.2/extra/vnet/intel_micveth.ko > > > /sep3.15/sep3_15-k1om-2.6.38.8+mpss3.4.2smp.ko > > > > > > If I'd have to guess, the debugging symbols are missing because I don't have any of the k1om debug packages installed on the MICs themselves, i.e. "dbg" RPMs from http://registrationcenter.intel.com/irc_nas/5017/mpss-3.4.2-k1om.tar. I try to keep the packages installed directly on the MICs to a minimum as they appear to sit in a RAM disk. > > > > > > Perhaps try running with CLI version of Vtune, amplxe-cl to see if it outputs anymore useful information? > > > > > > Example: > > > > > > export INTEL_LICENSE_FILE=28518@ftsn2 > > > source /soft/compilers/intel/vtune_amplifier_xe/amplxe-vars.sh > > > amplxe-cl -collect advanced-hotspots -target-system=mic-native ls -CF /usr/bin > > > > > > Ben > > > > > > > > > > On Jan 7, 2015, at 1:07 PM, John Tramm <jtramm(a)mcs.anl.gov> wrote: > > > > > > > > Ben, > > > > > > > > Thanks for re-deploying - it definitely seems to have helped. I'm now able to run the vtune GUI. > > > > > > > > However, now I'm running into a new issue. When I run vtune from the host, in order to analyze an app run on the device, it doesn't appear to be launching successfully (i.e., nothing is output to the terminal, and the resulting vtune analysis says the code only ran for a fraction of a second, when it should be at least several seconds on the MIC). > > > > > > > > It also outputs this set of errors when running my project: > > > > > > > > <Screen Shot 2015-01-07 at 12.56.50 PM.png> > > > > > > > > > > > > Can you think of anything that might cause this? > > > > > > > > Thanks again so much for your help! > > > > > > > > -John Tramm > > > > > > > > > > > > On Wed, Jan 7, 2015 at 12:47 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > > > Hi John, > > > > > > > > The RHEL6 image was missing libXxf86vm, I've re-deployed the three servers (Ruth, Lucie, Jenny) with libXxf86vm installed. I've also updated "sep", which is the sampling kernel module for both the MIC and the host. I doesn't look like Intel included the Power sampling kernel module with Vtune Amplifier XE 2015 however, ie. /soft/compilers/intel/vtune_amplifier_xe_2013/powerdk/src. > > > > > > > > Let me know if you run into any more problems. > > > > > > > > Ben > > > > > > > > > On Jan 6, 2015, at 2:50 PM, John Tramm <jtramm(a)mcs.anl.gov> wrote: > > > > > > > > > > Ben, > > > > > > > > > > I'm trying to use vtune to analyze performance of my application on the mic. I can't seem to get Vtune to run on the host node (either lucie or jenny). I keep getting the following error: > > > > > > > > > > amplxe-gui: error while loading shared libraries: libXxf86vm.so.1: cannot open shared object file: No such file or directory > > > > > > > > > > There seems to be a shared library file missing somewhere that I can't seem to find. Perhaps I need to add something to my linker path? > > > > > > > > > > Thanks for your help! > > > > > > > > > > -John > > > > > > > > > > > > > > > > > > > > On Tue, Jan 6, 2015 at 10:35 AM, John Tramm <jtramm(a)mcs.anl.gov> wrote: > > > > > Allen, > > > > > > > > > > Yes, everything seems to be working great now. Thanks! > > > > > > > > > > -John > > > > > > > > > > On Mon, Jan 5, 2015 at 2:26 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > > > > > Hi John, > > > > > > > > > > This should be fixed. Let me know if you run into further issues. > > > > > > > > > > Ben > > > > > > > > > > > On Jan 5, 2015, at 11:53 AM, John Tramm <jtramm(a)mcs.anl.gov> wrote: > > > > > > > > > > > > Greetings, > > > > > > > > > > > > I'm trying to use the JLSE MIC nodes, but I can't seem to get my home directory to show up on mic0 itself. It is available on the host, but it doesn't seem to be able to connect on the MIC. I recall a few months ago I was able to access my home dir on the mic. > > > > > > > > > > > > Do I need to run anything to get access to the home dir? > > > > > > > > > > > > Thanks! > > > > > > > > > > > > -John Tramm > > > > > > _______________________________________________ > > > > > > Admins mailing list > > > > > > Admins(a)lists.jlse.anl.gov > > > > > > https://lists.jlse.anl.gov/mailman/listinfo/admins > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >

1 0

Re: [JLSE-Admins] JLSE MIC home directory
by John Tramm 20 Jan '15

20 Jan '15

Hey Ben, I'm trying to compile my code, but I seem to be getting license errors when trying to use the intel compiler (I'm using the same environment that had been working previously, I believe): jtramm@lucie:~/SimpleMOC-kernel/src/cpu$ make icc -std=gnu99 -mmic -O3 -ansi-alias -no-prec-div -DINTEL -fp-model fast=2 -openmp -DOPENMP -c main.c -o main.o Error: A license for CCompL is not available now (-97,121). A connection to the license server could not be made. You should make sure that your license daemon process is running: both an lmgrd process and an INTEL process should be running if your license limits you to a specified number of licenses in use at a time. Also, check to see if the wrong port@host or the wrong license file is being used, or if the port or hostname in the license file has changed. License file(s) used were (in this order): 1. Trusted Storage ** 2. 28518@ftsn2 ** 3. /soft/compilers/intel/composer_xe_2013_sp1.1.106/licenses ** 4. /opt/intel/licenses ** 5. /home/jtramm/intel/licenses ** 6. /soft/compilers/intel/composer_xe_2013_sp1.1.106/Licenses ** 7. /Users/Shared/Library/Application Support/Intel/Licenses ** 8. /soft/compilers/intel/composer_xe_2013_sp1.1.106/bin/intel64/*.lic Please visit http://software.intel.com/sites/support/ if you require technical assistance. icc: error #10052: could not checkout FLEXlm license make: *** [main.o] Error 1 On Fri, Jan 16, 2015 at 10:33 AM, John Tramm <jtramm(a)mcs.anl.gov> wrote: > Ben, > > That sounds like a good plan. Thanks for the help! > > -John > > On Thu, Jan 15, 2015 at 3:05 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> > wrote: > >> Hi John, >> >> Assuming /home/jtramm/SimpleMOC-kernel/src/cpu/SimpleMOC-kernel is the >> actual app you're trying to profile, I can start with the packages >> referenced in the "Cannot locate debugging symbols" below. >> >> Ben >> >> > On Jan 14, 2015, at 8:15 PM, John Tramm <jtramm(a)mcs.anl.gov> wrote: >> > >> > Hey Ben, >> > >> > Thanks for doing this! I would ideally like all of the packages >> installed, as I'm not sure which ones I need and which I don't. My app >> doesn't have any outside dependencies, and it really just uses pretty >> normal C and OpenMP includes, so I'm guessing if it has issues with just >> that I will probably need them all (unless you have reason to believe >> otherwise). >> > >> > Thanks again! >> > >> > -John >> > >> > On Wed, Jan 14, 2015 at 6:58 PM, Allen, Benjamin S. < >> bsallen(a)alcf.anl.gov> wrote: >> > Hi John, >> > >> > Yea I can do this. I should be able to get to it sometime this week, at >> worst early next week. Do you want all of the dbg packages installed, or do >> you have a specific list you want? >> > >> > Ben >> > >> > > On Jan 14, 2015, at 11:21 AM, John Tramm <jtramm(a)mcs.anl.gov> wrote: >> > > >> > > Ben, >> > > >> > > I tried running through the CLI mode, and had the same problems. It >> would be really great if you could install the required packages, as Vtune >> offers a lot in terms of performance analysis that can't be done any other >> way on the MIC. I understand the desire to keep the install bloat to a >> minimum, but Vtune is pretty essential to working with the MIC. I'm really >> in the dark as to what's going on with my application without being able to >> use Vtune, as even simple things like grpof don't work on there. >> > > >> > > Here's what I'm getting: >> > > >> > > jtramm@lucie:~/SimpleMOC-kernel/src/cpu$ export >> INTEL_LICENSE_FILE=28518@ftsn2 >> > > jtramm@lucie:~/SimpleMOC-kernel/src/cpu$ source >> /soft/compilers/intel/vtune_amplifier_xe/amplxe-vars.sh >> > > Copyright (C) 2009-2014 Intel Corporation. All rights reserved. >> > > Intel(R) VTune(TM) Amplifier XE 2015 (build 380310) >> > > jtramm@lucie:~/SimpleMOC-kernel/src/cpu$ amplxe-cl -collect >> advanced-hotspots -target-system=mic-native:0 -- >> /home/jtramm/SimpleMOC-kernel/src/cpu/SimpleMOC-kernel >> > > amplxe: Using target: mic-native:0 >> > > amplxe: Collection started. To stop the collection, either press >> CTRL-C or enter from another console window: amplxe-cl -r >> /home/jtramm/SimpleMOC-kernel/src/cpu/r001ah -command stop. >> > > amplxe: Collection stopped. >> > > amplxe: Using result path >> `/home/jtramm/SimpleMOC-kernel/src/cpu/r001ah' >> > > amplxe: Executing actions 16 % Resolving module symbols >> > > amplxe: Locating file `/usr/lib/debug/lib/modules/2.6.38.8+mpss3.4.2/vmlinux' >> on the remote system >> > > amplxe: Locating file `/boot/vmlinuz-2.6.38.8+mpss3.4.2' on the >> remote system >> > > amplxe: Executing actions 16 % Resolving information for ` >> libc-2.14.90.so' >> > > amplxe: Warning: Cannot locate debugging symbols for file >> `/tmp/amplxe-tmp-jtramm/modules.mic-native_0/ >> libc-2.14.90.so/45397051311f95599054edd6ac8d616d/libc-2.14.90.so'. >> > > amplxe: Locating file `vmlinux-2.6.38.8+mpss3.4.2' on the remote >> system >> > > amplxe: Locating file `/boot/vmlinuz' on the remote system >> > > amplxe: Executing actions 16 % Resolving information for ` >> ld-2.14.90.so' >> > > amplxe: Warning: Cannot locate debugging symbols for file >> `/tmp/amplxe-tmp-jtramm/modules.mic-native_0/ >> ld-2.14.90.so/5152c3600c8161c8d9c8d152b5ad591e/ld-2.14.90.so'. >> > > amplxe: Executing actions 17 % Resolving information for ` >> ld-2.14.90.so' >> > > amplxe: Locating file `intel_micveth.ko' on the remote system >> > > amplxe: Warning: Cannot locate file `/usr/lib/debug/lib/modules/ >> 2.6.38.8+mpss3.4.2/vmlinux'. >> > > amplxe: Executing actions 18 % Resolving information for dangling >> locations >> > > amplxe: Warning: Cannot locate file `intel_micveth.ko'. >> > > amplxe: Executing actions 18 % Resolving information for >> `libgnutls.so.26.22.0' >> > > amplxe: Warning: Cannot locate debugging symbols for file >> `/tmp/amplxe-tmp-jtramm/modules.mic-native_0/libgnutls.so.26.22.0/9d99eae1cacffaa3f20e5a2312f69282/libgnutls.so.26.22.0'. >> > > amplxe: Executing actions 18 % Resolving information for >> `libgcrypt.so.11.7.0' >> > > amplxe: Warning: Cannot locate debugging symbols for file >> `/tmp/amplxe-tmp-jtramm/modules.mic-native_0/libgcrypt.so.11.7.0/587958b33eac1b807e852f640262da62/libgcrypt.so.11.7.0'. >> > > amplxe: Executing actions 19 % Resolving information for ` >> libnss_ldap-2.12.2.so >> > > amplxe: Warning: Cannot locate debugging symbols for file >> `/tmp/amplxe-tmp-jtramm/modules.mic-native_0/ >> libnss_ldap-2.12.2.so/0d6aa4bfd689f60ef5b0b94edf8d3ed1/libnss_ldap-2.12.2.so >> '. >> > > amplxe: Executing actions 19 % Resolving information for `sshd' >> > > amplxe: Warning: Cannot locate debugging symbols for file >> `/tmp/amplxe-tmp-jtramm/modules.mic-native_0/sshd/4354ab1d5cd53de91c85d8abe4000899/sshd'. >> > > amplxe: Executing actions 19 % Resolving information for >> `libcrypto.so.1.0.0' >> > > amplxe: Warning: Cannot locate debugging symbols for file >> `/tmp/amplxe-tmp-jtramm/modules.mic-native_0/libcrypto.so.1.0.0/2fc14f872d9ca6c49501676046029554/libcrypto.so.1.0.0'. >> > > amplxe: Executing actions 20 % Resolving information for >> `libtasn1.so.3.1.13' >> > > amplxe: Warning: Cannot locate debugging symbols for file >> `/tmp/amplxe-tmp-jtramm/modules.mic-native_0/libtasn1.so.3.1.13/0ba82903ca399644d103aed874e3ba7c/libtasn1.so.3.1.13'. >> > > amplxe: Executing actions 50 % Generating a report >> > > >> > > General Exploration Metrics >> > > --------------------------- >> > > Parameter r001ah >> > > -------------------- --------- >> > > CPU Time 0.300 >> > > Clockticks 372000558 >> > > Instructions Retired 95333238 >> > > CPI Rate 3.902 >> > > Cache Usage 0.0 >> > > Vectorization Usage 0.0 >> > > TLB Usage 0.0 >> > > >> > > Collection and Platform Info >> > > ---------------------------- >> > > Parameter r001ah >> > > ------------------------ >> ----------------------------------------------------------------- >> > > Application Command Line >> /home/jtramm/SimpleMOC-kernel/src/cpu/SimpleMOC-kernel >> > > User Name jtramm >> > > Operating System Intel MIC Platform Software Stack (Built by >> Poky 7.0) 3.4.2 \n \l >> > > Computer Name lucie-mic0.ftm.alcf.anl.gov >> > > Result Size 2272316 >> > > >> > > CPU >> > > --- >> > > Parameter r001ah >> > > ----------------- -------------------------- >> > > Name Intel Xeon Phi coprocessor >> > > Frequency 1238000000 >> > > Logical CPU Count 244 >> > > >> > > Summary >> > > ------- >> > > Elapsed Time: 0.392 >> > > CPU Time: 0.300 >> > > Average CPU Usage: 0.548 >> > > CPI Rate: 3.902 >> > > >> > > Event summary >> > > ------------- >> > > Hardware Event Type Hardware Event Count:Self Hardware Event >> Sample Count:Self Events Per Sample >> > > --------------------- ------------------------- >> -------------------------------- ----------------- >> > > CPU_CLK_UNHALTED 372000558 >> 186 2000003 >> > > INSTRUCTIONS_EXECUTED 95333238 >> 77 1238094 >> > > amplxe: Executing actions 100 % done >> > > >> > > >> > > Thanks for your help! >> > > >> > > -John >> > > >> > > On Wed, Jan 7, 2015 at 2:44 PM, Allen, Benjamin S. < >> bsallen(a)alcf.anl.gov> wrote: >> > > I have little to no experience with VTune. However looking at the >> screen cap the kernel modules referenced can be found on the MIC at: >> > > >> > > /lib/modules/2.6.38.8+mpss3.4.2/extra/vnet/intel_micveth.ko >> > > /sep3.15/sep3_15-k1om-2.6.38.8+mpss3.4.2smp.ko >> > > >> > > If I'd have to guess, the debugging symbols are missing because I >> don't have any of the k1om debug packages installed on the MICs themselves, >> i.e. "dbg" RPMs from >> http://registrationcenter.intel.com/irc_nas/5017/mpss-3.4.2-k1om.tar. I >> try to keep the packages installed directly on the MICs to a minimum as >> they appear to sit in a RAM disk. >> > > >> > > Perhaps try running with CLI version of Vtune, amplxe-cl to see if it >> outputs anymore useful information? >> > > >> > > Example: >> > > >> > > export INTEL_LICENSE_FILE=28518@ftsn2 >> > > source /soft/compilers/intel/vtune_amplifier_xe/amplxe-vars.sh >> > > amplxe-cl -collect advanced-hotspots -target-system=mic-native ls -CF >> /usr/bin >> > > >> > > Ben >> > > >> > > >> > > > On Jan 7, 2015, at 1:07 PM, John Tramm <jtramm(a)mcs.anl.gov> wrote: >> > > > >> > > > Ben, >> > > > >> > > > Thanks for re-deploying - it definitely seems to have helped. I'm >> now able to run the vtune GUI. >> > > > >> > > > However, now I'm running into a new issue. When I run vtune from >> the host, in order to analyze an app run on the device, it doesn't appear >> to be launching successfully (i.e., nothing is output to the terminal, and >> the resulting vtune analysis says the code only ran for a fraction of a >> second, when it should be at least several seconds on the MIC). >> > > > >> > > > It also outputs this set of errors when running my project: >> > > > >> > > > <Screen Shot 2015-01-07 at 12.56.50 PM.png> >> > > > >> > > > >> > > > Can you think of anything that might cause this? >> > > > >> > > > Thanks again so much for your help! >> > > > >> > > > -John Tramm >> > > > >> > > > >> > > > On Wed, Jan 7, 2015 at 12:47 PM, Allen, Benjamin S. < >> bsallen(a)alcf.anl.gov> wrote: >> > > > Hi John, >> > > > >> > > > The RHEL6 image was missing libXxf86vm, I've re-deployed the three >> servers (Ruth, Lucie, Jenny) with libXxf86vm installed. I've also updated >> "sep", which is the sampling kernel module for both the MIC and the host. I >> doesn't look like Intel included the Power sampling kernel module with >> Vtune Amplifier XE 2015 however, ie. >> /soft/compilers/intel/vtune_amplifier_xe_2013/powerdk/src. >> > > > >> > > > Let me know if you run into any more problems. >> > > > >> > > > Ben >> > > > >> > > > > On Jan 6, 2015, at 2:50 PM, John Tramm <jtramm(a)mcs.anl.gov> >> wrote: >> > > > > >> > > > > Ben, >> > > > > >> > > > > I'm trying to use vtune to analyze performance of my application >> on the mic. I can't seem to get Vtune to run on the host node (either lucie >> or jenny). I keep getting the following error: >> > > > > >> > > > > amplxe-gui: error while loading shared libraries: >> libXxf86vm.so.1: cannot open shared object file: No such file or directory >> > > > > >> > > > > There seems to be a shared library file missing somewhere that I >> can't seem to find. Perhaps I need to add something to my linker path? >> > > > > >> > > > > Thanks for your help! >> > > > > >> > > > > -John >> > > > > >> > > > > >> > > > > >> > > > > On Tue, Jan 6, 2015 at 10:35 AM, John Tramm <jtramm(a)mcs.anl.gov> >> wrote: >> > > > > Allen, >> > > > > >> > > > > Yes, everything seems to be working great now. Thanks! >> > > > > >> > > > > -John >> > > > > >> > > > > On Mon, Jan 5, 2015 at 2:26 PM, Allen, Benjamin S. < >> bsallen(a)alcf.anl.gov> wrote: >> > > > > Hi John, >> > > > > >> > > > > This should be fixed. Let me know if you run into further issues. >> > > > > >> > > > > Ben >> > > > > >> > > > > > On Jan 5, 2015, at 11:53 AM, John Tramm <jtramm(a)mcs.anl.gov> >> wrote: >> > > > > > >> > > > > > Greetings, >> > > > > > >> > > > > > I'm trying to use the JLSE MIC nodes, but I can't seem to get >> my home directory to show up on mic0 itself. It is available on the host, >> but it doesn't seem to be able to connect on the MIC. I recall a few months >> ago I was able to access my home dir on the mic. >> > > > > > >> > > > > > Do I need to run anything to get access to the home dir? >> > > > > > >> > > > > > Thanks! >> > > > > > >> > > > > > -John Tramm >> > > > > > _______________________________________________ >> > > > > > Admins mailing list >> > > > > > Admins(a)lists.jlse.anl.gov >> > > > > > https://lists.jlse.anl.gov/mailman/listinfo/admins >> > > > > >> > > > > >> > > > > >> > > > >> > > > >> > > >> > > >> > >> > >> >> >

1 0

[JLSE-Admins] What did we agree on?
by Kalyan Kumaran 20 Jan '15

20 Jan '15

Not all users on the JLSE users list are Argonne employees, correct? They could be collaborators from other institutions. Intel likes to set up monthly calls with us. I’m trying to arrange something similar with other vendors too. If these vendors present NDA information how do we handle it? I send an announcement to the users list saying that NDA information is going to be presented and please make sure you have an NDA with xxxx company? Thanks, Kumar

1 0

Re: [JLSE-Admins] The progress of stencil-
by Kazutomo Yoshii 19 Jan '15

19 Jan '15

Hi Ben, Thanks! (also thank you for helping IPMI!) We're currently interested in an explicit mapping, instead of transparent. Any setting is fine on transparent_hugepage/enabled as long as no other memory-intensive apps are running. - kaz On 01/19/2015 10:41 AM, Allen, Benjamin S. wrote: > Hi Kaz, > > I've pushed these changes to thing00-02,04-07 in JLSE: > > Added to /etc/fstab: > > none /hugetlbfs hugetlbfs mode=1770,gid=501 0 0 > > Created /etc/sysctl.d/02-hugetlbfs.conf: > > vm.hugetlb_shm_group = 501 > vm.nr_hugepages = 4096 > > I did not implement: $ echo always > /sys/kernel/mm/transparent_hugepage/enabled, let me know if this is indeed needed. > > Ben > > >> On Jan 12, 2015, at 4:35 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: >> >> Hi Ben, >> >> Thank you! >> >> 4096 would be enough for now. >> If you want to set a larger value, of course, no problem :-) >> >> - kaz >> >> On 01/12/2015 04:30 PM, Allen, Benjamin S. wrote: >>> Hi Kaz, >>> >>> Started working on this. I'm going to set the two sysctl values by default on the JLSE RHEL7 image, and add a noauto,user entry in fstab. This will let users mount and unmount it as will. >>> >>>> sysctl vm.nr_hugepages=Y >>> This takes a numerical value not "Y": >>> >>> /proc/sys/vm/nr_hugepages indicates the current number of "persistent" huge >>> pages in the kernel's huge page pool. "Persistent" huge pages will be >>> returned to the huge page pool when freed by a task. A user with root >>> privileges can dynamically allocate more or free some persistent huge pages >>> by increasing or decreasing the value of 'nr_hugepages'. >>> >>> Please advise on what it should be set to. >>> >>> Thanks, >>> >>> Ben >>> >>> >>>> On Jan 9, 2015, at 3:38 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: >>>> >>>> Hi Ben, >>>> >>>> Are you able to give a sudo access on Haswell nodes to Xiamin, Junchao (Pavan's group), >>>> or create a separate hugetlbfs mount for them (see the command below)? >>>> >>>> They may need an isolated environment for performance benchmark. >>>> At this point, it seems very few people uses haswell, probably we don't need any action >>>> for them. >>>> >>>> - kaz >>>> >>>> -------- Forwarded Message -------- >>>> Subject: Re: The progress of stencil- >>>> Date: Fri, 9 Jan 2015 14:17:00 -0600 >>>> From: woodslong <woodslong(a)gmail.com> >>>> To: Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> >>>> CC: Pavan <balaji(a)anl.gov>, Junchao Zhang <junchao.zhang(a)gmail.com>, Shigang Li <shigangli.cs(a)gmail.com>, Yunquan Zhang <yunquan.cas(a)gmail.com> >>>> >>>> Hi, Kaz >>>> The necessary operations to use huge page are shown below(most copy from your email): >>>> >>>> >>>> $ sysctl vm.hugetlb_shm_group=X # X >>>> is the group id for sharing, be sure that my account is the >>>> this group >>>> >>>> $ sysctl vm.nr_hugepages=Y >>>> $ echo always > /sys/kernel/mm/transparent_hugepage/enabled (maybe not needed) >>>> $ mkdir -p /mnt/hugetlbfs >>>> >>>> $ mount -t hugetlbfs none >>>> /mnt/hugetlbfs -o mode=1770,gid=X >>>> >>>> >>>> Thanks! >>>> Xiaomin Zhu >>>> >>>> >>>> >>>> >>>> Original Message >>>> >>>> >>>> >>>> Sender: woodslong<woodslong(a)gmail.com> >>>> Recipient: Pavan<balaji(a)anl.gov> >>>> Cc: Kazutomo Yoshii<kazutomo(a)mcs.anl.gov>; Junchao Zhang<junchao.zhang(a)gmail.com>; Shigang Li<shigangli.cs(a)gmail.com>; Yunquan Zhang<yunquan.cas(a)gmail.com> >>>> Date: Friday, Jan 9, 2015 10:03 >>>> Subject: Re: The progress of stencil >>>> >>>> Hi, all >>>> Just now, I discussed with Junchao. >>>> We think Cache miss and page fault are the two factors. I will try to verify that. >>>> The less floating computation for MPI3 is not accurate. Please just ignore it. >>>> >>>> Thanks, >>>> >>>> Xiaomin Zhu >>>> >>>> >>>> >>>> >>>> Original Message >>>> >>>> >>>> >>>> Sender: woodslong<woodslong(a)gmail.com> >>>> Recipient: Pavan<balaji(a)anl.gov> >>>> Cc: Kazutomo Yoshii<kazutomo(a)mcs.anl.gov>; Junchao Zhang<junchao.zhang(a)gmail.com>; Shigang Li<shigangli.cs(a)gmail.com>; Yunquan Zhang<yunquan.cas(a)gmail.com> >>>> Date: Friday, Jan 9, 2015 09:35 >>>> Subject: Re: The progress of stencil >>>> >>>> OK! Pavan >>>> I suddenly remembered another issue which affects the performance, that is the cache miss rate. >>>> >>>> As the MPI3 version removes the ghost area then changes the data structure of the matrix, as well as the data layout. According with the data access pattern of a grid update, this cause more cache miss if the matrix size happens to be power of 2. In the former profiling, I used 4096*4096 matrix, so the MPI3 version has more cache miss. >>>> >>>> Just now, I did another profiling with matrix 5034*5034, and MPI3 shows a better performance (only between MPI1 without huge page and MPI3 without huge page, noncontig), as MPI3 version has less floating point computations and the cache miss should be nearly the same with this random matrix size. >>>> >>>> I am sorry I forgot that. I will add more profiling which contains all the 6 versions. If all the mpi3 version showed a better performance, it will be good. >>>> >>>> Xiaomin Zhu >>>> >>>> >>>> >>>> >>>> Original Message >>>> >>>> >>>> >>>> Sender: Pavan" <balaji(a)anl.gov> >>>> Recipient: woodslong<woodslong(a)gmail.com> >>>> Cc: Kazutomo" <kazutomo(a)mcs.anl.gov>; Junchao Zhang<junchao.zhang(a)gmail.com>; Shigang Li<shigangli.cs(a)gmail.com>; Yunquan Zhang<yunquan.cas(a)gmail.com> >>>> Date: Friday, Jan 9, 2015 08:09 >>>> Subject: Re: The progress of stencil >>>> >>>> >>>> We should write the paper as-is. We can say that we identified some issues, but not all. That's OK. But we can't say that it's impossible to fix, unless we know what the reason is and can demonstrate that it's impossible to fix. >>>> >>>> Sent from my iPad >>>> >>>> On Jan 9, 2015, at 6:56 PM, woodslong <woodslong(a)gmail.com> wrote: >>>> >>>>> OK, Pavan >>>>> >>>>> Then how do you think about the future work? >>>>> >>>>> >>>>> I think if the reason is shared memory mechanism itself, it will be difficult to fix. Do you have any suggestions on the future work? >>>>> >>>>> >>>>> Xiaomin Zhu >>>>> >>>>> Original Message >>>>> >>>>> >>>>> >>>>> Sender: Pavan" <balaji(a)anl.gov> >>>>> Recipient: Xiaomin Zhu<woodslong(a)gmail.com> >>>>> Cc: Pavan" <balaji(a)anl.gov>; Kazutomo" <kazutomo(a)mcs.anl.gov>; Junchao Zhang<junchao.zhang(a)gmail.com>; Shigang Li<shigangli.cs(a)gmail.com>; Yunquan Zhang<yunquan.cas(a)gmail.com> >>>>> Date: Friday, Jan 9, 2015 00:55 >>>>> Subject: Re: The progress of stencil >>>>> >>>>>> On Jan 9, 2015, at 7:45 AM, woodslong <woodslong(a)gmail.com >>>>>> wrote: >>>>>> >>>>>> (3-4) Our aim is to make the shared memory have the same performance with malloc, but it seems impossible as the related overhead is inevitable. >>>>> I'm not sure how you came to this conclusion. >>>>> >>>>> So far, we identified two problems: (1) compiler aliasing, and (2) huge pages. You are saying that there is at least one more reason that's causing performance degradation that we haven't identified yet. But that doesn't mean it's *impossible* to fix. >>>>> >>>>> -- Pavan >>>>> >>>>> -- >>>>> Pavan Balaji ✉️ >>>>> >>>>> http://www.mcs.anl.gov/~balaji >>>>> >>>>> >>>>> >>>> _______________________________________________ >>>> Admins mailing list >>>> Admins(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/admins

1 0

Re: [JLSE-Admins] [JLSE-Users] Haswell-EP E5-2699v3 Nodes Available
by Allen, Benjamin S. 19 Jan '15

19 Jan '15

Hi Ray, We should be able to accommodate this. Please request a new JLSE resource via http://press3.mcs.anl.gov/jlse/projects/project-request-form/. Under Resources/Test Bed Needed, "Upgrade existing E5-2603v3 JLSE machine "snell" with E5-2620v3 CPUs add four more DIMMs to fully populate memory channels, cost will be $1610.52 via MNJ Tech in AMOS." I also suggest looking into the E5-2650L. It's actually the best flops/watt CPU in the entire line-up. Albeit it is quite a bit more expensive than the E5-2620v3. If your applications are memory bandwidth bound, I'd look at models that support DDR4-2133. Lastly, I'm not sure what vendor or what level/type of hardware you're looking for, but Intel has a really nice 4 node in 2RU system: http://www.intel.com/content/www/us/en/motherboards/server-motherboards/ser…. Thanks, Ben > On Jan 19, 2015, at 10:45 AM, Bair, Raymond A. <rbair(a)anl.gov> wrote: > > P.S. The aim is to determine the best price-performance for a 60-node > addition to Blues for reactor research. > > > > On 1/19/15, 10:38 AM, "Bair, Raymond A." <rbair(a)anl.gov> wrote: > >> We would like to compare the 6-core Haswell against our 8-core Sandy >> Bridge for a floating-point intensive benchmark. >> >> Ray >> >> >> >> >> >> On 1/19/15, 10:34 AM, "Allen, Benjamin S." <bsallen(a)alcf.anl.gov> wrote: >> >>> Hi Ray, >>> >>> Not specifically the E5-2620v3. JLSE has one machine with 2x E5-2603v3 >>> (6c, lower clock rate, and lower memory bandwidth), and 8 machines with >>> 2x E5-2699v3. What are you trying to test with the E5-2620v3? >>> >>> If needed we could upgrade the machine that has the E5-2603v3 CPUs to >>> E5-2620v3. Cost would be $843.72. >>> >>> Note, I'm pretty sure all 4-8 core Haswell-EP CPUs have a single memory >>> controller. >>> >>> Ben >>> >>>> On Jan 19, 2015, at 10:03 AM, Bair, Raymond A. <rbair(a)anl.gov> wrote: >>>> >>>> Hi Ben, >>>> >>>> Does anyone here have the 2620v3 Haswell with 6 cores and 1 memory >>>> controller on chip? >>>> >>>> Ray >>>> >>>> ----------------------------------------- >>>> Ray Bair >>>> Computing, Environment, and Life Sciences >>>> Argonne National Laboratory >>>> and the University of Chicago >>>> TCS Building 240, Room 4122 >>>> 9700 South Cass Avenue >>>> Argonne, IL 60439 >>>> email: rbair(at)anl.gov >>>> Phone: (630)252-5751 >>>> >>>> >>>> >>>> >>>> >>>> On 12/12/14, 2:18 PM, "Allen, Benjamin S." <bsallen(a)alcf.anl.gov> >>>> wrote: >>>> >>>>> All, >>>>> >>>>> We've added six systems that each have two Intel Haswell-EP 18c >>>>> E5-2699v3 >>>>> CPUs. These CPUs are interesting as they are the first Xeon based >>>>> system >>>>> that has greater than a teraflop of performance. >>>>> >>>>> Hostnames: thing{00,01,04,05,06,07}.ftm.alcf.anl.gov >>>>> >>>>> - SuperMicro SC747BTQ-R1K62B 1600W 4RU Chassis >>>>> - SuperMicro X10DRi Motherboard >>>>> - 2x Intel Xeon E5-2699v3 18c CPU >>>>> - RHEL7 (3.10.0-123.6.3.el7.x86_64) >>>>> >>>>> Note, currently these machines do not have an IB connection. We're >>>>> hoping >>>>> Mellanox will loan us new EDR hardware for these nodes. >>>>> >>>>> We're waiting on a replacement for a failed CPU on "thing02", and Kaz >>>>> has >>>>> "thing03" in Tinkerlab. >>>>> >>>>> Let me know if you run into any issues. >>>>> >>>>> Ben >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users(a)lists.jlse.anl.gov >>>>> https://lists.jlse.anl.gov/mailman/listinfo/users >>>> >>> >> >

1 0

Re: [JLSE-Admins] Project Request Form
by Allen, Benjamin S. 19 Jan '15

19 Jan '15

Hi Kaz, Quick update on this. I'm still working with LCF's networking folks to get JLSE's infrastructure IPMI interfaces off the same network as the nodes. Sorry for the delay. Ben > On Jan 12, 2015, at 11:24 AM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: > > Ben, > > Thank you for your help! > > I don't care much about the speed of hard drive. > Any drive would be fine as long as it works. > > - kaz > > On 01/09/2015 02:43 PM, Allen, Benjamin S. wrote: >> Hi Kaz, >> >> Yea its possible. I still need to move my infrastructure off the JLSE IPMI network before I give the login nodes access to it. This is to isolate production JLSE IPMI interfaces from user access. I hope to figure that out next week. >> >> The Haswell nodes currently don't have any storage installed, so we'll either need to order something or borrow. >> >> Ben >> >> >>> On Jan 9, 2015, at 1:58 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: >>> >>> Ben: >>> >>> Adam needs the followings. >>> >>> - IPMI access to change BIOS and reboot the machine >>> - a dedicated disk partition to boot a custom image ( e.g., centoos plus >>> a custom Linux kernel) >>> >>> Is this kind of stuff possible in the JLSE machine space? >>> >>> - kaz >>> >>> On 01/09/2015 12:44 PM, Thakur, Rajeev wrote: >>>>> E5-2699v3 (Haswell) nodes (http://ark.intel.com/products/81061/Intel-Xeon-Processor-E5-2699-v3-45M-Cac…) In particular, we would like isolated and IPMI access to change BIOS. Such access would allow us to explore the per-process strategies for thermal load balancing, and the subsequently necessary application optimization. >>>> Adam, >>>> You should be able to do what you want on the Haswell nodes we have in JLSE. Kaz can help you with that. >>>> >>>> Rajeev >>>> >>>> >>>> On Dec 23, 2014, at 12:59 PM, Adam Hammouda <ahammouda(a)cs.uchicago.edu> wrote: >>>> >>>>> From: Adam Hammouda <ahammouda(a)cs.uchicago.edu> >>>>> Energy Savings and Performance Optimizations With Pliable Algorithms on Intel’s E5-2699v3 (Haswell) Nodes >>>>> Adam Hammouda, Andrew Siegel, Pete Beckman >>>>> MCS >>>>> The goal of our research is to study operating system design decisions for next generation HPC architectures. In particular, we aim to study the impact which those decisions have on application performance characteristics. Further, we aim to understand how these decisions and performance characteristics relate to the thermal stability and energy effeciency of the architecture in question. Broad questions of interest to our research include the following: >>>>> >>>>> • Is there a layer of the HPC runtime environment best equipped for mitigating thermal load imbalances (i.e. OS, runtime system, or application)? >>>>> • If it is more optimal to share these responsibilities between layers, what is the optimal division of labor? >>>>> • What are the best metrics for the above referenced optimality? >>>>> • With such metrics, what are acceptable tradeoffs between performance and energy? >>>>> • What are generalizable strategies for the application layer to handle thermal load imbalances? What is a generalizable strategy for the application layer to receive and share information with the runtime system? >>>>> >>>>> In an earlier work, we reformulated an algorithm class of bulk-synchronous computation to account for nonuniformities in process execution rates. The challenge of energy and thermal efficiency presented by next generation machines is precisely what motivated our algorithmic developments in this previous work. The COOLR project presents an opportunity to pursue this line of inquiry even further by exploring the precise characteristics of these anticipated nonuniformities. >>>>> To this end we have already begun our explorations of the Intel Xeon E5-2670 machine available to us. We are exploring the effects of per-socket controls over cpu frequency, and pstate setings, and we have developed metrics to quantify the tradeoffs in performance and thermal load balance. However, our algorithmic work in Hammouda and Siegel 2013 was designed to exploit per-process nonuniformities in execution rates, and we therefore could benefit greatly from a machine which provides access to these controls. >>>>> COOLR, CESAR >>>>> Northwestern >>>>> Adam Hammouda, Kaicheng Zhang (Northwestern) >>>>> E5-2699v3 (Haswell) nodes (http://ark.intel.com/products/81061/Intel-Xeon-Processor-E5-2699-v3-45M-Cac…) In particular, we would like isolated and IPMI access to change BIOS. Such access would allow us to explore the per-process strategies for thermal load balancing, and the subsequently necessary application optimization. >>>>> [Schedule (when access is needed and for how long):] >>>>> >>>>> >>>>> -- >>>>> This e-mail was sent from a contact form on The Joint Laboratory for System Evaluation (http://press3.mcs.anl.gov/jlse) >>>>> >>>>> _______________________________________________ >>>>> Admins mailing list >>>>> Admins(a)lists.jlse.anl.gov >>>>> https://lists.jlse.anl.gov/mailman/listinfo/admins >>>> _______________________________________________ >>>> Admins mailing list >>>> Admins(a)lists.jlse.anl.gov >>>> https://lists.jlse.anl.gov/mailman/listinfo/admins >>> _______________________________________________ >>> Admins mailing list >>> Admins(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/admins >

1 0

Re: [JLSE-Admins] The progress of stencil-
by Allen, Benjamin S. 19 Jan '15

19 Jan '15

Hi Kaz, I've pushed these changes to thing00-02,04-07 in JLSE: Added to /etc/fstab: none /hugetlbfs hugetlbfs mode=1770,gid=501 0 0 Created /etc/sysctl.d/02-hugetlbfs.conf: vm.hugetlb_shm_group = 501 vm.nr_hugepages = 4096 I did not implement: $ echo always > /sys/kernel/mm/transparent_hugepage/enabled, let me know if this is indeed needed. Ben > On Jan 12, 2015, at 4:35 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: > > Hi Ben, > > Thank you! > > 4096 would be enough for now. > If you want to set a larger value, of course, no problem :-) > > - kaz > > On 01/12/2015 04:30 PM, Allen, Benjamin S. wrote: >> Hi Kaz, >> >> Started working on this. I'm going to set the two sysctl values by default on the JLSE RHEL7 image, and add a noauto,user entry in fstab. This will let users mount and unmount it as will. >> >>> sysctl vm.nr_hugepages=Y >> This takes a numerical value not "Y": >> >> /proc/sys/vm/nr_hugepages indicates the current number of "persistent" huge >> pages in the kernel's huge page pool. "Persistent" huge pages will be >> returned to the huge page pool when freed by a task. A user with root >> privileges can dynamically allocate more or free some persistent huge pages >> by increasing or decreasing the value of 'nr_hugepages'. >> >> Please advise on what it should be set to. >> >> Thanks, >> >> Ben >> >> >>> On Jan 9, 2015, at 3:38 PM, Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> wrote: >>> >>> Hi Ben, >>> >>> Are you able to give a sudo access on Haswell nodes to Xiamin, Junchao (Pavan's group), >>> or create a separate hugetlbfs mount for them (see the command below)? >>> >>> They may need an isolated environment for performance benchmark. >>> At this point, it seems very few people uses haswell, probably we don't need any action >>> for them. >>> >>> - kaz >>> >>> -------- Forwarded Message -------- >>> Subject: Re: The progress of stencil- >>> Date: Fri, 9 Jan 2015 14:17:00 -0600 >>> From: woodslong <woodslong(a)gmail.com> >>> To: Kazutomo Yoshii <kazutomo(a)mcs.anl.gov> >>> CC: Pavan <balaji(a)anl.gov>, Junchao Zhang <junchao.zhang(a)gmail.com>, Shigang Li <shigangli.cs(a)gmail.com>, Yunquan Zhang <yunquan.cas(a)gmail.com> >>> >>> Hi, Kaz >>> The necessary operations to use huge page are shown below(most copy from your email): >>> >>> >>> $ sysctl vm.hugetlb_shm_group=X # X >>> is the group id for sharing, be sure that my account is the >>> this group >>> >>> $ sysctl vm.nr_hugepages=Y >>> $ echo always > /sys/kernel/mm/transparent_hugepage/enabled (maybe not needed) >>> $ mkdir -p /mnt/hugetlbfs >>> >>> $ mount -t hugetlbfs none >>> /mnt/hugetlbfs -o mode=1770,gid=X >>> >>> >>> Thanks! >>> Xiaomin Zhu >>> >>> >>> >>> >>> Original Message >>> >>> >>> >>> Sender: woodslong<woodslong(a)gmail.com> >>> Recipient: Pavan<balaji(a)anl.gov> >>> Cc: Kazutomo Yoshii<kazutomo(a)mcs.anl.gov>; Junchao Zhang<junchao.zhang(a)gmail.com>; Shigang Li<shigangli.cs(a)gmail.com>; Yunquan Zhang<yunquan.cas(a)gmail.com> >>> Date: Friday, Jan 9, 2015 10:03 >>> Subject: Re: The progress of stencil >>> >>> Hi, all >>> Just now, I discussed with Junchao. >>> We think Cache miss and page fault are the two factors. I will try to verify that. >>> The less floating computation for MPI3 is not accurate. Please just ignore it. >>> >>> Thanks, >>> >>> Xiaomin Zhu >>> >>> >>> >>> >>> Original Message >>> >>> >>> >>> Sender: woodslong<woodslong(a)gmail.com> >>> Recipient: Pavan<balaji(a)anl.gov> >>> Cc: Kazutomo Yoshii<kazutomo(a)mcs.anl.gov>; Junchao Zhang<junchao.zhang(a)gmail.com>; Shigang Li<shigangli.cs(a)gmail.com>; Yunquan Zhang<yunquan.cas(a)gmail.com> >>> Date: Friday, Jan 9, 2015 09:35 >>> Subject: Re: The progress of stencil >>> >>> OK! Pavan >>> I suddenly remembered another issue which affects the performance, that is the cache miss rate. >>> >>> As the MPI3 version removes the ghost area then changes the data structure of the matrix, as well as the data layout. According with the data access pattern of a grid update, this cause more cache miss if the matrix size happens to be power of 2. In the former profiling, I used 4096*4096 matrix, so the MPI3 version has more cache miss. >>> >>> Just now, I did another profiling with matrix 5034*5034, and MPI3 shows a better performance (only between MPI1 without huge page and MPI3 without huge page, noncontig), as MPI3 version has less floating point computations and the cache miss should be nearly the same with this random matrix size. >>> >>> I am sorry I forgot that. I will add more profiling which contains all the 6 versions. If all the mpi3 version showed a better performance, it will be good. >>> >>> Xiaomin Zhu >>> >>> >>> >>> >>> Original Message >>> >>> >>> >>> Sender: Pavan" <balaji(a)anl.gov> >>> Recipient: woodslong<woodslong(a)gmail.com> >>> Cc: Kazutomo" <kazutomo(a)mcs.anl.gov>; Junchao Zhang<junchao.zhang(a)gmail.com>; Shigang Li<shigangli.cs(a)gmail.com>; Yunquan Zhang<yunquan.cas(a)gmail.com> >>> Date: Friday, Jan 9, 2015 08:09 >>> Subject: Re: The progress of stencil >>> >>> >>> We should write the paper as-is. We can say that we identified some issues, but not all. That's OK. But we can't say that it's impossible to fix, unless we know what the reason is and can demonstrate that it's impossible to fix. >>> >>> Sent from my iPad >>> >>> On Jan 9, 2015, at 6:56 PM, woodslong <woodslong(a)gmail.com> wrote: >>> >>>> OK, Pavan >>>> >>>> Then how do you think about the future work? >>>> >>>> >>>> I think if the reason is shared memory mechanism itself, it will be difficult to fix. Do you have any suggestions on the future work? >>>> >>>> >>>> Xiaomin Zhu >>>> >>>> Original Message >>>> >>>> >>>> >>>> Sender: Pavan" <balaji(a)anl.gov> >>>> Recipient: Xiaomin Zhu<woodslong(a)gmail.com> >>>> Cc: Pavan" <balaji(a)anl.gov>; Kazutomo" <kazutomo(a)mcs.anl.gov>; Junchao Zhang<junchao.zhang(a)gmail.com>; Shigang Li<shigangli.cs(a)gmail.com>; Yunquan Zhang<yunquan.cas(a)gmail.com> >>>> Date: Friday, Jan 9, 2015 00:55 >>>> Subject: Re: The progress of stencil >>>> >>>>> On Jan 9, 2015, at 7:45 AM, woodslong <woodslong(a)gmail.com >>>>> wrote: >>>>> >>>>> (3-4) Our aim is to make the shared memory have the same performance with malloc, but it seems impossible as the related overhead is inevitable. >>>> I'm not sure how you came to this conclusion. >>>> >>>> So far, we identified two problems: (1) compiler aliasing, and (2) huge pages. You are saying that there is at least one more reason that's causing performance degradation that we haven't identified yet. But that doesn't mean it's *impossible* to fix. >>>> >>>> -- Pavan >>>> >>>> -- >>>> Pavan Balaji ✉️ >>>> >>>> http://www.mcs.anl.gov/~balaji >>>> >>>> >>>> >>> >>> _______________________________________________ >>> Admins mailing list >>> Admins(a)lists.jlse.anl.gov >>> https://lists.jlse.anl.gov/mailman/listinfo/admins >

1 0

[JLSE-Admins] [Fail2Ban] SSH: banned 62.210.129.168 on
by Fail2Ban 18 Jan '15

18 Jan '15

The IP 62.210.129.168 has just been banned by Fail2Ban after 18 attempts against SSH. Latest lines containing IP:62.210.129.168 in /var/log/syslog/sshd.log Jan 18 22:43:15 ftlogin2 sshd[11324]: error: PAM: Authentication failure for root from 62-210-129-168.rev.poneytelecom.eu Jan 18 22:43:15 ftlogin2 sshd[11327]: Received disconnect from 62.210.129.168: 11: Bye Bye Jan 18 22:43:18 ftlogin2 sshd[11329]: error: PAM: Authentication failure for root from 62-210-129-168.rev.poneytelecom.eu Jan 18 22:43:18 ftlogin2 sshd[11332]: Received disconnect from 62.210.129.168: 11: Bye Bye Jan 18 22:43:19 ftlogin2 sshd[11334]: error: PAM: Authentication failure for root from 62-210-129-168.rev.poneytelecom.eu Jan 18 22:43:19 ftlogin2 sshd[11337]: Received disconnect from 62.210.129.168: 11: Bye Bye Jan 18 22:43:26 ftlogin2 sshd[11339]: error: PAM: Authentication failure for root from 62-210-129-168.rev.poneytelecom.eu Jan 18 22:43:26 ftlogin2 sshd[11342]: Received disconnect from 62.210.129.168: 11: Bye Bye Jan 18 22:43:28 ftlogin2 sshd[11344]: error: PAM: Authentication failure for root from 62-210-129-168.rev.poneytelecom.eu Jan 18 22:43:28 ftlogin2 sshd[11347]: Received disconnect from 62.210.129.168: 11: Bye Bye - Fail2ban on

1 0

[JLSE-Admins] [Fail2Ban] SSH: banned 115.231.222.42 on
by Fail2Ban 18 Jan '15

18 Jan '15

The IP 115.231.222.42 has just been banned by Fail2Ban after 18 attempts against SSH. Latest lines containing IP:115.231.222.42 in /var/log/syslog/sshd.log Jan 18 14:32:57 ftlogin1 sshd[7631]: Connection closed by 115.231.222.42 Jan 18 14:32:57 ftlogin1 sshd[7632]: Connection closed by 115.231.222.42 Jan 18 14:32:58 ftlogin1 sshd[7646]: Connection closed by 115.231.222.42 Jan 18 14:32:58 ftlogin1 sshd[7650]: Connection closed by 115.231.222.42 Jan 18 14:32:58 ftlogin1 sshd[7633]: Connection closed by 115.231.222.42 Jan 18 14:32:58 ftlogin1 sshd[7654]: Connection closed by 115.231.222.42 Jan 18 14:32:58 ftlogin1 sshd[7584]: error: PAM: Authentication failure for root from 115.231.222.42 Jan 18 14:32:59 ftlogin1 sshd[7595]: error: PAM: Authentication failure for root from 115.231.222.42 Jan 18 14:32:59 ftlogin1 sshd[7593]: Received disconnect from 115.231.222.42: 11: Bye Bye Jan 18 14:32:59 ftlogin1 sshd[7645]: Connection closed by 115.231.222.42 - Fail2ban on

1 0

[JLSE-Admins] [Fail2Ban] SSH: banned 62.210.82.189 on
by Fail2Ban 17 Jan '15

17 Jan '15

The IP 62.210.82.189 has just been banned by Fail2Ban after 18 attempts against SSH. Latest lines containing IP:62.210.82.189 in /var/log/syslog/sshd.log Jan 17 09:25:44 ftlogin2 sshd[16132]: error: PAM: Authentication failure for root from 62-210-82-189.rev.poneytelecom.eu Jan 17 09:25:44 ftlogin2 sshd[16135]: Received disconnect from 62.210.82.189: 11: Bye Bye Jan 17 09:25:47 ftlogin2 sshd[16188]: error: PAM: Authentication failure for root from 62-210-82-189.rev.poneytelecom.eu Jan 17 09:25:47 ftlogin2 sshd[16221]: Received disconnect from 62.210.82.189: 11: Bye Bye Jan 17 09:25:53 ftlogin2 sshd[16257]: error: PAM: Authentication failure for root from 62-210-82-189.rev.poneytelecom.eu Jan 17 09:25:53 ftlogin2 sshd[16260]: Received disconnect from 62.210.82.189: 11: Bye Bye Jan 17 09:25:57 ftlogin2 sshd[16262]: error: PAM: Authentication failure for root from 62-210-82-189.rev.poneytelecom.eu Jan 17 09:25:57 ftlogin2 sshd[16265]: Received disconnect from 62.210.82.189: 11: Bye Bye Jan 17 09:25:59 ftlogin2 sshd[16267]: error: PAM: Authentication failure for root from 62-210-82-189.rev.poneytelecom.eu Jan 17 09:25:59 ftlogin2 sshd[16270]: Received disconnect from 62.210.82.189: 11: Bye Bye - Fail2ban on

1 0