August 2016 - Intel-nda - mailman3.cels.anl.gov

Re: [JLSE-Intel-NDA] Testing Cobalt Scheduling on knl07 and knl08
by Alekseev, Yury 05 Aug '16

05 Aug '16

Hi Chris, Check if you are a part of "intel" UNIX group. Yuri ________________________________ From: intel-nda-bounces(a)lists.jlse.anl.gov [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Christopher Knight [knightc(a)anl.gov] Sent: Friday, August 05, 2016 10:40 AM To: Allen, Benjamin S. Cc: intel-nda(a)lists.jlse.anl.gov Subject: Re: [JLSE-Intel-NDA] Testing Cobalt Scheduling on knl07 and knl08 Hi Ben, I’m getting “Host key verification failed.” errors. Am I goofing somewhere? Is it necessary to set attributes for memory+cluster or will not setting those let the job run on any available node? chris =============== [knightc@jlselogin2 daxpy]$ pwd /home/knightc/blas/daxpy qsub -n 1 -q testingknl -t 5 ./simple.cobalt [knightc@jlselogin2 daxpy]$ cat simple.cobalt #!/bin/bash source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh env OMP_NUM_THREADS=1 mpirun -f ${COBALT_NODEFILE} -ppn 1 ./test_daxpy On Aug 2, 2016, at 5:08 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov<mailto:[email protected]>> wrote: All, We have the Cobalt scheduler setup for the KNL nodes. For initial testing I've setup a queue called testingknl, that is limited to knl07 and knl08. Anyone that is in the "intel" unix group should be able to submit to it. No other restrictions have been put on the queue. We've added the ability to set MCDRAM Memory Mode and Cluster Mode via qsub attrs. This goes through the syscfg and power cycle process for you. Example: qsub -n 2 -q testingknl -t 10 --attrs knl_memory_mode=Flat:knl_cluster_mode=Quadrant /home/bsallen/test_mpi.sh The scheduler detects the current settings and won't power cycle the node if they're already correct. However it doesn't have the ability to pick available nodes that already have the correct settings. When the nodes need to reboot it takes about 10min 30secs before the job starts. This time isn't deducted from your walltime usage. Note the values for knl_memory_mode and knl_cluster_mode are case sensitive. The full list of accepted values: knl_memory_mode: Cache Flat Hybrid Auto knl_cluster_mode: All2All SNC-2 SNC-4 Hemisphere Quadrant Auto Example submission script: #!/bin/bash source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh mpirun -f $COBALT_NODEFILE -ppn 64 $HOME/bin/mpi_hello Let me know what breaks. Once we're reasonably happy with it we'll schedule all the KNL nodes. Ben _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Testing Cobalt Scheduling on knl07 and knl08
by Christopher Knight 05 Aug '16

05 Aug '16

Hi Ben, I’m getting “Host key verification failed.” errors. Am I goofing somewhere? Is it necessary to set attributes for memory+cluster or will not setting those let the job run on any available node? chris =============== [knightc@jlselogin2 daxpy]$ pwd /home/knightc/blas/daxpy qsub -n 1 -q testingknl -t 5 ./simple.cobalt [knightc@jlselogin2 daxpy]$ cat simple.cobalt #!/bin/bash source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh env OMP_NUM_THREADS=1 mpirun -f ${COBALT_NODEFILE} -ppn 1 ./test_daxpy On Aug 2, 2016, at 5:08 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: All, We have the Cobalt scheduler setup for the KNL nodes. For initial testing I've setup a queue called testingknl, that is limited to knl07 and knl08. Anyone that is in the "intel" unix group should be able to submit to it. No other restrictions have been put on the queue. We've added the ability to set MCDRAM Memory Mode and Cluster Mode via qsub attrs. This goes through the syscfg and power cycle process for you. Example: qsub -n 2 -q testingknl -t 10 --attrs knl_memory_mode=Flat:knl_cluster_mode=Quadrant /home/bsallen/test_mpi.sh The scheduler detects the current settings and won't power cycle the node if they're already correct. However it doesn't have the ability to pick available nodes that already have the correct settings. When the nodes need to reboot it takes about 10min 30secs before the job starts. This time isn't deducted from your walltime usage. Note the values for knl_memory_mode and knl_cluster_mode are case sensitive. The full list of accepted values: knl_memory_mode: Cache Flat Hybrid Auto knl_cluster_mode: All2All SNC-2 SNC-4 Hemisphere Quadrant Auto Example submission script: #!/bin/bash source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh mpirun -f $COBALT_NODEFILE -ppn 64 $HOME/bin/mpi_hello Let me know what breaks. Once we're reasonably happy with it we'll schedule all the KNL nodes. Ben _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] cmake
by Delalondre Fabien Jonathan 04 Aug '16

04 Aug '16

We have also installed nix for all software management on jlse in user space. If there is interest we can in the future show you how that works as a possible alternative. Best, Fabien On 4 ao?t 2016, at 16:41, William Scullin <wscullin(a)alcf.anl.gov<mailto:[email protected]>> wrote: If there's a reasonable space to do installs, I would be willing to kick off installs via LLNL Spack: https://github.com/LLNL/spack. - William On Thu, Aug 4, 2016 at 1:34 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov<mailto:[email protected]>> wrote: You're on your own for newer versions of things. If you want to maintain a version in /soft that would also be great. Ben > On Aug 4, 2016, at 1:31 PM, Kenneth E Jansen <kenneth.jansen(a)colorado.edu<mailto:[email protected]>> wrote: > > Can we get a newer version of cmake? Our build environment exploits features not available in 2.8.11. 3.0 or above would probably work but the newer the better. > > Thanks, > > Ken > > ---------------------------------------------------------------- > Kenneth E. Jansen, Professor of Aerospace Engineering Sciences > ECAE 161 OFFICE (303) 492-4359<tel:%28303%29%20492-4359> > 429 UCB FAX (303) 492-4990 > University of Colorado at Boulder jansenke(a)colorado.edu<mailto:[email protected]> > Boulder, CO, 80309-0429 http://www.colorado.edu/aerospace > ---------------------------------------------------------------- > > > _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] cmake
by Allen, Benjamin S. 04 Aug '16

04 Aug '16

/soft/buildtools/cmake-<vers> sounds like a good directory. You're already in the software group. Thanks, Ben > On Aug 4, 2016, at 3:40 PM, William Scullin <wscullin(a)alcf.anl.gov> wrote: > > If there's a reasonable space to do installs, I would be willing to kick off installs via LLNL Spack: https://github.com/LLNL/spack. > > - William > > On Thu, Aug 4, 2016 at 1:34 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > You're on your own for newer versions of things. If you want to maintain a version in /soft that would also be great. > > Ben > > > On Aug 4, 2016, at 1:31 PM, Kenneth E Jansen <kenneth.jansen(a)colorado.edu> wrote: > > > > Can we get a newer version of cmake? Our build environment exploits features not available in 2.8.11. 3.0 or above would probably work but the newer the better. > > > > Thanks, > > > > Ken > > > > ---------------------------------------------------------------- > > Kenneth E. Jansen, Professor of Aerospace Engineering Sciences > > ECAE 161 OFFICE (303) 492-4359 > > 429 UCB FAX (303) 492-4990 > > University of Colorado at Boulder jansenke(a)colorado.edu > > Boulder, CO, 80309-0429 http://www.colorado.edu/aerospace > > ---------------------------------------------------------------- > > > > > > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] cmake
by William Scullin 04 Aug '16

04 Aug '16

If there's a reasonable space to do installs, I would be willing to kick off installs via LLNL Spack: https://github.com/LLNL/spack. - William On Thu, Aug 4, 2016 at 1:34 PM, Allen, Benjamin S. <bsallen(a)alcf.anl.gov> wrote: > You're on your own for newer versions of things. If you want to maintain a > version in /soft that would also be great. > > Ben > > > On Aug 4, 2016, at 1:31 PM, Kenneth E Jansen < > kenneth.jansen(a)colorado.edu> wrote: > > > > Can we get a newer version of cmake? Our build environment exploits > features not available in 2.8.11. 3.0 or above would probably work but > the newer the better. > > > > Thanks, > > > > Ken > > > > ---------------------------------------------------------------- > > Kenneth E. Jansen, Professor of Aerospace Engineering Sciences > > ECAE 161 OFFICE (303) 492-4359 > > 429 UCB FAX (303) 492-4990 > > University of Colorado at Boulder jansenke(a)colorado.edu > > Boulder, CO, 80309-0429 http://www.colorado.edu/aerospace > > ---------------------------------------------------------------- > > > > > > > > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] cmake
by Allen, Benjamin S. 04 Aug '16

04 Aug '16

You're on your own for newer versions of things. If you want to maintain a version in /soft that would also be great. Ben > On Aug 4, 2016, at 1:31 PM, Kenneth E Jansen <kenneth.jansen(a)colorado.edu> wrote: > > Can we get a newer version of cmake? Our build environment exploits features not available in 2.8.11. 3.0 or above would probably work but the newer the better. > > Thanks, > > Ken > > ---------------------------------------------------------------- > Kenneth E. Jansen, Professor of Aerospace Engineering Sciences > ECAE 161 OFFICE (303) 492-4359 > 429 UCB FAX (303) 492-4990 > University of Colorado at Boulder jansenke(a)colorado.edu > Boulder, CO, 80309-0429 http://www.colorado.edu/aerospace > ---------------------------------------------------------------- > > >

1 0

Re: [JLSE-Intel-NDA] Testing Cobalt Scheduling on knl07 and knl08
by Allen, Benjamin S. 03 Aug '16

03 Aug '16

Yea if I wasn't clear, qsub is on the login nodes. If needed, SSH to knl07 and knl08 (and eventually the rest of the nodes after they're integrated) won't be permitted until your job is running. We're setting this Cobalt instance up much like Cooley. Ben > On Aug 3, 2016, at 1:30 PM, Alekseev, Yury <yuri(a)alcf.anl.gov> wrote: > > > I see, I tried to find qsub on knl node. > > Yuri > ________________________________________ > From: Vitali A. Morozov [morozov(a)anl.gov] > Sent: Wednesday, August 03, 2016 1:28 PM > To: Alekseev, Yury; Allen, Benjamin S.; intel-nda(a)lists.jlse.anl.gov > Subject: Re: [JLSE-Intel-NDA] Testing Cobalt Scheduling on knl07 and knl08 > > [morozov@jlselogin2 Hello]$ which qsub > /usr/bin/qsub > > > > On 08/03/2016 01:25 PM, Alekseev, Yury wrote: >> Hi, >> >> I cannot find Cobalt qsub. In fact, I did find search on the whole JLSE file system and could not find it. >> Do you need to be on one of the knl nodes to submit jobs or it can be done from any JLSE computer? >> >> Yuri >> ________________________________________ >> From: intel-nda-bounces(a)lists.jlse.anl.gov [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Allen, Benjamin S. [bsallen(a)alcf.anl.gov] >> Sent: Tuesday, August 02, 2016 5:08 PM >> To: intel-nda(a)lists.jlse.anl.gov >> Subject: [JLSE-Intel-NDA] Testing Cobalt Scheduling on knl07 and knl08 >> >> All, >> >> We have the Cobalt scheduler setup for the KNL nodes. For initial testing I've setup a queue called testingknl, that is limited to knl07 and knl08. Anyone that is in the "intel" unix group should be able to submit to it. No other restrictions have been put on the queue. >> >> We've added the ability to set MCDRAM Memory Mode and Cluster Mode via qsub attrs. This goes through the syscfg and power cycle process for you. Example: >> >> qsub -n 2 -q testingknl -t 10 --attrs knl_memory_mode=Flat:knl_cluster_mode=Quadrant /home/bsallen/test_mpi.sh >> >> The scheduler detects the current settings and won't power cycle the node if they're already correct. However it doesn't have the ability to pick available nodes that already have the correct settings. When the nodes need to reboot it takes about 10min 30secs before the job starts. This time isn't deducted from your walltime usage. >> >> Note the values for knl_memory_mode and knl_cluster_mode are case sensitive. The full list of accepted values: >> >> knl_memory_mode: >> >> Cache >> Flat >> Hybrid >> Auto >> >> knl_cluster_mode: >> >> All2All >> SNC-2 >> SNC-4 >> Hemisphere >> Quadrant >> Auto >> >> Example submission script: >> >> #!/bin/bash >> >> source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh >> mpirun -f $COBALT_NODEFILE -ppn 64 $HOME/bin/mpi_hello >> >> Let me know what breaks. Once we're reasonably happy with it we'll schedule all the KNL nodes. >> >> Ben >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >> _______________________________________________ >> Intel-nda mailing list >> Intel-nda(a)lists.jlse.anl.gov >> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda >

1 0

Re: [JLSE-Intel-NDA] Testing Cobalt Scheduling on knl07 and knl08
by Alekseev, Yury 03 Aug '16

03 Aug '16

I see, I tried to find qsub on knl node. Yuri ________________________________________ From: Vitali A. Morozov [morozov(a)anl.gov] Sent: Wednesday, August 03, 2016 1:28 PM To: Alekseev, Yury; Allen, Benjamin S.; intel-nda(a)lists.jlse.anl.gov Subject: Re: [JLSE-Intel-NDA] Testing Cobalt Scheduling on knl07 and knl08 [morozov@jlselogin2 Hello]$ which qsub /usr/bin/qsub On 08/03/2016 01:25 PM, Alekseev, Yury wrote: > Hi, > > I cannot find Cobalt qsub. In fact, I did find search on the whole JLSE file system and could not find it. > Do you need to be on one of the knl nodes to submit jobs or it can be done from any JLSE computer? > > Yuri > ________________________________________ > From: intel-nda-bounces(a)lists.jlse.anl.gov [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Allen, Benjamin S. [bsallen(a)alcf.anl.gov] > Sent: Tuesday, August 02, 2016 5:08 PM > To: intel-nda(a)lists.jlse.anl.gov > Subject: [JLSE-Intel-NDA] Testing Cobalt Scheduling on knl07 and knl08 > > All, > > We have the Cobalt scheduler setup for the KNL nodes. For initial testing I've setup a queue called testingknl, that is limited to knl07 and knl08. Anyone that is in the "intel" unix group should be able to submit to it. No other restrictions have been put on the queue. > > We've added the ability to set MCDRAM Memory Mode and Cluster Mode via qsub attrs. This goes through the syscfg and power cycle process for you. Example: > > qsub -n 2 -q testingknl -t 10 --attrs knl_memory_mode=Flat:knl_cluster_mode=Quadrant /home/bsallen/test_mpi.sh > > The scheduler detects the current settings and won't power cycle the node if they're already correct. However it doesn't have the ability to pick available nodes that already have the correct settings. When the nodes need to reboot it takes about 10min 30secs before the job starts. This time isn't deducted from your walltime usage. > > Note the values for knl_memory_mode and knl_cluster_mode are case sensitive. The full list of accepted values: > > knl_memory_mode: > > Cache > Flat > Hybrid > Auto > > knl_cluster_mode: > > All2All > SNC-2 > SNC-4 > Hemisphere > Quadrant > Auto > > Example submission script: > > #!/bin/bash > > source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh > mpirun -f $COBALT_NODEFILE -ppn 64 $HOME/bin/mpi_hello > > Let me know what breaks. Once we're reasonably happy with it we'll schedule all the KNL nodes. > > Ben > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Testing Cobalt Scheduling on knl07 and knl08
by Vitali A. Morozov 03 Aug '16

03 Aug '16

[morozov@jlselogin2 Hello]$ which qsub /usr/bin/qsub On 08/03/2016 01:25 PM, Alekseev, Yury wrote: > Hi, > > I cannot find Cobalt qsub. In fact, I did find search on the whole JLSE file system and could not find it. > Do you need to be on one of the knl nodes to submit jobs or it can be done from any JLSE computer? > > Yuri > ________________________________________ > From: intel-nda-bounces(a)lists.jlse.anl.gov [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Allen, Benjamin S. [bsallen(a)alcf.anl.gov] > Sent: Tuesday, August 02, 2016 5:08 PM > To: intel-nda(a)lists.jlse.anl.gov > Subject: [JLSE-Intel-NDA] Testing Cobalt Scheduling on knl07 and knl08 > > All, > > We have the Cobalt scheduler setup for the KNL nodes. For initial testing I've setup a queue called testingknl, that is limited to knl07 and knl08. Anyone that is in the "intel" unix group should be able to submit to it. No other restrictions have been put on the queue. > > We've added the ability to set MCDRAM Memory Mode and Cluster Mode via qsub attrs. This goes through the syscfg and power cycle process for you. Example: > > qsub -n 2 -q testingknl -t 10 --attrs knl_memory_mode=Flat:knl_cluster_mode=Quadrant /home/bsallen/test_mpi.sh > > The scheduler detects the current settings and won't power cycle the node if they're already correct. However it doesn't have the ability to pick available nodes that already have the correct settings. When the nodes need to reboot it takes about 10min 30secs before the job starts. This time isn't deducted from your walltime usage. > > Note the values for knl_memory_mode and knl_cluster_mode are case sensitive. The full list of accepted values: > > knl_memory_mode: > > Cache > Flat > Hybrid > Auto > > knl_cluster_mode: > > All2All > SNC-2 > SNC-4 > Hemisphere > Quadrant > Auto > > Example submission script: > > #!/bin/bash > > source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh > mpirun -f $COBALT_NODEFILE -ppn 64 $HOME/bin/mpi_hello > > Let me know what breaks. Once we're reasonably happy with it we'll schedule all the KNL nodes. > > Ben > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda > _______________________________________________ > Intel-nda mailing list > Intel-nda(a)lists.jlse.anl.gov > https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0

Re: [JLSE-Intel-NDA] Testing Cobalt Scheduling on knl07 and knl08
by Alekseev, Yury 03 Aug '16

03 Aug '16

Hi, I cannot find Cobalt qsub. In fact, I did find search on the whole JLSE file system and could not find it. Do you need to be on one of the knl nodes to submit jobs or it can be done from any JLSE computer? Yuri ________________________________________ From: intel-nda-bounces(a)lists.jlse.anl.gov [intel-nda-bounces(a)lists.jlse.anl.gov] on behalf of Allen, Benjamin S. [bsallen(a)alcf.anl.gov] Sent: Tuesday, August 02, 2016 5:08 PM To: intel-nda(a)lists.jlse.anl.gov Subject: [JLSE-Intel-NDA] Testing Cobalt Scheduling on knl07 and knl08 All, We have the Cobalt scheduler setup for the KNL nodes. For initial testing I've setup a queue called testingknl, that is limited to knl07 and knl08. Anyone that is in the "intel" unix group should be able to submit to it. No other restrictions have been put on the queue. We've added the ability to set MCDRAM Memory Mode and Cluster Mode via qsub attrs. This goes through the syscfg and power cycle process for you. Example: qsub -n 2 -q testingknl -t 10 --attrs knl_memory_mode=Flat:knl_cluster_mode=Quadrant /home/bsallen/test_mpi.sh The scheduler detects the current settings and won't power cycle the node if they're already correct. However it doesn't have the ability to pick available nodes that already have the correct settings. When the nodes need to reboot it takes about 10min 30secs before the job starts. This time isn't deducted from your walltime usage. Note the values for knl_memory_mode and knl_cluster_mode are case sensitive. The full list of accepted values: knl_memory_mode: Cache Flat Hybrid Auto knl_cluster_mode: All2All SNC-2 SNC-4 Hemisphere Quadrant Auto Example submission script: #!/bin/bash source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh mpirun -f $COBALT_NODEFILE -ppn 64 $HOME/bin/mpi_hello Let me know what breaks. Once we're reasonably happy with it we'll schedule all the KNL nodes. Ben _______________________________________________ Intel-nda mailing list Intel-nda(a)lists.jlse.anl.gov https://lists.jlse.anl.gov/mailman/listinfo/intel-nda

1 0