Hi Chris, Check if you are a part of "intel" UNIX group. Yuri ________________________________ From: [email protected] [[email protected]] on behalf of Christopher Knight [[email protected]] Sent: Friday, August 05, 2016 10:40 AM To: Allen, Benjamin S. Cc: [email protected] Subject: Re: [JLSE-Intel-NDA] Testing Cobalt Scheduling on knl07 and knl08 Hi Ben, I’m getting “Host key verification failed.” errors. Am I goofing somewhere? Is it necessary to set attributes for memory+cluster or will not setting those let the job run on any available node? chris =============== [knightc@jlselogin2 daxpy]$ pwd /home/knightc/blas/daxpy qsub -n 1 -q testingknl -t 5 ./simple.cobalt [knightc@jlselogin2 daxpy]$ cat simple.cobalt #!/bin/bash source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh env OMP_NUM_THREADS=1 mpirun -f ${COBALT_NODEFILE} -ppn 1 ./test_daxpy On Aug 2, 2016, at 5:08 PM, Allen, Benjamin S. <[email protected]<mailto:[email protected]>> wrote: All, We have the Cobalt scheduler setup for the KNL nodes. For initial testing I've setup a queue called testingknl, that is limited to knl07 and knl08. Anyone that is in the "intel" unix group should be able to submit to it. No other restrictions have been put on the queue. We've added the ability to set MCDRAM Memory Mode and Cluster Mode via qsub attrs. This goes through the syscfg and power cycle process for you. Example: qsub -n 2 -q testingknl -t 10 --attrs knl_memory_mode=Flat:knl_cluster_mode=Quadrant /home/bsallen/test_mpi.sh The scheduler detects the current settings and won't power cycle the node if they're already correct. However it doesn't have the ability to pick available nodes that already have the correct settings. When the nodes need to reboot it takes about 10min 30secs before the job starts. This time isn't deducted from your walltime usage. Note the values for knl_memory_mode and knl_cluster_mode are case sensitive. The full list of accepted values: knl_memory_mode: Cache Flat Hybrid Auto knl_cluster_mode: All2All SNC-2 SNC-4 Hemisphere Quadrant Auto Example submission script: #!/bin/bash source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh mpirun -f $COBALT_NODEFILE -ppn 64 $HOME/bin/mpi_hello Let me know what breaks. Once we're reasonably happy with it we'll schedule all the KNL nodes. Ben _______________________________________________ Intel-nda mailing list [email protected]<mailto:[email protected]> https://lists.jlse.anl.gov/mailman/listinfo/intel-nda