Hi, I cannot find Cobalt qsub. In fact, I did find search on the whole JLSE file system and could not find it. Do you need to be on one of the knl nodes to submit jobs or it can be done from any JLSE computer? Yuri ________________________________________ From: [email protected] [[email protected]] on behalf of Allen, Benjamin S. [[email protected]] Sent: Tuesday, August 02, 2016 5:08 PM To: [email protected] Subject: [JLSE-Intel-NDA] Testing Cobalt Scheduling on knl07 and knl08 All, We have the Cobalt scheduler setup for the KNL nodes. For initial testing I've setup a queue called testingknl, that is limited to knl07 and knl08. Anyone that is in the "intel" unix group should be able to submit to it. No other restrictions have been put on the queue. We've added the ability to set MCDRAM Memory Mode and Cluster Mode via qsub attrs. This goes through the syscfg and power cycle process for you. Example: qsub -n 2 -q testingknl -t 10 --attrs knl_memory_mode=Flat:knl_cluster_mode=Quadrant /home/bsallen/test_mpi.sh The scheduler detects the current settings and won't power cycle the node if they're already correct. However it doesn't have the ability to pick available nodes that already have the correct settings. When the nodes need to reboot it takes about 10min 30secs before the job starts. This time isn't deducted from your walltime usage. Note the values for knl_memory_mode and knl_cluster_mode are case sensitive. The full list of accepted values: knl_memory_mode: Cache Flat Hybrid Auto knl_cluster_mode: All2All SNC-2 SNC-4 Hemisphere Quadrant Auto Example submission script: #!/bin/bash source /soft/compilers/intel/compilers_and_libraries/linux/mpi/bin64/mpivars.sh mpirun -f $COBALT_NODEFILE -ppn 64 $HOME/bin/mpi_hello Let me know what breaks. Once we're reasonably happy with it we'll schedule all the KNL nodes. Ben _______________________________________________ Intel-nda mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/intel-nda