La scorsa settimana ho installato un cluster Slurm per gestire un grosso numero di nodi di calcolo, Slurm è uno dei più supportati job scheduler per Linux e Unix.

E’ difficile trovare on-line un tutorial di installazione per CentOS 6, è più semplice partire con la distribuzione CentOS 7, ma nel mio landscape i server erano già configurati.

Spero che troverete utile l’articolo che copre l’installazione e configurazione del Master e del primo nodo di calcolo.

Installazione del master e del primo computing node

Fate attenzione che in tutto il cluster uid e gid coincidano.

creazione dell’utente munge

export MUNGEUSER=991
groupadd -g $MUNGEUSER munge
useradd -m -c “MUNGE Uid ‘N’ ” -d /var/lib/munge -u $MUNGEUSER -g munge -s /sbin/nologin munge

creazione dell’utente slurm

export SLURMUSER=990 #imposto a 990 la variabile SLURMUSER
groupadd -g $SLURMUSER slurm #lo aggiungo al Gruppo slurm
useradd -m -c “SLURM workload manager” -d /var/lib/slurm -u $SLURMUSER -g slurm -s /bin/bash slurm

installo epel-release

yum install epel-release

installo munge

yum install munge munge-libs munge-devel -y

installo rng-tools

yum install rng-tools -y
rngd -r /dev/urandom

genero sul master la random key

/usr/sbin/create-munge-key -r
dd if=/dev/urandom bs=1 count=1024 > /etc/munge/munge.key
chown munge: /etc/munge/munge.key
chmod 400 /etc/munge/munge.key

copio la chiave su tutti i computing nodes

scp /etc/munge/munge.key root@vm-calc-01:/etc/munge/munge.key
chown -R munge: /etc/munge/ /var/log/munge/
chmod 0700 /etc/munge/ /var/log/munge/

avvio munge

service munge start
chkconfig munge on
munge -n
munge -n | unmunge
munge -n | ssh vm-calc-01 unmunge

host benchmark

remunge

installo i prerequisiti per
slurm

yum install openssl openssl-devel pam-devel numactl numactl-devel hwloc hwloc-devel lua lua-devel readline-devel rrdtool-devel ncurses-devel man2html libibmad libibumad -y

download dell’ultima versione di slurm

wget https://download.schedmd.com/slurm/slurm-18.08.3.tar.bz2

installo slurm

tar –bzip -x -f slurm*tar.bz2
cd slurm-18.08.3
./configure
make
make install

creo la struttura directory per slurm

mkdir /etc/slurm
touch /var/log/slurm.log
touch /var/log/SlurmctldLogFile.log
touch /var/log/SlurmdLogFile.log
mkdir /var/spool/slurmctld
chown slurm: /var/spool/slurmctld
chmod 755 /var/spool/slurmctld
mkdir /var/spool/slurmld
chown slurm: /var/spool/slurmd
chmod 755 /var/spool/slurmd
touch /var/log/slurmctld.log
chown slurm: /var/log/slurmctld.log
touch /var/log/slurm_jobacct.log
chown slurm: /var/log/slurm_jobacct.log

creo sul master il file slurm.conf

Con il tool on-line tools https://slurm.schedmd.com/configurator.easy.html puoi generare il file slurm.conf

slurm.conf

slurm.conf file generated by configurator easy.html.
Put this file on all nodes of your cluster.
See the slurm.conf man page for more information.
#
SlurmctldHost=slurm-master

MailProg=/bin/mail
MpiDefault=none
MpiParams=ports=#-#
ProctrackType=proctrack/cgroup
ReturnToService=1
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/spool/slurmd
SlurmUser=slurm
SlurmdUser=root
StateSaveLocation=/var/spool/slurmctld
SwitchType=switch/none
TaskPlugin=task/cgroup


TIMERS
KillWait=30
MinJobAge=300
SlurmctldTimeout=120
SlurmdTimeout=300


SCHEDULING
FastSchedule=1
SchedulerType=sched/backfill
SelectType=select/cons_res
SelectTypeParameters=CR_Core


LOGGING AND ACCOUNTING
AccountingStorageType=accounting_storage/none
ClusterName=cluster
JobAcctGatherFrequency=30
JobAcctGatherType=jobacct_gather/none
SlurmctldDebug=3
SlurmctldLogFile=
SlurmdDebug=3
SlurmdLogFile=


COMPUTE NODES
NodeName=vm-calc-01 CPUs=36 State=UNKNOWN
PartitionName=test Nodes=vm-calc-01 Default=YES MaxTime=INFINITE State=UP

vi /etc/slurm/slurm.conf

copio la configurazione sui computing nodes

scp /etc/slurm/slurm.conf root@vm-calc-01:/etc/slurm/slurm.conf

test della configurazione sul master

slurmd -C

NodeName=slurm-master CPUs=8 Boards=1 SocketsPerBoard=8 CoresPerSocket=1ThreadsPerCore=1 RealMemory=32109

test della configurazione sul node-01

slurmd -C

NodeName=vm-calc-01 CPUs=36 Boards=1 SocketsPerBoard=18CoresPerSocket=2 ThreadsPerCore=1 RealMemory=129059

per fare le prove ho disabilito il firewall e il linux enhanced security 

service iptables stop
setenforce 0

creazione dei link ai file

ln -s /etc/slurm/slurm.conf /usr/local/etc/slurm.conf

da https://slurm.schedmd.com/faq.html Other exceptions:

On Centos 6, also set “ProcessUnpackaged = yes” in the file /etc/abrt/abrt-action-save-package-data.conf.

vi /etc/abrt/abrt-action-save-package-data.conf

avvio del control daemon sul master

slurmctld

avvio del demone slurm sul computing nodes

slurmd

verifica della configurazione

sinfo

PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
test*        up   infinite      1  idle vm-calc-01

installo cgroup

yum install libcgroup
service cgconfig start

vi /etc/slurm/cgroups.conf

cgroups.conf

CgroupMountpoint=”/cgroup”
CgroupAutomount=yes
CgroupReleaseAgentDir=”/etc/slurm/cgroup”
AllowedDevicesFile=”/etc/slurm/cgroup_allowed_devices_file.conf”
ConstrainCores=yes
TaskAffinity=no
ConstrainRAMSpace=yes
ConstrainSwapSpace=yes
ConstrainDevices=no
AllowedRamSpace=100
AllowedSwapSpace=0
MaxRAMPercent=100
MaxSwapPercent=100
MinRAMSpace=30

vi /etc/slurm/cgroup_allowed_device_file.conf

cgroup_allowed_devices_file.conf

/dev/null
/dev/urandom
/dev/zero
/dev/sda*
/dev/cpu//
/dev/pts/*

ln -s /etc/slurm/cgroup.conf /usr/local/etc/cgroup.conf
ln -s /etc/slurm/cgroup_allowed_devices_file.conf /usr/local/etc/cgroup_allowed_devices_file.conf

creo un batch di test sul master

vi submit.sh

submit.sh

!/bin/bash
#
SBATCH –job-name=test
SBATCH –output=res.txt
#
SBATCH –ntasks=1
SBATCH –time=10:00
SBATCH –output=slurm_%j.out
srun hostname
srun sleep 60

submit del batch dal master

sbatch submit.sh

verifico la coda dal master

squeue

JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
        8      test submit.s     root  R       0:50      1 rm-vm-calc-01

siamo alla fine… che è un nuovo inizio

Ricordate di abilitare e configurare il firewall secondo la topologia della vostra infrastruttura e di mettere in avvio automatico i servizi se vi serve (in rc.local), configurare l’accounting e il master di backup.