Cluster Deployment Management Tools

There are many deployment and management tools. Here is my recommended list :

• Big HPC cluster : Salt Stack
• Small HPC cluster : Ansible
• Very heterogeneous managed stuff : Puppet

My hearth goes to Salt and Ansible. Salt is really flexible and has an important thing for HPC or Cloud : loops. This is a personal choice.

Ensure all minion are up to date :

salt '*' state.highstate -v

Replace '*' by the hostname or the ip of the minion you want to update only the state.

To get all the process done on a specific minion, login on it with ssh, then run :

 salt-call -l debug state.highstate 

This is a good way to debug and see what is going on.

Use jinja in top files, and share a file containing all servers ip for both pillar and state top :

Pillar top file (/srv/pillar/top.sls):

{% import_yaml 'servers.sls' as vars %}
base:
  '{{vars.servers.dhcp.ip}}':
    - pkgs
    - services
    - dhcp-server

State top file (/srv/salt/top.sls):

{% import_yaml 'servers.sls' as vars %}
base:
  '{{vars.servers.dhcp.ip}}':
    - repository.client
    - dhcp.server

Then edit your server.sls file in /srv/pillar/server.sls :

servers:
    repository:
        name: repo0
        ip: 172.16.0.12
    dhcp:
        name: dhcp0
        ip: 172.16.0.12

To finish, add a link in linux to allow state top to read also this file :

ln -s /srv/pillar/servers.sls /srv/salt/servers.sls

Using this, all Salt is dynamic !!

Basic jinja2 syntax :

Get a value/string from pillar (here pillar network, value of subnet) :

 {{ salt['pillar.get']('network:subnet') }} 

Loop on a pillar last level list :

 {% for rangeip in salt['pillar.get']('network:dhcp:range') %}
 range {{rangeip}};
 {% endfor %}

Loop on a pillar non last level list :

 {% for host, args in salt['pillar.get']('nodes', {}).items() %}
 host {{ host }} {
 hardware ethernet {{ args.hwaddr }};
 fixed-address {{ args.ip }};
 }

 {% endfor %}

Split a string into a list using a specific character as separator (useful for DNS configuration files !):

 {% set list1 = salt['pillar.get']('network:subnet').split('.') %}
 {% if salt['pillar.get']('network:netmask') == '255.255.255.0' %}
 option broadcast-address {{ list1[0] }}.{{ list1[1] }}.{{ list1[2] }}.255;
 {% elif salt['pillar.get']('network:netmask') == '255.255.0.0' %}
 option broadcast-address {{ list1[0] }}.{{ list1[1] }}.225.255;
 {% elif salt['pillar.get']('network:netmask') == '255.0.0.0' %}
 option broadcast-address {{ list1[0] }}.255.225.255;
 {% else %}
 option broadcast-address CANNOT UNDERSTAND NETMASK !!! See dhcp/dhcpd.conf.jinja;
 {% endif %}

More : http://jinja.pocoo.org/docs/dev/templates/

git clone git://github.com/ansible/ansible.git --recursive
tar cvzf ansible.tar.gz ansible
mkdir /home/yourlogin/pip
pip install --ignore-installed --target=/home/yourlogin/pip --install-option="--install-purelib=/home/yourlogin/pip" paramiko PyYAML Jinja2 httplib2 six
tar cvzf pip.tar.gz /home/yourlogin/pip

Then on master node :

cd /root
tar xvzf ansible.tar.gz
tar xvzf pip.tar.gz

Add nodes into ansible nodes file (/root/ansible_hosts) :

[repo]
10.0.0.2

[dhcp]
10.0.0.3

[pxe]
10.0.0.4

Now launch ansible environnement (must be done each time you will use ansible) :

source /root/ansible/hacking/env-setup
export PYTHONPATH=/root/pip/:$PYTHONPATH

Set node list file :

export ANSIBLE_INVENTORY=/root/ansible_hosts

And check everything is ok (for this example, only 2 of the 3 other nodes are online, 4 is offline to display the error) :

ansible all -m ping 
10.0.0.2 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
10.0.0.3 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
10.0.0.4 | FAILED! => {
    "failed": true, 
    "msg": "ERROR! SSH encountered an unknown error during the connection. We recommend you re-run the command using -vvvv, which will enable SSH debugging output to help diagnose the issue"
}

To execute a playbook on nodes :

ansible-playbook /root/nodes/repo/playbooks/default.pb

If something goes wrong, you can see debug using -v (-vv -vvv -vvvv etc for level of debugs). Note also that ansible will stop when an error is detected. You can ignore errors using ignore_errors :

    - name: Say hello
      command: echo "hello"
      ignore_errors: true

---
- hosts: repo
  remote_user: root
  tasks:

    ###############################################################
    ########### vsftpd server installation
    ###

    - name: Installing vsftpd rpm
      command: chdir=/mnt/Packages/ rpm -ivh vsftpd-3.0.2-9.el7.x86_64.rpm 
    - name: Enable vsftpd on start
      command: systemctl enable vsftpd
    - name: Start vsftpd on start
      command: systemctl start vsftpd
    - name: Installing libxml2-python rpm
      command: chdir=/mnt/Packages/ rpm -ivh libxml2-python-2.9.1-5.el7_0.1.x86_64.rpm
    - name: Installing deltarpm rpm
      command: chdir=/mnt/Packages/ rpm -ivh deltarpm-3.6-3.el7.x86_64.rpm 
    - name: Installing python-deltarpm rpm
      command: chdir=/mnt/Packages/ rpm -ivh python-deltarpm-3.6-3.el7.x86_64.rpm
    - name: Installing createrepo rpm
      command: chdir=/mnt/Packages/ rpm -ivh createrepo-0.9.9-23.el7.noarch.rpm

    ###############################################################
    ########### copy rpm and create repository
    ###

    - file: path=/var/ftp/pub/localrepo state=directory mode=0755
    - name: Copy packages from DVD/iso to repository and repository configuration file
      shell: cp -ar /mnt/Packages/*.* /var/ftp/pub/localrepo/
    - copy: src=/root/nodes/repo/files/default.localrepo.repo dest=/etc/yum.repos.d/localrepo.repo owner=root group=root mode=0640
    - name: Create repository
      command: createrepo -v /var/ftp/pub/localrepo/
    - name: Restore SEl flags
      command: restorecon -R /var/ftp

    ###############################################################
    ########### remove online repositories and update yum
    ###

    - file: path=/etc/yum.repos.d.old state=directory mode=0755
    - name: Move online repo
      shell: mv /etc/yum.repos.d/CentOS-* /etc/yum.repos.d.old
    - name: Restore SEl flags
      command: restorecon -R /etc/yum.repos.d
    - name: Yum list repo
      command: yum repolist
    - name: Yum clean
      command: yum clean all
    - name: Yum update
      command: yum update

    ###############################################################
    ########### disable firewall and set selinux to permissive
    ###

    - name: Stop firewalld
      command: systemctl stop firewalld
    - name: Disable firewalld and copy selinx configuration file
      command: systemctl disable firewalld
    - copy: src=/root/nodes/repo/files/default.selinux dest=/etc/selinux/config owner=root group=root mode=0640
    - name: Restore SEl flags
      command: restorecon /etc/selinux/config

    ###############################################################
    ########### check and reboot
    ###

    - name: Make sure vsftpd is running
      service: name=vsftpd state=running

    - name: Restart node
      command: /sbin/reboot
      async: 0
      poll: 0
      ignore_errors: true
    - name: Waiting for node to come back
      local_action: wait_for host={{ inventory_hostname }}
                state=started
                port=22
                delay=1
                timeout=300
      sudo: false

---
- hosts: dhcp
  remote_user: root
  tasks:

    ###############################################################
    ########### add local repository, remove online repositories and update yum
    ###

    - copy: src=/root/nodes/dhcp/files/default.localrepo.repo dest=/etc/yum.repos.d/localrepo.repo owner=root group=root mode=0640
    - file: path=/etc/yum.repos.d.old state=directory mode=0755
    - name: Move online repo
      shell: mv /etc/yum.repos.d/CentOS-* /etc/yum.repos.d.old
    - name: Restore SEl flags
      command: restorecon -R /etc/yum.repos.d
    - name: Yum list repo
      command: yum repolist
    - name: Yum clean
      command: yum clean all
    - name: Yum update
      command: yum update

    ###############################################################
    ########### install dhcpd server
    ###

    - name: Install dhcpd server
      yum: name=dhcp state=latest
    - copy: src=/root/nodes/dhcp/files/default.dhcpd.conf dest=/etc/dhcp/dhcpd.conf owner=root group=root mode=0644
    - name: Restore SEl flags
      command: restorecon /etc/dhcp/dhcpd.conf
    - name: Enable dhcpd on start
      command: systemctl enable dhcpd.service
    - name: Start dhcpd on start
      command: systemctl start dhcpd.service

    ###############################################################
    ########### disable firewall and set selinux to permissive
    ###

    - name: Stop firewalld
      command: systemctl stop firewalld
    - name: Disable firewalld and copy selinx configuration file
      command: systemctl disable firewalld
    - copy: src=/root/nodes/dhcp/files/default.selinux dest=/etc/selinux/config owner=root group=root mode=0640
    - name: Restore SEl flags
      command: restorecon /etc/selinux/confi

    ###############################################################
    ########### check and reboot
    ###

    - name: Make sure dhcpd is running
      service: name=dhcpd state=running

    - name: Restart node
      command: /sbin/reboot
      async: 0
      poll: 0
      ignore_errors: true
    - name: Waiting for node to come back
      local_action: wait_for host={{ inventory_hostname }}
                state=started
                port=22
                delay=1
                timeout=300
      sudo: false

---
- hosts: pxe
  remote_user: root
  tasks:

    ###############################################################
    ########### add local repository, remove online repositories and update yum
    ###

    - copy: src=/root/nodes/pxe/files/default.localrepo.repo dest=/etc/yum.repos.d/localrepo.repo owner=root group=root mode=0640
    - file: path=/etc/yum.repos.d.old state=directory mode=0755
    - name: Move online repo
      shell: mv /etc/yum.repos.d/CentOS-* /etc/yum.repos.d.old
    - name: Restore SEl flags
      command: restorecon -R /etc/yum.repos.d
    - name: Yum list repo
      command: yum repolist
    - name: Yum clean
      command: yum clean all
    - name: Yum update
      command: yum update

    ###############################################################
    ########### install tftp, xinetd and vsftpd
    ###

    - name: Install tftp
      yum: name=tftp state=latest
    - name: Install tftp-server
      yum: name=tftp-server state=latest
    - name: Install xinetd
      yum: name=xinetd state=latest

    - copy: src=/root/nodes/pxe/files/default.tftp dest=/etc/xinetd.d/tftp owner=root group=root mode=0640
    - name: Restore SEl flags
      command: restorecon /etc/xinetd.d/tftp
    - name: Start xinetd
      command: systemctl start xinetd
    - name: Enable xinetd
      command: systemctl enable xinetd

    - name: Install syslinux
      yum: name=syslinux state=latest
    - name: Install wget
      yum: name=wget state=latest
    - name: Install vsftpd
      yum: name=vsftpd state=latest

    ###############################################################
    ########### copy files for pxe boot
    ###

    - name: Copy pxelinux.0
      command: cp -v /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot
    - name: Copy menu.c32
      command: cp -v /usr/share/syslinux/menu.c32 /var/lib/tftpboot
    - name: Copy memdisk
      command: cp -v /usr/share/syslinux/memdisk /var/lib/tftpboot
    - name: Copy mboot.c32
      command: cp -v /usr/share/syslinux/mboot.c32 /var/lib/tftpboot
    - name: Copy chain.c32
      command: cp -v /usr/share/syslinux/chain.c32 /var/lib/tftpboot

    - file: path=/var/lib/tftpboot/pxelinux.cfg state=directory mode=0755

    - file: path=/var/lib/tftpboot/netboot/ state=directory mode=0755

    - file: path=/var/ftp/pub/iso state=directory mode=0755
    - name: Copy vmlinuz
      command: cp /mnt/images/pxeboot/vmlinuz /var/lib/tftpboot/netboot/
    - name: Copy initrd.img
      command: cp /mnt/images/pxeboot/initrd.img /var/lib/tftpboot/netboot/
    - name: Restore SEl flags
      command: restorecon -R /var/lib/tftpboot

    - copy: src=/root/nodes/pxe/files/default.ks.cfg dest=/var/ftp/pub/ks.cfg owner=root group=root mode=0644
    - name: Restore SEl flags
      command: restorecon /var/ftp/pub/ks.cfg

    - copy: src=/root/nodes/pxe/files/default.pxelinux.cfg.default dest=/var/lib/tftpboot/pxelinux.cfg/default owner=root group=root mode=0644
    - name: Restore SEl flags
      command: restorecon /var/lib/tftpboot/pxelinux.cfg/default

    ###############################################################
    ########### copy minimal iso content and start services
    ###

    - name: Copy minimal iso to /var/ftp/pub/iso/
      shell: cp -Rv /mnt/* /var/ftp/pub/iso/
    - name: Restore SEl flags
      command: restorecon -R /var/ftp/pub/

    - name: Start vsftpd
      command: systemctl start vsftpd
    - name: Enable vsftpd
      command: systemctl enable vsftpd
    - name: Restart vsftpd

      command: systemctl restart vsftpd
    - name: Restart xinetd
      command: systemctl restart xinetd
    - name: Set rights on /var/lib/tftpboot 
      command: chmod 777 /var/lib/tftpboot 

    ###############################################################
    ########### disable firewall and set selinux to permissive
    ###

    - name: Stop firewalld
      command: systemctl stop firewalld
    - name: Disable firewalld and copy selinx configuration file
      command: systemctl disable firewalld
    - copy: src=/root/nodes/pxe/files/default.selinux dest=/etc/selinux/config owner=root group=root mode=0640
    - name: Restore SEl flags
      command: restorecon /etc/selinux/config

    ###############################################################
    ########### check and reboot
    ###

    - name: Make sure vsftpd is running
      service: name=vsftpd state=running
    - name: Make sure xinetd is running
      service: name=xinetd state=running

    - name: Restart node
      command: /sbin/reboot
      async: 0
      poll: 0
      ignore_errors: true
    - name: Waiting for node to come back
      local_action: wait_for host={{ inventory_hostname }}
                state=started
                port=22
                delay=1
                timeout=300
      sudo: false

How to debug YAML files: brute force with http://www.yamllint.com/

From : http://makandracards.com/makandra/29365-vague-puppet-error-messages-with-broken-yaml-files

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: (<unknown>): found character that cannot start any token while scanning for the next token at line 1297 column 3
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: undefined method `empty?' for nil:NilClass at /etc/puppet/environments/production/manifests/nodes.pp:1 on node example.makandra.de
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Need to check presence of “ when using %{ at start of a line : Bad:

foo: %{::fqdn}

Good:

foo: "%{::fqdn}"