How to Update a Container Cluster Node#
Problem#
How can I update a cluster node with a minimal risk by drain the node before any updates but only if updates are pending and rebooted only when there were updates processed?
Solution#
- hosts: nomad_nodes
serial: 1
tasks:
- name: Check updates available
ansible.builtin.package:
name: "*"
state: latest
check_mode: true
register: updates_available
- name: Process package updates
when: updates_available is changed
block:
- name: Nomad drain node
command: nomad node drain -self -enable
register: result
retries: 3
delay: 3
until: result.rc == 0
- name: Update all packages to their latest version
ansible.builtin.package:
name: "*"
state: latest
- name: Reboot host
ansible.builtin.reboot:
reboot_timeout: 3600
- name: Nomad eligibility enable node
command: nomad node eligibility -self -enable
register: result
retries: 3
delay: 3
until: result.rc == 0
- name: Pause for some time (reduces cluster pressure)
ansible.builtin.pause:
minutes: 1
Explanation#
The statment serial: 1 does not only run the play one node after the other but also stop the play if an error occures (max_fail_percentage=100). This ensures that only one node will might be unusable and the cluster is still healthy.
- hosts: nomad_nodes
serial: 1
tasks:
...
The first task "Check updates available" checks for updates without the system being updated because it has check_mode: true. We store the output of it in a variable so that it can be used in the rest of the Playbook.
- name: Check updates available
ansible.builtin.package:
name: "*"
state: latest
check_mode: true
register: updates_available
This way, the Block Task "Process package updates" is only executed when updates are pending.
- name: Process package updates
when: updates_available is changed
block:
...