Configuration
Management
Automation
2021-05-19
For lecture:
PV077: UNIX – programování a správa systémů II
Pavel Grochal
darkless@inuits.eu
Introduction
name: Pavel Grochal
nickname: darkless
email: darkless@inuits.eu ; pavel.grochal@inuits.eu
keyboard layouts: English(US), Czech (QWERTY)
favourite color: blue
favourite language: Python
favourite shell: fish
home igloo: Brno
Tech history
2004: HTML, CSS, PHP, SQL, JS – Websites (Firefox 1.0, IE 6, Opera 7)
2006: dualboot Linux - Ubuntu 6.10 (Edgy Eft)
2008: Bc. study FI MUNI, JAVA ecosystem, MVC PHP (CodeIgniter - thesis)
2011: Mgr. study FI MUNI, teaching - PB162 Programování Java
2012: PYTHON, DJANGO, IT Specialist @ Academy of Sciences
2013: Networking & Firewalls (CCNA certs)
2014: Virtualization (KVM, XEN, OpenShift, PXE), CFGMGMT (ANSIBLE, Chef)
2015+ OpenSource Consultant @ INUITS.eu
Overview
(1) Real-life story – don’t try this at home!
(2) Where does [CFG MGMT] fits in?
(3) Concepts of Configuration Management
(4) Building real-life example in Ansible
01
Real-life story – don’t try this at home!
Real-life story introduction
Task: Install Samba(SMB) server on Ubuntu
(used as network storage for employees)
Actions:
●
Prepare (Physical) Server, add DNS records, (setup other external services...)
●
boot Linux, install Linux – Ubuntu (USB / PXE)
●
Setup RAID storage (mdadm)
●
Setup Monitoring (Icinga)
●
Setup Backups (Tape DRIVE + Bacula)
●
Setup FW (iptables)
● Setup SMB server  Focus part!
How to setup SMB server?
1) Google “how to install SMB server”
(https://www.google.com/search?q=how+to+install+SMB+server)
2) Click on first link
(https://adrianmejia.com/how-to-set-up-samba-in-ubuntu-linux-and-access-it-in-mac-os-and-windows/)
3) Copy&paste commands to terminal
4) Profit !!!
FROM:
https://adrianmejia.com/how-to-set-up-samba-in-ub
untu-linux-and-access-it-in-mac-os-and-windows/
Production Overview
Ransomware Attack!
●
One of the employees computer was
compromised (Windows 7 Laptop)
●
Shared network disk was encrypted
demanding ransom
●
Gained access to server due to security
issue in SMB service
Attack Overview
First DAY mitigation
●
Remove server from Network
●
Inspect server via HW console in Data center
●
Backup whole OS disk (dd) - for the future analysis
●
(User’s data are backed on TAPE DRIVE and restorable)
●
Stop server
Difficulties
●
Server was running for 1 year periodically
maintained.
●
Nobody knew what was ACTUALLY installed
on server. Only what SHOULD be installed.
●
Custom-tweaked configurations for different
users.
Rinse & Repeat?
●
Reinstall compromised Linux server – Ubuntu (USB / PXE)
●
Setup RAID storage (mdadm)
●
Setup Monitoring (Icinga)
●
Setup Backups (Tape DRIVE + Bacula)
●
Setup FW (iptables)
●
Setup SMB server (again!)
There has to be another way!
Configuration
Management
Where does [CFG MGMT] fits in?
02
What is Automation?
Google: “What is Automation?”
“Automation is the use of technology to
perform tasks with reduced human assistance.”
* https://www.redhat.com/en/topics/automation
IT Automation maybe?
“IT automation, sometimes referred to as infrastructure
automation, is the use of software to create repeatable
instructions and processes to replace or reduce human
interaction with IT systems. Automation software works within the
confines of those instructions, tools, and frameworks to carry out
the tasks with little to no human intervention.”
* https://www.redhat.com/en/topics/automation/whats-it-automation
IT Automation topics
●
Provisioning
●
Configuration Management
●
(Container) Orchestration
●
IT migration
●
Application deployment (CI/CD)
●
Infrastructure as Code (IaC)
Provisioning
“Provisioning is the process of setting up IT infrastructure. It can also refer to the steps
required to manage access to data and resources, and make them available to users and
systems.
Provisioning is not the same thing as configuration, but they are both steps in the
deployment process. Once something has been provisioned, the next step is configuration.
When the term "provisioning" is used, it can mean many different types of provisioning,
such as server provisioning, network provisioning, user provisioning, service provisioning,
and more.”
* https://www.redhat.com/en/topics/automation/what-is-provisioning
Configuration Management
“Configuration management is a process for
maintaining computer systems, servers, and
software in a desired, consistent state. It’s a way
to make sure that a system performs as it’s expected
to as changes are made over time. ”
* https://www.redhat.com/en/topics/automation/what-is-configuration-management
Orchestration
“Orchestration is the automated configuration, management, and coordination of
computer systems, applications, and services. Orchestration helps IT to more easily
manage complex tasks and workflows.
Automation and orchestration are different, but related concepts.
In general, automation refers to automating a single task. This is different from
orchestration, which is how you can automate a process or workflow that
involves many steps across multiple disparate systems.”
* https://www.redhat.com/en/topics/automation/what-is-orchestration
Container Orchestration
“Container orchestration automates the deployment,
management, scaling, and networking of
containers. Enterprises that need to deploy and
manage hundreds or thousands of Linux® containers
and hosts can benefit from container orchestration.”
* https://www.redhat.com/en/topics/containers/what-is-container-orchestration
IT Migration
“An IT migration is the shifting of data or software
from one system to another. Depending on the
project, an IT migration could involve one or more kinds
of movement: Data migration, application migration,
operating system migration, and cloud migration.”
* https://www.redhat.com/en/topics/automation/what-is-it-migration
Application deployment (CI/CD)
“Continuous integration (CI) is the practice of merging all developers' working
copies to a shared mainline several times a day.”
“Continuous delivery (CD) is a software engineering approach in which teams
produce software in short cycles, ensuring that the software can be reliably
released at any time and, when releasing the software, without doing so manually.”
“Continuous deployment (CD) is a software engineering approach in which software
functionalities are delivered frequently through automated deployments.”
* https://en.wikipedia.org/wiki/Continuous_integration
* https://en.wikipedia.org/wiki/Continuous_delivery
* https://en.wikipedia.org/wiki/Continuous_deployment
Commonly Used Open Source Services for Ops
●
Provisioning: Packer, Terraform
●
Configuration Management: Puppet, Ansible, Chef, SaltStack
●
Container Orchestration: Kubernetes, Nomad
●
Continous Integration/Delivery: Jenkins, GitLab CI
●
Web servers: Apache, Nginx, Caddy
●
Load Balancers (including TLS termination): Nginx, HAProxy
●
Java application deployment: JBoss, Wildfly, Tomcat
●
Databases: MySQL, Postgres, CouchDB, MongoDB
●
Backup: rsync, Bacula
●
Central log aggregation: ELK, Fluentd, Graylog, Loki
●
Metrics and monitoring: Zabbix, Icinga, Prometheus + Grafana
●
DNS server: Bind, PowerDNS
●
Virtualization: qemu/kvm, VirtManager, OpenNebula, Proxmox
Infrastructure as Code (IaC)
“Infrastructure as Code (IaC) is the managing and provisioning of
infrastructure through code instead of through manual processes.
With IaC, configuration files are created that contain your
infrastructure specifications, which makes it easier to edit and
distribute configurations. It also ensures that you provision (and
configure) the same environment every time.”
* https://www.redhat.com/en/topics/automation/what-is-infrastructure-as-code-iac
* https://en.wikipedia.org/wiki/Infrastructure_as_code
General IaC Requirements
3 types of tools needed
Provisioning:
●
Create me an instance of asset X
●
Container instance
●
VM instance
●
Service X configuration via API
Configuration Management (Desired state, Continuous Configuration Automation, ...):
●
Ensure that this file present / service is always running
●
Set X with these permissions
●
Ensure User Removed
Orchestration:
●
Non frequent
●
Trigger action X on resource Y based on characteristics A,B and or C
●
First do X here then do Y there
●
One off actions
Brief intro into Provisioning Tools
Packer: Automates the creation of any type of machine
image.
Terraform: Codifies cloud APIs into declarative
configuration files, i.e.: Infrastructure as Code.
Pulumi: similar to Terraform, in that you create, deploy,
and manage Infrastructure as Code on any cloud.
Packer
Creates machine images from code:
1)launch virtual machine
2)install operating system
3)perform base configuration(Debian / Ubuntu: preseed or user-data, CentOS / RHEL: kickstart)
●
partitioning
●
base packages
●
users
●
networking
4)derives machine image from created virtual machine.
The resulting machine image can be used to launch new virtual machines from.
Terraform
●
Describes infrastructure as code and manages described infrastructure
●
init Terraform and providers used (e.g. Qemu, AWS, …)
●
validate if the configuration is correct
●
plan the required changes to achieve the described infrastructure state.
●
apply the required changes.
●
destroy previously declared infrastructure.
●
Functionality is not limited to virtual machines.
●
Hashicorp Configuration Language (HCL).
●
State Management stored in files
Pulumi
●
Similar to Terraform
●
Supports and uses general purpose languages (Python,
JavaScript, Go, C#, …)
●
Excellent code testing
●
Mid-sized community
●
State Management stored online
* https://www.pulumi.com/docs/intro/vs/terraform/#pulumi-vs-terraform
Concepts of Configuration Management
03
Concepts of CFG MGMT
●
Server-client / push-pull models
●
Imperative / Declarative models
●
Desired State
●
Idempotence
●
Comparison of Configuration Management tools
●
Puppet
●
Ansible
●
Chef
●
SaltStack
Topology Visualization
Push vs Pull model
Push model
Configuration Management Server pushes the configurations on the nodes.
1)Server obtain the current state of the node
2)Compare the state of the node with the stored configuration
3)Perform actions on the node to match the configuration
Pull model
Nodes (regularly) pull changes from the Configuration Management Server.
1)Node obtain the stored configuration from server - requires dedicated client
2)Compare the configuration with the current state of the node
3)Perform actions on the node to match the configuration
Imperative vs Declarative
Imperative model
●
Concept similar to Imperative Programming languages (Python, Java, PHP, ...)
●
States HOW things should be done.
●
Mostly includes Implementation details.
●
Example: Recipe for baking a cake.
Declarative model
●
Concept similar to Declarative Programming languages (Haskell, Proglog, Lisp, ...)
●
States WHAT the end result should be.
●
Should exclude Implementation details.
●
Example: Photo of how the cake should look like.
Desired State
●
Corresponds with Declarative model.
●
Setup Managed Nodes into an expected state.
●
Enforces consistency, reproducibility and
automation.
* https://puppet.com/docs/puppet/7.6/puppet_overview.html#why_use_puppet
Idempotence
“Idempotence (UK: /ˌɪdɛmˈpoʊtəns/, US: /ˌaɪdəm-/)
is the property of certain operations in mathematics
and computer science whereby they can be applied
multiple times without changing the result beyond the
initial application.”
* https://en.wikipedia.org/wiki/Idempotence
Idempotence Example - wrong
darkless@khajiit:~$ mkdir /var/backups/ ; echo $?
0
darkless@khajiit:~$ mkdir /var/backups/ ; echo $?
mkdir: cannot create directory ‘/var/backups/’: File exists
1
Create folder /var/backups/
Idempotence Example - better
darkless@khajiit:~$ rmdir /var/backups/
darkless@khajiit:~$ [ -d /var/backups/ ] || mkdir /var/backups ; echo $?
0
darkless@khajiit:~$ [ -d /var/backups/ ] || mkdir /var/backups ; echo $?
0
Puppet vs Ansible vs Chef vs SaltStack
●
DSL (Puppet DSL)
●
Pull model
●
Client required
●
YAML (Python)
●
Push model
●
No client (SSH)
●
Pull model
●
Client (Ansible Pull)
●
DSL(Ruby)
●
Pull model
●
Client required
* https://medium.com/successivetech/chef-vs-puppet-vs-ansible-vs-saltstack-a-complete-comparison-9af8f1790c0d
●
YAML(Python)
●
Pull model
●
Client required
●
Push model
●
No client (Salt
SSH)
Puppet Topology
* This is SIMPLIFIED overview. See detailed information in respective guide.
Puppet – create folder
* https://www.puppetcookbook.com/posts/creating-a-directory.html
class directories {
file { '/var/backups':
ensure => 'directory',
owner => 'darkless',
group => 'games',
mode => '0750',
}
}
Ansible Topology
* This is SIMPLIFIED overview. See detailed information in respective guide.
Ansible – create folder
- name: Create a /var/backups/ directory
ansible.builtin.file:
path: /var/backups
state: directory
owner: darkless
group: games
mode: '0755'
* https://docs.ansible.com/ansible/latest/collections/ansible/builtin/file_module.html
Chef Topology
* This is SIMPLIFIED overview. See detailed information in respective guide.
Chef – create folder
directory '/var/backups' do
owner 'darkless'
group 'games'
mode '0755'
action :create
end
* https://docs.chef.io/resources/directory/
SaltStack Topology
* This is SIMPLIFIED overview. See detailed information in respective guide.
SaltStack – create folder
/var/backups:
file.directory:
- user: darkless
- group: games
- mode: 755
* https://docs.saltproject.io/en/latest/ref/states/all/salt.states.file.html
Building real-life example in Ansible
04
Real-life example
●
Targeted result overview
●
Manual VM Provisioning
●
Automatic configuration management
Target result overview
Manual VM Provisioning
●
Install VirtualBox
●
Create new Virtual Machine
●
Install Ubuntu Server on Virtual Machine
●
Setup SSH Port Forwarding on VirtualBox
●
Test SSH connection from localhost
(Laptop)
Install Virtualbox
darkless@khajiit:~$ apt show virtualbox
Package: virtualbox
Version: 6.1.16-dfsg-6~ubuntu1.20.04.1
https://www.virtualbox.org/wiki/Downloads
Download Ubuntu Server
https://releases.ubuntu.com/20.04.2/ubuntu-20.04.2-live-server-amd64.iso
Create new Virtual Machine
Install Ubuntu Server on Virtual Machine
Setup SSH Port Forwarding on VirtualBox
Test SSH connection from localhost (Laptop)
Extras: VirtualBox bridged adapter
Virtual Box Networking Modes
* https://www.virtualbox.org/manual/ch06.html
Extras: Exporting Virtual Machine
Automatic configuration management
●
Basic Ansible concepts
●
Install Ansible
●
Ansible Best Practices
●
Setup simple Ansible Playbook
Basic Ansible Concepts
●
Control Node
●
Managed Nodes
●
Inventory
●
Playbook
●
Task
●
Modules
●
Collections
●
Handlers
●
Variables
Control Node
●
Any machine with Ansible installed.
●
You can use any computer that has a
Python installation as a control node.
●
You can have multiple control nodes.
https://docs.ansible.com/ansible/latest/network/getting_started/basic_concepts.ht
ml#control-node
Managed Nodes
●
The network devices (and/or servers) you
manage with Ansible.
●
Managed nodes are also sometimes called
“hosts”.
●
Ansible is not installed on managed nodes.
https://docs.ansible.com/ansible/latest/network/getting_started/basic_concepts.ht
ml#managed-nodes
Inventory
●
List of Managed Nodes and their groups
(Ansible knows how to reach those Nodes)
●
Sometimes called a “hostfile”
●
Defaults to /etc/ansible/hosts
●
Custom file can be used (eg. hosts.yml)
●
Dynamic Inventories – for Cloud solutions
https://docs.ansible.com/ansible/latest/user_guide/intro_inventory.html#inventory-
basics-formats-hosts-and-groups
Playbook
●
A playbook is composed of one or more ‘plays’
in an ordered list.
●
Each play executes part of the overall goal of the
playbook, running one or more tasks.
●
Each task calls an Ansible module.
https://docs.ansible.com/ansible/latest/user_guide/playbooks_intro.html
Task
●
The units of action in Ansible.
●
You can execute a single task once with an adhoc
command.
●
Each task calls an Ansible module.
https://docs.ansible.com/ansible/latest/network/getting_started/basic_concepts.ht
ml#tasks
Modules
●
The units of code Ansible executes.
●
Example: File Module
●
https://docs.ansible.com/ansible/latest/collections/ansible/builtin/file_module.html
https://docs.ansible.com/ansible/latest/network/getting_started/basic_concepts.ht
ml#modules
Ansible Task which uses File Module
- name: Create a /var/backups/ directory
ansible.builtin.file:
path: /var/backups
state: directory
owner: darkless
group: games
mode: '0755'
* https://docs.ansible.com/ansible/latest/collections/ansible/builtin/file_module.html
Collections
●
Collections are a distribution format for Ansible
content that can include playbooks, roles, modules, and
plugins.
●
You can install and use collections through Ansible
Galaxy.
https://docs.ansible.com/ansible/latest/network/getting_started/basic_concepts.ht
ml#collections
https://docs.ansible.com/ansible/latest/galaxy/user_guide.html
Handlers
●
Handlers are tasks that only run when notified.
●
Each handler should have a globally unique name.
●
Example:
●
restart a service if a task updates the configuration of that service
●
do nothing if the configuration is unchanged
https://docs.ansible.com/ansible/latest/user_guide/playbooks_handlers.html
Variables
●
Ansible uses variables to manage differences between systems.
●
You can define these variables in:
●
Playbooks
●
Inventory
●
Re-usable (variable) files
●
Roles
●
Command line
●
Define variable: remote_install_path: /opt/my_app_config
●
Use variable: dest: '{{ remote_install_path }}/foo.cfg'
https://docs.ansible.com/ansible/latest/user_guide/playbooks_variables.html
Install Ansible
https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html
darkless@khajiit:~$ apt show ansible
Package: ansible
Version: 2.9.6+dfsg-1
NOTE: Some functionaly used in example requires `sshpass` package installed
Ansible Best Practices
https://docs.ansible.com/ansible/latest/user_guide/playbooks_best_practices.html
●
General tips
●
Playbook tips
●
Inventory tips
●
Execution tips
Setup simple Ansible Playbook
●
Clone Simple Ansible Skeleton (Playbook)
●
Explore Skeleton Structure
●
Enhance playbook:
●
Setup basic server hardening for all hosts
●
Install Docker via roles from Ansible-Galaxy
●
Deploy example Docker app on subset of hosts
Simple Ansible Skeleton
Git clone:
https://github.com/Darkless012/ansible_tutorial
Or fork :)
Explore Skeleton Structure
tasks/
all/
hostname.yaml
vars/
all/
external_vars.yml.example → external_vars.yml
.gitignore
handlers.yml
hosts.yaml.example → hosts.yaml
playbook.yml
Setup basic server hardening for all hosts
IDK GOOGLE?
●
https://www.google.com/search?q=ubuntu+server+20.04+hardening
HERE YA GO:
●
https://www.informaticar.net/security-hardening-ubuntu-20-04/
●
https://implex.io/posts/ubuntu-20-04-setup/
●
https://gist.github.com/lokhman/cc716d2e2d373dd696b2d9264c0287a3
Hardening steps (from internet)
In this tutorial:
●
System Updates
●
Create Non-root User
●
Disable Root User (SSH and system)
●
Configure SSH settings
●
Firewall setup
●
Install Fail2Ban
●
Sysctl.conf
●
Secure Shared Memory
●
Set Hostname and Host File
●
Set Locale and Timezone
Out of scope
!!! DO try this at home !!!
●
2FA
●
Install AntiVirus
●
Add Swap
●
Set Security Limits
●
IP Spoofing
Install Docker via collections (Ansible-Galaxy)
●
Geerling Guy – check his playbooks
●
https://github.com/geerlingguy/ansible-role-docker
●
roles and collections defined requirements.yml
●
Install “dependencies” via:
ansible-galaxy install -r requirements.yml
Deploy example Docker app
IDK GOOGLE?
●
https://www.google.com/search?q=hello+world+web+app+docker
HERE YA GO:
●
browsing
●
browsing
●
This could be it: https://github.com/crccheck/docker-hello-world
INUITS bvba
Essensteenweg 31
2930 Brasschaat
Belgium
BE 0891.514.231
Contact:
+32.380.821.0
5
info@inuits.eu
inuits.eu
INUITS bvba
Essensteenweg 31
2930 Brasschaat
Belgium
BE 0891.514.231
Contact:
+32.380.821.05
info@inuits.eu
inuits.eu
Pavel Grochal
pavel.grochal@inuits.eu
INUITS s.r.o.
Brno Igloo
Hybešova 985/30
602 00 Brno
Prague Igloo
ImpactHub
Drtinova 557/10
150 00 Prague