Automation and Orchestration: Notes

Automation & Orchestration (Part 1)

• Goal: Provide a top-level view of orchestration.
• Incentives for automation and orchestration.
• Decoupling policy from deployment—declarative approaches.
• Data models and declarative tools.

Bibliography

• [1] John Strassner, Policy-Based Network Management: Solutions for the Next Generation, Morgan Kaufmann, 2003
• [2] OpenStack Heat documentation
  • https://docs.openstack.org/heat/wallaby/template_guide/hot_guide.html
  • https://docs.openstack.org/heat/latest/template_guide/hot_spec.html
  • https://docs.openstack.org/heat/latest/template_guide/openstack.html
  • https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/AWS_AutoScaling.html
• [3] Links scattered in the presentation slides.

5G Network PoC Example

• Components:
  • 5GC components (Vendor 2).
  • PHY infrastructure: network, PHY appliances, DC infrastructure (RAN Vendor 1).
• Characteristics:
  • End-to-end (multi-domain).
  • Functions/appliances: PHY/virtual.
  • Network/connectivity: PHY/virtual/CNF.
  • Multi-tenancy: network slicing (E2E).
  • Multiple abstraction levels for management/orchestration.
  • Lifecycle management at each layer (provision -> assure).
• Orchestration Layers:
  • E2E orchestrator.
  • Service orchestration.
  • RAN management (vendor 1).
  • 5GCore management (vendor 2).
  • NFV, SDN Inter-DC network, SDN - DC network, BSS
  • Service (domain) Resource

Intro: Service Architecture Evolution Paradigm

• Progression from monolithic apps to microservices.
• Monolithic App:
  • Client-server architecture.
  • Includes Web App, Application, and Database.
• Microservices:
  • Utilizes VMs.
  • Web Frontend, Application, Database
• Increasing complexity/size with the paradigm shift.

Virtualization and Service Platform Evolution

• Services in virtualized environments require:
  • Efficient deployment capabilities.
  • Dynamic management capabilities (life-cycle management).
• Lifecycle Management (LCM) automation is not new.
• LCM covers:
  • Physical/virtualized infrastructure.
  • Service functions (components).
  • Services (sets of components).
  • Different stages of LC (onboarding, deployment, operation).
• Well-known examples of LCM automation:
  • Configuration management (Puppet, Chef).
  • Remote server deployment (Capistrano, Fabric).
  • Ad-hoc task execution.
• Newer approaches: CI/CD, etc.

Automation of Operational Tasks (LCM Tasks)

• Scripts:
  • Used from early days for operational tasks.
  • Sequences of instructions/operations to execute.
  • Still a key component of automated solutions.
  • Abstraction level and execution have evolved, but their imperative nature remains.
• Models:
  • Emerged with increasing complexity of SW.
  • Direct the evolution towards greater abstractions.
  • Imperative => declarative.
  • (Partially) manual => fully automated.
• Result:
  • Scripts → templates/blueprints (templates express models).
  • Build a system (template) once and orchestrate "100" times.

Scripting Approaches vs. Declarative Approaches

Declarative Approach

• Declarativeness allows separation of concerns: WHAT and HOW.
• Two interrelated notions (roles):
  • Model (WHAT): Description of the goal of the activity, of what is desired.
  • Orchestrator (HOW): Automatic executor, cares for providing the desired state.
• Model provides the orchestrator with only necessary information on the WHAT and remains generic.
• Ideally, independent of the runtime platform.
• WHAT is descriptive and user-level, HOW is imperative and hidden.
• Declarativeness holds the allure of no coding (“programming”).
• Implementation is hidden by the orchestrator.
• Service designer focuses on “target” (“final product”), not on “instructions”.

Large-Scale Automation Requirements

• Reliable monitoring:
  • Telemetry.
  • Data lakes, big data; calls for the use of AI.
• Programmability:
  • Isolation of low-level details through intelligent models.
  • Software-like interfaces to the resource layers (network, computing).

Telemetry

• Continuous monitoring with incremental changes.
• Push-based model (contrasted with SNMP's poll-based model, though SNMP traps allow push).
• Data in the form of time series (streaming telemetry).
• Data (raw/processed/aggregated) available to upstream systems using pub/sub communication (e.g., Kafka).
• Devices from different vendors (data) modeled in a common way (e.g., YANG).
  • See https://github.com/YangModels/yang).
  • Easier to reuse monitoring tools.
• Efficient transport protocol needed (e.g., gRPC).
• Observability vs. monitoring/telemetry.
  • Polling: requests for data (ad-hoc/regular basis).

Programmability – Data Models vs. Model Representation

• Programmability provides a management abstraction layer.
• Data model is key:
  • Configuration & state of modeled entity.
  • e.g., YANG (Yet Another Next Generation) with NETCONF/RESTCONF.
  • Comes from big vendors; operational complexity is high.
  • Tools are needed for efficient use.
  • Not supported widely enough yet.
• Model representation needed for communication:
  • e.g., YAML (YAML Ain't Markup Language).
  • Modeling used to map items to native config.
  • e.g., NAPALM, K8s, Ansible.

Declarative Tools/Standards (Configurations/Orchestration)

• OpenStack Heat/HOT:
  • Deploy elements of virtualized infrastructure (OpenStack/AWS).
• TOSCA:
  • Deployment of virtualized services (Topology and Orchestration Specification for Cloud Applications, OASIS).
• Ansible Playbooks:
  • Configure + elements of PHY/virt infrastructure and more.
• Kubernetes & related:
  • e.g., HELM, kpt (newer tool from Google).
• Terraform:
  • Provision infra as code (hyperscalers, K8s, OpenStack).
• SDN Network Programming:
  • e.g., ONOS Intent API.
• Older:
  • Puppet-Chef/Juju: config management & infra as code/application orchestration.

Three Concepts Around Declarative Approaches

• Configuration drift:
  • Discrepancy between actual and desired state of resource.
• Mutability of resources:
  • Mutable vs. immutable.
• Idempotence of manager:
  • Do not touch if no change is needed.
• Interaction between these concepts:
  • Desired state (reference) reconciled to current state.
  • Manager at model and resource level.
  • Immutable vs mutable resources.

Automation & Orchestration with K3s and Ansible Automation

• Ansible - architecture and basic operation.
• Ansible - main constructs
• Inventory
• Variables
• Modules, Tasks, Handlers
• Playbooks, Plays
• Roles
• Collections
• Other constructs (just mentioned)
• Note on idempotence, mutability and configuration drift

Bibliography

• [1] Jeff Geerling, Ansible for DevOps: Server and configuration management for humans, 2nd edition, 2020 and a series of youtube videos related to the book https://www.youtube.com/playlist?list=PL2_OBreMn7FqZkvMYt6ATmgC0KAGGJNAN and the book examples on github https://github.com/geerlingguy/ansible-for-devops
• [2] Jason Edelman, Network Automation with Ansible, O’Reilly, 2016
• [3] Michael DeHaan, Ansible - A Simple Model-Driven Configuration Management and Command Execution Framework http://highscalability.com/blog/2012/4/18/ansible-a-simple-model-driven-configuration-management-and-c.html

Ansible Overview

• Formerly, separate tools for:
  • Configurations management (Puppet, Chef, cfengine).
  • Server deployment tools (Capistrano, Fabric).
  • Ad-hoc task execution (e.g., plain SSH).
• Ansible is a configuration management tool, deployment tool, and ad-hoc task execution tool all in one.

Aims of Ansible

• Clear:
  • Ansible uses a simple syntax (YAML) and is easy to understand.
  • APIs are simple and sensible.
• Fast:
  • Fast to learn, fast to set up—especially considering you don’t need to install extra agents or daemons on all your servers.
• Complete:
  • Ansible does three things in one and does them very well.
  • You have everything you need in one complete package.
• Efficient:
  • No extra software on your servers means more resources for your applications.
• Secure:
  • Ansible uses SSH, and requires no extra open ports or potentially vulnerable daemons on your servers.

Ansible: How it Works (in a Nutshell) - Part 1

• Basic structure:
  • Playbook
  • Play
  • Task (implemented by a Module - operational part).
• Operation bases on detailed knowledge, called Ansible facts.
• Play typically changes Ansible facts.
• By default, Play starts with gather-facts.
  • The outcome depends on the results of previous plays.
  • Can be skipped (gather_facts: no).
  • Can be run anywhere needed using ansible.builtin.setup.
• Example:
  • Inventory:

    ``[group_A] host1 host2

[group_B] host2 hostN``

*   Playbook:

```

• name: hosts: group_A connection: local tasks:
  • name: module1-name: (module1 params)
  • name: module2-name: (module2 params)

name: hosts: group_B connection: local gather_facts: no tasks:

• name: task1 name module3-name: (module1 params)
• name: task2 name module4-name: (module2 params)     ```

Ansible: How it Works (in a Nutshell) - Part 2

• No dedicated software, uses Python and SSH on managed hosts.
• Overall Operation:
  • Log in to the Managed Nodes (through SSH).
  • Install Modules (Ansible-specific and Playbooks-related), written in Python.
  • Execute Playbook and/or Remote ad-hoc Commands.
  • Sets of instructions to be executed by modules on the managed node.
• Illustrative Diagram:
  • Ansible control node interacts with Managed Nodes via SSH.
  • Modules are transferred and executed on the target hosts.
  • Inventory defines the target hosts.
  • Dependencies: Python + SSH

Inventory File in Ansible

• Set of hosts to manage in the Playbook.
• Can be a dedicated file or the default /etc/ansible/hosts.
• Example:

[master]
kpi091

[node]
kpi092
kpi093
kpi094

[cluster:children]
master
node

[cluster:vars]
ansible_user=ubuntu
ansible_become_method=sudo
ansible_become_pass=raspberry
ansible_ssh_private_key_file=/home/xubuntu/.ssh/id_ed25519

• Hosts are grouped; a host can belong to multiple groups.
• It is possible to store variables within an inventory file.
• We use an own file hosts.ini filled in by a script install.sh.
• Can use INI or YAML format.

Variables in Ansible

• Idea: Access to host data throughout the whole play/playbook.
• Host variables:
  • Specific to a particular host.
• Group variables:
  • Common for a group of hosts; variable inheritance works for groups of groups.
• Can be defined in multiple locations:
  • Inventory file (our example hosts.ini).
  • Paths relative to inventory file and playbook file:
• Jinja2 templating used for dynamic expressions and access to variables and facts:

k3s_version: v1.25.5+k3s2
ansible_user: ubuntu
gateway: "{{ ansible_default_ipv4.gateway }}"
master_ip: "{{ hostvars[groups['master'][0]]['ansible_default_ipv4']['address'] }}"
k3s_token: "{{ hostvars[groups['master'][0]]['token'] }}"
worker_number: "{{ groups['node'].index(inventory_hostname) }}"

Ansible - Special Variables (Predefined)

• Ansible facts:

  • Data related to your remote systems (OS, IP addresses, filesystems, etc.).
  • Accessed via ansible_facts or hostvars.
  • Gathered at the beginning of each play.

• Magic variables:

  • Reflect internal state of running playbook; can’t be set directly.
  • Most common: hostvars, groups, group_names, inventory_hostname.
  • Example: $master\_ip: "{{ hostvars[groups['master'][0]]['ansible\_default\_ipv4']['address'] }}"$

• Connection variables:

  • Specify how to execute on targets.
  • Example from our lab inventory:

  [cluster:vars] ansible_user=ubuntu ansible_become_method=sudo ansible_become_pass=zzzzzzz     

Ansible Tasks, Conditionals

• Task:

  • Definition of an ‘action’ to be applied to the managed host.
  • Contained in a Play.
  • Calls a single module with parameters.
  • Can return a result (register) or be executed conditionally (when).

• Examples:

  ```yaml

  • name: Check if k3s already installed stat: path: /usr/local/bin/k3s register: k3s_installed

  • name: Install k3s on master shell: cmd: > curl -sfL https://get.k3s.io | … when: not k3s_installed.stat.exists     ```

• stat module retrieves file or file system status.

• shell module executes shell commands.

Ansible Modules

• Module ( “task plugins” or “library plugins”):

  • Discrete unit of code used by task
  • Can be used from the command line (ad-hoc) or in a playbook task.

• Examples ad-hoc:

  shell $ ansible all -i hosts.yaml -m ping $ ansible all -i hosts.yaml -b -B 1 -P 0 -m shell -a "sleep 5 && shutdown now"     

• Ansible executes each module, usually on the remote managed node, and collects return values.

Ansible Playbooks and Plays

• Playbook:

  • Ansible’s metaphor for configurations file;
  • Contains Plays (basic unit of Ansible execution).

• Play:

  • Begins with - name: <play name>.
  • Maps managed nodes (hosts) to tasks.
  • Contains variables, roles, and an ordered list of tasks.
  • Implicit loop over the mapped hosts and tasks.
  • Can start with gather_facts: false/true.

• Example:

  ```yaml

  Play 1

  • name: play1 name hosts: group_A connection: local tasks:
    • name: task1 name module1-name: (module1 params)
    • name: task2 name module2-name: (module2 params)

  Play 2

  • name: play2 name hosts: group_B …     ```

Ansible Handlers

• Special form of a Task that executes only when notified by a previous task.
• Helps in ensuring idempotence.
• Handlers run at the end of the Play.

tasks:
  - name: Template configuration file
    ansible.builtin.template:
      src: template.j2
      dest: /etc/foo.conf
    notify:
      - Restart apache
      - Restart memcached

handlers:
  - name: Restart memcached
    ansible.builtin.service:
      name: memcached
      state: restarted
  - name: Restart apache
    ansible.builtin.service:
      name: apache
      state: restarted

• Handlers are notified if a task results in a 'changed' status.

Ansible Roles

• Breaks playbook into multiple files.

• Automatically loads vars, files, tasks, handlers.

• Defined directory structure.

• k3s case:

  ```yaml

  Play for all cluster nodes (common configs).

  • hosts: cluster gather_facts: yes become: yes roles: # OS update/upgrade/useful package install role: upgrade # k3s-oriented configs common for all hosts
    • role: preparation

  Play for the k3s master node (master-specific).

  • hosts: master become: yes roles:
    • role: master

  Play for k3s worker nodes (worker-specific).

  • hosts: node become: yes roles:
    • role: node     ``` Main goals : to break playbook into multiple files. Roles are compose a complete playbook