KUBERNETES

CONTINUOUS

INFRASTRUCTURE

PLATFORM

Donovan Finch, SASAG President

Powered by Vimdeck

About Me

Donovan Finch AWS Cloud Support Engineer - Deployment services github.com/finchd www.linkedin.com/in/finchd @positvpessimist

Who? What?

Who is this talk for?

• I, SysAdmin
• those dreaming about the ultimate treehouse
• admins of medium (10+) systems

Who is Kubernetes (k8s) for?

• Already using Docker/containers
• Switching to Docker vs. Config Management
• Looking for a community-run cloud middleware; can't afford to build their own

What's in the talk?

• inane rambling
• high-level overview
• technical operations
• obligatory shiny demo

Docker!

Set a program's operating environment like it was a VM:

• memory/CPU/network ports
• file contents
• dependencies unpacked in-place

Then distribute the result as a single object, just like a VM image.

That's it!

Design Note: Restarting a container or deleting and making new should have the same effect. Store persistent state outside.

What is Kubernetes

and why is it so hard to spell?

• Greek for 'helmsman'/'pilot'
• the new hotness... from 2014
• FOSS from Google, now the Cloud Native Computing Foundation
• "Production-Grade Container Orchestration"
• multi-computer distributed service with a master-worker architecture
• name? tl;dr - licensing and Intellectual Property

k8s is still rapidly changing - any book is immediately out of date, kubernetes.io/docs/ are excellent but incomplete

Words! pods, volumes, labels, annotations, nodes/workers, masters

Config Management

. vs.

Immutable Architecture

CM agents are in a race to enforcement. Maybe your change is applied everywhere, maybe you missed one?

Read-Only production systems keep people mutating changes in other, safer places:

If you don't have staging systems, local laptop/desktop historically didn't match OS of Prod

Realtime vs Human-time

• Docker is fast enough to download its images, < 1min
• images are small enough compared to modern HDD/SSD sizing, can fit plenty more than commonly used
• startup time < 10sec
• container infra automation only adds <1 min to these startup times

Orchestration was always also required, whether you used CM or not

• running commands on multiple systems with inter-system dependencies
• "start the DB on server1 before httpd on server2"
• ssh, pssh, Ansible
• manual orchestration has a conductor, a la orchestra

Continuous Orchestration

Active/reactive The goal of "Automation" in general

Restart on container exit Recreate new containers on other workers when a worker stops responding update to newest version / rollback

"publish" - make reachable from outside cluster's CNI network usually makes a load balancer in cloud account or bind to a publically-routable IP address

keep data - Volumes abstract a place for permanent data - admins can set what kinds (LVM, glusterfs, NFS) are allowed and how big, by whom

kubernetes elements

Things you make in kubernetes

Pod: group of containers sharing a network namespace (setns), localhost, and one IP on the CNI network Ex. httpd, nrpe plugins, logstash/filebeat/fluentd log shipper in seperate containers

Volume+VolumeClaim: storage abstraction for persistent data

Deployment: plural of pod. run X copies of the pod simultaneously on multiple nodes, and X is mutable, but manually and through automation

Rollout: Created by an update to a deployment, can walk the timeline of rollouts, and revert

Service: Allow inbound traffic to a Deployment's pods. Types:

1. ClusterIP uses virtual IP in CNI network to proxy traffic by kube-proxy, and in-cluster-only dns (kube-dns)
2. NodePort uses a port on the worker on its "public" IP
3. LoadBalancer creates an LB, if the cloud the cluster is on is known
4. ExternalName uses in-cluster-only dns (kube-dns) to map to a CNAME in an external DNS

DaemonSet - Always run one of this pod per node (log shipper, etc)

Kubernetes components

kubernetes is made of plugins!

Worker components:

container runtime: Docker/rkt/cri-o

Container Network Interface (CNI)

• unique to k8s
• most commonly flannel: containers have a 10.x.x.x address, a "VPN" that spans to workers
• cloud providers have their own as well

Kubelet: the worker

• nvidia supplies a fork for doing gpu work (Machine Learning modelling) inside containers
• otherwise stick to the latest stable one - currently no competitors to the reference implementation

cAdvisor: performance metrics gathering

kube-proxy:

• accepts Layer 4 TCP/UDP, Layer 7 HTTP traffic on all Services' ports,
• transparently redirects to a worker running a pod of that Service

Master components: Worker components +

• etcd database
• API Server (kubernetes the app)
• Scheduler:
• controller manager: "controllers" each handle a non-scheduled pod type, like DaemonSets

kubernetes extras

Kubernetes Dashboard - Service+Deployment that comes pre-installed on most hosted k8s

Prometheus - FLOSS pull (polling) metrics server

fluentd DaemonSet - log shipping agent

Horizontal Pod AutoScaler - dynamic update of Service's pod count based on container metrics

Cluster AutoScaling - dynamically create/destroy cluster nodes based on cluster/container metrics (AWS EC2 AutoScaling Group, etc)

Installation

a dozen different installers aimed at various skill-levels/node types (VMware, Azure, GCE, AWS)

minikube - local single-VM with all components (Hyper-V, VirtualBox, KVM, xhyve, etc)

kubeadm - generic by-hand installer runs per node, you have to make nodes first

AWS: kops, tectonic, kube-aws, AWS EKS (hosted)

GCE: kube-up.sh, Google GKE (hosted)

Azure: ACS-Engine, Azure ACS (hosted)

Ansible-based kubespray

Usage

everything talks to the kube-apiserver REST JSON API

** kubectl with YAML

jsonnet meta-JSON

place YAML files in a folder on the master

Demo!

minikube on KVM at home

kops on AWS

EKS on AWS