Persistent Storage with Containers & The Growing Need for K8s Configuration Management

Kaslin Fields, SA @ Oracle Cloud Infrastructure

container attractions: fast, portable, scalable, immutable

immutable means that you need to separate data out - into volumes

WordPress as a mutable data-driven example app:

numerous data folders on disk, as well as the MySQL DB

Most volumes use Docker Volumes, released in July 2015, either local folders or plugins to clouds

Docker Volume Plugin on OCI for their file storage service

just mention driver when making containers with -v

k8s

k8s 1.0, also July 2015

1. Volumes, tied to pod lifetime

2. Persistent Volumes + Persistent Volume Claims

OCI PV driver

• https://github.com/oracle/oci-flexvolume-driver

• volumeMount with path
• nodeSelector in pod-def to choose an AZ
• volume declaration needs fsType
• create volumes outside of k8s, or use provisioner:

OCI PVC provisioner

• https://github.com/oracle/oci-volume-provisioner

• needs provisioner on each container host
• persistent volume claim,
• gets made at pod-create-time

The Growing Need for K8s Configuration Management

Lee Briggs, Senior Infrastructure Eng @ Apptio, https://github.com/jaxxstorm/

Apptio Kubernetes Platform

Config Management is well-known in Ops, but missing in k8s

The host needs to be set up - by a CM; still got to patch it

k8s is API-driven, idempotent, convergent

k8s plugins, etc -> "components":

• monitoring, Prometheus
• ingress controller
• Sealed Secrets in git
• RBAC config
• Cluster AutoScaler

Each cluster is a little different: region, SSL cert, etc Templating!

Puppet doesn't do well with understanding cluster-ness

solutions?:

1. Helm

• moved differences from kubectl yaml, to helm charts
• helm is a secuity nightmare, RBAC full power (wait for helm 3)
• templates in golang are not fun
• how to apply it? ansible or puppet?
• helm has many new issues, merges PRs slowly
• helm chart versioning wasn't being used

1. Ansible

• use API endpoint as your destination
• k8s_raw module, helm module is out of date
• still yaml
• setting up python deps
• still moving templating to another tool

1. Terraform

• HCL JSON
• k8s (official) & helm providers, not active
• easy to extend
• providers are buggy, not keeping up with k8s API changes

1. ksonnet @ heptio

• promising
• dramatically overcomplicated
• jsonnet language

1. kapitan

• jinja2 templates
• only takes pure jsonnet, no helm

jsonnet

• config language
• native go wrapper
• external variables
• compiles to JSON

1. wrote their own 'kr8' "create"

• jsonnet
• golang
• automatic population of external vars
• auto concatenate jsonnet levels
• patch helm charts
• can use other tools too: kasane , kustomize that use jsonnet
• library of folders for components and for clusters, each has vars declared params.jsonnet, and more jsonnet to render with those vars to fill
• supports patching, can change out ConfigMaps to Secrets, etc.
• cluster config, etc can be hierarchical to join values.
• map components to the clusters you want them running on

github.com/go-task/task - like make, Taskfile.yaml or .json helm.sh github.com/ksonnet/kubecfg validate kubectl yaml, runs

running git-sync & kubecfg in containers as a replicaset to pull the kr8 output and run it, as self-enforcement

other alternatives:

git-kube? https://gitkube.sh/

weaveworks' flux https://github.com/weaveworks/flux