8/21/2016 - 8:47 AM

Introduction to DC/OS

dcos.md

Rendered
Source

DC/OS

The latest documentation at https://dcos.io/docs/latest.

Mesosphere DC/OS 1.9

Announcing Mesosphere DC/OS 1.9
DC/OS 1.9.0 Release Candidate 2
Bringing Production-Proven Data Services to DC/OS 1.9 with our Partners
- DC/OS provides one-click installation of data services such as databases, message queues, and analytics engines, on-par with cloud providers such as Amazon Web Services.
- DC/OS utilizes two-level scheduling with frameworks to implement application-aware lifecycle management
- Open source SDK provides a high-level interface for building new stateful services on DC/OS
- With the SDK, developers can write a stateful service complete with persistent volumes, fault tolerance, and configuration management in about 100 lines of code.
DC/OS, an open source project to bring the power of Mesos to the broader community
- OCI and the Container Storage Interface (CSI)
- OCI, CSI, and CNI

Open Container Initiative (OCI)

How to install DC/OS on Mac OS X

NOTE: For now use the quite old Vagrant 1.8.4 with VirtualBox 5.0.26 due to bugs in the recent versions of Vagrant

Installation steps:

Install DC/OS with Vagrant
1. Download DC/OS Installer
2. Install DC/OS Vagrant
Local DC/OS cluster provisioning
1. export DCOS_CONFIG_PATH=etc/config-1.8.yaml
2. Deploy
  - vagrant up --provision m1 a1 p1 boot (see below)
  - Read PROVISIONING in the official documentation of Vagrant.

Starting and Stopping DC/OS with Vagrant

Using Minimal Cluster below (as the default small cluster might not be needed yet).

➜  dcos-vagrant git:(master) vagrant up m1 a1 boot
➜  dcos-vagrant git:(master) vagrant up m1 a1 boot
Bringing machine 'm1' up with 'virtualbox' provider...
Bringing machine 'a1' up with 'virtualbox' provider...
Bringing machine 'boot' up with 'virtualbox' provider...
==> m1: Box 'mesosphere/dcos-centos-virtualbox' could not be found. Attempting to find and install...
    m1: Box Provider: virtualbox
    m1: Box Version: ~> 0.8.0
==> m1: Loading metadata for box 'https://downloads.dcos.io/dcos-vagrant/metadata.json'
    m1: URL: https://downloads.dcos.io/dcos-vagrant/metadata.json
==> m1: Adding box 'mesosphere/dcos-centos-virtualbox' (v0.8.0) for provider: virtualbox
    m1: Downloading: https://downloads.dcos.io/dcos-vagrant/dcos-centos-virtualbox-0.8.0.box
    m1: Calculating and comparing box checksum...
==> m1: Successfully added box 'mesosphere/dcos-centos-virtualbox' (v0.8.0) for 'virtualbox'!
...
==> boot: DC/OS Installation Complete
==> boot: Web Interface: http://m1.dcos/

Stopping a DC/OS cluster.

➜  dcos-vagrant git:(master) vagrant halt m1 a1 p1 boot
==> boot: Attempting graceful shutdown of VM...
==> p1: Attempting graceful shutdown of VM...
==> a1: Attempting graceful shutdown of VM...
==> a1: Forcing shutdown of VM...
==> m1: Attempting graceful shutdown of VM...
==> m1: Forcing shutdown of VM...

NOTE: Had to destroy the DC/OS cluster using vagrant destroy -f as I could not provision the machines.

NOTE: vagrant up m1 a1 p1 boot worked before quite fine, but I don't know if I need it.

DC/OS CLI

Install dcos CLI library

$ curl -O https://downloads.dcos.io/binaries/cli/darwin/x86-64/dcos-1.8/dcos
$ chmod +x dcos

$ ./dcos help
Command line utility for the Mesosphere Datacenter Operating
System (DC/OS). The Mesosphere DC/OS is a distributed operating
system built around Apache Mesos. This utility provides tools
for easy management of a DC/OS installation.

Available DC/OS commands:

    auth            Authenticate to DC/OS cluster
    config          Manage the DC/OS configuration file
    help            Display help information about DC/OS
    job             Deploy and manage jobs in DC/OS
    marathon        Deploy and manage applications to DC/OS
    node            Administer and manage DC/OS cluster nodes
    package         Install and manage DC/OS software packages
    service         Manage DC/OS services
    spark           Run and manage Spark jobs
    task            Manage DC/OS tasks

Get detailed command description with 'dcos <command> --help'.

$ ./dcos --version
dcoscli.version=0.4.13
dcos.version=1.8.4
dcos.commit=e64024af95b62c632c90b9063ed06296fcf38ea5
dcos.bootstrap-id=5b4aa43610c57ee1d60b4aa0751a1fb75824c083

DC/OS Spark Service

Read Apache DC/OS Spark
The latest version of Apache DC/OS Spark is Spark 1.1.1-2.2.0 (with Apache Spark 2.2.0 being the latest too)
See https://github.com/mesosphere/dcos-spark/issues/25

➜  dcos-vagrant git:(master) pwd
/Users/jacek/dev/oss/dcos-vagrant

➜  dcos-vagrant git:(master) ./dcos config set core.dcos_url http://m1.dcos
[core.dcos_url]: set to 'http://m1.dcos'

➜  dcos-vagrant git:(master) ./dcos package install spark --cli
Installing CLI subcommand for package [spark] version [1.0.2-2.0.0]
Unable to install CLI subcommand. Missing required program 'virtualenv'.
Please see installation instructions: https://virtualenv.pypa.io/en/latest/installation.html

➜  dcos-vagrant git:(master) pip install virtualenv
Collecting virtualenv
  Downloading virtualenv-15.0.3-py2.py3-none-any.whl (3.5MB)
    100% |████████████████████████████████| 3.5MB 314kB/s
Installing collected packages: virtualenv
Successfully installed virtualenv-15.0.3

// you could use the following instead to debug the command
// ./dcos --log-level=debug package install spark
➜  dcos-vagrant git:(master) ./dcos package install spark --cli
Installing CLI subcommand for package [spark] version [1.0.2-2.0.0]
New command available: dcos spark

➜  dcos-vagrant git:(master) ls ~/.dcos/subcommands
spark

➜  dcos-vagrant git:(master) ./dcos --log-level=debug spark
Usage:
    dcos spark --help
    dcos spark --info
    dcos spark --version
    dcos spark --config-schema
    dcos spark run --help
    dcos spark run --submit-args=<spark-args>
                   [--docker-image=<docker-image>]
                   [--verbose]
    dcos spark status <submissionId> [--verbose]
    dcos spark log <submissionId>
                   [--follow]
                   [--lines_count=<lines_count>]
                   [--file=<file>]
    dcos spark kill <submissionId> [--verbose]
    dcos spark webui

➜  dcos-vagrant git:(master) ./dcos --log-level=INFO spark webui
http://m1.dcos/service/spark/ui

Quoting a discussion in DCOS slack channel:

A complete wipe of the spark pkg via dcos package uninstall + deleted spark related entries from the zookeeper via its exhibitor web ui. After a fresh re-install of Spark I was able to start spark drivers.

Marathon and Spark DCOS Package