Hi! I'm Liz, a Developer Advocate at honeycomb.io, and I spent my first weeks at the company doing security hardening of our infrastructure. I'd like to share what I'd learned with you, so that you can benefit from my reading of dozens of scattered pages of documentation and my ruling out of numerous dead ends.

Why you should take security and usability seriously

Developers and administrators have historically used SSH keys to provide authentication between hosts. By adding passphrase encryption, the private keys become resistant to theft when at rest. But what about when in use? Unfortunately, the usability challenges of re-entering the passphrase on every connection means that engineers began caching keys unencrypted in memory of their workstations, and worse yet, forwarding the agent to allow remote hosts to use the cached keys without further confirmation. The recent breach at Matrix underscores how dangerous it is to allow authenticated sessions to propagate across hosts and environments without a human in the loop.

Thus, we need solutions that prevent key theft from the systems we connect to, while maintaining ease of use. Two-factor authentication stops malicious automated propagation in its tracks by having a second factor protect use of our keys. There are two primary ways of preventing an attacker from misusing our credentials: either using a separate device that generates, using a shared secret, numerical codes that we can transfer over out of band and enter alongside our key, or having the separate device perform all the cryptography for us only when physically authorized by us.

Google, where I previously worked, employs short-lived SSH certificates issued by a central piece of infrastructure, stored on secure hardware tokens. But this is a serious change to developer workflow, and requires extensive infrastructure to set up. What will work for a majority of developers who are used to simply loading their SSH key into the agent at the start of their login session and SSHing everywhere?

Design considerations & threat models

I'm assuming that you have a publicly exposed bastion host for each environment that intermediates accesses to the rest of each environment's VPC, and use SSH keys to authenticate from laptops to the bastion and from the bastion to each VM/container in the VPC. If you don't yet have a bastion host and a VPC, start there!

It was important to me to make Honeycomb safe from compromise, even if malicious worm-like code were executed on a developer's laptop while SSH keys were unlocked, or if a developer accidentally forwarded an SSH agent to a hostile remote system. I also thought it important to build on existing work to disk encrypt all endpoints by ensuring the loss of physical control over a phone or hardware token could not itself grant production access. However, I consider it out of scope to prevent active local intervention and session hijacking (since someone who controls your active console or keyboard has you pretty well pwned).

I'm also assuming you have a mix of operating systems, hardware, and preferences about carrying dongles vs. wanting to use phones for second factor, etc.

How to get started!

First, start by enabling numerical time-based one time password (TOTP) for SSH authentication. Is it perfect? No, since a malicious host could impersonate the real bastion (if strict host checking isn't on), intercept your OTP, and then use it to authenticate to the real bastion. But it's better than being wormed or compromised because you forgot to take basic measures against even a passive adversary.

Server-side setup

You'll want a root shell open just in case, and the following snippets added to your Chef cookbooks (from this gist):

• metadata.rb
• attributes/default.rb (from attributes.rb)
• files/sshd
• recipes/default.rb (copy from recipe.rb)
• templates/default/users.oath.erb

Okay, now we can set this running on our hosts… and go through the client setup for ourselves at least.

Client-side setup

Now, each user authenticating needs a shared key to be present, encrypted, in SSM (or equivalent for your choice of cloud provider). Have each user install an OTP app such as Google Authenticator, Authy, Duo, or Lastpass, then do the following on their laptop:

Install dependencies:

brew install oath-toolkit OR apt install oathtool openssl

Generate a random base16 string to use as your key:

➜ openssl rand -hex 10
22ea2966afefd82660e1
##### ^^^ that's an example output used here - don't use it!

Convert it and put it into a phone-based authenticator app: Run oathtool -v [key] to convert it to the format (“Base32 secret”) that mobile authenticators use.

➜ oathtool -v 22ea2966afefd82660e1
Hex secret: 22ea2966afefd82660e1
Base32 secret: ELVCSZVP57MCMYHB
... more stuff down here we don't need

• For 1Password, add a one time password and enter the “Base32 secret” output from oathtool -v [key]
• For Duo, select “other” and use the Base32 secret.
• for Authy click “Enter key manually” and use the Base32 secret

Verify that generated codes are correct: Run oathtool --totp [key] and check that it returns the same value as your authenticator application.

➜ oathtool --totp 22ea2966afefd82660e1
693439

Store our key into the cloud secrets manager: Run aws ssm put-parameter --name /2fa/totp/$USER --value [key] --type SecureString --key-id alias/parameter_store_key to put your key into SSM Parameter Store. $USER should be the same as the username you use when you log in to a bastion. If you are updating the key instead of pushing it for the first time, add the --overwrite flag to the end of the command.

➜ aws ssm put-parameter --name /2fa/totp/lizf --value 22ea2966afefd82660e1 --type SecureString --key-id alias/parameter_store_key 
{
    "Version": 1
}

Log in for the first time: Now, when we ssh to the bastion host, we can ensure that the SSH agent can only be trampolined to other hosts within the VPC, but any attempt to programatically use from the outside the forwarded agent (or loaded in-memory keys) to access a bastion will fail because no TOTP from the separate mobile device was provided.

Let's check that we're asking for TOTPs:

➜ ssh -A bastion
Enter passphrase for key '[snip]': 
One-time password (OATH) for '[user]': 
Welcome to Ubuntu 18.04.1 LTS...

Now there's a value proposition for hardware auth…

People might get sick and tired of entering a numerical OTP every time they have to log into the bastion! It's almost like the old days of passphrase-encrypted SSH keys that motivated us to use agents! So let's leverage this inherent laziness to get people more, rather than less, secure!

Server-side setup

Change the beginning of files/sshd in your Chef module to begin as follows:

auth    required    pam_permit.so
auth    optional    pam_cap.so

# If it's a hardware or secure enclave SSH key, no need for a numerical OTP.
auth    sufficient      pam_ssh_agent_auth.so file=/etc/2fa_token_keys

# Check a TOTP code as a second resort, using a time slip of +/- 150 seconds.
auth    sufficient  pam_oath.so usersfile=/etc/users.oath digits=6 window=5

# People without OTPs will need to add an OTP secret to AWS SSM and wait an hour.
auth    requisite   pam_deny.so
...

And add the following additional lines to recipes/default.rb (a note to the nervous: my source modifications to openssh-server and libpam-ssh-agent-auth are available from Launchpad):

apt_repository 'openssl-pam-bindings' do
  uri 'ppa:honeycomb.io/ssh-2fa'
end

packages = %w{ openssh-server libpam-ssh-agent-auth }
packages.each do |p|
  r = package p do
    action :upgrade
  end
end

service 'sshd' do
  subscribes :reload, 'package[openssh-server]'
end

Now you'll need to use Chef to populate /etc/2fa_token_keys with keys that you know are generated and stored securely (e.g. using one of the below methods). I don't know how you maintain your lists of ssh key mappings to users, nor how you add ssh keys to your ~/.ssh/authorized_keys files, so I can't provide general advice.

Mac client setup

People with Touchbar Macs should use TouchID to authenticate logins, as they'll have their laptop and their fingers with them anyways. sekey lets us support this.

Install the binary:

brew cask install sekey

Add to ~/.ssh/config on your local machine:

IdentityAgent ~/.sekey/ssh-agent.ssh

Generate a key and export it:

sekey --generate-keypair "bastion key"
sekey --export-key $(sekey --list-keys|grep "bastion key"|grep --only-matching -E '[a-f0-9]{40}')

And then store the resulting key to /etc/2fa_token_keys and ~/.ssh/authorized_keys in Chef.

Krypt.co setup for iOS and Android

Instead of generating OTPs and sending them over manually with our fingers, our mobile devices can securely store our SSH keys and only remotely authorize usage (and send the signed challenge to the remote server) if a human presses a button on the phone.

This is the theory behind krypt.co, and is even more secure than a TOTP app so long as you supply appropriate parameters to force hardware coprocessor storage (NIST P-256 for iOS, and 3072-bit RSA for Android, on new enough devices). Make sure people use screen locks!

Follow the instructions here: https://krypt.co/docs/start/upload-your-ssh-publickey.html and then supply the generated key to both ~/.ssh/authorized_keys and /etc/2fa_token_keys in your Chef automation, and you won't be prompted for a TOTP.

YubiKey hardware token & Linux/ChromeOS client setup

Initial per-YubiKey setup

Follow these instructions from a Linux host to set up a basic working hardened YubiKey SSH key:

Install Dependencies

sudo apt-add-repository ppa:yubico/stable && sudo apt-get update
sudo apt-get install gpg yubikey-manager-qt pinentry-curses scdaemon pcscd
echo "reader-port Yubico YubiKey" > .gnupg/scdaemon.conf

Hardening to prevent a rogue host from authenticating without your permission

ykman openpgp touch sig on
ykman openpgp touch aut on
ykman openpgp touch enc on

Hardening in case your security key is stolen

gpg --change-pin

Default user pin is 123456 and admin pin is 12345678, change both of them to something more secure; they can both be the same PIN.

Generate a random 24-byte hex-encoded reset key and save it somewhere, GPG encrypted with your normal daily use keys (ykman-gui can generate a 24-byte string for you in “PIV → Configure PINs → Change Management Key”)

Generating the keys:

gpg --card-edit
admin
generate

• Input 4096 for all three modes (you'll need to enter the admin and user pins)
• Don't back up the stubs when prompted.
• Enter your full name and email address; make sure you leave a comment (e.g. desk computer) so you know which stub key is which in your GPG keyring.

Wait a minute, then enter the user PIN one more time, then wait about 5-10 minutes for the generation process to complete. It will print the UID of the master key before returning you to the card-edit prompt.

quit

gpg --export-ssh-key UID_of_master_key

This will print out the ssh pubkey string you'll need to add to the remote ~/.ssh/authorized_keys and /etc/2fa_token_keys in Chef.

Usage for authentication

Linux

Once the per-key setup is done, the configured Yubikey can be used in a Linux machine configured like so: gpg --with-keygrip -K

Save the keygrip of the master key you just generated to .gnupg/sshcontrol

Ensure that you have gpg-agent configured correctly: Set curses pinentry. Why? So you don't randomly get X passphrase/passcode prompts all over the place (esp remotely):

Edit ~/.gnupg/gpg-agent.conf to contain:

pinentry-program /usr/bin/pinentry-curses
enable-ssh-support

You'll update your ~/.bashrc to contain the following lines:

export SSH_AUTH_SOCK=$(gpgconf --list-dirs agent-ssh-socket)
export GPG_TTY=$(tty)
gpg-connect-agent updatestartuptty /bye >/dev/null

Run ssh-add -l to confirm you see your key in the list (it'll show 4096 SHA256:... cardno:... (RSA) in the listing).

When you ssh from a terminal into a bastion (remember to ssh -A for agent forwarding!), it'll prompt in the terminal that you most recently opened for your PIN on initial usage of the key. You'll complete that step, then tap your key to confirm. You're in!

ChromeOS

Install these two Chrome apps:

• https://chrome.google.com/webstore/detail/secure-shell-app/pnhechapfaindjhompbnflcldabbghjo
• https://chrome.google.com/webstore/detail/smart-card-connector/khpfeaanjngmcnplbdlpegiifgpfgdco

Then open the Secure Shell App (this won't work yet from the Crostini Terminal app because Crostini doesn't have USB pass-through yet, although it's coming in Chrome 75!)

Within the secure shell app's configuration screen for the bastion host:

• relay server option: --ssh-agent=gsc
• ssh option: -A

You'll then enter the user PIN when prompted, and tap the security key to confirm when logging into the bastion.

Further reading

The folks at krypt.co have written some fantastic blogs on securing SSH that go beyond the basic hardening I recommend here.

Hope this helps! Send me a Twitter DM (@lizthegrey) or email (lizf@honeycomb.io) if you have improvements to suggest!