AWS Best Practices

I’m a heavy AWS user — so heavy that, as the saying goes, “a day without food is fine; a day without AWS is not.” My daily routine amounts to “eat, sleep, use AWS.” Looking back over the past few years at Juniper and Tubi, and over my first month at Arcblock, I’ve gradually worked out some best practices for using AWS — and although these practices are aimed at AWS, they may hold some lessons for the way you approach other clouds too.

Principles

First, when using cloud services, we need to define a few basic principles:

• R1. Security and ease of use are irreconcilably at odds. We cannot sacrifice security to gain usability. So the only path is to build the simplest possible tools, preserving some ease of use while guaranteeing security.
• R2. No resource should have external access unless it is truly necessary.
• R2. Test accounts and production accounts must be completely isolated. The test account serves as a sandbox where engineers can attempt all sorts of high-risk experiments. The production account is reachable only through scripts with specific permissions, minimizing the risk of human error.
• R3. Continuously train the whole team to reduce its dependence on the Console until, eventually, no one uses the Console at all.

Practices

MFA

Multi-Factor Authentication must be enabled at the earliest possible moment. Every IAM user we create carries one precondition on any service it can access: MFA must be enabled. You can add a condition like this to an IAM policy:

{
  "Condition": {
    "Effect": "Allow",
    "Action": [
      "ec2:*",
      "s3:*"
    ],
    "Bool": {
      "aws:MultiFactorAuthPresent": true
      }
  }
}

This way, engineers must present MFA credentials when accessing EC2 / S3. Of course, you also need to allow users to register an MFA device while MFA is not yet enabled — otherwise you’ve checkmated yourself.

The trouble is that this makes life miserable for engineers, especially when accessing APIs with access keys — the nuisance never ends. To fix that, we can wrap STS, so that every other day you run something like this to obtain temporary session-level credentials:

Bastion / VPN server

In the martial world, everything comes down to one word: concealment. The fewer points you expose, the safer you are.

Bastion and VPN servers both exist to grant access to internal resources. For internal servers you don’t want to expose certain interfaces — SSH, RPC, and so on — yet whether for internal testing or for ad-hoc administrative access, you need some way in. That’s where a dedicated jump host or VPN server strikes a balance between security and usability.

For extra safety, the bastion / VPN server can also move well-known ports elsewhere — so a scanner can’t discover your bastion / VPN server simply by port scanning. For example, SSH listens not on port 22 but on port 9527.

SSH

Accessing a server over SSH to get things done is often necessary, but as the number of machines grows, managing SSH keys becomes a headache — using one or a handful of SSH keys and sharing the private key with the whole team is clearly not safe. What happens when an engineer leaves? Should the key you shared with them be revoked? If you revoke it, what about all those servers already using it? If you don’t, how do you guarantee security?

So the best approach is for every engineer to use their own key; there is never a situation where everyone shares one. When a machine is provisioned, create an account for each engineer and enable their SSH public key. But then problems arise: 1) Where do you find each engineer’s SSH public key? 2) What happens when a new engineer joins — or when one leaves?

For 1), if we tie users to IAM, we can pull the SSH key each user has uploaded to IAM; for 2), when people come and go, we simply add or remove users on each machine, without affecting anyone else.

Many companies deploy a bastion server to protect internal resources, yet the bastion server and the other servers use the same key. It’s like building a security architecture as bewildering as a set of Russian nesting dolls, only to find that every nested lock opens with the same key. That is not security.