The story behind the OpenBSD AMI, and how it integrates with AWS, and a demo of dynamic VPN built using Consul (all running OpenBSD)

1. OpenBSD and AWS
September 23rd 2017EuroBSDcon

2. @eurobsdcon
Who am I?
2
Laurent Bernaille @d2si
• Linux background, getting to know OpenBSD
• Cloud enthusiast
• Love discovering, building (and breaking…) new things
@lbernail

3. @eurobsdcon
What is this presentation/demo about?
OpenBSD and AWS
• The first OpenBSD image and the ongoing work
• The integration in the AWS ecosystem
OpenBSD and microservices
• How we can leverage OpenBSD for cloud applications
• Examples and demo
OpenBSD and me
• A recent but interesting journey

4. @eurobsdcon
OpenBSD on AWS
First image by @ajacoutot (December 2015, in 5.9)
• Not straightforward due to Xen support (network, disk in particular)
• Intro: http://blog.d2-si.fr/2016/02/15/openbsd-on-aws/
• Details: https://github.com/ajacoutot/aws-openbsd
• More details: http://www.openbsd.org/papers/bsdcan2016-xen.pdf
=> The image worked, but without EBS (disk) support at first
=> Xen support was not perfect
An AWS hypervisor update broke the AMI (late 2016)
Fixed in 6.1, thanks to Mike Belopuhov and @esdenera
Many improvements in 6.2 (performances)

5. @eurobsdcon
Let's have a look

6. @eurobsdcon
Where does my public key come from?
AWS exposes a metadata web server at http://169.254.169.254

7. @eurobsdcon
OK but how did it get into authorized_keys?
Linux distributions rely on cloud-init
• http://cloudinit.readthedocs.io/
• Origin in ubuntu cloud
• Cloud-init does a lot of things and is very Linux specific
Enters ec2-init by @ajacoutot
• Minimal cloud-init implementation
• https://github.com/ajacoutot/aws-openbsd
When is it run?
• By netstart (very early in the boot process)

8. @eurobsdcon
A quick look at ec2-init
mock_pf open
if [[ $(mock meta-data/instance-id) != $(cat /var/db/ec2-init 2>/dev/null) ]]; then
ec2_instanceid
ec2_pubkey
ec2_hostname
ec2_userdata
ec2_fingerprints
sysclean
fi
mock_pf close
open pf to allow access to metadata server
check if already configured
write instance id to db file to set instance as "configured"
write public key in authorized_keys file
set hostname from AWS metadata
execute userdata (more on that later)
write rc.firsttime script to display ssh fingerprints after boot
clean up instance (remove old ssh keys, logs, dhcp data)

9. @eurobsdcon
What about this ec2-user?
Standard behavior on AWS
• No connection as root
• ec2-user is used for Amazon Linux, Redhat, Fedora, Centos, FreeBSD
• Debian uses "admin" and ubuntu, "ubuntu"
ec2-user has unlimited doas with "nopass"
$ cat /etc/doas.conf
permit nopass ec2-user

10. @eurobsdcon
Let's use this instance
Install terraform
$ pkg_info -Q terraform
terraform-0.9.2
$ doas pkg_add terraform
Terraform?
• Describe infrastructure components and build them
• « puppet » for infrastructure
• Alternatives: cloudformation / heat
OK let's set up something with it
$ doas pkg_add git
$ git clone git@github.com:lbernail/eurobsdcon2017.git
$ terraform init
$ terraform plan
$ terraform apply

11. @eurobsdcon
Under the hood
Bastion
eu-west-1a
Public subnets
Private subnets
eu-west-1b
Public subnets
Private subnets
resource "aws_vpc" "main" {
cidr_block = "10.100.0.0/16"
}
resource "aws_subnet" "public" {
vpc_id = "${aws_vpc.main.id}"
cidr_block = "10.100.1.0/24"
tags { Name = "Main" }
}
resource "aws_instance" "bastion" {
ami = "${var.bastion_ami}"
instance_type = "t2.micro”
subnet_id = "${aws_subnet.public.id}"
vpc_security_group_ids = [ "${aws_security_group.bastion.id}" ]
tags { Name = "bastion" }
}

12. @eurobsdcon 12
Bastion
Public subnets
NAT
GW
Public subnets Public subnets
What did we just build?
Private subnets Private subnetsPrivate subnets

13. @eurobsdcon 13
Bastion
Public subnets
NAT
GW
Public subnets Public subnets
CAg
(UI)
CS
Let’s create a consul cluster
10.0.128.100
consul0
CS
10.0.129.100
consul1
CS
10.0.130.100
consul2
10.0.128.200
consul-agent

14. @eurobsdcon
A quick intro to consul
From Hashicorp (authors of vagrant, packer, terraform, vault)
Used for microservices
• Service discovery
• Key-value store for configuration
Resilient
• Distributed system
• Built on RAFT

15. @eurobsdcon
Let's look at it
$ ssh 10.0.128.100
$ consul members

16. @eurobsdcon
OK but how did it all get configured?
Userdata: script to bootstrap AWS instances (executed by ec2-init)
$ ftp -MVo - http://169.254.169.254/latest/user-data
#!/bin/sh
pkg_add consul
cat > /etc/consul.d/config.json <<EOF
{
"bootstrap_expect": 3,
"server": true,
"node_name": "consul0",
"retry_join_ec2" :
{
"tag_key": "ConsulCluster",
"tag_value": "Consul"
}
}
EOF
rcctl enable consul
cat >> /etc/rc.firsttime <<EOF
rcctl start consul
EOF
install consul
this node is a server called consul0
it will wait for 2 other servers to bootstrap cluster
rely on AWS API to discover members
- instances have a "tag"
- instances have a role granting them access to AWS APIs
"enable" writes to /etc/rc.conf.local
but rc parses rc.conf.local very early so consul won't start
=> we use rc.firsttime

17. @eurobsdcon
What can we do with this?
Dynamic VPN configuration with consul-template
• A companion tool to Consul
• Watches for key changes in Consul
• Generates a file from a template
• Optionally executes a command when the file changes
Let's build a VPN gateway
$ cd ../vpn
$ terraform init
$ terraform apply

18. @eurobsdcon 18
Bastion
Public subnets
NAT
GW
Public subnets Public subnets
CAg
(UI)
CS
New architecture
10.0.128.100
consul0
CS
10.0.129.100
consul1
CS
10.0.130.100
consul2
10.0.128.200
consul-agent
VPN
10.0.0.10

19. @eurobsdcon
What is this VPN server? 1/2
$ ftp -MVo - http://169.254.169.254/latest/user-data
#!/bin/sh
rcctl enable ipsec
rcctl enable isakmpd
rcctl set isakmpd flags -K
install -m 0600 /dev/null /etc/ipsec.conf
pkg_add consul
cat > /etc/consul.d/config.json <<EOF
{
"server": false,
"node_name": "vpn",
"retry_join_ec2" :
{
"tag_key": "ConsulCluster",
"tag_value": "Consul"
}
}
EOF
enable ipsec
install consul
configure it as a client

20. @eurobsdcon
What is this VPN server? 2/2
pkg_add consul-template
cat > /etc/consul-template.d/default.conf << EOF
consul {
address = "127.0.0.1:8500"
}
template {
source = "/etc/consul-template.d/ipsec.ctmpl"
destination = "/etc/ipsec.conf"
perms = 0600
command = "ipsecctl -f /etc/ipsec.conf || echo Invalid ipsec configuration"
}
EOF
# Template
cat > /etc/consul-template.d/ipsec.ctmpl << 'EOF'
{{ range tree "vpn" | explode -}}
{{ if and .cidrblock .endpoint .psk -}}
ike esp from 10.0.0.0/16 to {{ .cidrblock }}
peer {{ .endpoint }}
srcid 34.252.210.92
psk "{{ .psk }}"
{{ end -}}
{{ end }}
EOF
install consul-template
use local consul
Template configuration
- template file
- target
- command to execute on change
template file to generate ipsec.conf

21. @eurobsdcon
The template file
{{ range tree "vpn" | explode -}}
{{ if and .cidrblock .endpoint .psk -}}
ike esp from 10.0.0.0/16 to {{ .cidrblock }}
peer {{ .endpoint }}
psk "{{ .psk }}"
srcid 34.252.210.92
{{ end -}}
{{ end }}
get all keys under "vpn"
iterate over them
transform items in maps
if we have values for all necessary keys
generate ipsec configuration
configuration keys
local public IP (injected by terraform)
vpn/
/us/
/cidrblock = 172.30.0.0/16
/endpoint = 32.32.32.32
/psk = demo
ike esp from 10.0.0.0/16 to 172.30.0.0/16
peer 32.32.32.32
psk "demo"
srcid 34.252.210.92

22. @eurobsdcon
Let's look at this
$ consul members
$ rcctl check consul consul_template
$ cat /etc/consul-template.d/ipsec.ctmpl
$ doas cat /etc/ipsec.conf
$ doas ipsecctl -s all

23. @eurobsdcon
Building our VPN
Bastion
Public subnets
NAT
GW
Public subnets Public subnets
CAg
(UI)
CS
10.0.128.100
consul0
CS
10.0.129.100
consul1
CS
10.0.130.100
consul2
10.0.128.200
consul-agent
VPN
10.0.0.10
Ireland, 10.0.0.0/16
Virginia, 172.30.0.0/16
EIP: 34.252.210.92
Demo 172.30.x.y
allow ICMP from 10.0.0.0/16

24. @eurobsdcon
Conclusion and perspectives
What could be improved in this example
• Security of consul: SSL / ACL
My (limited) usage of OpenBSD on AWS
• VPN Gateways
• DNS proxies
• And now consul
• Many potential other use-cases
Look at / Fork the code of this demo on github
https://github.com/lbernail/eurobsdcon2017
Questions ? @lbernail