Making Your Services More Reliable

Kevin Burke

@derivativeburke

What does Five Nines mean?

Five Nines = you can fail 0.001% of the time

Chasing Five Nines

5.26 min. downtime per year
1 request out of 100,000 fails
You can blow your yearly budget at one time

This might not be appropriate!

Microservices:

Rate Limiting
Auth
Fraud/IP address scoring
Validation

Database service 1
Database service 2
Billing
Feature flags

Many more opportunities for failure!

What happens when I type google.com into my browser?

DNS Lookup

`man getaddrinfo`

Establish TCP Connection

`man connect`

Write Request

`man 2 write`

Read Response

`man 2 read`

Parse Response

DNS

DNS Provider Outages

ENom - 5 million domains

Cloudfront - 100 Minutes Downtime

DNSimple - 11 Hour Downtime

DNS Lookup Failures

requests.get('https://api.acme.com')

DNS Server is Down/Unreachable

start = time.time()
try:
  requests.get('https://api.acme.com', 
               timeout=3)
except ConnectionError: pass
print time.time() - start

You Might Be Vulnerable If...

Mobile apps that use DNS to connect to an API
Client libraries that use DNS to connect to an API
Your service uses DNS to connect to third party API's (Stripe, Mailchimp...)

DNS Resolver is Down - Workarounds

Use multiple resolvers
Internally, connect directly to IP addresses instead of DNS
Set shorter timeout in /etc/resolv.conf
If you don't get a response, ignore TTL

DNS Provider is Down - Workarounds

Maintain DNS records at two hosts
Reference both in nameserver list

Timeouts

When something is taking too long, you abandon it

Your users have a timeout, whether your system does or not

Outside In

Setting Timeouts - 2 Questions

What's the maximum acceptable response time?
How long can my service afford to spend processing requests?

Hard Math Stuff

Thread pool with 100 workers
Each request takes 1 second. 20 requests come in per second.
Downstream server is slow, requests take 6 seconds each
Set timeout to prevent this

Fail early if you can't serve a request

Socket Timeouts Are Liars

Slow Read

timer = 29
# 13 minute long 204
for byte in 'HTTP/1.1 204 No Content\r\n\r\n':
    reactor.callLater(timer, self._send_byte, byte)
    timer += 29
reactor.callLater(timer, self.transport.loseConnection)

Remote Server Unreachable

start = time.time()
try:
  requests.get('http://google.com:81', 
               timeout=3)
except ConnectTimeout: pass
print time.time() - start

Why 18 Seconds?

Non-authoritative answer:
Name:	www.google.com
Address: 74.125.239.49
Name:	www.google.com
Address: 74.125.239.48
Name:	www.google.com
Address: 74.125.239.52
Name:	www.google.com
Address: 74.125.239.50
Name:	www.google.com
Address: 74.125.239.51

HAProxy

retries is the number of times a connection attempt should be retried on a server when a connection either is refused or times out. The default value is 3.

Timeouts

One Timeout Value = set on both Connect/Read

Separate Connect Timeout

import requests
requests.get('http://10.10.10.123', 
             timeout=(3, 31.5))

Not available in the standard library!

Timeouts - Workarounds

*Set timeouts*
Set shorter, separate, connect timeout
Know how your HTTP client behaves

Timeouts - Wall Clock Timeout

# net/http/transport
var respHeaderTimer <-chan time.Time
respHeaderTimer = time.After(d)
case <-respHeaderTimer:
  pc.close()
  re = errors.New("timeout")

Timeouts - Measure

Retries

Retries - Temporal Failures

Retries - Single Component Failures

Exponential Backoff

`1, 2, 4, 8...`

Exponential Backoff with Jitter

`1.01, 2.03, 3.9, 8.2...`

When can you retry?

Idempotence

Idempotent Actions

Retrieving a profile
Setting user's email to foo@bar.com
Emptying a bank account

Idempotent Requests

You can always retry idempotent actions!

Not Idempotent

Sending an email
Charging a credit card

When can you retry?

Not Idempotent Requests

You can retry if the data never made it (connection timeout, connection error)

Not Idempotent Requests

Determining whether the data made it is hard

Not Idempotent Requests

You can retry if you get a 429 or a 503 (carefully!)

How do I make things idempotent?

Not idempotent

POST /v1/<account_id>/Emails
To=foo@bar.com
From=me@me.com
Body=Lester, are we still cops?

Idempotent (retryable) request

PUT /v1/<account_id>/Emails/EM123
To=foo@bar.com
From=me@me.com
Body=Lester, are we still cops?

Idempotent (retryable) request

Requires sid collision handling!

Know your HTTP client

Testing your clients

Thanks!

Kevin Burke

kev.inburke.com

kev@inburke.com

@derivativeburke

These slides are available at: