Posts Tagged With: Code

AWS’s response to ALB internal validation failures

Last week I wrote about how AWS ALB's do not validate TLS certificates from internal services. Colm MacCárthaigh, the lead engineer for Amazon ELB, writes:

I’m the main author of Amazon s2n, our Open Source implementation of TLS/SSL, and a contributor to the TLS/SSL standards. Hopefully I’m qualified to chime in!

You’re right that ALB does not validate the certificates on targets, but it’s important to understand the context that ALBs run in to see why this is still a pending item on our roadmap, rather than something we’ve shipped already as a “must have”.

The role that server certificates play in TLS is to authenticate the server, so that it can’t be impersonated or MITM. ALBs run exclusively on our Amazon VPC network, a Software Defined Network where we encapsulate and authenticate traffic at the packet level. We believe that this protection is far stronger than certificate authentication. Every single packet is being checked for correctness, by both the sender and the recipient, even in Amazon-designed hardware if you’re using an Enhanced Networking interface. We think it’s better than the ecosystem where any CA can issue a certificate at any time, with still limited audit controls (though certificate transparency is promising!).

The short of it is that traffic simply can’t be man-in-the-middled or spoofed on the VPC network, it’s one of our core security guarantees. Instances, containers, lambda functions, and Elastic Network Interfaces can only be given IPs via the secure and audit-able EC2 APIs. In our security threat model, all of this API and packet level security is what plugs in the role performed by server certificates.

This contrasts with the older EC2 classic network, a big shared network, which is why classic load balancers do support backend authentication.

We actually find that many customers actually load their targets and backends with “invalid” certificates that are self-signed or expired, because it’s so operationally hard to stay up-to-date and it’s hard to automate, even with projects like LetsEncrypt, when your instances are inherently unreachable on the internet.

All that said, we’ll be adding support for certificate validation, probably including pinning and private CAs! Used well with good operational controls it can be a measure of defense in depth, and it’s important for cases such as targets hosted on less secure private networks such as on-premesis data-centers.

Liked what you read? I am available for hire.

Amazon’s ALB’s do not validate TLS certificates from internal services

If you are using an Amazon Application Load Balancer, and forwarding traffic to internal services using HTTPS, the ALB will not validate the certificate presented by the internal service before forwarding the traffic.

So we're clear here, let's say you are running a web server on Amazon ECS. The webserver is configured to present TLS certificates to incoming requests, receive encrypted TLS traffic. The web server is part of a ELB v2 Target Group. There are two hops in this flow:

  • Customer iPhone/laptop/whatever connects to Amazon ALB. You can upload a certificate to Amazon to present to the customer. The customer's iPhone/browser/whatever will (hopefully) verify that certificate before sending requests to the ALB.

  • The ALB forwards the request to your webserver. ALB will look up the right Listener for the request, and then forward it to a ELB v2 Target Group. You can configure the Target Group to receive requests over HTTP or HTTPS.

  • If you choose HTTPS, the ALB will establish a connection and request a certificate from a random host in the Target Group. It will not validate that certificate; it will just send the traffic.

Here is the configuration for a Target Group. There's no check box for "validate HTTPS traffic from internal service."

The entire point of HTTPS is to encrypt traffic. Otherwise, a random person snooping on the network could present a weak certificate and send back whatever data it wants. We learned in 2014 that the NSA was doing this, between and inside data centers and at key points in the US.

It's unacceptable for a major Internet service in 2018 to blindly accept certificates presented by an internal service without validating them.

For the moment, I suggest using the Network Load Balancer type, which forwards the raw TCP traffic to your machine. You don't get any of the nice features of an ALB, but at least you will have the ability to reject raw traffic. If you know of other providers that offer load balancers with TLS certificate validation, please send me an email.

Update: Please read the reply from AWS.

Liked what you read? I am available for hire.

Profile Anything in Any Language in Under a Minute

Want to know how to get rough profiling of any tool written in any language, whether you control the source or not? Keep reading.

For example, here's the output you get when you compile Go from source code. Each of these lines prints out a few seconds apart. How would you go about getting timings for each individual step, or how would you notice if one step was suddenly executing much more slowly than it did three days ago?

$ GOROOT_BOOTSTRAP=~/go1.10 ./make.bash
Building Go cmd/dist using /Users/kevin/go1.10.
Building Go toolchain1 using /Users/kevin/go1.10.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for darwin/amd64.
Installed Go for darwin/amd64 in /Users/kevin/go
Installed commands in /Users/kevin/go/bin

There are a few different methods you can use to get a sense for this, but I wanted to share my favorite one. Don't reach for a profiler, don't try to set a global variable, or do start/stop timing in code. Don't even start figuring out how to configure a logger to print timestamps, and use a log output format.

Then when you've done all this, rerun the command and pass it through a command that attaches timestamps to every output line. For example, I have a tool called tss (based on ts in the moreutils package) that does this. Pipe your command to tss and you'll get output like this:

$ GOROOT_BOOTSTRAP=~/go1.10 ./make.bash | tss
     11ms         Building Go cmd/dist using /Users/kevin/go1.10.
    690ms   679ms Building Go toolchain1 using /Users/kevin/go1.10.
  11.435s 10.745s Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
  18.483s  7.048s Building Go toolchain2 using go_bootstrap and Go toolchain1.
  40.788s 22.305s Building Go toolchain3 using go_bootstrap and Go toolchain2.
 1m0.112s 19.324s Building packages and commands for darwin/amd64.
1m19.053s 18.942s ---
1m19.053s      0s Installed Go for darwin/amd64 in /Users/kevin/go
1m19.053s      0s Installed commands in /Users/kevin/go/bin

The first column is the amount of time that has elapsed since the tool was invoked. The second column is the amount of time that has elapsed since the previous line was printed.

The good thing about this technique is you can use it in pretty much any situation - you don't have to know anything about the code besides:

  • how to invoke it
  • a rough idea of which code paths get hit
  • how to print things to stdout

Just start adding print lines around interesting bits of code.

For example, you might have this in your test suite in Ruby to empty the database. How much time does this add to every test in your suite? Put print lines before and after to find out.

config.after(:each) do
  puts "database clean start"
  puts "database clean end"

Similarly, if you wanted to find out why it takes so long for your test suite to start running or your web framework to boot, you can annotate your spec_helper or bootstrap file, the top of the test file, before and after require lines, and the start and end of the first test in the suite. You can even drill down into third party libraries, just find the source file on disk and start adding print statements.

You can also use it with strace! strace will show you a list of the syscalls being opened by your program. Passing the output through tss can show you which syscall activity is taking a lot of time, or when your program is doing a lot of stuff without going to the disk.


If you pipe program a to program b, and program a exits with a non-zero return code, Bash will by default report a return code of zero for the entire operation. This means if you are piping output to tss or ts you may accidentally change a failing program to a passing one. Use set -o pipefail in Bash scripts to ensure that Bash will return a non-zero return code if any part of a pipe operation fails. Or add this to a Makefile:

SHELL = /bin/bash -o pipefail

Second, the program being profiled may not flush output to stdout, that is, it may print something but it may not appear on the screen at the same time it was printed. If the timings on screen don't seem to match up with your intuition, check that the program is flushing print statements to the stdout file descriptor after printing them.

Liked what you read? I am available for hire.

You Shouldn’t Use Faker (or other test randomization libraries)

It's a common, and distressing, pattern to have factories in tests that call out to a library like Faker to "randomize" the returned object. So you might see something like this:

    id: uuid(),
    first_name: faker.random.firstName(),
    last_name: faker.random.lastName(),
    address: faker.random.address(),

package = Package.create({
    width: faker.math.range(100),
    height: faker.math.range(200),
    length: faker.math.range(300),

This is a bad practice and it's distressing to see it in such widespread use. I want to explain some reasons why it's bad.

The corpus is not large enough to avoid uniqueness failures

Let's say you write a test that creates two different users, and for whatever reason, the test fails if the users have the same name (due to a database constraint failure, write to same map key, array having 1 value instead of 2, &c). Faker has about three thousand different possible values for a first name. This means that every 3000th run on average, the randomly chosen names will collide, and your test will fail. In a CI environment or on a medium size team, it's easy to run the tests 3000 times in a week. Alternatively, if the same error exists in ten tests, it will emerge on average once every 300 runs.

The onus on catching failures falls on your team

Because the error only appears once in every 3000 test runs, the odds are you haven't fully or partially probed the space of possible test values when you submit your code and tests for review. This means that the other members of your team will be the ones who run into the error caused by your tests. They might lack the context necessary to effectively debug the problem. This is a very frustrating experience for your teammates.

It's hard to reproduce failures

If you don't have enough logging to see the problem on the first failure, it's going to be difficult to track down a failure that appears only once every 3,000 test runs. Run the test ten or even fifty times in a loop and it still won't appear.

An environment where tests randomly fail for unknown reasons is corrosive to build stability. It encourages people to just hit the "Rebuild" button and hope for a pass on the next try. Eventually this attitude will mask real problems with the build. If your test suite takes a long time to run (upwards of three minutes) it can be demoralizing to push the deploy or rebase button and watch your tests fail for an unrelated reason.

It's a large library (and growing)

It's common for me to write Node.js tests and have an entire test file with 50 or 100 tests exit in about 10 milliseconds.

By definition a "fixture" library needs to load a lot of fake data. In 2015 it took about 60 milliseconds to import faker. It now takes about 300 milliseconds:

var a =
console.log( - a);
$ node faker-testing/index.js

You can hack around this by only importing the locale you need, but I've never seen anyone do this in practice.

What you need instead

So faker is a bad idea. What should you use? It depends on your use case.

  • Unique random identifiers. If you need a random unique identifier, just use a uuid generator directly. If you create 103 trillion objects in a given test, the odds of a uuid collision are still only one in a billion, so the odds of a collision are much lower than using the faker library. Alternatively, you can use an incrementing integer each time you call your factory, so you get "Person 0", "Person 1", "Person 2", etc.

  • A fuzzer, or QuickCheck. Faker has tools for randomizing values within some input space, say a package width that varies between 0 and 100. The problem is that the default set up is to only exhaust one of those values each time you run the test, so you're barely covering the input space.

    What you really want is a fuzzer, that will generate a ton of random values within a set of parameters, and test a significant percentage of them each time the test runs. This gives a better guarantee that there are no errors. Alternatively, if there are known edge cases, you can add explicit tests for those edge cases (negative, 0, one, seven, max, or whatever). Tools like Haskell's QuickCheck or afl-fuzz make it easy to add this type of testing to your test suite.

If you want random human-looking data I suggest adding it yourself and using some scheme that makes sense for your company. For example, users named Alice and Bob are always friends that complete actions successfully, and Eve always makes errors. Riders with Android phones always want to get picked up in Golden Gate Park and taken to the Mission; riders with iPhones want to get picked up in the Tenderloin and taken to Russian Hill (or something). This scheme will provide more semantic value, and easily pinpoint errors (a "Bob" error means we should have submitted something successfully but failed to, for example).

That's all; happy testing, and may your tests never be flaky!

Liked what you read? I am available for hire.

Proxying to a subcommand with Go

There are a number of Unix commands that manipulate something about the environment or arguments in some way before starting a subcommand. For example:

  • xargs reads arguments from standard input and appends them to the end of the command.

  • chpst changes the process state before calling the resulting command.

  • envdir manipulates the process environment, loading variables from a directory, before starting a subcommand

Go is a popular language for writing tools that shell out to a subprocess, for example aws-vault or chamber, both of which have exec options for initializing environment variables and then calling your subcommand.

Unfortunately there are a number of ways that you can make mistakes when proxying to a subcommand, especially one where the end user is specifying the subcommand to run, and you can't determine anything about it ahead of time. I would like to list some of them here.

First, create the subcommand.

cmd := exec.Command("mybinary", "arg1", "-flag")
// All of our setup work will go here
if err := cmd.Run(); err != nil {

Standard Output/Input/Error

You probably want to proxy through the subcommand's standard output and standard error to the parent terminal. Just reassign stdout/stderr to point at the parent command. Obviously, you can redirect these later.

cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
// If you need
cmd.Stdin = os.Stdin

Environment Variables

By default the subcommand gets all of the environment variables that belong to the Go process. But if you reassign cmd.Env it will only get the variables you provide, so you have to be careful with your implementation.

cmd.Env = []string{"TZ=UTC", "MYVAR=myval"}


If someone sends a signal to the parent process, you probably want to forward it through to the child. This gets tricky because you can receive multiple signals, that all should be sent to the client and handled appropriately.

You also can't send signals until you start the process, so you need to use a goroutine to handle them. I've copied this code from aws-vault, with some modifications.

if err := cmd.Start(); err != nil {
    log.Fatal(err) // Command not found on PATH, not executable, &c.
// wait for the command to finish
waitCh := make(chan error, 1)
go func() {
    waitCh <- cmd.Wait()
sigChan := make(chan os.Signal, 1)

// You need a for loop to handle multiple signals
for {
    select {
    case sig := <-sigChan:
        if err := cmd.Process.Signal(sig); err != nil {
            // Not clear how we can hit this, but probably not
            // worth terminating the child.
            log.Print("error sending signal", sig, err)
    case err := <-waitCh:
        // Subprocess exited. Get the return code, if we can
        var waitStatus syscall.WaitStatus
        if exitError, ok := err.(*exec.ExitError); ok {
            waitStatus = exitError.Sys().(syscall.WaitStatus)
        if err != nil {

Just Use Exec

The above code block is quite complicated. If all you want to do is start a subcommand and then exit the process, you can avoid all of the above logic by calling exec. Unix has a notion of parent and children processes - when you start a process with cmd.Run, it becomes a child process. exec is a special system call that sees the child process become the parent. Consider the following program:

cmd := exec.Command("sleep", "3600")
if err := cmd.Run(); err != nil {

If I run it, the process tree will look something like this:

 | |   \-+= 06167 kevin -zsh
 | |     \-+= 09471 kevin go-sleep-command-proxy
 | |       \--- 09472 kevin sleep 3600

If I run exec, the sleep command will replace the Go command with the sleep command. In Go, this looks something like this:

 | |   \-+= 06167 kevin -zsh
 | |     \--= 09862 kevin sleep 3600

This is really useful for us, because:

  • We don't need to proxy through signals or stdout/stderr anymore! Those are automatically handled for us.

  • The return code of the exec'd process becomes the return code of the program.

  • We don't have to use memory running the Go program.

Another reason to use exec: even if you get the proxying code perfectly right, proxying commands (as we did above) can have errors. The Go process can't handle SIGKILL, so there's a chance that someone will send SIGKILL to the Go process and then the subprocess will be "orphaned." The subprocess will be reattached to PID 1, which makes it tougher to collect.

I should note: Syscalls can vary from operating system to operating system, so you should be careful that the arguments you pass to syscall.Exec make sense for the operating system you're using. Rob Pike and Ian Lance Taylor also think it's a bad idea.

Liked what you read? I am available for hire.

Running Bazel tests on Travis CI

Bazel is a build system that was recently open sourced by Google. Bazel operates on configuration files - a WORKSPACE file for your entire project, and then per-directory BUILD.bazel files. These declare all of the dependencies for your project, using a language called Skylark. There are a number of Skylark rules for things like downloading and caching HTTP archives, and language-specific rules.

This can be a pain to set up for a new project, but the advantage is you get builds that are perfectly hermetic. In addition, if your project has a lot of artifacts, Bazel has useful tools for caching those artifacts, and builds a dependency graph that makes it easy to see when things change. For example, if you compile Javascript from Typescript, you can cache the compiled Javascript. If the Typescript source file changes, Bazel knows that only the file that changed needs to be regenerated.

Anyway there are some gotchas about running Bazel on Travis CI that I wanted to cover, and when I was looking at instructions they weren't great.

1) Install Bazel

The official instructions ask you to add the Bazel apt repository, and then install from there. You can shave a few seconds by only updating that repository, instead of all of them.

echo "deb [arch=amd64] stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
curl --silent | sudo apt-key add -
sudo apt-get update -o Dir::Etc::sourcelist="sources.list.d/bazel.list" \
    -o Dir::Etc::sourceparts="-" \
    -o APT::Get::List-Cleanup="0"
sudo apt-get install openjdk-8-jdk bazel

You can shave a few more seconds by just getting and installing the deb directly, though this requires you to manually update the version number when Bazel releases a new version. (This is in a Makefile, you'll need to change the syntax for Bash or a travis.yml file).

BAZEL_VERSION := "0.7.0"
BAZEL_DEB := "bazel_$(BAZEL_VERSION)_amd64.deb"

curl --silent --output /tmp/$(BAZEL_DEB) "$(BAZEL_DEB)"
sudo dpkg --force-all -i /tmp/$(BAZEL_DEB)

2) Run Tests

Bazel ships with a ton of flags but these are probably the ones you want to know about/use:

  • --batch: Run in batch mode, as opposed to client-server mode.

  • --noshow_progress / --noshow_loading_progress: Travis pretends to be a TTY so it can show colors, but this causes Bazel to dump a whole bunch of progress messages to the screen. It would be nice if you could say "I can handle ANSI escape sequences relating to colors, but not screen redraws" but you only get the binary "I'm a TTY" / "I'm not", which is unfortunate. Anyway these options turn off the "show progress" log spam.

  • --test_output=errors: By default Bazel logs test failures to a log file. This option will print them to the screen instead.

  • --features: Enable features specific to a test runner. In particular for Go you may want the --features=race rule to run tests with the race detector enabled.

3) Cache Results

Bazel is much faster when it can load the intermediate results from a cache; on one library I maintain, tests run in 13 seconds (versus 47 seconds) when they are run with a full cache.

However, the Bazel caches are enormous. Caches for Go tests include the Go binary and source tree, which runs about 95MB alone. Add on other artifacts (including the Java JDK's used to run Bazel) and it's common to see over 200MB. If you are uploading or downloading caches from somewhere like S3, this can be a really slow operation. Also, it's difficult to remove large items from Bazel's cache without removing everything; the file structure makes the cache a little tough to remove parts of.

In particular, Travis's cache does go to S3, which means it skips the local caching proxies Travis sets up for e.g. apt and npm. Travis times out the cache download after 3 minutes, and Bazel quits if the cache is corrupt. In my experience Travis will not complete a 200MB cache download in 3 minutes, so your builds will fail.

That's it

For now it's probably best to eat the cold cache startup time. That's really unfortunate though; I hope we can find a better solution in the future. One way may be for Travis to run a local "remote cache", which responds to the WebDAV protocol. This would allow closer caching of Bazel objects.

It would also be nice if Travis had a "language: bazel" mode, which would come with the right version of Bazel installed for you.

Liked what you read? I am available for hire.

Make your Go Binaries Homebrew Installable

It's easier than you think to make a software package installable via Homebrew. If you depend on a very specific version of a software package (say, Postgres 9.5.3 with readline support), I highly recommend creating a Homebrew repository and publishing recipes to it. Then your team can install and update packages as easily as:

brew tap mycompany/packages
brew install mycompany/packages/postgresql

You can use the existing formulas as a jumping off point, and modify as you see fit. (Obviously this won't work for Linux folks on your team, however in my experience people running Linux in a Mac software shop have more experience building dependencies on their own).

Anyway, I wanted to describe how to install Go binaries via Homebrew. One way to do this is to compile binaries, upload them to Github releases, and install from there. However, the Homebrew core team requires that packages are buildable from the source code. (This helps check that a binary wasn't tampered with, and avoids compatibility problems with e.g. 32 bit and 64 bit systems).

If you vendor dependencies, and check in the vendor folder to Github, installation is super easy.

# Classname should match the name of the installed package.
class Hostsfile < Formula
  desc "CLI for manipulating /etc/hosts files"
  homepage ""

  # Source code archive. Each tagged release will have one
  url ""
  sha256 "cc1f3c1cb505536044cbe01f44ad7da997e6a3928fac1f64590ef69d73da8acd"
  head ""

  depends_on "go" => :build

  def install
    ENV["GOPATH"] = buildpath

    bin_path = buildpath/"src/"
    # Copy all files from their current location (GOPATH root)
    # to $GOPATH/src/
    bin_path.install Dir["*"]
    cd bin_path do
      # Install the compiled binary into Homebrew's `bin` - a pre-existing
      # global variable
      system "go", "build", "-o", bin/"hostsfile", "."

  # Homebrew requires tests.
  test do
    # "2>&1" redirects standard error to stdout. The "2" at the end means "the
    # exit code should be 2".
    assert_match "hostsfile version 1.2", shell_output("#{bin}/hostsfile version 2>&1", 2)

Basically, download some source code, move it to $GOPATH/src/path/to/binary, build it, and put the compiled binary in $(brew --prefix)/bin.

If you don't vendor dependencies, the story gets a little more complicated because you need to download a version of all of your dependencies. Say for example I had one dependency in my project, I would add a go_resource line for each dependency, and then call stage_deps to download/install all of them in the correct places.

require "language/go"

# Classname should match the name of the installed package.
class Hostsfile < Formula
  desc "CLI for manipulating /etc/hosts files"
  homepage ""

  # Source code archive. Each tagged release will have one
  url ""
  sha256 "cc1f3c1cb505536044cbe01f44ad7da997e6a3928fac1f64590ef69d73da8acd"
  head ""

  go_resource "" do
    url "",
        :revision => "40e4aedc8fabf8c23e040057540867186712faa5"

  depends_on "go" => :build

  def install
    ENV["GOPATH"] = buildpath

    bin_path = buildpath/"src/"
    # Copy all files from their current location (GOPATH root)
    # to $GOPATH/src/
    bin_path.install Dir["*"]

    # Stage dependencies. This requires the "require language/go" line above
    Language::Go.stage_deps resources, buildpath/"src"
    cd bin_path do
      # Install the compiled binary into Homebrew's `bin` - a pre-existing
      # global variable
      system "go", "build", "-o", bin/"hostsfile", "."

  # Homebrew requires tests.
  test do
    # "2>&1" redirects standard error to stdout. The "2" at the end means "the
    # exit code should be 2".
    assert_match "hostsfile version 1.2", shell_output("#{bin}/hostsfile version 2>&1", 2)

And that's it! You can test your new package by creating a symlink from /usr/local/Homebrew/Library/Taps/homebrew to wherever you keep your homebrew-core checkout:

ln -s ~/code/homebrew-core /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core

Then you can just use brew install commands and they'll work just as you expect.

Liked what you read? I am available for hire.

CircleCI trusts 8 analytics companies with your source code and API tokens

When you navigate to your project in CircleCI's UI, Javascript from eight different analytics companies gets loaded and executed in your browser.

  • Pusher
  • Intercom
  • Launch Darkly
  • Amplitude
  • Appcues
  • Quora (??)
  • Optimizely

You can see this in my Network tab here:

CircleCI network requests

This is a problem because the CircleCI browser context has full access to the CircleCI API, which is hosted on the same domain, so all eight of those companies' scripts can make requests to CircleCI API endpoints. Furthermore CircleCI customers frequently either include credentials in source code or as environment variables in CircleCI. Set these, and you are trusting that CircleCI won't get compromised, or at least, your application is at most as secure as CircleCI is.

However, with eight different companies running Javascript in your browser with access to the CircleCI API, your source code and secrets are at most as secure as the union of eight different analytics companies' Javascript environments. If any of those eight gets compromised, it's trivial to execute Javascript that creates a new API token for your account. Once that token is created, an attacker can easily export it to a domain controlled by the attacker. Once an attacker has the token, they can use the "Test Commands" API to add new commands that will dump your environment variables and/or all files in source code to the logs, then download your logs or artifacts via the same API.

Analytics companies frequently get hacked, serve malware or otherwise contain vulnerabilities. Just last week it was revealed that "Code Hive" was running coin mining code on CBS's website. In 2014, was compromised and used to serve malware. If the same happened to any of the eight companies, you would be screwed.

This is frankly unacceptable for a company that manages source code and secrets for a large number of companies in the industry. It's unacceptable enough that your browsing data on CircleCI is potentially exposed to eight different companies, let alone API access to your source code. There are a number of steps you can take to mitigate the issue:

  • Put all domains used by the above servers in /etc/hosts, for each machine and each person on your team that accesses CircleCI. I have a tool that you can use to automate this process. However, this may break Javascript on CircleCI or other sites that are not coded in a defensive style.

  • Only use command line tools + API's to access CircleCI. I have written one such tool.

  • If there is an API to disable the "Test Commands" setting, turn it off, as an attacker can use that to put data in logs or artifacts without being able to control the circle.yml file or push to Github.

  • Don't put any sensitive keys in CircleCI, or in source code; inject them directly into the production runtime.

  • Demand that CircleCI take steps to make their dashboard and your source code more secure.

Finally, there are a number of steps CircleCI should enable immediately:

  • Don't load analytics scripts from eight different companies on pages that contain sensitive content, or that have access to the CircleCI "create token" API. Host the dashboard and API on a separate domain from the marketing page.

  • Send an email any time an API access token is created. Add a setting to allow org-wide disabling of API token creation.

  • Add an option to disable the "Test Commands" API, as this would allow an attacker without access to Github to place whatever content they want (source code/environment variables) in logs.

  • Add an option to delete old logs. If you have ever dumped env vars to the log file, an attacker can export these.

  • Enable subresource integrity, or serve JS from each of these companies from the CircleCI domain.

With embarrassing compromises literally every week, and new Congressional scrutiny into Facebook and Google's advertising practices in the 2016 election, as an industry we need to do better on issues like this.

Update: CircleCI posted their response.


Why don't you include a POC? I have one that demonstrates this attack, but I don't want to show script kiddies how. If you can't figure out how to construct it by reading the above description and the network traffic, you don't deserve to know how.

Liked what you read? I am available for hire.

Let’s talk about Javascript string encoding

Node string encoding is all over the place. Let's try to straighten out how it works.

First, some very basics about string encoding. A string is a series of bytes. A byte is 8 bits, each of which can be 0 or 1, so a byte can have 28 or 256 different values. Encoding is the process of squashing the graphics you see on screen, say, 世 - into actual bytes. There are over a million possible Unicode characters - think about all of the different languages and emoji and symbols on the planet.

You could easily represent all of the characters in the Unicode set with an encoding that says simply "assign one number, 4 bytes (or 32 bits) long, for each character in the Unicode set." One 32-bit combo for each character means you can have 4 billion distinct characters. But this would be really inefficient for most documents. For example, the English Bible or dictionary or people's email folders are mostly the characters a-z, A-Z, 0-9, and punctuation. It would be inefficient to waste 4 bytes on every "a" in the document - we want a way to represent it more efficiently.

The most common encoding is UTF-8. The main advantage of UTF-8 is that it needs a single byte (instead of four) to encode Unicode characters 0-127 - the so-called ASCII set, which includes the ones I listed above. Less common characters are represented with two bytes, and even less common characters with three bytes, and so on.

Because Javascript was invented twenty years ago in the space of ten days, it uses an encoding that uses two bytes to store each character, which translates roughly to an encoding called UCS-2, or another one called UTF-16. The letter 'a' is Unicode code point 97, so stored in a Javascript string, the first byte of a UTF-16 code point would be 97 and the second byte would be 0. The euro symbol () is Unicode code point 8364, so the first byte would be 172 and the second byte would be 32. (The relationship between them: 8364 = (32 << 8) + 172).

That only gets you Unicode code points 0 through (256*256 = ) 65536, though, so: if the first two-byte character is between 55296 and 56319, it's a surrogate, and you have to read the second two-byte character to figure out what Unicode code point it represents. Javascript will handle this multi-character read for you if you use the new codePointAt operator, which can return all Unicode code points, up to 1.1 million. The old charCodeAt operator only reads one character at a time (between 0 and 65536) and you'll have to read/decode the second one yourself.

Where this gets complicated

Because everything else in the world is moving to UTF-8, Node is also trying to move to UTF-8. This gets confusing because Node sometimes asks you to choose which encoding you want, in places where you wouldn't really expect it to ask.

Frequently, you want to convert from a UTF-16 encoded Javascript string to UTF-8 bytes in some form, whether a Buffer or a Uint8Array. Unfortunately Node doesn't offer easy API's to do that; it buries the conversion logic in C++ code, which eventually calls out to the V8 binary to handle it.


Node offers a Buffer type, where a Buffer is an array of bytes, and an API, Buffer.from(string, encoding). encoding defaults to UTF-8. So far, so good! Unfortunately, sometimes encoding refers to the encoding of string, and sometimes it refers to the encoding of the bytes in the Buffer.

Buffer.from('7468697320697320612074c3a97374', 'hex') will decode the input as a series of hex characters, and store the bytes corresponding to each 2-digit hex character in the Buffer. But in var a = 'tést'; Buffer.from(a, 'utf8'); the 'utf8' refers to how the bytes will be stored in the resulting Buffer. Remember, strings are always two bytes per character, so declaring they are 'utf8' doesn't make sense.

Similarly, the encoding parameter in buf.toString(encoding) does not refer to the encoding of the output string - it refers to the encoding of the data in the buffer. Unless of course you specify hex or base64, in which case it does refer to the encoding of the output string. Got it?


Incoming HTTP requests are often encoded using UTF-8. Node will give you data as a Buffer via the HTTP request's .on('data') event handler, which can be decoded using the content-type from the HTTP headers. In Express, this is handled in the body-parser module, which defers to the iconv-lite module for the actual encoding and decoding.

But if you expect e.g. a JSON or XML input, you will probably eventually want to turn that Buffer into a string - see the JSON section below.

When writing string data as an HTTP response, Node will wait until the last possible minute to convert that string into a Buffer that can be written to the socket. This leads to odd behaviors like the net library needing to know what encoding to use for a string you pass to it.


JSON.parse operates on a string and returns a JSON object, which may be a string or contain strings. JSON.parse and JSON.stringify don't do anything with the string encoding - if you pass in a garbage string, you get a garbage string back out.

So you need to depend on something else to determine the character encoding, before you pass a string to JSON.parse. Usually this will be your HTTP middleware; you have to trust (or verify) that it handles character sets correctly, before creating UTF-8 strings.

Files on disk

Node source files are expected to be encoded with UTF-8. That means you can encode UTF-8 source characters in a string, like this:

var x = "¢"

Where the cent character is the UTF-8 encoded byte sequence "\xc2\xa2". When Node starts and you try to reference x in your program, it will be re-encoded as a UTF-16 string. If you type the literal characters:

var x = "\xc2\xa2";

This will be turned into the UTF-16 string "\xc2\x00\xa2\x00". So be careful to mind your inputs and outputs.


Encoding in Node is extremely confusing, and difficult to get right. It helps, though, when you realize that Javascript string types will always be encoded as UTF-16, and most of the other places strings in RAM interact with sockets, files, or byte arrays, the string gets re-encoded as UTF-8.

This is all massively inefficient, of course. Most strings are representable as UTF-8, and using two bytes to represent their characters means you are using more memory than you need to, as well as paying an O(n) tax to re-encode the string any time you encounter a HTTP or filesystem boundary.

There's nothing stopping us from packing UTF-8 bytes into a UTF-16 string: to use each of the two bytes to store one UTF-8 character. We would need custom encoders and decoders, but it's possible. And it would avoid the need to re-encode the string at any system boundary.

Liked what you read? I am available for hire.

Things to Use Instead of JWT

You might have heard that you shouldn't be using JWT. That advice is correct - you really shouldn't use it. In general, specifications that allow the attacker to choose the algorithm for negotiation have more problems than ones that don't (see TLS). N libraries need to implement M different encryption and decryption algorithms, and an attacker only needs to find a vulnerability in one of them, or a vulnerability in their combination. JWT has seen both of these errors; unlike TLS, it hasn't already been deployed onto billions of devices around the world.

This is a controversial opinion, but implementation errors should lower your opinion of a specification. An error in one implementation means other implementations are more likely to contain the same or different errors. It implies that it's more difficult to correctly implement the spec. JWT implementations have been extremely buggy.

But The Bad Implementations Were Written by Bad Authors

In the 1800's rail cars were coupled by an oval link on one end and a socket on the other. A railway worker would drop a pin down through the socket, keeping the link in place.

The train engineer could not see the coupler at the time of coupling, and the operation was fraught. Many couplers had their hands mangled. Worse, there was no buffer between the cars, and it was easy to get crushed if the coupling missed. Tens of thousands of people died.

Link-and-pin railway coupler

Still, the railroads stuck with them because link-and-pin couplers were cheap. You could imagine excuses being made about the people who died, or lost their fingers or hands; they were inattentive, they weren't following the right procedure, bad luck happens and we can't do anything about it, etc.

In 1893 Congress outlawed the link-and-pin coupler and deaths fell by one third within a year, and that's despite the high cost of switching to automatic couplers.


Update, January 2018: PAST is an excellent library implementing many of the suggestions here. Use PAST. Follow its development for use in your language.

What should you be using instead of JWT? That depends on your use case.

I want users to authenticate with a username and secret token

Have them make a request to your server over TLS; you don't need any additional encryption. TLS provides an encryption layer, you don't need any additional encryption or hashing besides TLS.

I want to post a public key and have users send me encrypted messages with it

The technical name for this is asymmetric encryption; only the user with the private key can decrypt the message. This is pretty magical; the magic is that people don't need the private key to send you messages that you can read. It was illegal to ship this technology outside of the US for most of the 90's.

JWT supports public key encryption with RSA, but you don't want to use it for two reasons. One, RSA is notoriously tricky to implement, especially when compared with elliptic curve cryptography. Thomas Ptáček explains:

The weight of correctness/safety in elliptic curve systems falls primarily on cryptographers, who must provide a set of curve parameters optimized for security at a particular performance level; once that happens, there aren't many knobs for implementors to turn that can subvert security. The opposite is true in RSA. Even if you use RSA-OAEP, there are additional parameters to supply and things you have to know to get right.

You don't want the random person implementing your JWT library to be tuning RSA. Two, the algorithm used by JWT doesn't support forward secrecy. With JWT, someone can slurp all of your encrypted messages, and if they get your key later, they can decrypt all of your messages after the fact. With forward secrecy, even if your keys are exposed later, an attacker can't read previous messages.

A better elliptic curve library is Nacl's box, which only uses one encryption primitive, and doesn't require any configuration. Or you can have users send you messages with TLS, which also uses public key encryption.

I want to encrypt some data so third parties can't read it, and then be able to decrypt it later

You might use this for browser cookies (if you don't want the user to be able to read or modify the payload), or for API keys / other secrets that need to be stored at rest.

You don't want to use JWT for this because the payload (the middle part) is unencrypted. You can encrypt the entire JWT object, but if you are using a different, better algorithm to encrypt the JWT token, or the data in it, there's not much point in using JWT.

The best algorithm to use for two-way encryption is Nacl's secretbox. Secretbox is not vulnerable to downgrade or protocol switching attacks and the Go secretbox implementation was written by a world-renowned cryptographer who also writes and verifies cryptographic code for Google Chrome.

I want to send some data and have users send it back to me and verify that it hasn't been tampered with

This is the JWT use case. The third part of a JWT is the signature, which is supposed to verify that the header and the payload have not been tampered with since you signed them.

The problem with JWT is the user gets to choose which algorithm to use. In the past, implementations have allowed users to pass "none" as the verification algorithm. Other implementations have allowed access by mixing up RSA and HMAC protocols. In general, implementations are also more complicated than they need to be because of the need to support multiple different algorithms. For example in jwt-go, it's not enough to check err == nil to verify a good token, you also have to check the Valid parameter on a token object. I have seen someone omit the latter check in production.

The one benefit of JWT is a shared standard for specifying a header and a payload. But the server and the client should support only a single algorithm, probably HMAC with SHA-256, and reject all of the others.

If you are rejecting all of the other algorithms, though, you shouldn't leave the code for them lying around in your library. Omitting all of the other algorithms makes it impossible to commit an algorithm confusion error. It also means you can't screw up the implementations of those algorithms.

For fun, I forked jwt-go and ripped out all of the code not related to the HMAC-SHA256 algorithm. That library is currently 2600 lines of Go, and supports four distinct verification algorithms. My fork is only 720 lines and has much simpler API's.

// old
func Parse(tokenString string, keyFunc func(*Token) (interface{}, error)) (*Token, error)
func (m *SigningMethodHMAC) Sign(signingString string, key interface{}) (string, error)
func (m *SigningMethodHMAC) Verify(signingString, signature string, key interface{}) error

// new
func Parse(tokenString string, key *[32]byte) (*Token, error)
func Sign(signingString string, key *[32]byte) string
func Verify(signingString, signature string, key *[32]byte) error

These changes increased the type safety and reduced the number of branches in the code. Reducing the number of branches helps reduce the chance of introducing a defect that compromises security.

It's important to note that my experiment is not JWT. When you reduce JWT to a thing that is secure, you give up the "algorithm agility" that is a proud part of the specification.

We should have more "JWT"-adjacent libraries that only attempt to implement a single algorithm, with a 256-bit random key only, for their own sake and for their users. Or we should give up on the idea of JWT.

Liked what you read? I am available for hire.