Posts Tagged With: Code

Stepping Up Your Pull Request Game

Okay! You had an idea for how to improve the project, the maintainers indicated they'd approve it, you checked out a new branch, made some changes, and you are ready to submit it for review. Here are some tips for submitting a changeset that's more likely to pass through code review quickly, and make it easier for people to understand your code.

A Very Brief Caveat

If you are new to programming, don't worry about getting these details right! There are a lot of other useful things to learn first, like the details of a programming language, how to test your code, how to retrieve data you need, then parse it and transform it into a useful format. I started programming by copy/pasting text into the WordPress "edit file" UI.

Write a Good Commit Message

A big, big part of your job as an engineer is communicating what you're doing to other people. When you communicate, you can be more sure that you're building the right thing, you increase usage of things that you've built, and you ensure that people don't assign credit to someone else when you build things. Part of this job is writing a clear message for the rest of the team when you change something.

Fortunately you are probably already doing this! If you write a description of your change in the pull request summary field, you're already halfway there. You just need to put that great message in the commit instead of in the pull request.

The first thing you need to do is stop typing git commit -m "Updated the widgets" and start typing just git commit. Git will try to open a text editor; you'll want to configure this to use the editor of your choice.

A lot of words have been written about writing good commit messages; Tim Pope wrote my favorite post about it.

How big should a commit be? Bigger than you think; I rarely use more than one commit in a pull request, though I try to limit pull requests to 400 lines removed or added, and sometimes break a multiple-commit change into multiple pull requests.

Logistics

If you use Vim to write commit messages, the editor will show you if the summary goes beyond 50 characters.

If you write a great commit message, and the pull request is one commit, Github will display it straight in the UI! Here's an example commit in the terminal:

And here's what that looks like when I open that commit as a pull request in Github - note Github has autofilled the subject/body of the message.

Note if your summary is too long, Github will truncate it:

Review Your Pull Request Before Submitting It

Before you hit "Submit", be sure to look over your diff so you don't submit an obvious error. In this pass you should be looking for things like typos, syntax errors, and debugging print statements which are left in the diff. One last read through can be really useful.

Everyone struggles to get code reviews done, and it can be frustrating for reviewers to find things like print statements in a diff. They might be more hesitant to review your code in the future if it has obvious errors in it.

Make Sure The Tests Pass

If the project has tests, try to verify they pass before you submit your change. It's annoying that Github doesn't make the test pass/failure state more obvious before you submit.

Hopefully the project has a test convention - a make test command in a Makefile, instructions in a CONTRIBUTING.md, or automatic builds via Travis CI. If you can't get the tests set up, or it seems like there's an unrelated failure, add a separate issue explaining why you couldn't get the tests running.

If the project has clear guidelines on how to run the tests, and they fail on your change, it can be a sign you weren't paying close enough attention.

The Code Review Process

I don't have great advice here, besides a) patience is a virtue, and b) the faster you are to respond to feedback, the easier it will go for reviewers. Really you should just read Glen Sanford's excellent post on code review.

Ready to Merge

Okay! Someone gave you a LGTM and it's time to merge. As a part of the code review process, you may have ended up with a bunch of commits that look like this:

These commits are detritus, the prototypes of the creative process, and they shouldn't be part of the permanent record. Why fix them? Because six months from now, when you're staring at a piece of confusing code and trying to figure out why it's written the way it is, you really want to see a commit message explaining the change that looks like this, and not one that says "fix tests".

There are two ways to get rid of these:

Git Amend

If you just did a commit and have a new change that should be part of the same commit, use git amend to add changes to the current commit. If you don't need to change the message, use git amend --no-edit (I map this to git alter in my git config).

Git Rebase

You want to squash all of those typo fix commits into one. Steve Klabnik has a good guide for how to do this. I use this script, saved as rb:

    local branch="$1"
    if [[ -z "$branch" ]]; then
        branch='master'
    fi
    BRANCHREF="$(git symbolic-ref HEAD 2>/dev/null)"
    BRANCHNAME=${BRANCHREF##refs/heads/}
    if [[ "$BRANCHNAME" == "$branch" ]]; then
        echo "Switch to a branch first"
        exit 1
    fi
    git checkout "$branch"
    git pull origin "$branch"
    git checkout "$BRANCHNAME"
    if [[ -n "$2" ]]; then
        git rebase "$branch" "$2"
    else
        git rebase "$branch"
    fi
    git push origin --force "$BRANCHNAME"

If you run that with rb master, it will pull down the latest master from origin, rebase your branch against it, and force push to your branch on origin. Run rb master -i and select "squash" to squash your commits down to one.

As a side effect of rebasing, you'll resolve any merge conflicts that have developed between the time you opened the pull request and the merge time! This can take the headache away from the person doing the merge, and help prevent mistakes.

Logging Database Queries in CircleCI

Our test suite has been failing maybe 2% of the time with a pretty opaque error message. I thought the database logs would have more information about the failure so I figured out how to turn them on with Circle.

Add the following to the database section of your circle.yml:


database:
  override:
    - sudo sed -i "s/#logging_collector = off/logging_collector = on/" /etc/postgresql/9.4/main/postgresql.conf
    - sudo sed -i "s/#log_statement = 'none'/log_statement = 'all'/" /etc/postgresql/9.4/main/postgresql.conf
    - sudo sed -i "s/#log_duration = off/log_duration = on/" /etc/postgresql/9.4/main/postgresql.conf
    - sudo sed -i "s/#log_disconnections = off/log_disconnections = on/" /etc/postgresql/9.4/main/postgresql.conf
    - sudo mkdir -p /var/lib/postgresql/9.4/main/pg_log
    - sudo chown -R postgres:postgres /var/lib/postgresql/9.4/main/pg_log
    - sudo service postgresql restart

Those settings will:

  • enable various logging settings in the Postgres config
  • create the pg_log directory (Postgres can't write if the directory doesn't exist)
  • change ownership of the directory to the postgres user
  • restart Postgres (needed to enable logging).

Finally to trigger the error you're going to want to run the tests a bunch of times.


test:
  override:
    - while true; do your_test_command | tee -a test.log ; if [[ ${PIPESTATUS[0]} -ne 0 ]]; then break; fi; done; exit 1

Note you need to use the ${PIPESTATUS} array instead of $? because you're piping the output of your test runner to the test log. You're going to want to pipe the output of your test runner to the test log because Circle can't render text beyond a certain threshold.

Finally, enable SSH on your build so you can view the test log file you've been writing, and your database logs, which will be in /var/lib/postgresql/9.4/main/pg_log.

Happy hunting!

Node’s `require` is dog slow

Our test environment takes 6-9 seconds to load before any tests get run. I tire of this during the ~30 times I run the test suite a day,1 so I wanted to make it faster.

For better or worse, the API runs on Sails.js. Before running model/controller tests, a bootstrap file in our tests calls sails.lift.

require('sails').lift(function(err) {
    // Run the tests
});

This lift call generates about 400 queries against Postgres to retrieve database schema, that each look like this:

SELECT x.nspname || '.' || x.relname as "Table", x.attnum
as "#", x.attname as "Column", x."Type", case x.attnotnull
when true then 'NOT NULL' else '' end as "NULL",
r.conname as "Constraint", r.contype as "C", r.consrc,
fn.nspname || '.' || f.relname as "F Key", d.adsrc as "Default"
FROM (SELECT c.oid, a.attrelid, a.attnum, n.nspname, c.relname,
a.attname, pg_catalog.format_type(a.atttypid, a.atttypmod) as "Type",
a.attnotnull FROM pg_catalog.pg_attribute a, pg_namespace n,
pg_class c WHERE a.attnum > 0 AND NOT a.attisdropped
AND a.attrelid = c.oid and c.relkind not in ('S','v')
and c.relnamespace = n.oid and n.nspname
not in ('pg_catalog','pg_toast','information_schema')) x
left join pg_attrdef d on d.adrelid = x.attrelid
and d.adnum = x.attnum left join pg_constraint r on r.conrelid = x.oid
and r.conkey[1] = x.attnum left join pg_class f on r.confrelid = f.oid
left join pg_namespace fn on f.relnamespace = fn.oid
where x.relname = 'credits' order by 1,2;

I'm not really sure what those queries do, since Sails seems to ignore the schema that's already in the database when generating table queries. Since we use the smallest, safest subset of the ORM we can find, I tried commenting out the sails-postgresql module code that makes those queries to see if the tests would still pass, and the tests did pass... but the load time was still slow.

The next step was to instrument the code to figure out what was taking so long. I wanted to have the Sails loader log the duration of each part of the load process, but this would have required a global variable, and a whole bunch of calls to console.log. It turns out the Unix function ts can do this for you, if log lines are generated at the appropriate times. Basically, it's an instant awesome tool for generating insight into a program's runtime, without needing to generate timestamps in the underlying tool.

NAME
       ts - timestamp input
SYNOPSIS
       ts [-r] [-i | -s] [format]
DESCRIPTION
       ts adds a timestamp to the beginning of each line of input.

I set the Sails logging level to verbose and piped the output through ts '[%Y-%m-%d %H:%M:%.S]'. I pretty quickly found a culprit..

[2015-04-19 21:53:45.730679] verbose: request hook loaded successfully.
[2015-04-19 21:53:45.731032] verbose: Loading the app's models and adapters...
[2015-04-19 21:53:45.731095] verbose: Loading app models...
[2015-04-19 21:53:47.928104] verbose: Loading app adapters...
[2015-04-19 21:53:47.929343] verbose: Loading blueprint middleware...

That's a 2 second gap between loading the models and loading the adapters.

I started adding profiling to the code near the "Loading app models..." line. I expected to see that attaching custom functions (findOne, update, etc) to the Sails models was the part that took so long. Instead I found out that a module called include-all accounted for almost all of the 2.5 seconds. That module simply requires every file in the models directory, about 30 files.

Further reading/logging revealed that each require call was being generated in turn. I've found it, I thought, just require them all at the same time and see a speedup. Unfortunately, the require operation is synchronous in Node, so it doesn't matter if you throw async pixie dust at it, the process can still only perform one require at a time.

I tried just loading one model to see how slow that would be. This script took an average of 700 milliseconds to run, on my high end Macbook Pro:

var start = Date.now();
require('api/models/Drivers');
console.log(Date.now() - start);

700 milliseconds to require a model file, and that file's dependencies! I can send a packet to and from New York 8 times in that amount of time. What the hell is it actually doing? For this I turned to good old strace (or dtruss, as it's ported on Macs). First start up a shell to record syscalls for any process that is called node.

# (You'll want to kill any other node processes before running this.)
sudo dtruss -d -n 'node' > /tmp/require.log 2>&1

Open up another shell session and run your little script that calls require and prints the startup time and then exits. You should have a few thousand lines in a file called /tmp/require.log. Here's what I found near the start:

   1186335 stat64("/Users/burke/code/api/api/models/node_modules/async\0", 0x7FFF5FBFE608, 0x9)             = -1 Err#2
   1186382 stat64("/Users/burke/code/api/api/models/node_modules/async.js\0", 0x7FFF5FBFE5B8, 0x9)          = -1 Err#2
   1186405 stat64("/Users/burke/code/api/api/models/node_modules/async.json\0", 0x7FFF5FBFE5B8, 0x9)                = -1 Err#2
   1186423 stat64("/Users/burke/code/api/api/models/node_modules/async.node\0", 0x7FFF5FBFE5B8, 0x9)                = -1 Err#2
   1186438 stat64("/Users/burke/code/api/api/models/node_modules/async.coffee\0", 0x7FFF5FBFE5B8, 0x9)              = -1 Err#2
   1186473 open("/Users/burke/code/api/api/models/node_modules/async/package.json\0", 0x0, 0x1B6)           = -1 Err#2
   1186501 stat64("/Users/burke/code/api/api/models/node_modules/async/index.js\0", 0x7FFF5FBFE5B8, 0x1B6)          = -1 Err#2
   1186519 stat64("/Users/burke/code/api/api/models/node_modules/async/index.json\0", 0x7FFF5FBFE5B8, 0x1B6)                = -1 Err#2
   1186534 stat64("/Users/burke/code/api/api/models/node_modules/async/index.node\0", 0x7FFF5FBFE5B8, 0x1B6)                = -1 Err#2
   1186554 stat64("/Users/burke/code/api/api/models/node_modules/async/index.coffee\0", 0x7FFF5FBFE5B8, 0x1B6)              = -1 Err#2
   1186580 stat64("/Users/burke/code/api/api/node_modules/async\0", 0x7FFF5FBFE608, 0x1B6)          = -1 Err#2
   1186598 stat64("/Users/burke/code/api/api/node_modules/async.js\0", 0x7FFF5FBFE5B8, 0x1B6)               = -1 Err#2
   1186614 stat64("/Users/burke/code/api/api/node_modules/async.json\0", 0x7FFF5FBFE5B8, 0x1B6)             = -1 Err#2
   1186630 stat64("/Users/burke/code/api/api/node_modules/async.node\0", 0x7FFF5FBFE5B8, 0x1B6)             = -1 Err#2
   1186645 stat64("/Users/burke/code/api/api/node_modules/async.coffee\0", 0x7FFF5FBFE5B8, 0x1B6)           = -1 Err#2
   1186670 open("/Users/burke/code/api/api/node_modules/async/package.json\0", 0x0, 0x1B6)          = -1 Err#2
   1186694 stat64("/Users/burke/code/api/api/node_modules/async/index.js\0", 0x7FFF5FBFE5B8, 0x1B6)                 = -1 Err#2
   1186712 stat64("/Users/burke/code/api/api/node_modules/async/index.json\0", 0x7FFF5FBFE5B8, 0x1B6)               = -1 Err#2
   1186727 stat64("/Users/burke/code/api/api/node_modules/async/index.node\0", 0x7FFF5FBFE5B8, 0x1B6)               = -1 Err#2
   1186742 stat64("/Users/burke/code/api/api/node_modules/async/index.coffee\0", 0x7FFF5FBFE5B8, 0x1B6)             = -1 Err#2
   1186901 stat64("/Users/burke/code/api/node_modules/async\0", 0x7FFF5FBFE608, 0x1B6)              = 0 0
   1186963 stat64("/Users/burke/code/api/node_modules/async.js\0", 0x7FFF5FBFE5B8, 0x1B6)           = -1 Err#2
   1187024 stat64("/Users/burke/code/api/node_modules/async.json\0", 0x7FFF5FBFE5B8, 0x1B6)                 = -1 Err#2
   1187050 stat64("/Users/burke/code/api/node_modules/async.node\0", 0x7FFF5FBFE5B8, 0x1B6)                 = -1 Err#2
   1187074 stat64("/Users/burke/code/api/node_modules/async.coffee\0", 0x7FFF5FBFE5B8, 0x1B6)               = -1 Err#2
      1656 __semwait_signal(0x10F, 0x0, 0x1)                = -1 Err#60
   1187215 open("/Users/burke/code/api/node_modules/async/package.json\0"0x0, 0x1B6)              = 11 0

That's a lot of wasted open()'s, and that's just to find one dependency.2 To load the single model, node had to open and read 300 different files,3 and every require which wasn't a relative dependency did the same find-the-node-module-folder dance. The documentation seems to indicate this is desired behavior, and it doesn't seem like there is any way around this.

Now, failed stat's are not the entire reason require is slow, but if I am reading the timestamps correctly in the strace log, they are a fair amount of it, and the most obviously wasteful thing. I could rewrite every require to be relative, e.g. require('../../node_modules/async') but that seems cumbersome and wasteful, when I can define the exact rule I want before hand: if it's not a relative file path, look in node_modules in the top level of my project.

So that's where we are; require for a single model takes 700 milliseconds, require for all the models takes 2.5 seconds, and there don't seem to be great options for speeding that up. There's some discouraging discussion here from core developers about the possibility of speeding up module import.

You are probably saying "load fewer dependencies", and you are right and I would love to, but that is not an option at this point in time, since "dependencies" basically equals Sails, and while we are trying to move off of Sails, we're stuck on it for the time being. Where I can, I write tests that work with objects in memory and don't touch our models/controllers, but Destroy All Software videos only get you so far.

I will definitely think harder about importing another small module vs. just copying the code/license wholesale into my codebase, and I'll definitely look to add something that can warn about unused imports or variables in the code base.

I'm left concluding that importing modules in Node is dog slow. Yes, 300 imports is a lot, but a 700ms load time seems way too high. Python imports can be slow, but as far as I remember, it's mostly for things that compile C on the fly, and for everything else you can work around it by rearranging sys.path to match the most imports first (impossible here). If there's anything I can do - some kind of compile option, or saving the V8 bytecode or something, I'm open to suggestions.

1. If you follow along with the somewhat-crazy convention of crashing your Node process on an error and having a supervisor restart it, that means that your server takes 6-9 seconds of downtime as well.

2. The story gets even worse if you are writing Coffeescript, since require will also look for files matching .litcoffee and .coffee.md at every level. You can hack require.extensions to delete these keys.

3. For unknown reasons, node occasionally decided not to stat/open some imports that were require'd in the file.

Profiling ZSH startup time

Recently I had a very weird problem with iTerm where new login shells were being created with environment variables already present. Restarting my machine made the issue go away, and I wasn't able to reproduce it again.

But I got curious about how long ZSH spends in various parts of the startup process. A new /bin/bash login shell loads instantaneously and I was wondering why my ZSH startup was so slow, and if there was anything I could do to make it faster. My goal was to get to a command prompt in under 100 milliseconds.

What files run when you start a ZSH login shell?

In order, your machine will load/execute the following files when ZSH starts:

/etc/zshenv
~/.zshenv
/etc/zprofile
~/.zprofile
/etc/zshrc
~/.zshrc
/etc/zlogin
~/.zlogin

On my machine only /etc/zshenv (which adds paths in /etc/paths* to the $PATH variable) and ~/.zshrc actually existed, which simplified the problem somewhat.

I used the following two scripts to check execution time. First, this snippet logged the execution time of every command run by my startup script in ~/tmp/startlog.<pid>.

    PROFILE_STARTUP=false
    if [[ "$PROFILE_STARTUP" == true ]]; then
        # http://zsh.sourceforge.net/Doc/Release/Prompt-Expansion.html
        PS4=$'%D{%M%S%.} %N:%i> '
        exec 3>&2 2>$HOME/tmp/startlog.$$
        setopt xtrace prompt_subst
    fi
    # Entirety of my startup file... then
    if [[ "$PROFILE_STARTUP" == true ]]; then
        unsetopt xtrace
        exec 2>&3 3>&-
    fi

Essentially this prints the command that zsh is running before you run it. The PS4 variable controls the output of this print statement, and allows us to insert a timestamp each time we run it.

I then wrote a short python script to print all lines above a certain threshold. This is okay, but can show a lot of detail and won't show you if one line in your .zshrc is causing thousands of lines of execution, which take a lot of time in total.

Observations

At first I used the GNU date program in the PS4 prompt to get millisecond information about the time, however this program consistently used 3 milliseconds to run, so it got costly to run it thousands of times during system profiling.

The first thing you find is that shelling out is very expensive, especially to high-level languages. Running something like brew --prefix takes 50 milliseconds which is half of the budget. I found that Autojump's shell loader ran brew --prefix twice so I submitted this pull request to cache the output of that and run it again.

Another timing method

Ultimately what I want is to time specific lines/blocks of my zshrc, instead of get profiling information for specific lines. I did this by wrapping them in time commands, like this:

    { time (
        # linux
        (( $+commands[gvim] )) && {
            alias vi=gvim;
            alias svi='sudo gvim'
        }
        # set up macvim, if it exists
        (( $+commands[mvim] )) && {
            alias vi=mvim;
            alias svi='sudo mvim'
        }
    ) }

This will time commands in a sub-shell, which means that any environment variables set in the sub-shell won't be set in the environment. However the purpose is to get timing information and it's good enough for that.

Lazy loading

By far the worst offenders were the various "Add this to your .zshrc" scripts that I had added in the past - virtualenvwrapper, travis, git-completion, autojump, pyenv, and more. I wanted to see if there was a way to load these only when I needed them (I don't, frequently). Turns out there is! Most of these set functions in zsh, so I can shadow them with my own functions in a zshrc. Once the file with the actual function definition is sourced, it'll replace the shim and I'll be fine. Here's an example for autojump:

    function j() {
        (( $+commands[brew] )) && {
            local pfx=$(brew --prefix)
            [[ -f "$pfx/etc/autojump.sh" ]] && . "$pfx/etc/autojump.sh"
            j "$@"
        }
    }

Or for pyenv:

pyenv() {
    eval "$( command pyenv init - )"
    pyenv "$@"
}

Essentially on the first invocation, these functions source the actual definition and then immediately call it with the arguments passed in.

Conclusion

Ultimately I was able to shave my zsh startup time from 670 milliseconds to about 390 milliseconds and I have ideas on how to shave it further (rewriting my weirdfortune program in Go for example, to avoid the Python/PyPy startup cost). You can probably get similar gains from examining your own zshrc.

Hacking Roller Coaster Tycoon with Genetic Algorithms

I used to play a ton of Roller Coaster Tycoon when I was a kid. I loved the game but I was never very good at making the roller coasters. They always felt too spread out, or too unnatural looking. As a ten year old I idly wondered about writing a computer program that was really good at playing the game. What sort of parks would it make? How would a computer approach the freedoms inherent in an empty park? What would we learn about the game engine from doing so?

A cool coaster

Sadly, this one wasn't generated by a computer

In case you're not familiar, Roller Coaster Tycoon is a amusement park simulation game most notable because the entire game was written in x86 assembler by Chris Sawyer.

Finally a few months ago, I had the tools and the free time available to work on this, and I made some progress toward writing a program that would generate cool looking roller coasters. Let's examine the parts of this program in turn.

Interacting with the Game

So let's say you have a program that can generate roller coasters. How do you actually put them in the game, or integrate them into your parks?

Fortunately, Roller Coaster Tycoon has a format for saving track layouts to disk. Even more amazingly, this format has been documented.

4B: departure control flags
4C number of trains
4D number of cars per train
4E: minimum wait time in seconds
4F: maximum wait time in seconds

Once you decode the ride data, it follows a format. Byte 0 stores the ride type - 00000010 is a suspended steel roller coaster, for example. Some bytes indicate the presence of flags - the 17th bit tells you whether the coaster can have a vertical loop. And so on, and so on.

To compress space, RCT used a cheap run-length encoding algorithm that would compress duplicates of the same byte. But once you encoded/decoded the file, it was super easy to get it running in the game.

So, great! I could write my coaster generator in any language I wanted, write out the file to disk, then load it from any of the parks.

Getting Track Data

There are a lot of track pieces in the game, and I needed to get a lot of data about each of them to be able to make assertions about generated roller coasters. As an example, if the track is currently banked left, which pieces are even possible to construct next?

A steep upward slope track piece increases the car height 4 units. A sharp left turn would advance the car 3 squares forward and 3 squares left, and also rotate the car's direction by 90 degrees. I had less than zero interest in doing this all by hand, and would probably make a mistake doing it. So I went looking for the source of truth in the game..

OpenRCT2

Literally the same week that I started looking at this, Ted John started decompiling the Roller Coaster Tycoon 2 source from x86 into C, and posting the results on Github. More importantly (for me), Ted and the source code actually showed how to read and decompile the source of the game.

The repository shipped with an EXE that would load the C sources before the x86 game code. From there, the C code could (and did, often) use assembler calls to jump back into the game source, for parts that hadn't been decompiled yet.

This also introduced me to the tools you use to decompile x86 into C. We used the reverse engineering tool IDA Pro to read the raw assembly, with a shared database that had information on subroutines that had been decompiled. Using IDA is probably as close as I will come to a profession in code-breaking and/or reverse engineering.

Most of the time with IDA involved reading, annotating the code, and then double checking your results against other parts of the code, the same way you might annotate a crossword puzzle. Other times I used guess and check - change a value in the code, then re-run the game and see what specifically had changed, or use debugging statements to see what went on.

So I started looking for the track data in the game. This turned out to be really, really difficult. You would have a hunch, or use the limited search capability in the game to search for something you thought should be there. Ultimately, I figured out where the strings "Too high!" and "Too low!" were being called in the engine, figuring that track height data would have been computed or used near those points.

This was only part of the solution - it turns out that track data is not stored in one big map but in several maps all around the code base. Some places store information about track bank, some store information about heights and it's tricky to compile it all together. Ultimately, I was able to figure it out by spending enough time with the code and testing different addresses to see if the values there lined up with the pre-determined track order.

Visualizing rides

With a genetic algorithm you are going to be generating a lot of roller coasters. I wanted a quick way to see whether those roller coasters were getting better or not by plotting them. So I used Go's image package to draw roller coasters. To start I didn't try for an isometric view, although that would be fun to draw. Instead I just plotted height change in one image and x/y changes in another image. Running this against existing roller coasters also revealed some flaws in my track data.

A fitness function

A good fitness function will have penalties/rewards for various pieces of behavior.

  • Is the ride complete?

  • Does the ride intersect itself at any points?

  • Does the ride respect gravity, e.g. will a car make it all the way around the track?

  • How exciting is the ride, per the in-game excitement meter?

  • How nauseating is the ride, per the in-game excitement meter?

The first two points on that list are easy; the last three are much more difficult. Finding the excitement data was very tricky. I eventually found it by getting the excitement for a "static" ride with no moving parts (the Crooked House) and searching for the actual numbers used in the game. Here's the function that computes excitement, nausea and intensity for a Crooked House ride.

sub_65C4D4 proc near
or      dword ptr [edi+1D0h], 2
or      dword ptr [edi+1D0h], 8
mov     byte ptr [edi+198h], 5
call    sub_655FD6
mov     ebx, 0D7h ; ''
mov     ecx, 3Eh ; '>'
mov     ebp, 22h ; '"'
call    sub_65E7A3
call    sub_65E7FB
mov     [edi+140h], bx
mov     [edi+142h], cx
mov     [edi+144h], bp
xor     ecx, ecx
call    sub_65E621
mov     dl, 7
shl     dl, 5
and     byte ptr [edi+114h], 1Fh
or      [edi+114h], dl
retn
sub_65C4D4 endp

Got that? In this case 0xD7 in hex is 215 in decimal, which is the ride's excitement rating. I got lucky that this value is static and not changed by anything, which meant I could search for it from outside the binary. This is then stored in the ride's location in memory (register edi), at the offset 0x140. In between there are a few subroutine calls, which shows that nothing is ever really easy when you are reading x86, as well as calls to functions that I have nothing besides hunches about.

Anyway, when you turn this into C, you get something like this:

void crooked_house_excitement(rct_ride *ride)
{
    // Set lifecycle bits
    ride->lifecycle_flags |= RIDE_LIFECYCLE_TESTED;
    ride->lifecycle_flags |= RIDE_LIFECYCLE_NO_RAW_STATS;
    ride->var_198 = 5;
    sub_655FD6(ride);
    ride_rating excitement  = RIDE_RATING(2,15);
    ride_rating intensity   = RIDE_RATING(0,62);
    ride_rating nausea      = RIDE_RATING(0,34);
    excitement = apply_intensity_penalty(excitement, intensity);
    rating_tuple tup = per_ride_rating_adjustments(ride, excitement, intensity, nausea);
    ride->excitement = tup.excitement;
    ride->intensity = tup.intensity;
    ride->nausea = tup.nausea;
    ride->upkeep_cost = compute_upkeep(ride);
    // Upkeep flag? or a dirtiness flag
    ride->var_14D |= 2;
    // clear all bits except lowest 5
    ride->var_114 &= 0x1f;
    // set 6th,7th,8th bits
    ride->var_114 |= 0xE0;
}

And we're lucky in this case that the function is relatively contained; many places in the code feature jumps and constructs that make following the code pretty tricky.

So this one wasn't too bad, but I got bogged down trying to compute excitement for a ride that had a track. The function gets orders of magnitude more complex than this. One positive is, as far as I can tell, excitement and nausea ratings are wholly functions of overall ride statistics like the vertical and lateral G-forces, and there's no accumulator per track segment.

Most of the computation involves multiplying a ride statistic by a constant, then bit shifting the value so it can't be too high/influence the final number by too much.

And sadly, this is where the project stalled. It was impossible to test the C code, because the track computation functions were buried four subroutines deep, and each of those subroutines had at least 500 lines of code. Decompiling each of these correctly, just to get to the code I wanted, was going to be a massive pain. There are ways around this, but ultimately I got back from vacation and had to focus on more pressing issues.

Conclusion

You can hack Roller Coaster Tycoon! There are a bunch of people doing interesting stuff with the game, including improving the peep UI, working on cross compilation (you can play it on Macs!), adding intrigues like the possibility of a worker strike, removing limitations based on the number of bytes (you can only have 255 rides, for example), and more.

It's been really fun having an utterly useless side project. I learned a lot about registers, calling conventions, bit shifting tricks, and other things that probably won't be useful at all, for anything.

I will definitely revisit this project at some point, hopefully when more of the game has been decompiled, or I might try to dig back into the x86/C more on my own.

Storing Photos for the Long Term

You have photos on your computer. You would probably be really sad if you lost them. Let's discuss some strategies for ensuring that doesn't happen.

What you are doing now is probably not enough.

Your current strategy is to have photos and critical files stored on your most current laptop and maybe some things in Dropbox. This means you will probably lose your data sometime in the next three years.

Hard drives fail, often, and are unrecoverable, or expensive to recover.

The software that manages your photos is written by people who are bad at their jobs, or didn't anticipate the software running on (insert hardware/software condition 10 years from now here) and it breaks, destroying your photos in the process.

Backup services fail, often enough to make you worry. Apple can't be trusted to reliably deliver messages with iMessage, I don't trust Time Machine at all.

Apartments and cars get broken into and laptops/external drives are good candidates for theft. You can buy renters insurance, but you can't get your photos back.

What you should do

I'm going to focus specifically on keeping photos for a long time. For files and folders which change more often, I use git and source code tools like Github or Bitbucket. For work environments that take a while to set up, I try to automate installation with shell scripts instead of trying to store the entire environment via Time Machine or similar.

I'm also going to pick a thirty year time horizon, just for fun. If you want something to last for thirty years, you can't just store it on your local machine because your hard drive will probably fail between now and then, and then you lose your photos. So you want to store it on your machine and then also somewhere offsite (e.g. not in the same 5-mile radius as your machine).

Storage Tool

Which raises the question, which storage/backup companies will be around in 30 years? Any small enough candidate, even Dropbox, could get acquired or shut down in that time frame. My guess right now is that Amazon Web Services will be around the longest, because it is profitable, already part of a large, growing company, and the service is growing rapidly.

Specifically, I am putting a bet on Amazon Glacier, which has extremely low storage costs - $0.01 per GB - and is one of the most reliable services Amazon runs. Glacier is a subset of Simple Storage Service (S3) which has had extraordinarily good availability in the 7-8 years it's been available. In addition Amazon regularly publishes information on the technology underpinning S3, see "Amazon Dynamo paper" for example.

I use Arq to back up photos to Glacier. It seems fairly stable, has software preferences for the right things, and I am encouraged that the author charges for the software, which means he/she has an incentive to continue developing the product and making sure that it works. This is $30 but this is still much, much cheaper than any other tool (iCloud for example would be $100 per year).

I have 52 GB of files, which means I'll pay roughly $7 per year to have 11 years of photos stored safely in the cloud. Not bad.

I would consider Tarsnap as well, which encrypts your data, but it's currently 2.5x the price of Glacier. I expect this price to decline soon.

Photo Management Software

The second piece is you need to choose a stable piece of software for viewing and managing your photos. This is orders of magnitude more risky than Glacier. The ideal piece of software would have:

  • An easy to understand file format, published online

  • Source code available online

  • Some support for grouping photos into albums and importing them from my phone/camera/whatever

  • Some sort of tool for auto-adjusting the layers on a photo, cropping it, editing the brightness/contrast. Not Lightroom, but enough to make a photo better than it was.

  • Supports hundreds of gigabytes of photos without falling over.

I haven't done as much research into this as with backup solutions, but there are a few tools. Picasa is supported by Google and relies on Google's charity to stay running and supported. Lightroom is very nice, but overshoots my needs, is very expensive, and Adobe may run out of money and fold within the next 30 years. This leaves iPhoto, which isn't well documented, or open source, but mostly works, some of the time, can crop/edit photos while saving the originals, and is a core component of Apple's Mac product, which may also die, when we all have tablets. At least iPhoto stores the files on disk.

If I was a better person, I'd manually arrange photos in files on my filesystem and use that. But I am not.

Problems

In this game it's a question of how paranoid you want to be, and how much your photos are worth to you. If they're worth a lot it's worth investing significant time and resources into redundant backup systems. Specifically, the 3-2-1 rule suggests storing backups in 3 different places, 2 different file systems (or photo management tools), with at least one offsite. For photos, backing up to CD's or tapes is not a terrible option though it's not very efficient and CD's also die. Using a 2nd photo management software solution, storing the data for that one on an external drive, and backing one of them up to Glacier every night is not a terrible idea.

My photos are worth a lot to me, but I'm not going to go insanely overboard; if my hard drive dies, and Glacier loses my data, that would suck, but I'm willing to gamble on the one-in-a-million odds of them not both happening simultaneously.

I am most worried about the management software; recently iPhoto decided I had no photos in my library and deleted all of the metadata for those photos (albums, rotation information, etc), leaving only the raw copies in my library. This was very sad, though I didn't lose the photos themselves, and might be able to recover the metadata if I am lucky. So yeah, I am not too happy with this. I now store important iPhoto databases in git, backed up to Bitbucket, so I can pull them back down in the event of another catastrophic failure.

Conclusion

This is too long already. Ensuring your photos aren't lost is difficult, and probably requires professional experience as a computer programmer to get right, or for you to part with a significant amount of money, or achieve a large amount of good luck, to avoid the bad things that happen to most people who just hope their photos stick around. I wish you the best of luck.

Figure out when long-running jobs finish, without stopping them

You kick off a long running job - maybe a data migration script that operates on a large data set, or you're copying a large file from one disk to another, or from the Internet to your local computer.

Then a few minutes in, you realize the job is going to take longer than you thought, and you'd like to trigger some action when it's done - notify you, or remove a temp directory, or something.

Like this: wget reallybigfile.com/bigfile.mp3 && say "file done downloading"

But you can't queue an action without hitting Ctrl+C and restarting the job, setting you back minutes or hours. Or can you?

With most modern shells on Unix, you can suspend the running process, and the Unix machine will freeze the state of the process for you. Simply hit Ctrl+Z while any process is running and you will get a message like this:

$ sleep 10
^Z
[1]  + 72277 suspended  sleep 10

You can then resume it with the fg command, which tells Unix to resume operations with the suspended process. You can then combine fg with the notification command of your choice. So let's say you've suspended the process with Ctrl+Z, you can bring it back to the foreground and attach actions afterwards like so:

fg; say -vzarvox "Job complete."

Of course, you can do whatever you want instead of using the say command; trigger another long running operation or whatever.

I use this probably about once a day, it never fails and it's always useful. Hope it helps you too!

Easy maintenance of your AUTHORS file

It's important to recognize the people that have contributed to your project, but it can be annoying to keep your project's AUTHORS file up to date, and annoying to ask everyone to add themselves in the correct format.

So I did what any good engineer should do, and automated the process! I added a simple make target that will update the authors file based on the current Git history:

    # Run this command to update AUTHORS.md with the latest contributors.
    authors:
        echo "Authors\n=======\nWe'd like to thank the following people for their contributions.\n\n" > AUTHORS.md
        git log --raw | grep "^Author: " | sort | uniq | cut -d ' ' -f2- | sed 's/^/- /' >> AUTHORS.md

Essentially, that snippet gets every author from the git history, sorts and unique-ifies the list, strips the word "Author: " from the line, and outputs the new list to the AUTHORS.md file. Now updating the authors list is as easy as running make authors from the command line.

The SSL library ecosystem needs diversity

The Heartbleed bug was really bad for OpenSSL - it let you ask a server a simple question like "How are you" and then have the server tell you anything it wants (password data, private keys that could be used to decrypt all traffic), and the server would have no idea it was happening.

A lot of people have said that we should ditch OpenSSL because this bug is so bad, and because there are parts of the codebase that are odd, and would usually indicate bad programmers, except that they are found in a library that is deployed everywhere.

Ditching OpenSSL is not going to happen any time soon because it is the standard implementation for any server that has to terminate SSL traffic, and writing good crypto libraries is very difficult. So this is not a promising approach.

However this bug and the subsequent panic (as well as the flood of emails telling me to reset passwords etc) indicate the problem with having every software company in the world rely on the same library. Imagine that there were three different SSL software tools that each had a significant share of the market. A flaw in one could affect, at most, the users of that library. Diversification reduces the value of any one exploit and makes it more difficult to find general attacks against web servers.

This diversity is what makes humans so robust against things like the Spanish Flu, which killed ~100 million people but didn't make a dent on the overall human population. Compare that with the banana, which is susceptible to a virus that could wipe out the entire stock of bananas around the world.

You can see the benefits of diversity in two places. One, even inside the OpenSSL project, users had different versions of the library installed on their servers. This meant that servers that didn't have versions 1.0.1a-f installed (like Twilio) were not vulnerable, which was good.

The second is that servers use different programming languages and different frameworks. This means that the series of Rails CVE's were very bad for Rails deployments but didn't mean anything for anyone else (another good thing).

After Heartbleed I donated $100 to the OpenSSL Foundation, in part because it is really important and in part because it's saved me from having to think about encrypting communication with clients (most of the time) which is really, really neat. I will match that donation to other SSL libraries, under these conditions:

  • The library's source code is available to the public.

  • There is evidence that the code has been used in a production environment to terminate SSL connections.

  • The project has room for more funding.

This is not a very large incentive, but it's at least a step in the right direction; if you want to join my pledge, I'll update the dollar amounts and list your name in this post. A prize of $10 million put a rocket into space; I'm hoping it will help spur diversity in the SSL ecosystem as well.

A look at the new retry behavior in urllib3

"Build software like a tank." I am not sure where I read this, but I think about it a lot, especially when writing HTTP clients. Tanks are incredible machines - they are designed to move rapidly and protect their inhabitants in any kind of terrain, against enemy gunfire, or worse.

HTTP clients often run in unfriendly territory, because they usually involve a network connection between machines. Connections can fail, packets can be dropped, the other party may respond very slowly, or with a new unknown error message, or they might even change the API from under you. All of this means writing an HTTP client like a tank is difficult. Here are some examples of things that a desirable HTTP client would do for you, that are never there by default.

  • If a request fails to reach the remote server, we would like to retry it no matter what. We don't want to wait around for the server forever though, so we want to set a timeout on the connection attempt.

  • If we send the request but the remote server doesn't respond in a timely manner, we want to retry it, but only on requests where it is safe to send the request again - so called idempotent requests.

  • If the server returns an unexpected response, we want to always retry if the server didn't do any processing - a 429, 502 or a 503 response usually indicate this - as well as all idempotent requests.

  • Generally we want to sleep between retries to allow the remote connection/server to recover, so to speak. To help prevent thundering herd problems, we usually sleep with an exponential back off.

Here's an example of how you might code this:

def resilient_request(method, uri, retries):
    while True:
        try:
            resp = requests.request(method, uri)
            if resp.status < 300:
                break
            if resp.status in [429, 502, 503]:
                retries -= 1
                if retries <= 0:
                    raise
                time.sleep(2 ** (3 - retries))
                continue
            if resp.status >= 500 and method in ['GET', 'PUT', 'DELETE']:
                retries -= 1
                if retries <= 0:
                    raise
                time.sleep(2 ** (3 - retries))
                continue
        except (ConnectionError, ConnectTimeoutError):
            retries -= 1
            if retries <= 0:
                raise
            time.sleep(2 ** (3 - retries))
        except TimeoutError:
            if method in ['GET', 'PUT', 'DELETE']:
                retries -= 1
                if retries <= 0:
                    raise
                time.sleep(2 ** (3 - retries))
                continue

Holy cyclomatic complexity, Batman! This suddenly got complex, and the control flow here is not simple to follow, reason about, or test. Better hope we caught everything, or we might end up in an infinite loop, or try to access resp when it has not been set. There are some parts of the above code that we can break into sub-methods, but you can't make the code too much more compact than it is there, since most of it is control flow. It's also a pain to write this type of code and verify its correctness; most people just try once, as this comment from the pip library illustrates. This is a shame and the reliability of services on the Internet suffers.

A better way

Andrey Petrov and I have been putting in a lot of work make it really, really easy for you to write resilient requests in Python. We pushed the complexity of the above code down into the urllib3 library, closer to the request that goes over the wire. Instead of the above, you'll be able to write this:

def retry_callable(method, response):
    """ Determine whether to retry this
    return ((response.status >= 400 and method in IDEMPOTENT_METHODS)
            or response.status in (429, 503))
retry = urllib3.util.Retry(read=3, backoff_factor=2,
                           retry_callable=retry_callable)
http = urllib3.PoolManager()
resp = http.request(method, uri, retries=retry)

You can pass a callable to the retries object to determine the retry behavior you'd like to see. Alternatively you can use the convenience method_whitelist and codes_whitelist helpers to specify which methods to retry.

retry = urllib3.util.Retry(read=3, backoff_factor=2,
                           codes_whitelist=set([429, 500]))
http = urllib3.PoolManager()
resp = http.request(method, uri, retries=retry)

And you will get out the same results as the 30 lines above. urllib3 will do all of the hard work for you to catch the conditions mentioned above, with sane (read: non-intrusive) defaults.

This is coming soon to urllib3 (and with it, to Python Requests and pip); we're looking for a bit more review on the pull request before we merge it. We hope this makes it easier for you to write high performance HTTP clients in Python, and appreciate your feedback!

Thanks to Andrey Petrov for reading a draft of this post.