Posts Tagged With: Code

Profiling ZSH startup time

Recently I had a very weird problem with iTerm where new login shells were being created with environment variables already present. Restarting my machine made the issue go away, and I wasn't able to reproduce it again.

But I got curious about how long ZSH spends in various parts of the startup process. A new /bin/bash login shell loads instantaneously and I was wondering why my ZSH startup was so slow, and if there was anything I could do to make it faster. My goal was to get to a command prompt in under 100 milliseconds.

What files run when you start a ZSH login shell?

In order, your machine will load/execute the following files when ZSH starts:

/etc/zshenv
~/.zshenv
/etc/zprofile
~/.zprofile
/etc/zshrc
~/.zshrc
/etc/zlogin
~/.zlogin

On my machine only /etc/zshenv (which adds paths in /etc/paths* to the $PATH variable) and ~/.zshrc actually existed, which simplified the problem somewhat.

I used the following two scripts to check execution time. First, this snippet logged the execution time of every command run by my startup script in ~/tmp/startlog.<pid>.

    PROFILE_STARTUP=false
    if [[ "$PROFILE_STARTUP" == true ]]; then
        # http://zsh.sourceforge.net/Doc/Release/Prompt-Expansion.html
        PS4=$'%D{%M%S%.} %N:%i> '
        exec 3>&2 2>$HOME/tmp/startlog.$$
        setopt xtrace prompt_subst
    fi
    # Entirety of my startup file... then
    if [[ "$PROFILE_STARTUP" == true ]]; then
        unsetopt xtrace
        exec 2>&3 3>&-
    fi

Essentially this prints the command that zsh is running before you run it. The PS4 variable controls the output of this print statement, and allows us to insert a timestamp each time we run it.

I then wrote a short python script to print all lines above a certain threshold. This is okay, but can show a lot of detail and won't show you if one line in your .zshrc is causing thousands of lines of execution, which take a lot of time in total.

Observations

At first I used the GNU date program in the PS4 prompt to get millisecond information about the time, however this program consistently used 3 milliseconds to run, so it got costly to run it thousands of times during system profiling.

The first thing you find is that shelling out is very expensive, especially to high-level languages. Running something like brew --prefix takes 50 milliseconds which is half of the budget. I found that Autojump's shell loader ran brew --prefix twice so I submitted this pull request to cache the output of that and run it again.

Another timing method

Ultimately what I want is to time specific lines/blocks of my zshrc, instead of get profiling information for specific lines. I did this by wrapping them in time commands, like this:

    { time (
        # linux
        (( $+commands[gvim] )) && {
            alias vi=gvim;
            alias svi='sudo gvim'
        }
        # set up macvim, if it exists
        (( $+commands[mvim] )) && {
            alias vi=mvim;
            alias svi='sudo mvim'
        }
    ) }

This will time commands in a sub-shell, which means that any environment variables set in the sub-shell won't be set in the environment. However the purpose is to get timing information and it's good enough for that.

Lazy loading

By far the worst offenders were the various "Add this to your .zshrc" scripts that I had added in the past - virtualenvwrapper, travis, git-completion, autojump, pyenv, and more. I wanted to see if there was a way to load these only when I needed them (I don't, frequently). Turns out there is! Most of these set functions in zsh, so I can shadow them with my own functions in a zshrc. Once the file with the actual function definition is sourced, it'll replace the shim and I'll be fine. Here's an example for autojump:

    function j() {
        (( $+commands[brew] )) && {
            local pfx=$(brew --prefix)
            [[ -f "$pfx/etc/autojump.sh" ]] && . "$pfx/etc/autojump.sh"
            j "$@"
        }
    }

Or for pyenv:

pyenv() {
    eval "$( command pyenv init - )"
    pyenv "$@"
}

Essentially on the first invocation, these functions source the actual definition and then immediately call it with the arguments passed in.

Conclusion

Ultimately I was able to shave my zsh startup time from 670 milliseconds to about 390 milliseconds and I have ideas on how to shave it further (rewriting my weirdfortune program in Go for example, to avoid the Python/PyPy startup cost). You can probably get similar gains from examining your own zshrc.

Hacking Roller Coaster Tycoon with Genetic Algorithms

I used to play a ton of Roller Coaster Tycoon when I was a kid. I loved the game but I was never very good at making the roller coasters. They always felt too spread out, or too unnatural looking. As a ten year old I idly wondered about writing a computer program that was really good at playing the game. What sort of parks would it make? How would a computer approach the freedoms inherent in an empty park? What would we learn about the game engine from doing so?

A cool coaster

Sadly, this one wasn't generated by a computer

In case you're not familiar, Roller Coaster Tycoon is a amusement park simulation game most notable because the entire game was written in x86 assembler by Chris Sawyer.

Finally a few months ago, I had the tools and the free time available to work on this, and I made some progress toward writing a program that would generate cool looking roller coasters. Let's examine the parts of this program in turn.

Interacting with the Game

So let's say you have a program that can generate roller coasters. How do you actually put them in the game, or integrate them into your parks?

Fortunately, Roller Coaster Tycoon has a format for saving track layouts to disk. Even more amazingly, this format has been documented.

4B: departure control flags
4C number of trains
4D number of cars per train
4E: minimum wait time in seconds
4F: maximum wait time in seconds

Once you decode the ride data, it follows a format. Byte 0 stores the ride type - 00000010 is a suspended steel roller coaster, for example. Some bytes indicate the presence of flags - the 17th bit tells you whether the coaster can have a vertical loop. And so on, and so on.

To compress space, RCT used a cheap run-length encoding algorithm that would compress duplicates of the same byte. But once you encoded/decoded the file, it was super easy to get it running in the game.

So, great! I could write my coaster generator in any language I wanted, write out the file to disk, then load it from any of the parks.

Getting Track Data

There are a lot of track pieces in the game, and I needed to get a lot of data about each of them to be able to make assertions about generated roller coasters. As an example, if the track is currently banked left, which pieces are even possible to construct next?

A steep upward slope track piece increases the car height 4 units. A sharp left turn would advance the car 3 squares forward and 3 squares left, and also rotate the car's direction by 90 degrees. I had less than zero interest in doing this all by hand, and would probably make a mistake doing it. So I went looking for the source of truth in the game..

OpenRCT2

Literally the same week that I started looking at this, Ted John started decompiling the Roller Coaster Tycoon 2 source from x86 into C, and posting the results on Github. More importantly (for me), Ted and the source code actually showed how to read and decompile the source of the game.

The repository shipped with an EXE that would load the C sources before the x86 game code. From there, the C code could (and did, often) use assembler calls to jump back into the game source, for parts that hadn't been decompiled yet.

This also introduced me to the tools you use to decompile x86 into C. We used the reverse engineering tool IDA Pro to read the raw assembly, with a shared database that had information on subroutines that had been decompiled. Using IDA is probably as close as I will come to a profession in code-breaking and/or reverse engineering.

Most of the time with IDA involved reading, annotating the code, and then double checking your results against other parts of the code, the same way you might annotate a crossword puzzle. Other times I used guess and check - change a value in the code, then re-run the game and see what specifically had changed, or use debugging statements to see what went on.

So I started looking for the track data in the game. This turned out to be really, really difficult. You would have a hunch, or use the limited search capability in the game to search for something you thought should be there. Ultimately, I figured out where the strings "Too high!" and "Too low!" were being called in the engine, figuring that track height data would have been computed or used near those points.

This was only part of the solution - it turns out that track data is not stored in one big map but in several maps all around the code base. Some places store information about track bank, some store information about heights and it's tricky to compile it all together. Ultimately, I was able to figure it out by spending enough time with the code and testing different addresses to see if the values there lined up with the pre-determined track order.

Visualizing rides

With a genetic algorithm you are going to be generating a lot of roller coasters. I wanted a quick way to see whether those roller coasters were getting better or not by plotting them. So I used Go's image package to draw roller coasters. To start I didn't try for an isometric view, although that would be fun to draw. Instead I just plotted height change in one image and x/y changes in another image. Running this against existing roller coasters also revealed some flaws in my track data.

A fitness function

A good fitness function will have penalties/rewards for various pieces of behavior.

  • Is the ride complete?

  • Does the ride intersect itself at any points?

  • Does the ride respect gravity, e.g. will a car make it all the way around the track?

  • How exciting is the ride, per the in-game excitement meter?

  • How nauseating is the ride, per the in-game excitement meter?

The first two points on that list are easy; the last three are much more difficult. Finding the excitement data was very tricky. I eventually found it by getting the excitement for a "static" ride with no moving parts (the Crooked House) and searching for the actual numbers used in the game. Here's the function that computes excitement, nausea and intensity for a Crooked House ride.

sub_65C4D4 proc near
or      dword ptr [edi+1D0h], 2
or      dword ptr [edi+1D0h], 8
mov     byte ptr [edi+198h], 5
call    sub_655FD6
mov     ebx, 0D7h ; ''
mov     ecx, 3Eh ; '>'
mov     ebp, 22h ; '"'
call    sub_65E7A3
call    sub_65E7FB
mov     [edi+140h], bx
mov     [edi+142h], cx
mov     [edi+144h], bp
xor     ecx, ecx
call    sub_65E621
mov     dl, 7
shl     dl, 5
and     byte ptr [edi+114h], 1Fh
or      [edi+114h], dl
retn
sub_65C4D4 endp

Got that? In this case 0xD7 in hex is 215 in decimal, which is the ride's excitement rating. I got lucky that this value is static and not changed by anything, which meant I could search for it from outside the binary. This is then stored in the ride's location in memory (register edi), at the offset 0x140. In between there are a few subroutine calls, which shows that nothing is ever really easy when you are reading x86, as well as calls to functions that I have nothing besides hunches about.

Anyway, when you turn this into C, you get something like this:

void crooked_house_excitement(rct_ride *ride)
{
    // Set lifecycle bits
    ride->lifecycle_flags |= RIDE_LIFECYCLE_TESTED;
    ride->lifecycle_flags |= RIDE_LIFECYCLE_NO_RAW_STATS;
    ride->var_198 = 5;
    sub_655FD6(ride);
    ride_rating excitement  = RIDE_RATING(2,15);
    ride_rating intensity   = RIDE_RATING(0,62);
    ride_rating nausea      = RIDE_RATING(0,34);
    excitement = apply_intensity_penalty(excitement, intensity);
    rating_tuple tup = per_ride_rating_adjustments(ride, excitement, intensity, nausea);
    ride->excitement = tup.excitement;
    ride->intensity = tup.intensity;
    ride->nausea = tup.nausea;
    ride->upkeep_cost = compute_upkeep(ride);
    // Upkeep flag? or a dirtiness flag
    ride->var_14D |= 2;
    // clear all bits except lowest 5
    ride->var_114 &= 0x1f;
    // set 6th,7th,8th bits
    ride->var_114 |= 0xE0;
}

And we're lucky in this case that the function is relatively contained; many places in the code feature jumps and constructs that make following the code pretty tricky.

So this one wasn't too bad, but I got bogged down trying to compute excitement for a ride that had a track. The function gets orders of magnitude more complex than this. One positive is, as far as I can tell, excitement and nausea ratings are wholly functions of overall ride statistics like the vertical and lateral G-forces, and there's no accumulator per track segment.

Most of the computation involves multiplying a ride statistic by a constant, then bit shifting the value so it can't be too high/influence the final number by too much.

And sadly, this is where the project stalled. It was impossible to test the C code, because the track computation functions were buried four subroutines deep, and each of those subroutines had at least 500 lines of code. Decompiling each of these correctly, just to get to the code I wanted, was going to be a massive pain. There are ways around this, but ultimately I got back from vacation and had to focus on more pressing issues.

Conclusion

You can hack Roller Coaster Tycoon! There are a bunch of people doing interesting stuff with the game, including improving the peep UI, working on cross compilation (you can play it on Macs!), adding intrigues like the possibility of a worker strike, removing limitations based on the number of bytes (you can only have 255 rides, for example), and more.

It's been really fun having an utterly useless side project. I learned a lot about registers, calling conventions, bit shifting tricks, and other things that probably won't be useful at all, for anything.

I will definitely revisit this project at some point, hopefully when more of the game has been decompiled, or I might try to dig back into the x86/C more on my own.

Storing Photos for the Long Term

You have photos on your computer. You would probably be really sad if you lost them. Let's discuss some strategies for ensuring that doesn't happen.

What you are doing now is probably not enough.

Your current strategy is to have photos and critical files stored on your most current laptop and maybe some things in Dropbox. This means you will probably lose your data sometime in the next three years.

Hard drives fail, often, and are unrecoverable, or expensive to recover.

The software that manages your photos is written by people who are bad at their jobs, or didn't anticipate the software running on (insert hardware/software condition 10 years from now here) and it breaks, destroying your photos in the process.

Backup services fail, often enough to make you worry. Apple can't be trusted to reliably deliver messages with iMessage, I don't trust Time Machine at all.

Apartments and cars get broken into and laptops/external drives are good candidates for theft. You can buy renters insurance, but you can't get your photos back.

What you should do

I'm going to focus specifically on keeping photos for a long time. For files and folders which change more often, I use git and source code tools like Github or Bitbucket. For work environments that take a while to set up, I try to automate installation with shell scripts instead of trying to store the entire environment via Time Machine or similar.

I'm also going to pick a thirty year time horizon, just for fun. If you want something to last for thirty years, you can't just store it on your local machine because your hard drive will probably fail between now and then, and then you lose your photos. So you want to store it on your machine and then also somewhere offsite (e.g. not in the same 5-mile radius as your machine).

Storage Tool

Which raises the question, which storage/backup companies will be around in 30 years? Any small enough candidate, even Dropbox, could get acquired or shut down in that time frame. My guess right now is that Amazon Web Services will be around the longest, because it is profitable, already part of a large, growing company, and the service is growing rapidly.

Specifically, I am putting a bet on Amazon Glacier, which has extremely low storage costs - $0.01 per GB - and is one of the most reliable services Amazon runs. Glacier is a subset of Simple Storage Service (S3) which has had extraordinarily good availability in the 7-8 years it's been available. In addition Amazon regularly publishes information on the technology underpinning S3, see "Amazon Dynamo paper" for example.

I use Arq to back up photos to Glacier. It seems fairly stable, has software preferences for the right things, and I am encouraged that the author charges for the software, which means he/she has an incentive to continue developing the product and making sure that it works. This is $30 but this is still much, much cheaper than any other tool (iCloud for example would be $100 per year).

I have 52 GB of files, which means I'll pay roughly $7 per year to have 11 years of photos stored safely in the cloud. Not bad.

I would consider Tarsnap as well, which encrypts your data, but it's currently 2.5x the price of Glacier. I expect this price to decline soon.

Photo Management Software

The second piece is you need to choose a stable piece of software for viewing and managing your photos. This is orders of magnitude more risky than Glacier. The ideal piece of software would have:

  • An easy to understand file format, published online

  • Source code available online

  • Some support for grouping photos into albums and importing them from my phone/camera/whatever

  • Some sort of tool for auto-adjusting the layers on a photo, cropping it, editing the brightness/contrast. Not Lightroom, but enough to make a photo better than it was.

  • Supports hundreds of gigabytes of photos without falling over.

I haven't done as much research into this as with backup solutions, but there are a few tools. Picasa is supported by Google and relies on Google's charity to stay running and supported. Lightroom is very nice, but overshoots my needs, is very expensive, and Adobe may run out of money and fold within the next 30 years. This leaves iPhoto, which isn't well documented, or open source, but mostly works, some of the time, can crop/edit photos while saving the originals, and is a core component of Apple's Mac product, which may also die, when we all have tablets. At least iPhoto stores the files on disk.

If I was a better person, I'd manually arrange photos in files on my filesystem and use that. But I am not.

Problems

In this game it's a question of how paranoid you want to be, and how much your photos are worth to you. If they're worth a lot it's worth investing significant time and resources into redundant backup systems. Specifically, the 3-2-1 rule suggests storing backups in 3 different places, 2 different file systems (or photo management tools), with at least one offsite. For photos, backing up to CD's or tapes is not a terrible option though it's not very efficient and CD's also die. Using a 2nd photo management software solution, storing the data for that one on an external drive, and backing one of them up to Glacier every night is not a terrible idea.

My photos are worth a lot to me, but I'm not going to go insanely overboard; if my hard drive dies, and Glacier loses my data, that would suck, but I'm willing to gamble on the one-in-a-million odds of them not both happening simultaneously.

I am most worried about the management software; recently iPhoto decided I had no photos in my library and deleted all of the metadata for those photos (albums, rotation information, etc), leaving only the raw copies in my library. This was very sad, though I didn't lose the photos themselves, and might be able to recover the metadata if I am lucky. So yeah, I am not too happy with this. I now store important iPhoto databases in git, backed up to Bitbucket, so I can pull them back down in the event of another catastrophic failure.

Conclusion

This is too long already. Ensuring your photos aren't lost is difficult, and probably requires professional experience as a computer programmer to get right, or for you to part with a significant amount of money, or achieve a large amount of good luck, to avoid the bad things that happen to most people who just hope their photos stick around. I wish you the best of luck.

Figure out when long-running jobs finish, without stopping them

You kick off a long running job - maybe a data migration script that operates on a large data set, or you're copying a large file from one disk to another, or from the Internet to your local computer.

Then a few minutes in, you realize the job is going to take longer than you thought, and you'd like to trigger some action when it's done - notify you, or remove a temp directory, or something.

Like this: wget reallybigfile.com/bigfile.mp3 && say "file done downloading"

But you can't queue an action without hitting Ctrl+C and restarting the job, setting you back minutes or hours. Or can you?

With most modern shells on Unix, you can suspend the running process, and the Unix machine will freeze the state of the process for you. Simply hit Ctrl+Z while any process is running and you will get a message like this:

$ sleep 10
^Z
[1]  + 72277 suspended  sleep 10

You can then resume it with the fg command, which tells Unix to resume operations with the suspended process. You can then combine fg with the notification command of your choice. So let's say you've suspended the process with Ctrl+Z, you can bring it back to the foreground and attach actions afterwards like so:

fg; say -vzarvox "Job complete."

Of course, you can do whatever you want instead of using the say command; trigger another long running operation or whatever.

I use this probably about once a day, it never fails and it's always useful. Hope it helps you too!

Easy maintenance of your AUTHORS file

It's important to recognize the people that have contributed to your project, but it can be annoying to keep your project's AUTHORS file up to date, and annoying to ask everyone to add themselves in the correct format.

So I did what any good engineer should do, and automated the process! I added a simple make target that will update the authors file based on the current Git history:

    # Run this command to update AUTHORS.md with the latest contributors.
    authors:
        echo "Authors\n=======\nWe'd like to thank the following people for their contributions.\n\n" > AUTHORS.md
        git log --raw | grep "^Author: " | sort | uniq | cut -d ' ' -f2- | sed 's/^/- /' >> AUTHORS.md

Essentially, that snippet gets every author from the git history, sorts and unique-ifies the list, strips the word "Author: " from the line, and outputs the new list to the AUTHORS.md file. Now updating the authors list is as easy as running make authors from the command line.

The SSL library ecosystem needs diversity

The Heartbleed bug was really bad for OpenSSL - it let you ask a server a simple question like "How are you" and then have the server tell you anything it wants (password data, private keys that could be used to decrypt all traffic), and the server would have no idea it was happening.

A lot of people have said that we should ditch OpenSSL because this bug is so bad, and because there are parts of the codebase that are odd, and would usually indicate bad programmers, except that they are found in a library that is deployed everywhere.

Ditching OpenSSL is not going to happen any time soon because it is the standard implementation for any server that has to terminate SSL traffic, and writing good crypto libraries is very difficult. So this is not a promising approach.

However this bug and the subsequent panic (as well as the flood of emails telling me to reset passwords etc) indicate the problem with having every software company in the world rely on the same library. Imagine that there were three different SSL software tools that each had a significant share of the market. A flaw in one could affect, at most, the users of that library. Diversification reduces the value of any one exploit and makes it more difficult to find general attacks against web servers.

This diversity is what makes humans so robust against things like the Spanish Flu, which killed ~100 million people but didn't make a dent on the overall human population. Compare that with the banana, which is susceptible to a virus that could wipe out the entire stock of bananas around the world.

You can see the benefits of diversity in two places. One, even inside the OpenSSL project, users had different versions of the library installed on their servers. This meant that servers that didn't have versions 1.0.1a-f installed (like Twilio) were not vulnerable, which was good.

The second is that servers use different programming languages and different frameworks. This means that the series of Rails CVE's were very bad for Rails deployments but didn't mean anything for anyone else (another good thing).

After Heartbleed I donated $100 to the OpenSSL Foundation, in part because it is really important and in part because it's saved me from having to think about encrypting communication with clients (most of the time) which is really, really neat. I will match that donation to other SSL libraries, under these conditions:

  • The library's source code is available to the public.

  • There is evidence that the code has been used in a production environment to terminate SSL connections.

  • The project has room for more funding.

This is not a very large incentive, but it's at least a step in the right direction; if you want to join my pledge, I'll update the dollar amounts and list your name in this post. A prize of $10 million put a rocket into space; I'm hoping it will help spur diversity in the SSL ecosystem as well.

A look at the new retry behavior in urllib3

"Build software like a tank." I am not sure where I read this, but I think about it a lot, especially when writing HTTP clients. Tanks are incredible machines - they are designed to move rapidly and protect their inhabitants in any kind of terrain, against enemy gunfire, or worse.

HTTP clients often run in unfriendly territory, because they usually involve a network connection between machines. Connections can fail, packets can be dropped, the other party may respond very slowly, or with a new unknown error message, or they might even change the API from under you. All of this means writing an HTTP client like a tank is difficult. Here are some examples of things that a desirable HTTP client would do for you, that are never there by default.

  • If a request fails to reach the remote server, we would like to retry it no matter what. We don't want to wait around for the server forever though, so we want to set a timeout on the connection attempt.

  • If we send the request but the remote server doesn't respond in a timely manner, we want to retry it, but only on requests where it is safe to send the request again - so called idempotent requests.

  • If the server returns an unexpected response, we want to always retry if the server didn't do any processing - a 429, 502 or a 503 response usually indicate this - as well as all idempotent requests.

  • Generally we want to sleep between retries to allow the remote connection/server to recover, so to speak. To help prevent thundering herd problems, we usually sleep with an exponential back off.

Here's an example of how you might code this:

def resilient_request(method, uri, retries):
    while True:
        try:
            resp = requests.request(method, uri)
            if resp.status < 300:
                break
            if resp.status in [429, 502, 503]:
                retries -= 1
                if retries <= 0:
                    raise
                time.sleep(2 ** (3 - retries))
                continue
            if resp.status >= 500 and method in ['GET', 'PUT', 'DELETE']:
                retries -= 1
                if retries <= 0:
                    raise
                time.sleep(2 ** (3 - retries))
                continue
        except (ConnectionError, ConnectTimeoutError):
            retries -= 1
            if retries <= 0:
                raise
            time.sleep(2 ** (3 - retries))
        except TimeoutError:
            if method in ['GET', 'PUT', 'DELETE']:
                retries -= 1
                if retries <= 0:
                    raise
                time.sleep(2 ** (3 - retries))
                continue

Holy cyclomatic complexity, Batman! This suddenly got complex, and the control flow here is not simple to follow, reason about, or test. Better hope we caught everything, or we might end up in an infinite loop, or try to access resp when it has not been set. There are some parts of the above code that we can break into sub-methods, but you can't make the code too much more compact than it is there, since most of it is control flow. It's also a pain to write this type of code and verify its correctness; most people just try once, as this comment from the pip library illustrates. This is a shame and the reliability of services on the Internet suffers.

A better way

Andrey Petrov and I have been putting in a lot of work make it really, really easy for you to write resilient requests in Python. We pushed the complexity of the above code down into the urllib3 library, closer to the request that goes over the wire. Instead of the above, you'll be able to write this:

def retry_callable(method, response):
    """ Determine whether to retry this
    return ((response.status >= 400 and method in IDEMPOTENT_METHODS)
            or response.status in (429, 503))
retry = urllib3.util.Retry(read=3, backoff_factor=2,
                           retry_callable=retry_callable)
http = urllib3.PoolManager()
resp = http.request(method, uri, retries=retry)

You can pass a callable to the retries object to determine the retry behavior you'd like to see. Alternatively you can use the convenience method_whitelist and codes_whitelist helpers to specify which methods to retry.

retry = urllib3.util.Retry(read=3, backoff_factor=2,
                           codes_whitelist=set([429, 500]))
http = urllib3.PoolManager()
resp = http.request(method, uri, retries=retry)

And you will get out the same results as the 30 lines above. urllib3 will do all of the hard work for you to catch the conditions mentioned above, with sane (read: non-intrusive) defaults.

This is coming soon to urllib3 (and with it, to Python Requests and pip); we're looking for a bit more review on the pull request before we merge it. We hope this makes it easier for you to write high performance HTTP clients in Python, and appreciate your feedback!

Thanks to Andrey Petrov for reading a draft of this post.

How to create rich links in your Sphinx documentation

This will be short, but it seems there's some difficulty doing this, so I thought I'd share.

The gist is, any time you reference a class or method in your own library, in the Python standard library, or in another third-party extension, you can provide a link directly to that project's documentation. This is pretty amazing and only requires a little bit of extra work from you. Here's how.

The Simplest Type of Link

Just create a link using the full import path of the class or attribute or method. Surround it with backticks like this:

Use :meth:`requests.Request.get` to make HTTP Get requests.

That link will show up in text as:

Use requests.Request.get to make HTTP Get requests.

There are a few different types of declarations you can use at the beginning of that phrase:

:attr:
:class:
:meth:
:exc:

The full list is here.

I Don't Want to Link the Whole Thing

To specify just the method/attribute name and not any of the modules or classes that precede it, use a squiggly, like this:

Use :meth:`~requests.Request.get` to make HTTP Get requests.

That link will show up in text as:

Use get to make HTTP Get requests.

I Want to Write My Own Text

This gets a little trickier, but still doable:

Use :meth:`the get() method <requests.Request.get>` to make HTTP Get requests.

That link will show up in text as:

Use the get() method to make HTTP Get requests.

I want to link to someone else's docs

In your docs/conf.py file, add 'sphinx.ext.intersphinx' to the end of the extensions list near the top of the file. Then, add the following anywhere in the file:

    # Add the "intersphinx" extension
    extensions = [
        'sphinx.ext.intersphinx',
    ]
    # Add mappings
    intersphinx_mapping = {
        'urllib3': ('http://urllib3.readthedocs.org/en/latest', None),
        'python': ('http://docs.python.org/3', None),
    }

You can then link to other projects' documentation and then reference it the same way you do your own projects, and Sphinx will magically make everything work.

I want to write the documentation inline in my source code and link to it

Great! I love this as well. Add the 'sphinx.ext.autodoc' extension, then write your documentation. There's a full guide to the inline syntax on the Sphinx website; confusingly, it is not listed on the autodoc page.

    # Add the "intersphinx" extension
    extensions = [
        'sphinx.ext.autodoc',
    ]

Hope that helps! Happy linking.

New blog post about HAProxy

Over on the Twilio Engineering Blog, I have a new post about optimizing your HAProxy configuration. I wrote this mostly because we had some confusion in our configuration about setting options, and if I had it I figured others would as well. Here's a sample:

retries 2
option redispatch
When I said a 30 second connect timeout meant HAProxy would try a bad connection for 30 seconds, I lied. It turns out that by default HAProxy will retry the connect attempt 3 times. So our 30 second connect timeout is actually a 120 second connect timeout, blowing through our SLA and meaning we're returning an empty response to the customer.

Read the full post to learn more about HAProxy.

Automating your IPython Notebook Setup (and getting launchctl to work)

Recently I've fallen in love with the IPython Notebook. It's the Python REPL on steroids and I've probably just scratched the surface of what it can actually do. This will be a short post because long posts make me feel pain when I think about blogging more again. This is also really more about setting up launchctl than IPython, but hopefully that's useful too.

Starting it from the command line is kind of a pain (it tries to save .ipynb files in your current directory, it warns you to save files before closing tabs) so I thought I'd just set it up to run in the background each time I run ipython. Here's how you can get that set up.

Create a virtualenv with iPython

First, you need to install the ipython binary, and the other packages you need to run IPython Notebook.

    # Install virtualenvwrapper, then source it
    pip install virtualenvwrapper
    source /path/to/virtualenvwrapper.sh

mkvirtualenv ipython
pip install ipython tornado pyzmq

Starting IPython When Your Mac Boots

Open a text editor and add the following:

    <?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
    <plist version="1.0">
    <dict>
      <key>Label</key>
      <string>com.kevinburke.ipython</string>
      <key>ProgramArguments</key>
      <array>
          <string>/Users/kevin/.envs/ipython/bin/ipython</string>
          <string>notebook</string>
      </array>
      <key>RunAtLoad</key>
      <true/>
      <key>StandardOutPath</key>
      <string>/Users/kevin/var/log/ipython.log</string>
      <key>StandardErrorPath</key>
      <string>/Users/kevin/var/log/ipython.err.log</string>
      <key>ServiceDescription</key>
      <string>ipython notebook runner</string>
      <key>WorkingDirectory</key>
      <string>/Users/kevin/.ipython_notebooks</string>
    </dict>
    </plist>

You will need to replace the word kevin with your name and relevant file locations on your file system. I also save my notebooks in a directory called .ipython_notebooks in my home directory, you may want to add that as well.

Save that in /Library/LaunchDaemons/<yourname>.ipython.plist. Then change the owner to root:

sudo chown root:wheel /Library/LaunchDaemons/<yourname>.ipython.plist

Finally load it:

sudo launchctl load -w /Library/LaunchDaemons/<yourname>.ipython.plist

If everything went ok, IPython should open in a tab. If it didn't go okay, check /var/log/system.log for errors, or one of the two logfiles specified in your plist.

Additional Steps

That's it! I've also found it really useful to run an nginx redirecter locally, as well as a new rule in /etc/hosts, so I can visit http://ipython and get redirected to my notebooks. But that is a topic for a different blog post.