Tag Archives: code

How to find a Pearson’s correlation and ordinary least squares in Python

I’m going to try to blog more technical stuff, as well as simple recipes, here more. I’m working out the best way to present it – whether to bundle everything together or separate out the technical posts.

Today we’re going to use Python to find a simple correlation, and then fit a straight line to the curve.

First you want to install the SciPy and NumPy libraries – they have a lot of cool functions for Python. On Mac, if you have MacPorts installed, this is trivial:

$ sudo port install py27-numpy py27-scipy

Then you find a Pearson’s correlation as follows:

# The dependent variable
>>> x = [1, -2, 2, 3, 1]
# The independent variable
>>> y = [7.5, -3.5, 14.5, 19, 6.6]
#Find a correlation
>>> from scipy.stats.stats import pearsonr
#First value is the r-value, 2nd is the p-value
>>> pearsonr(x,y)
(0.98139984935586166, 0.0030366388199721478)
# To find the best-fit line, use the numpy directory
>>> from numpy.linalg import lstsq
# Put the x variable in the correct format.
>>> A = numpy.vstack([x, numpy.ones(len(x))]).T
>>> A
array([[ 1.,  1.],
       [-2.,  1.],
       [ 2.,  1.],
       [ 3.,  1.],
       [ 1.,  1.]])
>>> lstsq(A, y)
(array([ 4.5 ,  4.32]), array([ 10.848]), 2, array([ 4.53897844,  1.84327826]))
# 4.5 is the slope, 4.32 is the y-intercept. 10.848 is the sum of the residuals.
# 2 is the rank. The last array are the singular values.

For more details, including how to plot the correlation, see the NumPy documentation here.

Auto-lightbox images in WordPress

I’m starting a new side project, where I’ll have to insert lots of screenshots for each blogpost. I’d like to allow users to enlarge images without having to leave the current page. The best way to do this is with a “lightbox,” which darkens the background and displays the image in the center of the screen.

What I would like is just to post an image to the blog and have the lightboxing and image resizing occur automatically. Unfortunately none of the WordPress plugins let you do this; most of them require you to add an extra link with a special tag. So I wrote this short bit of Javascript which extends a popular WordPress lightbox plugin, Lightbox 2. The script takes an image and wraps an “a” tag around it with the rel=”lightbox” attribute set, which triggers the Lightbox code. It also resizes the image to a maximum of 600 pixels wide or high. Here’s the code:

//get all images inside entry content
jQuery(document).ready(function(){
    jQuery(".entry-content img").each(function(){
//get the URL of the image
        var the_url = jQuery(this).attr('src');
//insert a href before image where rel=lightbox
        var a = jQuery("<a></a>").attr({"href": the_url, "rel": "lightbox"});
        jQuery(this).wrap(a);
        jQuery(this).load(function(){
            var the_width = jQuery(this).width();
            var the_height = jQuery(this).height();
            var max_dimension = 600;
            if (the_width && the_height){
                var ratio = the_width / the_height;
                if (ratio >= 1 && the_width > max_dimension){
                    jQuery(this).css('width', max_dimension);
                    jQuery(this).css('height', the_height * max_dimension / the_width);
                }
                else if (ratio > 1 && the_height > max_dimension){
                    jQuery(this).css('height', max_dimension);
                    jQuery(this).css('width', the_width * max_dimension / the_height);
                }
            }
        });
    });
});

You can also download the file here. To install, edit the header.php file of your WordPress directory and add the following lines above the

<?php wp_head(); ?>

line:

<script src="//ajax.googleapis.com/ajax/libs/jquery/1.4.4/jquery.min.js" type="text/javascript"></script>
<script src="/path/to/img.js file" type="text/javascript"></script>

Then you’re done!

Google Finance killed my weekend project

One annoying thing about Google Finance is that there’s no way to see how your portfolio’s value has changed over time. You can see your cost basis and the current value but none of the intermediate values. Way back when I used to log into MSN Money and record the value in a spreadsheet every day. That didn’t last very long.

Over the weekend I built a tool that keeps track of how much your portfolio’s worth. Every night at midnight it would grab the current value of all your portfolios and store it in my database, and then when you visited my site again, it would display a chart with all of the daily values of your portfolios. I got the entire backend working, and got to the point where the app was displaying all of the daily values on the page in plain text. Here’s my cost basis and a week’s worth of daily values, with some numbers blurred.

This weekend I was going to turn those numbers into a pretty Annotated Time Line using the Google Chart API. Unfortunately (or fortunately), I checked Google Finance today and it turns out that they just built the portfolio value chart into their main product.

Google Finance portfolio view

It’s pretty cool that they put the chart in there; obviously it would have been nice to have visitors to my own site, and I built it so that anyone can log in and check the values, but also a little deflating, that my project won’t be useful for people. Guess I’ll have more time to work on my other projects. In the meantime though, I got better at XML parsing, learned a ton about OAuth, implemented an app with user-specific data for the first time, and got better at programming.

You can grab the code I was using here, or check your own portfolio here.

To learn how you can outperform the Dow by 10%, or to learn more about the app, you should email me here.