Posts Tagged With: Today’s World

Ethical Considerations for Software Engineers

The next president of the United States showed a willingness to violate historical norms while campaigning, and there's little evidence that he has any moral compass - the examples of this are legion, one of the worst is him cutting off medical treatment to his sick nephew over a legal dispute. His kids are going to run his businesses (with his name on them) while he is in office. He has also asked for security clearances for them. This is at best an unusual arrangement and at worst opens the door to massive corruption.

During the election the Russian government hacked and leaked the DCC's emails, then hacked and leaked the email of Hillary Clinton's campaign chief. Trump denied Russia's involvement publicly at a debate even though he'd been briefed on it. Trump has taken many sides on many issues but praise for Putin and Russia has been consistent. Trump just promoted a paid Russia Today commentator to his National Security Adviser. It is likely that Russian (and Chinese, Iranian, etc) hacking of US government offices and US companies will be tolerated over the next four years, especially if it benefits Trump and hurts his political allies.

It's important to note these attacks won't come out of the blue. It's not sunny one day and the next there are men in suits asking for data center access. There will probably be some pretext - a foreign war, a terror attack, something else, that'll be used to justify the unethical request. It's easy to imagine "Of course I will identify the racist thing!" and much harder in the moment, or in a context that's surrounded by fear.

Note also that if you are an engineer, these requests may come outside of normal channels. Last year, Yahoo fielded a request to search all emails for a given term. Yahoo's C-level executives went around the security team and asked engineers to implement this directly, at an extremely low level. Alex Stamos, Yahoo's CSO, resigned when he found out. You should be prepared to do the same. Don't expect unethical requests to show up on the backlog - it'll be a meeting you're pulled into with the CTO, or a man showing up at your apartment and threatening your immigration status unless you insert a backdoor.

Employees (and especially engineers) will be the key people to push back. Customers aren't always aware of shenanigans, and management can be under more pressure to make their company succeed. Especially in Silicon Valley, most employees have multiple job options, which gives us unique leverage. Every employee at a Silicon Valley company should be prepared for unethical or illegal requests, and (where appropriate) be prepared for state sponsored attacks, from the US government or another one. Every employee should be prepared to put pressure on management, and the legal team, to deny requests.

Here are some examples of ethical problems you might run into. I'd encourage you to have these discussions internally before you get put in the situation discussed below, and lay out bright lines for everyone in the company to follow, to make it clear where you stand and what's not acceptable. I would also encourage you to ask about these when you interview.

All

The pledge at neveragain.tech has covered this in more detail but here are some good questions to ask in an interview:

  • Do you encrypt messages that go from datacenter to datacenter? The NSA has spied on this data in the past.

  • Do you offer end-to-end encryption of messages sent between users?

  • Do you destroy sensitive data if it's not needed anymore? Do you destroy user data if they delete their accounts?

  • What is your policy to responding to requests from the US government and other governments?

  • Do you have data that would be valuable to foreign governments, or embarrassing to customers if it was made public? What's your strategy for protecting that data against sophisticated nation states?

  • Would you take money from the Trump Organization or its affiliates in exchange for an explicit or implicit guarantee of "protection"?

Venture Capitalists / CEO's

  • Donald Trump's children or their representatives may ask for a share in your fund, in exchange for favorable treatment from the federal government. Would you accept such a request? Note they may ask after they have successfully applied this approach to other companies.

  • You may be approached for an investment by a company or entity that has ties to the Russian government, or ties to the Trump Organization. This may be accompanied by a threat of harassment from the federal government, hacking, DDOS, or other. Would you accept the investment?

Slack

  • By default you store a company's entire conversation history, including DM's. Private information like this is easy to distort and take out of context. Russians hacked from the DCC and trickled emails to the press, with devastating effects. Should the default behavior for a Slack installation be to store a company's entire history?

  • What efforts are you making to educate users about the risks of storing their entire conversation history on Slack? What are the highest-value targets for hackers who'd like to compromise the Slack network?

  • What progress have you made on end-to-end encryption for Slack messages?

  • Is there a way to store the data where a compromise would not allow a hacker to access every message for every company in your system? Say you had three different datastore designs.

Uber/Lyft

  • Your companies store a massive amount of data on where users have been and where they are going. If exposed, this data could be used to embarrass people - why is this married Congressman requesting a ride from outside a gay bar, or a hotel in the middle of the day?

  • What options do users have for removing their trip history from your site?

  • What employees can access user data, and under what circumstances? What tools do you have for anonymizing data that's not viewed in aggregate?

  • Many Trump voters cited a feeling of being left behind as a reason to vote for him. Uber drivers are 1099 contractors, which means you are prohibited from providing them with training. What responsibility do corporations have to put their workers on an upwards career path?

  • Many of your 1099 contractors get health care from the government, or on government-mandated exchanges. These exchanges are being threatened by Republican governors in many states, and Republicans in Congress. What responsibility does Uber have to work for healthcare for its drivers?

  • Your legal page says "We generally require a valid request issued in accordance with applicable law before we can process private requests for information." What does "generally" mean in this context? If China passes a law that says "we can ask for everything," would Uber comply?

  • You've taken money from Saudi Arabia's public investment arm. Would you be say no to that money if the Saudi Arabian government asked for data on customers as a condition of the deal?

Stripe/Braintree

  • You collected millions of dollars in revenue from the Trump campaign in 2016. If Trump acts like an authoritarian in office, or severely restricts the rights of minorities or immigrants, will you support his campaign again in 2020?

  • Does Stripe receive requests from law enforcement? What is your policy for responding to subpoenas?

  • If Stripe processes a credit card payment, who can see the record of that transaction? Who should be able to see it, and/or remove it?

Twilio

  • Do you encrypt messages passing from datacenter to datacenter?

Facebook

  • Historically newspapers and other media organizations have had a strong understanding of their role in promoting democracy and enforcing accountability from the government and our business leaders. Facebook has become a very important part of how people figure out what's going on in the world around them. What responsibility does Facebook have to ensure people have a mostly-correct view of the world? Should Facebook have a role in promoting democracy and in rejecting authoritarianism?

  • Facebook tells advertisers that their ads can change users' minds. But Facebook also insists that the algorithms it uses to show information didn't sway the US election (or overseas elections). Which is it?

  • Has Facebook responded to queries from governments on the lines of "Muslims/blacks/immigrants living in state/city/county X"?

  • Facebook's current policy is to censor/restrict content according to local laws. If a law was passed to restrict speech in the United States, would Facebook comply?

  • Does Facebook encrypt data being sent from datacenter to datacenter?

Twitter

  • What line would Donald Trump have to cross for you to suspend or ban his account?

In sum

You are the most likely agent of change at your company. A lot of stuff may happen in the next four years and it's good to think and declare now, when things are relatively sane, what you'll agree to do or not do, because in the aftermath of another 9/11, or similar event, you may be asked to do a lot.

I've laid out my own consulting ethics guide here.

Liked what you read? I am available for hire.

Election Guide (Part 2) – CA Ballot Propositions, State Senate, more

This is Part 2 of my voter guide. Part 1 covers the 24 San Francisco ballot propositions and city supervisor races.

The deadline to register to vote in California is October 24. I highly recommend you sign up. Click here to register to vote.

A few notes I cover in more detail in Part 1: More housing is the most important issue for me on this year's ballot, and by default I vote "no" on ballot propositions, since I think we shouldn't be deciding policy by statewide or citywide ballot.

California State Initiatives

Prop 51 (School Bonds): Yes

The real story here is that Proposition 13, passed decades ago, limits the state's ability to collect property taxes, enriching a generation of homeowners at everyone else's expense. This is why our schools constantly need more money.

I also wish the Legislature should be able to figure out its budget and prioritize and we didn't have to vote on things like this. I don't feel too strongly in either direction.

Prop 52 (Medi-cal): No

Hospitals pay a required fee to the CA State government (about $5 billion a year). When the State allocates this money for Medi-cal, the federal government provides about $4 billion in matching funds.

In the past the State has diverted some of the hospital fee money to the general fund which hurts 2x - not only does Medical miss out on the fee money, it misses out on the federal matching funds.

This measure would require the hospital fee money to be spent on Medical, which seems reasonable.

I'm upset that we have to vote on this; I would rather the legislature do the right thing. I'm also upset that this amends the state Constitution; I don't think the Constitution should get into the specifics of how things should be funded. I also think we should be trying to loosen the hands of our legislators, not restrict them further, and that they're as aware of the cost of giving up matching funds as voters are.

Prop 53 (Voter Approval for Megaprojects): No

I'm really torn on this. On the one hand, you are putting voters in charge of deciding even more things about what the government does. On the other, megaprojects frequently fail and the majority come in at least 50% over budget (high speed rail is only the most prominent example of this). Politicians also like to build big things so they can have a "legacy" and the history of big things lately has been really mixed - see high speed rail and also the Bay Bridge which has required frequent fixes and may be cracking.

Our politicians might not make great decisions with our money but I think voters would make worse decisions. Note voters approved the first $9 billion of a high speed rail project whose final cost may be upwards of $60 billion, when no real funding source for the other $51 billion was in sight. This would also increase uncertainty and delay the start of any project until the next statewide vote.

Prop 54 (72 Hr Bill Freeze): Yes

I am unhappy that this bill amends the Constitution. But apparently there are numerous instances of state legislators shoehorning special-interest-friendly language at the last second.

There was that budget measure that limited the amount of reserves local school districts could maintain as a cushion against lean times (a gift to the teachers union, which wanted to make those dollars available for immediate spending); the 2009 waiver of environmental rules for a downtown Los Angeles football stadium (on the argument that time was of the essence to secure an NFL team ... the project never broke ground); or the 2011 bill that Democrats rushed through to force all voter initiatives on the November ballot, thus breaking a deal with Republicans to put spending reform on the June 2012 ballot.

Prop 55 (Extending Income Taxes on High Earners): No

The share of tax each resident pays is something that the Legislature should resolve. I also agree with the Chronicle that this measure will increase the variability of revenue in the state budget, which isn't great.

Prop 56 ($2 Cigarette Tax): Yes

In general taxes are a good way to discourage behavior you don't want. Cigarettes are unhealthy and incur significant spillover costs due to secondhand smoke, and the additional burden on the healthcare system from insuring/treating patients with cancer and emphysema.

I would have preferred for the Legislature to vote for this tax as well.

Prop 57 (Parole): No

Many people are serving sentences that are too long and the prisons are overcrowded. But the language is confusing and I don't see why the Legislature can't pass legislation to deal with this issue.

Prop 58 (Local Language Education Flexibility): Yes

Apparently this is on the ballot because it repeals a previous voter-passed initiative from 1998. The worry is that voting Yes will allow students to graduate without mastering English at all, which isn't good. But it seems like all of parents, students and schools want students to learn English, they just don't agree that "all English classes, all the time" is the best way to do it.

Prop 59 (Citizens United): No

I'm voting No because this is a waste of energy and we shouldn't be voting on things like this, not based on any opinion about Citizens United.

Prop 60 (Porn Stars Wear Condoms): No

The practical effect of this bill would be to shift the porn industry in California to Nevada or another nearby state. The porn industry also requires performers to get tested every two weeks. There are problems that probably deserve more scrutiny - the exploitation of performers in some scenarios - but it's not clear that this initiative is the vehicle or the method to fix them.

Prop 61 (Drug Prices): No

I agree with the Chronicle that the right solution here is to make drug prices (and the rates each agency pays) public, instead of ensuring that the prices Medi-cal and the VA pay are the same. I also think there are legitimate concerns about reduced access to necessary drugs and the ability of the Legislature to override this initiative if there are unforeseen problems.

Prop 62 (Repeal Death Penalty): Yes

Prop 66 (Quicken Death Penalty): No

Leaving aside whether it is ethical to put someone to death for crimes they have committed, I am against the death penalty for the following, more practical reasons:

  • It's entirely possible we have put an innocent person to death, an monstrous miscarriage of justice that should never be allowed to happen.

  • It's argued that the death penalty deters people from violent crimes. But there's a lot of evidence that deterrence depends much more on the severity and the certainty of punishment. Death, if it comes at all for death row inmates, is applied years or decades after the fact.

  • There are legitimate concerns about whether execution can be done "humanely" and a number of states have had problems sourcing the drugs used to put people to death.

  • It's expensive to execute someone, both in pure cost and in the cost of the appeals process - a death sentence must be appealed to the Supreme Court.

Repeal would also save California a significant amount of money.

Prop 63 (Ammunition): No

The biggest effect of a Yes vote would add additional charges for people who would like to buy ammunition. I don't think we need to vote on this.

Prop 64 (Marijuana Legalization): No Position

In general I'd prefer for drugs to be legalized and heavily regulated + taxed, instead of illegal, especially when you consider the potential revenue. I also think criminal sentences for possessing or distributing marijuana should be smaller than they are (the initiative provides for this). However, I'm concerned that marijuana is only as expensive as it is because it is illegal. Marijuana is not an expensive crop, and if it becomes legal to grow the price per ounce could go really low. I'm worried the flat taxes per ounce are too low, and the 15% sales tax should be a flat tax or a guaranteed minimum price per ounce.

The results on public health so far are mixed; one study reports a 7% increase in traffic fatalities for every 1% increase in marijuana consumption. The penalties for drunk drivers are not currently high enough and I'm worried we don't know how to measure whether a driver is high.

On top of this I am worried that the Legislature won't have the flexibility to override a state initiative; any amendments require a 2/3 vote.

Prop 65 (Money from Paper Bags to Environment): No

This directs revenue from grocery bag fees to specific environmental causes. I don't think we should put additional constraints on where the Legislature should direct money, and I don't think we should pass things by state initiative.

Prop 66: No (see #62 above)

Prop 67 (Affirm Plastic Bag Ban): Yes

Proposition 67 is a referendum on the existing bag law (10 cents a bag); a "Yes" vote says "Yes, please keep the law the way it is." I prefer the Legislature to write laws, not California voters, so I am voting Yes.

Superior Court

Victor Hwang, who has experience working as a public defender.

Board of Education

Stevon Cook, Matt Haney, Rachel Norton, Jill Wynns.

Community College Board

Rafael Mandelman, Amy Bacharach, Alex Randolph, Shanell Williams.

Bart Director

Gwyneth Borden, who has been endorsed by the Chronicle and is open to a ban on BART strikes.

California State Senate: Scott Wiener

This is one of the most important races on the ballot due to the difference in quality between the candidates. Wiener is running against Jane Kim, who has opposed numerous housing projects, and is sponsoring some of the poorer propositions on the city ballot. Scott Wiener understands how to build more housing in San Francisco.

Kim also recently sponsored "legacy status" for Luxor Cab Company, which gives them a permanent subsidy from the City of San Francisco. This is a terrific waste of money compounded by the fact the benefit won't do anything for the company's cab drivers, only its 20 or so full time employees. Vote for Scott Wiener.

California State Representative: David Chiu

Chiu is running against Matthew Del Carlo, who does not appear to have policy positions listed anywhere publicly; it's not clear what he would run for, or do in office.

Chiu slammed Governor Brown for including $0 in affordable housing in this year's budget. The housing measures were tied to the Governor's "by right" housing legislation, which would have done more to lower rent/housing prices in San Francisco than any other legislative measure in a decade. It's not clear whether Chiu supported or opposed this measure.

Chiu is running against Matthew Del Carlo, who doesn't have any information about his policy positions listed publicly. I reached out to him multiple times asking him to post these publicly and he's refused to do so.

United States Senate: Kamala Harris

Harris is running for Barbara Boxer's old seat. We really need a California Senator who understands the technology industry and is willing to fight for it; who understands you can't just make a "golden key" to read messages that only the US government can access, as in Dianne Feinstein's horrible encryption bill.

President

Hillary Clinton.

Liked what you read? I am available for hire.

San Francisco Voting Guide – Propositions and Supervisors

I think this is useful and the ballot's complicated so I wanted to share how I'm voting this year. I used several sources to compile this guide:

  • The SF Chronicle's endorsements - they follow these issues every day.

  • The ballot book mailed to every voter, especially the text of the law and the main pro/con arguments.

I highly recommend voting by mail. You can feel too rushed or disorganized in the ballot booth, especially in this election, when there are so many things to vote on.

San Francisco Ballot Initiatives

The #1 issue for me in this election is housing. People make a fundamental mistake when analyzing the SF housing market; they see lots of increased demand (maybe 10%) and little increased supply (maybe 1%), and conclude "We're building housing but prices are still rising; the new housing must be causing the price increases." In reality if demand is outpacing supply you'd expect to see prices rise and supply rise, and the new housing stock is preventing the price from rising even faster than it currently is!

I also see a lot of hypocrisy. SF is full of liberals, and social mobility is a traditional liberal plank. In one of the hottest economies in the country, high rents are preventing poor people from moving here and establishing a foothold. Lowering the price of housing in our fastest growing economies is a moral imperative.

San Francisco added 5000 new units this year, and SF condos are 8% cheaper this year than last year. The market rate of rent also slowed from its normal double-digit increase. We need to build on this progress.

I want there to be more housing in San Francisco, of all shapes and sizes. In this election, anything that makes housing more complicated to build is a No; anything that makes housing easier is a Yes. Affordable housing is admirable but isn't a full answer, and gets more expensive as market rent rises. The easiest path to more affordable housing is to lower market rent.

I'll say two other things; in general I am opposed to deciding things by ballot initiative that could be resolved by the Board of Supervisors or the State Senate, since election votes tend to tie the hands of our elected officials, and can require supermajorities to unwind. So all other things being equal I am more likely to vote No on any given ballot measure.

I am also generally opposed to measures that set aside percentages of the budget, or specific dollar amounts, for any cause, no matter how noble. They reduce the flexibility of our elected officials to balance a budget, which is why we elect them in the first place. The percentage of the city budget each interest group would like to reserve for itself would well exceed 100%.

Measure A (School Bond): Yes

Measure B ($99 Parcel Tax for City College): Yes

Measure C (Repurpose Earthquake Bonds for Housing): Yes

The City is sitting on $261 million in unspent earthquake safety bonds and would like to redirect it to housing. This will increase the supply of housing.

Measure D (Short Term Appointment Rules): No

Some replacement public officials are named by the mayor to replace someone else who left their term. This measure would prevent them from running for a full term. I see no reason why appointees should not be allowed to run for a full term. The SF Chronicle opposes this measure.

Measure E (Trees Fund): No

$19 million per year for trees. In the words of the SF Chronicle, "San Francisco is running a near $10 billion budget. The civic bill for tree care is pegged at $20 million. There should be room for this expense without carving out a program that can't be changed."

Measure F (Youth Vote): No

This would let 16 and 17 year olds vote in local elections.

Measure G (Police Oversight): Yes

This would grant additional powers to a citizen review board. I think police organizations have trouble regulating themselves and this is a good step in the right direction.

Measure H (Public Advocate): No

This creates a new elected position with no power to do anything. "It's posturing minus responsibility, a dream job in the political world," according to the Chronicle.

Measure I (Senior Citizen Fund): No

This measure would set aside $38 million a year for programs for senior citizens and adults with disabilities. I support programs for senior citizens, but would rather our elected officials make decisions about the budget, instead of voters.

Measure J (Homeless Housing and Services): No Position

Measure K (Sales Tax Increase): Yes

In general I'd like to see more parcel tax increases and fewer sales tax increases, since the former hit property owners, who have been granted great gifts by Prop 13. They are also politically unpalatable.

Measure L (Muni Board): No

This would let the Board of Supervisors appoint three of the seven members of the Muni Agency. I don't see why the mayor shouldn't appoint members of the Muni Agency.

Measure M (New Housing Committee): No

This would add another layer of approval in the housing approval process, which would make it more difficult to add housing. I am against measures that would make it more difficult to add housing.

Measure N (Noncitizen Resident Voting): No

I'm sympathetic but this would likely be subject to a legal challenge.

Measure O (Office Exemptions): Yes

The city limits new office construction to 950,000 square feet. This is a silly rule, which makes it hard for startups, among others, to rent in San Francisco. This would exempt Candlestick Point development from that square footage rule.

I would like to see similar rules applied to speed housing growth, but there you go.

Measure P (Competitive Bidding for Affordable Housing): No

This makes it more difficult to build housing by discouraging projects that can't get at least three bids. From the Chronicle:

Prop. P obliges the city to seek three bids when offering city land to affordable housing builders. But City Hall already beats the bushes for multiple contenders. By one count, the last 10 projects had at least two bidders. Locking in a three-bid minimum could kill projects which don’t attract that threshold number of entrants. The measure has the potential to stop promising deals, the last thing San Francisco needs.

Measure Q (Prohibit Tent Placement): No

This wouldn't have much practical effect, and won't really help much to address the shortage of beds.

Measure R (Neighborhood Crime Unit): No

This would allocate 3% of the police force for neighborhood crime. Even if this is an issue that could be addressed by this allocation, I don't think the right answer is for the voters to make allocation decisions for the police department.

Measure S (Hotel Money Allocation): No

This would allocate the 8% hotel tax for the arts and for the homeless. In general I'm against allocating revenue for specific purposes; this isn't an exception. I doubt this will matter; the Chronicle has no position and there are no arguments against the measure in the ballot book.

Measure T (Lobbying Rules): Yes

This would add tighter restrictions on what lobbyists are allowed to do and spend to influence votes.

Measure U (Median Income): No

This would help middle income families qualify for affordable housing at the expense of lower income families. Per [the Chronicle][measure-u], "The guidelines for competitive bidding and income qualifications are better left to a process of legislative hearings, study and political compromise that balances the competing goals and concerns. These are not issues to be settled at the ballot box."

The solution here is more housing of all stripes, and hopefully market rate housing that is affordable to middle income families. This wouldn't help.

Measure V (Soda Tax): Yes

Charging a higher price for something is a good way to discourage people from getting it. This strategy has been used very successfully with cigarettes, which cause cancer in others via secondhand smoke; raising the price of cigarettes makes it an expensive habit. The fact that this raises money for the City is an ancillary benefit. The goal of this bill is to make sugary drinks more expensive and non-sugary drinks cheaper by comparison.

I'm also dismayed by the efforts of bill opponents, who have sent volumes of mail and mislead when they call this a "grocery tax." It's a 1 cent per ounce tax on sugary drinks.

Measure W (Higher City Transfer Taxes): No Opinion

The arguments for this measure all seem to say "this will help make City College free", which is very odd since it seems the tax money will go into the General Fund.

The arguments against point out that this also applies to rent controlled buildings and large buildings.

Measure X (Arts Use in New Buildings): No

This would add restrictions if you want to build housing in an area that was formerly used for the arts or certain types of small businesses. We shouldn't be voting on this, and it makes it more difficult to build housing, maybe more so than any other measure on the ballot.

San Francisco Board of Supervisors

District 1: Marjan Philhour

Marjan wants to build more housing of all shapes and sizes to address the area's housing crisis. She's also been endorsed by the Chronicle.

District 3: No Recommendation

Aaron Peskin is the incumbent who is going to win going away. Peskin has held up new housing on several occasions. He's also supported symbolic efforts to oppose Governor Brown's by right legislation, which would have done more for housing growth than any other proposal in a long time. Peskin also believes that you should only be allowed to exceed existing density limits if you build 100% affordable housing, which is a great way to grandstand for affordable housing while ensuring no new housing gets built.

He is being opposed by Tim Donnelly, who supports "respecting building limits", increasing parking, expanding rent control, and "giving residents a voice" because changes have been made "despite overwhelming opposition from the local community." It does not sound like Mr Donnelly is in favor of more housing.

District 5: No Recommendation

London Breed voted against Governor Brown's by right legislation, which would have helped increase the market-rate and affordable housing stock in San Francisco by letting developers build any project that followed local zoning rules and had 20% affordable housing. She also supports affordability requirements that make it difficult to build more housing.

She is being opposed by Dean Preston, who is running against "rent gouging", and supports an "anti-demolition" ordinance for "historic" buildings. Mr. Preston would not make it easier to build more housing in San Francisco.

District 7: Joel Engardio

Engardio is running against Norman Yee, who supports CEQA, a law that is frequently abused to oppose housing. Engardio supports building more housing. "I also know that building more housing will help middle income residents become homeowners -- and we want to keep families from leaving San Francisco. Restricting supply only drives prices higher," he writes.

District 9: No Endorsement

Hillary Ronen pledges to "fight for an affordable San Francisco" and wants to build 5000 units of affordable housing in 10 years. There was a very easy way to have accomplished 5000 units of affordable housing - support Governor Brown's by right housing legislation, which would have guaranteed that 20% of every new building in San Francisco would have been affordable. Her boss, David Campos, voted against it. She also wants to leverage state and federal funds to build affordable housing. Her boss's vote against by right legislation helped remove $400 million for affordable housing from the state budget.

District 11: No Endorsement

None of the candidates in either of these districts seem to agree that building more housing of any shape and size is the best way to alleviate our affordability crisis for everyone. Notably bad is District 11's Kim Alvarenga, running on a platform of "more parking" and "100% affordable housing", which is very difficult to build.

Coming Soon!

California State Propositions, BART director, judicial elections, State Senate and US Senate.

Liked what you read? I am available for hire.

More Comment-Preserving Configuration Parsers

For the past few weeks I've been on the hunt for a configuration file format with the following three properties:

  1. You can use a library to parse the configuration. Most configuration formats allow for this, though some (nginx, haproxy, vim) aren't so easy.

  2. You can manipulate the keys and values, using the same library.

  3. When that library writes the file to disk, any comments that were present in the original config file are preserved.

Why bother? First, allowing programs to read/write configuration files allows for automated cleanup/manipulation. Go ships with a first-class parser/AST, and as a consequence there are many programs that can lint/edit/vet your source code. These wouldn't be possible without that ast package and a set of related tools that make parsing and manipulating the source easy.

You can imagine installers that could automatically make a change to your configuration; for example, certbot from the Let's Encrypt project tries to automatically edit your Apache or Nginx configuration. This is an incredibly difficult task, due to the complexity of the configuration that have piled up over the years, and that those configuration files weren't built with automatic editing in mind.

Backwards incompatible changes are never good, but their downsides can be mitigated by effective tools for parsing and updating configuration.

You want comments in your configuration file because configurations tend to accumulate over the years and it can be incredibly unclear where values came from, or why values were set the way they were. At Twilio, the same HAProxy config got copied from service to service to service, even though the defined timeouts led to bad behavior. Comments allow you to provide more information about why a value is set the way it is, and note values where you weren't sure what they should be, but had to pick something besides "infinity" before deploying.

What problems do you run into when you try to implement a comment-preserving configuration parser? A lot of config parsers try to turn the file into a simple data type like a dictionary or an array, which immediately loses a lot of the fidelity that was present in the original file. The second problem there is that dictionaries in most languages do not preserve ordering so you might write out the configuration in a different order than you read it, which messes up git diffs, and the comment order.

You are going to need to implement something that is a lot closer to an abstract syntax tree than a dictionary; at the very least maps of keys and values should be stored as an array of tuples and not a dictionary type.

The next problem you run into is that syntax trees are great for preserving the fidelity of source code but tend to be unwieldy when all you want to do is index into an array, or get the value for a key, especially when the type of that value may take any number of values - a number, a string, a date, or an array of the above. The good news is configuration files tend to only need a subset of the syntax/fidelity necessary for a programming language (you don't need/want functions, for example) so you can hopefully get away with defining a simpler set of interfaces for manipulating data.

(Incidentally I realized in the course of researching this that I have written two libraries to do this - one is a library for manipulating your /etc/hosts file, and the other is a library for bumping versions in Go source code. Of course those are simpler problems than the one I am trying to solve here).

So let's look at what's out there.

  • JSON is very popular, but it's a non-starter because there's no support for comments, and JSON does not define an ordering for keys and values in a dictionary; they could get written in a different order than they are read. JSON5 is a variant of JSON that allows for code comments. Unfortunately I haven't seen a JSON5 parser that maintains comments in the representation.

  • YAML is another configuration format used by Ansible, Salt, Travis CI, CircleCI and others. As far as I can tell there is exactly one YAML parser that preserves comments, written in Python.

  • XML is not the most popular format for configuration, but the structure makes it pretty easy to preserve comments. For example, the Go standard library parser contains tools for reading and writing comments. XML seems to have the widest set of libraries that preserve comments - I also found libraries in Python and Java and could probably find more if I looked harder.

  • TOML is a newer format that resembles YAML but has a looser grammar. There are no known parsers for TOML that preserve comments.

  • INI files are used by windows programs, and the Python configparser module, among others. I have found one parser in Perl that tries to preserve comments.

  • Avro is another configuration tool that is gaining in popularity for things like database schema definitions. Unfortunately it's backed by JSON, so it's out for the same reasons JSON is out.

  • You can use Go source code for your configuration. Unfortunately the tools for working with Go syntax trees are still pretty forbidding, for tasks beyond extremely simple ones, especially if you want to go past the token representation of a file into actually working with e.g. a struct or an array.

I decided on [a configuration file format called hcl], from Hashicorp. It resembles nginx configuration syntax, but ships with a Go parser and printer. It's still a little rough around the edges to get values out of it, so I wrote a small library for getting and setting keys in a configuration map.

This is difficult - it's much easier to write a parser that just converts to an array or a dictionary, than one that preserves the structure of the underlying file. But I think we've only scratched the surface of the benefits, with tools like Go's auto code rewriter and npm init/npm version patch. Hopefully going forward, new configuration formats will ship with a proper parser from day one.

Liked what you read? I am available for hire.

Cleaning up Parallel Tests in Go 1.7

I have a lot of tests in Go that integrate with Postgres, and test the interactions between Go models and the database.

A lot of these tests can run in parallel. For example, any test that attempts to write a record, but fails with a constraint failure, can run in parallel with all other tests. A test that tries to read a random database ID and expects to not fetch a record can run in parallel with other tests. If you write your tests so they all use random UUID's, or all run inside of transactions, you can run them in parallel. You can use this technique to keep your test suite pretty fast, even if each individual test takes 20-40 milliseconds.

You can mark a test to run in parallel by calling t.Parallel() at the top of the test. Here's an example test from the job queue Rickover:

func TestCreateMissingFields(t *testing.T) {
  t.Parallel()
  test.SetUp(t)
  job := models.Job{
    Name: "email-signup",
  }
  _, err := jobs.Create(job)
  test.AssertError(t, err, "")
  test.AssertEquals(t, err.Error(), "Invalid delivery_strategy: \"\"")
}

This test will run in parallel with other tests marked Parallel and only with other tests marked Parallel; all other tests run sequentially.

The problem comes when you want to clear the database. If you have a t.Parallel() test clean up after it has made its assertions, it might try to clear the database while another Parallel() test is still running! That wouldn't be good at all. Presumably, the sequential tests are expecting the database to be cleared. (They could clear it at the start of the test, but this might lead to unnecessary extra DB writes; it's better for tests that alter the database to clean up after themselves).

(You can also run every test in a transaction, and roll it back at the end. Which is great, and gives you automatic isolation! But you have to pass a *sql.Tx around everywhere, and make two additional roundtrips to the database, which you probably also need to do in your application).

Go 1.7 adds the ability to nest tests. Which means we can run setup once, run every parallel test, then tear down once. Something like this (from the docs):

func TestTeardownParallel(t *testing.T) {
  // This Run will not return until the parallel tests finish.
  t.Run("group", func(t *testing.T) {
    t.Run("Test1", parallelTest1)
    t.Run("Test2", parallelTest2)
    t.Run("Test3", parallelTest3)
  })
  // <tear-down code>
}

Note you have to lowercase the function names for the parallel tests, or they'll run inside of the test block, and then again, individually. I settled on this pattern:

var parallelTests = []func(*testing.T){
  testCreate,
  testCreateEmptyPriority,
  testUniqueFailure,
  testGet,
}
func TestAll(t *testing.T) {
  test.SetUp(t)
  defer test.TearDown(t)
  t.Run("Parallel", func(t *testing.T) {
    for _, parallelTest := range parallelTests {
      test.F(t, parallelTest)
    }
  })
}

The test mentioned there is the set of test helpers from the Let's Encrypt project, plus some of my own. test.F finds the defined function name, capitalizes it, and passes the result to test.Run:

// capitalize the first letter in the string
func capitalize(s string) string {
  r, size := utf8.DecodeRuneInString(s)
  return fmt.Sprintf("%c", unicode.ToTitle(r)) + s[size:]
}
func F(t *testing.T, f func(*testing.T)) {
  longfuncname := runtime.FuncForPC(reflect.ValueOf(f).Pointer()).Name()
  funcnameparts := strings.Split(longfuncname, ".")
  funcname := funcnameparts[len(funcnameparts)-1]
  t.Run(capitalize(funcname), f)
}

The result is a set of parallel tests that run a cleanup action exactly once. The downside is the resulting tests have two levels of nesting; you have to define a second t.Run that waits for the parallel tests to complete.

=== RUN   TestAll
=== RUN   TestAll/Parallel
=== RUN   TestAll/Parallel/TestCreate
=== RUN   TestAll/Parallel/TestCreateEmptyPriority
=== RUN   TestAll/Parallel/TestUniqueFailure
=== RUN   TestAll/Parallel/TestGet
--- PASS: TestAll (0.03s)
    --- PASS: TestAll/Parallel (0.00s)
        --- PASS: TestAll/Parallel/TestCreate (0.01s)
        --- PASS: TestAll/Parallel/TestCreateEmptyPriority (0.01s)
        --- PASS: TestAll/Parallel/TestUniqueFailure (0.01s)
        --- PASS: TestAll/Parallel/TestGet (0.02s)

The other thing that might trip you up: If you add print statements to your tear down lines, they'll appear in the console output before the PASS lines. However, I verified they run after all of your parallel tests are finished running.

Liked what you read? I am available for hire.

Six Years of Hacker News Comments about Twilio

(I'm omitting the many, many, astroturf posts - "Why X is Better than Twilio", "Why I'm Ditching Twilio for X" - and comments from employees at competitors between 2010 and 2014.)

Twilio Raises $12m Series B

"Can something like Twilio really become a $100m+ company? I hope so but my ignorance blinkers me to how this could happen.."

Twilio Launches UK SMS

"I'm not sure what the big deal is. (Competitor) has much better international coverage and costs less."

Twilio Launches in Europe

"Great when it will have SMS support."

Twilio Raises $17m Series C

"When are they going to get SMS enabled numbers in Canada?"

"Their move to the UK was very half-assed, still no SMS support for the UK... :( I hope they fix that with this new funding before they expand elsewhere."

"It's a good service, but unless they bring prices way down, some big provider (Cough Amazon..) is going to come in and eat their lunch. Granted they probably wouldn't offer the level of detailed APIs that Twillio does. We started using them for SMS sending but went with (Competitor) at a fraction of the cost."

"I believe their exit strategy is to be acquired by Amazon or someone of their scale... anyone can do it once they learn how to handle the SMS messages between the web and the carrier gateways, as well as how to handle call flows with Asterisk or Freeswitch." - (ed "Anyone can do it once [the entire value proposition])

Twilio Launches MMS

"Well, all the kudos should go to Bandwidth.com the primary carrier of of Twilio, (Competitor), (free, irrelevant service), etc ..." - (ed. This is completely incorrect)

"MMS died years ago. Terrible technically and never actually worked the four times in someone tried to send one."

Twilio Launches WebRTC Support

"I came to the conclusion that on mobile phones, WebRTC video is not yet usable."

"Out of all the WebRTC products I've tried, (competitor) was the easiest one for me to use."

Twilio Launches International SMS

"Love how Hacker News will post anything Twilio like their pricing, but competitors posts are instantly removed. More proof that HN is a silicon valley whore"

Twilio Launches SIP Support

"( Competitor ) has been supporting SIP for a very long time."

Twilio Acquires Authy

"Authy is one of the worst-designed iOS applications I have ever used."

"Authy was only a marginal improvement in technology."

Twilio Files S-1

"This IPO is an exit for their VCs. They were all the way up to a series E round, and since they grew fast by losing money, the early investors had to pour in a lot of cash."

"I can only imagine that as more and more developers flock to "free" SMS verification services provided by companies like Facebook (Account Kit) and Twitter (Digits), their long term outlook is even more unsure."

"Looking at their escalating losses, I have to wonder if this IPO is a desperation play after failing to raise private money at an acceptable valuation in the current climate."

"Not even cash flow positive. Stay away."

This Post Gets Submitted to HN

"What is the point of this post?"

"It seems like a running [joke] that their primary (ed. struggling) competitor is ahead on features, and has better prices. I think those are completely fair criticisms of any company. It's a little strange that this compilation goes to great lengths to never mention the competitor by name."

Twilio Sells 10m Shares on Open Market, Stock Rises 90% on Opening Day, Hope Restored for Other Tech Company IPO's

(In this thread someone posted links to a spreadsheet with 500 other SMS API's. Oh. Shut down the company)

"Good service but still very overvalued."

"For a company with a yearly gross profit of USD92 millions and net losses of USD35 millions, I cannot imagine who is buying shares at a USD1.8 billion market cap."

"I'm not sure how defensible this is against AWS or another infrastructure provider cross-selling to existing customers."

"I used to work for a company that vaguely competed with them. It was frustrating to have people ask "why does your service cost so much compared to Twilio" and to bite my tongue and say something diplomatic rather than "because we're running an actual business that makes money rather than a VC-funded playpen". Apparently they're not even cash-flow positive at this point?"

Keep doubting! And don't forget to crap on any new thing that gets launched.

Liked what you read? I am available for hire.

The TSA Randomizer iPad App Cost $1.4 Million

You may have seen the TSA Randomizer on your last flight. A TSA agent holds an iPad. The agent taps the iPad, a large arrow points right or left, and you follow it into a given lane.

How much does the TSA pay for an app that a beginner could build in a day? It turns out the TSA paid IBM $1.4 million dollars for it.

It's not hard! I searched on Google for "TSA FOIA" and found this page, which describes exactly how to reach the FOIA team at the TSA. Ignore the part about how they will get back to you in 20 days. I sent them this email:

I request that a copy of documents concerning the following subject matter be provided to me:

The RFP (Request for Proposal) issued by the TSA for designing and implementing the "TSA Randomizer" iPad application, described here and currently in use by TSA agents at security checkpoints at many airports, including Terminal 2 at Oakland International Airport.

Details of any submitted bids from contractors or internal government agencies to design and construct the "TSA Randomizer" iPad app.

The final signed contract between the TSA and a contractor to implement the "TSA Randomizer" iPad app.

I also included a little bit about how I thought this request should be eligible for a fee waiver, and how I wasn't going to profit from knowing this information, but it didn't seem particularly difficult to get the data, so I'm not sure it mattered.

They just got back to me! They sent me two documents. The first is a disclaimer about how they had to black out some of the information. The second is the contract between the TSA and IBM. And there's the payment:

Later today Pratheek Rebala reached out to mention that this data is available publicly, and there were 8 other payments as part of the same award, totaling $1.4 million; the document I have is one part, totaling $336,000. Furthermore, there were 4 bids for the contract and IBM won the bidding.

Unfortunately we don't know everything the TSA got for that $1.4 million. They might have just gotten the iPad app; they might have gotten iPads, or work on multiple different apps, including the TSA Randomizer. We only know it's associated with the TSA Randomizer based on the FOIA request that returned this document.

TSA randomizer payment

I should mention that the Obama Administration, the "most transparent", has set numerous records for delays in turning over files and refusing to fulfill requests for access, and none of the candidates seem likely to reverse that trend. If you think this is important, consider writing your elected officials and asking them to prioritize this, or making decisions in November based on this.

Liked what you read? I am available for hire.

YOUR ASSISTANCE IS NEEDED: STOCK AWARD NOTICE (Ref: LSUK/2031/8161/05)

DEAR SIR,

I HAVE THE BELIEVE YOU ARE A REPUTABLE AND RESPONSIBLE AND TRUSTWORTHY PERSON I CAN DO BUSINESS WITH FROM THE LITTLE INFORMATION SO FAR I GATHERED ABOUT YOU DURING MY SEARCH FOR A PARTNER AND BY MATTER OF TRUST I MUST NOT HESITATE TO CONFIDE IN YOU FOR THIS SIMPLE AND SINCERE BUSINESS.

LET ME START BY INTRODUCING MYSELF PROPERLY , I AM MR. IRVING TRUBE, LOAN OFFICER WITH SILICON VALLEY BANK PLC SAN FRANCISCO BRANCH, I CAME TO KNOW OF YOU IN MY PRIVATE SEARCH FOR A RELIABLE AND REPUTABLE PERSON TO HANDLE THIS CONFIDENTIAL TRANSACTION,WHICH INVOLVES TRANSFERING HUGE SUM OF MONEY TO A FOREIGN ACCOUNT REQUIRING MAXIMUM CONFIDENCE.

THE PREPOSITION:

A FOREIGNER AND AN AMERICAN , MR KEVIN BURKE (SWE) AN ENGINEER WITH A UNICORN STARTUP IN SAN FRANCISCO, UNTIL HIS DEPARTURE WEEKS AGO DUE TO BOREDOM AND A TWENTY FIVE CENT SURCHARGE ON DRINKS FROM THE SODA MACHINE BANKED WITH US AT SILICON VALLEY BANK OF SAN FRANCISCO PLC. A YOUNG WHITE MAN OF MEDIOCRE ABILITY, ENGR BURKE HIRED DUE TO HIS APPARENT WILLINGNESS TO WORK LONG HOURS FOR BEER, JUNK FOOD AND A PINBALL TABLE, WAS IMPRUDENTLY GRANTED A LARGE NUMBER OF SHARES OF COMMON STOCK. THROUGH NO FAULT OF HIS OWN, THE END OF DECEMBER 2015 THE SHARES WERE WORTH $8,432,000USD.

THE MANAGEMENT OF UNICORN STARTUP PLC UNDER THE INFLUENCE OF THEIR CHAIRMAN AND MEMBERS OF THE BOARD OF DIRECTORS ARRANGEMENTS HAVE BEEN MADE FOR THE STOCK TO BE DECLARED "UNCLAIMED" AFTER 90 DAYS AND SUBSEQUENTLY DONATE THE FUNDS TO THE VENTURE CAPITALISTS, AND THIS WILL FUTHER ENHANCE THE CAUSE OF RICH WHITE MEN IN SILICON VALLEY AND SCHOOL TEACHER PENSION FUNDS IN GENERAL . THE COMPANY NOW EXPECTS BURKE TO EXERCISE HIS OPTIONS WITHIN 90 DAYS OR LOSE THE SHARES.

POOR ENGR BURKE EXPLAINED TO ME THAT HE HAS NO CASH ON HAND TO PURCHASE THE SHARES, ( HE PUT HIS LIFE SAVINGS IN AN ONLINE BITCOIN ACCOUNT THAT WAS FOUNDED AS A TRADING SITE FOR MAGIC THE GATHERING CARDS ).

HE SAID I SHOULD SEEK FOR A GOD FEARING PARTNER IN A BAY AREA COUNTY OF MY CHOICE WHERE I WILL TRANSFER THIS STOCK AND USE IT FOR INVESTMENT PURPOSE, (SUCH AS FURTHER INCREASING THE ALREADY RIDICULOUS PRICES OF BAY AREA REAL ESTATE). SIR, WE ARE HONOURABLY SEEKING YOUR ASSISTANCE IN THE FOLLOWING WAYS.

IN OTHER TO AVERT THIS NEGATIVE DEVELOPMENT SOME OF MY TRUSTED COLLEAGUES AND I NOW SEEK YOUR PERMISSION TO LOAN MONEY TO THE DESPONDENT MR. KEVIN BURKE SO THAT THE STOCK WILL BE RELEASED AND A CERTIFICATE WILL BE ISSUED AS THE STARTUP'S THIRD EMPLOYEE . WE HAVE BEEN MANDATED BY THE BANK TO OFFICIALY PROVIDE THE FUNDS WITHIN THE SHORTEST POSSIBLE TIME (90 DAYS), THAT IS WHY WE HAVE DECIDED TO CONTACT YOU AND MORE SO WE ARE ASSURING YOU THAT YOU THAT THE BUSINESS IS 100% RISK FREE INVOLVEMENT .

MOREOVER SIR, WE ARE WILLING TO OFFER YOU 15% OF THE SUM AS COMPENSATION FOR EFFORT INPUT AFTER THE SUCCESSFUL LOAN OF THIS MONEY TO MR. KEVIN BURKE'S ACCOUNT, UNTIL UNICORN STARTUP RIDES OUT ADVERSE MARKET CONDITIONS, UNDERGOES SEVERAL POINTLESS BRANDING EXERCISES, HIRES ITS OLD CEO BACK AND TENDERS AN INITIAL PUBLIC OFFERING. PLEASE FEEL FREE TO CONTACT ,ME VIA THIS GPG PUBLIC KEY: 8F A0 A0 2F 1A D1 C0 2F

THANK YOU IN ADVANCED FOR YOUR ANTICIPATED CO-OPERATION

BEST REGARDS

MR IRVING TRUBE

Liked what you read? I am available for hire.

Ready Player One and a Dystopian View of Future Oil Prices

I read Ready Player One recently and I enjoyed it; it was a pretty fast read and I finished it in a day. It presents a view of the future where the possibilities allowed by a virtual reality world surpass those of the real world, so most people spend as much time as possible connected to a system called OASIS.

One part of the book's prediction of the future bothered me. The author describes the price of oil skyrocketing, causing the decay of American roads, and cities with cars stacked around their exteriors, as the unaffordable price of oil made them too expensive to drive.

This bothered me and my economics degree. People hugely value the ability to move around and travel for many reasons:

  • For reasons we cannot yet explain in-classroom teaching produces better results than online teaching, meaning the teacher and students need to travel to the same place (There's a study showing this result but I don't know where it is).

  • People migrate for work, to bring their skills to an area of the world where there's more demand for them, see for example the millions of Filipinos who work overseas. People will still need food delivered and haircuts and their plumbing fixed and their cars driven even if they spend twelve hours a day with a headset strapped to their face.

  • The same is mostly true of relationships and friendship bonding; the face to face time is a costly signal that helps show the other person you care and are committed to the relationship.

  • People enjoy traveling for tourism, to see beaches, mountains, etc. Sure virtual reality could put you on a beach and it might be "good enough" but a device that can replicate the feeling of the sand beneath your toes is still a ways away.

And many, many more; sure, virtual reality may chip away at the margins here but there still will be a vast demand for people to fly around the planet.

The vast demand means that there's a huge incentive for folks to figure out cost-effective ways to get around. If gas gets too expensive, we see people figure out ways to create gas from other minerals, as the large expansion of activity in North Dakota shows. Or there will be more of an incentive to figure out how to store solar energy to use at night, or more of an incentive to use electric power to get around, and we'd see more trains and less planes.

How can we turn this into a testable prediction? One obvious answer is to look at the price of oil. The author writes that the price will be sky-high, causing people to abandon their cars. Well another group of people bet on the price of oil every day. I tend to trust them more, because if they are wrong, they will lose tons of money.

Data from the CME Group indicates that the best prediction for the price of oil in December 2022 is $86.20, lower than it is today. Now, there are some caveats that would affect this. There are reasons to believe the futures price might be lower than the true price in 2026, because a buyer today is assuming risk and the seller is shedding it. On the flip side at least one source states that oil futures with long maturities tend to be priced higher than the actual price.

So that's the best prediction of the price of oil 8 years from now - about what it is today, not drastically higher. I would encourage anyone who thinks oil will be very expensive (including the author) to bet on that outcome, and profit from all of the people who currently think it will be cheap.

In general we look to science fiction authors for a vision of what the future will look like. In some places they may be better than average at predicting what the future will look like. But clearly in others, they're not.

Liked what you read? I am available for hire.