Monday, March 20, 2017

gsutil: No module named google_compute_engine

While trying to run gsutil ls on a Compute Engine VM, I received a stack trace like this:
Traceback (most recent call last):
ImportError: No module named google_compute_engine
It turns out that gsutil doesn't like Python installed from Linuxbrew.  It really wants to use the system Python.  The following works fine:
env CLOUDSDK_PYTHON=/usr/bin/python gsutil ls ...
Easy enough to fix.  Nobody else had documented trouble with this configuration so here you go Internet.

Wednesday, March 15, 2017

AGI Script completed, returning 4

I use Asterisk to run my phone system.  Most of my dialplan is an AGI script written in Perl.  In certain circumstances, I kept finding the following message in Asterisk's logs:
AGI Script ... completed, returning 4
My script should always return 0, so seeing an exit code of 4 was unexpected.  After this log message appeared, no further code in my AGI script was executed, also unexpected.

After a bunch of debugging, I discovered that Asterisk sends SIGHUP to your script if a caller hangs up.  The exit code 4 was Asterisk's interpretation of Perl's exit code when SIGHUP wasn't handled.  I installed a handler for that signal and now everything's grand.

Prior to tonight's debugging, I had never used the sigtrap package.  Its stack traces were very helpful.

Tuesday, March 07, 2017

Serving Git repositories from Go

I have a small web server written in Go.  I wanted to serve Git repositories from this server.  It turned out to be surprisingly easy since git-http-backend and the cgi package do all the hard work.

First, define a function for handling Git's HTTP requests:

func git(w http.ResponseWriter, r *http.Request) {
        username, password, ok := r.BasicAuth()
        if !ok || username != "john" || password != "secret" {
                w.Header().Set("Content-Type", "text/plain")
                w.Header().Set("WWW-Authenticate", "Basic realm=\" git\"")
                fmt.Fprintln(w, "Not authorized")

        const gitRoot = "/path/to/git/repositories/"
        h := &cgi.Handler{
                Path: "/usr/lib/git-core/git-http-backend",
                Root: "/git",
                Dir:  gitRoot,
                Env: []string{
                        "GIT_PROJECT_ROOT=" + gitRoot,
                        "GIT_HTTP_EXPORT_ALL=", // we require auth above
                        "REMOTE_USER=" + username,
        h.ServeHTTP(w, r)

Second, connect that function to the mux you're already using:

mux.HandleFunc("/git/", git)

Now I can clone from and push to my little Go server.

Saturday, March 04, 2017

Sharing is dangerous

Last week I tried pass as an open source replacement for 1Password.  It was almost exactly what I wanted.  Unfortunately, it uses GnuPG for encryption.  GnuPG is a pain and bloated (444k lines of code).  Since pass just runs the gpg binary, I wrote a quick script that implements the gpg shell interface but does encryption with a much smaller library.  The proof of concept worked.  I made a note to migrate to pass later and uninstalled it.  I forgot to remove my fake gpg script.

This morning I tried running "brew update".  It fetched Git repositories then stalled without hints about the cause.  After too much debugging time, I discovered that it was running my fake gpg which blocked waiting for input.  I deleted fake gpg then "brew update" proceeded fine.

Too much sharing

The wasted debug time was clearly my fault, but it reminded me how dangerous sharing is.  Too much of our software is a wobbly tower of dependencies.  Sure, you can change the bottom block, but it's risky.

The well-known costs of global mutable state are a symptom of problematic sharing.  Package management becomes NP-complete dependency hell when sharing is mandatory (assumption 4). The recent Cloudbleed vulnerability was mostly a problem because of sharing:

Because Cloudflare operates a large, shared infrastructure, an HTTP request to a Cloudflare web site that was vulnerable to this problem could reveal information about an unrelated other Cloudflare site. (emphasis added)

In other words, FitBit and Uber have a security vulnerability because some random WordPress blog generated bad HTML.

Don't share, copy

Some cultures don't like to share.  Docker containers isolate applications from each other.  Node prefers to load a module that's the least shared.  OpenBSD doesn't like to share file system partitions.  Chrome sandboxes each site to prevent sharing.  This is pretty nice.  If my Youtube tab has a problem, I don't have to consider debugging the Reddit tab.  If I upgrade one project's Node dependency, it doesn't break other Node projects.

Some cultures that used to favor sharing are moving away from it.  Perl (local::lib) and Go (vendor directories) come to mind.  Of course, everyone can still share (in the sense of using identical code), they just get their own local copy of it.  Go's default of building static binaries is a breath of fresh air.

All of this is just a reminder to me: if my code shares anything, find a way to make it either local (not shared) or immutable.

Thursday, March 02, 2017

Caller ID for SMS in XMPP

When you receive an SMS on a traditional mobile device, your phone looks up the number in your local address book and displays the corresponding name.  That works great for a small social network where you can maintain the list yourself.  As the number of potential contacts (coworkers, telemarketers, etc) increases, it becomes unwieldy.

For example, I routinely receive SMS from people in my ward or stake who I'm unlikely to contact on a regular basis.  That's a few thousand people and the population changes weekly.  I don't want all that info in my regular contact list.

Fortunately, I run my own little phone system that uses XMPP for sending and receiving SMS.  I wrote a gateway to handle the conversion back and forth between those networks.  That gateway can run a script to perform caller ID however I want.  So now I have something like this when an SMS arrives:

  1. check my personal address book
  2. check the stake directory
  3. check 800notes for records of telemarketing
I also have multiple external phone numbers.  One of them is a junk number that I give out when I fill out online forms.  If an SMS comes in to that number, the caller ID appears with a "(junk)" prefix.

It's a lot of fun to be able to write a couple shell scripts to customize how your phone works.

Wednesday, March 01, 2017

Signing up for Project Fi

In preparation for an extended visit to Europe, I signed up for Project Fi today.  Prior to this I had been using Ting.  I love Ting.  I do all my SMS and voice communications over IP, so I really just need a reliable, affordable data plan.  If Ting had better data rates in Europe, I would have stuck with them.  Anyway, here are some first impressions while signing up for Project Fi.

Charging $20 per month per phone line seems expensive.  That costs $6 on Ting.  Fi gives you unlimited calls and texts but since I always use zero of those, I'd rather have a cheaper base rate.

Fi charges $10 per GB for data.  As they say, "No matter your budget, you'll only pay for data you use."  So why are they asking me to set a budget?  It seems like an extra step without a purpose.  I chose the smallest budget allowed, 1 GB.  This combination costs $30 per month on Fi and $22 on Ting.  An extra $8 per month to get excellent data rates across Europe seems reasonable to me.

I already have a Google Pixel so I only needed a new SIM.  That was free.  As I recall, Ting charges $12 up front for that, so $0 is nice.

Since my public phone numbers are all managed through Twilio, it doesn't really matter to me what phone number Fi assigns me.  They gave me the choice of porting a number or choosing a new one.  I chose to receive a new one.  Unfortunately, they only offered numbers with area codes in my local area.  Some people might like to choose an arbitrary area code.  They didn't let me choose the specific number at this point.

Fi asked for my primary service address.  Since I'll be using this phone all over Europe, I wasn't entirely sure what to put.  I used my home address in Wyoming.  Fi didn't seem picky about that.  They even let me use a different shipping address for the SIM card.  That was convenient since I was traveling at the time I placed the order.

Overall the sign up process was smooth and the prices seem reasonable.  When the SIM arrives in 7 days, I'll put it through its paces.

Tuesday, February 07, 2017

Selfhosted Google Voice

I've used Google Voice since 2010.  During that time, it's been my main telephone number.  Besides Search, it's my favorite Google product.  Google has even renewed their commitment to the product recently.  Unfortunately, Google Voice only works for US customers and my family and I are moving to Europe later this year.  GV has spoiled me and I don't want to return to the old days when I used my mobile carrier's number directly.  This is how I'm selfhosting an alternative to Google Voice.


Today I finished porting my Google Voice number to Twilio.  They offer a great platform on which to build telephony apps.  I've been very impressed with their customer support too.  It's refreshing to be able to discuss telephony problems with your provider.  That was never possible with Google Voice.

They offer phone numbers in every country we plan to live.  They provide numerous telephony services, but we're only using a few right now.


I selfhost an XMPP server for my family.  We use it all the time for messaging each other.  We run Conversations on mobile and use other XMPP clients on desktop.  It works great.  I wrote a gateway between XMPP and SMS.  You send an XMPP message and it converts it into an SMS.  When someone sends me an SMS, it converts it into an XMPP message.

This all works through Twilio's SMS API.  Since the whole system is under my control, I can add arbitrary logic for handling SMS.


To replace the voice component of Google Voice, I set up an Asterisk instance.  I use Twilio SIP trunking to connect to the telephone network.  I run Zoiper on my Android phone.  To make an outgoing call, I use the Zoiper dialer and Asterisk routes the SIP call through Twilio to the public telephone network.  When someone calls my Twilio number, Asterisk routes the call to Zoiper on my Android device.  The call quality is great.

I have no previous experience with PBX systems, but it was all surprisingly easy to configure.  Once again, since it's all in my control, I can add whatever features I want.


For now I'm using a very basic setup.  SMS accounts for 90% of my Google Voice usage and that part works really well.  Even if I never had more than this, I think I'd be content.  I have a bunch of features in mind and this setup makes it really easy to add them as I need them.