World tour: Marseilles, Italy, Athens

After leaving Spain, we spent ten days in Marseilles, France.  Since we live in apartments in most cities, we frequent nearby parks where the younger boys can burn off some steam.  Their favorite playground in Marseilles (in Parc Borely) had a small, car-themed roller coaster for kids.  I don't know which they enjoyed more: the roller coaster or riding the subway to get there.

On the first night in a new city, I usually walk around the neighborhood trying to find groceries for breakfast for the next day. In Marseilles the closest store was a Turkish grocer with plenty of dried fish heads and exotic olives. Fortunately for the children, they also carried peanut butter. That's a rare treat in Europe so I stocked up and we ate it with our baguettes for a few days.  On the way home from the Turkish grocery, I happened across a cathedral with its stained glass windows all lit up.  These kinds of random encounters are some of my favorite parts of an unplanned, spontaneous trip.

Most places in Europe have a surprisingly familiar feel. They're not much different than big cities in the States.  Aside from the local language, it's often small differences that seem most memorable.  For instance, sliced bread in Marseilles (and Athens too) has no heels.  They're removed at the factory.  At church on Sunday, everyone kisses hello instead of shaking hands: one kiss on the left cheek, then one on the right.  This was the custom throughout Italy too. Alcohol is used more freely as an ingredient without any obvious labeling.  So far, we've accidentally purchased alcohol while trying to buy pop, birthday cake and candy.

After Marseilles, we spent ten days each in Genoa, Florence and Rome.  Our house in Genoa was above a bakery and a pizza restaurant so it always smelled delicious.  Because Genoa is so hilly, they have public elevators that serve the same purpose as public buses but bring you from one part of town to another by traveling vertically.  The hills also make for some nice views of the city:

Florence had some great statues and architecture.  It also had many more tourists than our previous destinations.  As we walked through the city, we heard more English and Chinese than we heard Italian.  Since we're all early risers, we visited most sites as soon as they opened and before most tourists were awake.  That worked great.  When we arrived at Galleria dell'Accademia to see Michelangelo's David, we were the only ones in the room.  By the time we left, the place was packed.

We also celebrated Thanksgiving in Florence.  We couldn't find turkey or pie, but rotisserie chicken, dates, figs and cookies were delicious:

The age of things in Europe is always surprising for an American. For example, our house in Florence was constructed in 1470.  It's been around longer than Europeans have known about our continent.  I've also been impressed with the kindness of people throughout Italy.  For example, on our second Sunday in Florence, it rained on us the entire time we walked to church.  When we arrived, our coats were soaked.  The people at church couldn't speak English, but they welcomed us with a smile, helped us out of our wet coats and gave us warm, dry sweatshirts.

Rome is always fun.  I've been there a few times and always enjoy wandering around and seeing random ruins in the middle of the city.  The kids have been really interested in Roman mythology for the last couple years, so it was particularly fun to explore the city with them.  They'd narrate each statute with a full genealogy of the relevant gods and their myths.  We were also in St Peter's Square at the same time that Pope Francis was giving a general audience, so the timing on that worked out great.  Oh, and Roman broccoli is snazzy:

Athens was our last stop on this leg of the journey.  Jericho accidentally left his backpack, with his laptop in the taxi from the airport.  We had no luck recovering it, so my first stop in the city was to find a cheap, used laptop for him to do his school work.  We ate baklava and cheesecake and visited the Acropolis.  Jericho passed the sacrament in the small English-speaking branch of the Church that we attended in Athens.  We also met a family at church that had just moved from Jordan, our next stop, so they gave us some pointers.  All around, it's been a fun month.

World tour: Spain

We started our time in Spain with a brief stop in Madrid followed by ten days in each of Toledo, Cordoba, Valencia and Barcelona.  We spent about five weeks in Spain altogether, covering all of October.

The stop in Madrid was a logistical detail leftover from Ryanair canceling our flights to Spain.  Our real journey began in Toledo.  When we arrive in a new place, we usually try to walk from the train station to our house so that we can get a feel for the city.  That's not always possible, but it usually works well.  In Toledo, it was a great walk.  The route took us near a couple of ancient bridges into the city, past some great views of the Alcazar on the hill and through a canyon along the Tagus river:

One of our goals for living abroad is for our kids to learn how to navigate through places they're unfamiliar with.  We take turns having each kid lead us to our destination by following signs or maps.  Sometimes we get lost, sometimes we arrive unscathed and we almost always have fun.  Here's Gideon leading the way through Toledo's old city:

One challenge we've noticed with traveling and living abroad is that all seven of us are in tight quarters most of the time.  In Wyoming or Iceland, when you want some private time, you walk into the wilderness, sit on a rock and stare at the clouds.  Elbow room is easy to come by in those geographies.  In the Spanish cities where we traveled, personal space is harder to come by.  Our houses in both Toledo and Cordoba had courtyards where we could be outside while still being walled off from the hustle of the surrounding city.  We instituted a family rule that the courtyards were silent places.  If anyone wanted quiet, they could retire to the courtyard for a little peace.  It's not quite a rock in the prairie, but it got the job done.  In Toledo, Haven also found some peaceful spots along the Tagus:

In Cordoba, our house was a couple blocks away from the Mezquita-Catedral.  This building had been a mosque for almost 500 years during the time that Muslims controlled Spain.  After the Reconquista, it was converted into a Catholic church.  They basically just changed the sign on the door and converted the minaret into a bell tower.  Walking through the church, it looks just like a mosque but with Christian chapels around the outside wall.  It was fascinating.  Anyway, the Cathedral has a large irrigated courtyard planted with orange trees and date palms.  During a morning visit, a branch full of ripe dates fell from a palm tree, so we all ate dates in the shade of the Mezquita.  It's a beautiful place:

In Valencia, our house was near the beach and the city's main port.  We watched container ships come and go while playing in the sand:

Valencia also has a giant playground inspired by Gulliver's time with the Lilliputians.  A giant statue of Gulliver is tied to the ground so that children can climb and slide and jump all over him.  It's hard to communicate the scale of the playground, but here's a picture of all five children scattered along Gulliver's left leg:

The timing worked out great to travel through Spain during the Catalan referendum on independence.  We were in Toledo on the day of the vote and in Barcelona when the Catalan Parliament declared independence from Spain.  The politics had very little impact on people's daily lives.  We saw no protests and only talked to one person who mentioned the situation.  The most visible sign of the process was flags flying from apartment balconies.  In Cordoba, the apartments draped a Spanish flag over the balcony to signal support for Spanish unity.  In Valencia, about half the flags were Spanish and half were the Estelada, signaling support for Catalonian independence.  In Barcelona, we saw the Estelada almost exclusively.  It was joined by flags saying "Si" or "Hola Republica".

Next stop: Marseille, France.  We'll only be there for 10 days, so I might lump it in with a blog post about Italy next month.

This may not seem like much excitement for an entire month in a foreign country.  That's partly by design.  We want this year to be more like living abroad than an extended vacation.  Most days for us are just like most days for you: we exercise, go to work, buy groceries, cook dinner, help the kids with school work.  We explore the city a couple days each week and go to church on Sunday.

The surprises of travel spice things up a bit, but not as much as you might expect.  I broke one of my ribs in Boston, so exercise has been tricky.  We ate galleta Maria with hot chocolate for breakfast every morning since that's what the Spaniards do. While trying to catch a train from Barcelona to France, we discovered that we were departing on All Saints' Day and buses to the train station were canceled. We're always glad to find a pay toilet since we know it'll be clean and fully stocked.  Small surprises like this, against a backdrop of structure, make it fun and educational without being stressful.

World tour: Iceland

After our last minute pivot to Boston, we left for our first destination abroad: Iceland.  When scheduling flights from the US to Europe, Icelandair had some of the least expensive flights.  They also let us stopover in Iceland for up to 7 days for the same cost as flying directly to Europe.  Since we wanted to see Iceland anyway, the stopover worked great.  They turned out to be a really comfortable airline too; more pleasant than others I've tried.

We rented a small house on the south coast near Strandarkirkja.  It's about an hour's drive south of Reykjavik (which we didn't visit).  We loved the rural feel in this area.  The closest town was about 15 minutes away.  There were only about 5-10 houses near us.  Everything else was open fields, lava rocks, ocean and the occasional rainbow:

During the five days we were here, the sun shined for about 12 hours total.  It rained the rest of the time.  I thought that was pretty good weather for autumn in the north Atlantic. The kids enjoyed walking along the rocky beach near the house on one sunny day.  The water was too cold for swimming, but they enjoyed throwing rocks and watching sea critters:

When we travel, we try to live like the locals and to avoid the most popular destinations for foreign visitors.  So instead of eating at restaurants, we buy groceries at a local store and cook our meals at home.  Instead of visiting Blue Lagoon, we went to a small community pool in Hveragerði, etc.  A slow pace in small venues suits us well.  Anyway, as you can imagine, seven Americans wandering around a small town grocery store in Iceland draws plenty of attention.  Our meager attempts at Icelandic were laughable, but everyone we encountered spoke excellent English.  Iceland is roughly the same size and population as Wyoming and they speak their own language so very few goods are manufactured specifically for the Icelandic market. That meant that their grocery store is an amalgamation of items imported from UK, Norway, Sweden and Denmark with labels in the same assortment of languages.  It was fun to decipher all the languages.  We ate horse sausage with dinner one night and had Icelandic lamb another night.  We also enjoyed the name of local milk brand:

Iceland is a remarkably beautiful place.  It reminds me a lot of Wyoming: open, windy, treeless expanses in a harsh climate.  The barren landscape that we saw may not be for everyone, but we really liked it.  Other areas, like Hveragerði were gorgeous in a more traditional way:

Our only regret about Iceland is that the stopover prevented us from spending more time here.  We enjoyed the country so much that we think we'll stay in Iceland again on our way home in the spring.

After Iceland we had planned to stay in London for a few days before flying to Madrid.  However, three days before departing Iceland, Ryanair canceled our flight to Madrid so we had a quick, overnight turnaround in London instead (the only flight remaining).  We should be able to see London in the spring when we return to the UK.  We did enjoy seeing London at night as the plane landed:

Next stop: Spain

World tour: Boston

First step in the world tour: pack our bags.  With the many planes, trains and buses we'll use over the next year, there's not much room for luggage.  Each person gets a carry-on and a laptop bag.  Brigham helped me load it all up:

And Gideon created a small token of our departure:

To kick things off, we had planned to surprise the kids with a trip to Orlando.  The ever thoughtful Hurricane Irma had her own surprise in mind: cancel our flights three days before departure.  Since we couldn't reach Orlando, we did a quick pivot to spend the week in Boston (from which our Europe flights departed anyway).  It all worked out surprisingly well.  We found a small house near Revere Beach.  Haven and Jericho insisted that the water was warm enough for swimming:

Wonderland station on the Blue Line was only a 20-minute walk from our house, so we used the T most days to travel into the city.  Our daily walks near the ocean to and from the subway became a pleasant feature of the trip.  On one walk, Brigham got bored and decided to use a small shell to collect sand from the beach and construct his own private sandbox in the shade of a gazebo.  As you can imagine, this was a very slow process:

In the city, we visited the New England Aquarium, some historic sites and Boston Common.  Walking through Boston Common one evening after dinner, we were 150 ft away from a shooting.  We heard the shots being fired and saw some of the participants dive to the ground.  The crowd scattered and we followed suit.  We joked later that we dodged a hurricane and bullets within a few days and that this trip was turning out to be more exciting than we'd expected.

We also toured Plimouth Plantation to learn about Pilgrims and contemporary natives.  The Plantation is probably the best living museum I've ever been to.  There's a lot to explore and the staff is all very knowledgeable and glad to answer questions.

On our last day in Boston, we tried to attend a local ward's Sunday meetings.  It turned out that they had stake conference that day so nobody was at the meetinghouse.  Two other groups of travelers showed up with the same plans.  After they all left, we stood outside the locked building in the shade and sang primary songs and had our own little Sunday school lesson as a family.  It was a pretty great way to spend a Sunday:

Next stop: Iceland

Summer 2017 Road Trip

After Kinsey graduated from school in Johnson City, we spent the next couple months slowly making our way back to Wyoming.  We'd drive for 4-6 hours, stay for 7-10 days and repeat.  Except for a visit to Memphis (for Kinsey to take her TN licensing exam) and the ocean we had no planned itinerary.  We picked locations as we went.  We stayed in VRBO and Airbnb houses at most stops.  Our route ended up looking like this:



Our first stop was to spend some more time in Appalachia.  This was the view from our porch every morning:

Next to our house was a patch of wild blackberries.  The kids loved to pick and eat berries whenever they wanted a snack:

We all really enjoyed Appalachia.  There were lots of trails for hiking, beautiful scenery to explore and quiet places to enjoy.  Unlike the Rockies, the forests are dense and trails were typically a narrow path surrounded by thick foliage.  In this picture, there are three children ahead of Jericho and Brigham, but you can't see them for the trees:

Only about 70,000 people live in Johnson City, TN but that was a big city by our standards.  We'd had enough of the urban hustle by the time we left, so we spent our first month in small towns throughout the northern parts of Dixie.  In Middle Tennessee, we stayed on 50 acres near a small stream.  In the Arkansas Delta, we were in an old farmhouse surrounded by fields of cotton and rice.  The owner's 10 children lived nearby so our younger kids passed the time playing with them in irrigation ponds and climbing trees.  Kinsey and I ran the nearby farm roads each morning:

One of the kids' favorite stops was in Hot Springs, Arkansas.  We had a small house on the edge of a lake.  They spent every day jumping off the dock and getting sunburn.

While fighting with pool noodles near the water's edge, Kinsey knocked Jericho's glasses right off his face and fifteen feet into the lake (as best we could tell) so Jericho spent the next month squinting at all the scenery.  For our upcoming trip to Europe, we've packed a couple extra pair of glasses for each kid.

Hot Springs also had a small putt-putt golf course in town.  One rainy morning we decided to play nine holes and had the entire course to ourselves.  Brigham (3 years old) scored four holes-in-one on the course, three of them consecutively on a single hole. After the second hole-in-one, I bet the kids $10 that he wouldn't score another.  Nobody took the bet.

The rest of our trip had a more urban flavor.  The kids had never tried a performance Japanese restaurant, so we made an obligatory stop at Benihana:

Same story for the beach.  Padre Island was nice because we could park right on the sand.  The mornings were quiet with only a few people around.  We left the beach each day around 11 am to eat lunch, avoid the crowds and retire to the pool (which was less popular than the beach).

After Padre Island, most of our trip was devoted to visiting family.  On the way, we tried to visit the Riverwalk in Pueblo, CO and met with an unexpected and heavy rainstorm during the walk.  We took shelter under a bridge where Aravis and Brigham passed the time with a little dancing:

We had a lot of fun, made great memories and loved having months together as a family without distractions or external responsibilities.  The naps were nice too:

Of all the places we visited, Wyoming was our favorite.  On my first walk after being home, I was reminded how much I love it here.

Parenting? Roll a Dice

The Problem

I have five children between the ages of 13 and 3.  Everywhere we go, there's a scarce resource that all the kids want.  Maybe it's an elevator button that everyone wants to push.  Maybe it's a favorite spot in the car where everyone wants to sit.  On the flip side, there are chores that nobody wants to do: take out the trash, load the dishwasher, etc.  I used to make assignments round-robin, trying to give each kid a turn at fun and chores.  I've tried other scheduling algorithms too, but I inevitably forget whose turn it is and then everyone tries to straighten me out: "But it was Gideon's turn last time. No, that was Haven not me. ..."

I've tried for years to convince them that these things don't matter and that life is not fair.  I'm apparently poor at communicating that message.  So I decided to take my own sermon to heart:

My Solution

If it really doesn't matter, why not pick kids at random?  I installed Random Name Picker on my phone.  I created a list named Children and added each child's name to that list.  In the list settings, I chose "With replacement" and set "# of names chosen" to 5.  When I click Choose, the app shuffles all the children into a random order.  Now that little Choose button resolves all trivial, family disputes.  We've been doing this for months and it works great.

Only space for one kid to help Dad at the store? Click Choose and the top name wins.  Choosing 2 ice cream flavors at the store? Click Choose and the top two names win.  On vacation? Click Choose in the morning and read everyone the full list to assign priority for pushing elevator buttons that day.  Assigning all children a random priority has proven especially helpful.  Each kid remembers their place in line.  No matter which subset of children happens to travel in the elevator at any given time, they instantly know whose turn it is to push the buttons.

One extra rule proved useful: Dad always clicks Choose.  Otherwise, things get meta really fast: "I get to click Choose to see who gets to click Choose to see who gets to pick the ice cream"

Goodbye Duolingo

Today I'm giving up a 665 day streak in Duolingo.  I've removed the app from my phone.  Duolingo is still a helpful tool for many and they have potential to become great again.  Unfortunately, the trend in their recent changes has encouraged me to leave.  I'm mostly writing this to warn myself about practices that can drive customers away.

I didn't mind when Duolingo started showing ads.  I knew they had to do it eventually.  They even did a good job of showing ads in predictable locations and using ads of relatively high quality.  I anticipated a subscription model allowing me to pay $1 per month to remove the ads and expected no other functionality for my subscription.  Unfortunately, the subscription costs 10x more than I was willing to pay.  For reference, Duolingo charges $9 per month.  Netflix charges $7 per month and provides much greater value.

No big deal.  I'll skip the subscription and just look at the ad after each lesson.  I'm sure that 90% of users do the same thing.  Duolingo doesn't seem content with that, so they randomly move the location of the "close this ad" button on each page.  Sometimes it's in the top left corner.  Sometimes it's in the lower right corner.  This inconsistency is disrespectful.  It's playing games with my time in an apparent attempt to manipulate me into clicking their ads or subscription button.  If most users are going to skip the ads, the "close this ad" button should be predictable and convenient (on the bottom where a thumb can quickly reach it).

It was annoying, but I was willing to live with it.  Duolingo has to make money.  They're being a little pushy about it, but I can overlook that.  Then they changed the "strengthen my skills" button so that it only performs a timed practice.  I thought, "maybe they've seen that this produces better results, so I'll try it".  After using it for 2-3 weeks, I found myself skipping any question which took longer than 1-2 seconds to answer (listening comprehension, free text response).  The new rules of the game were encouraging me to learn a language less effectively.  For people whose focus is conversational speech, it might make sense to practice under time pressure.  My goal has always been reading comprehension.  In that context, time is not an issue.  I can spend as much time as I want on a German Wikipedia page and nobody will care.

Android has always been a second-class citizen for Duolingo.  Useful features come to iPhone first and often don't make it to Android for months or years.  It's no big deal since the core functionality is available on Android.  Duolingo makes it very clear that they're trying to hire an Android developer.  They've been trying for as long as I can remember.  A couple days ago, I installed an update to the Android app.  Even though I only use the app for 10 minutes each morning, it now consumes 9% of my battery each day.

In a market for user attention that's so competitive, you really have to play at the top of your game.  Anything pushy or disrespectful can have large costs.  Repeated minor issues like this can push a user away.  I hope I can remember that as I work on software.

Duolingo.  Thanks for 665 days of fun.  Sorry that things didn't work out between us.

Toggle syntax highlighting to catch bugs

I usually write code with syntax highlighting enabled.  While preparing my final commit message, I view the proposed diff in its own color scheme (red for removed lines, green for added lines, white for context lines).  Even though I've spent hours working on a patch, I often spot mistakes in my newly added lines as soon as the color scheme changes.  Apparently, psychologists already knew about this phenomenon: “Once you’ve learned something in a particular way, it’s hard to see the details without changing the visual form.”  The article suggests other visual changes like using a different font or printing to paper.

I use two other, related hacks for helping myself find mistakes in my code:

• go to sleep and review my code in the morning
• watch a video, play a game, read a book, work on a completely unrelated problem to force my mind to lose as much of its mental model as possible then review my code again

In each case, the new perspective often reveals details that I overlooked before.

Sensitive survey questions

Do you steal from your employer? Do you lie on your taxes? Have you cheated on your wife?  If you want to gather statistical information about these questions, you can't ask directly.  Most respondents will lie.  I'm aware of three methods for addressing the problem, two of them are quite clever.

Bogus Pipeline

The first one is not particularly clever.  Hook the subject to a machine.  Tell them it's a lie detector even though it's not. Ask them to respond honestly and pose a few baseline questions to which you know the answer (What's your name? What day is it? etc). After each answer, have the machine indicate that it detected truth.  Now ask the subject to respond deceptively and ask more baseline questions. After each response, have the machine indicate that it detected a lie.  Now hide the machine's truth/lie indicator and ask your questions.  Most subjects will tell the truth.

This is called a bogus pipeline. It's complicated to implement, requires physical access to the subject and not as accurate as other techniques.

Randomized Response

Ask the subject to flip a coin but don't tell you what it is.  If it's heads, they should answer truthfully. If it's tails, they should answer yes (or whatever the socially unfavorable answer is).  Now ask your question.  Applying some simple math to the aggregate responses, you can accurately calculate the percentages you want to know.

This one's pretty helpful, but it requires the subject to have a coin (who uses coins anymore?).  The subject must also be smart enough to recognize that the coin gives him deniability. It seems obvious, but it's not obvious to everyone.

Unmatched Count

Construct an innocuous survey along these lines: "How many of the following statements are true about you? I own a dog. I drink coffee. I've been married. I have brown hair."  Construct a second survey, identical to the first but add your sensitive statement, "I cheat on my taxes".  For each subject, randomly give them one survey or the other.  Calculate the average answer for each type of survey.  The difference between the two averages tells you the percentages you want to know.

This one's my favorite. Since the subject only tells you their final count, it's obvious to them that they've divulged no sensitive information.  The math for analyzing the results is similarly easy.

Do you know of any other techniques?

Switching to OpenBSD

Short story:

After 12 years, I switched from macOS to OpenBSD.  It's clean, focused, stable, consistent and lets me get my work done without any hassle.

Long story:

When I first became interested in computers, I thought operating systems were fascinating. For years I would reinstall an operating system every other weekend just to try a different configuration: MS-DOS 3.3, Windows 3.0, Linux 1.0 (countless hours recompiling kernels).  In high school, I settled down and ran OS/2 for 5 years until I graduated college. I switched to Linux after college and used it exclusively for 5 years. I got tired of configuring Linux, so I switched to OS X for the next 12 years, where things just worked.

I was pretty happy with OS X.  It gave me Unix and mostly got out of the way so that I could write software.  I wrote about enjoying Apple's simplicity.  Snow Leopard even spent an entire release cycle just fixing bugs and improving performance.

But Snow Leopard was 7 years ago. These days, OS X is like running a denial of service attack against myself.  macOS has a dozen apps I don't use but can't remove. Updating them requires a restart.  Frequent updates to the browser require a restart.  A minor XCode update requires me to download a 4.3 GB file.  My monitors frequently turn off and require a restart to fix.  A system's availability is a function of mean time between failure and mean time to repair.  For macOS, both numbers are heading in the wrong direction for me. I don't hold any hard feelings about it, but it's time for me to get off this OS and back to productive work.

So where do I go now?  We own 5 Chromebooks and they have great availability.  Updates are infrequent, small, fast and nearly transparent.  Unfortunately, I need an OS where I can write and compile code.  I also want it to run on older, commodity hardware so I can replace a broken laptop for $400 instead of $2,000.

I considered several Linux distributions.  Lubuntu seemed promising, but it was too bloated for my taste.  A couple years ago, I tried Ubuntu on a Dell XPS Developer Edition for a few months.  Even with hardware designed for Linux, it was too fragile. Desktop Linux has also become even more complex than when I used it a decade ago.  I just want to get my work done, not feed and maintain an OS.

I was reminded of OpenBSD during the Heartbleed scare.  While everyone else was complaining about OpenSSL and claiming that open source had failed, the OpenBSD developers quietly drew their machetes and hacked out hundreds of thousands of lines of bad code, forking off LibreSSL where they can keep it clean and stable.  The OpenBSD community is like that: focus on what's really important, hold your code to a high standard, ignore all the distractions.  They're not trying to live in the past, just trying to make the future a place worth living.

Anyway, I found OpenBSD very refreshing, so I created a bootable thumb drive and within an hour had it up and running on a two-year old laptop.  I've been using it for my daily work for the past two weeks and it's been great.  Simple, boring and productive.  Just the way I like it.  The documentation is fantastic.  I've been using Unix for years and have learned quite a bit just by reading their man pages.  OS releases come like clockwork every 6 months and are supported for 12.  Security and other updates seem relatively rare between releases (roughly one small patch per week during 6.0).  With syspatch in 6.1, installing them should be really easy too.

I also enjoy that most things are turned off in OpenBSD by default.  The base installation is sparse.  It assumes that I'll enable a service or install a tool if I want it.   So I'm not constantly facing updates for software I never use.

My experience with OpenBSD is still young, but I really like what I see so far.

Using Project Fi

I signed up for Project Fi last month.  It's been a real pleasure to use.  I expected it to be a step down compared to Ting, but I was wrong.  For my use cases, Fi is slightly better in a couple ways.  Fi's customer support isn't as good, but it's acceptable and better than most phone companies.

Billing

Project Fi canceled my Google Voice account during sign up, but it transferred all my GV account credit over to the new account.  The credit wasn't visible on the first bill but appeared on the second.  It was nice not to lose those GV funds.  (Fi also transferred my voicemail greetings and blocked numbers from GV).

I really like the way that Fi charges for data.  I chose the 1 GB plan (since there's no penalty for overages).  Last month I consumed 1.165 GB of data.  On Ting I would have paid $10 for crossing into the second GB.  On Fi, I paid only $1.65 extra; exactly covering my overage.

Fi provides free data-only SIM cards whose usage is just added to your account.  Since my family uses VoIP for all phone calls (either SIP or Google Voice) and SMS, everyone just needs data.  I gave everyone a data-only SIM card and it's been working great.  The SIM cards work in every device we've tried, old and new.  This avoids the $20/month charge per phone line.  Fi billing breaks out usage for each device.  The only downside is that you can only order one data SIM at a time.  It took me a few weeks to place orders for all the cards I needed.

Coverage

Ting is built on T-Mobile's network.  It has great coverage almost everywhere I go.  The one exception is the northwest quarter of my grocery store.  Data signal in that part of the store was always missing.

Since Project Fi automatically switches between T-Mobile, Sprint and US Cellular networks, depending on signal strength, I expected Fi to do better.  It did.  This dead spot is no longer a problem.  In this particular store, Fi often switches to the Sprint network then switches back to T-Mobile when I go elsewhere.  Never underestimate the power of a layer of abstraction.  Data-only SIM cards only use T-Mobile, so their coverage is identical to what I had on Ting.

I use Signal Spy to see which network my phone is currently using.  I find it gratifying to drive through town and watch Fi switch networks.  I look forward to some road trips this summer to see how effective it is in that scenario.

Fi's switching algorithm sometimes sticks with a network whose signal is slightly weaker than the alternatives.  Theoretically this could hurt battery life, but I've never had it impact connectivity.  It does annoy my OCD a little.

Conclusion

Overall, I'm very happy with Project Fi and will probably stick with it after I return from Europe.  I hope Fi's international coverage is as good as the US coverage.

Cancel a Stripe subscription in App Engine

Stripe's API for canceling subscriptions requires that you send an HTTP DELETE request.  If you want to cancel the subscription at the end of the current billing period (instead of immediately), you need to include a body parameter at_period_end=true in the request.  That's fine.  RFC 7231 section 4.3.5 says that a DELETE request is allowed to have a body but it "has no defined semantics".

Unfortunately, App Engine's urlfetch service silently removes the body from all outgoing DELETE requests.  Trying to cancel a Stripe subscription in App Engine always does it immediately, even if you asked to cancel at the end of the billing period.   Google's known about the problem since 2008, but never fixed it.  The glitch impacts many APIs other than Stripe.

Fortunately, you can use App Engine's sockets API to construct a workaround.  In Go, you build an HTTP client whose transport uses the socket API to make outbound network connections.  That uses Go's built in support for HTTP DELETE requests and avoids the bugs in App Engine's urlfetch service.

httpClient := &http.Client{
 Transport: &http.Transport{
  Dial: func(network, addr string) (net.Conn, error) {
   return socket.Dial(c, network, addr)
  },
 },
}

Programming Languages by Spec Size

I was curious which programming language has the smallest specification.  Which one has the largest?  For each language, I printed the spec to a PDF and counted the pages.  Self 2017.1 is the smallest.  C++ is the largest.

This is a very rough estimate of language complexity since each specification varies in style and purpose.  I tried to count only the language specification. For example, I didn't count Annex A in the Prolog spec since it's only informative.  For Haskell 2010, I didn't count Part II - Libraries which defines the standard library, not the language.  Anyway, here's the list:

• Self 2017.1 (see Chapters 3-4) - 37 pages
• Newspeak - 49 pages
• Lua 5.0 - 71 pages
• Go - 75 pages
• Shell - 83 pages
• OCaml 4.0 (see Part II) - 88 pages
• Scheme R7Rs - 88 pages
• Smalltalk-80 (see Part One) - 89 pages
• Prolog (not freely available) - 124 pages
• Nim 0.19 - 133 pages
• Standard ML '97 - 136 pages
• Haskell 2010 - 152 pages
• Dart 2.2 - 206 pages
• Ruby - 331 pages
• F# 4.1 - 333 pages
• C# - 553 pages
• JavaScript - 586 pages
• Java - 670 pages
• C 2011 - 701 pages
• Ada 2012 - 947 pages
• C++ - 1368 pages

If you'd like me to add other languages, comment with a link to the spec and I'll update this post.

gsutil: No module named google_compute_engine

While trying to run gsutil ls on a Compute Engine VM, I received a stack trace like this:

Traceback (most recent call last):
  ...
ImportError: No module named google_compute_engine

It turns out that gsutil doesn't like Python installed from Linuxbrew.  It really wants to use the system Python.  The following works fine:

env CLOUDSDK_PYTHON=/usr/bin/python gsutil ls ...

Easy enough to fix.  Nobody else had documented trouble with this configuration so here you go Internet.

AGI Script completed, returning 4

I use Asterisk to run my phone system.  Most of my dialplan is an AGI script written in Perl.  In certain circumstances, I kept finding the following message in Asterisk's logs:

AGI Script ... completed, returning 4

My script should always return 0, so seeing an exit code of 4 was unexpected.  After this log message appeared, no further code in my AGI script was executed, also unexpected.

After a bunch of debugging, I discovered that Asterisk sends SIGHUP to your script if a caller hangs up.  The exit code 4 was Asterisk's interpretation of Perl's exit code when SIGHUP wasn't handled.  I installed a handler for that signal and now everything's grand.

Prior to tonight's debugging, I had never used the sigtrap package.  Its stack traces were very helpful.

Serving Git repositories from Go

I have a small web server written in Go.  I wanted to serve Git repositories from this server.  It turned out to be surprisingly easy since git-http-backend and the cgi package do all the hard work.

First, define a function for handling Git's HTTP requests:

func git(w http.ResponseWriter, r *http.Request) {
        username, password, ok := r.BasicAuth()
        if !ok || username != "john" || password != "secret" {
                w.Header().Set("Content-Type", "text/plain")
                w.Header().Set("WWW-Authenticate", "Basic realm=\"example.com git\"")
                w.WriteHeader(http.StatusUnauthorized)
                fmt.Fprintln(w, "Not authorized")
                return
        }

        const gitRoot = "/path/to/git/repositories/"
        h := &cgi.Handler{
                Path: "/usr/lib/git-core/git-http-backend",
                Root: "/git",
                Dir:  gitRoot,
                Env: []string{
                        "GIT_PROJECT_ROOT=" + gitRoot,
                        "GIT_HTTP_EXPORT_ALL=", // we require auth above
                        "REMOTE_USER=" + username,
                },
        }
        h.ServeHTTP(w, r)
}

Second, connect that function to the mux you're already using:

mux.HandleFunc("/git/", git)

Now I can clone from and push to my little Go server.

Sharing is dangerous

Last week I tried pass as an open source replacement for 1Password.  It was almost exactly what I wanted.  Unfortunately, it uses GnuPG for encryption.  GnuPG is a pain and bloated (444k lines of code).  Since pass just runs the gpg binary, I wrote a quick script that implements the gpg shell interface but does encryption with a much smaller library.  The proof of concept worked.  I made a note to migrate to pass later and uninstalled it.  I forgot to remove my fake gpg script.

This morning I tried running "brew update".  It fetched Git repositories then stalled without hints about the cause.  After too much debugging time, I discovered that it was running my fake gpg which blocked waiting for input.  I deleted fake gpg then "brew update" proceeded fine.

Too much sharing

The wasted debug time was clearly my fault, but it reminded me how dangerous sharing is.  Too much of our software is a wobbly tower of dependencies.  Sure, you can change the bottom block, but it's risky.

The well-known costs of global mutable state are a symptom of problematic sharing.  Package management becomes NP-complete dependency hell when sharing is mandatory (assumption 4). The recent Cloudbleed vulnerability was mostly a problem because of sharing:

Because Cloudflare operates a large, shared infrastructure, an HTTP request to a Cloudflare web site that was vulnerable to this problem could reveal information about an unrelated other Cloudflare site. (emphasis added)

In other words, FitBit and Uber have a security vulnerability because some random WordPress blog generated bad HTML.

Don't share, copy

Some cultures don't like to share.  Docker containers isolate applications from each other.  Node prefers to load a module that's the least shared.  OpenBSD doesn't like to share file system partitions.  Chrome sandboxes each site to prevent sharing.  This is pretty nice.  If my Youtube tab has a problem, I don't have to consider debugging the Reddit tab.  If I upgrade one project's Node dependency, it doesn't break other Node projects.

Some cultures that used to favor sharing are moving away from it.  Perl (local::lib) and Go (vendor directories) come to mind.  Of course, everyone can still share (in the sense of using identical code), they just get their own local copy of it.  Go's default of building static binaries is a breath of fresh air.

All of this is just a reminder to me: if my code shares anything, find a way to make it either local (not shared) or immutable.

Caller ID for SMS in XMPP

When you receive an SMS on a traditional mobile device, your phone looks up the number in your local address book and displays the corresponding name.  That works great for a small social network where you can maintain the list yourself.  As the number of potential contacts (coworkers, telemarketers, etc) increases, it becomes unwieldy.

For example, I routinely receive SMS from people in my ward or stake who I'm unlikely to contact on a regular basis.  That's a few thousand people and the population changes weekly.  I don't want all that info in my regular contact list.

Fortunately, I run my own little phone system that uses XMPP for sending and receiving SMS.  I wrote a gateway to handle the conversion back and forth between those networks.  That gateway can run a script to perform caller ID however I want.  So now I have something like this when an SMS arrives:

1. check my personal address book
2. check the stake directory
3. check 800notes for records of telemarketing

I also have multiple external phone numbers.  One of them is a junk number that I give out when I fill out online forms.  If an SMS comes in to that number, the caller ID appears with a "(junk)" prefix.

It's a lot of fun to be able to write a couple shell scripts to customize how your phone works.

Signing up for Project Fi

In preparation for an extended visit to Europe, I signed up for Project Fi today.  Prior to this I had been using Ting.  I love Ting.  I do all my SMS and voice communications over IP, so I really just need a reliable, affordable data plan.  If Ting had better data rates in Europe, I would have stuck with them.  Anyway, here are some first impressions while signing up for Project Fi.

Charging $20 per month per phone line seems expensive.  That costs $6 on Ting.  Fi gives you unlimited calls and texts but since I always use zero of those, I'd rather have a cheaper base rate.

Fi charges $10 per GB for data.  As they say, "No matter your budget, you'll only pay for data you use."  So why are they asking me to set a budget?  It seems like an extra step without a purpose.  I chose the smallest budget allowed, 1 GB.  This combination costs $30 per month on Fi and $22 on Ting.  An extra $8 per month to get excellent data rates across Europe seems reasonable to me.

I already have a Google Pixel so I only needed a new SIM.  That was free.  As I recall, Ting charges $12 up front for that, so $0 is nice.

Since my public phone numbers are all managed through Twilio, it doesn't really matter to me what phone number Fi assigns me.  They gave me the choice of porting a number or choosing a new one.  I chose to receive a new one.  Unfortunately, they only offered numbers with area codes in my local area.  Some people might like to choose an arbitrary area code.  They didn't let me choose the specific number at this point.

Fi asked for my primary service address.  Since I'll be using this phone all over Europe, I wasn't entirely sure what to put.  I used my home address in Wyoming.  Fi didn't seem picky about that.  They even let me use a different shipping address for the SIM card.  That was convenient since I was traveling at the time I placed the order.

Overall the sign up process was smooth and the prices seem reasonable.  When the SIM arrives in 7 days, I'll put it through its paces.

Selfhosted Google Voice

I've used Google Voice since 2010.  During that time, it's been my main telephone number.  Besides Search, it's my favorite Google product.  Google has even renewed their commitment to the product recently.  Unfortunately, Google Voice only works for US customers and my family and I are moving to Europe later this year.  GV has spoiled me and I don't want to return to the old days when I used my mobile carrier's number directly.  This is how I'm selfhosting an alternative to Google Voice.

Twilio

Today I finished porting my Google Voice number to Twilio.  They offer a great platform on which to build telephony apps.  I've been very impressed with their customer support too.  It's refreshing to be able to discuss telephony problems with your provider.  That was never possible with Google Voice.

They offer phone numbers in every country we plan to live.  They provide numerous telephony services, but we're only using a few right now.

SMS

I selfhost an XMPP server for my family.  We use it all the time for messaging each other.  We run Conversations on mobile and use other XMPP clients on desktop.  It works great.  I wrote a gateway between XMPP and SMS.  You send an XMPP message and it converts it into an SMS.  When someone sends me an SMS, it converts it into an XMPP message.

This all works through Twilio's SMS API.  Since the whole system is under my control, I can add arbitrary logic for handling SMS.

Voice

To replace the voice component of Google Voice, I set up an Asterisk instance.  I use Twilio SIP trunking to connect to the telephone network.  I run Zoiper on my Android phone.  To make an outgoing call, I use the Zoiper dialer and Asterisk routes the SIP call through Twilio to the public telephone network.  When someone calls my Twilio number, Asterisk routes the call to Zoiper on my Android device.  The call quality is great.

I have no previous experience with PBX systems, but it was all surprisingly easy to configure.  Once again, since it's all in my control, I can add whatever features I want.

Future

For now I'm using a very basic setup.  SMS accounts for 90% of my Google Voice usage and that part works really well.  Even if I never had more than this, I think I'd be content.  I have a bunch of features in mind and this setup makes it really easy to add them as I need them.