All of the dominant commercial operating systems in desktop and mobile computing — macOS, iOS, Windows, Android, Chrome OS — are made in the US, on the West Coast. Except for Microsoft, they are heavily concentrated in the SF Bay Area. (Note that I am not talking about free software such as GNU/Linux, BSD, etc., which is probably more diverse regarding the location of their developers.)

Flight Tracking

All of these OSes are very eager to support you with flight tickets and boarding passes. For instance, you can add your boarding pass to Apple Wallet, or Google Wallet, and it will automatically surface when you are at the airport. Very convenient!

I have a flight to Tallinn, Estonia, in a few weeks. Ever since the flight appeared in my airline’s mobile app, my iPhone has been sending me reminders — daily, sometimes even multiple times a day — that it has found data about a flight, added it to my calendar and will be tracking it. Similarly, on iOS, you can pull up the OS search field and enter a flight number to get instant tracking of that flight. The OS will even remember it for a while.

Culture?

Here is the thing though: Try to enter, say, “ICE 79” into the same search field, and you get — nothing.

This made me realize how much cultural influence bleeds into the features that they provide. It’s not like these companies do not care about public transit. Both mobile Wallet apps have good support for things line the OMNY pass in New York City and the Clipper card in the Bay area. It’s just that long-distance, inter-city travel by train is all but nonexistent in the US. A few people take Amtrak, but they are not worth building support for. In the US, people just fly or drive. Rail, transit in general, is something you might use for your commute and that’s it.

Now, there are some technical hurdles to providing similarly good support for trains. For one, the data formats are not as standardized as in air traffic. But that could be changed. Google and Apple do have transit teams. They did make most transit agencies output feeds in GTFS format for real-time updates, plans of which metro line goes where and whatnot.

So if there was a push for a standardized way of supporting inter-city rail tickets, it would probably succeed. But it doesn’t look as if this is a priority for any of them.

I recently made two LinkedIn posts about meetings. I thought it would make sense to reproduce them here for posterity.

Normalize ending meetings early.

If you have scheduled a 1-hour meeting but you are done with your agenda after 30 minutes, there is no need to scramble to think about more things to say. No one will judge you (hopefully) if you just end the meeting at that point and make a more productive use of the remaining time.

Unless you are interviewing a candidate, I guess.

Another post about meetings:

Larry Page, Google founder, famously wrote a memo on effective meetings. It contains the sentence:

“Attendance in meetings is not a badge of honor.”

This has resonated a lot with me. What it means is: when you realize your presence in a meeting is not needed, you find yourself disconnecting or even checking emails on your phone, then leave. Of course, if you know this before the meeting, then you can just decline the invitation. The hard part is getting out while you are sitting in it.

There are a few techniques to do this, such as simply saying sorry and leaving the room. It takes guts (and psychological safety!) to do that though. I know that some people who are on call have paged themselves to have an excuse to leave a meeting – that may or may not have been me at some point :)

Last weekend, at FOSDEM 2023, I watched a Lightning Talk by Frank Vanbever titled “Lua for the Lazy C developer”. I had recently suggested at work that we should be using Lua to script the behavior of some systems which are written in Go, so the talk strongly resonated with me.

Lua is an ideal scripting language for embedding into other programs because it is small and provides excellent bindings in both directions – Lua code can call native code, and vice versa. Because Go has good hash tables as part of the language, using Lua as a container for your hash tables is probably less interesting than from C though.

During the talk, I tried porting the “Hello World” example to Go as the native language. It turns out that Shopify published a pure-Go reimplementation of the Lua interpreter that is byte code compatible with the original C implementation! They use this for scripting the behavior (ha!) of their load testing tools. The package is at github.com/Shopify/go-lua.

It turns out that the example from the talk translates 1:1 to Go and Lua:

package main

import lua "github.com/Shopify/go-lua"

func main() {
	l := lua.NewState()
	lua.OpenLibraries(l)
	l.Global("print")
	l.PushString("Hello World!\n")
	l.Call(1, 0)
}

And it does exactly what you would expect:

$ go build .
$ ./luatest 
Hello World!

The full example is available as a GitHub Gist.

I have been a pkgsrc developer for several years. For what it’s worth, I think pkgsrc is wonderful: a large selection of third-party software, packaged so that it is easy to install with a single command – either building everything from source, or relying on binary packages. pkgsrc supports dozens of OSes – not just NetBSD but also other BSDs, macOS, Linux, Illumos and more.

On the other hand, unfortunately, pkgsrc and NetBSD in general are suffering from what I would call a loss of mindshare. When I joined the NetBSD Foundation as a developer, I had the impression that NetBSD had more users and community than OpenBSD or, say, Dragonfly. In recent years however, it seems to me that people have more or less forgotten about NetBSD and pkgsrc.

Here is where you may come in

Here is a simple idea that occurred to me a while ago and that I finally am getting around to writing up, now that I am sitting in the train to FOSDEM 2023:

These days, a lot of upstream software has a manual with a section on installation that does not give instructions for actually building it. (It seems like building from source is so 1995, or something.) Instead, it goes roughly like this:

If you are using a Mac or a Linux/x86_64 machine, see our own binary packages here. Or use distro packages:

For Debian, run apt-get install somepackage.

For Fedora Linux, run some yum command.

etc. etc.

There is typically a long list of install commands for a bunch of Linux distributions and OSes. pkgsrc is almost never mentioned.

So if you are looking for a simple way of helping out pkgsrc and NetBSD, look for the README of your favorite Open Source tools and add a section for installing from pkgsrc. Typically, for a package from pkgsrc itself, the simplest is to add instructions for using pkgin:

$ pkgin install somepackage

For packages in pkgsrc-wip, you could first ask on the mailing list for an import so that there may be binary packages in the future :) Then, give the typical instructions for installing from source:

$ cd /usr/pkgsrc/wip/somepackage
$ make package-install

I have created a few pull requests for such changes in the past, but I think we need a lot more of that!

It took me quite a while to realize that I have been here before.

When Google+ (the greatest social network I have used, by the way) was killed, somebody spun up Pluspora as a refuge. It was a Pod of something called diaspora. It was nice in the beginning, then gradually I interacted with it less – since, TBH, there wasn’t much content, and I also didn’t post much. Eventually, the person running the Pod died, and their family ended up switching it off.

So when that guy bought Twitter and proceeded to run it into the ground at unprecedented speed, the number of Twitter postings saying “find me on Mastodon @x@y.z” multiplied on my timeline.

Also, at work, we occasionally had a moment where we all shared our gratitude not to work for Twitter. But I digress.

Being a NetBSD dev, it seemed natural to me to register an account on mastodon.sdf.org. SDF was founded as a Public Unix System, giving out free shell accounts to people. They have been running on NetBSD for many years and form a valuable part of the NetBSD community.

The beginning was rough: The SDF Mastodon server had surpassed 15,000 users, just days after breaking through the 10,000 mark. Service was slow, there were lots of errors, and it was generally not super pleasant. However, within a few days, the SDF admins started adding a LOT more hardware to the service, and these days, latency and functionality are top-notch. I think instead of paying $8 for a blue checkmark, that money is better spent with a recurring SDF donation!

The Fediverse

I alluded to this earlier, but diaspora and Mastodon are part of the Fediverse, so they federate. Just like in email, you can follow people from other servers, even using different software. For instance, Mastodon has a 500-character limit for posts, but you could follow someone using Plume for blogging, and their full blog posts would appear right on your timeline. Or I suppose you could follow someone’s PeerTube channel and have their videos appear on your timeline.

This is also how I eventually realized “Mastodon” and “diaspora” are really the same network: in the federated (global) timeline, a number of bots and people I already knew appeared.

Coming over

If you are looking to come over to Mastodon, you probably want to do the following things:

1. Create an account on some server. It’s nice if you like the “theme” or the other people on the same server, since then you can look at the list of all public local posts when you are looking for something to read.
2. Add your new address to your Twitter account – name, bio, link, all are fine.
3. Head over to https://fedifinder.glitch.me and let it find the Fediverse accounts for all the folks you follow on Twitter. You can download a CSV and import it in the Mastodon settings page, making you follow everyone automatically.

Side note

diaspora still exists. Hilariously, if you follow the “Sign up now!” link on their homepage, you eventually arrive at https://diaspora.fediverse.observer/go&software=diaspora, which lists … no open Pods for signup.

I have used a Mi Band as a smartwatch / fitness tracking device for the last couple years. Compared to, say, a Wear OS device, it offers a vastly better deal:

• It’s cheap, at around 30-40 CHF.
• The battery lasts several weeks.
• It shows the time and tracks steps and movement.

The Wear OS device costs 5-10 times as much and needs daily charging. To be fair, in addition to showing the time and tracking fitness data, it does many other things that I don’t need.

My old Mi Band would no longer hold a charge, so I upgraded to the Mi Band 6, the latest iteration. I had read that it would be compatible with Strava, which I like for tracking bike rides. Turns out it is, but only around several corners!

The native app for the Mi Band is called Zepp Life, formerly known as Mi Fit. There is a separate app called Mi Fitness that is completely different (but also supports the Mi Band for some reason?). You can track workouts in Zepp Life but they do not appear anywhere else by default, except (maybe) in Apple Health on iOS. Zepp Life requires a Mi account.

The trick is one more app

You need to connect your Mi account to your Strava account to make workouts sync across the two. But Zepp Life does not support this. However, there is the following amazing workaround:

1. Install the Zepp app (which apparently used to be called Amazfit).
2. Log in to Zepp with your Mi account, the same as in Zepp Life.
3. In the Zepp app, connect the Mi account to Strava by logging in to Strava from Zepp’s “Integration” settings menu.

That’s it! You can now deinstall Zepp, the connection has been established. Now, you can start a bike ride from Zepp Life or from the band, and the following information will appear in Strava:

• GPS trace, speed and elevation – as usual
• Heart rate
• Cadence (I did not expect this!)

Strava will show “Amazfit” as the data source. You can edit the workout after the fact to add comments, description, photos etc.

Thoughts

Fitness tracking seems like a nice thing. I like having the trace of where I went. I like having the comparison to previous rides along the same stretch of road that Strava provides.

What I don’t like is walled gardens. There are so many of them!

Every manufacturer would like to store your fitness data in their cloud. Your past data, the months or years worth of workouts, all this is a source of vendor lock-in that tech companies are all too eager to embrace. After all, why would you buy a different brand (say, an Android phone when coming from iOS) if you cannot transfer over this history?

There are all sorts of neat statistics available that only make sense if all your fitness data is in the same walled garden and whatever device is always on you. Look at how many kilometers you biked in the last month! You were active on this many days! You used this many calories in your activities today! But if you ever walk around without your phone, or without your fitness band, these activities don’t count for the stats.

This way of coupling accounts via some sort of Auth token so that data automatically transfers over is good, but it should not be so difficult and non-obvious to set it up.

Recently, I was thinking about one of my previous software development teams at work. Our program manager was a former Scrum master, so he taught us the basics of the Scrum method, which is one of the Agile development methodologies.

Now, the funny thing was that Scrum includes a number of rituals and techniques, and the book essentially states that you must do all of them or it won’t work; despite that, we used a subset and were somewhat successful with it. We also practiced continuous delivery by accident: because we didn’t use experiments or another way to gate new features, they became available to users as soon as they got merged, with the next daily release.

We decided to do two-week “sprints” (iterations). At the beginning of each iteration, there would be a sprint planning meeting where we decided on the tasks that should be done. Each task (= ticket) was assigned a size in story points, following the Fibonacci sequence, though without the use of cards or other nonsense. Someone just guessed a number, basically, and the others stated if they thought it was too low or too high.

At the end of the sprint, there was a retrospective meeting centered around three basic questions:

1. What went well?
2. What did not go well?
3. What do we want to do differently in the future?

If nothing else, these retrospectives are probably the best thing, and I will do them with any team no matter how they work. Some amount of introspection and thinking about your own performance and that of your teammates is definitely healthy.

Why do any of this?

It took me some time to understand the fundamental reason for following this model. That’s because the literature presents the wrong dichotomy.

In a project driven by software engineers, the natural state (if not following Agile principles) is not the Waterfall. It’s unstructured development.

What I mean by that is that the Software Engineers will work on whatever comes up, or seems important. Someone files a bug? It might just get fixed right away! There is a spontaneous hallway conversation about feature X? Let’s work on feature X then. This “development by Random Walk” seems convenient, but its results are not reliable. Worse, engineers tend to lose motivation over time, in my experience.

In our project, we had stakeholders (well, users) that wanted to see us chip away at the features they needed the most. In some cases, they were literally paying for it by providing headcount to my team. So the solution is to put the tasks with the highest priority firmly into the sprint. Then we assigned an initial task to each person, often by them picking up the thing that they want to work on most.

Micromanagement?

“But wait”, I hear you say, “isn’t this just micromanagement?” Why yes, it is. But crucially, you want to arrive at a point where the individual software engineer micromanages themselves.

The goal is really this:

• to get everyone to agree on a set of tasks that need to be done in a unit of time;
• to get everyone to state what they will be working on;
• and for everyone to actually work on what they said they would be working on.

That last point is crucial.

Non-Fungible Engineers

(Side note: I really wanted to use this heading.)

What did not work in our project was having extra, unassigned tasks for anyone to pick up that has spare cycles. Scrum assumes that every team member can do every task equally well; in practice, that’s just not true.

Every team member has a corner of the code that they are most familiar with. For instance, some folks may know JavaScript (and thus frontend development) better than others. Some folks may understand the monitoring and alerting configs and metrics better than others. Some folks are just plain uninterested in some aspects. Engineers are not fungible in general, in my experience.

Towards reliable delivery

Zooming out from the nitty-gritty of individual tasks: How do you motivate the team to reach a quarterly or yearly goal?

A failure mode that I see often is that a team with n engineers has n goals, one per person. At the end of the quarter, each goal is, say, 2/3 done. The team has worked hard, but nothing has launched. The stakeholders and the team members are frustrated.

What worked for us to get out of this hole was swarming: try to get as many people as possible to work on one set of tasks related to the same goal, finish it and launch. Celebrate the launch, congratulate folks, announce it far and wide. Maybe even give folks some swag, a cash bonus or something. The positive vibes increase everyone’s motivation, and the next thing might go even better!

If you overpromised for whatever reason, at the end of the quarter, you might have some things launched, one or two in flight, and some things not started at all. That’s totally fine. If stakeholders see progress somewhere, they are likely not very upset that the team didn’t get to the thing they wanted. It might just come a bit later.

In our case, we decided on two small features to work on all together. Each took about three weeks. They were the easiest things on our list, but the extra motivation from launching these made the team work much better and taught them how important it is to work together.

I have been running the BulkTracker web app for keeping track of pkgsrc bulk package build results since about 2015.

After running without problems since the start (!!), the BulkTracker app had its first outage in November of 2021. It turns out that the function that renders the home page returns a 500 if it gets an error from Datastore. The error that was returned was:

rpc error: code = ResourceExhausted desc = Quota exceeded.

After complaining on Twitter, I fixed the handler to return at least a degraded home page in case of Datastore errors. But why was there an error in the first place? According to the “Quotas” page on the Google Cloud Console, there are no Datastore-related quotas that can be exhausted – essentially, you can use as much as you need as long as you pay for it. I fixed the handler to return at least a degraded home page in case of Datastore errors.

But why was there an error in the first place? I filed an issue (bsiegert/BulkTracker#26) and paused ingestion for the time being.

After several days, I ended up finding a daily spending limit for App Engine. This setting is deprecated and thus kind of hidden in the UI. It turns out that the storage costs were eating almost the entire daily budget! As soon as one build was ingested in a day, the spending limit would be reached, and Cloud Datastore just fails all requests at that point. Not a good failure mode, if you ask me!

After deleting the spending limit, I watched the situation for a day, then I tentatively re-enabled ingestion of new bulk builds. It turns out that solved it! I have since deleted a bunch of old builds, so that there is not as much stored data.

Unfortunately, I never implemented automatic expiration of results, so they have been accumulating ever since. I built a small tool that deletes the detailed data for the n oldest builds in the database, and I notice it would run fairly slowly: deleting a set of 20,000 records would take about 3 minutes. This seems like a lot. The next post in this series will talk about how I made this faster.

Background: Datastore

As an App Engine application, using the App Engine datastore (a NoSQL database) was a logical choice at the time. A SQL database may have been more appropriate for the workload, as I later found out, but the pricing for Google Cloud SQL is prohibitive for a small application.

Small side note: Through my job I learned how App Engine Datastore was actually implemented, and it was horrifying.

Over the years, App Engine Datastore became first Cloud Datastore – an offering independent from App Engine that can be used with any Cloud application – then something called “Cloud Firestore in Datastore Mode”. Firestore is the Firebase NoSQL database; its API is very different from the Datastore one, but there is a shim layer that provides the Datastore API on top of it. The migration was transparent and flawlessly done by the Google Cloud folks – kudos.

Some Statistics

Most of the space in the BulkTracker datastore is used by indices, since all queries must use an index. So in early December, before I started deleting old data, the storage contained:

• data for 8800 bulk builds
• the oldest build was from June 2015
• total size of the data itself was about 30 GiB in 165,000,000 records
• total index size was another whopping 300 GiB

The BSD build system in general, and pkgsrc in particular, have a large number of Makefiles ending in .mk.

Recently, I was looking at a commit message in Gmail and noticed that these names are linkified. At the time, I was looking at a Go module package, where there is a go-modules.mk file containing details about dependencies. This got me thinking: Why is this file name turned into a link?

It turns out that .mk is the ccTLD of the Republic of North Macedonia! So I did what I had to do: I went to the website of a registrar in Skopje and reserved the go-modules.mk domain.

For the DNS, I created a zone in Google Cloud DNS, which was simple and extremely cheap, on the order of a few cents per month. For the contents, I added the domain as a redirect to my existing blog on the Firebase console.

So there you have it: next time, you see one of those emails, clicking the file name will bring you directly to this blog.

I have not checked if other standard Makefile names are still free as domains. So if you are interested in putting your pages at bsd.prog.mk or similar, here is your chance! :D

This weekend, I made a series of somewhat unusual changes to pkgsrc.

I removed a bunch of Go packages.

Why? Because of Go modules.

What are Go modules?

Since my series of design-ish blog posts(part 1, part 2), Go module builds have fully landed in pkgsrc, to the point that they are now the preferred way to build Go packages.

To recap: There are two ways to use the go tool to build Go code.

• The old way is to have a tree, below $GOPATH, that has all the dependencies in a directory tree according to their import path. For instance, the golang.org/x/net package would be placed in a $GOPATH/src/golang.org/x/net directory. This is what lang/go/go-package.mk implements in pkgsrc.
• The new way is to extract the source code wherever you want, just like any C source. The top-level source directory contains a go.mod file that specifies dependencies and their versions. The go tool then downloads a bunch of .zip and .mod files for those dependencies and unpacks them as needed. This is similar to how Cargo works for Rust code.

Like with Rust, in pkgsrc, we specify a list of dependent module files to be downloaded from the module proxy.

In actual practice, a useful pattern has emerged, where the list of modules is in a separate file named go-modules.mk in the package directory. To create or update the file, simply run

$ make patch
$ make show-go-modules > go-modules.mk

and then .include the file from the main Makefile.

But why remove all these packages?

A pkgsrc package built with lang/go/go-module.mk does not install any source code or .a files. Only the binaries are packaged, just like for C. Go packages that just correspond to intermediate libraries and do not contain any useful binaries are simply no longer needed. They can be deleted as soon as nothing depends on them any more.

In particular, I changed all the packages depending on golang.org/x/tools to be modules, then migrated the go-tools package itself. go-tools depends on a number of other libraries that nothing else depends on.

By the way, it is fairly simple to make a non-module into a module, even if the source does not contain a go.mod:

1. Change go-package.mk to go-module.mk.
2. Run make patch and change into the top-level source directory.
3. Run go mod init github.com/foo/bar or whatever the import path is.
4. Update the file with go get and/or go mod tidy.
5. Copy the generated go.mod and go.sum files to a files directory and copy them into place in pre-patch.

Future

Some future Go release will deprecate GOPATH builds, so we must convert all Go code in pkgsrc to modules at some point. By the way, if upstream has not made the jump to modules yet, they might be happy about your pull request :)