It has been a bit over a year since I wrote Some nuanced thoughts on AI. Much of what I said then still stands.

On the other hand, the AI models have continued improving. Even more so, the “harness” running it has gone through multiple iterations. Back then, the main mode of interacting with an LLM during coding was through autocomplete. The LLM would guess what you wanted to write and propose some text. That UI paradigm is gone now.

The major game changer has been making it agentic. Yeah, I can hear you groan, dear reader. But the LLM can now run tools: it can build stuff. It can run tests. It can ingest test failures and use that for improvements. Most importantly, it can search documentation and can run CLI tools, e.g. for your framework.

The current UI paradigms that I see are:

• a sidebar in an IDE, typically based on VS Code. This is what Antigravity does. IMHO, this pattern is on the way out, as you need to make fewer edits to LLM generated code, and you can make them by prompting.
• a conversation view that can occasionally branch out to show some diffs, or a design doc. Choose your medium: TUI in the terminal, web app, or chatbot right in your Slack or similar.

The last one is what I am now working with (at my dayjob).

With the ability to run tools comes a big security problem: you don’t want the LLM to take down production because it has decided that your deployment is fucked. You don’t want it to be able to delete a database, or all those other things your account can do. What does not work is stopping every couple of seconds and asking the user to approve a command. You’ll get tired of it and simply end up allowing everything. Or the LLM goes around your guardrails, e.g. by writing a Python script that does a dangerous thing.

At work, we have solved (solved-ish) the issue by giving everyone’s LLM agent its own user ID, clearly marked as not-human. The harness runs in an ephemeral VM. The user and the VM are more limited in their access. Code produced by the LLM has to be reviewed by two humans, typically the owner of the bot and a second person. This is the modern version of the “four eyes” principle that apparently has something to do with compliance with the Sarbanes-Oxley act.

So how has my job changed? If you had told me at the beginning of this year that I would be writing almost no code by hand at work, I would not have believed you.

Anyway, I now write almost no code by hand. I still do quick edits, config changes and the like by hand – but mostly in vim in the terminal. Do I mind? No. As a Staff Engineer, I have always been reviewing 10x more code than I’d been writing. So I don’t mind reviewing code. Reviewing code is important.

I have been involved with packaging and patching software (in pkgsrc) and with large scale changes at work, so only having superficial knowledge of a code base is also fine for me. It helps that my main language is Go, which (a) LLMs are very good at and (b) usually does exactly what the code says.

A positive change that I do see: “less important” coding tasks actually get done! I see quarterly goals that sat at “we won’t have time to do this” get to completion in time, because someone can churn out that code “in the background”, or on a calm Friday afternoon. I see tech debt actually decreasing because you can let the LLM tackle those tech debt reduction tasks.

I am still pretty uncomfortable with the AI maximalists at work that want to upend the entire development lifecycle: agents review code produced by other agents with no human involvement; agents writing tickets for other agents that autonomously design a feature. YOLOing your deployment because you “don’t have time” to do it properly. That way still lies madness.

Do I use LLMs for coding in my spare time? Mostly no. I can ask Gemini to write me a complex SQL query, and it does a much better job than I do. I tried feeding it some hard-to-debug crash, and it came up with a plausible theory why it happens and a patch. But the patch did not solve the issue, which is a bummer.

Also, somehow, my motivation for computing in general has decreased. Should I spend the time learning Rust properly? Does it matter anymore?

I don’t know if people are still genuinely enthusiastic about tech. It feels like tech is not sparking joy any more.

And don’t get me started about the prices. A Raspberry Pi 5 with 16G of RAM is 260 CHF. 260! You used to be able to buy more computer for that amount of money. The RAM price madness is ostensibly because of AI, but in reality it’s more about the various people involved lining their pockets, at the expense of all of us.

A while ago, I made an off-hand remark that got some followup questions. I said that in AI applications, Model Context Protocol (MCP) is no longer a favored extension model for tools. The question was essentially – Really? What to use instead?

What is the problem with MCP?

MCP is a chatty protocol. Connecting to any MCP endpoint will hand the AI model a full description of all the possible calls, including some explanatory text and parameters.

The problem with this is that it consumes a lot of tokens. An LLM has something akin to a working memory, called the context window. Adding several MCPs, together with the system prompt, might consume most of the context window already. What tends to happen then is that the LLM forgets about earlier stuff, e.g. what’s in the system prompt. Suddenly, it might start talking to the user in a foreign language, for instance.

Are native tools better?

Agentic development frameworks such as Google ADK let you add tools natively by writing code. This lets you get around some of the high token usage issues.

You can also scale up the number of usable tools by making more specialized sub-agents. The main agent can decide which sub-agent should do the task and hand it off. The A2A protocol (agent to agent) enables this. A2A is usually a native feature of these agent frameworks.

Fundamentally, if you want your system to be able to do a lot of different things, writing these specialized tools for every purpose does not scale well. At work, we have found an approach that works suprisingly well for doing “arbitrary” things though.

“Meta” tools to the rescue

You can create an agent that can do a large variety of tasks with a very narrow subset of tools:

1. Command line interfaces for all the integrations that you would like. For example, a CLI to query the bug tracker, one to send emails, etc.
2. Documentation for all the required CLI tools, ideally in Markdown.
3. An agent tool to search the corpus of documentation.
4. An agent tool to run shell commands.

With this combination of things, LLMs are pretty capable of formulating a search term, reading the results and calling the tool. I think this is because shell syntax is part of their training set.

If the LLM has the capability of creating a shell or Python script and running it, it can do actual computation in this way, e.g. for grepping over a result or calculating “2+2” by asking the Python interpreter.

The amazing thing about points 1 and 2 is that CLIs and docs are super useful to humans too! So by making a system logical, easy to understand and well documented, you are making it easier for the LLM and for the human. Win-win.

I recently spent a week in London, as one does. As I walked through the King’s Cross train station, it struck me: if anything, it should be track 8 3/4 and not 9 3/4. Let me explain.

In case you have been living under a rock: in the Harry Potter book series, the protagonists run through a wall to reach the “hidden” platform 9 3/4, where the Hogwarts Express departs.

In real life, King’s Cross consists of a large, magnificent main hall housing tracks 0 to 8. Yes, numbering starts at 0. Then, there are tracks 9 and 10. They are a bit shorter and live in their own hall sharing a wall with the main building. I assume that these two tracks have been added later.

In order to get to tracks 9 and 10, you have to exit through the fare gates, walk through the station hall – a recent construction with a large round roof – and re-enter through a different set of fare gates.

Next to track 9 and 10, there is a Harry Potter shop selling souvenirs and a photo point that has a perpetual line of people waiting. There is half a luggage cart stuck to the wall, they hand you a scarf and a wand, and your friends take a photo of you pretending to be a Hogwarts student. Cringe.

But the thing is this: we all agree (I hope) that track 9 3/4 should be located between 9 and 10. But tracks 9 and 10 share a common platform with no wall whatsoever! The wall is between track 9, in the smaller hall, and track 8 in the big hall. Thus, the Hogwarts Express should really be departing from track 8 3/4.

My email inbox is like the pile of documents on my desk. Things that I wanted to get back to ends up moving towards the bottom, into the never-ending pile of … stuff. For the first time in a while, I have looked at the bottom – and found an inquiry from someone who had seen my presentation at FOSDEM 2024.

They had a question for me, which I am going to paraphrase below. I am going to reproduce my answer here because it may be interesting for others.

FreeBSD is apparently considering incorporating Rust code into the kernel, in order to appeal to a younger developer audience. Will NetBSD do similarly odd decisions in the future, or can I trust them to remain conservative in this regard?

First of all: I’m not so sure that Rust in the kernel would have been chosen to cater to a younger developer audience. It’s probably more on merits such as memory-safety. Rust has a number of fans who are power users. I would be interested to learn why you think this is an odd decision.

NetBSD already has taken a much odder decision in the same space, by adding Lua into the kernel! This was also much discussed at the time. I have not used it, but apparently Lua is useful for rapid development and prototyping of kernel drivers.

Note that in general, NetBSD is more conservative in its technical decisions than FreeBSD or Linux. Rust in the core of NetBSD is probably a non-starter for multiple reasons:

• There are many architectures that NetBSD supports where Rust is not available. This is probably the most important argument against Rust.

• Today, even getting a Rust compiler running in the first place is hard. Keeping Rust working (in pkgsrc) is quite a bit of work, resting on the shoulders of a few developers.

• In general, the bootstrap relies on a binary package of the previous version. This is unacceptable for an otherwise source-only, self-contained distribution like the NetBSD sources.

• For Rust to be used in base or in the kernel, the compiler would also have to be part of the base system. This means adding a lot of code and maintenance. It would not be impossible though – we already have LLVM (a major dependency) in base. But again, the binary bootstrap kits are a significant hurdle.

• Finally, the release cycles of Rust are not compatible with the NetBSD ones. We support the last two major releases. Today, that’s NetBSD 9 and 10. NetBSD 9.0 was released in 2020. Rust 1.41 was the current version back then. If Rust 1.41 had been part of NetBSD 9, it would be useless for anything except compiling NetBSD itself – Rust 1.41 is so old that basically no modern code would compile with it. Not great.

Last week, I visited the Bay Area for a business trip. Most of my time was spent in the South Bay, but I also spent a day in SF.

In the afternoon, I went to Ocean beach and dipped my feet in the Pacific Ocean for a tiny bit. The water was pretty cold! Where I live, we have beautiful landscapes but the country is landlocked, so going to ocean shore is special for me.

After walking along a beautiful path over the cliffs for a while, it was time to go back to the city center. I chose a Muni bus, line 1 as it happens. The bus mostly goes in a straight line, but it takes a long time as there are many stops. But I saw something fascinating happening on that bus. Let me explain.

The prevailing narrative in the media about SF is that it’s somehow “dead” and a crime-ridden hellhole, and that there are homeless people everywhere. And I can kind of see how you would get this impression if all you do is drive a car and spend time in the financial district.

But in this bus, I saw a different kind of San Francisco. I saw a city that is busy and thriving. I saw a slice of normal people living normal lives.

Most of all, I saw kindness. Many of the seats were reserved for disabled and senior people. Every time an older person came in, immediately someone got up and offered their seat. Someone came with a bag full of groceries and got help carrying it.

Most endearing of all, a little girl was taking the bus with her dad, and a lady made her press the stop button so that she could get out. She turned it into a fun little game and made the girl understand that she can take actions that have an actual consequence – stopping the bus in this case. The girl was delighted.

These residential neighborhoods – Richmonds, Nob Hill, etc. – look like great places to live, with great people. From what I saw, San Francisco is doing a lot better than you might think. And the people are much more sane than the distorted image that Social Media is showing you.

It has been a good reality check.

I have a Raspberry Pi 3 with NetBSD 10, running CI jobs. Because SD cards are notoriously unreliable, I attached a USB hard drive to it. The HDD has a swap partition and scratch space for the builds, while root is on the SD. Unfortunately, some writes end up going to the root file system after all, which meant that the SD card was destroyed after only about a year!

So I set out to put the root file system onto the HDD as well, only using an SD card for loading the kernel.

This is an RPi3 running in 32-bit mode with the “classic” RPi bootloader, no UEFI. So it will load config.txt from the FAT partition on the SD card and load the kernel that’s in there. In theory, the cmdline.txt file is supposed to give the root file system, but somehow, changes to this file had no effect (???). But we can force the kernel to use a certain root file system location!

Some theory: disklabels, partitions, wedges

There are two partitioning schemes in use, MBR and GPT.

In MBR partitioning, there is one MBR partition of type NetBSD that contains a disklabel. The disklabel permits splitting the space further, into NetBSD partitions. These are identified by a letter – for example, the a partition on disk ld0 is /dev/ld0a. Usually, a is root, b is swap, and c is the entire disk. As an example, here is the disklabel of my RPi’s SD card:

8 partitions:
#        size    offset     fstype [fsize bsize cpg/sgs]
a:   2459648    196608     4.2BSD      0     0     0  # (Cyl.     96 -   1296)
c:   2656256         0     unused      0     0        # (Cyl.      0 -   1296)
e:    163840     32768      MSDOS                     # (Cyl.     16 -     95)

In GPT partitioning, there is no disklabel. Instead, there is just a list of partitions. NetBSD partitions, swap, EFI, etc. are directly created as partitions. Partitions have a GUID (that’s why it’s called a GUID partition table) and optionally a label. For example, this is the GPT of the RPi’s hard drive:

# gpt show -l sd0
      start        size  index  contents
          0           1         PMBR
          1           1         Pri GPT header
          2          32         Pri GPT table
         34       32734         Unused
      32768      163840      1  GPT part - 
     196608   838860800      2  GPT part - rpi3-root
  839057408   134217728      3  GPT part - rpi3-swap
  973275136   980249999      4  GPT part - scratch2
 1953525135          32         Sec GPT table
 1953525167           1         Sec GPT header

As an additional complication, the kernel will create wedges for each GPT partition. These are numbered dk devices. Numbers are assigned in the order in which partitions are discovered:

[     5.463320] dk0 at sd0: "8e1462a7-d2f5-431c-a8f8-9165d455f631", 163840 blocks at 32768, type: msdos
[     5.473321] dk1 at sd0: "rpi3-root", 838860800 blocks at 196608, type: ffs
[     5.483322] dk2 at sd0: "rpi3-swap", 134217728 blocks at 839057408, type: swap
[     5.483322] dk3 at sd0: "scratch2", 980249999 blocks at 973275136, type: ffs

As you can probably imagine, relying on these wedge numbers is brittle: today, /dev/dk0 might be the root, tomorrow you might have some other device plugged in on boot, so dk0 is something else. Thus, NetBSD allows you to refer to them by name (= label) – yes, the terminology is inconsistent. The syntax for this is NAME=somelabel and gets resolved to the correct dk device. So your /etc/fstab might contain lines like:

NAME=rpi3-root	/		ffs	rw	1 1
NAME=rpi3-swap	swap		swap	sw,dp

How the kernel finds the root FS

But wait! fstab is only active once the system is booting. Before that, the kernel needs to find the correct root filesystem. By default, it will guess! This is controlled by a line in the kernel configuration file. The GENERIC kernel contains this bit:

config netbsd root on ? type ?

This means that a kernel with the name netbsd will be built, and it will probe for the root FS location and type. You can also provide a location on the kernel command line, by adding something like root=NAME=rpi3-root.

As said above, this didn’t work for me. So I looked into baking the location in to the kernel. The kernel image contains what is called a rootspec, with a syntax that is almost but not quite the one for the root= command line parameter (of course!). You can set the rootspec in the kernel config.

I discovered that dk devices are not allowed in the kernel config – sensible, since their number can change. (Note that the kernel itself supports this.) But how to get a name in there?

According to the cpu_rootconf(9) manpage, the rootconf can be the prefix wedge: followed by the name of the disk wedge. And config supports quoted strings, which are directly passed to the kernel, as root.

Putting it all together

Inside of the NetBSD source tree, I created a new file, sys/arch/evbarm/conf/MYKERNEL. The filename can be anything you want. In it, I just override bits from GENERIC:

include "arch/evbarm/conf/GENERIC"

no config netbsd
config netbsd root on "wedge:rpi3-root" type ffs

The “no config netbsd” line removes the existing root config from GENERIC, so we can have a different one.

You compile this kernel on a machine of your choice by running

$ ./build.sh kernel=MYKERNEL

resulting in netbsd, netbsd.img and netbsd.ub files. AIUI, the file with no extension is used with UEFI, the .ub one with the U-Boot bootloader, and the .img one is for the RPi bootloader. So I copied the .img file to the boot partition, set its file name in config.txt and voilà! The system now boots from the partition on the HDD.

I have stuggled with writing this for a while. This has the potential to be controversial: Some people have very strong opinions on this, while my own opinion is more … nuanced. –

I live in two different worlds regarding AI.

During the day, I am working in a big tech company that is betting big on AI. AI is everywhere, and (at least some) people are excited about the possibilities.

Outside of work, when I log in on Mastodon, the typical poster there thinks that anyone using AI for anything should be ashamed, and that AI is unequivocally bad.

Going with the times

In Tech, things change all the time. If there is some new development that you don’t like, you might decide to “boycott” it – however, you risk looking very silly down the line when the thing you despise is just normal, while you made disagreeing with it your whole personality. For example:

• In the late 90s, some people were very upset about computers no longer having a floppy drive, or only supporting USB for peripherals, so they swore that they would never buy that new-fangled stuff. Try finding a laptop with a floppy drive today.
• Also in the 90s, you might have decided to ignore that new-fangled “internet” and “WWW” thing. Anyone can edit Wikipedia, right? That clearly makes the printed encyclopedia in the shelf much more useful!! You would have missed a lot of exciting stuff in tech in the following decades.
• In the 2010s, you might have decided that “There is no cloud, only other people’s computers”, but you would have missed out on a number of cool innovations. (That’s another post for the future.)
• On the other hand, in the early 2020s, you might have decided early on that NFTs are stupid, and you would have been right.

So what about AI? I feel the jury is still out, overall. There are aspects of the hype that are clearly stupid. There are also AI-related things that are useful. I feel that dismissing everything containing AI means depriving myself of some useful tools. Not using the best tools is going to mean that I cannot do my best work.

Maybe that’s the FOMO argument. But it is also the “lifelong learning” argument, particularly in a professional context.

Example: Coding assistants

Coding assistants can be a giant productivity boost. At Google and Microsoft, statistics claim that up to half of coding keystrokes are “autocompleted” from an AI assistant. I can believe it! Never mind that you sometimes end up deleting most of what the assistant has written because it has misunderstood what you wanted to do.

Some situations in which a coding assistant is helpful for me:

• Framework boilerplate. For instance, I recently ported a Go command-line tool to Cobra. The assistent could spit out Command structs and Cobra registration functions based on the other code in the file. I would imagine that JS frameworks work really well for this too.
• The boring parts of the code. “Create an argument parser supporting the -f, -d and -h options”. This, too, is kinda boilerplate.
• You can narrate your code by using comments like subheadings. You write something like // Connect to the server. and let the assistant propose a dozen lines.
• You can ask the chatbot for ideas on how to approach something, like “How do I make the JS on this page communicate back and forth with the server?”, let it suggest a websocket and make a skeleton implementation of the two sides.

What these things have in common: I want to understand the code that the assistent generates. And I have the last word and proofread everything. Typically, there are some bugs in what is generated. I don’t like the Vibe Coding approach, where I just let the machine make random edits to make errors go away.

If – and only if – you are the one that stays in control, an AI assistant can be like “a bicycle for the mind”.

Don’t buy the hype

While all of the above is pretty positive, there are many aspects of the AI hype that I really do not like. Consider “slop” – the endless stream of low-quality, uninspiring content created from LLMs, image generators, and more. Grifters seem to really like those.

Then there is the societal argument about the consequences of large-scale deployment of AI. There is clearly an effort by some to replace knowledge workers with unaccountable machines.

You apply for a credit card, the AI declines your application, and there is no way to appeal.

Translators and linguists are laid off because “AI does the job well enough” (cf. Duolingo).

Personally, I find the endless boasting about GenAI in particular just boring. Half the posts on my LinkedIn now are people leaving their jobs (which is expected), while the other half is discussing generative AI. Are there no other topics?? But look at me writing this, I am part of that problem now :/

Conclusion

As said above, I think the jury is still out. It is probable that the AI bubble will burst at some point, with painful consequences for the entire tech industry. It is also probable that GenAI of some sort will be here to stay. So in my opinion, the best option is to find tools that are useful but do not believe all the hype.

I stopped developing pkgsrc with CVS.

Quick bit of background: NetBSD is still using CVS as its version control system. The decision to move to something else has been taken long ago, but the switch has not happened as of today.

Working with CVS is painful for many reasons. For instance, there is no way to see your local changes without waiting several minutes for a cvs up -n. A full tree update (cvs up) churns for quite a while before it even starts updating any files.

I met Taylor (riastradh@) last year, and he told me about his git-based workflow. I must say I have become a convert! I use the GitHub mirror as my main source tree. Yes, it adds GitHub (and thus Microsoft) as an intermediary, but I don’t mind. Below, I am going to describe my workflow and play through an example.

My setup

I have two pkgsrc source trees:

• a CVS checkout at ~/pkgsrc-cvs, based on the writable repo and only used for writing
• a git checkout at ~/pkgsrc

The git checkout is relatively quick when skipping history:

$ git clone --depth 1 https://github.com/NetBSD/pkgsrc

The first thing is to create a local/${hostname} branch. Every time I rebase this branch onto the trunk, I run pkgrrxx -u to update all packages to that state. This gives me a stable base from which to work. For work on packages, I create a new branch (e.g. update/${pkgname}) and delete it when done. Or I do the change directly in the local branch if I am lazy :)

For committing to CVS, I use the git-cvsexportcommit script from the devel/git-perlscripts package.

Worked example: a package update

Here, I am updating the net/gh package (the GitHub CLI) to a new version. Once I have made my changes, I commit:

$ git commit -a
[local/fedorakumori 668ebe1fa] gh: update to 2.68.1
 3 files changed, 473 insertions(+), 509 deletions(-)

Note the commit hash; we will use it for exporting the commit.

$ cd ~/pkgsrc-cvs
$ GIT_DIR=~/pkgsrc/.git git cvsexportcommit -v -c 668ebe1fa
Applying to CVS commit 668ebe1fa32d778ce19fe81bc3529f15488a048d from parent 7fa1acd873b88c6b36796133c7b692023a611f61
Checking if patch will apply
Applying
error: patch failed: net/gh/Makefile:2
error: net/gh/Makefile: patch does not apply
error: patch failed: net/gh/distinfo:1
error: net/gh/distinfo: patch does not apply
error: patch failed: net/gh/go-modules.mk:1
error: net/gh/go-modules.mk: patch does not apply
cannot patch at /usr/pkg/libexec/git-core/git-cvsexportcommit line 338.

There has been some other commit in pkgsrc since my last sync. To resolve, let’s first switch over to a new branch, based on the trunk:

$ cd ~/pkgsrc
$ git checkout trunk
$ git pull
$ git checkout -b update/gh
$ git cherry-pick 668ebe1fa
Auto-merging net/gh/Makefile
CONFLICT (content): Merge conflict in net/gh/Makefile
Auto-merging net/gh/distinfo
CONFLICT (content): Merge conflict in net/gh/distinfo
Auto-merging net/gh/go-modules.mk
CONFLICT (content): Merge conflict in net/gh/go-modules.mk
error: could not apply 668ebe1fa... gh: update to 2.68.1
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
$ cd net/gh
$ git status
On branch update/gh
You are currently cherry-picking commit 668ebe1fa.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Unmerged paths:
  (use "git add <file>..." to mark resolution)
        both modified:   Makefile
        both modified:   distinfo
        both modified:   go-modules.mk

no changes added to commit (use "git add" and/or "git commit -a")

Now I open the three files in my editor and merge, thanks to my years-long routine of doing this. Confusingly, the messages above show that there are two ways to continue the cherrypick: either with git commit or with git cherry-pick --continue. As far as I can tell, they do the exact same thing.

So, after resolving the conflict and running git commit -a, I have a commit with a new hash:

$ git log | head -1
commit 2b51308c74912f41974d2267f58bb35987bde7e8

Of course, it is a good idea to build the package again at this point and verify that it still works.

Let’s commit this one to CVS now. You don’t need to copy the whole hash, just the first 6-8 characters.

$ cd ~/pkgsrc-cvs
$ GIT_DIR=~/pkgsrc/.git git cvsexportcommit -v -c 2b51308c7
Applying to CVS commit 2b51308c74912f41974d2267f58bb35987bde7e8 from parent 44a7067c8ab30c8ec0e58668ec86ed0edaaba1de
Checking if patch will apply
Applying
Patch applied successfully. Adding new files and directories to CVS
Commit to CVS
Patch title (first comment line): gh: update to 2.68.1
Autocommit
  cvs commit -F .msg 'net/gh/Makefile' 'net/gh/distinfo' 'net/gh/go-modules.mk'
/cvsroot/pkgsrc/net/gh/Makefile,v  <--  net/gh/Makefile
new revision: 1.90; previous revision: 1.89
/cvsroot/pkgsrc/net/gh/distinfo,v  <--  net/gh/distinfo
new revision: 1.45; previous revision: 1.44
/cvsroot/pkgsrc/net/gh/go-modules.mk,v  <--  net/gh/go-modules.mk
new revision: 1.39; previous revision: 1.38
Committed successfully to CVS

The change is now live in CVS. The normal thing to do now is to add to the changelog. I do this directly in CVS for simplicity.

$ cd ~/pkgsrc-cvs/net/gh
$ bmake changes-entry && bmake changes-entry-commit
=> Updating CHANGES-2025 and TODO
P CHANGES-2025
=> Adding the change
=> Committing the change
/cvsroot/pkgsrc/doc/CHANGES-2025,v  <--  CHANGES-2025
new revision: 1.2125; previous revision: 1.2124

This works well for cross-cutting changes, like revbumps, too.

Why

In the beginning, I alluded to some of the pain points with CVS. But this also enables me to do some new things. In CVS, working copies are expensive, so you use the same one each time. It has happened to me more than once that I committed something I did not intend to because I had the change lying around in my tree, uncommitted.

Uncommitted changes are liabilities.

In a DVCS tree, you can commit everything right away and upstream when you are ready to. This means that you write change descriptions while you still remember what you did :) For a cross-cutting or independent change, you can just “reset” your tree to a clean state in a few seconds. This is a great thing.

My current work project started last summer, as a bit of experimentation. A few of us sat together in a room and started writing down a hypothetical piece of configuration. Within less than a week, we had actually written a prototype-quality piece of software accepting exactly the configuration we had brainstormed.

A few months later however, this project went through a difficult phase where we realized that we actually needed to write down a plan for bringing the software to a stable, usable state. This ended up a painful couple of weeks of planning ahead and realizing that our goal of shipping this was slipping.

I had a conversation about this with my Engineering Director the other day, and I thought what he told me was intriguing. He said that we were struggling with going from what he called “founder mode” to an actual planned development process.

As an analogy, imagine you are painting a wall green. In the beginning, you can just put paint wherever you like. After a while, you can make a break and proudly exclaim that you already painted 30% of that wall. But sooner or later, you will have to check carefully where there is no paint yet. And you will need to go with a smaller brush, so that you can paint the corners properly.

In software development, some people can only do the first phase. They have awesome ideas and they can churn out an early version in record time, maybe even working nights. But they are not good at doing the other 90%: bug fixes, support, identifying areas for improvement, and continual maintenance.

Some other people can only do that second phase. They are very good at maintaining something, sometimes for years, but they might not have the sort of idea to jump-start something completely new. Of course, that’s perfectly fine.

And some people (hopefully) can do both, but they might need to be reminded when it is time to switch to the refinement phase :)

As a leader, if you have people on your team who are the perpetual “founders” (startup, open source, big tech, it doesn’t matter), there is a trick you can pull to keep development going while not antagonizing that “10x engineer”. The trick is to get them interested in some new, shiny problem at the right moment. They might fall in love with solving the next problem and come back wanting to hand off their current thing to someone who will finish building it. And thus, they can do their magic one more time. Even better is when the people who are part of this rochade don’t even notice it happening. –

… Now that I wrote the previous paragraph, I actually wonder if one of my leads has done that with me before, and if so, how many times it has happened!

This is part 3 of my blog post series about emulating BSD operating systems for 32-bit ARM with QEMU. Buckle up, today we will need to do an actual OS installation!

In OpenBSD/armv7, the miniroot image is an installer, so we also need a new empty drive image to install to. I recommend the qcow2 format, since it consumes only the space that is actually occupied. The 10G image created below is only 192 kilobytes initially. Here is how you create the root image:

qemu-img create -f qcow2 root.qcow2 10G

Next, we will need the pflash0.img and pflash1.img files created in the previous part.

As for the install image, there are several choices on the download page for OpenBSD 7.6/armv7. I went with miniroot-cubox-76.img but I don’t think it matters much which one you choose. They only differ in the platform-specific bootloader, and we will be booting from UEFI instead anyway.

Note that when booting, you might get an error message complaining that the size of the image is not a power of two. If that happens, just resize it to the next bigger one:

qemu-img resize miniroot-cubox-76.img 64M

Running the installer

Here is the command line to launch the installer then:

#!/bin/sh
qemu-system-arm \
	-M virt \
	-m 1024 \
	-nographic \
	-drive file=miniroot-cubox-76.img,format=raw \
	-drive file=root.qcow2,format=qcow2 \
	-drive file=pflash0.img,format=raw,if=pflash,readonly=on \
	-drive file=pflash1.img,format=raw,if=pflash \
	-device virtio-gpu-pci \
	-nic user,model=rtl8139

This time, we don’t have to type anything in the UEFI shell, it loads the bootloader directly, followed by the OpenBSD installer! At the boot> prompt, just press Enter.

I won’t go over the steps to install OpenBSD itself. The only thing to keep in mind is that OpenBSD calls the boot image (the miniroot) sd0 and the root disk we created sd1, so be sure to install onto sd1.

As always, you can get out of QEMU and back to the terminal by “powering down”, e.g. running halt -p from a shell.

After installation

To boot your newly installed system, use the exact same QEMU command line as before, except that you remove the miniroot disk entry:

#!/bin/sh
qemu-system-arm \
	-M virt \
	-m 1024 \
	-nographic \
	-drive file=root.qcow2,format=qcow2 \
	-drive file=pflash0.img,format=raw,if=pflash,readonly=on \
	-drive file=pflash1.img,format=raw,if=pflash \
	-device virtio-gpu-pci \
	-nic user,model=rtl8139

A dmesg output of this freshly installed system can be found here. Notably, it looks like the virtio GPU is not supported, so I am not sure if you can get X11 to work.

When I did this installation, a short time after the OpenBSD 7.6 release, there were no binary packages avaiable for 32-bit ARM. But it looks like this has changed! There is a full set of packages in https://cdn.openbsd.org/pub/OpenBSD/7.6/packages/arm/ now, so that a command like

pkg_add vim

will just work. That’s nice!

Some thoughts on machine types and emulation

When I started fiddling with this, I tried to get qemu to emulate a real machine corresponding to one of the OpenBSD miniroot image. But that’s fairly limiting: these boards came with too little RAM, too slow storage, too few cores, all of it.

The nice property of an emulator like QEMU is that you can kind of build your own machine from scratch. The virt machine type is a generic thing that you can build out just like you want it. Want 8G of RAM? More cores? No worries!

You can add a bunch of virtio devices that are virtualization-aware; if one of them doesn’t work right, replace with some other emulated hardware. This is also why my command lines above use an rtl8139 network device instead of a virtio-net.

And for booting this contraption, using the EDKII UEFI firmware allows just booting a generic kernel and bootloader, which is supported almost everywhere.

And that concludes my series of blog posts on emulating ARM systems with QEMU. I hope you enjoyed it!

For reference, here are links to the other parts in the series:

• Emulating *BSD on ARM, Part 1: Introduction
• Emulating *BSD on ARM, Part 2: FreeBSD