I stopped developing pkgsrc with CVS.

Quick bit of background: NetBSD is still using CVS as its version control system. The decision to move to something else has been taken long ago, but the switch has not happened as of today.

Working with CVS is painful for many reasons. For instance, there is no way to see your local changes without waiting several minutes for a cvs up -n. A full tree update (cvs up) churns for quite a while before it even starts updating any files.

I met Taylor (riastradh@) last year, and he told me about his git-based workflow. I must say I have become a convert! I use the GitHub mirror as my main source tree. Yes, it adds GitHub (and thus Microsoft) as an intermediary, but I don’t mind. Below, I am going to describe my workflow and play through an example.

My setup

I have two pkgsrc source trees:

• a CVS checkout at ~/pkgsrc-cvs, based on the writable repo and only used for writing
• a git checkout at ~/pkgsrc

The git checkout is relatively quick when skipping history:

$ git clone --depth 1 https://github.com/NetBSD/pkgsrc

The first thing is to create a local/${hostname} branch. Every time I rebase this branch onto the trunk, I run pkgrrxx -u to update all packages to that state. This gives me a stable base from which to work. For work on packages, I create a new branch (e.g. update/${pkgname}) and delete it when done. Or I do the change directly in the local branch if I am lazy :)

For committing to CVS, I use the git-cvsexportcommit script from the devel/git-perlscripts package.

Worked example: a package update

Here, I am updating the net/gh package (the GitHub CLI) to a new version. Once I have made my changes, I commit:

$ git commit -a
[local/fedorakumori 668ebe1fa] gh: update to 2.68.1
 3 files changed, 473 insertions(+), 509 deletions(-)

Note the commit hash; we will use it for exporting the commit.

$ cd ~/pkgsrc-cvs
$ GIT_DIR=~/pkgsrc/.git git cvsexportcommit -v -c 668ebe1fa
Applying to CVS commit 668ebe1fa32d778ce19fe81bc3529f15488a048d from parent 7fa1acd873b88c6b36796133c7b692023a611f61
Checking if patch will apply
Applying
error: patch failed: net/gh/Makefile:2
error: net/gh/Makefile: patch does not apply
error: patch failed: net/gh/distinfo:1
error: net/gh/distinfo: patch does not apply
error: patch failed: net/gh/go-modules.mk:1
error: net/gh/go-modules.mk: patch does not apply
cannot patch at /usr/pkg/libexec/git-core/git-cvsexportcommit line 338.

There has been some other commit in pkgsrc since my last sync. To resolve, let’s first switch over to a new branch, based on the trunk:

$ cd ~/pkgsrc
$ git checkout trunk
$ git pull
$ git checkout -b update/gh
$ git cherry-pick 668ebe1fa
Auto-merging net/gh/Makefile
CONFLICT (content): Merge conflict in net/gh/Makefile
Auto-merging net/gh/distinfo
CONFLICT (content): Merge conflict in net/gh/distinfo
Auto-merging net/gh/go-modules.mk
CONFLICT (content): Merge conflict in net/gh/go-modules.mk
error: could not apply 668ebe1fa... gh: update to 2.68.1
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
$ cd net/gh
$ git status
On branch update/gh
You are currently cherry-picking commit 668ebe1fa.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Unmerged paths:
  (use "git add <file>..." to mark resolution)
        both modified:   Makefile
        both modified:   distinfo
        both modified:   go-modules.mk

no changes added to commit (use "git add" and/or "git commit -a")

Now I open the three files in my editor and merge, thanks to my years-long routine of doing this. Confusingly, the messages above show that there are two ways to continue the cherrypick: either with git commit or with git cherry-pick --continue. As far as I can tell, they do the exact same thing.

So, after resolving the conflict and running git commit -a, I have a commit with a new hash:

$ git log | head -1
commit 2b51308c74912f41974d2267f58bb35987bde7e8

Of course, it is a good idea to build the package again at this point and verify that it still works.

Let’s commit this one to CVS now. You don’t need to copy the whole hash, just the first 6-8 characters.

$ cd ~/pkgsrc-cvs
$ GIT_DIR=~/pkgsrc/.git git cvsexportcommit -v -c 2b51308c7
Applying to CVS commit 2b51308c74912f41974d2267f58bb35987bde7e8 from parent 44a7067c8ab30c8ec0e58668ec86ed0edaaba1de
Checking if patch will apply
Applying
Patch applied successfully. Adding new files and directories to CVS
Commit to CVS
Patch title (first comment line): gh: update to 2.68.1
Autocommit
  cvs commit -F .msg 'net/gh/Makefile' 'net/gh/distinfo' 'net/gh/go-modules.mk'
/cvsroot/pkgsrc/net/gh/Makefile,v  <--  net/gh/Makefile
new revision: 1.90; previous revision: 1.89
/cvsroot/pkgsrc/net/gh/distinfo,v  <--  net/gh/distinfo
new revision: 1.45; previous revision: 1.44
/cvsroot/pkgsrc/net/gh/go-modules.mk,v  <--  net/gh/go-modules.mk
new revision: 1.39; previous revision: 1.38
Committed successfully to CVS

The change is now live in CVS. The normal thing to do now is to add to the changelog. I do this directly in CVS for simplicity.

$ cd ~/pkgsrc-cvs/net/gh
$ bmake changes-entry && bmake changes-entry-commit
=> Updating CHANGES-2025 and TODO
P CHANGES-2025
=> Adding the change
=> Committing the change
/cvsroot/pkgsrc/doc/CHANGES-2025,v  <--  CHANGES-2025
new revision: 1.2125; previous revision: 1.2124

This works well for cross-cutting changes, like revbumps, too.

Why

In the beginning, I alluded to some of the pain points with CVS. But this also enables me to do some new things. In CVS, working copies are expensive, so you use the same one each time. It has happened to me more than once that I committed something I did not intend to because I had the change lying around in my tree, uncommitted.

Uncommitted changes are liabilities.

In a DVCS tree, you can commit everything right away and upstream when you are ready to. This means that you write change descriptions while you still remember what you did :) For a cross-cutting or independent change, you can just “reset” your tree to a clean state in a few seconds. This is a great thing.

When I used OpenBSD, I was a big fan of bsd.rd: a kernel that includes a root file system with an installer and a few tools. When I invariably did something bad to my root file system, I could use that to repair things. bsd.rd is also helpful for OS updates. And there is only a single file involved.

On NetBSD however, there is usually no netbsd.rd kernel installed, or even available by default. The facility is there, it’s just not standard. To be fair, there are a number of architectures that use kernels with a ramdisk for installation.

Recently, I have been toying with NetBSD on an Orange Pi 5. This is a 64-bit ARM board, using the evbarm-aarch64 architecture. I am booting from an SD card (details in a followup post) but once booted, the kernel does not see the card any more, only the NVMe SSD. So my thoughts went back to bsd.rd and I decided that I want one!

It turns out that there are two ways of getting a kernel with a ramdisk:

1. Building the ramdisk image into the kernel itself, bsd.rd-style. This is called an “instkernel” in NetBSD terminology.
2. A loadable kernel module, miniroot.kmod. The modules.tar.xz set contains an “empty” miniroot module, to which you can apparently add your own image.

I was unable to make #2 work, so I will only be talking about #1 here.

How to create a NetBSD ramdisk kernel

For the ramdisk image, you can use a pre-built “daily” image, which is available at http://nycdn.netbsd.org/pub/NetBSD-daily/HEAD/latest/evbarm-aarch64/installation/ramdisk/ as ramdisk.fs. Its size is 3072 KiB. Keep that number in mind for later.

For the next steps, you need a NetBSD source tree. The default location is /usr/src but any other location works just as well.

The Orange Pi 5 uses the GENERIC64 kernel configuration but there is no “install” variant. So you need to add a new configuration file, sys/arch/evbarm/conf/GENERIC64_INSTALL, with these contents:

include "arch/evbarm/conf/GENERIC64"

no options MEMORY_DISK_DYNAMIC
options MEMORY_DISK_IS_ROOT
options MEMORY_DISK_RBFLAGS=RB_SINGLE
options MEMORY_DISK_ROOT_SIZE=6144

This disables the dynamic ramdisk size and use a fixed size of 6144 sectors x 512 bytes (= 3072 KiB) instead. It also sets the ramdisk as the root filesystem and defaults to single-user mode.

Now let’s build the kernel. I did this on a Mac but any OS would do:

$ cd /usr/src
$ ./build.sh -O /Volumes/obj/ -m evbarm64 -N1 -j 7 -U tools
$ ./build.sh -O /Volumes/obj/ -m evbarm64 -N1 -j 7 -U -u kernel=GENERIC64_INSTALL

Side note: the obj directory needs to be on a case-sensitive filesystem. On a Mac, you can create a case-sensitive APFS dataset named obj, which is mounted under /Volumes/obj.

Now the magic bit, adding the image into the kernel:

$ /Volumes/obj/tools/mdsetimage/mdsetimage -v /Volumes/obj/sys/arch/evbarm/compile/GENERIC64_INSTALL/netbsd ~/Downloads/ramdisk.fs
mapped /Volumes/obj/sys/arch/evbarm/compile/GENERIC64_INSTALL/netbsd
got symbols from /Volumes/obj/sys/arch/evbarm/compile/GENERIC64_INSTALL/netbsd
root @ 0xc6e328/3145728
copying image /Users/bsiegert/Downloads/ramdisk.fs into /Volumes/obj/sys/arch/evbarm/compile/GENERIC64_INSTALL/netbsd (3145728 bytes)
done copying image
exiting

Now you can copy the sys/arch/evbarm/compile/GENERIC64_INSTALL/netbsd file to the SD card and boot from it. This boots into the NetBSD installer, which also has a (limited) shell available.

In my case, I was able to install the system to the SSD. Success! 🎉

Whenever I send around a Call for Papers for an open source conference, some people reply something like

Unfortunately, I don’t have anything to present right now. My work on XYZ is simply not far enough along.

Or similar. However, this does not matter.

I’ll let you in on a secret: When I was in academia, when someone was talking about their results at a conference, I mostly did not give a shit about the results themselves. (Unless I was working on something narrowly related, which was rare.)

Instead, what I was really interested in was the Intro and the Conclusion. Why did you do this research? What is the bigger picture? What have you learned?

What to talk about in an (open source) conference

The idea for this post came to me after EuroBSDCon 2023, where this type of conversation happened more than once, with different NetBSD and other developers. (I hung out mainly with the NetBSD crowd, as expected.)

Many of the successful presentations in the conference weren’t all that much about own work, the work of developing some driver, subsystem or whatever. They were about cool stuff they do with the OS. What cool stuff can I, the person watching your talk, do with this OS, or with some other OS? What are you using your operating system, or programming language, or whatever, for? Surely you are using it for something? So you are running some web services behind a firewall? You could talk about that! Explain how, say, npf in NetBSD helps you with that.

You are fiddling with your OS on a tiny single-board computer? Bring it into the talk, connect it to your laptop and boot it live!

Example: OccamBSD

One of the more inspiring talks that I watched at EuroBSDCon 2023 was the excellent Michael Dexter explaining OccamBSD. At its core, OccamBSD is a minimal FreeBSD – you build the world and disable all the knobs that can be disabled, so you get a system that’s minimal in size. Why?

• as a study object, to take apart and see what you can do without
• for jails, where you don’t need a mailer daemon to run your webserver

And more. I thought the idea was neat, and I am now experimenting with doing something similar on NetBSD.

So what, I hear you object, but this was presenting the author’s own, finished development work! Well yeah. But development work is never finished. Don’t be afraid of showing things which are not quite ready for prime time. You might find people that like the idea and want to help you.

Example: “NetBSD, not just for Toasters”

At FOSDEM 2020, I gave a somewhat generic talk introducing NetBSD and pkgsrc. Its title was “NetBSD: Not just for Toasters”. To be honest, when I started writing the talk, I was a bit worried that it would be too generic and hence boring.

But to my great surprise, this was my most successful FOSDEM talk ever. The hall was packed, and I had about an hour’s worth of audience questions in the hallway outside. It turns out that people who are not from the BSD community were coming in to this talk to find out what cool things your can do with this, just as I alluded to above.

And I did have some cool things in there, mentioning various emulation and cloud support layers, plus some fun ARM hardware that you can run NetBSD on. And clearly, even if you are never going to try this OS, maybe you got inspired to buy a Pinebook Pro and use an open hardware laptop?

Conclusion

It is hard to judge from your own perspective how interesting your topic will be for the audience. You, as the author, tend to overindex on your own work and take the context in which your work happens totally for granted. Yet, the audience perspective is very different.

• The audience is looking for ideas and inspiration.
• The audience might be looking for a cause that they can join and that sounds fun. It doesn’t have to be finished.
• Even if they will not use the exact combination of things you present, they are looking for an aspect that translates to their own world.

I hope this gives you some ideas for when you are feeling writer’s block when the next call for papers arrives.

PS: The Twitter button is gone from this blog.

I have been a pkgsrc developer for several years. For what it’s worth, I think pkgsrc is wonderful: a large selection of third-party software, packaged so that it is easy to install with a single command – either building everything from source, or relying on binary packages. pkgsrc supports dozens of OSes – not just NetBSD but also other BSDs, macOS, Linux, Illumos and more.

On the other hand, unfortunately, pkgsrc and NetBSD in general are suffering from what I would call a loss of mindshare. When I joined the NetBSD Foundation as a developer, I had the impression that NetBSD had more users and community than OpenBSD or, say, Dragonfly. In recent years however, it seems to me that people have more or less forgotten about NetBSD and pkgsrc.

Here is where you may come in

Here is a simple idea that occurred to me a while ago and that I finally am getting around to writing up, now that I am sitting in the train to FOSDEM 2023:

These days, a lot of upstream software has a manual with a section on installation that does not give instructions for actually building it. (It seems like building from source is so 1995, or something.) Instead, it goes roughly like this:

If you are using a Mac or a Linux/x86_64 machine, see our own binary packages here. Or use distro packages:

For Debian, run apt-get install somepackage.

For Fedora Linux, run some yum command.

etc. etc.

There is typically a long list of install commands for a bunch of Linux distributions and OSes. pkgsrc is almost never mentioned.

So if you are looking for a simple way of helping out pkgsrc and NetBSD, look for the README of your favorite Open Source tools and add a section for installing from pkgsrc. Typically, for a package from pkgsrc itself, the simplest is to add instructions for using pkgin:

$ pkgin install somepackage

For packages in pkgsrc-wip, you could first ask on the mailing list for an import so that there may be binary packages in the future :) Then, give the typical instructions for installing from source:

$ cd /usr/pkgsrc/wip/somepackage
$ make package-install

I have created a few pull requests for such changes in the past, but I think we need a lot more of that!

My new NAS at home is running TrueNAS Core. So far, it has been excellent, however I struggled a bit setting up a NetBSD VM on it. Part of the problem is that a lot of the docs and how-tos I found are stale, and the information in it no longer applies.

TrueNAS Core allows running VMs using bhyve, which is FreeBSD’s hypervisor. NetBSD is not an officially supported OS, at least according to the guest OS chooser in the TrueNAS web UI :) But since the release of NetBSD 9 a while ago, things have become far simpler than they used to be – with one caveat (see below).

UEFI Boot

NetBSD 9 and newer fully support booting through UEFI. This simplifies things because (as far as I understand) bhyve does not really support BIOS boot, it prefers loading the kernel directly.

It used to be the case that it was hard to get the installer working, so people started with an image of an already installed system, plus GRUB for bhyve. This is all very clunky and, to be clear, it is no longer needed.

Starting the Installer

Begin by downloading a CD image for the installer – the regular NetBSD 9_STABLE/amd64 installation CD image from https://nycdn.netbsd.org/pub/NetBSD-daily/netbsd-9/latest/amd64/installation/cdrom/ – and storing it on the ZFS volume.

In the web UI, under “Virtual Machines”, create a new one with the following settings:

• Guest OS: FreeBSD
• Boot method: UEFI
• VNC: enabled
• “wait with boot until VNC connects”: enabled
• Use VirtIO for disks and network
• When asked for an install CD, select the CD image downloaded earlier

Fallout

When you boot now, you will find that VNC disconnects after about five seconds. Further investigation shows that it’s actually the bhyve hypervisor that exits with a segfault.

This turns out to be an issue with the USB 3 (xhci) driver in the NetBSD kernel – or rather, it is probably a bhyve bug but disabling the xhci driver on the guest side works around it. Disabling xhci is not a big deal because the VM does not need native USB 3 anyway.

To work around:

1. Start the VM, then connect to VNC.
2. Once the bootloader appears, press (3) to go to the prompt. Enter boot -c.
3. In the userconf shell, enter disable xhci*, then quit.
4. The installer should appear, letting you install normally.
5. After installation, shut down the VM and remove the CD from the list of devices.
6. Start it again and repeat steps 2 and 3 to boot into the installed system.

To make the workaround permanent, edit the file /boot.cfg so that the first line reads

menu=Boot normally:rndseed /var/db/entropy-file;userconf disable xhci*;boot

At this point, you should also uncheck the “wait for boot until VNC connects” checkbox in the settings so the VM can start unattended in the future.

Success!

Here is the obligatory dmesg output for this system:

Copyright (c) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005,
    2006, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017,
    2018, 2019, 2020 The NetBSD Foundation, Inc.  All rights reserved.
Copyright (c) 1982, 1986, 1989, 1991, 1993
    The Regents of the University of California.  All rights reserved.

NetBSD 9.1_STABLE (GENERIC) #0: Thu Apr 22 10:08:46 UTC 2021
	mkrepro@mkrepro.NetBSD.org:/usr/src/sys/arch/amd64/compile/GENERIC
total memory = 8191 MB
avail memory = 7926 MB
cpu_rng: RDSEED
rnd: seeded with 256 bits
timecounter: Timecounters tick every 10.000 msec
Kernelized RAIDframe activated
running cgd selftest aes-xts-256 aes-xts-512 done
xhci* disabled
xhci* already disabled
timecounter: Timecounter "i8254" frequency 1193182 Hz quality 100
efi: systbl at pa bfb7cf18
  BHYVE (1.0)
mainbus0 (root)
ACPI: RSDP 0x00000000BFB88014 000024 (v02 BHYVE )
ACPI: XSDT 0x00000000BFB870E8 00004C (v01 BHYVE  BVFACP   00000001      01000013)
ACPI: FACP 0x00000000BFB86000 0000F4 (v04 BHYVE  BVFACP   00000001 BHYV 00000001)
ACPI: DSDT 0x00000000BEA98000 00191A (v02 BHYVE  BVDSDT   00000001 INTL 20200430)
ACPI: FACS 0x00000000BFB8C000 000040
ACPI: HPET 0x00000000BFB85000 000038 (v01 BHYVE  BVHPET   00000001 BHYV 00000001)
ACPI: APIC 0x00000000BFB84000 000062 (v01 BHYVE  BVMADT   00000001 BHYV 00000001)
ACPI: MCFG 0x00000000BFB83000 00003C (v01 BHYVE  BVMCFG   00000001 BHYV 00000001)
ACPI: SPCR 0x00000000BFB82000 000050 (v01 BHYVE  BVSPCR   00000001 BHYV 00000001)
ACPI: 1 ACPI AML tables successfully acquired and loaded
ioapic0 at mainbus0 apid 4: pa 0xfec00000, version 0x11, 32 pins
cpu0 at mainbus0 apid 0
cpu0: Intel(R) Pentium(R) Gold G5420 CPU @ 3.80GHz, id 0x906ea
cpu0: package 0, core 0, smt 0
cpu1 at mainbus0 apid 1
cpu1: Intel(R) Pentium(R) Gold G5420 CPU @ 3.80GHz, id 0x906ea
cpu1: package 0, core 0, smt 1
cpu2 at mainbus0 apid 2
cpu2: Intel(R) Pentium(R) Gold G5420 CPU @ 3.80GHz, id 0x906ea
cpu2: package 1, core 0, smt 0
cpu3 at mainbus0 apid 3
cpu3: Intel(R) Pentium(R) Gold G5420 CPU @ 3.80GHz, id 0x906ea
cpu3: package 1, core 0, smt 1
acpi0 at mainbus0: Intel ACPICA 20190405
acpi0: X/RSDT: OemId <BHYVE ,BVFACP  ,00000001>, AslId <    ,01000013>
acpi0: MCFG: segment 0, bus 0-255, address 0x00000000e0000000
acpi0: SCI interrupting at int 9
acpi0: fixed power button present
timecounter: Timecounter "ACPI-Safe" frequency 3579545 Hz quality 900
hpet0 at acpi0: high precision event timer (mem 0xfed00000-0xfed00400)
timecounter: Timecounter "hpet0" frequency 16777216 Hz quality 2000
pckbc1 at acpi0 (KBD, PNP0303) (kbd port): io 0x60,0x64 irq 1
pckbc2 at acpi0 (MOU, PNP0F03) (aux port): irq 12
SIO (PNP0C02) at acpi0 not configured
COM1 (PNP0501) at acpi0 not configured
COM2 (PNP0501) at acpi0 not configured
attimer1 at acpi0 (TIMR, PNP0100): io 0x40-0x43 irq 0
pckbd0 at pckbc1 (kbd slot)
pckbc1: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard
pms0 at pckbc1 (aux slot)
pckbc1: using irq 12 for aux slot
wsmouse0 at pms0 mux 0
pci0 at mainbus0 bus 0: configuration mode 1
pci0: i/o space, memory space enabled, rd/line, rd/mult, wr/inv ok
pchb0 at pci0 dev 0 function 0: vendor 1275 product 1275 (rev. 0x00)
virtio0 at pci0 dev 3 function 0
virtio0: Virtio Block Device (rev. 0x00)
ld0 at virtio0: Features: 0x10000244<INDIRECT_DESC,FLUSH,BLK_SIZE,SEG_MAX>
virtio0: allocated 270336 byte for virtqueue 0 for I/O request, size 128
virtio0: using 262144 byte (16384 entries) indirect descriptors
virtio0: config interrupting at msix0 vec 0
virtio0: queues interrupting at msix0 vec 1
ld0: 10240 MB, 5201 cyl, 64 head, 63 sec, 512 bytes/sect x 20971520 sectors
virtio1 at pci0 dev 4 function 0
virtio1: Virtio Network Device (rev. 0x00)
vioif0 at virtio1: Features: 0x11010020<INDIRECT_DESC,NOTIFY_ON_EMPTY,STATUS,MAC>
vioif0: Ethernet address xx:xx:xx:xx:xx:xx
virtio1: allocated 32768 byte for virtqueue 0 for rx0, size 1024
virtio1: allocated 311296 byte for virtqueue 1 for tx0, size 1024
virtio1: using 278528 byte (17408 entries) indirect descriptors
virtio1: config interrupting at msix1 vec 0
virtio1: queues interrupting at msix1 vec 1
genfb0 at pci0 dev 29 function 0: vendor fb5d product 40fb (rev. 0x00)
genfb0: framebuffer at 0xc1000000, size 1024x768, depth 32, stride 4096
genfb0: shadow framebuffer enabled, size 3072 KB
wsdisplay0 at genfb0 kbdmux 1: console (default, vt100 emulation), using wskbd0
wsmux1: connecting to wsdisplay0
drm at genfb0 not configured
vendor 8086 product 1e31 (USB serial bus, xHCI) at pci0 dev 30 function 0 not configured
pcib0 at pci0 dev 31 function 0: vendor 8086 product 7000 (rev. 0x00)
isa0 at pcib0
com0 at isa0 port 0x3f8-0x3ff irq 4: ns16550a, working fifo
com1 at isa0 port 0x2f8-0x2ff irq 3: ns16550a, working fifo
timecounter: Timecounter "clockinterrupt" frequency 100 Hz quality 0
timecounter: Timecounter "TSC" frequency 3792882480 Hz quality 3000
ld0: GPT GUID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
dk0 at ld0: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", 262144 blocks at 64, type: msdos
dk1 at ld0: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", 16512960 blocks at 262208, type: ffs
dk2 at ld0: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx", 4196319 blocks at 16775168, type: swap
IPsec: Initialized Security Association Processing.
boot device: ld0
root on dk1 dumps on dk2
root file system type: ffs
kern.module.path=/stand/amd64/9.1/modules
wsdisplay0: screen 1 added (default, vt100 emulation)
wsdisplay0: screen 2 added (default, vt100 emulation)
wsdisplay0: screen 3 added (default, vt100 emulation)
wsdisplay0: screen 4 added (default, vt100 emulation)

Like many ARM CPUs, the one in the Pinebook Pro has a “big.LITTLE” architecture, where some cores are more powerful than others:

[     1.000000] cpu0 at cpus0: Arm Cortex-A53 r0p4 (v8-A), id 0x0
[     1.000000] cpu1 at cpus0: Arm Cortex-A53 r0p4 (v8-A), id 0x1
[     1.000000] cpu2 at cpus0: Arm Cortex-A53 r0p4 (v8-A), id 0x2
[     1.000000] cpu3 at cpus0: Arm Cortex-A53 r0p4 (v8-A), id 0x3
[     1.000000] cpu4 at cpus0: Arm Cortex-A72 r0p2 (v8-A), id 0x100
[     1.000000] cpu5 at cpus0: Arm Cortex-A72 r0p2 (v8-A), id 0x101

The A72 is a more powerful than the efficiency-oriented A53, it has out-of-order execution, plus it reaches a higher maximum clock rate (1.4 GHz for the A53 and 2.0 GHz for the A72 in the Pinebook Pro).

On NetBSD-current, the kernel scheduler prefers the big cores to the little ones. However, when building software, you may want to force the build process onto the big cores only. One advantage is that you still have the little cores to deal with user input and such, yet your build has the highest performance. Also, building with all cores at the highest clock rate will quickly lead to overheating.

NetBSD has a somewhat obscure tool named psrset that allows creating “sets” of cores and running tasks on one of those sets. Let’s try it:

$ psrset
system processor set 0: processor(s) 0 1 2 3 4 5

Now let’s create a set that comprises cpu4 and cpu5. You will have to do that as root for obvious reasons:

# psrset -c 4 5
1
# psrset
system processor set 0: processor(s) 0 1 2 3
user processor set 1: processor(s) 4 5

The first invocation printed “1”, which is the ID of our new processor set. Now we can run something on this set. Everything run below only sees two cores, cpu4 and cpu5. Note the “1” in the command below. This is the ID from before.

# psrset -e 1 make package-install MAKE_JOBS=2

If you run htop or similar while your package is building, you will see that only cpu4 and cpu5 are busy. If you have installed estd to automatically adjust CPU clocks, you will notice that cpu4 and cpu5 are at 2 GHz while the four little cores are running at a cool 400 MHz.

If you buy a Pinebook Pro now, it comes with Manjaro Linux on the internal eMMC storage. Let’s install NetBSD instead!

The easiest way to get started is to buy a decent micro-SD card (what sort of markings it should have is a science of its own, by the way) and install NetBSD on that. On a warm boot (i.e. when rebooting a running system), the micro-SD card has priority compared to the eMMC, so the system will boot from there.

As for which version to run, there is a conundrum:

• There are binary packages but only for NetBSD-9. On -current, you have to compile everything yourself, which takes a long time.
• Hardware support is better in NetBSD-current.

The solution is to run a userland from NetBSD-9 with a NetBSD-current kernel.

As the Pinebook Pro is a fully 64-bit capable machine, we are going to run the evbarm-aarch64 NetBSD port on it. Head over to https://armbsd.org/arm/ (thanks, Jared McNeill!) and grab a NetBSD 9 image for the Pinebook Pro. Then (assuming you are under Linux), extract it onto the memory card with the following command:

zcat netbsd-9.img.gz | dd of=/dev/mmcblk2 bs=1m status=progress

Be sure to check that mmcblk2 is the correct device, e.g. by examining dmesg output! Once the command is done, you can reboot. Once booting is finished, you can log in as root with no password. The first thing you should do is to set one, using passwd.

To the eMMC!

Would you like to replace the pre-installed Manjaro Linux on the eMMC?

It makes sense to have your main OS on the built-in storage, since it is quite a bit faster than the typical micro-SD card. In my tests, I get write speeds of about 70 MiB per second on the eMMC.

By the way, if you want more and even faster Storage, PINE64 will sell you an adapter board for adding an NVMe drive (a fast SSD).

Once you have booted NetBSD from the memory card, mount the Linux volume and copy over the image file from before, then unmount and extract it in exactly the same way as above. The only difference is that the target device is called /dev/rld0. Shut down the system, remove the memory card, switch it back on and watch NetBSD come up :)

Getting a -current Kernel

To have better driver support, I recommend installing a NetBSD-current kernel. To do that, you just need to replace the /netbsd file with the new kernel – no changes to the bootloader are required.

You can find a pre-built kernel under https://nycdn.netbsd.org/pub/NetBSD-daily/HEAD/latest/evbarm-aarch64/binary/kernel. Download the file and install it:

cp /netbsd /netbsd.old
gunzip netbsd-GENERIC64.gz
install -o root -g wheel -m 555 netbsd-GENERIC64 /netbsd

You will find that there is now a driver for the built-in Broadcom Wi-Fi (as the bwfm0 interface) but the firmware is missing. To fix this, download the base.tgz set from the same location and extract the firmware blobs only (as root):

cd /
tar xvpfz /path/to/base.tgz libdata/firmware

In my experience however, the Broadcom Wi-Fi driver is extremely likely to make the system crash or hang. I tend to rely mostly on an old Apple Ethernet-USB adapter (an axe interface).

NOTE: I have since back-pedaled and returned to NetBSD 9. Other than the unstable Wi-Fi, I also had crashes when running npm install and other issues.

Update 2022-08-27: changed the device name for writing the image, thanks Nikita!

Note: This post was written on the Pinebook Pro :)

After seeing it in action at FOSDEM (from afar, as the crowd was too large), I decided to buy a Pinebook Pro for personal use. From the beginning, the intention was to use it for pkgsrc development, with NetBSD as the main OS. It was finally delivered on Thursday, one day earlier than promised, so I thought I would write down my first impressions.

If you have never heard of the Pinebook Pro: It is a cheap, open, hackable laptop with an ARM processor, the successor of the original Pinebook (which I thought was too low-end to be a useful daily driver) with generally more premium components.

As I alluded to in the first paragraph, the enthusiasm of the Free Software community is incredibly strong! Nothing showed this better than the incredible resonance from a tweet with a quick snapshot after the first boot:

It's here, a day earlier than expected! @thepine64 #PineBookPro pic.twitter.com/HIMdnFtxIG

— benz (@bentsukun) May 28, 2020

You’ll note, in passing, that on the photo, I am downloading the NetBSD install image :)

What makes this device attractive, apart from the price, is the ARM architecture without the baggage of the PC world. What’s more, an open and hackable system in the age of Macbooks with soldered everything, tablets that you cannot open, “secure” boot that severely limits what you can run on it, is something of a counterculture device. The Pine64 folks have built a great community that embodies the true hacker spirit.

The Hardware

Here is where I am going to be harsh: in some ways, using this device feels like a regression.

I was previously using a Samsung Chromebook Pro, a Pixelbook and a Pixel Slate as laptops. Compared to these, the Pinebook Pro has

• no HiDPI screen,
• no touchscreen,
• a barrel connector power supply,

and peripherals are mostly using USB-A port. To be fair, there is USB-C, and it can be used to charge the machine, so I haven’t used the original charger yet.

The display resolution is 1920x1080, equivalent to about 100 dpi. While I regret the absence of HiDPI, it is well lit and well readable. The viewing angles are fairly large, and the colors are crisp. The default Manjaro Linux wallpaper is a great showcase for this.

The Pinebook Pro is vaguely shaped like a MacBook Air from a few years ago, with the same curved bottom. At 14", it is surprisingly large – the machines mentioned above are 12 to 13". Compared to the MacBook and Pixelbook in their quest for ever thinner devices, the Pinebook Pro feels strangely empty. I guess there is actually free space on the inside that you can use for upgrades and such.

The most similar laptop I have used is the HP Chromebook 14. Against this, the Pinebook Pro holds up really well though: it is lighter, has a better keyboard and display and is actually cheaper!

Keyboard and Trackpad

The keyboard is an absolute joy to use. Really, it’s great. The keys have a large amount of travel, comparable to older MacBook Pros, before they introduced the terrible keyboard. The layout (I have ANSI, i.e. US) is exactly what you would expect. This is definitely made for typing a lot.

On the other hand, I am not friends with that trackpad. I am hoping I can get used to it at some point. The way it tracks small finger movement is … weird and counter-intuitive, and I am having a hard time hitting small click targets. It has two mouse buttons under the bottom left and bottom right, so clicking in the middle usually has no effect. For dragging, you need to keep one finger in the corner on the button and move another finger, which sometimes triggers multi-touch gestures.

Battery life

I have not done detailed measurements, but it seems pretty good at about 7 hours. Here is the envstat output while writing this:

                               Current  CritMax  WarnMax  WarnMin  CritMin  Unit
[cwfg0]
            battery voltage:     3.935                                         V
            battery percent:        79                                      none
  battery remaining minutes:       319        0        0        0        0  none
[rktsadc0]
                        CPU:    42.778   95.000   75.000                    degC
                        GPU:    43.333   95.000   75.000                    degC

Performance

Compared to all those ARM SBCs I have used (Raspberry Pi, Orange Pi, Pine-A64), the Pinebook Pro feels really fast. Storage (at this point I am using a memory card) is decent speed-wise, and compilations are reasonably fast – though my five year old Intel NUC with an i7 still beats it by far, of course. But my workload involves compiling lots of stuff, so this seems like a good fit.

Graphics performance has not blown me away. Animations on Manjaro stutter a bit, Midori on NetBSD (the first browser that I tried) is really testing my patience.

Hackability?

I noticed that opening the bottom of the housing is screwed on with standard Philips head screws and easy to open. There are no rubber feet glued on top of the screws, no special tools needed.

You can easily boot your custom OS from the micro-SD card reader. As mentioned, I bought this machine for running NetBSD on it, which works well.

There are upgrade kits available, for example an adapter to add an NVMe disk instead of the eMMC. For the original Pinebook, there has been an upgrade kit with a better processor even.

Because we are on ARM, there is no “Intel Inside”, and consequently, there are also no stickers, except for one on the underside that gives the model number.

Conclusion

This has become longer than I intended. Despite my criticism, I really like this machine. I am hoping I can get some good development work done with it and use NetBSD for my daily computing tasks.

Stay tuned for another post with some NetBSD tips!

After talking to Sijmen Mulder on IRC (thanks, TGV Wi-Fi!), I began thinking more about how you could automate the pkgsrc release engineers away.

The basic idea for a buildbot would be this:

1. Download and unpack latest pkgsrc.tar.gz for the stable branch.
2. Run the pullup script with the ticket number, then run whatever pullup script it outputs.
3. Figure out the package that this concerns (perhaps from filenames).
4. Go to the package in question, install its dependencies from binary packages.
5. Build (make package is probably enough, or perhaps also install?).
6. Upload build log to Cloud Storage.
7. Post an email to the pullup thread with status and a link to the log.

For extra points, do this in a fresh, ephemeral VM, triggered by an incoming mail.

You would also need a buildbot supervisor that receives mails (to know that it should build something) and that launches the VM. I know that Google App Engine could do it, as it can receive emails. But maybe Cloud Functions would be the way to go?

In any case, this would be a cool project for someone, maybe myself :)

Issues with Pull-up Ticket Tracking

This project is largely orthogonal to improvements in the pullup script. Right now, there are a number of issues with it that make it require manual intervention in many cases:

• The tracker (req) doesn’t do MIME, so sometimes mails are encoded with base64 or quoted-printale. This breaks parsing the commit mails.
• Sometimes, submitters of tickets insert mail-index.netbsd.org URLs instead of copies of the message.
• Some pullup tickets include a patch instead of, or in addition to, a list of commits. For instance, this may happen when backporting a fix to an older release instead of pulling up a bigger update.
• Sometimes, commit messages are truncated, or there are merge conflicts. This mostly happens when there has been a revbump before the change that is to be committed – in the majority of cases, the merge conflicts only concern PKGREVISION lines.

I am wondering how much we could gain, e.g. in terms of MIME support, from changing the request tracking software. admins@ uses RT, which has more features. Perhaps that could be brought to pullup tickets?

Today, I deleted my Amazon AWS account.

I had been on AWS since about 2011. My usage was mainly for two things:

1. Saving large amounts of files (build logs and such) on S3;
2. Running NetBSD VMs on EC2.

EC2 is based on Xen, and NetBSD runs really well in PV (paravirtualized) mode on Xen. However, XSA-240 means that a malicious PV guest may crash (or even otherwise exploit) the hypervisor, with the recommended fix being to not run untrusted PV guests. Over night, Amazon disabled PV, making NetBSD VMs useless.

In general, EC2 has been moving away from Xen. The newer instance types already no longer supported PV; there are two higher-performance paravirtualized modes (PVH and PVHVM) that are preferred these days, and that NetBSD does not support. The newest machine types use a custom hypervisor based on KVM.

The way the PV change was rolled out highlighted another long-standing EC2 problem: instances would continue running until the server they ran on got rebooted, at which point they were migrated to a random machine. If the target machine had PV disabled, the VM simply did not come up again. I have had the same type of issue in the past, where your VM randomly landed on a “good” or a “bad” machine and did not come up on the bad one. There is no way (AFAIK) to constrain to a certain subset of servers, e.g. running a certain hypervisor version.

Also, of course, there was no warning or announcement, just that VMs stopped working all of a sudden. A bunch of people were completely caught by surprise when their service became unavailable. I hope you have monitoring!?

The Alternative

Which brings me to where I did take my workloads: Google Cloud Platform.

(This has nothing to do whatsoever with who my employer is. I pay for my GCP usage with my own money.)

These days, NetBSD (8+) runs great on Google Compute Engine. There is a script (that I created) to stage instances at https://github.com/google/netbsd-gce, though there are no official NetBSD images around. My S3 usage works equally well using Google Cloud Storage. And I have always been a fan of App Engine, particularly because of its great Go support. https://bulktracker.appspot.com/ runs on App Engine.

Conclusion

My general impression is: Features are roughly on par, prices on GCP are a bit cheaper, and the Google Cloud SDK and command-line tools are better. So rather than let old, unusable VM images continue to rot and pay Amazon 2$ a month for that bit of storage, I let go of that AWS account. Bye, Amazon.