A while ago, I installed Fedora Asahi Remix on my M2 MacBook Air, and I was very positive about it. So positive, in fact, that I ended up making it the default partition in the bootloader. I haven’t used macOS in weeks. But then, a few days ago, something weird happened:

In the middle of some development work, running cvs update on the pkgsrc repository, the screen suddenly filled with a bunch of “read-only file system” errors. It turned out that both / and /home had remounted themselves read-only, without my intervention.

Running journalctl --system showed some warning messages in an alarming orange-red: btrfs, the file system that the system was running from, had found inconsistencies and had chosen to remount itself read-only. Now what?

I tried running btrfsck /dev/nvme0n1p6. Now, by default, btrfsck will only check, not repair. But even the check aborted with an error about “errors found in FS roots”.

From my BSD experience, I thought booting into single-user mode and running fsck with a r/o root file system would do the trick. A quick init S later … it turns out that you can only get a shell in single-user mode if you have a root password set. By default, logins as root are forbidden, even in single-user mode. But good luck setting a password while / is read-only!

After several reboots, I got the system to work r/w for a while, so I could set a root password. So within single-user mode, I ran

# btrfsck --repair --force /dev/nvme0n1p6

Decent idea, but like the check before, the repair option also aborted with an error, saying “parent transid verify failed”. According to some random search results, this means that the file system is likely damaged beyond repair. Or, as @ParadeGrotesque aptly remarked on Mastodon:

It’s dead, Jim.

Don’t blame me, I don’t make the rules.

So in the end, I managed to back up my home directory with Restic (good thing I had restic installed before!), then wiped the entire installation and re-installed. Bummer.

More on btrfs

Why is btrfs the default file system in the first place? The Fedora Asahi Remix installer doesn’t even let you choose the file system type! I wish it did!

The way I understand btrfs is that the developers are aiming for a feature set on par with ZFS, except fully under the GPL and more “Linux-native” – ZFS was originally developed for Solaris, after all, and its licensing situation is at least a bit dodgy.

btrfs has long had a reputation of playing fast and loose with your data. In the early days, I heard several stories of spectacularly losing data. People have assured me that those days are over, and that btrfs is a reliable system these days.

Honestly, this experience has shown me that btrfs, even today, is anything but reliable. The corruption I witnessed is surely not due to broken hardware. The hardware is absolutely fine.

I suspect that the root cause is probably several unclean shutdowns. Asahi Linux does have an unfortunate bug where it will consume too much battery when in standby, so I ran out of battery several times. And, to be fair, finding the corruption and going read-only is sensible. But what I find absolutely infuriating is the utter inability of its fsck to actually repair anything. The fs roots are damaged, so what? There are incorrect transids, so what? Telling me my FS is fucked and I need to recreate it from scratch is not useful!

As it is, what happened with btrfs made me recall the previous low point of Linux file systems: years ago, I ran a file server with ReiserFS in production. It corrupted itself to such a degree that reiserfsck kept segfaulting. We lost so much user data. Good times.

I have been following Asahi Linux for a while. Linux for my MacBook Air M2 – sure, why not? But I wasn’t particularly interested in a distribution based on Arch Linux.

In late 2023, the Asahi folks presented a new distro that they called Fedora Asahi Remix. The promise is to combine the ground-breaking Kernel development of Asahi with the polish of Fedora Linux. I thought I would give it a go.

tl;dr for the rest of this post: the installer feels very hackish but once installed, it’s great!

Installation

The installer follows the infamous curl | sudo bash paradigm. I don’t trust this sort of thing so I took a look at what is downloaded: it is mostly a launcher for the rest of the installation, which is also a collection of shell scripts. It turns out that Fedora has a graphical installer, which appears very late in the installation routine, for about two minutes, the rest happens in the terminal.

The first thing the installer does is shrink the macOS partition to make space. You choose how much to give it – I chose a roughly 60:40 split. Aside: technically, it’s not a partition but more of a pool, as APFS is similar to ZFS in its interface.

Step number two is really the special sauce of Asahi Linux: installing a UEFI environment, the U-Boot bootloader and making it bootable from the boot selector screen, including some firmware setting changes.

Unlike a PC, an ARM Mac has a specialized firmware which is made to boot macOS and nothing else. On other types of ARM machines such as the Pinebook Pro, U-Boot (the Universal Bootloader) takes care of running the OS. On the Mac, Asahi sets some firmware variables that allow this behavior as well. The Linux boot partition needs to be “blessed”.

Here is where the most hackish part of the installation happens: the installer reboots into the macOS recovery system. Before it does that, it prints instructions asking you to run a series of commands inside the rescue system and accept the warnings that it prints. I don’t think you can brick your machine if you do it wrong but I would not give this to someone who is not good at this computer thing.

The next reboot is the first time that the machine is actually running a Linux kernel. The graphical Fedora installer appears and does a bit of configuration. That’s it.

After installation, my system still boots into macOS by default. To boot Fedora, I hold the power button in the first boot phase, so that the firmware shows the boot selector screen. Then, the options are macOS, Fedora and the macOS rescue system.

Using the system

This my first time using Fedora. The only RPM-based distro that I used in the past was SuSE, about 25 years ago. But from afar, Fedora always seemed to be fairly polished. So I was curious.

There are several flavors of desktops that can be selected in the installer: KDE, GNOME, or no desktop / do-it-yourself. The documentation says that KDE is the most polished option, so I went with KDE Plasma. Again, many years have passed since I last used KDE.

KDE still feels cluttered to me, as if its designers think what power users want is lots of buttons in lots of toolbars everywhere. But luckily, you can disable them and gain some space.

On the other hand, the UI in general (based on Wayland) looks gorgeous on the 2x Hi-DPI screen of the MacBook Air! I have struggled to configure Hi-DPI X11 desktops properly in the past (both on NetBSD and Debian), but Fedora has really nailed the setup out of the box. Kudos!

Another thing I noticed is the amount of updates. Fedora’s equivalent to apt used to be called yum and is now dnf, though there is a symlink – yum update still works. Contrary to apt, dnf update fetches package descriptions and updates packages.

New updates are added all the time. As I quipped on Mastodon the other day:

I have not booted this Fedora system in a few days, so 750 MB of updates it is

This was literally the amount of updates I had at one point. I am not sure if this is good or bad. I guess it’s not a big deal unless you are concerned about the amount of SSD writes, or the speed of your internet connection (FTTH FTW!).

Installing more software

• Bootstrapping pkgsrc went fine, so that’s another 35000 packages at my fingertips :)
• I tried installing Steam but it seems like there are no Linux/aarch64 packages available. This works better on macOS with the Game Development kit doing some kind of ad-hoc x86 emulation, though I was able to make it work exactly once and it stopped working after a reboot.
• I wanted Visual Studio Code (don’t judge me!), and the graphical software catalog proposed a Flatpak version. As I quickly discovered, the VS Code Flatpak is useless, at least for my purposes. I suppose you can make it work with a generous sprinkling of Dev Containers, but I don’t want to.

Hardware support

Almost everything works! I cannot overstate what a big deal this is, and I expected the experience to be much less refined in that regard. The function keys do the right thing, you can control brightness and keyboard backlight, suspend/resume works perfectly, etc.

Using Fedora on this hardware feels super snappy. File system operations are shockingly fast. Compiling software seems much faster than on macOS, though I did not benchmark this. My hunch is that the process management is a bit slower in macOS, due to the Mach / XNU underpinnings. 3D graphics performance is good too.

I only found two minor caveats in my testing:

• External displays over USB-C do not work, which means I cannot do my conference presentation from Linux. This is a known limitation.
• The system uses more power while it is suspended. In macOS, you can close the lid and have almost no battery use during the time. On Linux, the battery is empty after a few days.
• In general, battery life is a bit worse but still amazing. This may be due to me running more demanding workloads while on Linux. I did not do a scientific comparison.

Conclusion

I really like it!

I might find myself using Linux as my main OS on the MacBook – though in the last couple of days, I have been using macOS more, maybe half the time. Still, I had not expected this distro to be so good. And I had not expected that I would like it so much.

So is 2024 finally the year of Linux on the desktop? I guess for me it is.

I installed Fedora 39 the other day. (More on that in one of the next posts.) It has a nifty software installer thing named “Discover”. When I typed “Visual Studio Code” into the search box, it dutifully installed VS Code. As a Flatpak.

This was the first time I interacted with Flatpak, and it did not go well.

Aside: Why VS Code

I want to use VS Code for editing Go code, with gopls, since it provides a really good integration. It turns out that a majority of Go developers use VS Code, so the language server integration is well tested and complete. In short, it’s the well-lit path.

Specifically for myself, there is a second reason: At work, we use a bespoke web-based IDE based on VS Code, with all sorts of integrations to make developers more productive. I have gotten very familiar with this setup, so it makes sense for me to write open source code in this way too and benefit from the muscle memory, so to speak.

There is a philosophical argument that you should avoid VS Code because it is controlled by Microsoft, and it gives Microsoft a certain leverage over the open source Go ecosystem. For what it’s worth, the same argument applies to using GitHub: Such a large percentage of open source code is developed on GitHub these days, and Microsoft could “enshittify” it at any moment if they wanted to. But for both of these, I personally think that it would be easily doable to switch away from them to something free – for instance, move over to Neovim’s LSP integration. In the meantime, I remain pragmatic and use what works for me.

Why not Flatpak

Back to Fedora. I installed Go and gopls from pkgsrc, as you do. However, the Go plugin tells me that it cannot find gopls, or any of the toolchain. Why!? Issue golang/vscode-go#263 has a bunch of people rather confused about this failure mode.

Ultimately, it comes down to this: Flatpaks run isolated from the host OS, which is fundamentally incompatible with developing code in an IDE. Despite the allowlist for access to OS paths, there is no way to allow access to /usr, /lib, etc. This is fundamental to the Flatpak security model. The app runs in a container that is cordoned off from the main OS.

But an IDE for developing native code does need host OS access! It needs to run a build tool, toolchain, native tools, etc. Not only does the VS Code Flatpak not allow doing this, it also does not provide an easy way to install a toolchain into the container. Never mind that I do not want a second copy of my tools. An IDE also needs to be able to run a shell in a terminal.

At this point, I am left wondering: who can actually use this Flatpak, if native compilation is all but impossible? Is it made for frontend devs writing only JS, with all the tooling in JS as well?

Fixes

I am also wondering why the Flatpak is the default type of installation for VS Code on Fedora, given that Upstream has a repository of perfectly fine RPM packages. See https://code.visualstudio.com/docs/setup/linux#_rhel-fedora-and-centos-based-distributions.

The other container-like option would be a Snap, if you are into additional packaging systems on your distro. You can install the Snap in “classic” mode, where it has file system access:

sudo snap install --classic code

My choice has been the native package, and now everything is running fine.