Last weekend, at FOSDEM 2023, I watched a Lightning Talk by Frank Vanbever titled “Lua for the Lazy C developer”. I had recently suggested at work that we should be using Lua to script the behavior of some systems which are written in Go, so the talk strongly resonated with me.

Lua is an ideal scripting language for embedding into other programs because it is small and provides excellent bindings in both directions – Lua code can call native code, and vice versa. Because Go has good hash tables as part of the language, using Lua as a container for your hash tables is probably less interesting than from C though.

During the talk, I tried porting the “Hello World” example to Go as the native language. It turns out that Shopify published a pure-Go reimplementation of the Lua interpreter that is byte code compatible with the original C implementation! They use this for scripting the behavior (ha!) of their load testing tools. The package is at github.com/Shopify/go-lua.

It turns out that the example from the talk translates 1:1 to Go and Lua:

package main

import lua "github.com/Shopify/go-lua"

func main() {
	l := lua.NewState()
	lua.OpenLibraries(l)
	l.Global("print")
	l.PushString("Hello World!\n")
	l.Call(1, 0)
}

And it does exactly what you would expect:

$ go build .
$ ./luatest 
Hello World!

The full example is available as a GitHub Gist.

This weekend, I made a series of somewhat unusual changes to pkgsrc.

I removed a bunch of Go packages.

Why? Because of Go modules.

What are Go modules?

Since my series of design-ish blog posts(part 1, part 2), Go module builds have fully landed in pkgsrc, to the point that they are now the preferred way to build Go packages.

To recap: There are two ways to use the go tool to build Go code.

• The old way is to have a tree, below $GOPATH, that has all the dependencies in a directory tree according to their import path. For instance, the golang.org/x/net package would be placed in a $GOPATH/src/golang.org/x/net directory. This is what lang/go/go-package.mk implements in pkgsrc.
• The new way is to extract the source code wherever you want, just like any C source. The top-level source directory contains a go.mod file that specifies dependencies and their versions. The go tool then downloads a bunch of .zip and .mod files for those dependencies and unpacks them as needed. This is similar to how Cargo works for Rust code.

Like with Rust, in pkgsrc, we specify a list of dependent module files to be downloaded from the module proxy.

In actual practice, a useful pattern has emerged, where the list of modules is in a separate file named go-modules.mk in the package directory. To create or update the file, simply run

$ make patch
$ make show-go-modules > go-modules.mk

and then .include the file from the main Makefile.

But why remove all these packages?

A pkgsrc package built with lang/go/go-module.mk does not install any source code or .a files. Only the binaries are packaged, just like for C. Go packages that just correspond to intermediate libraries and do not contain any useful binaries are simply no longer needed. They can be deleted as soon as nothing depends on them any more.

In particular, I changed all the packages depending on golang.org/x/tools to be modules, then migrated the go-tools package itself. go-tools depends on a number of other libraries that nothing else depends on.

By the way, it is fairly simple to make a non-module into a module, even if the source does not contain a go.mod:

1. Change go-package.mk to go-module.mk.
2. Run make patch and change into the top-level source directory.
3. Run go mod init github.com/foo/bar or whatever the import path is.
4. Update the file with go get and/or go mod tidy.
5. Copy the generated go.mod and go.sum files to a files directory and copy them into place in pre-patch.

Future

Some future Go release will deprecate GOPATH builds, so we must convert all Go code in pkgsrc to modules at some point. By the way, if upstream has not made the jump to modules yet, they might be happy about your pull request :)

This announcement dropped today:

I realized that this is the missing piece for supporting Go modules in pkgsrc. If you go back and reread the “fetch” section in Supporting Go Modules in pkgsrc, it seems a bit awkward compared to a standard fetch action. The reason is that go mod download re-packs the source into its own zip format archive.

The module proxy (https://proxy.golang.org/) solves this problem and enables a simple solution for modules, very similar to lang/rust/cargo.mk. Basically, a target similar to show-cargo-depends that outputs a Makefile fragment containing the names of modules that the current package depends upon. All these become distfiles fetched from a hypothetical $MASTER_SITES_GOPROXY. Crucially, this means that the distfiles do not have to be stored in a LOCAL_PORTS subdirectory but can use the normal fetch infrastructure.

Now all that remains is implementing this :) There is some more time to do that: Go 1.13 (to be released some time in summer) will use module support by default. What’s more, a bunch of new software (including the various golang.org/x/* repositories) has go.mod files these days, using module-based builds by default.

Go 1.11 introduced a new way of building Go code that no longer needs a GOPATH at all. In due course, this will become the default way of building. What’s more, sooner or later, we are going to want to package software that only builds with modules.

There should be some package-settable variable that controls whether you want to use modules or not. If you are going to use modules, then the repo should have a go.mod file. Otherwise (e.g. if there is a dep file or something), the build could start by doing go mod init (which needs to be after make extract).

fetch

There can be two implementations of the fetch phase:

1. Run go mod download.

   It should download required packages into a cache directory, $GOPATH/pkg/mod/cache/download. Then, I propose tarring up the whole tree into a single .tar.gz and putting that into the distfile directory for make checksum. Alternatively, we could have the individual files from the cache as “distfiles”. Note however (see below) that the filenames alone do not contain the module name, so there will be tons of files named v1.0.zip and so on.

2. “Regular fetch”

   Download the .tar.gz (or the set of individual files) from above from the LOCAL_PORTS directory on ftp.n.o, as usual.

The files that go mod download creates are different from any of the ones that upstream provides. Notably, the zip files are based on a VCS checkout followed by re-zipping. Here is an example for the piece of a cache tree corresponding to a single dependency (ignore the lock files):

./github.com/nsf/termbox-go/@v:
list                                           v0.0.0-20180613055208-5c94acc5e6eb.lock        v0.0.0-20180613055208-5c94acc5e6eb.ziphash
list.lock                                      v0.0.0-20180613055208-5c94acc5e6eb.mod
v0.0.0-20180613055208-5c94acc5e6eb.info        v0.0.0-20180613055208-5c94acc5e6eb.zip

As an additional complication, (2) needs to run after “make extract”. Method (1) cannot always be the default, as it needs access to some kind of hosting. A non-developer cannot easily upload the distfile.

extract

In a GOPATH build, we do some gymnastics to move the just-extracted source code into the correct place in a GOPATH. This is no longer necessary, and module builds can just use the same $WRKSRC logic as other software.

build

The dependencies tarball (or individual dependencies files) should be extracted into $GOPATH, which in non-mod builds is propagated through buildlink3.mk files of dependent packages. After this, in all invocations of the go tool, we set GOPROXY=file://$GOPATH/pkg/mod/cache/download, as per this comment from the help:

A Go module proxy is any web server that can respond to GET requests for URLs of a specified form. The requests have no query parameters, so even a site serving from a fixed file system (including a file:/// URL) can be a module proxy.

Even when downloading directly from version control systems, the go command synthesizes explicit info, mod, and zip files and stores them in its local cache, $GOPATH/pkg/mod/cache/download, the same as if it had downloaded them directly from a proxy.