Replacing packr with go:embed to Embed Files & Directories

With Go 1.16 we can natively embed static files into binaries. The common alternative to this day was using community alternatives, of which one of the most commonly used ones is gobuffalo/packr. There are issues with the development experience. Packages such as packr require us to add/update assets every time they change and the tool to package files up to be available on every machine.

We usually deploy Go apps with ease as we only need to deliver a single binary. There could however be cases for run time files and assets. Instead of shipping such files through build pipelines and into containers, native file embedding support allows for a more idiomatic way to access such dependencies and makes it easier to deploy our applications. Some examples I toyed with when switching from packr2.

Consider the following directory structure:

datadir
├── file1.txt
└── subdir
    └── file2.txt

We can embed a read only collection of these files and walk the directory tree as follows:

//go:embed datadir
var data embed.FS

func main() {
	err := fs.WalkDir(data, ".", func(path string, d fs.DirEntry, err error) error {
		if err != nil {
			return err
		}
		fmt.Printf("path=%q, isDir=%v\n", path, d.IsDir())
		return nil
	})

	if err != nil {
		log.Fatal(err)
	}
}

embed.FS implements fs.FS, which allows usage with any package that understands file system interfaces. Running this with go run *.go returns

path=".", isDir=true
path="datadir", isDir=true
path="datadir/file1.txt", isDir=false
path="datadir/file2.txt", isDir=false

We already have access to the file name:

fmt.Printf("path=%q, name=%s, isDir=%v\n", path, d.Name(), d.IsDir())

So a list function to return a list of all files could look like this:

func list(dir embed.FS) ([]string, error) {
	var out []string
	err := fs.WalkDir(dir, ".", func(path string, d fs.DirEntry, err error) error {
		if err != nil {
			return err
		}
		if d.IsDir() {
			return nil
		}

		out = append(out, d.Name())

		return nil
	})

	return out, err
}

it returns

list=["file1.txt" "file2.txt"]

A find function to return the byte contents of an embedded file if found we can construct analogously:

func find(dir embed.FS, filename string) ([]byte, error) {
	var out []byte
	err := fs.WalkDir(dir, ".", func(path string, d fs.DirEntry, err error) error {
		if err != nil {
			return err
		}

		if d.IsDir() {
			return nil
		}

		if d.Name() == filename {
			out, err = fs.ReadFile(dir, path)
			if err != nil {
				return err
			}
		}

		return nil
	})

	return out, err
}

Since we traverse the entire tree, we will find a file here recursively, from sub directories too.

Symbolic Links in go:embed

Where this falls short are symlinks

datadir
├── file1.txt
├── file3.txt -> subdir/file2.txt
└── subdir
    └── file2.txt

1 directory, 3 files

The find() function wouldn’t return any results. Looking for file3.txt here would result in the file not found error which we should add to the above example as an exercise for the reader.

We can fix this by resolving the symlink however in a build directory and embed that from the sources.

cp -rL source build/destination
# cp -RL source build/destination # on MacOS

Compile time step with go:generate

If we’re dealing with symlinks it would be nice if we can add them with our native tooling in a build step. There is the go:generate directive that can assist us and executes these commands with the go generate command:

//go:generate  mkdir -p build
//go:generate  cp -RL datadir ./build/datadir
//go:embed build/datadir
var data embed.FS

Embedded files in packages

Go packages can embed files which will be vendored with go mod and not purged because mod is sensitive to the embed statements. This works out of the box.

Armed with all this, we hopefully now have even better tooling at our disposal.