Surveyor

Registry sweeps and CBOMs

A registry sweep starts from the artifacts your builds already publish. Point it at an artifact repository, such as Artifactory or Nexus, or at a folder of files, and it lists every package version there. Then it downloads each version, maps every file inside it, and writes a CBOM (a cryptographic bill of materials) for it, on this machine. No team has to change its build pipeline, and the sweep remembers what it has done, so the next run builds only the versions that are new.

A sweep takes three steps:

1. Write a registry configuration file that says where the artifacts are.
2. Run spice registry discover to list the package versions into a discovery file, and edit it to choose what to build.
3. Run spice registry cbom --process to build the CBOMs.

Run every command from one folder that holds the configuration file, the discovery file and the output. The wrapper makes that folder, and every file you name on the command line, visible to the CLI’s container; paths inside the configuration file are read relative to that folder.

Write a registry configuration file

The file is TOML. Each [[repositories]] table is one repository: its format (the package type), the server’s url, the repository’s name on that server, and which product serves it. For a Maven repository on Sonatype Nexus:

[[repositories]]
id = "corp-maven"                     # your name for it, used in output
format = "maven"
url = "https://nexus.example.internal"
repository = "maven-releases"

[repositories.host]
kind = "nexus"

[repositories.host.credential]
type = "basic"
username = "survey-bot"
password = { file = "nexus-password.txt" }

Save it as registry.toml. The password is read from nexus-password.txt in the same folder, so the configuration file holds no secret. A secret can also be written in the file as a plain string, which is simplest to test with but is then in the file for anyone who can read it.

For JFrog Artifactory, url includes the context path, and Artifactory also accepts a bearer token:

[[repositories]]
id = "corp-libs"
format = "maven"
url = "https://artifactory.example.internal/artifactory"
repository = "libs-release-local"

[repositories.host]
kind = "artifactory"

[repositories.host.credential]
type = "bearer"
token = { file = "artifactory-token.txt" }

format can be maven, npm, pypi, nuget, helm, rubygems, cargo, apt, rpm, conda and others; kind can be nexus, artifactory, gitlab, standard (any server that speaks only the format’s own protocol) and others. A repository that needs no sign-in leaves out the credential table. Add one [[repositories]] table for each repository to sweep.

Sweep a folder of files

To sweep files you already have, such as firmware images, device images or compiled binaries copied from elsewhere, use format = "local" with a path:

[[repositories]]
id = "device-images"
format = "local"
path = "artifacts"

Every file under artifacts/ is a package with one version, named by the first twelve hex digits of its SHA-256. Keep the folder inside the one you run spice from.

Discover what is there

spice registry discover --config registry.toml --output discovery.toml

This lists every package and its versions into discovery.toml. Without --output, the list is printed instead; add --max 20 to try a few packages first. The discovery file is a snapshot: run discover again to refresh it.

To choose what to build, edit the rules near the top of the file. Each pattern is matched against the package URL (such as pkg:maven/com.example/payments-api); with no rules, everything is built:

[[include]]
pattern = "pkg:maven/com.example/"

[[exclude]]
pattern = "-test"

Do not edit the package list below the rules; discover rewrites it.

Build the CBOMs

spice registry cbom --process --config registry.toml --discovery discovery.toml --output cboms

For each selected version, this downloads the artifacts from the repository, maps every file inside them, and writes the results to cboms/<namespace>/<name>/<version>/. Each version’s folder holds a cbom folder with the CBOM of the cryptographic material found, such as certificates, keys and algorithms, in CycloneDX JSON, beside the inventory of every file.

When the run ends, it prints a summary: how many packages it processed, how many artifacts it downloaded, how many CBOM files it wrote, and any errors. --threads 4 processes four versions at once.

Run it again

The sweep keeps track of every version it has processed, in a state folder it creates beside the configuration file. Run the same --process command again after discover has found new versions, and it builds only those. To see or reset what it remembers:

# List the versions already processed
spice registry cbom --config registry.toml --export-processed

# Forget one package, or everything, so the next --process builds it again
spice registry cbom --config registry.toml --reset --purl pkg:maven/com.example/payments-api
spice registry cbom --config registry.toml --reset

spice registry status --config registry.toml summarizes what the sweeps have done so far.

On Windows

The commands are the same in PowerShell. Run them from the folder that holds registry.toml, with Docker Desktop running:

spice registry discover --config registry.toml --output discovery.toml
spice registry cbom --process --config registry.toml --discovery discovery.toml --output cboms