Skip to content

Initial go_modules table and mcp_servers refactor/fixes - #132

Merged
grahamgilbert merged 10 commits into
macadmins:mainfrom
arubdesu:go_modules
Oct 29, 2026
Merged

grahamgilbert merged 10 commits into
macadmins:mainfrom
arubdesu:go_modules

Conversation

@arubdesu

@arubdesu arubdesu commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds a go_modules table, the Go module inventory (without resolving actually built versions or traversing/recursing the cache) proposed in #129, built to the schema, discovery, limits and acceptance criteria agreed there. The new table and mcp_servers share project discovery, credential redaction and constraint handling through packages under pkg/. mcp_servers also gets fixes that decoupling that code turned up, and the same 'completeness' model when it comes to communicating the state of how successful an inventory run was, and both get a new workflow that runs their tests on GitHub's Ubuntu, macOS and Windows runners.

Deferred/not in this pull request

  • Reading/recursing the Go module cache, a walk from a named root, and parsing go.work workspaces are intentionally left out as discussed. Other future 'supply chain' tables with better user guide docs for query writers/admins/etc in this series may necessitate or increase our confidence that this can be done strategically and responsibly, but they don't need to be in the v1.
  • The osquery-go bump. It fixes the transport each client opens and adds the column options, and will follow as its own pull request (which will need some fixes adapted to the other older tables, signing myself up for that work as well).

mcp_servers table fix details

Schema. Changes are additive:

  • a uid column - mostly for parity when writing queries as used in core with the pattern LEFT JOIN users USING (uid), but specifically for Windows to also better differentiate from the pre-existing user_id (which remains holding the SID and remains the same across Mac/Linux) to instead just be the normal shortened RID user account identification
  • the column order in a pattern I chose for go_modules, with the meat of what we're looking for/pertinent details/'item' columns first, and the correlation/account and 'completeness' columns last
  • five new warning codes (none are removed or renamed) that shook out when reviewing the shared implementation across tables

Behavior.

  • Completeness. Both tables use one model: scan_complete describes a row's own files, and a problem finding a user's configuration is a warning row that changes no 'server' row. The README lists every warning code in the array with its sentence, its details and its conditional effect on scan_complete.
  • Discovery. The per-account project cap rises from 200 to 1,000, which is also what go_modules uses (per-table, not pooled) with the devclients.MaxProjects constant. Multi-root VS Code workspaces are now read. A configuration file or project list that is a symbolic link is followed one hop on macOS and Linux when its target is in the same home (an exception considered worthwhile since this is how dotfile managers tend to create them). The Windows implementation declines links and refuses a junctioned parent directory altogether, also one project directory that two tools recorded with upper/lower case variation are read once because duh. Projects on a network filesystem that the home is not on are counted and not read. A home on a network filesystem has its projects read, and on Linux 9p is not treated as a network filesystem, because WSL2 mounts the Windows drives over it. A Windows project path in extended-length (\\?\C:\…) or device-namespace (\\.\C:\…) form, which Codex records, is read as the local drive's path; \\?\UNC\… is still counted as another host. (Yes these are progressively very excessively esoteric details.)
  • Launcher options. To report which package a server runs, such as the one npx -y @acme/mcp-server@1.2.3 downloads, the table has to skip the launcher's options, and some options take a value. Each launcher (npm/npx, uv, pipx, bun, Docker, podman) now has its own list of options, taken from that tool's source or CLI. An option the table doesn't recognize leaves the package columns empty with an unknown_launcher_option warning rather than a guess, except for npx, which itself treats an unknown option as taking the next word.
  • Plugins. A Claude Code plugin's manifest can point at other files that declare servers. The table reads at most 64 of those per pass and skips overlong paths, and a problem in one plugin's manifest marks only that plugin's rows incomplete.
  • Approval and trust. The settings files that record which project servers a user approved are read only for projects that declare servers. "mcpServers": null counts as no servers, but a file that is only null is reported as unreadable. If Codex's config lists one project twice with different trust, it counts as untrusted.
  • Panics. Both tables are registered through utils.RecoverPanics, so a panic reaches osquery as fixed text and never as the panic value.

Approach

There is one row for each module version that a project on this host requires in go.mod, records a hash for in go.sum, or both. The table inventories declarations and checksum records. It does not compute the build list or verify downloaded/referenced content independently. A requirement is a minimum under minimal version selection, and go.sum can keep versions that are no longer needed, so a 'match' would still e.g. require being confirmed with the go command in that project.

SELECT project_path, version, mod_relationship FROM go_modules
WHERE module_path = 'golang.org/x/net' AND scan_complete = 1;
  • Discovery. Golang code repos/projects come from the lists that IDE's/software clients/agents record, as in mcp_servers, and the tables to-date do not walk directory trees. The sources are:

    • the VS Code family's workspaceStorage records, including multi-root .code-workspace files
    • JetBrains IDEs' recentProjects.xml
    • ~/.claude.json
    • ~/.codex/config.toml

    Only the project root is read. A root go.work is reported as go_workspace_unsupported, and modules in subdirectories are not read.

  • Parsing. It uses golang.org/x/mod v0.37.0, a new dependency in go.mod, go.sum and deps.bzl. The strict go.mod parser runs first, and the lenient parser runs only to recover requirements, reported as go_mod_partial. Module paths and versions are checked against their formats before they are published. A value that contains a credential format with a distinctive prefix (carefully considered to avoid common/well-known-enough false-positives) is dropped.

  • Limits. Each go.mod and go.sum is capped at 2 MiB and 5,000 entries. A scan follows at most 1,000 recorded projects per account. A query returns at most 50,000 module rows and 32 MiB, within one MACADMINS_EXTENSION_WALK_TIMEOUT budget for all accounts, which is checked before each file. Each limit that is reached produces a warning row.

  • Completeness. scan_complete = 1 marks a row whose project's go.mod and go.sum were read in full. A problem finding a user's projects is a warning row for that user, chosen from a fixed list, and it changes no module row. Projects in cloud storage, outside the home, or on a network filesystem the home is not on are counted and not read.

The table README has the column reference, every warning code, the limits, and what the project lists do not cover.

Operating system support

Actually exercised/confirmed working (~once, on my machinelol, please let me know if I should include a ci-local.sh I'm using for the ubuntu container and macOS testing setup):

  • [✅] macOS(27) / Windows(11) / Linux(ubuntu-latest)

Testing/Validation

.github/workflows/native-tests.yml runs the tests for pkg/..., go_modules and mcp_servers on ubuntu-latest, macos-latest and windows-latest. It runs on push and pull request, as test.yml and lint.yml do. On Windows it doesn't use the race detector, and it also skips the not-applicable pkg/utils TestExecCmd* tests there, which run echo, cat, sleep, sh, etc.

As mentioned above, ci-local.sh harnesses support local/native testing, with Ubuntu/macOS handled from a host with Docker-compatible tooling (like OrbStack) installed, and has been manually QA'd thorough-ish including in osqueryi on Win11. On macOS osqueryi, when run as an unprivileged user (2026-10-15), returned 55 module rows for that user and three diagnostic rows: root's unreadable home, an account whose home does not exist, and four recorded projects outside the home.

  • The lint, test and build checks and the three native jobs pass on the fork/my branch.
  • Not yet run: osqueryd as root on any platform, and osqueryi on a Linux host.

Checklist

  • I have read the Mac Admins contribution guidance: https://macadmins.io/contributing/
  • This change was discussed in an issue before code was submitted, unless it is a small documentation fix, typo fix, or clearly isolated bug fix.
  • [🤥] I understand the code I am submitting and can explain how it works.
  • This change fits the existing direction, style, and maintenance expectations of the project.
  • I have considered operating system support, including Linux support where practical.
  • I have added tests, updated tests, or explained why tests are not applicable.
  • I have documented any known limitations or follow-up work.

Addresses macadmins#129 : a go.mod/go.sum inventory per recorded project. Moves project discovery, credential redaction and constraint handling into pkg/devclients, pkg/redact and pkg/utils; adds uid and the item-first column order to both tables; documents every warning code; runs the tests natively on Ubuntu, macOS and Windows; stops mcp_servers from putting a panic value in the error osquery logs.
…ed in macadmins#129, still open but syncing for windows local test running
… warning counts, further mcp_server and testing tweaks
@arubdesu arubdesu changed the title Go modules Initial go_modules table and mcp_servers refactor/fixes Oct 28, 2026
Comment thread tables/mcp_servers/identity.go Resolved
@grahamgilbert
grahamgilbert merged commit 8011605 into macadmins:main Oct 29, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants