Weekly engineering notes — what shipped, what was decided, what's next.
The morning after we demoed at Canopy, I pulled up the waitlist over coffee on Sansome Street. Yesterday's signups were there. Today's weren't. I opened the site and got a server error. That made no sense; the site is static, and a static page can't throw a server error. The only thing that could was the form service behind it. It was down, and every signup that day had gone straight to nowhere.
- Swapped it for Listmonk, an open-source list manager. One binary on a box we already run. Signups go into our own database now.
- No third-party service sitting in that path anymore. The old one threw 521s and never told us; we'd rather not relive that.
- We're fine renting things we can watch. This one we couldn't see fail, and that's the part that cost us.
Next: bring back the longer form and wire the email side, then go looking for whatever else we won't notice when it breaks.
- Operator endpoints shipped. Bind is loopback; cross-host access is a deliberate operator choice, not a default. Closed a tracked invariant violation while we were there.
- Alert bundle shipped. Each rule maps to a counter or gauge that exists for its purpose. Conservative thresholds; we tighten when production tells us which ones page usefully.
- Killed a legacy counter that overlapped with newer inventory metrics. Less drift surface to keep honest.
- Divergence-signal event for cases where intent and outcome diverge. Auditable cleanly without coupling to the state-machine walk.
- Wrote and shipped the canonical system overview doc. Legacy architecture doc is marked acknowledged-stale and kept for git-blame continuity.
- Self-review on the prep work caught a doc/code contradiction one of the changes had introduced. Atomic fix.
- Canopy Festival demo prep: booth path tested end-to-end, fallback recording ready if the network bounces.
Next: post-demo conversations. Then capacity testing — first numbers we can stand behind in public.
- VC demo midweek. Two days of prep. Three live runs through the week. Cut scope twice. Recorded a fallback in case the box bounced. Demo ran clean; two follow-up conversations scheduled.
- Wrote the spec for a public system overview doc. Reader is an external technical leader who reads it cold; the writing constraint is that no follow-up conversation should be required.
- Profiled the agent under sustained load. Allocation pressure traced to a snapshot-copy hot path; fix in branch, lands next week with the operator endpoints.
- Started the metrics registry refactor on a branch. Tests now run with their own registry instead of colliding on the global; merge sequenced with the endpoint shipping.
- Decided against publishing a public metrics surface alongside the system doc. Classification catalog stays internal; the right shape of public observability waits on what operators ask for first.
Next: ship the documented endpoints, the alerting, and the system doc.
- Spec'd the operator endpoint surface: response shape, bind discipline, empty-projection contract. Lands the week after.
- Spec'd the alert bundle. Hardest call was which states deserve a stuck-state alert and which are legitimately long-running. Three excluded explicitly, with reasons inline.
- Triaged seven open issues. Took four for immediate work; deferred three with named trigger conditions instead of TBDs.
- Reviewed an external team's proposal that would have silently collapsed our audit-trail emit sites if applied naively. Wrote up the conflict with specifics so the conversation has a shared artifact.
- Read four AWS postmortems on Lambda-adjacent incidents to recalibrate alert thresholds for our first production weeks.
Next: ship the spec'd work, top to bottom.
- VM lifecycle refactored from ad-hoc booleans to a declared state model. The transition table is canonical; an event the table rejects fails loudly instead of silently mutating state.
- Closed-enum reasons for every classifiable failure on the VM side. New values fail a snapshot test until they're explicitly added.
- Structured event emitter on the VM side, with a published wire format that backs both observability and operator queries.
- Same shape on the job side: parallel state model, parallel emitter, same closed-enum discipline.
- Threaded the emitter through the executor end-to-end. Every transition in production goes through the same code path the tests cover.
- Closed a long-standing gap where one of the active states was tracked outside the state model. Everything walks the model now.
- vsock connection serialization fix found by the race detector under contention. We had been getting lucky; the test makes that explicit.
- Honored the configured boot timeout in the wait-for-boot path. Earlier code path swallowed it.
- Capped tarball size on the host at 100 MiB. Generous for our workloads, bounds host exposure to attacker-controlled inputs.
Next: operator-facing views, and the alerting bundle.
Two threads. Half the week on network controls, half on writing decisions down before they walk out the door.
- Per-VM bandwidth shaping via netlink. Verified from inside a real VM, not at the host side of the link.
- Per-source connection rate limits and conntrack caps. Cheap; we should have shipped them earlier.
- Made Unbound the sole DNS resolver on the host. Standard hardening.
- Intercepted guest DNS via DNAT. Every guest lookup walks through our resolver regardless of what the guest's
resolv.conf claims. Made resolv.conf immutable inside the guest while we were there.
- Narrowed the DNAT rule after the test caught some host-internal traffic the first draft would have touched.
- Stale-namespace cleanup on agent startup. Production found this for us, on a host that had been kicked by oom-killer mid-job.
- Wrote the first five ADRs into a directory in the repo. Backfilled reasoning behind decisions made over the last two months while it's still fresh.
- Added a gotchas file capturing operational surprises that aren't in any one commit message.
- Shipped a preflight script for repeatable agent deploys. Replaces a checklist that used to live in a colleague's head.
Next: state machine refactor. Spec is mostly written; the cuts are next.
Closing audit findings and standing up the daemon.
- Bridge-veth topology shipped after a week chasing TAP connectivity. The diagnostics path we built finding the bug stays in the repo.
- Closed several host-isolation vectors flagged by the audit. Each one has a test that runs every CI cycle.
- Locked the root account and disabled serial-console autologin in the guest rootfs.
- Pinned the guest interface to a stable name. The kernel was renaming it under us depending on module load order.
- Stood up the agent daemon under systemd. Eight commits to satisfy the linter — gci and gofumpt have opinions and we took them.
- Found a Postgres URL parsing bug inherited from the v1 codepath. Build the URL from env vars directly now.
- Started the job execution layer. Compatibility adapter for v1 in place.
- Configured the agent to treat each VM as single-use for the initial release. Reuse paths stay implemented; flipping them on waits for an audit.
Next: per-VM bandwidth shaping, rate limiting verified from inside a real boot.