Case 01 · Seed-stage AI data platform · Oct 2025 — present

The platform layer of an agent-native data platform

The client builds an agent-native data platform for science and private data — scoped data connections, domain tools, knowledge, and evaluations with an analyst agent on top. SoulTech was brought in at seed stage to turn a contractor-built proof of concept into a platform a real engineering team could build a company on, and has led the technical direction of that work since.

Where it started

A proof-of-concept monolith built by contractors before the engineering team existed. No developer platform, no infrastructure standards, no service boundaries — and one long-lived executor doing every piece of compute the product needed.

What we own

The platform layer and its technical direction: the developer platform and shared infrastructure, the decomposition into services, the event-driven compute and control planes, and the security, release, and operability standards the team builds against.

Where it is now

An agent-native data platform on a shared event fabric, in production since June 2026, with a measured scaling model to 25k users and a public CLI in beta — built and operated by an engineering team of eight.

The architecture

Three planes on one event fabric

The rule the whole design turns on is a simple one: large bytes move between the browser, object storage, and compute workers without detouring through chat servers or long-lived executor nodes. Everything else follows from refusing to let one machine's CPU, RAM, disk, and network become the platform's ceiling.

Control plane

product-api

User-facing API. Publishes root run + cancel.

event fabric

Shared EventBridge + SQS. Work routing, cancel delivery.

run-ledger

Durable run ledger. Consume-only read model.

control-plane

Reconciler framework. Policy plugins: tiering, timeouts.

↓ work routing↑ lifecycle · progress · terminal

Compute plane

compute-runtime

Planners · shard workers · reducers · finalizers. Executes work and emits events — deliberately non-authoritative for state.

↕ manifests · inputs · outputs

Data plane

artifacts-api

Manifests, output promotion, analysis-ready metadata.

object storage

Large-byte inputs, partials, durable worker-readable data.

Not shown: control-plane reads the run ledger over HTTP to find its policy triggers — the one deliberate non-bus edge in the design. The web client sits above product-api; large uploads go browser-direct to object storage, bypassing the API tier entirely.

Designed to support a 25k-user and 250 GB scaling goal, after aggressive load testing clarified what real-time chat workloads cost. The split between the two services is deliberate and asymmetric: compute-runtime owns every lever that changes how much capacity exists, and control-plane owns only where a run lands. The section that follows is how those two decisions fit together; the capability APIs described later are how they stay honest with each other.

In depth

Compute that has to answer while someone is waiting

The ledger below is a survey, and a survey fragments the one mechanism that mattered most. It is worth stating once, in order. Two co-operating controls put a machine underneath a user’s question fast enough that the question still feels answered — a purpose-built autoscaler that decides how much capacity exists, and a routing policy that decides which tier a run lands on when one is backed up. We designed and built both, and the interesting part is why they are deliberately not the same controller.

The constraint

A user asks a question in chat. Behind that question, code the agent generated seconds ago has to execute on a worker that does not exist yet, and the user is watching a cursor blink while it happens. Interactive latency is the product, so every second of scheduling is taken directly out of the experience.

The platform was scaling on AWS’s managed autoscaler — Application Auto Scaling target tracking against the ECS service, driven by a CloudWatch alarm on queue depth per running task. Measured properly, in a controlled in-VPC campaign, it took 360 seconds to react to a burst, against a prior assertion that 30 to 90 seconds was plausible. Four to twelve times optimistic — and the fleet design had already been justified against it.

The dominant term was never the machine. A container reaches running state in 38 to 50 seconds, and the worker process inside it boots in about twelve milliseconds. The delay was the decision: target tracking will not fire until a CloudWatch alarm has several sustained one-minute datapoints, and one minute is the finest granularity that alarm can have. The platform was not slow to start compute. It was slow to admit it needed any — and no amount of tuning a managed policy fixes a controller whose input resolution is coarser than the event being reacted to.

The mechanism

Segment the work so one shape cannot starve another

Four worker pools, each with its own queue, dead-letter queue, routing rule, service and task size — sized from measured job duration: 3.7 seconds average for platform tools called inside a chat turn, 5.7 seconds for dynamically generated compute, 108.9 seconds for heavy analysis. Without that split, one agentic burst of long analyses sits in front of every interactive request behind it.

The fourth pool is a security boundary, not a size class. Credentialed shell execution gets its own task role, its own queue and a runtime-capability set that is exclusive in both directions — the ordinary pools refuse credentialed work, and the credentialed pool refuses ordinary work.

Size the warm floor from measurement, not intuition

A cold start is 38 to 50 seconds, which is simply not available on an interactive path. The floor that avoids it was calculated: measured per-tier duration and traffic share, run through Little’s law, projected as a ladder from 5,000 to 25,000 users.

The result is the kind that only falls out of arithmetic. The interactive tool tier needs roughly one warm worker even at 25,000 users, because 3.7-second jobs barely accumulate concurrency. The generated-compute tier is the only real warm driver, climbing from two to eight. The heavy tier stays at a floor of zero and instead needs a very wide burst ceiling — it is the one place where waiting is acceptable, because the job itself runs for minutes.

Designed and built a replacement autoscaler

The fix was not faster containers — it was taking ownership of the decision. We designed and implemented a purpose-built autoscaler: a 256 MB Go Lambda that reads each tier’s queue depth and running count directly, computes that tier’s desired count itself, and writes it to the service. AWS’s managed policy was removed from the path rather than tuned, because the problem was its reaction model, not its configuration.

The control law is three steps, and the order matters: required capacity is depth divided by jobs per worker; clamp to the tier’s floor and ceiling; then clamp again to within five of what is currently running. Two of the inputs are where the correctness lives. Depth counts in-flight messages as well as visible ones, because visible-only collapses the fleet to its floor the instant workers pick work up. And running count is never added to demand — running workers are capacity, not work.

That second rule was learned the hard way. An earlier formulation added running count to depth and used running as the lower clamp; during a rolling deploy, where old and new tasks briefly overlap, that ratcheted the desired count upward by one every tick against an empty queue until it reached 432 tasks and exhausted the subnet’s free addresses. The regression is now pinned by a test that asserts a fleet above its own ceiling must drain, not hold. A second guard rejects a negative per-tick clamp at load time instead of coercing it, because that value inverts the clamp window and silently defeats the ceiling — and a Lambda that refuses to start freezes the fleet and trips an alarm, where a silently corrected one scales on a number nobody wrote.

The single symmetric clamp is the only damping in the system: no cooldown, no hysteresis, no smoothing. One number bounds how fast a burst can be answered and how fast an idle fleet drains. Because the scheduler’s floor is one minute, the Lambda loops eleven times at five-second spacing inside a single sixty-second invocation, skipping the trailing sleep so invocations can never overlap — a five-second decision cadence on a scheduler that only sells sixty. And because clamping to the ceiling is lossy exactly where an operator needs the truth, every tick also publishes the shortfall between demanded and permitted capacity, so a tier pinned at its maximum reports how far past it demand went.

Designed the traffic shift that covers what scaling cannot

Even a five-second decision cannot conjure a worker in under forty seconds. So we designed a second, independent control in the routing service to cover that window: it polls the same per-tier queues every twenty seconds, derives a saturation posture per tier from backlog measured against running workers instead of a flat depth, and at classification time moves a run onto a tier that has room. A backlog on one tier shifts new traffic to a more available one while the first tier’s own workers are still booting.

The policy around that shift is the careful part. A run moves up one tier only, never down, because a downgrade can run a job out of memory — and that property is enforced structurally: a misordered tier table is rejected when configuration is parsed, since a table in the wrong order would silently invert an upgrade into a downgrade. A shift only happens if the target tier actually has room. During a correlated multi-tier stall the run stays exactly where it is, because rerouting into an equally saturated queue gains nothing and feeds the hottest backlog. Nothing is ever rejected: an oversized request routes to the largest available tier and raises a flag rather than failing.

The posture is also published to consumers, and the published value and the reroute the classifier performs are the same function called twice against the same snapshot. The contract therefore cannot advertise a shift that will not happen — when a tier is saturated but has nowhere to go, consumers are told it is throttled, not that it is upgrading.

The reason this is a second mechanism and not a competing one is arithmetic we did before building it. The autoscaler reacts to roughly four outstanding messages within five seconds; the router would not shift traffic until around eighteen, after a twenty-second poll. It can therefore only ever engage inside the cold-start window, or when a tier is genuinely pinned at its ceiling. It is a bridge, not a controller — and two controllers on one signal would have fought each other.

Make scale-in safe enough that scaling hard is safe

Aggressive scaling is only responsible if removing a worker cannot destroy work. Load testing proved it could: fourteen in-flight jobs were killed across two staging runs, every one of them seventy to ninety seconds into a ninety-second job, by scale-in terminating the task underneath them.

Two layers fixed it. Tasks acquire scale-in protection for the duration of a job, reference-counted and bounded by a per-tier expiry sized just above that tier’s job ceiling, so a crashed worker is still reclaimed quickly. And because a serial worker briefly drops protection between jobs, the shutdown path also decouples the running job from the termination signal, letting it finish and acknowledge its own message instead of being redelivered.

The consequence worth noticing is the one that bites later: a protected task will not stop for a deployment either, so the rolling-deploy configuration has to allow a tier to run at twice its desired count during a release. At the default it would have deadlocked — waiting to stop a task that has been told not to stop.

360s

Measured reaction, managed autoscaling

~10s

Observed first scale-up, replacement

Autoscaler decision cadence

0.16s

Warm per-job overhead

12ms

Worker boot inside the container

The 360s baseline is measured. The replacement’s reaction is the observed first scale-up in a staging load test, asserted at ≤15s against a ≤30s milestone target — 5s is the decision cadence, not an end-to-end guarantee.

The division of labour is the part worth copying. One service owns everything that changes how much capacity exists — pools, floors, ceilings, the autoscaler itself. The other owns only where a run lands; it cannot start a worker and does not try to. When it reroutes, it is buying the forty seconds a new worker needs in order to exist.

The work

Eleven domains, ten months

Not a sequence — these ran concurrently, and the scaling work above cut across most of them.

Developer platform
(DevEx)

Built the developer platform from zero

Authored a Go CLI that scaffolds a production-ready service end to end — repo, GitHub environments, infrastructure stacks, CI/CD, OCI image pipeline — from eight golden templates. Paired it with a shared IaC library: 8 stacks, 13 reusable workflows, 30 composite actions, and 80 shared scripts across 15 operational domains, all consumed as versioned releases by repos across a 64-repo org.

The scripts hook straight into consumer Makefiles and workflows, so an engineer gets the capability — security scanning, runner fleet health, release cutting, telemetry, VPN and cloud setup — by including a target, not by learning how it works. New services go from init to deployed instead of days of hand-wired infrastructure, and the cognitive load of the platform stays with the platform.

Decomposition

Drove the monolith toward a service platform

Led the eight-phase extraction of the web app out of the monorepo into a standalone repo, replacing monorepo coupling with a build-time OpenAPI spec registry — one bucket in a dedicated platform account, served over PrivateLink and an internal load balancer with no public exposure, so an exact staging image promotes to production unchanged. Then split compute execution out of the API monolith into two new services, and stood up the platform's C4, standards, ADR, and break-glass documentation home.

Compute plane

Designed and shipped the event-driven execution plane

compute-runtime is worker-only: events → per-tier queues → segmented Fargate pools, tiered by what the work is, with the scaling model above sitting on top of it. The original public HTTP API and submission shim were removed once the event fabric carried everything, so a job has exactly one way in.

Underneath the tiering, the unglamorous properties that make a worker safe to run untrusted code on: panic recovery, output capture that streams in constant memory regardless of how much a child process produces, process-group kill on timeout, a non-root child uid with an unconditional environment scrub, and a loopback internal API with per-job tokens for script-local storage.

Measured,
then built

Measured the thing everyone had estimated

The scale-out campaign above is the clearest example of a habit worth naming on its own. The cold-start number had been asserted as “30 to 90 seconds plausible” and used as an architectural input. Measuring it properly — a controlled in-VPC campaign instead of reasoning from documentation — returned 360 seconds, four to twelve times worse, and the conclusion it supported was wrong as a result.

The same pass caught a policy bug nobody had noticed. The smallest tier was selected by a rule requiring a sub-30-second timeout, but nothing in the corpus configured one — the minimum was 120 seconds — so the tier was structurally unreachable and carried 0% of traffic while appearing healthy. Redefining it on code provenance turned it into a real lane at roughly 14% of runs across staging and production.

Neither of those is a clever fix. Both are the same move: measure the quantity the design depends on, before the design depends on it.

Control plane

Built a reconciler framework the routing policy lives in

control-plane runs the pattern poll durable state → apply policy → publish event on a tick loop, deliberately named for that shape so it can host policies that have nothing to do with scheduling. Its first tenant is the compute classifier described above.

Two of its reconcilers are the ones that keep the routing honest. A capabilities drift guard verifies that every policy ceiling the service publishes still fits inside the physical capacity the compute service reports, and it flags drift instead of adapting to it — serve config, flag loudly, never silently follow infrastructure. It fails open on any field the other service has not published, because a missing number and a genuine zero are identical on the wire.

The other is a supervisor that owns terminal state for stranded runs. The ledger is a passive event-sourced store and the chat tier only times out its own rows, so without it nothing in the platform could declare a lost run dead.

Closing
the loop

Told the model what it was actually running on

The agent had no authoritative description of its own execution runtime, so it guessed — dependencies that couldn't load, output-contract violations, and a tool prompt hardcoding “estimate runtime on a 16 GB machine” that was wrong for two of the three tiers.

Built capability APIs on both services under a strict domain rule — compose, never merge: compute-runtime publishes physical and image facts, control-plane publishes policy facts. The API tier polls both into Redis (no TTL — the cache is the durable last-known fallback) and renders a capability slice into the compute tool's model-facing description at every consumer boundary, so the agent frames jobs against real tier ceilings and they land on the right tier. Rendering is guardrailed against leaking routing posture or denylist contents, and emits in stable order so the provider's prompt cache still hits.

Security

Hardened the security posture org-wide

We own the platform's security and access architecture across a multi-account cloud organization — sandbox, dev, staging, production, and a dedicated platform account. That covers identity-center SSO, cross-account deployment roles, CI OIDC federation, client VPN with SAML and certificate auth, and the public/private network boundary of VPC endpoints, PrivateLink, and internal load balancers.

The least-privilege IAM layer is the part we are most deliberate about: ~1,900 lines of policy as code, backed by a resource registry that scans every stack for resource types and fails CI when a policy is missing an action a resource requires — so permission drift surfaces at build time instead of as a failed deploy or an over-broad grant nobody noticed. Org-wide across 64 repositories: rulesets, required checks, break-glass bypass, enforced job timeouts, and scoped App tokens in place of long-lived personal tokens, including a full migration to platform-native dependency and code scanning with repeatable per-repo onboarding.

At the service level: user code runs as a non-root uid with an unconditional child-env credential scrub and ambient-credential probe tests; and in the connections service, write-time SSRF guards plus authenticated connection probing, so a passing connection test proves the credentials actually worked.

Release
engineering

Automated release and change management across the fleet

Changelogs generated from commits at release time in six-plus repos, pushed by a scoped App composite, not a long-lived token. A CalVer release line for the platform API. A self-hosted CI runner fleet on autoscaling groups with an ephemeral ops pool, job-aware drain so a rolling refresh can't kill an in-flight job, and a deep fleet-health check across the cloud, CI, and instance planes. Plus a telemetry pipeline so the build tooling reports its own latency and failure modes.

Local
development

Owned the local-development and testing platform

A developer sidecar that orchestrates eleven services across switchable workspaces — dev-managed checkouts alongside tool-managed clones — with a dependency-graph UI, live health, log streaming and search, a cross-environment release dashboard, and synthetic browser plus k6 suites. Secrets resolve through a password manager at the moment they're needed, never at rest. Underneath it, a shared local cloud-emulation and Postgres fabric every service's local stack now builds on.

Public
surface

Shipped the public CLI

Empty repo to public beta in under six weeks: one Go binary that is both a CLI and a stdio MCP server, so any coding agent can drive platform workspaces directly. macOS, Linux, and Windows (native and WSL); keyring-backed tokens with an encrypted-file fallback; silent 401 recovery; and a checksum-verifying public installer.

Cross-service
delivery

Delivered bring-your-own-MCP end to end across three services

Customers connect their own MCP servers to the platform. The connections service got the source type, SSRF guard, and credential brokering; the API tier got the auth bridge, fan-out registration, and tool auto-attach; the web app got the connect wizard and management views. Designed the contract, wrote the handoff docs each repo worked from, then removed the feature gate to GA.

Systems owned

Primary or sole author

Counts as of 2026-08-20.

scaffolding CLI

Developer platform CLI · Go
95 commits · 45 PRs
sole primary author

The lifecycle owner for every service in the org. init scaffolds a new repo end to end from golden templates — project files, repo creation, environment configuration, infrastructure stack provisioning, first push. init-infra retrofits repos that predate the automation. pkg and release build and push OCI images through one standardized pipeline. common sync keeps every consumer converged on a pinned shared-IaC release.

shared-iac

Shared infrastructure & CI/CD · Pulumi, Bash, Actions
194 commits · 114 PRs
~70% of repo history

The org's shared infrastructure substrate: 8 stacks, 13 reusable workflows, 30 composite actions, and 80 shared scripts across 15 domains — plus runbooks, incident forensics, and managed agent skills shipped down to consumer repos. Every asset is versioned and pinned, so a consumer repo upgrades deliberately instead of drifting.

Reverse-generation with drift enforcement — golden templates are regenerated from real repos and drift-checked in CI, so templates can't silently rot away from what services actually run.
Intent-scoped migrations replacing three-way-merge template re-application, which real diverged repos can't do safely — shipped with a reusable apply harness with plan and preview-diff modes.
Self-hosted runner fleet — autoscaling-group runners with a canary tier and an ephemeral single-job ops pool, job-aware drain, per-job workspace hygiene, rolled canary-first. Diagnosed and fixed a disk-prune race deleting images mid-job.
Platform account + spec registry — a single artifact bucket behind an internal load balancer and PrivateLink endpoint service, replacing a per-environment CDN design that had made API specs an environment-coupled runtime concern instead of a build-time artifact.

compute-runtime

Event-driven code execution · Go
133 commits · 73 PRs
primary author

Executes untrusted Python for the platform's analyst agent. Worker-only as of milestone five — the original public HTTP API and MCP submission shim were removed; jobs now enter exclusively through the event fabric and per-tier queues.

Tier segmentation — small (platform tools inside interactive chat, trusted provenance, under 300s), standard (dynamically generated compute, and the catch-all), large (heavy analysis, beyond 1800s), plus a credentialed CLI tier. Each is its own service, queue, DLQ, and routing rule, and the catch-all rule is derived from the tier list, not hardcoded, so adding a tier cannot produce double-dispatch.
The measurement campaign — measured scale-out at 360s against an asserted estimate of “30–90s plausible.” The real figure was 4–12× worse and changed the architectural conclusion. A follow-up run showed raising the warm floor from 3 to 8 eliminated cold start entirely for a typical 20-job burst.
Reliability — graceful SIGTERM drain and task scale-in protection, after load testing proved scale-in was killing in-flight jobs mid-execution; SQS visibility heartbeats bounded by a maximum in-flight duration; panic recovery; O(1)-memory streaming output capture. An investigation into orphaned running jobs is worth naming for its conclusion more than its fix: the first draft blamed this service, and the correction proved it could not have been the source — the ledger row is owned upstream, and nothing there timed it out.
Security — non-root child uid, unconditional child-env scrub, process-group kill on timeout, and a loopback internal API with per-job HMAC tokens for script-local blob storage; scenario probes runnable against any environment.

control-plane

Reconciliation framework · Go
29 commits · 23 PRs
sole author

Owns policies that need periodic evaluation against durable state in other services, applies them by publishing events the owning service consumes, and dispatches downstream work onto the event bus. Deliberately named for its tick-based shape, not for compute, so the extension point is open to policies that have nothing to do with scheduling — today it carries one reconciler package, the compute classifier.

Two-lane classification — a resource-fit lane routes agent-marked intentional requests to the smallest enabled tier whose admission ceiling fits; everything else falls back to the timeout-and-provenance lane. Activation is traffic-driven, so there was no flag day.
Real-time saturation rerouting — a poller snapshots per-tier queue depth plus desired and running counts every 20s and derives a routing posture (open / throttled / upgrading). The published posture and the classify-time reroute are the same function called twice, so they cannot disagree. Staleness is judged per tier, so one failed poll carries its last-known load forward instead of flapping the busiest tier to open.
Capabilities drift guard — verifies every policy ceiling stays within real physical capacity; fails open on missing fields and flags drift instead of silently adapting. Serve config, flag loudly, never silently follow infra.
Operability — a run supervisor with recovery sweeps and a per-tick heartbeat that never reads green over a partial failure; alarms and runbooks shipped alongside the service, not after it. Every replica runs every reconciler, which makes posture telemetry a share-of-samples question, not a count — a distinction that shipped as a bug fix.

dev sidecar

Developer sidecar & test platform · TypeScript
118 commits · co-lead
1,200+ tests

The two things CI can't give an engineer: expensive tests that don't belong in every PR pipeline, and multi-workspace service management across an eleven-service fleet. Dependency graph in canonical startup order with cycle-safe layering, per-system sync, log history paging and cross-system search, health checks that resolve independently of local DNS, and a per-environment releases dashboard showing what's deployed, when, by whom, and from which branch. Along the way: root-caused a recurring editor lockup that had been costing days.

public CLI + distribution

Public CLI & MCP server · Go
57 commits · 16 PRs
empty repo → beta in 6 weeks

One static binary that is both a CLI and a stdio MCP server — no separately hosted MCP service — so a terminal user or a coding agent can drive workspace-scoped agents: ask questions, run analyses, generate reports, move files in and out. Keyring-backed credentials with an encrypted-file fallback and env-var override for headless use, refresh-token revocation on logout, silent mid-command 401 recovery, and Windows as a first-class target alongside macOS and Linux on both architectures.

This is the level of detail you would get on your own systems — written down, measured, and handed over.

Start a conversation