enkly-ai/ARCHITECTURE.md

# Architecture

This document records the bounded-context decomposition of the
**enkly-ai** monolith, why each boundary is drawn where it is, the
communication rules enforced throughout, and the handful of places
where splitting a single process into eight forced a small, deliberate
change in behavior.

## 1. Bounded contexts (the context map)

The monolith's packages were grouped by **technical layer**
(`internal/handlers`, `internal/db`, `internal/llm`, ...); this port
regroups them by **business capability**, each with its own data and
its own deployable:

```
                         ┌─────────────┐
                         │   gateway   │  (HTTP/HTMX edge — no DB)
                         └──────┬──────┘
             ┌──────────┬───────┼────────┬──────────┬──────────┐
             ▼          ▼       ▼         ▼          ▼          │
         ┌───────┐  ┌───────┐┌──────┐┌─────────┐┌─────────┐     │
         │  iam  │  │billing││ conv │ │workspace││knowledge│     │
         └───┬───┘  └───┬───┘└──────┘└─────────┘└────┬────┘     │
             │          │                             │          │
             │ events   │ events                      │          ▼
             ▼          ▼                              │    ┌───────┐
        ┌──────────────────┐                            └──▶│ agent │
        │   notification    │                                └───┬───┘
        └──────────────────┘                                     │
                                          (agent also calls billing,
                                           conversation, workspace,
                                           knowledge — see §3)
```

| Context | Aggregate(s) | Replaces (monolith) |
|---|---|---|
| **IAM** | User, Session, VerificationCode, PasswordResetToken | `internal/handlers/auth.go`, `internal/db/users.go`, `internal/auth/*` |
| **Billing** | Subscription, StripeEvent, DailyUsage | `internal/handlers/billing.go`, `internal/billing/*`, `internal/db/subscriptions.go` |
| **Conversation** | Conversation, Message, ToolCall | `internal/db/conversations.go` |
| **Workspace** | (the sandboxed filesystem itself) | `internal/llm/tools.go` + `sandbox.go`, `internal/handlers/files.go` |
| **Knowledge** | CodeChunk (pgvector document) | `internal/rag/*` |
| **Agent** | Stream (an in-flight run) — a saga coordinator, owns no table | `internal/llm/agent.go` + `client.go`, the run-orchestration half of `internal/handlers/chat.go` |
| **Notification** | (stateless consumer) | the email-sending half of `internal/handlers/auth.go`, `internal/mail/*` |
| **Gateway (BFF)** | (no domain state) | `internal/handlers/{pages,ui,deliverables,middleware}.go`, `internal/render/*` |

**Why Agent is its own context, not folded into Workspace or
Conversation:** the agent run loop is a process that *uses* workspace
(tool execution), knowledge (semantic search) and conversation
(persistence) to do its job, but owns none of their data. That's a
textbook application service / saga, not an aggregate of any of the
three — giving it its own deployable keeps "what tools exist and how a
turn is orchestrated" independently scalable and independently
restartable from "where files live" or "what was said."

**Why Workspace is separate from Agent**, even though today only Agent
calls it: the sandboxed filesystem is also directly reachable by a
human through the gateway's upload/download/explorer panel. Coupling
it to Agent would mean every file-browsing click goes through the LLM
orchestration service for no reason, and would block scaling
"concurrent agent runs" independently of "workspace storage."

**Why rendering lives only in Gateway:** `pkg/renderkit` (Markdown,
syntax highlighting, code-snippet extraction) and every `html/template`
file are a presentation concern, not a domain concept — no bounded
context's *business rule* depends on HTML. Agent streams plain text and
structured tool-call data; Gateway is the one place that turns that
into HTML. This is also what makes Agent's gRPC contract
(`api/proto/agent/v1/agent.proto`) reusable by a future non-HTML client
(a CLI, a different front end) without change.

## 2. Database-per-service

Every context that owns data gets its **own Postgres database** (not
just its own schema/tables in a shared instance):

- `iam` → `enkly_iam` (users, sessions, email_verification_codes, password_reset_tokens)
- `billing` → `enkly_billing` (subscriptions, stripe_events, usage_daily)
- `conversation` → `enkly_conversation` (conversations, messages, tool_calls)
- `knowledge` → `enkly_knowledge` (documents, pgvector)

`workspace` has no database — the sandboxed directory tree itself
*is* its data, on its own Docker volume.

**Consequence worth naming explicitly:** the monolith's `schema.sql`
had `user_id BIGINT REFERENCES users(id) ON DELETE CASCADE` on
`conversations`, `subscriptions`, and `usage_daily`. Postgres cannot
enforce a foreign key across two separate databases, so every one of
those is now a bare `user_id BIGINT` with **no cascade delete**. The
monolith never actually exposed an "delete my account" feature, so no
behavior is lost today — but if one is added later, deleting a user
must become a saga (iam deletes the user, then publishes a
`user.deleted` event for billing/conversation to react to and delete
their own rows), not a single SQL statement. This is the standard,
unavoidable cost of database-per-service and is called out here so it
isn't rediscovered the hard way.

## 3. Communication rules

> *"gRPC for synchronous communication, RabbitMQ for asynchronous
> communication"* — applied throughout as: **if the caller needs an
> answer (or needs to know the write succeeded) before it can proceed,
> it's a gRPC call; if the caller doesn't need to wait for the result,
> it's an event.**

### Synchronous (gRPC) — `api/proto/*/v1/*.proto`

Every cross-context call that's part of a request's critical path:

- Gateway → IAM: session validation on (almost) every request, login, signup, ...
- Gateway → every other service: composing a page from their data
- Agent → Billing: check-and-increment the daily quota before a run starts
- Agent → Conversation: persist each message/tool-call as a turn proceeds
- Agent → Workspace: every filesystem/bash tool call
- Agent → Knowledge: the `search_codebase` tool
- `dispatch_subagent` makes **no new outbound call of its own kind**: it runs a second,
  nested `agentrun.Agent` in-process, sharing the Agent service's own vLLM-backed client and
  making the *same* Workspace/Knowledge gRPC calls an ordinary tool call would, just via a
  `toolexec.restrictedExecutor` that only forwards the read-only subset
  (`read_file`/`list_directory`/`glob_search`/`grep`/`search_codebase`). No new service, no
  new proto — see `services/agent/internal/infrastructure/toolexec/subagent.go`.
- `solve_ml_math` (like `write_todos`) makes **no** outbound call at all: it's pure
  computation over `pkg/mathkit`, a dependency-free numerical engine (linear algebra,
  calculus/gradients, statistics, optimization, ML primitives) that runs entirely inside
  the agent process — see `pkg/mathkit`'s package doc and
  `services/agent/internal/infrastructure/toolexec/mlmath.go`.
- Agent `StreamTurn`: a **server-streaming** RPC — the gRPC replacement
  for the monolith's Server-Sent Events, carrying the same
  token/thinking/tool-call/done event sequence; the gateway re-emits
  each message as an actual SSE frame to the browser (see
  `services/gateway/internal/handlers/chat.go`).

### Asynchronous (RabbitMQ) — `pkg/eventbus`

One topic exchange (`enkly.events`); routing keys of the form
`<context>.<aggregate>.<event>`. Today's only real consumer is
`notification`, bound to:

- `iam.user.verification_code_issued` → send the signup/resend email
- `iam.user.password_reset_requested` → send the reset-link email

Also published, with no consumer yet (kept so the *pattern* — every
aggregate commit gets an event — is consistent across every context,
and so a future service can bind without a producer-side change):
`iam.user.registered`, `iam.user.email_verified`,
`billing.subscription.changed`, `conversation.message.created`.

**Trust boundary of the event bus:** `VerificationCodeIssued` and
`PasswordResetRequested` carry the raw code/token on the bus, not just
an ID. This is a deliberate trade-off (documented in
`pkg/eventbus/events/iam_events.go`) rather than an oversight: it lets
`notification` send the exact email the monolith sent without needing
read access to `iam`'s token tables (which would violate
database-per-service far more seriously than this does). It's safe
*only* because RabbitMQ is never exposed outside the private service
network — the same trust level the monolith implicitly gave its own
in-process function call. A production hardening pass should put this
exchange behind TLS + per-service credentials (RabbitMQ supports both)
even on a private network, as defense in depth.

## 4. One Go module, many deployables

Every service shares a single `module enkly` (one `go.mod` at the repo
root) rather than one `go.mod` per service. This is a deliberate,
pragmatic choice: true microservice *independence* comes from separate
processes, separate databases, separate containers, and a versioned
API contract between them (all of which this repo has) — not from
separate Go dependency graphs, which mainly adds `replace`-directive
and version-skew overhead in a monorepo without buying any of the
actual deployability benefits. Each service still has its own
`cmd/server/main.go`, its own `Dockerfile` that builds only that one
binary, and starts, scales, and fails independently of the others.

## 5. Layering within a service

```
internal/domain/            entities + repository interfaces (port). No sql, no net/http, no generated gRPC code.
internal/application/       use cases. Depends on domain only. Transport-agnostic — unit-testable without a server.
internal/infrastructure/    adapters: Postgres repos, the Stripe/vLLM HTTP clients, the sandboxed filesystem.
internal/interfaces/grpc/   the ONLY files that import this service's generated `gen/<ctx>/v1` package.
```

Dependency direction is inward: `interfaces` and `infrastructure` both
depend on `domain` (through its interfaces); `domain` depends on
nothing. This is what lets, e.g., `services/iam/internal/application`
be tested with a fake `UserRepository` and no database at all.

## 6. What's verified, what isn't

Every `.proto` file in `api/proto/` was compiled for real with
`protoc` + the actual `protoc-gen-go`/`protoc-gen-go-grpc` plugins —
`gen/` is genuine generated code, not hand-written stand-ins. Every
hand-written `.go` file in this repo passes `gofmt -l` (verified file
by file as it was written).

What this port could **not** do, for lack of outbound network access
to `proxy.golang.org`/`google.golang.org` in the authoring sandbox: run
`go build`/`go mod tidy` end-to-end across the whole dependency graph
(`google.golang.org/grpc`, `github.com/lib/pq`,
`github.com/rabbitmq/amqp091-go`, `github.com/pgvector/pgvector-go`,
`github.com/gomarkdown/markdown`), and so no `go.sum` is committed.
Every service's Dockerfile therefore runs `go mod tidy` at build time
(after copying in `gen/`, `pkg/`, and that service's own directory) to
resolve and lock the graph on first build — this needs ordinary
internet access, same as any other Go project, and is expected to
succeed on any normal developer machine or CI runner. `make build`
(plain `go build`, no Docker) needs a `go.sum` to exist first, so run
`go mod tidy` at the repo root once before using it locally. Once a
`go.sum` is committed, each Dockerfile's `go mod tidy` line should be
swapped back to `go mod download` (noted inline in
`services/iam/Dockerfile`) for faster, better-cached rebuilds.

The same constraint applied to the later Ollama→vLLM migration
(`pkg/vllmkit`, replacing `pkg/ollamakit`): `pkg/vllmkit` itself has no
dependency beyond the Go standard library, so it was verified for real
— `go build`/`go test` both pass in an isolated module — but every
file that imports it still depends on the same ungenerated `go.sum`,
so those files could only be checked with `gofmt -l` (a full syntax
parse; it does not resolve imports or type-check across packages), not
a full `go build ./...`. Run `go build ./...` (or `make build`) as the
first gate on that change in an environment with normal network
access, before trusting it beyond what `gofmt` already confirms. See
`CHANGES.md`'s most recent entry for the full migration writeup.

## 7. Known gaps / simplifications

Called out here rather than silently shipped, each with why it's safe
to defer:

- **Session validation latency.** `gateway`'s `RequireAuth` now makes
  one `IAM.ValidateSession` gRPC call per request, where the monolith
  did one local Postgres query in the same process. For most
  deployments this is low-single-digit-millisecond overhead on an
  internal network and is the correct place to start; if it becomes a
  bottleneck, the standard fix is a short-TTL cache in the gateway (or
  switching sessions to signed, self-contained tokens) — not
  re-merging the services.
- **Reattaching to a run after a page reload.** The monolith could show
  a still-in-flight reply immediately on reload
  (`StreamManager.activeFor`). The gateway doesn't re-wire this today
  (it would need a new "is there an active run for this conversation"
  Agent RPC with no other caller) — a reload mid-reply shows the
  completed answer on next load instead of resuming the live view. The
  reply itself is unaffected; this is UI polish, not correctness.
- **Deleting a conversation mid-run.** The monolith proactively
  cancelled any in-flight stream for a conversation being deleted.
  Agent's API has no "stop by conversation" RPC (only by `stream_id`),
  so a straggling run's writes will fail (logged) against the deleted
  conversation rather than being pre-empted. Low-impact and easy to
  close later by adding that RPC.
- **Verification/reset email confirmation.** Because sending email is
  now asynchronous (RabbitMQ), `iam`'s `SignUp`/`ForgotPassword` RPCs
  can no longer report back "the email failed to send" synchronously
  the way the monolith occasionally could — they report success once
  the code/token is persisted and the event is published. This is the
  expected, intentional cost of decoupling a side effect onto a
  message bus, not a bug.
- **Per-call bash timeout.** `WorkspaceService.RunBashCommand` accepts
  a `timeout_seconds` field for forward-compatibility, but (matching
  the monolith, which only ever had one process-wide timeout) does not
  yet apply it per call.
- **Chat attachments aren't a schema field.** The composer's "Add
  files or photos" (`services/gateway/internal/handlers/attachments.go`)
  uploads into the workspace under `attachments/` and folds the
  resulting paths into the prompt string itself, as a trailing
  `[Attached files]` block — `conversation.Message` has no attachments
  field of its own, and adding one would be a schema change across the
  conversation/gateway boundary. The trade-off: attachments only ever
  round-trip as a specific text shape inside `content`, parsed back out
  by `renderUserMessage`, instead of being structured data the
  conversation service understands natively. Also: removing a staged
  chip before sending, or never sending at all, leaves the uploaded
  file behind in the workspace — nothing currently reaps attachments
  that were uploaded but never attached to a sent message.
- **Zip attachments extract one level deep.** A `.zip` attached to the
  chat is unpacked into `attachments/<name>/` (gateway-side, with its
  own zip-slip guard on top of workspace's existing path jail, plus
  caps on entry count and total uncompressed size) so the agent's
  existing `read_file`/`list_directory`/`grep`/`glob_search` tools can
  read what's inside it — no new agent tool needed. A `.zip` nested
  inside that `.zip` is written out as-is rather than recursively
  extracted; `unzip` isn't in `DefaultAllowedCommands`
  (`services/workspace/internal/infrastructure/sandboxfs/sandbox.go`),
  so the agent can't shell out to unpack it further unless an operator
  adds it to `ENKLY_BASH_ALLOWLIST`.
- **Language preference storage.** The language switcher (`pkg/i18n`,
  `services/gateway/internal/handlers/i18n.go`) persists a visitor's
  chosen UI locale in a year-long `enkly_lang` cookie, not on the IAM
  `User` row. That was a deliberate choice, not an oversight: adding a
  `preferred_language` field means extending `iam.proto`'s `User`
  message and an RPC, which means regenerating every language's
  generated client/server code under `gen/` — this repo's `gen/`
  directory is checked in and hand-maintained, and doing that
  regeneration safely needs `protoc` and the project's `buf`/codegen
  toolchain, which isn't always available in every environment this
  code is edited in. A cookie needs neither: it survives logins and
  browser restarts, just not a switch to a second device. The same
  reasoning is why the visitor's locale reaches the agent service as
  gRPC metadata (`x-enkly-lang`, set in `chat.go`'s `withLangMetadata`,
  read in `services/agent/internal/interfaces/grpc/server.go`'s
  `langFromIncomingContext`) rather than as a new `StartTurnRequest`
  field. Moving either onto the User row/proto later is a clean,
  backward-compatible addition — `localeFrom`/`langFromIncomingContext`
  are the only two places that would need a new "check the account
  first" branch — just not one to do without `protoc` on hand to
  verify the regenerated code.
- **UI translation coverage.** `pkg/i18n` ships English and French
  complete; Spanish, German and Portuguese cover the nav/auth/composer
  chrome and fall back to English for the rest (quota/error copy, the
  subscription and files panels) — see `pkg/i18n/locales/*.json` and
  `T`'s fallback behavior in `pkg/i18n/i18n.go`. `subscription.html`
  and `files.html` aren't wired to the switcher yet (no `Lang` field on
  their page-data structs). None of this affects how the agent itself
  talks to a user — see `agentrun.SystemPrompt`'s "## Language" section
  and `agentrun.WithPreferredLanguage`, which work from any locale
  regardless of how much of the surrounding chrome has a catalog entry
  for it.

## 8. Operational hardening before a real rollout

Deliberately out of scope for this restructuring (it's infrastructure
polish orthogonal to the DDD boundary work), but worth a checklist:

- mTLS between services (today: plaintext gRPC on a private Docker
  network — fine inside a single trusted VPC, not fine across one)
- RabbitMQ publisher confirms + a dead-letter exchange for poison
  messages (today: at-least-once with a bounded retry via `Nack`)
- Replacing the in-memory `authsupport.Limiter`/`domain.Manager`
  (agent's stream registry) with something shared across replicas
  (Redis) once any service runs more than one instance
  — both are currently per-process, same as the monolith's own
  in-memory limiter was