This commit is contained in:
enkly
332
ARCHITECTURE.md
Normal file
332
ARCHITECTURE.md
Normal file
@@ -0,0 +1,332 @@
|
||||
# Architecture
|
||||
|
||||
This document records the bounded-context decomposition of the
|
||||
**enkly-ai** monolith, why each boundary is drawn where it is, the
|
||||
communication rules enforced throughout, and the handful of places
|
||||
where splitting a single process into eight forced a small, deliberate
|
||||
change in behavior.
|
||||
|
||||
## 1. Bounded contexts (the context map)
|
||||
|
||||
The monolith's packages were grouped by **technical layer**
|
||||
(`internal/handlers`, `internal/db`, `internal/llm`, ...); this port
|
||||
regroups them by **business capability**, each with its own data and
|
||||
its own deployable:
|
||||
|
||||
```
|
||||
┌─────────────┐
|
||||
│ gateway │ (HTTP/HTMX edge — no DB)
|
||||
└──────┬──────┘
|
||||
┌──────────┬───────┼────────┬──────────┬──────────┐
|
||||
▼ ▼ ▼ ▼ ▼ │
|
||||
┌───────┐ ┌───────┐┌──────┐┌─────────┐┌─────────┐ │
|
||||
│ iam │ │billing││ conv │ │workspace││knowledge│ │
|
||||
└───┬───┘ └───┬───┘└──────┘└─────────┘└────┬────┘ │
|
||||
│ │ │ │
|
||||
│ events │ events │ ▼
|
||||
▼ ▼ │ ┌───────┐
|
||||
┌──────────────────┐ └──▶│ agent │
|
||||
│ notification │ └───┬───┘
|
||||
└──────────────────┘ │
|
||||
(agent also calls billing,
|
||||
conversation, workspace,
|
||||
knowledge — see §3)
|
||||
```
|
||||
|
||||
| Context | Aggregate(s) | Replaces (monolith) |
|
||||
|---|---|---|
|
||||
| **IAM** | User, Session, VerificationCode, PasswordResetToken | `internal/handlers/auth.go`, `internal/db/users.go`, `internal/auth/*` |
|
||||
| **Billing** | Subscription, StripeEvent, DailyUsage | `internal/handlers/billing.go`, `internal/billing/*`, `internal/db/subscriptions.go` |
|
||||
| **Conversation** | Conversation, Message, ToolCall | `internal/db/conversations.go` |
|
||||
| **Workspace** | (the sandboxed filesystem itself) | `internal/llm/tools.go` + `sandbox.go`, `internal/handlers/files.go` |
|
||||
| **Knowledge** | CodeChunk (pgvector document) | `internal/rag/*` |
|
||||
| **Agent** | Stream (an in-flight run) — a saga coordinator, owns no table | `internal/llm/agent.go` + `client.go`, the run-orchestration half of `internal/handlers/chat.go` |
|
||||
| **Notification** | (stateless consumer) | the email-sending half of `internal/handlers/auth.go`, `internal/mail/*` |
|
||||
| **Gateway (BFF)** | (no domain state) | `internal/handlers/{pages,ui,deliverables,middleware}.go`, `internal/render/*` |
|
||||
|
||||
**Why Agent is its own context, not folded into Workspace or
|
||||
Conversation:** the agent run loop is a process that *uses* workspace
|
||||
(tool execution), knowledge (semantic search) and conversation
|
||||
(persistence) to do its job, but owns none of their data. That's a
|
||||
textbook application service / saga, not an aggregate of any of the
|
||||
three — giving it its own deployable keeps "what tools exist and how a
|
||||
turn is orchestrated" independently scalable and independently
|
||||
restartable from "where files live" or "what was said."
|
||||
|
||||
**Why Workspace is separate from Agent**, even though today only Agent
|
||||
calls it: the sandboxed filesystem is also directly reachable by a
|
||||
human through the gateway's upload/download/explorer panel. Coupling
|
||||
it to Agent would mean every file-browsing click goes through the LLM
|
||||
orchestration service for no reason, and would block scaling
|
||||
"concurrent agent runs" independently of "workspace storage."
|
||||
|
||||
**Why rendering lives only in Gateway:** `pkg/renderkit` (Markdown,
|
||||
syntax highlighting, code-snippet extraction) and every `html/template`
|
||||
file are a presentation concern, not a domain concept — no bounded
|
||||
context's *business rule* depends on HTML. Agent streams plain text and
|
||||
structured tool-call data; Gateway is the one place that turns that
|
||||
into HTML. This is also what makes Agent's gRPC contract
|
||||
(`api/proto/agent/v1/agent.proto`) reusable by a future non-HTML client
|
||||
(a CLI, a different front end) without change.
|
||||
|
||||
## 2. Database-per-service
|
||||
|
||||
Every context that owns data gets its **own Postgres database** (not
|
||||
just its own schema/tables in a shared instance):
|
||||
|
||||
- `iam` → `enkly_iam` (users, sessions, email_verification_codes, password_reset_tokens)
|
||||
- `billing` → `enkly_billing` (subscriptions, stripe_events, usage_daily)
|
||||
- `conversation` → `enkly_conversation` (conversations, messages, tool_calls)
|
||||
- `knowledge` → `enkly_knowledge` (documents, pgvector)
|
||||
|
||||
`workspace` has no database — the sandboxed directory tree itself
|
||||
*is* its data, on its own Docker volume.
|
||||
|
||||
**Consequence worth naming explicitly:** the monolith's `schema.sql`
|
||||
had `user_id BIGINT REFERENCES users(id) ON DELETE CASCADE` on
|
||||
`conversations`, `subscriptions`, and `usage_daily`. Postgres cannot
|
||||
enforce a foreign key across two separate databases, so every one of
|
||||
those is now a bare `user_id BIGINT` with **no cascade delete**. The
|
||||
monolith never actually exposed an "delete my account" feature, so no
|
||||
behavior is lost today — but if one is added later, deleting a user
|
||||
must become a saga (iam deletes the user, then publishes a
|
||||
`user.deleted` event for billing/conversation to react to and delete
|
||||
their own rows), not a single SQL statement. This is the standard,
|
||||
unavoidable cost of database-per-service and is called out here so it
|
||||
isn't rediscovered the hard way.
|
||||
|
||||
## 3. Communication rules
|
||||
|
||||
> *"gRPC for synchronous communication, RabbitMQ for asynchronous
|
||||
> communication"* — applied throughout as: **if the caller needs an
|
||||
> answer (or needs to know the write succeeded) before it can proceed,
|
||||
> it's a gRPC call; if the caller doesn't need to wait for the result,
|
||||
> it's an event.**
|
||||
|
||||
### Synchronous (gRPC) — `api/proto/*/v1/*.proto`
|
||||
|
||||
Every cross-context call that's part of a request's critical path:
|
||||
|
||||
- Gateway → IAM: session validation on (almost) every request, login, signup, ...
|
||||
- Gateway → every other service: composing a page from their data
|
||||
- Agent → Billing: check-and-increment the daily quota before a run starts
|
||||
- Agent → Conversation: persist each message/tool-call as a turn proceeds
|
||||
- Agent → Workspace: every filesystem/bash tool call
|
||||
- Agent → Knowledge: the `search_codebase` tool
|
||||
- `dispatch_subagent` makes **no new outbound call of its own kind**: it runs a second,
|
||||
nested `agentrun.Agent` in-process, sharing the Agent service's own vLLM-backed client and
|
||||
making the *same* Workspace/Knowledge gRPC calls an ordinary tool call would, just via a
|
||||
`toolexec.restrictedExecutor` that only forwards the read-only subset
|
||||
(`read_file`/`list_directory`/`glob_search`/`grep`/`search_codebase`). No new service, no
|
||||
new proto — see `services/agent/internal/infrastructure/toolexec/subagent.go`.
|
||||
- `solve_ml_math` (like `write_todos`) makes **no** outbound call at all: it's pure
|
||||
computation over `pkg/mathkit`, a dependency-free numerical engine (linear algebra,
|
||||
calculus/gradients, statistics, optimization, ML primitives) that runs entirely inside
|
||||
the agent process — see `pkg/mathkit`'s package doc and
|
||||
`services/agent/internal/infrastructure/toolexec/mlmath.go`.
|
||||
- Agent `StreamTurn`: a **server-streaming** RPC — the gRPC replacement
|
||||
for the monolith's Server-Sent Events, carrying the same
|
||||
token/thinking/tool-call/done event sequence; the gateway re-emits
|
||||
each message as an actual SSE frame to the browser (see
|
||||
`services/gateway/internal/handlers/chat.go`).
|
||||
|
||||
### Asynchronous (RabbitMQ) — `pkg/eventbus`
|
||||
|
||||
One topic exchange (`enkly.events`); routing keys of the form
|
||||
`<context>.<aggregate>.<event>`. Today's only real consumer is
|
||||
`notification`, bound to:
|
||||
|
||||
- `iam.user.verification_code_issued` → send the signup/resend email
|
||||
- `iam.user.password_reset_requested` → send the reset-link email
|
||||
|
||||
Also published, with no consumer yet (kept so the *pattern* — every
|
||||
aggregate commit gets an event — is consistent across every context,
|
||||
and so a future service can bind without a producer-side change):
|
||||
`iam.user.registered`, `iam.user.email_verified`,
|
||||
`billing.subscription.changed`, `conversation.message.created`.
|
||||
|
||||
**Trust boundary of the event bus:** `VerificationCodeIssued` and
|
||||
`PasswordResetRequested` carry the raw code/token on the bus, not just
|
||||
an ID. This is a deliberate trade-off (documented in
|
||||
`pkg/eventbus/events/iam_events.go`) rather than an oversight: it lets
|
||||
`notification` send the exact email the monolith sent without needing
|
||||
read access to `iam`'s token tables (which would violate
|
||||
database-per-service far more seriously than this does). It's safe
|
||||
*only* because RabbitMQ is never exposed outside the private service
|
||||
network — the same trust level the monolith implicitly gave its own
|
||||
in-process function call. A production hardening pass should put this
|
||||
exchange behind TLS + per-service credentials (RabbitMQ supports both)
|
||||
even on a private network, as defense in depth.
|
||||
|
||||
## 4. One Go module, many deployables
|
||||
|
||||
Every service shares a single `module enkly` (one `go.mod` at the repo
|
||||
root) rather than one `go.mod` per service. This is a deliberate,
|
||||
pragmatic choice: true microservice *independence* comes from separate
|
||||
processes, separate databases, separate containers, and a versioned
|
||||
API contract between them (all of which this repo has) — not from
|
||||
separate Go dependency graphs, which mainly adds `replace`-directive
|
||||
and version-skew overhead in a monorepo without buying any of the
|
||||
actual deployability benefits. Each service still has its own
|
||||
`cmd/server/main.go`, its own `Dockerfile` that builds only that one
|
||||
binary, and starts, scales, and fails independently of the others.
|
||||
|
||||
## 5. Layering within a service
|
||||
|
||||
```
|
||||
internal/domain/ entities + repository interfaces (port). No sql, no net/http, no generated gRPC code.
|
||||
internal/application/ use cases. Depends on domain only. Transport-agnostic — unit-testable without a server.
|
||||
internal/infrastructure/ adapters: Postgres repos, the Stripe/vLLM HTTP clients, the sandboxed filesystem.
|
||||
internal/interfaces/grpc/ the ONLY files that import this service's generated `gen/<ctx>/v1` package.
|
||||
```
|
||||
|
||||
Dependency direction is inward: `interfaces` and `infrastructure` both
|
||||
depend on `domain` (through its interfaces); `domain` depends on
|
||||
nothing. This is what lets, e.g., `services/iam/internal/application`
|
||||
be tested with a fake `UserRepository` and no database at all.
|
||||
|
||||
## 6. What's verified, what isn't
|
||||
|
||||
Every `.proto` file in `api/proto/` was compiled for real with
|
||||
`protoc` + the actual `protoc-gen-go`/`protoc-gen-go-grpc` plugins —
|
||||
`gen/` is genuine generated code, not hand-written stand-ins. Every
|
||||
hand-written `.go` file in this repo passes `gofmt -l` (verified file
|
||||
by file as it was written).
|
||||
|
||||
What this port could **not** do, for lack of outbound network access
|
||||
to `proxy.golang.org`/`google.golang.org` in the authoring sandbox: run
|
||||
`go build`/`go mod tidy` end-to-end across the whole dependency graph
|
||||
(`google.golang.org/grpc`, `github.com/lib/pq`,
|
||||
`github.com/rabbitmq/amqp091-go`, `github.com/pgvector/pgvector-go`,
|
||||
`github.com/gomarkdown/markdown`), and so no `go.sum` is committed.
|
||||
Every service's Dockerfile therefore runs `go mod tidy` at build time
|
||||
(after copying in `gen/`, `pkg/`, and that service's own directory) to
|
||||
resolve and lock the graph on first build — this needs ordinary
|
||||
internet access, same as any other Go project, and is expected to
|
||||
succeed on any normal developer machine or CI runner. `make build`
|
||||
(plain `go build`, no Docker) needs a `go.sum` to exist first, so run
|
||||
`go mod tidy` at the repo root once before using it locally. Once a
|
||||
`go.sum` is committed, each Dockerfile's `go mod tidy` line should be
|
||||
swapped back to `go mod download` (noted inline in
|
||||
`services/iam/Dockerfile`) for faster, better-cached rebuilds.
|
||||
|
||||
The same constraint applied to the later Ollama→vLLM migration
|
||||
(`pkg/vllmkit`, replacing `pkg/ollamakit`): `pkg/vllmkit` itself has no
|
||||
dependency beyond the Go standard library, so it was verified for real
|
||||
— `go build`/`go test` both pass in an isolated module — but every
|
||||
file that imports it still depends on the same ungenerated `go.sum`,
|
||||
so those files could only be checked with `gofmt -l` (a full syntax
|
||||
parse; it does not resolve imports or type-check across packages), not
|
||||
a full `go build ./...`. Run `go build ./...` (or `make build`) as the
|
||||
first gate on that change in an environment with normal network
|
||||
access, before trusting it beyond what `gofmt` already confirms. See
|
||||
`CHANGES.md`'s most recent entry for the full migration writeup.
|
||||
|
||||
## 7. Known gaps / simplifications
|
||||
|
||||
Called out here rather than silently shipped, each with why it's safe
|
||||
to defer:
|
||||
|
||||
- **Session validation latency.** `gateway`'s `RequireAuth` now makes
|
||||
one `IAM.ValidateSession` gRPC call per request, where the monolith
|
||||
did one local Postgres query in the same process. For most
|
||||
deployments this is low-single-digit-millisecond overhead on an
|
||||
internal network and is the correct place to start; if it becomes a
|
||||
bottleneck, the standard fix is a short-TTL cache in the gateway (or
|
||||
switching sessions to signed, self-contained tokens) — not
|
||||
re-merging the services.
|
||||
- **Reattaching to a run after a page reload.** The monolith could show
|
||||
a still-in-flight reply immediately on reload
|
||||
(`StreamManager.activeFor`). The gateway doesn't re-wire this today
|
||||
(it would need a new "is there an active run for this conversation"
|
||||
Agent RPC with no other caller) — a reload mid-reply shows the
|
||||
completed answer on next load instead of resuming the live view. The
|
||||
reply itself is unaffected; this is UI polish, not correctness.
|
||||
- **Deleting a conversation mid-run.** The monolith proactively
|
||||
cancelled any in-flight stream for a conversation being deleted.
|
||||
Agent's API has no "stop by conversation" RPC (only by `stream_id`),
|
||||
so a straggling run's writes will fail (logged) against the deleted
|
||||
conversation rather than being pre-empted. Low-impact and easy to
|
||||
close later by adding that RPC.
|
||||
- **Verification/reset email confirmation.** Because sending email is
|
||||
now asynchronous (RabbitMQ), `iam`'s `SignUp`/`ForgotPassword` RPCs
|
||||
can no longer report back "the email failed to send" synchronously
|
||||
the way the monolith occasionally could — they report success once
|
||||
the code/token is persisted and the event is published. This is the
|
||||
expected, intentional cost of decoupling a side effect onto a
|
||||
message bus, not a bug.
|
||||
- **Per-call bash timeout.** `WorkspaceService.RunBashCommand` accepts
|
||||
a `timeout_seconds` field for forward-compatibility, but (matching
|
||||
the monolith, which only ever had one process-wide timeout) does not
|
||||
yet apply it per call.
|
||||
- **Chat attachments aren't a schema field.** The composer's "Add
|
||||
files or photos" (`services/gateway/internal/handlers/attachments.go`)
|
||||
uploads into the workspace under `attachments/` and folds the
|
||||
resulting paths into the prompt string itself, as a trailing
|
||||
`[Attached files]` block — `conversation.Message` has no attachments
|
||||
field of its own, and adding one would be a schema change across the
|
||||
conversation/gateway boundary. The trade-off: attachments only ever
|
||||
round-trip as a specific text shape inside `content`, parsed back out
|
||||
by `renderUserMessage`, instead of being structured data the
|
||||
conversation service understands natively. Also: removing a staged
|
||||
chip before sending, or never sending at all, leaves the uploaded
|
||||
file behind in the workspace — nothing currently reaps attachments
|
||||
that were uploaded but never attached to a sent message.
|
||||
- **Zip attachments extract one level deep.** A `.zip` attached to the
|
||||
chat is unpacked into `attachments/<name>/` (gateway-side, with its
|
||||
own zip-slip guard on top of workspace's existing path jail, plus
|
||||
caps on entry count and total uncompressed size) so the agent's
|
||||
existing `read_file`/`list_directory`/`grep`/`glob_search` tools can
|
||||
read what's inside it — no new agent tool needed. A `.zip` nested
|
||||
inside that `.zip` is written out as-is rather than recursively
|
||||
extracted; `unzip` isn't in `DefaultAllowedCommands`
|
||||
(`services/workspace/internal/infrastructure/sandboxfs/sandbox.go`),
|
||||
so the agent can't shell out to unpack it further unless an operator
|
||||
adds it to `ENKLY_BASH_ALLOWLIST`.
|
||||
- **Language preference storage.** The language switcher (`pkg/i18n`,
|
||||
`services/gateway/internal/handlers/i18n.go`) persists a visitor's
|
||||
chosen UI locale in a year-long `enkly_lang` cookie, not on the IAM
|
||||
`User` row. That was a deliberate choice, not an oversight: adding a
|
||||
`preferred_language` field means extending `iam.proto`'s `User`
|
||||
message and an RPC, which means regenerating every language's
|
||||
generated client/server code under `gen/` — this repo's `gen/`
|
||||
directory is checked in and hand-maintained, and doing that
|
||||
regeneration safely needs `protoc` and the project's `buf`/codegen
|
||||
toolchain, which isn't always available in every environment this
|
||||
code is edited in. A cookie needs neither: it survives logins and
|
||||
browser restarts, just not a switch to a second device. The same
|
||||
reasoning is why the visitor's locale reaches the agent service as
|
||||
gRPC metadata (`x-enkly-lang`, set in `chat.go`'s `withLangMetadata`,
|
||||
read in `services/agent/internal/interfaces/grpc/server.go`'s
|
||||
`langFromIncomingContext`) rather than as a new `StartTurnRequest`
|
||||
field. Moving either onto the User row/proto later is a clean,
|
||||
backward-compatible addition — `localeFrom`/`langFromIncomingContext`
|
||||
are the only two places that would need a new "check the account
|
||||
first" branch — just not one to do without `protoc` on hand to
|
||||
verify the regenerated code.
|
||||
- **UI translation coverage.** `pkg/i18n` ships English and French
|
||||
complete; Spanish, German and Portuguese cover the nav/auth/composer
|
||||
chrome and fall back to English for the rest (quota/error copy, the
|
||||
subscription and files panels) — see `pkg/i18n/locales/*.json` and
|
||||
`T`'s fallback behavior in `pkg/i18n/i18n.go`. `subscription.html`
|
||||
and `files.html` aren't wired to the switcher yet (no `Lang` field on
|
||||
their page-data structs). None of this affects how the agent itself
|
||||
talks to a user — see `agentrun.SystemPrompt`'s "## Language" section
|
||||
and `agentrun.WithPreferredLanguage`, which work from any locale
|
||||
regardless of how much of the surrounding chrome has a catalog entry
|
||||
for it.
|
||||
|
||||
## 8. Operational hardening before a real rollout
|
||||
|
||||
Deliberately out of scope for this restructuring (it's infrastructure
|
||||
polish orthogonal to the DDD boundary work), but worth a checklist:
|
||||
|
||||
- mTLS between services (today: plaintext gRPC on a private Docker
|
||||
network — fine inside a single trusted VPC, not fine across one)
|
||||
- RabbitMQ publisher confirms + a dead-letter exchange for poison
|
||||
messages (today: at-least-once with a bounded retry via `Nack`)
|
||||
- Replacing the in-memory `authsupport.Limiter`/`domain.Manager`
|
||||
(agent's stream registry) with something shared across replicas
|
||||
(Redis) once any service runs more than one instance
|
||||
— both are currently per-process, same as the monolith's own
|
||||
in-memory limiter was
|
||||
Reference in New Issue
Block a user