# enkly-ai — Kubernetes deployment

Plain manifests (no Helm) for the 8 Go microservices + RabbitMQ + 4
Postgres databases + two CPU-only vLLM inference servers, for the
`kmaster` / `kworker1` / `kworker2` cluster described in
`docker-compose.yml`. One file per concern, applied individually or
via `kustomization.yaml`.

## vLLM on a cluster with no GPU

An earlier version of this manifest set ran `vllm-chat`/`vllm-embed`
on the CUDA `vllm/vllm-openai` image, each requesting
`nvidia.com/gpu: "1"`. **This cluster has no physical GPU on any of
its three nodes.** `nvidia.com/gpu: "1"` is not a resource any node
here can ever advertise — with no node reporting it, both pods sat
`Pending` permanently (`Insufficient nvidia.com/gpu`), not just slow
to schedule.

`11-vllm.yaml` runs vLLM's CPU backend instead — same two Deployments,
same OpenAI-compatible API, same `--api-key` enforcement, no
application code changes. Two real, *observed* problems (not just
theoretical GPU-vs-CPU ones) shaped the rest of the fix:

1. **`avx512f` genuinely missing on this cluster's nodes.** vLLM's
   official pre-built CPU image
   (`public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo`) hard-requires
   that flag and crashes with `Illegal instruction` without it — and
   an init container added to both Deployments confirmed it's absent
   here, consistent with VirtualBox/Vagrant nested virtualization
   (this project's own `vagrant-provisioning/` setup) historically not
   exposing AVX-512 to guest VMs at all. There's no runtime flag for
   this, only a build-time one, so `image:` on both Deployments now
   points at a **custom build you must create first**:
   `registry.enkly.org/enkly/vllm-cpu:latest`, built with
   `--build-arg VLLM_CPU_DISABLE_AVX512=true` — see `11-vllm.yaml`'s
   header for the exact command. That build falls back to plain AVX2,
   universal on any x86_64 node since ~2013, so the init container now
   only checks for that (a much lower, already-satisfied bar) instead
   of gating on `avx512f`.
2. **The chat model's memory request alone exceeded a node's total
   RAM.** `10-rabbitmq.yaml`'s memory-watermark fix already
   established each node has ~7949Mi (~7.76Gi) total — `vllm-chat`'s
   original `8Gi` request could never be satisfied by *any* node here,
   which is exactly why it sat `Pending` instead of crashing. Fixed by
   dropping the chat model to `Qwen/Qwen2.5-1.5B-Instruct` (~3GB bf16
   vs. the 3B variant's ~6GB) and cutting
   `resources`/`VLLM_CPU_KVCACHE_SPACE`/`--max-model-len` on both
   Deployments to fit comfortably inside ~7.76Gi alongside everything
   else already running on that node (`vllm-chat` requests `4Gi`/limits
   `6Gi` — sized to the realistic weights + KV-cache + runtime-overhead
   total, not the bare minimum, so there's headroom left to the limit
   during startup warmup).
3. **A shared `ReadWriteMany` PVC was an unverified assumption, not a
   confirmed fix.** `vllm-chat` and `vllm-embed` originally bind-mounted
   one PVC for their Hugging Face cache. `managed-nfs-storage`
   (`nfs-subdir-external-provisioner`) does advertise RWX, so that
   probably binds fine — but "probably" produces exactly the kind of
   pod that's silently stuck `Pending` on an unbound PVC, easy to
   mis-diagnose as a repeat of the GPU/memory issues above. Split into
   `vllm-chat-cache` and `vllm-embed-cache`, both plain `ReadWriteOnce`
   — the same access mode every other PVC in this project already binds
   with — so neither Deployment's scheduling can depend on the other's
   claim or on RWX ever having been exercised here.

If real GPU nodes get added to this cluster later: point `image:` on
both Deployments back at `vllm/vllm-openai`, add back
`nvidia.com/gpu` requests + a GPU-pool `nodeSelector`/`toleration` +
an NVIDIA device-plugin DaemonSet, and size the model/resources back
up.

## Other issues found in the manifest set and how they were fixed here

1. **Postgres on NFS-backed PVCs fails `initdb`.**
   `nfs-subdir-external-provisioner` volumes aren't always empty at
   the Postgres data-dir mount point, and `initdb` refuses to
   initialize a non-empty directory. All four databases set
   `PGDATA=/var/lib/postgresql/data/pgdata` (a subdirectory of the
   mount) to sidestep this — see `12-databases.yaml`.
2. **`agent`/`knowledge` never received the vLLM API key.** Both
   services' own code (`services/agent/cmd/server/main.go`,
   `services/knowledge/cmd/server/main.go`) reads
   `ENKLY_VLLM_API_KEY` from the environment and defaults to `""` if
   unset — but neither `24-knowledge.yaml` nor `25-agent.yaml` sourced
   it from `enkly-secrets`. Since `11-vllm.yaml` starts both vLLM
   servers with `--api-key=$(VLLM_API_KEY)`, every chat/embedding call
   from these two services would get rejected with `401`. Both files
   now set `ENKLY_VLLM_API_KEY` from `secretKeyRef: {name:
   enkly-secrets, key: ENKLY_VLLM_API_KEY}`.
3. **`02-secrets.yaml` had what read as live credentials committed as
   if they were placeholders.** `STRIPE_SECRET_KEY`/
   `STRIPE_PUBLISH_KEY` were filled with what look like real Stripe
   test-mode keys, and `ENKLY_SMTP_USERNAME`/`ENKLY_SMTP_PASSWORD`
   held a real-looking Gmail address and a 16-character app password.
   **Rotate the Gmail app password and the Stripe keys if this file
   was ever pushed anywhere outside a local sandbox** — replacing the
   values here doesn't undo prior exposure. Replace all four with
   your own before applying. Separately, `ENKLY_VLLM_API_KEY` should
   be a real random value before applying — YAML `stringData` doesn't
   run shell commands, so a literal `"$(openssl rand -hex 32)"` string
   would become the actual, non-random key; see that key's comment in
   `02-secrets.yaml` for the correct one-liner.
4. **Registry auth/TLS is cluster-level, not manifest-level.** Every
   Deployment references `imagePullSecrets: [enkly-registry]`, but
   that Secret must be created out-of-band (real credentials can't
   live in a file meant for a repo — see `02-secrets.yaml`'s header
   comment). Likewise, if `registry.enkly.org` serves a self-signed or
   internal CA cert, every node's containerd needs that CA trusted via
   `/etc/containerd/certs.d/registry.enkly.org/hosts.toml` — that's
   node provisioning, outside what any Kubernetes manifest can
   express.

## Layout

| File | What |
|---|---|
| `00-namespace.yaml` | the `enkly` namespace |
| `01-config.yaml` | non-secret `ENKLY_*` config, shared by every service |
| `02-secrets.yaml` | **template** — DB/AMQP/Stripe/SMTP/HF secret + instructions for the registry pull secret |
| `10-rabbitmq.yaml` | RabbitMQ, with a PVC (so broker state survives a pod restart) |
| `11-vllm.yaml` | vLLM chat + embed Deployments/Services, CPU backend (no GPU), each with its own RWO Hugging Face cache PVC — see above |
| `12-databases.yaml` | iam-db, billing-db, conversation-db, knowledge-db (pgvector), each with its own PVC |
| `20`–`27` | iam, billing, conversation, workspace, knowledge, agent, notification, gateway |
| `30-ingress.yaml` | Ingress for the gateway (+ commented NodePort fallback) |
| `kustomization.yaml` | ties the above together for `kubectl apply -k .` |

Every Deployment's env vars are a 1:1 mapping from
`docker-compose.yml` — service hostnames are identical to the compose
service names, so the only real change the application sees is DNS
resolving inside a namespace instead of a compose network.

## Deploy

Two things have to be true *before* any apply, because nothing in
Kubernetes can discover them on its own: your image registry has to
have the 8 service images, and your real Stripe/SMTP credentials have
to come from you.

```bash
# 1. Build & push the 8 images
REG=registry.enkly.org/enkly
for svc in iam billing conversation workspace knowledge agent notification gateway; do
  docker build -f services/$svc/Dockerfile -t $REG/$svc:latest .
  docker push $REG/$svc:latest
done

# 2. Registry pull secret (every Deployment references this)
kubectl create secret docker-registry enkly-registry -n enkly \
  --docker-server=registry.enkly.org \
  --docker-username=<user> --docker-password=<token> --docker-email=<email>

# 3. Edit 02-secrets.yaml: fill in STRIPE_*/ENKLY_SMTP_* with your own
#    rotated values (see issue #3 above), and generate a real
#    ENKLY_VLLM_API_KEY (the file documents the exact one-liner).

# 4. Build & push the CPU-only, no-AVX512 vLLM image — this cluster's
#    nodes were confirmed to lack avx512f, so the official pre-built
#    image (public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo) will crash
#    with "Illegal instruction"; 11-vllm.yaml's image: already points
#    at the custom tag this produces:
git clone https://github.com/vllm-project/vllm.git /tmp/vllm-build && cd /tmp/vllm-build
docker build -f docker/Dockerfile.cpu \
  --build-arg VLLM_CPU_DISABLE_AVX512=true \
  --tag registry.enkly.org/enkly/vllm-cpu:latest \
  --target vllm-openai .
docker push registry.enkly.org/enkly/vllm-cpu:latest

# 5. Apply everything
kubectl apply -k k8s/

# 6. Watch the rollout — first boot downloads ~3.5GB of Hugging Face
#    weights total onto vllm-chat-cache/vllm-embed-cache, so the
#    startupProbe budget is generous.
kubectl -n enkly get pods -w
kubectl -n enkly rollout status deploy/vllm-chat --timeout=15m
kubectl -n enkly rollout status deploy/vllm-embed --timeout=15m
kubectl -n enkly rollout status deploy/gateway
```

Then point DNS/`/etc/hosts` at the Ingress controller's address for
the host set in `30-ingress.yaml` / `ENKLY_BASE_URL` in
`01-config.yaml`, or use the commented NodePort Service in
`30-ingress.yaml` if no ingress controller is installed
(`kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.11.3/deploy/static/provider/baremetal/deploy.yaml`
installs one on bare metal).

### Verify the LLM path specifically

```bash
kubectl -n enkly logs deploy/vllm-chat -f   # watch the HF download + CPU backend warmup
kubectl -n enkly logs deploy/vllm-embed -f
kubectl -n enkly port-forward svc/gateway 8080:8080   # then run a chat turn end to end
```

If a pod is stuck in `Init:Error` / `Init:CrashLoopBackOff`, that's
the `check-cpu-avx2` init container — this cluster already needed the
custom no-AVX512 image (see above), so seeing this now would mean a
node lacks even AVX2, which no mainstream x86_64 CPU built since
~2013 should be missing; read its log for which node and double-check
`lscpu` there directly. If instead the main `vllm-chat`/`vllm-embed`
container itself is crashing after the init container passed, check
for `OOMKilled` in `kubectl -n enkly describe pod` specifically (this
cluster's nodes only have ~7.76Gi RAM total each — raise
`resources.limits.memory` cautiously, if at all, and prefer lowering
`VLLM_CPU_KVCACHE_SPACE` or `--max-model-len` instead) over assuming
it's a repeat of the AVX-512 issue.

## Scaling beyond the defaults

`iam`, `billing`, `conversation`, `knowledge`, `agent`, `notification`
and `gateway` are stateless and safe to scale (`kubectl -n enkly scale
deploy/gateway --replicas=3`). `workspace`, the four Postgres
Deployments, `vllm-chat` and `vllm-embed` are all pinned to
`replicas: 1` with `strategy: Recreate` because each owns a
`ReadWriteOnce` PVC — don't raise their replica counts without first
either moving to a clustered/replicated backend (e.g. Patroni for
Postgres) or, for the vLLM pair specifically, converting
`vllm-chat-cache`/`vllm-embed-cache` to `ReadWriteMany` (cheap to try:
`managed-nfs-storage` already advertises it) and switching
`strategy.type` to `RollingUpdate`. Either way, remember each replica
is a full, independent CPU-bound process (no GPU to share), so scaling
trades throughput for a proportional increase in CPU/RAM footprint,
not a free win the way request-level load balancing sometimes is.