223 lines
12 KiB
Markdown
223 lines
12 KiB
Markdown
# enkly-ai — Kubernetes deployment
|
||
|
||
Plain manifests (no Helm) for the 8 Go microservices + RabbitMQ + 4
|
||
Postgres databases + two CPU-only vLLM inference servers, for the
|
||
`kmaster` / `kworker1` / `kworker2` cluster described in
|
||
`docker-compose.yml`. One file per concern, applied individually or
|
||
via `kustomization.yaml`.
|
||
|
||
## vLLM on a cluster with no GPU
|
||
|
||
An earlier version of this manifest set ran `vllm-chat`/`vllm-embed`
|
||
on the CUDA `vllm/vllm-openai` image, each requesting
|
||
`nvidia.com/gpu: "1"`. **This cluster has no physical GPU on any of
|
||
its three nodes.** `nvidia.com/gpu: "1"` is not a resource any node
|
||
here can ever advertise — with no node reporting it, both pods sat
|
||
`Pending` permanently (`Insufficient nvidia.com/gpu`), not just slow
|
||
to schedule.
|
||
|
||
`11-vllm.yaml` runs vLLM's CPU backend instead — same two Deployments,
|
||
same OpenAI-compatible API, same `--api-key` enforcement, no
|
||
application code changes. Two real, *observed* problems (not just
|
||
theoretical GPU-vs-CPU ones) shaped the rest of the fix:
|
||
|
||
1. **`avx512f` genuinely missing on this cluster's nodes.** vLLM's
|
||
official pre-built CPU image
|
||
(`public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo`) hard-requires
|
||
that flag and crashes with `Illegal instruction` without it — and
|
||
an init container added to both Deployments confirmed it's absent
|
||
here, consistent with VirtualBox/Vagrant nested virtualization
|
||
(this project's own `vagrant-provisioning/` setup) historically not
|
||
exposing AVX-512 to guest VMs at all. There's no runtime flag for
|
||
this, only a build-time one, so `image:` on both Deployments now
|
||
points at a **custom build you must create first**:
|
||
`registry.enkly.org/enkly/vllm-cpu:latest`, built with
|
||
`--build-arg VLLM_CPU_DISABLE_AVX512=true` — see `11-vllm.yaml`'s
|
||
header for the exact command. That build falls back to plain AVX2,
|
||
universal on any x86_64 node since ~2013, so the init container now
|
||
only checks for that (a much lower, already-satisfied bar) instead
|
||
of gating on `avx512f`.
|
||
2. **The chat model's memory request alone exceeded a node's total
|
||
RAM.** `10-rabbitmq.yaml`'s memory-watermark fix already
|
||
established each node has ~7949Mi (~7.76Gi) total — `vllm-chat`'s
|
||
original `8Gi` request could never be satisfied by *any* node here,
|
||
which is exactly why it sat `Pending` instead of crashing. Fixed by
|
||
dropping the chat model to `Qwen/Qwen2.5-1.5B-Instruct` (~3GB bf16
|
||
vs. the 3B variant's ~6GB) and cutting
|
||
`resources`/`VLLM_CPU_KVCACHE_SPACE`/`--max-model-len` on both
|
||
Deployments to fit comfortably inside ~7.76Gi alongside everything
|
||
else already running on that node (`vllm-chat` requests `4Gi`/limits
|
||
`6Gi` — sized to the realistic weights + KV-cache + runtime-overhead
|
||
total, not the bare minimum, so there's headroom left to the limit
|
||
during startup warmup).
|
||
3. **A shared `ReadWriteMany` PVC was an unverified assumption, not a
|
||
confirmed fix.** `vllm-chat` and `vllm-embed` originally bind-mounted
|
||
one PVC for their Hugging Face cache. `managed-nfs-storage`
|
||
(`nfs-subdir-external-provisioner`) does advertise RWX, so that
|
||
probably binds fine — but "probably" produces exactly the kind of
|
||
pod that's silently stuck `Pending` on an unbound PVC, easy to
|
||
mis-diagnose as a repeat of the GPU/memory issues above. Split into
|
||
`vllm-chat-cache` and `vllm-embed-cache`, both plain `ReadWriteOnce`
|
||
— the same access mode every other PVC in this project already binds
|
||
with — so neither Deployment's scheduling can depend on the other's
|
||
claim or on RWX ever having been exercised here.
|
||
|
||
If real GPU nodes get added to this cluster later: point `image:` on
|
||
both Deployments back at `vllm/vllm-openai`, add back
|
||
`nvidia.com/gpu` requests + a GPU-pool `nodeSelector`/`toleration` +
|
||
an NVIDIA device-plugin DaemonSet, and size the model/resources back
|
||
up.
|
||
|
||
## Other issues found in the manifest set and how they were fixed here
|
||
|
||
1. **Postgres on NFS-backed PVCs fails `initdb`.**
|
||
`nfs-subdir-external-provisioner` volumes aren't always empty at
|
||
the Postgres data-dir mount point, and `initdb` refuses to
|
||
initialize a non-empty directory. All four databases set
|
||
`PGDATA=/var/lib/postgresql/data/pgdata` (a subdirectory of the
|
||
mount) to sidestep this — see `12-databases.yaml`.
|
||
2. **`agent`/`knowledge` never received the vLLM API key.** Both
|
||
services' own code (`services/agent/cmd/server/main.go`,
|
||
`services/knowledge/cmd/server/main.go`) reads
|
||
`ENKLY_VLLM_API_KEY` from the environment and defaults to `""` if
|
||
unset — but neither `24-knowledge.yaml` nor `25-agent.yaml` sourced
|
||
it from `enkly-secrets`. Since `11-vllm.yaml` starts both vLLM
|
||
servers with `--api-key=$(VLLM_API_KEY)`, every chat/embedding call
|
||
from these two services would get rejected with `401`. Both files
|
||
now set `ENKLY_VLLM_API_KEY` from `secretKeyRef: {name:
|
||
enkly-secrets, key: ENKLY_VLLM_API_KEY}`.
|
||
3. **`02-secrets.yaml` had what read as live credentials committed as
|
||
if they were placeholders.** `STRIPE_SECRET_KEY`/
|
||
`STRIPE_PUBLISH_KEY` were filled with what look like real Stripe
|
||
test-mode keys, and `ENKLY_SMTP_USERNAME`/`ENKLY_SMTP_PASSWORD`
|
||
held a real-looking Gmail address and a 16-character app password.
|
||
**Rotate the Gmail app password and the Stripe keys if this file
|
||
was ever pushed anywhere outside a local sandbox** — replacing the
|
||
values here doesn't undo prior exposure. Replace all four with
|
||
your own before applying. Separately, `ENKLY_VLLM_API_KEY` should
|
||
be a real random value before applying — YAML `stringData` doesn't
|
||
run shell commands, so a literal `"$(openssl rand -hex 32)"` string
|
||
would become the actual, non-random key; see that key's comment in
|
||
`02-secrets.yaml` for the correct one-liner.
|
||
4. **Registry auth/TLS is cluster-level, not manifest-level.** Every
|
||
Deployment references `imagePullSecrets: [enkly-registry]`, but
|
||
that Secret must be created out-of-band (real credentials can't
|
||
live in a file meant for a repo — see `02-secrets.yaml`'s header
|
||
comment). Likewise, if `registry.enkly.org` serves a self-signed or
|
||
internal CA cert, every node's containerd needs that CA trusted via
|
||
`/etc/containerd/certs.d/registry.enkly.org/hosts.toml` — that's
|
||
node provisioning, outside what any Kubernetes manifest can
|
||
express.
|
||
|
||
## Layout
|
||
|
||
| File | What |
|
||
|---|---|
|
||
| `00-namespace.yaml` | the `enkly` namespace |
|
||
| `01-config.yaml` | non-secret `ENKLY_*` config, shared by every service |
|
||
| `02-secrets.yaml` | **template** — DB/AMQP/Stripe/SMTP/HF secret + instructions for the registry pull secret |
|
||
| `10-rabbitmq.yaml` | RabbitMQ, with a PVC (so broker state survives a pod restart) |
|
||
| `11-vllm.yaml` | vLLM chat + embed Deployments/Services, CPU backend (no GPU), each with its own RWO Hugging Face cache PVC — see above |
|
||
| `12-databases.yaml` | iam-db, billing-db, conversation-db, knowledge-db (pgvector), each with its own PVC |
|
||
| `20`–`27` | iam, billing, conversation, workspace, knowledge, agent, notification, gateway |
|
||
| `30-ingress.yaml` | Ingress for the gateway (+ commented NodePort fallback) |
|
||
| `kustomization.yaml` | ties the above together for `kubectl apply -k .` |
|
||
|
||
Every Deployment's env vars are a 1:1 mapping from
|
||
`docker-compose.yml` — service hostnames are identical to the compose
|
||
service names, so the only real change the application sees is DNS
|
||
resolving inside a namespace instead of a compose network.
|
||
|
||
## Deploy
|
||
|
||
Two things have to be true *before* any apply, because nothing in
|
||
Kubernetes can discover them on its own: your image registry has to
|
||
have the 8 service images, and your real Stripe/SMTP credentials have
|
||
to come from you.
|
||
|
||
```bash
|
||
# 1. Build & push the 8 images
|
||
REG=registry.enkly.org/enkly
|
||
for svc in iam billing conversation workspace knowledge agent notification gateway; do
|
||
docker build -f services/$svc/Dockerfile -t $REG/$svc:latest .
|
||
docker push $REG/$svc:latest
|
||
done
|
||
|
||
# 2. Registry pull secret (every Deployment references this)
|
||
kubectl create secret docker-registry enkly-registry -n enkly \
|
||
--docker-server=registry.enkly.org \
|
||
--docker-username=<user> --docker-password=<token> --docker-email=<email>
|
||
|
||
# 3. Edit 02-secrets.yaml: fill in STRIPE_*/ENKLY_SMTP_* with your own
|
||
# rotated values (see issue #3 above), and generate a real
|
||
# ENKLY_VLLM_API_KEY (the file documents the exact one-liner).
|
||
|
||
# 4. Build & push the CPU-only, no-AVX512 vLLM image — this cluster's
|
||
# nodes were confirmed to lack avx512f, so the official pre-built
|
||
# image (public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo) will crash
|
||
# with "Illegal instruction"; 11-vllm.yaml's image: already points
|
||
# at the custom tag this produces:
|
||
git clone https://github.com/vllm-project/vllm.git /tmp/vllm-build && cd /tmp/vllm-build
|
||
docker build -f docker/Dockerfile.cpu \
|
||
--build-arg VLLM_CPU_DISABLE_AVX512=true \
|
||
--tag registry.enkly.org/enkly/vllm-cpu:latest \
|
||
--target vllm-openai .
|
||
docker push registry.enkly.org/enkly/vllm-cpu:latest
|
||
|
||
# 5. Apply everything
|
||
kubectl apply -k k8s/
|
||
|
||
# 6. Watch the rollout — first boot downloads ~3.5GB of Hugging Face
|
||
# weights total onto vllm-chat-cache/vllm-embed-cache, so the
|
||
# startupProbe budget is generous.
|
||
kubectl -n enkly get pods -w
|
||
kubectl -n enkly rollout status deploy/vllm-chat --timeout=15m
|
||
kubectl -n enkly rollout status deploy/vllm-embed --timeout=15m
|
||
kubectl -n enkly rollout status deploy/gateway
|
||
```
|
||
|
||
Then point DNS/`/etc/hosts` at the Ingress controller's address for
|
||
the host set in `30-ingress.yaml` / `ENKLY_BASE_URL` in
|
||
`01-config.yaml`, or use the commented NodePort Service in
|
||
`30-ingress.yaml` if no ingress controller is installed
|
||
(`kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.11.3/deploy/static/provider/baremetal/deploy.yaml`
|
||
installs one on bare metal).
|
||
|
||
### Verify the LLM path specifically
|
||
|
||
```bash
|
||
kubectl -n enkly logs deploy/vllm-chat -f # watch the HF download + CPU backend warmup
|
||
kubectl -n enkly logs deploy/vllm-embed -f
|
||
kubectl -n enkly port-forward svc/gateway 8080:8080 # then run a chat turn end to end
|
||
```
|
||
|
||
If a pod is stuck in `Init:Error` / `Init:CrashLoopBackOff`, that's
|
||
the `check-cpu-avx2` init container — this cluster already needed the
|
||
custom no-AVX512 image (see above), so seeing this now would mean a
|
||
node lacks even AVX2, which no mainstream x86_64 CPU built since
|
||
~2013 should be missing; read its log for which node and double-check
|
||
`lscpu` there directly. If instead the main `vllm-chat`/`vllm-embed`
|
||
container itself is crashing after the init container passed, check
|
||
for `OOMKilled` in `kubectl -n enkly describe pod` specifically (this
|
||
cluster's nodes only have ~7.76Gi RAM total each — raise
|
||
`resources.limits.memory` cautiously, if at all, and prefer lowering
|
||
`VLLM_CPU_KVCACHE_SPACE` or `--max-model-len` instead) over assuming
|
||
it's a repeat of the AVX-512 issue.
|
||
|
||
## Scaling beyond the defaults
|
||
|
||
`iam`, `billing`, `conversation`, `knowledge`, `agent`, `notification`
|
||
and `gateway` are stateless and safe to scale (`kubectl -n enkly scale
|
||
deploy/gateway --replicas=3`). `workspace`, the four Postgres
|
||
Deployments, `vllm-chat` and `vllm-embed` are all pinned to
|
||
`replicas: 1` with `strategy: Recreate` because each owns a
|
||
`ReadWriteOnce` PVC — don't raise their replica counts without first
|
||
either moving to a clustered/replicated backend (e.g. Patroni for
|
||
Postgres) or, for the vLLM pair specifically, converting
|
||
`vllm-chat-cache`/`vllm-embed-cache` to `ReadWriteMany` (cheap to try:
|
||
`managed-nfs-storage` already advertises it) and switching
|
||
`strategy.type` to `RollingUpdate`. Either way, remember each replica
|
||
is a full, independent CPU-bound process (no GPU to share), so scaling
|
||
trades throughput for a proportional increase in CPU/RAM footprint,
|
||
not a free win the way request-level load balancing sometimes is.
|