enkly-ai/k8s

enkly-ai — Kubernetes deployment

Plain manifests (no Helm) for the 8 Go microservices + RabbitMQ + 4 Postgres databases + two CPU-only vLLM inference servers, for the kmaster / kworker1 / kworker2 cluster described in docker-compose.yml. One file per concern, applied individually or via kustomization.yaml.

vLLM on a cluster with no GPU

An earlier version of this manifest set ran vllm-chat/vllm-embed on the CUDA vllm/vllm-openai image, each requesting nvidia.com/gpu: "1". This cluster has no physical GPU on any of its three nodes. nvidia.com/gpu: "1" is not a resource any node here can ever advertise — with no node reporting it, both pods sat Pending permanently (Insufficient nvidia.com/gpu), not just slow to schedule.

11-vllm.yaml runs vLLM's CPU backend instead — same two Deployments, same OpenAI-compatible API, same --api-key enforcement, no application code changes. Two real, observed problems (not just theoretical GPU-vs-CPU ones) shaped the rest of the fix:

1. avx512f genuinely missing on this cluster's nodes. vLLM's official pre-built CPU image (public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo) hard-requires that flag and crashes with Illegal instruction without it — and an init container added to both Deployments confirmed it's absent here, consistent with VirtualBox/Vagrant nested virtualization (this project's own vagrant-provisioning/ setup) historically not exposing AVX-512 to guest VMs at all. There's no runtime flag for this, only a build-time one, so image: on both Deployments now points at a custom build you must create first: registry.enkly.org/enkly/vllm-cpu:latest, built with --build-arg VLLM_CPU_DISABLE_AVX512=true — see 11-vllm.yaml's header for the exact command. That build falls back to plain AVX2, universal on any x86_64 node since ~2013, so the init container now only checks for that (a much lower, already-satisfied bar) instead of gating on avx512f.
2. The chat model's memory request alone exceeded a node's total RAM. 10-rabbitmq.yaml's memory-watermark fix already established each node has ~7949Mi (~7.76Gi) total — vllm-chat's original 8Gi request could never be satisfied by any node here, which is exactly why it sat Pending instead of crashing. Fixed by dropping the chat model to Qwen/Qwen2.5-1.5B-Instruct (~3GB bf16 vs. the 3B variant's ~6GB) and cutting resources/VLLM_CPU_KVCACHE_SPACE/--max-model-len on both Deployments to fit comfortably inside ~7.76Gi alongside everything else already running on that node (vllm-chat requests 4Gi/limits 6Gi — sized to the realistic weights + KV-cache + runtime-overhead total, not the bare minimum, so there's headroom left to the limit during startup warmup).
3. A shared ReadWriteMany PVC was an unverified assumption, not a confirmed fix. vllm-chat and vllm-embed originally bind-mounted one PVC for their Hugging Face cache. managed-nfs-storage (nfs-subdir-external-provisioner) does advertise RWX, so that probably binds fine — but "probably" produces exactly the kind of pod that's silently stuck Pending on an unbound PVC, easy to mis-diagnose as a repeat of the GPU/memory issues above. Split into vllm-chat-cache and vllm-embed-cache, both plain ReadWriteOnce — the same access mode every other PVC in this project already binds with — so neither Deployment's scheduling can depend on the other's claim or on RWX ever having been exercised here.

If real GPU nodes get added to this cluster later: point image: on both Deployments back at vllm/vllm-openai, add back nvidia.com/gpu requests + a GPU-pool nodeSelector/toleration + an NVIDIA device-plugin DaemonSet, and size the model/resources back up.

Other issues found in the manifest set and how they were fixed here

1. Postgres on NFS-backed PVCs fails initdb. nfs-subdir-external-provisioner volumes aren't always empty at the Postgres data-dir mount point, and initdb refuses to initialize a non-empty directory. All four databases set PGDATA=/var/lib/postgresql/data/pgdata (a subdirectory of the mount) to sidestep this — see 12-databases.yaml.
2. agent/knowledge never received the vLLM API key. Both services' own code (services/agent/cmd/server/main.go, services/knowledge/cmd/server/main.go) reads ENKLY_VLLM_API_KEY from the environment and defaults to "" if unset — but neither 24-knowledge.yaml nor 25-agent.yaml sourced it from enkly-secrets. Since 11-vllm.yaml starts both vLLM servers with --api-key=$(VLLM_API_KEY), every chat/embedding call from these two services would get rejected with 401. Both files now set ENKLY_VLLM_API_KEY from secretKeyRef: {name: enkly-secrets, key: ENKLY_VLLM_API_KEY}.
3. 02-secrets.yaml had what read as live credentials committed as if they were placeholders. STRIPE_SECRET_KEY/ STRIPE_PUBLISH_KEY were filled with what look like real Stripe test-mode keys, and ENKLY_SMTP_USERNAME/ENKLY_SMTP_PASSWORD held a real-looking Gmail address and a 16-character app password. Rotate the Gmail app password and the Stripe keys if this file was ever pushed anywhere outside a local sandbox — replacing the values here doesn't undo prior exposure. Replace all four with your own before applying. Separately, ENKLY_VLLM_API_KEY should be a real random value before applying — YAML stringData doesn't run shell commands, so a literal "$(openssl rand -hex 32)" string would become the actual, non-random key; see that key's comment in 02-secrets.yaml for the correct one-liner.
4. Registry auth/TLS is cluster-level, not manifest-level. Every Deployment references imagePullSecrets: [enkly-registry], but that Secret must be created out-of-band (real credentials can't live in a file meant for a repo — see 02-secrets.yaml's header comment). Likewise, if registry.enkly.org serves a self-signed or internal CA cert, every node's containerd needs that CA trusted via /etc/containerd/certs.d/registry.enkly.org/hosts.toml — that's node provisioning, outside what any Kubernetes manifest can express.

Layout

File	What
00-namespace.yaml	the enkly namespace
01-config.yaml	non-secret ENKLY_* config, shared by every service
02-secrets.yaml	template — DB/AMQP/Stripe/SMTP/HF secret + instructions for the registry pull secret
10-rabbitmq.yaml	RabbitMQ, with a PVC (so broker state survives a pod restart)
11-vllm.yaml	vLLM chat + embed Deployments/Services, CPU backend (no GPU), each with its own RWO Hugging Face cache PVC — see above
12-databases.yaml	iam-db, billing-db, conversation-db, knowledge-db (pgvector), each with its own PVC
20–27	iam, billing, conversation, workspace, knowledge, agent, notification, gateway
30-ingress.yaml	Ingress for the gateway (+ commented NodePort fallback)
kustomization.yaml	ties the above together for kubectl apply -k .

Every Deployment's env vars are a 1:1 mapping from docker-compose.yml — service hostnames are identical to the compose service names, so the only real change the application sees is DNS resolving inside a namespace instead of a compose network.

Deploy

Two things have to be true before any apply, because nothing in Kubernetes can discover them on its own: your image registry has to have the 8 service images, and your real Stripe/SMTP credentials have to come from you.

# 1. Build & push the 8 images
REG=registry.enkly.org/enkly
for svc in iam billing conversation workspace knowledge agent notification gateway; do
  docker build -f services/$svc/Dockerfile -t $REG/$svc:latest .
  docker push $REG/$svc:latest
done

# 2. Registry pull secret (every Deployment references this)
kubectl create secret docker-registry enkly-registry -n enkly \
  --docker-server=registry.enkly.org \
  --docker-username=<user> --docker-password=<token> --docker-email=<email>

# 3. Edit 02-secrets.yaml: fill in STRIPE_*/ENKLY_SMTP_* with your own
#    rotated values (see issue #3 above), and generate a real
#    ENKLY_VLLM_API_KEY (the file documents the exact one-liner).

# 4. Build & push the CPU-only, no-AVX512 vLLM image — this cluster's
#    nodes were confirmed to lack avx512f, so the official pre-built
#    image (public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo) will crash
#    with "Illegal instruction"; 11-vllm.yaml's image: already points
#    at the custom tag this produces:
git clone https://github.com/vllm-project/vllm.git /tmp/vllm-build && cd /tmp/vllm-build
docker build -f docker/Dockerfile.cpu \
  --build-arg VLLM_CPU_DISABLE_AVX512=true \
  --tag registry.enkly.org/enkly/vllm-cpu:latest \
  --target vllm-openai .
docker push registry.enkly.org/enkly/vllm-cpu:latest

# 5. Apply everything
kubectl apply -k k8s/

# 6. Watch the rollout — first boot downloads ~3.5GB of Hugging Face
#    weights total onto vllm-chat-cache/vllm-embed-cache, so the
#    startupProbe budget is generous.
kubectl -n enkly get pods -w
kubectl -n enkly rollout status deploy/vllm-chat --timeout=15m
kubectl -n enkly rollout status deploy/vllm-embed --timeout=15m
kubectl -n enkly rollout status deploy/gateway

Then point DNS//etc/hosts at the Ingress controller's address for the host set in 30-ingress.yaml / ENKLY_BASE_URL in 01-config.yaml, or use the commented NodePort Service in 30-ingress.yaml if no ingress controller is installed (kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.11.3/deploy/static/provider/baremetal/deploy.yaml installs one on bare metal).

Verify the LLM path specifically

kubectl -n enkly logs deploy/vllm-chat -f   # watch the HF download + CPU backend warmup
kubectl -n enkly logs deploy/vllm-embed -f
kubectl -n enkly port-forward svc/gateway 8080:8080   # then run a chat turn end to end

If a pod is stuck in Init:Error / Init:CrashLoopBackOff, that's the check-cpu-avx2 init container — this cluster already needed the custom no-AVX512 image (see above), so seeing this now would mean a node lacks even AVX2, which no mainstream x86_64 CPU built since ~2013 should be missing; read its log for which node and double-check lscpu there directly. If instead the main vllm-chat/vllm-embed container itself is crashing after the init container passed, check for OOMKilled in kubectl -n enkly describe pod specifically (this cluster's nodes only have ~7.76Gi RAM total each — raise resources.limits.memory cautiously, if at all, and prefer lowering VLLM_CPU_KVCACHE_SPACE or --max-model-len instead) over assuming it's a repeat of the AVX-512 issue.

Scaling beyond the defaults

iam, billing, conversation, knowledge, agent, notification and gateway are stateless and safe to scale (kubectl -n enkly scale deploy/gateway --replicas=3). workspace, the four Postgres Deployments, vllm-chat and vllm-embed are all pinned to replicas: 1 with strategy: Recreate because each owns a ReadWriteOnce PVC — don't raise their replica counts without first either moving to a clustered/replicated backend (e.g. Patroni for Postgres) or, for the vLLM pair specifically, converting vllm-chat-cache/vllm-embed-cache to ReadWriteMany (cheap to try: managed-nfs-storage already advertises it) and switching strategy.type to RollingUpdate. Either way, remember each replica is a full, independent CPU-bound process (no GPU to share), so scaling trades throughput for a proportional increase in CPU/RAM footprint, not a free win the way request-level load balancing sometimes is.