Init Project

k8s/00-namespace.yaml

@@ -0,0 +1,6 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: enkly
+  labels:
+    app.kubernetes.io/part-of: enkly

k8s/01-config.yaml

@@ -0,0 +1,103 @@
+# Non-secret configuration shared by every service. Keys mirror the
+# ENKLY_* env vars from docker-compose.yml / .env.example 1:1 so the
+# application code needs zero changes — only the hostnames change,
+# from docker-compose service names to in-cluster Kubernetes Service
+# names (which are identical, so most values don't even change).
+# NOTE: the three *_POSTGRES_DSN values are NOT here — ConfigMap data
+# is static text and cannot expand $(POSTGRES_PASSWORD); each DSN is
+# instead composed inline in its Deployment's env: list (see the
+# per-service files), where Kubernetes does support that expansion.
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: enkly-config
+  namespace: enkly
+data:
+  # --- shared ---
+  ENKLY_BASE_URL: "https://ai.enkly.org"            # set to your real public URL / Ingress host
+  # Back on vLLM (per your request), still on a GPU-less cluster: both
+  # Deployments in 11-vllm.yaml now run vLLM's CPU backend — a custom
+  # build with AVX-512 disabled (registry.enkly.org/enkly/vllm-cpu),
+  # not vLLM's official pre-built image, after vllm-embed's init
+  # container confirmed this cluster's nodes actually lack avx512f —
+  # instead of the CUDA vllm/vllm-openai image, with no nvidia.com/gpu
+  # request and no GPU-pool nodeSelector/toleration. See 11-vllm.yaml's
+  # header for the build command and the memory-sizing fix that went
+  # with it (vllm-chat's original 8Gi request exceeded this cluster's
+  # confirmed ~7.76Gi per-node RAM outright).
+  # Two Deployments, two Services, same as the original GPU manifest
+  # (one vLLM engine process serves exactly one model — see
+  # pkg/vllmkit's doc comment) — unlike the brief Ollama detour this
+  # cluster also tried, which could multiplex both models behind one
+  # daemon. Internal-only ClusterIP Services this time (no public
+  # vllm-chat.enkly.org/vllm-embed.enkly.org Ingress) since agent and
+  # knowledge are the only callers and both are in-cluster.
+  ENKLY_VLLM_CHAT_URL: "http://vllm-chat:8000/v1"
+  ENKLY_VLLM_EMBED_URL: "http://vllm-embed:8000/v1"
+  # qwen3.5 (the original GPU deployment's model, ~35B-parameter MoE,
+  # served via a GPTQ-Marlin checkpoint) isn't viable here twice over:
+  # Marlin's INT4 kernels are CUDA-only, and 35B active-weight decode
+  # at CPU speeds wouldn't be usable latency regardless. Went with
+  # qwen2.5-1.5b-instruct rather than the 3B variant tried first —
+  # unquantized bf16 (CPU quantized-GPTQ/AWQ kernel support exists but
+  # is far less battle-tested than Marlin's GPU path, so plain bf16 is
+  # the safer default here) — specifically because this cluster's
+  # nodes only have ~7.76Gi RAM total each (see 11-vllm.yaml's header):
+  # the 3B model's 8Gi request alone couldn't fit on any node, full
+  # stop. 1.5B still supports OpenAI-style tool calling via vLLM's
+  # hermes parser (required by
+  # services/agent/internal/infrastructure/agentrun). See
+  # 11-vllm.yaml's resources/VLLM_CPU_KVCACHE_SPACE for how this maps
+  # to actual CPU/RAM, and its header for going back up to 3B or
+  # higher once `kubectl describe node` confirms real free headroom.
+  ENKLY_CHAT_MODEL: "qwen2.5-1.5b-instruct"
+  # Note one real behavior gap versus the original Qwen3.5 GPU
+  # deployment: Qwen2.5 has no equivalent to --reasoning-parser=qwen3's
+  # separate "thinking" trace (Message.Thinking / wire
+  # reasoning_content) — vLLM won't emit that field for this model, so
+  # Thinking comes back empty. Doesn't break chat or tool calling,
+  # just drops the reasoning-trace UI panel's content.
+  #
+  # Unchanged — nomic-embed-text was already the right size/cost
+  # trade-off on CPU (see pkg/vllmkit's DefaultEmbedModel comment and
+  # 11-vllm.yaml's vllm-embed Deployment), and no schema.sql
+  # vector(768) dimension change is needed either.
+  ENKLY_EMBED_MODEL: "nomic-embed-text"
+
+  # --- iam ---
+  ENKLY_IAM_GRPC_ADDR: ":9001"
+  ENKLY_SMTP_HOST: "smtp.gmail.com"
+  ENKLY_SMTP_PORT: "587"
+
+  # --- billing ---
+  ENKLY_BILLING_GRPC_ADDR: ":9002"
+
+  # --- conversation ---
+  ENKLY_CONVERSATION_GRPC_ADDR: ":9003"
+
+  # --- workspace ---
+  ENKLY_WORKSPACE_GRPC_ADDR: ":9004"
+  ENKLY_WORKSPACE: "/data/workspace"
+  ENKLY_BASH_TIMEOUT_SECONDS: "120"
+  # Only tools actually present in the workspace image (see its
+  # Dockerfile: ca-certificates + git). Extend the image before
+  # widening this list.
+  ENKLY_BASH_ALLOWLIST: "git,ls,cat,grep,find,pwd,echo"
+
+  # --- knowledge ---
+  ENKLY_KNOWLEDGE_GRPC_ADDR: ":9005"
+
+  # --- agent ---
+  ENKLY_AGENT_GRPC_ADDR: ":9006"
+
+  # --- notification ---
+  ENKLY_NOTIFICATION_HEALTH_ADDR: ":9007"
+  ENKLY_SMTP_FROM: "noreply <contact.enkly@gmail.com>"
+
+  # --- gateway ---
+  ENKLY_GATEWAY_HTTP_ADDR: ":8080"
+  ENKLY_TEMPLATES_DIR: "/app/web/templates"
+  ENKLY_STATIC_DIR: "/app/web/static"
+
+  # --- rabbitmq (non-secret half; the password half is in enkly-secrets) ---
+  RABBITMQ_DEFAULT_USER: "enkly"

k8s/02-secrets.yaml

@@ -0,0 +1,67 @@
+# TEMPLATE — replace every REPLACE_ME before applying, then either
+# `kubectl apply -f 02-secrets.yaml` or, better, keep this file out of
+# git and manage it with `kubectl create secret generic` / sealed-secrets
+# / SOPS instead. Postgres/RabbitMQ passwords are pre-filled with the
+# same "enkly" default the app ships with (see .env.example) purely so
+# the stack comes up unmodified on first try — rotate them before any
+# real deployment.
+apiVersion: v1
+kind: Secret
+metadata:
+  name: enkly-secrets
+  namespace: enkly
+type: Opaque
+stringData:
+  POSTGRES_PASSWORD: "enkly"
+  RABBITMQ_DEFAULT_PASS: "enkly"
+  ENKLY_RABBITMQ_URL: "amqp://enkly:enkly@rabbitmq:5672/"
+
+  STRIPE_SECRET_KEY: "sk_test_51Sk6S809EijW3hh9YFenSRh3gT2R8QzQ1XwDb4GpgwZx7k1zADSY4ApXh92fP39Clbn5vieX5vKYRM8Ts4t878S800OJfRc1zf"
+  STRIPE_PUBLISH_KEY: "pk_test_51Sk6S809EijW3hh95Tu0y6PyjbYXEKNus9dPlgFFCFqm3WYNRgO8dK2qoUGSbaRT4fDCQko1wyUQ2PrqbfP1FQD100xrrf9gzY"
+  STRIPE_WEBHOOK_SECRET: ""
+  STRIPE_PRICE_BASIC: "5"
+  STRIPE_PRICE_PRO: "8"
+  STRIPE_PRICE_PREMIUM: "12"
+
+  ENKLY_SMTP_USERNAME: "contact.enkly@gmail.com"
+  ENKLY_SMTP_PASSWORD: "ntqjsksezwjqpcac"
+
+  # vLLM (11-vllm.yaml). ENKLY_VLLM_API_KEY is the bearer token both
+  # vllm-chat and vllm-embed require (--api-key) and that agent/
+  # knowledge authenticate with (see services/agent/cmd/server/main.go,
+  # services/knowledge/cmd/server/main.go — both read this exact env
+  # var and default to "" if it's unset, which vLLM's --api-key will
+  # then reject with a 401 on every single request). Generate a real
+  # value BEFORE applying this file — YAML stringData does not run
+  # shell commands, so a literal "$(openssl rand -hex 32)" string here
+  # would silently become the actual, non-random API key instead of
+  # being executed:
+  #   kubectl create secret generic enkly-secrets -n enkly \
+  #     --from-literal=ENKLY_VLLM_API_KEY=$(openssl rand -hex 32) --dry-run=client -o yaml
+  # or just replace the line below by hand with a real random string.
+  ENKLY_VLLM_API_KEY: "OWVmOGNiZjM1YzFjNjUxNTY0YWJlMWZmYTg0M2FiYWM5ZTkwMjI3NDBmZjAwMzVjMDdlZDUwMjNlZTg1MTk1OQ=="
+  # HUGGING_FACE_HUB_TOKEN is only required if Qwen/Qwen2.5-3B-Instruct
+  # or nomic-ai/nomic-embed-text-v1.5 is gated/private for this org's
+  # HF account; both are public weights today, so leave this as an
+  # empty string unless that changes. Re-added now that 11-vllm.yaml
+  # pulls straight from the Hugging Face Hub again (unlike the brief
+  # Ollama detour, which pulled from Ollama's own registry instead).
+  HUGGING_FACE_HUB_TOKEN: ""
+
+# Pull secret for registry.enkly.org. Do NOT hand-edit this one — it's
+# generated, not templated, because .dockerconfigjson must be valid
+# base64 JSON. Create it with:
+#
+#   kubectl create secret docker-registry enkly-registry \
+#     --namespace enkly \
+#     --docker-server=registry.enkly.org \
+#     --docker-username=<user> \
+#     --docker-password=<password-or-token> \
+#     --docker-email=<email>
+#
+# Every Deployment below references it as imagePullSecrets: [enkly-registry].
+# If registry.enkly.org serves a self-signed/internal CA certificate,
+# that trust also has to be configured at the containerd level on
+# every node (/etc/containerd/certs.d/registry.enkly.org/hosts.toml) —
+# that's node config, not something a k8s manifest can do; see the
+# README's "Known issues to fix" section.

k8s/10-rabbitmq.yaml

@@ -0,0 +1,174 @@
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata: {name: rabbitmq-data, namespace: enkly}
+spec:
+  accessModes: [ReadWriteOnce]
+  storageClassName: managed-nfs-storage
+  resources: {requests: {storage: 10Gi}}
+---
+# Root cause of the alarm in the pod logs: RabbitMQ's default memory
+# watermark is 40% of what it detects as "total system memory". Inside
+# this container that detection comes back as the *node's* memory
+# (7949 MiB), not the 512Mi the container is actually capped to — so
+# the watermark it set (3179 MiB) is six times higher than the limit
+# Kubernetes will actually enforce. In practice that means the broker
+# never throttles publishers in time and gets OOMKilled by the kubelet
+# instead of gracefully blocking connections, which is what the
+# watermark is supposed to prevent. Pinning the watermark explicitly,
+# below the container's real memory limit, fixes that.
+apiVersion: v1
+kind: ConfigMap
+metadata: {name: rabbitmq-conf, namespace: enkly}
+data:
+  20-enkly.conf: |
+    # Keep this comfortably under the container memory limit below so
+    # RabbitMQ blocks publishers well before Kubernetes OOMKills the pod.
+    vm_memory_high_watermark.absolute = 550MB
+    vm_memory_calculation_strategy = allocated
+    disk_free_limit.absolute = 2GB
+    # NOTE: management_metrics_collection logs its deprecation warning on
+    # every boot regardless of the deprecated_features.permit setting while
+    # it's in the "permitted by default" phase — there is no config to
+    # silence it. It's informational, not actionable, until RabbitMQ moves
+    # it to "denied by default" in a future release. The real work item is
+    # migrating anything scraping the management API for metrics (Grafana,
+    # alerting) over to the rabbitmq_prometheus plugin before that happens.
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: rabbitmq
+  namespace: enkly
+  labels: {app: rabbitmq}
+spec:
+  replicas: 1
+  strategy: {type: Recreate}
+  selector: {matchLabels: {app: rabbitmq}}
+  template:
+    metadata:
+      labels: {app: rabbitmq}
+    spec:
+      imagePullSecrets: [{name: enkly-registry}]
+      # RabbitMQ resolves hostnames during boot even with RABBITMQ_NODENAME
+      # pinned to rabbit@localhost (the management plugin and Erlang's
+      # networking stack still do their own lookups). The default resolver
+      # in a pod walks every *.svc.cluster.local search-domain suffix
+      # before giving up on each failed query, and RabbitMQ's own k8s
+      # deployment docs call out exactly this as a cause of ~30-50s boot
+      # stalls. Shortening the per-query timeout/attempts here is the
+      # pod-level fix that doesn't require touching cluster-wide CoreDNS.
+      dnsPolicy: ClusterFirst
+      dnsConfig:
+        options:
+          - {name: ndots, value: "2"}
+          - {name: timeout, value: "2"}
+          - {name: attempts, value: "2"}
+      # Runs before the rabbitmq container every single boot, sharing the
+      # same PVC. Deletes any leftover mnesia directory that isn't the one
+      # actually in use (rabbit@localhost). Those directories are what
+      # caused the ~120s boot time and the resulting liveness-probe kills:
+      # every pod restart before RABBITMQ_NODENAME was pinned minted a new
+      # directory named after that pod's hostname, and the entrypoint's
+      # ownership pass over /var/lib/rabbitmq scans all of them. This is
+      # idempotent — once clean, it's a no-op on every future boot, and it
+      # means boot time is never again at the mercy of accumulated cruft,
+      # so the probe timings below don't need to keep being widened.
+      initContainers:
+        - name: clean-stale-mnesia-dirs
+          image: rabbitmq:3.13-management-alpine
+          command:
+            - sh
+            - -c
+            - |
+              set -e
+              cd /var/lib/rabbitmq/mnesia 2>/dev/null || exit 0
+              for d in rabbit@*; do
+                [ -e "$d" ] || continue
+                case "$d" in
+                  rabbit@localhost|rabbit@localhost-plugins-expand|rabbit@localhost-feature_flags)
+                    continue ;;
+                esac
+                echo "removing stale mnesia dir: $d"
+                rm -rf -- "$d"
+              done
+          volumeMounts:
+            - {name: data, mountPath: /var/lib/rabbitmq}
+      containers:
+        - name: rabbitmq
+          image: rabbitmq:3.13-management-alpine
+          ports:
+            - {name: amqp, containerPort: 5672}
+            - {name: mgmt, containerPort: 15672}
+          env:
+            # Pins the Erlang node identity so it does NOT follow the pod's
+            # hostname. Without this, every rollout/reschedule gets a new
+            # pod name -> new node name (rabbit@<pod-name>) -> RabbitMQ
+            # mounts the same PVC but looks for mnesia data under a path
+            # for a node it's never seen before, so it reports "virgin
+            # node" and silently reseeds an empty vhost every time. Safe
+            # as a static value here because replicas is 1 and strategy is
+            # Recreate, so there's never two pods needing distinct identities.
+            - name: RABBITMQ_NODENAME
+              value: "rabbit@localhost"
+            - name: RABBITMQ_DEFAULT_USER
+              valueFrom: {configMapKeyRef: {name: enkly-config, key: RABBITMQ_DEFAULT_USER}}
+            - name: RABBITMQ_DEFAULT_PASS
+              valueFrom: {secretKeyRef: {name: enkly-secrets, key: RABBITMQ_DEFAULT_PASS}}
+          resources:
+            # request == limit (Guaranteed QoS): a Burstable broker pod is
+            # one of the kubelet's first eviction targets under node memory
+            # pressure, which is the last thing you want for the message
+            # bus. 768Mi gives the 550MB watermark above real headroom
+            # before the container limit, instead of the old 512Mi limit
+            # sitting below the watermark that was actually configured.
+            # Raised from 250m/500m: gives the liveness probe's
+            # rabbitmq-diagnostics CLI (a separate, short-lived BEAM VM
+            # each time it runs) enough CPU headroom to finish inside its
+            # 5s timeout without contending with the broker process itself.
+            requests: {cpu: 500m, memory: 768Mi}
+            limits: {cpu: 500m, memory: 768Mi}
+          readinessProbe:
+            # Switched from an exec-based "rabbitmq-diagnostics
+            # check_port_connectivity" to a plain TCP check on the AMQP
+            # port. The exec form spins up a brand-new Erlang/BEAM VM on
+            # every single invocation just to run the check — under this
+            # container's CPU limit that overhead can exceed the probe's
+            # own timeout even while the broker itself is completely
+            # healthy, which reads as a permanently-failing readiness
+            # probe with no corresponding error in the app's own logs.
+            # RabbitMQ's own Kubernetes deployment docs recommend a
+            # TCP-based readiness check for exactly this reason — it's
+            # handled by the kubelet directly with no process spawned
+            # inside the container, so it isn't subject to CPU contention
+            # with the broker or the probe's own VM startup cost.
+            tcpSocket: {port: amqp}
+            initialDelaySeconds: 10
+            periodSeconds: 10
+            timeoutSeconds: 5
+            failureThreshold: 10
+          livenessProbe:
+            exec: {command: ["rabbitmq-diagnostics", "-q", "ping"]}
+            initialDelaySeconds: 30
+            periodSeconds: 20
+            timeoutSeconds: 5
+            failureThreshold: 6
+          volumeMounts:
+            - {name: data, mountPath: /var/lib/rabbitmq}
+            - {name: conf, mountPath: /etc/rabbitmq/conf.d/20-enkly.conf, subPath: 20-enkly.conf}
+      volumes:
+        - name: data
+          persistentVolumeClaim: {claimName: rabbitmq-data}
+        - name: conf
+          configMap: {name: rabbitmq-conf}
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: rabbitmq
+  namespace: enkly
+  labels: {app: rabbitmq}
+spec:
+  selector: {app: rabbitmq}
+  ports:
+    - {name: amqp, port: 5672, targetPort: amqp}
+    - {name: mgmt, port: 15672, targetPort: mgmt}

k8s/11-vllm.yaml

@@ -0,0 +1,326 @@
+# vLLM, per your request — but still on kmaster/kworker1/kworker2,
+# which have no physical GPU. The original vllm-chat/vllm-embed
+# (restored structurally below: two Deployments, one model each, same
+# as pkg/vllmkit's doc comment describes) ran the CUDA vllm/vllm-openai
+# image and requested nvidia.com/gpu: "1" — that's what made both pods
+# sit Pending forever with no node able to advertise that resource.
+#
+# This version runs vLLM's CPU backend instead, built from source
+# with AVX-512 disabled, rather than vLLM's official pre-built
+# public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo image. Two real,
+# observed problems drove both the image choice and the sizing below
+# — not hypothetical ones:
+#
+# 1. vllm-embed's init container (`check-cpu-avx512` further down)
+#    reported no `avx512f` CPU flag on the node it landed on. That's
+#    consistent with VirtualBox/Vagrant nested virtualization, which
+#    historically doesn't expose AVX-512 to guest VMs at all — and
+#    this cluster's own vagrant-provisioning/ setup is exactly that.
+#    The pre-built image hard-requires avx512f and crashes with
+#    "Illegal instruction" without it; there is no runtime flag to
+#    turn that off, only a build-time one. So `image:` below now
+#    points at a custom build instead — **you must build and push it
+#    before applying this file**:
+#      git clone https://github.com/vllm-project/vllm.git && cd vllm
+#      docker build -f docker/Dockerfile.cpu \
+#        --build-arg VLLM_CPU_DISABLE_AVX512=true \
+#        --tag registry.enkly.org/enkly/vllm-cpu:latest \
+#        --target vllm-openai .
+#      docker push registry.enkly.org/enkly/vllm-cpu:latest
+#    This build falls back to plain AVX2, which is universal on any
+#    x86_64 node (physical or virtualized) built since ~2013 — there's
+#    no equivalent flag check needed for it the way avx512f needed
+#    one, so the init container that used to gate on avx512f is gone;
+#    only a lightweight informational one remains (see below).
+#
+# 2. vllm-chat's 8Gi memory *request* alone exceeded a single node's
+#    total capacity: 10-rabbitmq.yaml's memory-watermark fix already
+#    established each node has ~7949Mi (~7.76Gi) total RAM, so an 8Gi
+#    request could never be satisfied regardless of what else was
+#    scheduled — that's what "Pending" (not a crash) meant. Fixed two
+#    ways together: the chat model dropped from Qwen2.5-3B-Instruct to
+#    Qwen2.5-1.5B-Instruct (bf16 weights ~3GB vs. ~6GB), and
+#    resources.requests/limits and VLLM_CPU_KVCACHE_SPACE below were
+#    cut to fit comfortably inside a ~7.76Gi node alongside everything
+#    else already running there (4Gi/6Gi request/limit — realistic for
+#    ~3GB weights + 1GiB KV cache + CPU-backend runtime overhead, not
+#    the bare-minimum 3Gi this previously requested, which left too
+#    little room to the limit during startup warmup). If
+#    `kubectl describe node` shows more real headroom than that once
+#    this is stable, both the model and these numbers can be raised
+#    again — see the sizing comments inline.
+#
+# 3. The two Deployments previously shared one ReadWriteMany PVC for
+#    the Hugging Face cache. managed-nfs-storage (nfs-subdir-external-
+#    provisioner) does advertise RWX, so that likely would have bound
+#    — but an unverified "likely" is exactly the kind of thing that
+#    produces a silent Pending pod (a PVC stuck Pending blocks its
+#    Pod the same way insufficient node resources does, just with a
+#    different `kubectl describe pod` event, easy to mis-read as a
+#    repeat of problem #2). Split into vllm-chat-cache and
+#    vllm-embed-cache below, both plain ReadWriteOnce — the same
+#    access mode every other PVC in this project already binds with
+#    (10-rabbitmq.yaml, 12-databases.yaml, 23-workspace.yaml) — so
+#    neither Deployment's scheduling can ever depend on the other's
+#    PVC or on RWX support this cluster's storage class may or may
+#    not actually have been exercised with yet.
+#
+# Everything else about the original design is unchanged: two
+# Deployments (one vLLM engine process serves exactly one model), the
+# OpenAI-compatible /v1/chat/completions + /v1/embeddings API,
+# --api-key, tool calling — so pkg/vllmkit needs no code changes,
+# only this file and 01-config.yaml's model/URL values.
+#
+# If real GPU nodes get added to this cluster later: point `image:` on
+# both Deployments back at `vllm/vllm-openai`, add back
+# `nvidia.com/gpu` requests + a GPU-pool nodeSelector/toleration + an
+# NVIDIA device-plugin DaemonSet, and size the model/resources up.
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata: {name: vllm-chat-cache, namespace: enkly}
+spec:
+  # ReadWriteOnce, deliberately not a single shared ReadWriteMany PVC
+  # across both Deployments. nfs-subdir-external-provisioner (behind
+  # managed-nfs-storage) does advertise RWX, so a shared PVC would
+  # probably bind — but "probably" is exactly the kind of unverified
+  # assumption that produces a silent, hard-to-diagnose Pending pod
+  # (PVC stuck Pending blocks the Pod the same way insufficient node
+  # resources does, just with a different `kubectl describe pod`
+  # event). Two independent RWO claims remove that failure mode
+  # entirely: each is the same well-proven access mode every other
+  # PVC in this project already binds with (10-rabbitmq.yaml,
+  # 12-databases.yaml, 23-workspace.yaml), neither Deployment's
+  # scheduling depends on the other's PVC, and losing the (small,
+  # one-time-download) benefit of a shared cache between a 3GB chat
+  # model and a 550MB embed model costs nothing worth trading that
+  # reliability for.
+  accessModes: [ReadWriteOnce]
+  storageClassName: managed-nfs-storage
+  # Qwen2.5-1.5B-Instruct at bf16 is ~3GB on disk; sized with headroom
+  # to try Qwen2.5-3B-Instruct (~6GB bf16) later without resizing
+  # first, once `kubectl describe node` confirms real free RAM.
+  resources: {requests: {storage: 10Gi}}
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata: {name: vllm-embed-cache, namespace: enkly}
+spec:
+  accessModes: [ReadWriteOnce]
+  storageClassName: managed-nfs-storage
+  # nomic-embed-text-v1.5 is ~550MB on disk.
+  resources: {requests: {storage: 5Gi}}
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: vllm-chat, namespace: enkly, labels: {app: vllm-chat}}
+spec:
+  replicas: 1
+  strategy: {type: Recreate}
+  selector: {matchLabels: {app: vllm-chat}}
+  template:
+    metadata: {labels: {app: vllm-chat}}
+    spec:
+      imagePullSecrets: [{name: enkly-registry}]
+      # Deliberately NO nodeSelector/toleration/nvidia.com/gpu request
+      # here (unlike the original GPU version of this file) — this
+      # cluster has no node that could ever satisfy either.
+      initContainers:
+        - name: check-cpu-avx2
+          image: busybox:1.36
+          command: ["sh", "-c"]
+          args:
+            - |
+              set -e
+              if ! grep -q avx2 /proc/cpuinfo; then
+                echo "FATAL: this node's CPU has no avx2 flag." >&2
+                echo "registry.enkly.org/enkly/vllm-cpu (built with VLLM_CPU_DISABLE_AVX512=true) still needs AVX2 at minimum." >&2
+                echo "This is a much lower bar than the avx512f this cluster was already confirmed to lack, but if it's missing too, CPU inference isn't viable on this node." >&2
+                exit 1
+              fi
+              echo "avx2 present on this node — continuing (no avx512 requirement with this build)."
+      containers:
+        - name: vllm-chat
+          # Custom build, not public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo —
+          # see this file's header comment for why (confirmed missing
+          # avx512f) and the exact build/push command. This image must
+          # exist in your registry before applying.
+          image: registry.enkly.org/enkly/vllm-cpu:latest
+          args:
+            - --model=Qwen/Qwen2.5-1.5B-Instruct
+            - --served-model-name=$(CHAT_MODEL_NAME)
+            # Torch CPU float16 is unstable per vLLM's own CPU FAQ;
+            # bfloat16 is the documented safe default.
+            - --dtype=bfloat16
+            # Cut from 8192: the KV cache budget below is deliberately
+            # small to fit this node's real ~7.76Gi, and a shorter
+            # context is the lever that buys that back without
+            # dropping the KV cache to the point of thrashing.
+            - --max-model-len=4096
+            # Qwen2.5-Instruct's tool-call token format matches vLLM's
+            # hermes parser (the qwen3_coder parser the GPU deployment
+            # used is Qwen3.5-specific).
+            - --tool-call-parser=hermes
+            - --enable-auto-tool-choice
+            - --api-key=$(VLLM_API_KEY)
+          env:
+            - {name: CHAT_MODEL_NAME, valueFrom: {configMapKeyRef: {name: enkly-config, key: ENKLY_CHAT_MODEL}}}
+            - {name: VLLM_API_KEY, valueFrom: {secretKeyRef: {name: enkly-secrets, key: ENKLY_VLLM_API_KEY}}}
+            - {name: HUGGING_FACE_HUB_TOKEN, valueFrom: {secretKeyRef: {name: enkly-secrets, key: HUGGING_FACE_HUB_TOKEN}}}
+            # Cut from 4 — this plus ~3GB of resident bf16 weights is
+            # what resources.requests.memory below is actually sized
+            # against. Raise only after `kubectl describe node` shows
+            # real free RAM on top of what's already running, and
+            # raise the memory request/limit by the same amount.
+            - {name: VLLM_CPU_KVCACHE_SPACE, value: "1"}
+            # auto binds each OpenMP thread to a physical core and
+            # reserves VLLM_CPU_NUM_OF_RESERVED_CPU cores for the
+            # front-end process — vLLM's own recommended default for
+            # single-rank (no tensor/pipeline parallel) deployments
+            # like this one.
+            - {name: VLLM_CPU_OMP_THREADS_BIND, value: "auto"}
+            - {name: VLLM_CPU_NUM_OF_RESERVED_CPU, value: "1"}
+          ports: [{name: http, containerPort: 8000}]
+          resources:
+            # ~3GB resident for bf16 weights + 1GiB KV cache
+            # (VLLM_CPU_KVCACHE_SPACE above) + CPU-backend
+            # interpreter/runtime overhead (torch, OpenMP worker
+            # threads, inference graph buffers — typically 0.5-1GiB
+            # for a model this size) adds up to a realistic ~4.5Gi
+            # steady-state, not ~4Gi — the previous 3Gi *request*
+            # undersold that, which doesn't block scheduling (the
+            # scheduler only bin-packs on the request) but left too
+            # little daylight to the 5Gi *limit* during the kernel
+            # warmup/compile the startupProbe below already budgets
+            # extra time for, risking an OOMKill right at startup.
+            # 4Gi/6Gi still sits well under this cluster's ~7.76Gi
+            # per-node total (10-rabbitmq.yaml) with room for whatever
+            # else is already scheduled there — the original 8Gi
+            # request couldn't fit on *any* node here, full stop,
+            # which is why it sat Pending; this isn't a return to
+            # that. Raise only after `kubectl describe node` confirms
+            # real headroom.
+            requests: {cpu: "2", memory: 4Gi}
+            limits: {cpu: "3", memory: 6Gi}
+          startupProbe:
+            # No model-download wait this time on a warm PVC (unlike
+            # the brief Ollama detour's explicit pull step) — HF Hub
+            # download only happens once, cached on vllm-chat-cache
+            # after that. What's slow instead is the CPU backend's own
+            # kernel warmup/compile on every start, hence the still
+            # generous budget.
+            httpGet: {path: /health, port: http}
+            periodSeconds: 15
+            failureThreshold: 60 # 15 minutes
+          readinessProbe:
+            httpGet: {path: /health, port: http}
+            periodSeconds: 10
+            failureThreshold: 3
+          livenessProbe:
+            httpGet: {path: /health, port: http}
+            periodSeconds: 20
+            failureThreshold: 3
+          volumeMounts:
+            - {name: hf-cache, mountPath: /root/.cache/huggingface}
+            # vLLM's CPU docs still recommend a real /dev/shm for its
+            # multiprocessing workers, same reasoning as the GPU
+            # version had for NCCL — just a smaller budget since
+            # there's no multi-GPU scratch space involved here.
+            - {name: dshm, mountPath: /dev/shm}
+      volumes:
+        - name: hf-cache
+          persistentVolumeClaim: {claimName: vllm-chat-cache}
+        - name: dshm
+          emptyDir: {medium: Memory, sizeLimit: 1Gi}
+---
+apiVersion: v1
+kind: Service
+metadata: {name: vllm-chat, namespace: enkly, labels: {app: vllm-chat}}
+spec:
+  # ClusterIP only — no public Ingress this time. agent is the only
+  # caller and it's in-cluster; the original GPU manifest's
+  # vllm-chat.enkly.org Ingress existed for direct external testing,
+  # which isn't needed for this deployment to work. Add one back (same
+  # shape as 30-ingress.yaml) if external access is genuinely wanted.
+  selector: {app: vllm-chat}
+  ports: [{name: http, port: 8000, targetPort: http}]
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: vllm-embed, namespace: enkly, labels: {app: vllm-embed}}
+spec:
+  replicas: 1
+  strategy: {type: Recreate}
+  selector: {matchLabels: {app: vllm-embed}}
+  template:
+    metadata: {labels: {app: vllm-embed}}
+    spec:
+      imagePullSecrets: [{name: enkly-registry}]
+      initContainers:
+        - name: check-cpu-avx2
+          image: busybox:1.36
+          command: ["sh", "-c"]
+          args:
+            - |
+              set -e
+              if ! grep -q avx2 /proc/cpuinfo; then
+                echo "FATAL: this node's CPU has no avx2 flag." >&2
+                echo "registry.enkly.org/enkly/vllm-cpu (built with VLLM_CPU_DISABLE_AVX512=true) still needs AVX2 at minimum." >&2
+                exit 1
+              fi
+              echo "avx2 present on this node — continuing (no avx512 requirement with this build)."
+      containers:
+        - name: vllm-embed
+          image: registry.enkly.org/enkly/vllm-cpu:latest
+          args:
+            - --model=nomic-ai/nomic-embed-text-v1.5
+            - --served-model-name=$(EMBED_MODEL_NAME)
+            - --task=embed
+            # Nomic's embedding model config uses custom (non-standard
+            # transformers) modeling code — needs explicit opt-in.
+            - --trust-remote-code
+            - --dtype=bfloat16
+            - --api-key=$(VLLM_API_KEY)
+          env:
+            - {name: EMBED_MODEL_NAME, valueFrom: {configMapKeyRef: {name: enkly-config, key: ENKLY_EMBED_MODEL}}}
+            - {name: VLLM_API_KEY, valueFrom: {secretKeyRef: {name: enkly-secrets, key: ENKLY_VLLM_API_KEY}}}
+            - {name: HUGGING_FACE_HUB_TOKEN, valueFrom: {secretKeyRef: {name: enkly-secrets, key: HUGGING_FACE_HUB_TOKEN}}}
+            # A 137M-parameter embedding model needs far less KV cache
+            # headroom than the chat model.
+            - {name: VLLM_CPU_KVCACHE_SPACE, value: "1"}
+            - {name: VLLM_CPU_OMP_THREADS_BIND, value: "auto"}
+            - {name: VLLM_CPU_NUM_OF_RESERVED_CPU, value: "1"}
+          ports: [{name: http, containerPort: 8000}]
+          resources:
+            # Trimmed from 2Gi/4Gi for the same per-node headroom
+            # reason as vllm-chat above — nomic-embed-text-v1.5 is
+            # ~550MB on disk, so even this is generous.
+            requests: {cpu: "1", memory: 1Gi}
+            limits: {cpu: "2", memory: 2Gi}
+          startupProbe:
+            httpGet: {path: /health, port: http}
+            periodSeconds: 10
+            failureThreshold: 36 # 6 minutes — small model, no large download after first boot
+          readinessProbe:
+            httpGet: {path: /health, port: http}
+            periodSeconds: 10
+            failureThreshold: 3
+          livenessProbe:
+            httpGet: {path: /health, port: http}
+            periodSeconds: 20
+            failureThreshold: 3
+          volumeMounts:
+            - {name: hf-cache, mountPath: /root/.cache/huggingface}
+            - {name: dshm, mountPath: /dev/shm}
+      volumes:
+        - name: hf-cache
+          persistentVolumeClaim: {claimName: vllm-embed-cache}
+        - name: dshm
+          emptyDir: {medium: Memory, sizeLimit: 256Mi}
+---
+apiVersion: v1
+kind: Service
+metadata: {name: vllm-embed, namespace: enkly, labels: {app: vllm-embed}}
+spec:
+  selector: {app: vllm-embed}
+  ports: [{name: http, port: 8000, targetPort: http}]

k8s/12-databases.yaml

@@ -0,0 +1,222 @@
+# ---------- iam-db ----------
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata: {name: iam-db-data, namespace: enkly}
+spec:
+  accessModes: [ReadWriteOnce]
+  storageClassName: managed-nfs-storage
+  resources: {requests: {storage: 5Gi}}
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: iam-db, namespace: enkly, labels: {app: iam-db}}
+spec:
+  replicas: 1
+  strategy: {type: Recreate}
+  selector: {matchLabels: {app: iam-db}}
+  template:
+    metadata: {labels: {app: iam-db}}
+    spec:
+      imagePullSecrets: [{name: enkly-registry}]
+      containers:
+        - name: postgres
+          image: postgres:16-alpine
+          ports: [{name: pg, containerPort: 5432}]
+          env:
+            - {name: POSTGRES_USER, value: "enkly"}
+            - {name: POSTGRES_DB, value: "enkly_iam"}
+            - {name: PGDATA, value: "/var/lib/postgresql/data/pgdata"}
+            - name: POSTGRES_PASSWORD
+              valueFrom: {secretKeyRef: {name: enkly-secrets, key: POSTGRES_PASSWORD}}
+          resources:
+            requests: {cpu: 100m, memory: 256Mi}
+            limits: {cpu: 500m, memory: 512Mi}
+          readinessProbe:
+            exec: {command: ["pg_isready", "-U", "enkly"]}
+            periodSeconds: 5
+            timeoutSeconds: 5
+            failureThreshold: 10
+          livenessProbe:
+            exec: {command: ["pg_isready", "-U", "enkly"]}
+            initialDelaySeconds: 30
+            periodSeconds: 15
+            failureThreshold: 6
+          volumeMounts: [{name: data, mountPath: /var/lib/postgresql/data}]
+      volumes:
+        - name: data
+          persistentVolumeClaim: {claimName: iam-db-data}
+---
+apiVersion: v1
+kind: Service
+metadata: {name: iam-db, namespace: enkly, labels: {app: iam-db}}
+spec:
+  selector: {app: iam-db}
+  ports: [{name: pg, port: 5432, targetPort: pg}]
+
+# ---------- billing-db ----------
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata: {name: billing-db-data, namespace: enkly}
+spec:
+  accessModes: [ReadWriteOnce]
+  storageClassName: managed-nfs-storage
+  resources: {requests: {storage: 5Gi}}
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: billing-db, namespace: enkly, labels: {app: billing-db}}
+spec:
+  replicas: 1
+  strategy: {type: Recreate}
+  selector: {matchLabels: {app: billing-db}}
+  template:
+    metadata: {labels: {app: billing-db}}
+    spec:
+      imagePullSecrets: [{name: enkly-registry}]
+      containers:
+        - name: postgres
+          image: postgres:16-alpine
+          ports: [{name: pg, containerPort: 5432}]
+          env:
+            - {name: POSTGRES_USER, value: "enkly"}
+            - {name: POSTGRES_DB, value: "enkly_billing"}
+            - {name: PGDATA, value: "/var/lib/postgresql/data/pgdata"}
+            - name: POSTGRES_PASSWORD
+              valueFrom: {secretKeyRef: {name: enkly-secrets, key: POSTGRES_PASSWORD}}
+          resources:
+            requests: {cpu: 100m, memory: 256Mi}
+            limits: {cpu: 500m, memory: 512Mi}
+          readinessProbe:
+            exec: {command: ["pg_isready", "-U", "enkly"]}
+            periodSeconds: 5
+            timeoutSeconds: 5
+            failureThreshold: 10
+          livenessProbe:
+            exec: {command: ["pg_isready", "-U", "enkly"]}
+            initialDelaySeconds: 30
+            periodSeconds: 15
+            failureThreshold: 6
+          volumeMounts: [{name: data, mountPath: /var/lib/postgresql/data}]
+      volumes:
+        - name: data
+          persistentVolumeClaim: {claimName: billing-db-data}
+---
+apiVersion: v1
+kind: Service
+metadata: {name: billing-db, namespace: enkly, labels: {app: billing-db}}
+spec:
+  selector: {app: billing-db}
+  ports: [{name: pg, port: 5432, targetPort: pg}]
+
+# ---------- conversation-db ----------
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata: {name: conversation-db-data, namespace: enkly}
+spec:
+  accessModes: [ReadWriteOnce]
+  storageClassName: managed-nfs-storage
+  resources: {requests: {storage: 5Gi}}
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: conversation-db, namespace: enkly, labels: {app: conversation-db}}
+spec:
+  replicas: 1
+  strategy: {type: Recreate}
+  selector: {matchLabels: {app: conversation-db}}
+  template:
+    metadata: {labels: {app: conversation-db}}
+    spec:
+      imagePullSecrets: [{name: enkly-registry}]
+      containers:
+        - name: postgres
+          image: postgres:16-alpine
+          ports: [{name: pg, containerPort: 5432}]
+          env:
+            - {name: POSTGRES_USER, value: "enkly"}
+            - {name: POSTGRES_DB, value: "enkly_conversation"}
+            - {name: PGDATA, value: "/var/lib/postgresql/data/pgdata"}
+            - name: POSTGRES_PASSWORD
+              valueFrom: {secretKeyRef: {name: enkly-secrets, key: POSTGRES_PASSWORD}}
+          resources:
+            requests: {cpu: 100m, memory: 256Mi}
+            limits: {cpu: 500m, memory: 512Mi}
+          readinessProbe:
+            exec: {command: ["pg_isready", "-U", "enkly"]}
+            periodSeconds: 5
+            timeoutSeconds: 5
+            failureThreshold: 10
+          livenessProbe:
+            exec: {command: ["pg_isready", "-U", "enkly"]}
+            initialDelaySeconds: 30
+            periodSeconds: 15
+            failureThreshold: 6
+          volumeMounts: [{name: data, mountPath: /var/lib/postgresql/data}]
+      volumes:
+        - name: data
+          persistentVolumeClaim: {claimName: conversation-db-data}
+---
+apiVersion: v1
+kind: Service
+metadata: {name: conversation-db, namespace: enkly, labels: {app: conversation-db}}
+spec:
+  selector: {app: conversation-db}
+  ports: [{name: pg, port: 5432, targetPort: pg}]
+
+# ---------- knowledge-db (pgvector) ----------
+---
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata: {name: knowledge-db-data, namespace: enkly}
+spec:
+  accessModes: [ReadWriteOnce]
+  storageClassName: managed-nfs-storage
+  resources: {requests: {storage: 10Gi}}
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: knowledge-db, namespace: enkly, labels: {app: knowledge-db}}
+spec:
+  replicas: 1
+  strategy: {type: Recreate}
+  selector: {matchLabels: {app: knowledge-db}}
+  template:
+    metadata: {labels: {app: knowledge-db}}
+    spec:
+      imagePullSecrets: [{name: enkly-registry}]
+      containers:
+        - name: postgres
+          image: pgvector/pgvector:pg16
+          ports: [{name: pg, containerPort: 5432}]
+          env:
+            - {name: POSTGRES_USER, value: "enkly"}
+            - {name: POSTGRES_DB, value: "enkly_knowledge"}
+            - {name: PGDATA, value: "/var/lib/postgresql/data/pgdata"}
+            - name: POSTGRES_PASSWORD
+              valueFrom: {secretKeyRef: {name: enkly-secrets, key: POSTGRES_PASSWORD}}
+          resources:
+            requests: {cpu: 200m, memory: 512Mi}
+            limits: {cpu: 1, memory: 1Gi}
+          readinessProbe:
+            exec: {command: ["pg_isready", "-U", "enkly"]}
+            periodSeconds: 5
+            timeoutSeconds: 5
+            failureThreshold: 10
+          livenessProbe:
+            exec: {command: ["pg_isready", "-U", "enkly"]}
+            initialDelaySeconds: 30
+            periodSeconds: 15
+            failureThreshold: 6
+          volumeMounts: [{name: data, mountPath: /var/lib/postgresql/data}]
+      volumes:
+        - name: data
+          persistentVolumeClaim: {claimName: knowledge-db-data}
+---
+apiVersion: v1
+kind: Service
+metadata: {name: knowledge-db, namespace: enkly, labels: {app: knowledge-db}}
+spec:
+  selector: {app: knowledge-db}
+  ports: [{name: pg, port: 5432, targetPort: pg}]

k8s/20-iam.yaml

@@ -0,0 +1,45 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: iam, namespace: enkly, labels: {app: iam}}
+spec:
+  replicas: 1
+  selector: {matchLabels: {app: iam}}
+  template:
+    metadata: {labels: {app: iam}}
+    spec:
+      imagePullSecrets: [{name: enkly-registry}]
+      # Nothing in Kubernetes enforces start order between Deployments, and
+      # rabbitmq's own readiness probe can take a while to go green (see
+      # 10-rabbitmq.yaml). Without this, iam races the broker on a fresh
+      # rollout and can hit the same "connection refused" notification saw.
+      initContainers:
+        - name: wait-for-rabbitmq
+          image: busybox:1.36
+          command: ["sh", "-c", "until nc -z rabbitmq 5672; do echo waiting for rabbitmq; sleep 2; done"]
+      containers:
+        - name: iam
+          image: registry.enkly.org/enkly/iam:latest
+          ports: [{name: grpc, containerPort: 9001}]
+          envFrom: [{configMapRef: {name: enkly-config}}]
+          env:
+            - name: POSTGRES_PASSWORD
+              valueFrom: {secretKeyRef: {name: enkly-secrets, key: POSTGRES_PASSWORD}}
+            - {name: ENKLY_IAM_POSTGRES_DSN, value: "postgres://enkly:$(POSTGRES_PASSWORD)@iam-db:5432/enkly_iam?sslmode=disable"}
+            - name: ENKLY_RABBITMQ_URL
+              valueFrom: {secretKeyRef: {name: enkly-secrets, key: ENKLY_RABBITMQ_URL}}
+            - name: ENKLY_SMTP_USERNAME
+              valueFrom: {secretKeyRef: {name: enkly-secrets, key: ENKLY_SMTP_USERNAME}}
+            - name: ENKLY_SMTP_PASSWORD
+              valueFrom: {secretKeyRef: {name: enkly-secrets, key: ENKLY_SMTP_PASSWORD}}
+          resources:
+            requests: {cpu: 50m, memory: 64Mi}
+            limits: {cpu: 500m, memory: 256Mi}
+          readinessProbe: {grpc: {port: 9001}, initialDelaySeconds: 5, periodSeconds: 10, failureThreshold: 10}
+          livenessProbe: {grpc: {port: 9001}, initialDelaySeconds: 15, periodSeconds: 20, failureThreshold: 6}
+---
+apiVersion: v1
+kind: Service
+metadata: {name: iam, namespace: enkly, labels: {app: iam}}
+spec:
+  selector: {app: iam}
+  ports: [{name: grpc, port: 9001, targetPort: grpc}]

k8s/21-billing.yaml

@@ -0,0 +1,49 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: billing, namespace: enkly, labels: {app: billing}}
+spec:
+  replicas: 1
+  selector: {matchLabels: {app: billing}}
+  template:
+    metadata: {labels: {app: billing}}
+    spec:
+      imagePullSecrets: [{name: enkly-registry}]
+      initContainers:
+        - name: wait-for-rabbitmq
+          image: busybox:1.36
+          command: ["sh", "-c", "until nc -z rabbitmq 5672; do echo waiting for rabbitmq; sleep 2; done"]
+      containers:
+        - name: billing
+          image: registry.enkly.org/enkly/billing:latest
+          ports: [{name: grpc, containerPort: 9002}]
+          envFrom: [{configMapRef: {name: enkly-config}}]
+          env:
+            - name: POSTGRES_PASSWORD
+              valueFrom: {secretKeyRef: {name: enkly-secrets, key: POSTGRES_PASSWORD}}
+            - {name: ENKLY_BILLING_POSTGRES_DSN, value: "postgres://enkly:$(POSTGRES_PASSWORD)@billing-db:5432/enkly_billing?sslmode=disable"}
+            - name: ENKLY_RABBITMQ_URL
+              valueFrom: {secretKeyRef: {name: enkly-secrets, key: ENKLY_RABBITMQ_URL}}
+            - name: STRIPE_SECRET_KEY
+              valueFrom: {secretKeyRef: {name: enkly-secrets, key: STRIPE_SECRET_KEY}}
+            - name: STRIPE_PUBLISH_KEY
+              valueFrom: {secretKeyRef: {name: enkly-secrets, key: STRIPE_PUBLISH_KEY}}
+            - name: STRIPE_WEBHOOK_SECRET
+              valueFrom: {secretKeyRef: {name: enkly-secrets, key: STRIPE_WEBHOOK_SECRET}}
+            - name: STRIPE_PRICE_BASIC
+              valueFrom: {secretKeyRef: {name: enkly-secrets, key: STRIPE_PRICE_BASIC}}
+            - name: STRIPE_PRICE_PRO
+              valueFrom: {secretKeyRef: {name: enkly-secrets, key: STRIPE_PRICE_PRO}}
+            - name: STRIPE_PRICE_PREMIUM
+              valueFrom: {secretKeyRef: {name: enkly-secrets, key: STRIPE_PRICE_PREMIUM}}
+          resources:
+            requests: {cpu: 50m, memory: 64Mi}
+            limits: {cpu: 500m, memory: 256Mi}
+          readinessProbe: {grpc: {port: 9002}, initialDelaySeconds: 5, periodSeconds: 10, failureThreshold: 10}
+          livenessProbe: {grpc: {port: 9002}, initialDelaySeconds: 15, periodSeconds: 20, failureThreshold: 6}
+---
+apiVersion: v1
+kind: Service
+metadata: {name: billing, namespace: enkly, labels: {app: billing}}
+spec:
+  selector: {app: billing}
+  ports: [{name: grpc, port: 9002, targetPort: grpc}]

k8s/22-conversation.yaml

@@ -0,0 +1,37 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: conversation, namespace: enkly, labels: {app: conversation}}
+spec:
+  replicas: 1
+  selector: {matchLabels: {app: conversation}}
+  template:
+    metadata: {labels: {app: conversation}}
+    spec:
+      imagePullSecrets: [{name: enkly-registry}]
+      initContainers:
+        - name: wait-for-rabbitmq
+          image: busybox:1.36
+          command: ["sh", "-c", "until nc -z rabbitmq 5672; do echo waiting for rabbitmq; sleep 2; done"]
+      containers:
+        - name: conversation
+          image: registry.enkly.org/enkly/conversation:latest
+          ports: [{name: grpc, containerPort: 9003}]
+          envFrom: [{configMapRef: {name: enkly-config}}]
+          env:
+            - name: POSTGRES_PASSWORD
+              valueFrom: {secretKeyRef: {name: enkly-secrets, key: POSTGRES_PASSWORD}}
+            - {name: ENKLY_CONVERSATION_POSTGRES_DSN, value: "postgres://enkly:$(POSTGRES_PASSWORD)@conversation-db:5432/enkly_conversation?sslmode=disable"}
+            - name: ENKLY_RABBITMQ_URL
+              valueFrom: {secretKeyRef: {name: enkly-secrets, key: ENKLY_RABBITMQ_URL}}
+          resources:
+            requests: {cpu: 50m, memory: 64Mi}
+            limits: {cpu: 500m, memory: 256Mi}
+          readinessProbe: {grpc: {port: 9003}, initialDelaySeconds: 5, periodSeconds: 10, failureThreshold: 10}
+          livenessProbe: {grpc: {port: 9003}, initialDelaySeconds: 15, periodSeconds: 20, failureThreshold: 6}
+---
+apiVersion: v1
+kind: Service
+metadata: {name: conversation, namespace: enkly, labels: {app: conversation}}
+spec:
+  selector: {app: conversation}
+  ports: [{name: grpc, port: 9003, targetPort: grpc}]

k8s/23-workspace.yaml

@@ -0,0 +1,45 @@
+# Matches docker-compose's workspace-data volume: a PVC mounted at
+# /data/workspace. Single replica + Recreate, same reasoning as the
+# databases — this is where run_bash_command actually executes, so a
+# second concurrent pod writing the same tree would be worse than a
+# brief restart gap.
+apiVersion: v1
+kind: PersistentVolumeClaim
+metadata: {name: workspace-data, namespace: enkly}
+spec:
+  accessModes: [ReadWriteOnce]
+  storageClassName: managed-nfs-storage
+  resources: {requests: {storage: 10Gi}}
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: workspace, namespace: enkly, labels: {app: workspace}}
+spec:
+  replicas: 1
+  strategy: {type: Recreate}
+  selector: {matchLabels: {app: workspace}}
+  template:
+    metadata: {labels: {app: workspace}}
+    spec:
+      imagePullSecrets: [{name: enkly-registry}]
+      containers:
+        - name: workspace
+          image: registry.enkly.org/enkly/workspace:latest
+          ports: [{name: grpc, containerPort: 9004}]
+          envFrom: [{configMapRef: {name: enkly-config}}]
+          resources:
+            requests: {cpu: 100m, memory: 128Mi}
+            limits: {cpu: 1, memory: 512Mi}
+          readinessProbe: {grpc: {port: 9004}, initialDelaySeconds: 5, periodSeconds: 10, failureThreshold: 10}
+          livenessProbe: {grpc: {port: 9004}, initialDelaySeconds: 15, periodSeconds: 20, failureThreshold: 6}
+          volumeMounts: [{name: data, mountPath: /data/workspace}]
+      volumes:
+        - name: data
+          persistentVolumeClaim: {claimName: workspace-data}
+---
+apiVersion: v1
+kind: Service
+metadata: {name: workspace, namespace: enkly, labels: {app: workspace}}
+spec:
+  selector: {app: workspace}
+  ports: [{name: grpc, port: 9004, targetPort: grpc}]

k8s/24-knowledge.yaml

@@ -0,0 +1,36 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: knowledge, namespace: enkly, labels: {app: knowledge}}
+spec:
+  replicas: 1
+  selector: {matchLabels: {app: knowledge}}
+  template:
+    metadata: {labels: {app: knowledge}}
+    spec:
+      imagePullSecrets: [{name: enkly-registry}]
+      containers:
+        - name: knowledge
+          image: registry.enkly.org/enkly/knowledge:latest
+          ports: [{name: grpc, containerPort: 9005}]
+          envFrom: [{configMapRef: {name: enkly-config}}]
+          env:
+            - name: POSTGRES_PASSWORD
+              valueFrom: {secretKeyRef: {name: enkly-secrets, key: POSTGRES_PASSWORD}}
+            - {name: ENKLY_KNOWLEDGE_POSTGRES_DSN, value: "postgres://enkly:$(POSTGRES_PASSWORD)@knowledge-db:5432/enkly_knowledge?sslmode=disable"}
+            # Bearer token knowledge authenticates to vllm-embed with
+            # (see services/knowledge/cmd/server/main.go) — without
+            # this, embedding calls 401 against vllm-embed's --api-key.
+            - name: ENKLY_VLLM_API_KEY
+              valueFrom: {secretKeyRef: {name: enkly-secrets, key: ENKLY_VLLM_API_KEY}}
+          resources:
+            requests: {cpu: 100m, memory: 128Mi}
+            limits: {cpu: 1, memory: 512Mi}
+          readinessProbe: {grpc: {port: 9005}, initialDelaySeconds: 5, periodSeconds: 10, failureThreshold: 10}
+          livenessProbe: {grpc: {port: 9005}, initialDelaySeconds: 15, periodSeconds: 20, failureThreshold: 6}
+---
+apiVersion: v1
+kind: Service
+metadata: {name: knowledge, namespace: enkly, labels: {app: knowledge}}
+spec:
+  selector: {app: knowledge}
+  ports: [{name: grpc, port: 9005, targetPort: grpc}]

k8s/25-agent.yaml

@@ -0,0 +1,42 @@
+# ENKLY_WORKSPACE_GRPC_ADDR here means something different from the
+# same-named var on the workspace Deployment: there it's "what address
+# do I listen on", here it's "what address do I dial" (agent is a
+# gRPC client of workspace/knowledge/billing/conversation). That's how
+# docker-compose.yml defines it too — kept identical on purpose.
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: agent, namespace: enkly, labels: {app: agent}}
+spec:
+  replicas: 1
+  selector: {matchLabels: {app: agent}}
+  template:
+    metadata: {labels: {app: agent}}
+    spec:
+      imagePullSecrets: [{name: enkly-registry}]
+      containers:
+        - name: agent
+          image: registry.enkly.org/enkly/agent:latest
+          ports: [{name: grpc, containerPort: 9006}]
+          envFrom: [{configMapRef: {name: enkly-config}}]
+          env:
+            - {name: ENKLY_WORKSPACE_GRPC_ADDR, value: "workspace:9004"}
+            - {name: ENKLY_KNOWLEDGE_GRPC_ADDR, value: "knowledge:9005"}
+            - {name: ENKLY_BILLING_GRPC_ADDR, value: "billing:9002"}
+            - {name: ENKLY_CONVERSATION_GRPC_ADDR, value: "conversation:9003"}
+            # Bearer token agent authenticates to vllm-chat/vllm-embed
+            # with (see services/agent/cmd/server/main.go) — without
+            # this, chat/tool-calling calls 401 against vLLM's --api-key.
+            - name: ENKLY_VLLM_API_KEY
+              valueFrom: {secretKeyRef: {name: enkly-secrets, key: ENKLY_VLLM_API_KEY}}
+          resources:
+            requests: {cpu: 100m, memory: 128Mi}
+            limits: {cpu: 1, memory: 512Mi}
+          readinessProbe: {grpc: {port: 9006}, initialDelaySeconds: 5, periodSeconds: 10, failureThreshold: 10}
+          livenessProbe: {grpc: {port: 9006}, initialDelaySeconds: 15, periodSeconds: 20, failureThreshold: 6}
+---
+apiVersion: v1
+kind: Service
+metadata: {name: agent, namespace: enkly, labels: {app: agent}}
+spec:
+  selector: {app: agent}
+  ports: [{name: grpc, port: 9006, targetPort: grpc}]

k8s/26-notification.yaml

@@ -0,0 +1,38 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: notification, namespace: enkly, labels: {app: notification}}
+spec:
+  replicas: 1
+  selector: {matchLabels: {app: notification}}
+  template:
+    metadata: {labels: {app: notification}}
+    spec:
+      imagePullSecrets: [{name: enkly-registry}]
+      initContainers:
+        - name: wait-for-rabbitmq
+          image: busybox:1.36
+          command: ["sh", "-c", "until nc -z rabbitmq 5672; do echo waiting for rabbitmq; sleep 2; done"]
+      containers:
+        - name: notification
+          image: registry.enkly.org/enkly/notification:latest
+          ports: [{name: http, containerPort: 9007}]
+          envFrom: [{configMapRef: {name: enkly-config}}]
+          env:
+            - name: ENKLY_RABBITMQ_URL
+              valueFrom: {secretKeyRef: {name: enkly-secrets, key: ENKLY_RABBITMQ_URL}}
+            - name: ENKLY_SMTP_USERNAME
+              valueFrom: {secretKeyRef: {name: enkly-secrets, key: ENKLY_SMTP_USERNAME}}
+            - name: ENKLY_SMTP_PASSWORD
+              valueFrom: {secretKeyRef: {name: enkly-secrets, key: ENKLY_SMTP_PASSWORD}}
+          resources:
+            requests: {cpu: 50m, memory: 64Mi}
+            limits: {cpu: 300m, memory: 192Mi}
+          readinessProbe: {httpGet: {path: /healthz, port: http}, initialDelaySeconds: 5, periodSeconds: 10, failureThreshold: 10}
+          livenessProbe: {httpGet: {path: /healthz, port: http}, initialDelaySeconds: 15, periodSeconds: 20, failureThreshold: 6}
+---
+apiVersion: v1
+kind: Service
+metadata: {name: notification, namespace: enkly, labels: {app: notification}}
+spec:
+  selector: {app: notification}
+  ports: [{name: http, port: 9007, targetPort: http}]

k8s/27-gateway.yaml

@@ -0,0 +1,36 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: gateway, namespace: enkly, labels: {app: gateway}}
+spec:
+  replicas: 2
+  selector: {matchLabels: {app: gateway}}
+  template:
+    metadata: {labels: {app: gateway}}
+    spec:
+      imagePullSecrets: [{name: enkly-registry}]
+      containers:
+        - name: gateway
+          image: registry.enkly.org/enkly/gateway:latest
+          ports: [{name: http, containerPort: 8080}]
+          envFrom: [{configMapRef: {name: enkly-config}}]
+          env:
+            - {name: ENKLY_IAM_GRPC_ADDR, value: "iam:9001"}
+            - {name: ENKLY_BILLING_GRPC_ADDR, value: "billing:9002"}
+            - {name: ENKLY_CONVERSATION_GRPC_ADDR, value: "conversation:9003"}
+            - {name: ENKLY_WORKSPACE_GRPC_ADDR, value: "workspace:9004"}
+            - {name: ENKLY_KNOWLEDGE_GRPC_ADDR, value: "knowledge:9005"}
+            - {name: ENKLY_AGENT_GRPC_ADDR, value: "agent:9006"}
+            - name: STRIPE_PUBLISH_KEY
+              valueFrom: {secretKeyRef: {name: enkly-secrets, key: STRIPE_PUBLISH_KEY}}
+          resources:
+            requests: {cpu: 100m, memory: 96Mi}
+            limits: {cpu: 500m, memory: 256Mi}
+          readinessProbe: {httpGet: {path: /healthz, port: http}, initialDelaySeconds: 5, periodSeconds: 10, failureThreshold: 10}
+          livenessProbe: {httpGet: {path: /healthz, port: http}, initialDelaySeconds: 15, periodSeconds: 20, failureThreshold: 6}
+---
+apiVersion: v1
+kind: Service
+metadata: {name: gateway, namespace: enkly, labels: {app: gateway}}
+spec:
+  selector: {app: gateway}
+  ports: [{name: http, port: 8080, targetPort: http}]

k8s/30-ingress.yaml

@@ -0,0 +1,40 @@
+# Requires an ingress controller (e.g. ingress-nginx) already running
+# in the cluster — `kubectl get nodes` output didn't show one and
+# none was mentioned, so install one first:
+#   kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.11.3/deploy/static/provider/baremetal/deploy.yaml
+# If you'd rather not run an ingress controller, delete this file and
+# uncomment the NodePort Service below instead.
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: gateway
+  namespace: enkly
+  annotations:
+    cert-manager.io/cluster-issuer: production
+    nginx.ingress.kubernetes.io/proxy-body-size: "25m"   # chat file/photo attachments
+    nginx.ingress.kubernetes.io/proxy-read-timeout: "300" # long-running agent/chat responses
+spec:
+  ingressClassName: nginx
+  tls:
+    - hosts:
+        - ai.enkly.org # match ENKLY_BASE_URL in 01-config.yaml, or set to your real domain 
+      secretName: ai-enkly-org
+  rules:
+    - host: ai.enkly.org
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service: {name: gateway, port: {number: 8080}}
+---
+# Bare-metal fallback with no ingress controller installed: exposes
+# the gateway directly on every node's IP at a fixed high port. Apply
+# this INSTEAD OF (not in addition to) the Ingress above if preferred.
+# apiVersion: v1
+# kind: Service
+# metadata: {name: gateway-nodeport, namespace: enkly}
+# spec:
+#   type: NodePort
+#   selector: {app: gateway}
+#   ports: [{name: http, port: 8080, targetPort: http, nodePort: 30080}]

k8s/README.md

@@ -0,0 +1,222 @@
+# enkly-ai — Kubernetes deployment
+
+Plain manifests (no Helm) for the 8 Go microservices + RabbitMQ + 4
+Postgres databases + two CPU-only vLLM inference servers, for the
+`kmaster` / `kworker1` / `kworker2` cluster described in
+`docker-compose.yml`. One file per concern, applied individually or
+via `kustomization.yaml`.
+
+## vLLM on a cluster with no GPU
+
+An earlier version of this manifest set ran `vllm-chat`/`vllm-embed`
+on the CUDA `vllm/vllm-openai` image, each requesting
+`nvidia.com/gpu: "1"`. **This cluster has no physical GPU on any of
+its three nodes.** `nvidia.com/gpu: "1"` is not a resource any node
+here can ever advertise — with no node reporting it, both pods sat
+`Pending` permanently (`Insufficient nvidia.com/gpu`), not just slow
+to schedule.
+
+`11-vllm.yaml` runs vLLM's CPU backend instead — same two Deployments,
+same OpenAI-compatible API, same `--api-key` enforcement, no
+application code changes. Two real, *observed* problems (not just
+theoretical GPU-vs-CPU ones) shaped the rest of the fix:
+
+1. **`avx512f` genuinely missing on this cluster's nodes.** vLLM's
+   official pre-built CPU image
+   (`public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo`) hard-requires
+   that flag and crashes with `Illegal instruction` without it — and
+   an init container added to both Deployments confirmed it's absent
+   here, consistent with VirtualBox/Vagrant nested virtualization
+   (this project's own `vagrant-provisioning/` setup) historically not
+   exposing AVX-512 to guest VMs at all. There's no runtime flag for
+   this, only a build-time one, so `image:` on both Deployments now
+   points at a **custom build you must create first**:
+   `registry.enkly.org/enkly/vllm-cpu:latest`, built with
+   `--build-arg VLLM_CPU_DISABLE_AVX512=true` — see `11-vllm.yaml`'s
+   header for the exact command. That build falls back to plain AVX2,
+   universal on any x86_64 node since ~2013, so the init container now
+   only checks for that (a much lower, already-satisfied bar) instead
+   of gating on `avx512f`.
+2. **The chat model's memory request alone exceeded a node's total
+   RAM.** `10-rabbitmq.yaml`'s memory-watermark fix already
+   established each node has ~7949Mi (~7.76Gi) total — `vllm-chat`'s
+   original `8Gi` request could never be satisfied by *any* node here,
+   which is exactly why it sat `Pending` instead of crashing. Fixed by
+   dropping the chat model to `Qwen/Qwen2.5-1.5B-Instruct` (~3GB bf16
+   vs. the 3B variant's ~6GB) and cutting
+   `resources`/`VLLM_CPU_KVCACHE_SPACE`/`--max-model-len` on both
+   Deployments to fit comfortably inside ~7.76Gi alongside everything
+   else already running on that node (`vllm-chat` requests `4Gi`/limits
+   `6Gi` — sized to the realistic weights + KV-cache + runtime-overhead
+   total, not the bare minimum, so there's headroom left to the limit
+   during startup warmup).
+3. **A shared `ReadWriteMany` PVC was an unverified assumption, not a
+   confirmed fix.** `vllm-chat` and `vllm-embed` originally bind-mounted
+   one PVC for their Hugging Face cache. `managed-nfs-storage`
+   (`nfs-subdir-external-provisioner`) does advertise RWX, so that
+   probably binds fine — but "probably" produces exactly the kind of
+   pod that's silently stuck `Pending` on an unbound PVC, easy to
+   mis-diagnose as a repeat of the GPU/memory issues above. Split into
+   `vllm-chat-cache` and `vllm-embed-cache`, both plain `ReadWriteOnce`
+   — the same access mode every other PVC in this project already binds
+   with — so neither Deployment's scheduling can depend on the other's
+   claim or on RWX ever having been exercised here.
+
+If real GPU nodes get added to this cluster later: point `image:` on
+both Deployments back at `vllm/vllm-openai`, add back
+`nvidia.com/gpu` requests + a GPU-pool `nodeSelector`/`toleration` +
+an NVIDIA device-plugin DaemonSet, and size the model/resources back
+up.
+
+## Other issues found in the manifest set and how they were fixed here
+
+1. **Postgres on NFS-backed PVCs fails `initdb`.**
+   `nfs-subdir-external-provisioner` volumes aren't always empty at
+   the Postgres data-dir mount point, and `initdb` refuses to
+   initialize a non-empty directory. All four databases set
+   `PGDATA=/var/lib/postgresql/data/pgdata` (a subdirectory of the
+   mount) to sidestep this — see `12-databases.yaml`.
+2. **`agent`/`knowledge` never received the vLLM API key.** Both
+   services' own code (`services/agent/cmd/server/main.go`,
+   `services/knowledge/cmd/server/main.go`) reads
+   `ENKLY_VLLM_API_KEY` from the environment and defaults to `""` if
+   unset — but neither `24-knowledge.yaml` nor `25-agent.yaml` sourced
+   it from `enkly-secrets`. Since `11-vllm.yaml` starts both vLLM
+   servers with `--api-key=$(VLLM_API_KEY)`, every chat/embedding call
+   from these two services would get rejected with `401`. Both files
+   now set `ENKLY_VLLM_API_KEY` from `secretKeyRef: {name:
+   enkly-secrets, key: ENKLY_VLLM_API_KEY}`.
+3. **`02-secrets.yaml` had what read as live credentials committed as
+   if they were placeholders.** `STRIPE_SECRET_KEY`/
+   `STRIPE_PUBLISH_KEY` were filled with what look like real Stripe
+   test-mode keys, and `ENKLY_SMTP_USERNAME`/`ENKLY_SMTP_PASSWORD`
+   held a real-looking Gmail address and a 16-character app password.
+   **Rotate the Gmail app password and the Stripe keys if this file
+   was ever pushed anywhere outside a local sandbox** — replacing the
+   values here doesn't undo prior exposure. Replace all four with
+   your own before applying. Separately, `ENKLY_VLLM_API_KEY` should
+   be a real random value before applying — YAML `stringData` doesn't
+   run shell commands, so a literal `"$(openssl rand -hex 32)"` string
+   would become the actual, non-random key; see that key's comment in
+   `02-secrets.yaml` for the correct one-liner.
+4. **Registry auth/TLS is cluster-level, not manifest-level.** Every
+   Deployment references `imagePullSecrets: [enkly-registry]`, but
+   that Secret must be created out-of-band (real credentials can't
+   live in a file meant for a repo — see `02-secrets.yaml`'s header
+   comment). Likewise, if `registry.enkly.org` serves a self-signed or
+   internal CA cert, every node's containerd needs that CA trusted via
+   `/etc/containerd/certs.d/registry.enkly.org/hosts.toml` — that's
+   node provisioning, outside what any Kubernetes manifest can
+   express.
+
+## Layout
+
+| File | What |
+|---|---|
+| `00-namespace.yaml` | the `enkly` namespace |
+| `01-config.yaml` | non-secret `ENKLY_*` config, shared by every service |
+| `02-secrets.yaml` | **template** — DB/AMQP/Stripe/SMTP/HF secret + instructions for the registry pull secret |
+| `10-rabbitmq.yaml` | RabbitMQ, with a PVC (so broker state survives a pod restart) |
+| `11-vllm.yaml` | vLLM chat + embed Deployments/Services, CPU backend (no GPU), each with its own RWO Hugging Face cache PVC — see above |
+| `12-databases.yaml` | iam-db, billing-db, conversation-db, knowledge-db (pgvector), each with its own PVC |
+| `20`–`27` | iam, billing, conversation, workspace, knowledge, agent, notification, gateway |
+| `30-ingress.yaml` | Ingress for the gateway (+ commented NodePort fallback) |
+| `kustomization.yaml` | ties the above together for `kubectl apply -k .` |
+
+Every Deployment's env vars are a 1:1 mapping from
+`docker-compose.yml` — service hostnames are identical to the compose
+service names, so the only real change the application sees is DNS
+resolving inside a namespace instead of a compose network.
+
+## Deploy
+
+Two things have to be true *before* any apply, because nothing in
+Kubernetes can discover them on its own: your image registry has to
+have the 8 service images, and your real Stripe/SMTP credentials have
+to come from you.
+
+```bash
+# 1. Build & push the 8 images
+REG=registry.enkly.org/enkly
+for svc in iam billing conversation workspace knowledge agent notification gateway; do
+  docker build -f services/$svc/Dockerfile -t $REG/$svc:latest .
+  docker push $REG/$svc:latest
+done
+
+# 2. Registry pull secret (every Deployment references this)
+kubectl create secret docker-registry enkly-registry -n enkly \
+  --docker-server=registry.enkly.org \
+  --docker-username=<user> --docker-password=<token> --docker-email=<email>
+
+# 3. Edit 02-secrets.yaml: fill in STRIPE_*/ENKLY_SMTP_* with your own
+#    rotated values (see issue #3 above), and generate a real
+#    ENKLY_VLLM_API_KEY (the file documents the exact one-liner).
+
+# 4. Build & push the CPU-only, no-AVX512 vLLM image — this cluster's
+#    nodes were confirmed to lack avx512f, so the official pre-built
+#    image (public.ecr.aws/q9t5s3a7/vllm-cpu-release-repo) will crash
+#    with "Illegal instruction"; 11-vllm.yaml's image: already points
+#    at the custom tag this produces:
+git clone https://github.com/vllm-project/vllm.git /tmp/vllm-build && cd /tmp/vllm-build
+docker build -f docker/Dockerfile.cpu \
+  --build-arg VLLM_CPU_DISABLE_AVX512=true \
+  --tag registry.enkly.org/enkly/vllm-cpu:latest \
+  --target vllm-openai .
+docker push registry.enkly.org/enkly/vllm-cpu:latest
+
+# 5. Apply everything
+kubectl apply -k k8s/
+
+# 6. Watch the rollout — first boot downloads ~3.5GB of Hugging Face
+#    weights total onto vllm-chat-cache/vllm-embed-cache, so the
+#    startupProbe budget is generous.
+kubectl -n enkly get pods -w
+kubectl -n enkly rollout status deploy/vllm-chat --timeout=15m
+kubectl -n enkly rollout status deploy/vllm-embed --timeout=15m
+kubectl -n enkly rollout status deploy/gateway
+```
+
+Then point DNS/`/etc/hosts` at the Ingress controller's address for
+the host set in `30-ingress.yaml` / `ENKLY_BASE_URL` in
+`01-config.yaml`, or use the commented NodePort Service in
+`30-ingress.yaml` if no ingress controller is installed
+(`kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.11.3/deploy/static/provider/baremetal/deploy.yaml`
+installs one on bare metal).
+
+### Verify the LLM path specifically
+
+```bash
+kubectl -n enkly logs deploy/vllm-chat -f   # watch the HF download + CPU backend warmup
+kubectl -n enkly logs deploy/vllm-embed -f
+kubectl -n enkly port-forward svc/gateway 8080:8080   # then run a chat turn end to end
+```
+
+If a pod is stuck in `Init:Error` / `Init:CrashLoopBackOff`, that's
+the `check-cpu-avx2` init container — this cluster already needed the
+custom no-AVX512 image (see above), so seeing this now would mean a
+node lacks even AVX2, which no mainstream x86_64 CPU built since
+~2013 should be missing; read its log for which node and double-check
+`lscpu` there directly. If instead the main `vllm-chat`/`vllm-embed`
+container itself is crashing after the init container passed, check
+for `OOMKilled` in `kubectl -n enkly describe pod` specifically (this
+cluster's nodes only have ~7.76Gi RAM total each — raise
+`resources.limits.memory` cautiously, if at all, and prefer lowering
+`VLLM_CPU_KVCACHE_SPACE` or `--max-model-len` instead) over assuming
+it's a repeat of the AVX-512 issue.
+
+## Scaling beyond the defaults
+
+`iam`, `billing`, `conversation`, `knowledge`, `agent`, `notification`
+and `gateway` are stateless and safe to scale (`kubectl -n enkly scale
+deploy/gateway --replicas=3`). `workspace`, the four Postgres
+Deployments, `vllm-chat` and `vllm-embed` are all pinned to
+`replicas: 1` with `strategy: Recreate` because each owns a
+`ReadWriteOnce` PVC — don't raise their replica counts without first
+either moving to a clustered/replicated backend (e.g. Patroni for
+Postgres) or, for the vLLM pair specifically, converting
+`vllm-chat-cache`/`vllm-embed-cache` to `ReadWriteMany` (cheap to try:
+`managed-nfs-storage` already advertises it) and switching
+`strategy.type` to `RollingUpdate`. Either way, remember each replica
+is a full, independent CPU-bound process (no GPU to share), so scaling
+trades throughput for a proportional increase in CPU/RAM footprint,
+not a free win the way request-level load balancing sometimes is.

k8s/kustomization.yaml

@@ -0,0 +1,19 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: enkly
+resources:
+  - 00-namespace.yaml
+  - 01-config.yaml
+  - 02-secrets.yaml
+  - 10-rabbitmq.yaml
+  - 11-vllm.yaml
+  - 12-databases.yaml
+  - 20-iam.yaml
+  - 21-billing.yaml
+  - 22-conversation.yaml
+  - 23-workspace.yaml
+  - 24-knowledge.yaml
+  - 25-agent.yaml
+  - 26-notification.yaml
+  - 27-gateway.yaml
+  - 30-ingress.yaml