en
new
0
Avatar
Tuấn Kiệt @kevinram
1.1K 101 15
Published about 6 hours ago 10 min read
11

Banking Demo - GITOPS : Phần 2 - Triển khai GitOps trên OpenShift

Phase 9 — Triển khai GitOps trên OpenShift (nhánh dev-ocp)

image.png

Bài này mình viết ra, để mọi người có thể thấy được sự khác nhau trong việc triển khai giữa OCP và K8s. Đặc biệt là những thứ liên quan đến quyền cước trên OCP, thật sự là khá đau đầu.

Hướng dẫn end-to-end cho OpenShift Container Platform (OCP) đã có sẵn — nhánh Git dev-ocp.

  • CI: Jenkins + Kaniko → Harbor (in-cluster)
  • CD: ArgoCD upstream (App of Apps)
  • App: Phase 8 + infra Phase 5
  • Secret: Vault + External Secrets Operator (ESO)
  • Expose: OpenShift Route (không Traefik / Nginx)
  • Storage: NFS CSI (nfs-csi)

Tài liệu liên quan (nhánh dev-ocp):

Quan trọng — Thứ tự triển khai (5 giai đoạn)

ArgoCD chỉ deploy banking app ở giai đoạn cuối, sau khi platform + infra + CI/CD đã sẵn sàng.

Giai đoạn Việc cần làm ArgoCD deploy app?
0 NFS CSI driver + StorageClass nfs-csi Không
1 ArgoCD upstream + SCC + Route + kết nối GitHub Không
2 Platform: Harbor, Vault, ESO, Jenkins + Routes Không
2b Observability (tùy chọn): Coroot, OTEL, Linkerd Không
3 Infra: Postgres, Redis, RabbitMQ, Kong + secret Không
4 CI/CD: Jenkins build → Harbor → commit values-images.yaml Không
5 ArgoCD sync banking app → rollout Phase 8 Có

image.png

Giai đoạn 0–3              Giai đoạn 4              Giai đoạn 5
─────────────────          ─────────────            ─────────────
NFS CSI + ArgoCD           git push dev-ocp    →    ArgoCD sync
Harbor, Vault, ESO         Jenkins + Kaniko         banking-app-of-apps
Jenkins + Routes           push Harbor              pods Running
Postgres, Redis            commit values-images     https://npd-banking.co
RabbitMQ, Kong             verify pipeline green

Không apply banking-app-of-apps cho đến khi Giai đoạn 4 hoàn tất (Harbor đã có image).

0. Kiến trúc sau khi hoàn tất

Browser / oc
    │
    ▼
OpenShift Router (HAProxy) — TLS edge
    │
    ├── argocd-server-argocd.apps.ocp01.npd.co   → ArgoCD (ns argocd)
    ├── harbor-platform.apps.ocp01.npd.co         → Harbor (ns platform)
    ├── jenkins-platform.apps.ocp01.npd.co       → Jenkins (ns platform)
    ├── vault-platform.apps.ocp01.npd.co          → Vault (ns vault)
    ├── kong.apps.ocp01.npd.co                   → Kong proxy (ns kong)
    └── npd-banking.co (/ , /api , /ws)          → frontend + Kong (ns banking)

GitHub (dev-ocp) ──webhook──► Jenkins ──Kaniko──► Harbor
         ▲                           │
         └── commit values-images.yaml ◄┘
                    │
                    ▼
              ArgoCD sync ──► ns banking   ← CHỈ sau Giai đoạn 4
Namespace Thành phần URL Giai đoạn
argocd ArgoCD upstream argocd-server-argocd.apps.<domain> 1
platform Jenkins, Harbor jenkins-platform..., harbor-platform... 2
vault Vault vault-banking... 2
external-secrets ESO — 2
observability Coroot, OTEL (tùy chọn — oc expose hoặc Route) 2b
linkerd / linkerd-viz Linkerd mesh (tùy chọn) 2b
postgres Postgres HA 3
redis Redis HA 3
kong Kong HA kong.apps.<domain> 3
rabbit RabbitMQ 3
banking App Phase 8 https://npd-banking.co 5

Chi tiết sơ đồ: OCP-ARCHITECTURE.md

1. Điều kiện tiên quyết

Cluster OpenShift

Khuyến nghị
Quyền cluster-admin (cài NFS CSI, SCC, ArgoCD)
RAM worker ≥ 32 GB cho full stack (Harbor + Coroot + Linkerd)
CLI oc đã login
NFS Server 10.100.1.180 reachable từ worker (subnet 10.100.1.0/24)

Công cụ trên bastion

oc version
git --version
helm version   # cài NFS CSI driver

Login cluster

oc login --token=<TOKEN> --server=https://api.<cluster>:6443
oc get nodes
export CLUSTER_DOMAIN=$(oc get ingresses.config cluster -o jsonpath='{.spec.domain}')
echo "Cluster domain: $CLUSTER_DOMAIN"
# Lab NPD: ocp01.npd.co → route pattern *.apps.ocp01.npd.co

Biến môi trường (dùng xuyên suốt doc)

export REPO_ROOT=/path/to/banking-demo   # chỉnh path thật
export GIT_BRANCH=dev-ocp
export ARGOCD_NS=argocd
export CLUSTER_DOMAIN=ocp01.npd.co       # hoặc lấy từ oc ở trên

2. Clone repo — nhánh dev-ocp

git clone https://github.com/kevinram164/banking-demo.git
cd banking-demo
git checkout dev-ocp
git pull

Repo: https://github.com/kevinram164/banking-demo.git

Cấu hình môi trường: environments/dev-ocp/gitops-env.yaml

Tùy chỉnh cluster domain (nếu khác ocp01.npd.co)

chmod +x phase9-gitops-platform/environments/dev-ocp/scripts/customize-cluster-domain.sh
./phase9-gitops-platform/environments/dev-ocp/scripts/customize-cluster-domain.sh "$CLUSTER_DOMAIN"
git diff phase9-gitops-platform/environments/dev-ocp
# commit + push nếu đổi domain

3. Giai đoạn 0 — NFS CSI Storage

Mọi PVC (Harbor, Jenkins, Postgres, Redis, RabbitMQ, Coroot…) dùng StorageClass nfs-csi.

Hướng dẫn đầy đủ: INSTALL-NFS-CSI.md

# Namespace + PodSecurity privileged
oc apply -f phase9-gitops-platform/environments/dev-ocp/ocp-values/nfs-csi/00-namespace.yaml

# Helm install csi-driver-nfs (xem INSTALL-NFS-CSI.md §3)
helm repo add csi-driver-nfs https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts
helm install csi-driver-nfs csi-driver-nfs/csi-driver-nfs -n csi-driver-nfs --version v4.11.0

# SCC cho CSI pods
chmod +x phase9-gitops-platform/environments/dev-ocp/ocp-values/nfs-csi/scc.sh
./phase9-gitops-platform/environments/dev-ocp/ocp-values/nfs-csi/scc.sh

# StorageClass
oc apply -f phase9-gitops-platform/environments/dev-ocp/ocp-values/nfs-csi/storageclass.yaml

# Test PVC
oc apply -f phase9-gitops-platform/environments/dev-ocp/ocp-values/nfs-csi/test-pvc.yaml
oc get pvc -n nfs-test

Kiểm tra:

oc get sc nfs-csi
oc get pods -n csi-driver-nfs

Checkpoint Giai đoạn 0: nfs-csi là default SC (hoặc chỉ định rõ trong values); PVC test Bound.

4. Giai đoạn 1 — ArgoCD bootstrap

ArgoCD cài ngoài GitOps lần đầu (bootstrap). Chi tiết: INSTALL-ARGOCD-UPSTREAM.md

4.1 Cài ArgoCD upstream

oc create namespace argocd
oc apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.4.4/manifests/install.yaml

oc wait -n argocd --for=condition=Ready pod -l app.kubernetes.io/name=argocd-server --timeout=600s

4.2 SCC — theo namespace (khuyến nghị)

Sau khi pod ArgoCD cài xong, patch UID vào dải namespace + gán nonroot:

chmod +x phase9-gitops-platform/environments/dev-ocp/scripts/namespace-scc-setup.sh
./phase9-gitops-platform/environments/dev-ocp/scripts/namespace-scc-setup.sh argocd

Chi tiết: INSTALL-SCC-HARDENED.md

(Tùy chọn) Tắt Dex nếu không dùng SSO — script trên đã scale 0 mặc định.

4.3 Route + insecure mode

oc patch configmap argocd-cmd-params-cm -n argocd --type merge \
  -p '{"data":{"server.insecure":"true"}}'
oc patch configmap argocd-cm -n argocd --type merge \
  -p '{"data":{"url":"https://argocd-server-argocd.apps.'"$CLUSTER_DOMAIN"'"}}'
oc rollout restart deployment argocd-server -n argocd

oc apply -f phase9-gitops-platform/environments/dev-ocp/ocp-values/routes/argocd-route.yaml

Lấy password admin:

oc get secret argocd-initial-admin-secret -n argocd \
  -o jsonpath='{.data.password}' | base64 -d; echo
# user: admin

Mở https://argocd-server-argocd.apps.ocp01.npd.co (đổi domain nếu cần).

4.4 Kết nối GitHub + AppProject

  1. ArgoCD UI → Settings → Repositories → Connect repo
  2. URL: https://github.com/kevinram164/banking-demo.git
  3. Nếu private: username + PAT (scope repo)
oc apply -f phase9-gitops-platform/environments/dev-ocp/appproject.yaml -n argocd

Checkpoint Giai đoạn 1: ArgoCD UI login OK; repo connected; AppProject banking-platform tồn tại.

5. Giai đoạn 2 — Platform (Harbor, Vault, ESO, Jenkins)

ArgoCD quản lý platform; chưa deploy banking app.

5.1 Apply platform App of Apps

oc apply -f phase9-gitops-platform/environments/dev-ocp/argocd/applications/platform-app-of-apps.yaml -n argocd

ArgoCD UI → platform-app-of-apps-dev-ocp → Sync.

Thứ tự sync wave:

Wave App Ghi chú
0 Harbor, Vault, External Secrets (controller) Chờ pod Running
1 External Secrets config (jenkins-platform-credentials) Seed Vault secret/platform/jenkins trước
2 Jenkins JCasC đọc credential từ K8s secret (Vault)

Theo dõi:

oc get pods -n platform
oc get pods -n vault
oc get pods -n external-secrets

5.2 SCC cho platform

Sau khi Harbor/Jenkins sync xong:

./phase9-gitops-platform/environments/dev-ocp/scripts/namespace-scc-setup.sh platform

5.3 Routes — expose UI platform

Sau khi Harbor, Jenkins, Vault Running:

oc apply -f phase9-gitops-platform/environments/dev-ocp/argocd/applications/platform-routes-app-of-apps.yaml -n argocd
# ArgoCD UI → Sync platform-routes-dev-ocp
Route Host Backend
Harbor harbor-platform.apps.ocp01.npd.co harbor:80 (ns platform)
Jenkins jenkins-platform.apps.ocp01.npd.co jenkins:http
Vault vault-platform.apps.ocp01.npd.co vault:8200 (ns vault)
Coroot coroot-platform.apps.ocp01.npd.co coroot-coroot:8080 (ns observability)
Linkerd Viz linkerd-viz-platform.apps.ocp01.npd.co web:8084 (ns linkerd-viz)

Manifest: environments/dev-ocp/ocp-values/routes/

5.4 Harbor

  1. UI: https://harbor-platform.apps.ocp01.npd.co — đổi admin password
  2. Tạo project banking-demo
  3. Robot accounts: ci-push (Jenkins push), k8s-pull (cluster pull)

Pull secret — seed Vault secret/platform/harbor-pull (robot k8s-pull); ESO tự tạo harbor-pull-creds trong ns banking + platform (xem vault/README.md mục 5.5). Không oc create secret thủ công.

# Trong pod vault-0 (sau khi tạo robot k8s-pull trên Harbor UI)
vault kv put secret/platform/harbor-pull \
  registry='harbor-platform.apps.ocp01.npd.co' \
  username='robot$banking-demo+k8s-pull' \
  password='ROBOT_TOKEN'

Sau sync platform-external-secrets-config:

oc get externalsecret harbor-pull-creds -n banking
oc get secret harbor-pull-creds -n banking

Trust TLS Harbor (bắt buộc trước khi banking pull image): Route edge dùng cert ingress — kubelet báo x509: certificate signed by unknown authority nếu chưa trust CA.

# 1) Seed CA vào Vault (từ router-ca)
export VAULT_TOKEN=root
./phase9-gitops-platform/environments/dev-ocp/scripts/vault-seed-harbor-registry-ca.sh

# 2) Sync ESO (ArgoCD UI → platform-external-secrets-config) — Secret trong openshift-config
oc get secret harbor-registry-ca-vault -n openshift-config

# 3) ConfigMap + image.config (CRI-O không đọc Vault trực tiếp)
./phase9-gitops-platform/environments/dev-ocp/scripts/harbor-registry-trust-setup.sh
oc get mcp

Chi tiết Vault path: vault/README.md §5.6b. Lab nhanh không Vault: INSECURE=1 ./…/harbor-registry-trust-setup.sh

OCP vs k3d: Không cần registry mirror registries.yaml. Kaniko dùng kanikoSkipTlsVerify: true trong Jenkinsfile; kubelet pull qua Route TLS của OpenShift Router.

5.5 Vault + External Secrets

Chi tiết CLI: vault/README.md. Thứ tự bắt buộc — đảo bước sẽ lỗi InvalidProviderConfig / SecretSyncedError.

Bước 1 — Seed KV trong pod Vault

oc get pods -n vault   # vault-0 Running
oc exec -it vault-0 -n vault -- sh

Trong pod:

export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='root'

vault kv put secret/banking/db \
  DATABASE_URL='postgresql://banking:bankingpass@postgres-ha-postgresql-primary.postgres.svc.cluster.local:5432/banking' \
  REDIS_URL='redis://redis-ha-redis-master.redis.svc.cluster.local:6379/0'

vault kv put secret/banking/rabbitmq \
  RABBITMQ_URL='amqp://banking:bankingpass@rabbitmq.rabbit.svc.cluster.local:5672/'

vault kv put secret/rabbitmq/admin \
  username='banking' \
  password='bankingpass'

vault kv put secret/platform/harbor-pull \
  registry='harbor-platform.apps.ocp01.npd.co' \
  username='robot$banking-demo+k8s-pull' \
  password='HARBOR_K8S_PULL_TOKEN'

UI lab: https://vault-platform.apps.ocp01.npd.co, token root.

Bước 2 — Sync ESO controller

ArgoCD: platform-external-secrets → Synced / Healthy.

Bước 3 — Tạo vault-token trước ClusterSecretStore

oc create secret generic vault-token \
  --from-literal=token=root \
  -n external-secrets

Bước 4 — Namespace + sync ExternalSecret

oc create ns banking --dry-run=client -o yaml | oc apply -f -
oc create ns rabbit --dry-run=client -o yaml | oc apply -f -

ArgoCD: platform-external-secrets-config

Bước 5 — Kiểm tra

oc get clustersecretstore vault-backend
oc get externalsecret -A
oc get secret banking-db-secret -n banking
oc get secret harbor-pull-creds -n banking

Force reconcile nếu cần:

oc annotate clustersecretstore vault-backend force-sync=$(date +%s) --overwrite
oc annotate externalsecret banking-db-secret -n banking force-sync=$(date +%s) --overwrite

5.6 Jenkins — cấu hình CI

Mục tiêu: push nhánh dev-ocp → Jenkins Kaniko → Harbor → commit values-images.yaml.

Bước 1 — Đợi Jenkins Running

oc get pods -n platform -l app.kubernetes.io/component=jenkins-controller

Bước 2 — Đăng nhập
URL https://jenkins-platform.apps.ocp01.npd.co
User admin
Password ChangeMe-Jenkins (hoặc lấy từ secret) 
oc get secret jenkins -n platform \
  -o jsonpath='{.data.jenkins-admin-password}' | base64 -d; echo

Bước 3 — Shared Library (tự động JCasC)

Tham số Giá trị
Library banking-demo
Branch dev-ocp
Path phase9-gitops-platform/jenkins-shared-library

Bước 4 — Secret trong Vault (admin + CI)

Seed Vault (trong pod vault-0):

# Robot Harbor — pipeline đọc trực tiếp (không Jenkins credential)
vault kv put secret/platform/harbor \
  username='robot$banking-demo+ci-push' \
  password='HARBOR_ROBOT_TOKEN'

# GitHub PAT — push GitOps
vault kv put secret/platform/github \
  username='kevinram164' \
  pat='github_pat_xxxx'

# Chỉ admin Jenkins (+ webhook tùy chọn)
vault kv put secret/platform/jenkins \
  admin_username='admin' \
  admin_password='YOUR_JENKINS_ADMIN_PASSWORD'

Bật Vault Kubernetes auth cho agent jenkins-kaniko (một lần, trên bastion):

export VAULT_TOKEN=root
chmod +x phase9-gitops-platform/environments/dev-ocp/scripts/vault-setup-jenkins-k8s-auth.sh
./phase9-gitops-platform/environments/dev-ocp/scripts/vault-setup-jenkins-k8s-auth.sh

Kiểm tra ESO (chỉ admin):

oc get externalsecret jenkins-platform-credentials -n platform
oc get secret jenkins-platform-credentials -n platform

Sync Jenkins → restart pod nếu cần:

oc delete pod jenkins-0 -n platform

Bước 5 — ServiceAccount Kaniko + SCC (UID 0)

Kaniko trên OpenShift cần UID 0 để chown layer (operation not permitted nếu chỉ dùng restricted SCC).

chmod +x phase9-gitops-platform/environments/dev-ocp/scripts/jenkins-kaniko-scc-setup.sh
./phase9-gitops-platform/environments/dev-ocp/scripts/jenkins-kaniko-scc-setup.sh

Script tạo SA jenkins-kaniko + SCC jenkins-kaniko-root (chỉ SA này, không anyuid cả namespace).

Bước 6 — Multibranch Pipeline

  1. New Item → banking-demo → Multibranch Pipeline
  2. Git: https://github.com/kevinram164/banking-demo.git
  3. Filter by name: Include dev-ocp
  4. Script Path: Jenkinsfile (root — đã cấu hình Harbor OCP)

Bước 7 — Webhook (tùy chọn)

GitHub → Webhooks → https://jenkins-platform.apps.ocp01.npd.co/github-webhook/

Checkpoint Giai đoạn 2: Harbor UI + project banking-demo; Vault seeded (harbor/github/jenkins); Jenkins login OK; Vault K8s auth cho jenkins-kaniko.

6. Giai đoạn 2b — Observability (tùy chọn)

Coroot + OpenTelemetry + Linkerd. Chi tiết: observability/README.md

oc apply -f phase9-gitops-platform/environments/dev-ocp/argocd/applications/observability-app-of-apps.yaml -n argocd

ArgoCD UI → observability-app-of-apps-dev-ocp → Sync.

AppProject: Chart linkerd-viz tạo ClusterRoleBinding trong kube-system (tap-auth-reader). AppProject banking-platform phải cho phép destination kube-system — xem environments/dev-ocp/appproject.yaml. Nếu thiếu → namespace kube-system is not permitted in project 'banking-platform'.

oc apply -f phase9-gitops-platform/environments/dev-ocp/appproject.yaml -n argocd
Wave App
0 coroot-operator, linkerd-crds, linkerd-identity-bootstrap
1 otel-collector, linkerd-control-plane
2 coroot-ce, linkerd-viz

Linkerd trên OCP cần SCC privileged (UID 2102/2103/65534 + NET_ADMIN/NET_RAW). Ns banking cũng cần (sidecar: linkerd-init UID 0 + proxy 2102 — linkerd.io/inject: enabled):

chmod +x phase9-gitops-platform/environments/dev-ocp/scripts/linkerd-scc-setup.sh
./phase9-gitops-platform/environments/dev-ocp/scripts/linkerd-scc-setup.sh
# tương đương:
# oc adm policy add-scc-to-group privileged system:serviceaccounts:linkerd
# oc adm policy add-scc-to-group privileged system:serviceaccounts:linkerd-viz
# oc adm policy add-scc-to-group privileged system:serviceaccounts:banking

Nếu RS banking / linkerd-destination-* báo Forbidden / runAsUser 0|2102 / NET_ADMIN → chạy script trên. Không chạy namespace-scc-setup.sh banking sau đó (gỡ privileged).

linkerd-init Init:CrashLoopBackOff (pods đã tạo được, nhưng init exit 1) — thường do iptables-legacy trên RHCOS. Values phải có:

proxyInit:
  iptablesMode: nft
  runAsRoot: true

# Chẩn đoán
oc logs -n linkerd deploy/linkerd-proxy-injector -c linkerd-init --tail=40

# Sync sau khi push values (hoặc hotfix Helm param)
argocd app sync observability-linkerd-control-plane
# hotfix tạm (chưa push Git):
# argocd app set observability-linkerd-control-plane \
#   -p proxyInit.iptablesMode=nft -p proxyInit.runAsRoot=true
# argocd app sync observability-linkerd-control-plane --force

oc get pods -n linkerd

Nếu log còn Extension multiport … missing kernel module → load xt_* trên node.

xt_* là gì? Kernel modules xtables (iptables extensions). linkerd-init dùng iptables-nft nhưng vẫn cần match/target kiểu cũ trong network namespace của pod:

Module Việc Linkerd dùng
xt_multiport Bỏ qua nhiều port (4190,4191,…) một rule
xt_owner Không redirect traffic của UID proxy (2102)
xt_comment Comment trên rule (debug)
xt_REDIRECT Redirect TCP vào port proxy (4140/4143)

Trên RHCOS các module đôi khi không tự load → rule fail dù đã iptablesMode=nft. modprobe (lab) hoặc MachineConfig (persist sau reboot).

# Lab nhanh (không reboot) — modprobe trên mọi node rồi restart pods
chmod +x phase9-gitops-platform/environments/dev-ocp/scripts/linkerd-load-xt-modules.sh
./phase9-gitops-platform/environments/dev-ocp/scripts/linkerd-load-xt-modules.sh

# Persist (MCP rolling reboot worker + master)
oc apply -f phase9-gitops-platform/environments/dev-ocp/ocp-values/machineconfig/linkerd-xt-modules.yaml
oc get mcp -w

Hoặc chuyển sang Linkerd CNI (cniEnabled: true) — vẫn có thể cần cùng xt_* modules trên host.

Coroot Prometheus (embedded) chạy UID 65534 — không khớp dải UID namespace observability:

chmod +x phase9-gitops-platform/environments/dev-ocp/scripts/coroot-scc-setup.sh
./phase9-gitops-platform/environments/dev-ocp/scripts/coroot-scc-setup.sh

Nếu ReplicaSet prometheus-* báo runAsUser: Invalid value: 65534 → chạy script trên.

Coroot node-agent (DaemonSet — metrics từng node). UI báo no agent installed khi:

  1. Values còn nodeSelector: coroot-node-agent: enabled mà worker chưa label — đã bỏ trong observability/values-coroot-ce-k3d.yaml.
  2. Thiếu SCC privileged cho SA node-agent trên OCP.
# Sau khi sync observability-coroot-ce (push dev-ocp + ArgoCD Sync)
chmod +x phase9-gitops-platform/environments/dev-ocp/scripts/coroot-node-agent-scc-setup.sh
./phase9-gitops-platform/environments/dev-ocp/scripts/coroot-node-agent-scc-setup.sh

oc get ds -n observability | grep node-agent
oc get pods -n observability -l app.kubernetes.io/name=coroot-node-agent -o wide

Đợi 1–2 phút rồi refresh Coroot → Nodes (CPU/Memory/Network sẽ có data). eBPF tracer/profiler vẫn tắt trên lab — đủ metrics cơ bản.

Routes UI (GitOps — cùng app platform-routes-dev-ocp, sync sau observability pods Running):

UI URL
Coroot https://coroot-platform.apps.ocp01.npd.co
Linkerd Viz https://linkerd-viz-platform.apps.ocp01.npd.co 
oc apply -f phase9-gitops-platform/environments/dev-ocp/argocd/applications/platform-routes-app-of-apps.yaml -n argocd
# ArgoCD → Sync platform-routes-dev-ocp
oc get route -n observability coroot-platform
oc get route -n linkerd-viz linkerd-viz-platform

Checkpoint Giai đoạn 2b: oc get pods -n observability; linkerd check pass (nếu dùng mesh).

coroot-cluster-agent OOMKilled: Values mặc định 256Mi quá thấp trên OCP — đã tăng lên 1Gi trong observability/values-coroot-ce-k3d.yaml. Sync observability-coroot-ce hoặc oc rollout restart deploy/coroot-cluster-agent -n observability.

coroot-node-agent OOMKilled: Limit 512Mi không đủ (eBPF trên OCP) — values đã tăng 512Mi request / 2Gi limit. Patch nhanh trên cluster:

oc set resources ds/coroot-node-agent -n observability \
  --requests=cpu=100m,memory=512Mi --limits=memory=2Gi
oc rollout restart ds/coroot-node-agent -n observability

Coroot Nodes — no agent installed: Sync values mới (bỏ nodeSelector) + coroot-node-agent-scc-setup.sh. Nếu DaemonSet 0/6 READY — oc describe ds -n observability xem FailedCreate / SCC.

7. Giai đoạn 3 — Infra (Postgres, Redis, RabbitMQ, Kong)

StorageClass

Trên dev-ocp, mọi StatefulSet dùng nfs-csi (đã cấu hình trong ArgoCD Application manifests).

Thành phần StorageClass
Harbor, Jenkins nfs-csi
Postgres, Redis nfs-csi
RabbitMQ nfs-csi

Nếu PVC cũ sai SC — xóa STS + PVC rồi sync lại (chỉ khi chưa có data quan trọng):

oc delete sts -n redis redis-ha-node --cascade=orphan
oc delete pvc -n redis --all

7.1 Apply infra App of Apps

oc apply -f phase9-gitops-platform/environments/dev-ocp/argocd/applications/infra-app-of-apps.yaml -n argocd

ArgoCD UI → infra-app-of-apps-dev-ocp → Sync.

Thứ tự sync wave:

Wave App Ghi chú
0 Postgres, Redis PVC Bound + pod Running
1 RabbitMQ Sau postgres/redis
2 Kong PreSync hook chờ PG Ready + tạo DB kong

Kong image cố định UID 1000 — restricted-v2 sẽ Forbidden. Chạy trước/sau sync infra-kong:

chmod +x phase9-gitops-platform/environments/dev-ocp/scripts/kong-scc-setup.sh
./phase9-gitops-platform/environments/dev-ocp/scripts/kong-scc-setup.sh

Theo dõi:

oc get pods -n postgres
oc get pods -n redis
oc get pods -n rabbit
oc get pods -n kong

Bitnami OCI 401: Chart phải dùng https://charts.bitnami.com/bitnami, không oci://registry-1.docker.io/bitnamicharts.

ImagePullBackOff: Values dùng bitnamilegacy/* + global.security.allowInsecureImages: true.

Checkpoint Giai đoạn 3: Tất cả pod infra Running; ESO secret banking / rabbit đã sync.

8. Giai đoạn 4 — CI/CD (Jenkins → Harbor → Git)

Mục tiêu: pipeline build image và commit tag vào Git trước khi ArgoCD deploy app.

8.1 Chạy pipeline lần đầu

# Sửa code Phase 8 hoặc trigger build trên Jenkins UI
git add phase8-application-v3/
git commit -m "feat: first CI build on dev-ocp"
git push origin dev-ocp

Jenkins pipeline kỳ vọng:

  1. Kaniko build từ Dockerfile Phase 8
  2. Push harbor-platform.apps.ocp01.npd.co/banking-demo/<service>:<short-sha>
  3. Commit + push phase9-gitops-platform/gitops/values-images.yaml

Verify:

# Harbor UI — project banking-demo có image
git pull origin dev-ocp
git log -1 --oneline -- phase9-gitops-platform/gitops/values-images.yaml

8.2 Checklist CI/CD

  • [ ] Vault secret/platform/harbor + secret/platform/github seeded
  • [ ] Vault secret/platform/harbor-pull + ESO harbor-pull-creds (ns banking)
  • [ ] Vault K8s auth + role jenkins-kaniko (vault-setup-jenkins-k8s-auth.sh)
  • [ ] Vault secret/platform/jenkins (admin) + ESO sync
  • [ ] SA jenkins-kaniko tồn tại
  • [ ] Multibranch filter dev-ocp
  • [ ] Pipeline green end-to-end

Checkpoint Giai đoạn 4: Harbor có image; values-images.yaml committed; Jenkins green.

9. Giai đoạn 5 — ArgoCD deploy banking app

Chỉ thực hiện sau Giai đoạn 4.

9.1 Apply banking App of Apps

oc apply -f phase9-gitops-platform/environments/dev-ocp/argocd/applications/banking-app-of-apps.yaml -n argocd

ArgoCD UI → banking-app-of-apps-dev-ocp → Sync (lần đầu sync thủ công).

Apps:

  • banking-namespace
  • banking-api-producer, auth, account, transfer, notification
  • banking-frontend, banking-ingress (ingress Helm tắt — Route thay thế)
oc get applications -n argocd | grep banking
oc get pods -n banking

ImagePullBackOff → kiểm tra:

  1. Vault platform/harbor-pull + ESO harbor-pull-creds
  2. Tag trong values-images.yaml, CI đã push
  3. x509: unknown authority → harbor-registry-trust-setup.sh (trust router CA / insecureRegistries)

9.2 Banking Routes

Routes banking đã nằm trong platform-routes-dev-ocp (sync ở Giai đoạn 2):

Route Path Backend
npd-banking.co / frontend
npd-banking.co /api kong-proxy-ext → Kong
npd-banking.co /ws kong-proxy-ext → Kong

DNS: npd-banking.co trỏ A/CNAME tới OpenShift Router IP, hoặc thêm vào hosts file máy client.

oc get route -n banking
oc get route -n kong kong-proxy

9.3 Kong Phase 8 routes

oc apply -f phase8-application-v3/kong-ha/kong-import-job.yaml
oc wait -n kong --for=condition=complete job/kong-config-import-phase8 --timeout=300s
oc logs -n kong job/kong-config-import-phase8 --tail=20
oc rollout restart deployment -n kong -l app.kubernetes.io/name=kong

CORS origins trong job đã trỏ https://npd-banking.co.

Verify:

curl -sk https://npd-banking.co/ | head
curl -sk https://npd-banking.co/api/health

9.4 (Tùy chọn) Root App of Apps

Sau khi mọi thứ ổn:

./phase9-gitops-platform/environments/dev-ocp/apply-argocd.sh
# hoặc:
oc apply -f phase9-gitops-platform/environments/dev-ocp/argocd/app-of-apps.yaml -n argocd

Root adopt các nhóm app-of-apps đã apply trước.

Checkpoint Giai đoạn 5: Banking pods Running; https://npd-banking.co mở được.

10. Luồng phát triển hàng ngày

git push dev-ocp  →  Jenkins build  →  Harbor  →  commit values-images.yaml
                                                          ↓
                                              ArgoCD auto-sync banking apps
                                                          ↓
                                              rollout pods ns banking

git push origin dev-ocp
oc get applications -n argocd -w
oc get pods -n banking

11. Checklist hoàn tất

Giai đoạn 0

  • [ ] oc get sc nfs-csi — default hoặc chỉ định trong values
  • [ ] PVC test Bound

Giai đoạn 1

Giai đoạn 2 — Platform

  • [ ] Harbor UI + project banking-demo + robot accounts
  • [ ] Routes Harbor / Jenkins / Vault truy cập được
  • [ ] Vault seeded + ESO sync
  • [ ] Jenkins + Multibranch job dev-ocp

Giai đoạn 3 — Infra

  • [ ] Postgres, Redis, RabbitMQ, Kong pods Running
  • [ ] PVC dùng nfs-csi

Giai đoạn 4 — CI/CD

  • [ ] Push code → Jenkins green
  • [ ] Image trên Harbor
  • [ ] values-images.yaml committed

Giai đoạn 5 — App

  • [ ] Banking pods Running trong banking
  • [ ] Kong routes imported
  • [ ] https://npd-banking.co — Banking Demo UI

12. Xử lý lỗi nhanh (OCP)

Runbook đầy đủ: environments/dev-ocp/INSTALL-TROUBLESHOOTING.md (Vault image UBI, Harbor SCC/NFS, CSR kubelet, Jenkins ESO).

Các lỗi có thể gặp Cách sửa
Pod Forbidden SCC namespace-scc-setup.sh <ns> — xem INSTALL-SCC-HARDENED.md
Kong runAsUser 1000 Forbidden kong-scc-setup.sh (SCC kong-uid1000)
Linkerd 2102 / NET_ADMIN Forbidden linkerd-scc-setup.sh (privileged ns linkerd + linkerd-viz + banking)
Banking RS: init runAsUser: 0 / NET_ADMIN Forbidden Sidecar Linkerd — ./linkerd-scc-setup.sh banking rồi oc delete pods -n banking --all
Linkerd Viz metrics-api UID 2103 Forbidden Cùng script — hoặc oc adm policy add-scc-to-group privileged system:serviceaccounts:linkerd-viz
linkerd-init CrashLoopBackOff proxyInit.iptablesMode=nft + runAsRoot=true — sync control-plane
linkerd-init missing xt_multiport linkerd-load-xt-modules.sh (+ MachineConfig persist)
Prometheus runAsUser 65534 invalid coroot-scc-setup.sh (Coroot embedded Prometheus)
Coroot Nodes no agent installed Sync values (bỏ nodeSelector) + coroot-node-agent-scc-setup.sh
coroot-node-agent OOMKilled oc set resources ds/coroot-node-agent -n observability --limits=memory=2Gi --requests=memory=512Mi
Harbor Permission denied entrypoint harbor-scc-setup.sh — không patch harbor-* bằng namespace-scc-setup
Harbor DB initdb Permission denied chmod 777 subdir PVC trên NFS server — xem INSTALL-TROUBLESHOOTING.md §3.3
oc logs tls internal error Approve CSR Pending: oc get csr -o name \| xargs oc adm certificate approve
Vault ImagePullBackOff Red Hat Image *-ubi; tắt injector — xem INSTALL-TROUBLESHOOTING.md §2
ArgoCD Route 502 server.insecure=true + Route TLS edge + targetPort: http — xem INSTALL-ARGOCD-UPSTREAM.md
PVC Pending Kiểm tra NFS CSI pods; worker mount được NFS server
jenkins-platform-credentials not found Seed Vault + vault-token + sync ESO config — xem INSTALL-TROUBLESHOOTING.md §6
vault-token not found Tạo secret trước ClusterSecretStore
ImagePullBackOff Vault platform/harbor-pull seeded? ESO harbor-pull-creds synced? CI đã push image?
ImagePull x509: unknown authority Harbor harbor-registry-trust-setup.sh — trust router CA (hoặc INSECURE=1)
Banking thiếu uvicorn / CreateContainerError Kaniko miss venv — multi-stage + venv --copies + CMD /app/venv/bin/uvicorn; rebuild BUILD_TARGET=all
Kaniko push OK nhưng Jenkins exit -1 / JENKINS-48300 Heartbeat NFS — sync platform-jenkins (javaOpts HEARTBEAT) + shared lib; image đã có trên Harbor
Jenkins pod restart + startup 503 Thường do Argo sync STS (đổi javaOpts/values). Log Failed Loading plugin = lệch version Pipeline — pin pipeline-model-* cùng bản trong jenkins.yaml
Init CrashLoop / stage-view 404 Plugin id sai — không tồn tại stage-view; dùng pipeline-stage-view hoặc bỏ hẳn. Kiểm tra oc get cm jenkins -n platform -o jsonpath='{.data.plugins\.txt}'
uvicorn not found / thiếu /app/venv/bin/uvicorn Kaniko miss symlink venv — multi-stage + venv --copies; rebuild BUILD_TARGET=all
Kaniko symlink lib .../venv/lib64: file exists State multi-stage còn lại + lib64→lib — rm -f venv/lib64 trong Dockerfile + rm -rf /kaniko/0 giữa builds
Name or service not known (Postgres) Sai host trong Vault secret/banking/db — OCP: postgres-ha-postgresql-primary.postgres.svc.cluster.local
Name or service not known (Redis) Sai host REDIS_URL — oc get svc -n redis; thường redis-ha hoặc redis-ha-master (không phải redis-ha-redis-master); auth: redis://:Mbfs%402025@...
secret "banking-db-secret" not found ESO chưa tạo Secret — seed Vault secret/banking/db + oc describe externalsecret banking-db-secret -n banking
Frontend nginx [emerg] host not found in upstream "kong" nginx.conf FQDN kong-kong-proxy.kong.svc.cluster.local (không short name kong)
Frontend nginx Permission denied /var/cache/nginx hoặc nginx.pid No such file OCP: /var/run tmpfs rỗng — pid/temp dùng /tmp; listen 8080; Helm targetPort: 8080
Route npd-banking.co 503, pod Ready nhưng Endpoints: <none> Route dùng targetPort: http (tên Service port), không dùng số 80 khi container listen 8080
Banking sync quá sớm Quay lại Giai đoạn 4
ArgoCD OutOfSync Sync từng app; kiểm tra branch dev-ocp
Kaniko push 401 Robot Harbor sai user/token
Route Missing / not found Sync platform-routes-dev-ocp — xem INSTALL-TROUBLESHOOTING.md §9

13. Lệnh tham chiếu nhanh

# Trạng thái tổng
oc get pods -A | grep -v Running
oc get applications -n argocd

# Apply theo giai đoạn (dev-ocp)
oc apply -f phase9-gitops-platform/environments/dev-ocp/appproject.yaml -n argocd
oc apply -f phase9-gitops-platform/environments/dev-ocp/argocd/applications/platform-app-of-apps.yaml -n argocd
oc apply -f phase9-gitops-platform/environments/dev-ocp/argocd/applications/platform-routes-app-of-apps.yaml -n argocd
oc apply -f phase9-gitops-platform/environments/dev-ocp/argocd/applications/observability-app-of-apps.yaml -n argocd
oc apply -f phase9-gitops-platform/environments/dev-ocp/argocd/applications/infra-app-of-apps.yaml -n argocd
# ... CI/CD pipeline green ...
oc apply -f phase9-gitops-platform/environments/dev-ocp/argocd/applications/banking-app-of-apps.yaml -n argocd

# Logs app
oc logs -n banking -l app=auth-service --tail=50

14. Test thôi nào 😄

image.png Vẫn đẹp như ngày nào. Tracing trên Coroot : image.png image.png Vào check giao dịch nào : image.png image.png image.png

Đến đây, thì mình tạm gác lại project này, vì từ giờ đến cuối năm, mình tập trung vào 2 project thú vị hơn.

Xin chào, và hẹn gặp lại vào năm 2027 😉

gitops Kubernetes openshift DevOps

All Rights Reserved

Table of contents


Viblo
Let's register a Viblo Account to get more interesting posts.
Register