Using ArgoCD to Manage AI Model Deployments with GitOps

Models are not container images, and that one difference reshapes the whole pipeline. Here is the GitOps setup that accounts for it.

Every engineering team that runs both microservices and models tends to have the same split personality. The services deploy through pull requests, automated reconciliation, and audit trails, while the models deploy because somebody ran a notebook cell or dropped a message in Slack asking for a restart. That gap is not a tooling shortage anymore. The CNCF moved KServe to Incubating status back in September 2025, ArgoCD has been production standard for years, and the pattern that joins them is well documented.

What makes model deployment genuinely different from application deployment is worth stating before any YAML. The deployable artifact is not a container image, it is a multi gigabyte blob of weights sitting in object storage, and that single fact changes what belongs in Git, how long a rollout takes, what a readiness probe should check, and what a rollback actually costs you. This tutorial builds the pipeline with those differences designed in rather than discovered later.

What GitOps for AI model deployments means

GitOps for AI model deployments is a delivery pattern where the desired state of every model serving workload, including which model version is live in which environment, exists as declarative manifests in Git, and a reconciler such as ArgoCD continuously applies that state to the cluster, so every model change arrives through a reviewed commit and every rollback is a revert.

The critical design choice sits inside that definition. Git holds a reference to the model, never the model itself, which means a versioned URI plus a content digest plus the serving configuration. Weights live in object storage or an OCI registry where they belong.

What actually lives in Git

This table is the whole mental model, and getting it wrong is how teams end up with a repository nobody can clone.

That last row deserves emphasis because it is the one people get wrong under deadline pressure. A plain Kubernetes Secret committed to Git is base64, which is encoding rather than encryption, so it is a readable credential in your history forever.

The repository layout

Keep environments as overlays and the model reference as the only thing that changes between them.

ml-platform/
├── base/
│   ├── inferenceservice.yaml        # shape of the service, no version pinned
│   └── kustomization.yaml
├── overlays/
│   ├── staging/
│   │   ├── kustomization.yaml
│   │   └── model-version.yaml       # patch: which model version is live here
│   └── production/
│       ├── kustomization.yaml
│       └── model-version.yaml       # the file automation edits, humans review
└── applications/
    ├── staging.yaml                 # ArgoCD Application
    └── production.yaml

The point of isolating model-version.yaml is that it becomes the one file your promotion automation writes to, and therefore the one file a reviewer has to read carefully. A pull request that changes a single digest is a pull request people actually review, unlike a two hundred line manifest diff.

The InferenceService, pinned and annotated

Here is the serving manifest. Every field that looks like boilerplate is doing something specific for models.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: recommender
  namespace: ml-production
  annotations:
    # Lineage. This is what makes an audit answerable a year from now.
    model.registry/version: "2.7.1"
    model.registry/training-data-sha256: "9f2c41a8b7e3d6f0c5a94b2e8d1f7c30"
    model.registry/training-code-sha: "a41f9c2"
    model.registry/eval-score: "0.9142"
    argocd.argoproj.io/sync-wave: "2"        # after warm up, before traffic shift
spec:
  predictor:
    minReplicas: 2
    maxReplicas: 8
    scaleTarget: 60
    model:
      modelFormat:
        name: sklearn
      # Pinned by digest, never by a floating tag such as :latest
      storageUri: "s3://ml-models/recommender/2.7.1"
      resources:
        requests:
          cpu: "2"
          memory: 8Gi
        limits:
          cpu: "4"
          memory: 12Gi
      readinessProbe:
        httpGet:
          path: /v1/models/recommender    # proves the MODEL loaded
          port: 8080
        initialDelaySeconds: 45           # weights take time to load
        periodSeconds: 10
        failureThreshold: 12
    # Canary: 10 percent of traffic to the new revision
    canaryTrafficPercent: 10

Three of those fields carry most of the safety, and each one exists because of a specific failure.

Pinning storageUri to an explicit version rather than a moving pointer means the manifest tells you exactly what is running. A floating tag reintroduces the snowflake problem you adopted GitOps to remove, because the cluster and the repository can silently disagree.

The readiness probe targets the model endpoint rather than a generic health port. A model server frequently accepts TCP connections and returns 200 on /healthz while the weights are still loading, so a naive probe marks the pod ready and Kubernetes routes real traffic to a model that cannot serve it. Probing the model path is the difference between a clean rollout and a burst of 503s.

Generous resource requests and a patient initialDelaySeconds reflect that model containers are heavy. Underprovisioned memory shows up as OOM kills during load rather than a clear error, which is a genuinely confusing failure to debug at speed.

Build the ArgoCD foundation this assumes

Everything above rests on knowing how ArgoCD reconciles, syncs, and reports health. The GitOps with ArgoCD course on KodeKloud covers GitOps principles, application health, sync strategies, declarative setup, and notifications across about six hours of hands on work, which maps directly onto the pipeline in this tutorial.

The ArgoCD Application, and the gotcha that blocks most first attempts

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: recommender-production
  namespace: argocd
spec:
  project: ml-platform
  source:
    repoURL: https://github.com/acme/ml-platform
    targetRevision: main
    path: overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: ml-production
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=false
  # THE GOTCHA: KServe writes to its own status and spec fields at runtime.
  # Without this, selfHeal sees drift on every reconcile and fights the
  # controller forever, so the Application never settles on Synced.
  ignoreDifferences:
    - group: serving.kserve.io
      kind: InferenceService
      jsonPointers:
        - /status
        - /spec/predictor/model/runtime

That ignoreDifferences block is the single most common reason a model Application sits in a permanent OutOfSync or Progressing state. KServe's controller mutates fields that Git does not specify, ArgoCD's self heal reads that as drift and reverts it, the controller writes again, and the two loop indefinitely. Left unfixed it also masks real drift, because everything looks broken all the time.

Ordering the rollout with sync waves

Model rollouts have a dependency order that application rollouts usually do not, and the three ordering mechanisms in the Argo ecosystem solve different scopes.

For a model, the useful wave order puts a warm up job ahead of anything that receives traffic.

apiVersion: batch/v1
kind: Job
metadata:
  name: recommender-warmup
  annotations:
    argocd.argoproj.io/sync-wave: "1"        # runs BEFORE the InferenceService
    argocd.argoproj.io/hook: PreSync
spec:
  backoffLimit: 2
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: warmup
          image: ghcr.io/acme/model-warmup:1.4.0
          args:
            - "--storage-uri=s3://ml-models/recommender/2.7.1"
            - "--verify-digest=sha256:9f2c41a8b7e3d6f0c5a94b2e8d1f7c30"
            - "--run-smoke-inference"        # one real prediction, not a ping

Verifying the digest before serving matters more for models than for images, because object storage has no immutability guarantee unless you configure one. Someone can overwrite 2.7.1 in a bucket, and without a digest check your pipeline will deploy different weights under the same version number and report success.

Promotion gates need evaluation scores, not just health checks

Here is where model delivery diverges most sharply from application delivery. An application either works or throws errors. A model can be perfectly healthy, return well formed responses within its latency budget, and still be worse than the version it replaced.

So the promotion gate has to read model quality, not just cluster health.

#!/usr/bin/env python3
"""Promotion gate. Runs in CI, writes to Git only if the model earns it."""
import json
import subprocess
import sys

MIN_EVAL_SCORE = 0.90        # absolute floor
MAX_REGRESSION = 0.005       # tolerated drop against the live model
MAX_P99_MS = 250

def gate(candidate: dict, live: dict) -> list[str]:
    """Return a list of reasons to block. Empty list means promote."""
    blocks = []
    if candidate["eval_score"] < MIN_EVAL_SCORE:
        blocks.append(f"eval {candidate['eval_score']:.4f} below floor {MIN_EVAL_SCORE}")
    regression = live["eval_score"] - candidate["eval_score"]
    if regression > MAX_REGRESSION:
        blocks.append(f"regression of {regression:.4f} against live model")
    if candidate["p99_latency_ms"] > MAX_P99_MS:
        blocks.append(f"p99 {candidate['p99_latency_ms']}ms over budget {MAX_P99_MS}ms")
    if candidate["training_data_sha256"] == live["training_data_sha256"]:
        blocks.append("training data unchanged, so this is not a new model")
    return blocks

def promote(version: str, digest: str) -> None:
    """Write the pinned reference and open a pull request. Never apply directly."""
    subprocess.run(["yq", "-i",
                    f'.spec.predictor.model.storageUri = "s3://ml-models/recommender/{version}"',
                    "overlays/production/model-version.yaml"], check=True)
    subprocess.run(["git", "checkout", "-b", f"promote-recommender-{version}"], check=True)
    subprocess.run(["git", "commit", "-am",
                    f"promote recommender {version} (digest {digest[:12]})"], check=True)
    subprocess.run(["gh", "pr", "create", "--fill"], check=True)

if __name__ == "__main__":
    candidate = json.load(open("candidate-metrics.json"))
    live = json.load(open("live-metrics.json"))
    reasons = gate(candidate, live)
    if reasons:
        print("PROMOTION BLOCKED:")
        for r in reasons:
            print(f"  - {r}")
        sys.exit(1)
    promote(candidate["version"], candidate["digest"])
    print(f"opened promotion PR for {candidate['version']}")

Notice that the script opens a pull request rather than applying anything. Automation writes to Git, ArgoCD writes to the cluster, and that separation is what keeps GitOps honest. The moment your CI system also has cluster credentials, you have two sources of truth and no reliable audit trail.

The training data hash check is a small guard with outsized value. A retraining job that silently reused yesterday's dataset produces a new version number with identical inputs, and catching that in the gate saves a confusing week of wondering why the metrics never moved.

Rollback is fast in Git and slow in the cluster

GitOps gives models a genuinely excellent rollback story, with one caveat worth planning around.

# Roll back the model version. ArgoCD reconciles within its sync interval.
git revert 4f2a1c9 && git push        # the promotion commit

# Or force reconciliation immediately
argocd app sync recommender-production

# Confirm which model version the cluster now believes is live
kubectl get inferenceservice recommender -n ml-production \
  -o jsonpath='{.spec.predictor.model.storageUri}'

The revert takes seconds and the reconcile takes seconds. Pulling several gigabytes of weights onto pods that no longer have them cached does not. If your previous model's pods have already been scaled to zero and their node caches evicted, a rollback can take minutes rather than seconds, which is exactly the wrong discovery to make during an incident.

Three mitigations handle it. Keep the previous revision warm rather than immediately scaling it to zero, which is precisely what a canary with a retained stable revision gives you. Cache model weights on nodes so a rollback rehydrates from local disk instead of object storage. And practice a rollback on purpose in staging, because a documented rollback procedure that nobody has executed is a hypothesis rather than a plan.

Practice model serving on Kubernetes

The InferenceService shape, canary traffic, and troubleshooting a model that will not become ready are all much clearer once you have deployed one yourself. The KServe Fundamentals course on KodeKloud covers deploying predictive and generative models, managing InferenceServices, and troubleshooting failures, and the KodeKloud playgrounds give you clusters where a broken rollout costs nothing.

Fanning out across environments

Once one environment works, ApplicationSet removes the copy and paste.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: recommender
  namespace: argocd
spec:
  # Progressive sync: staging must go Healthy before production begins.
  strategy:
    type: RollingSync
    rollingSync:
      steps:
        - matchExpressions:
            - key: env
              operator: In
              values: ["staging"]
        - matchExpressions:
            - key: env
              operator: In
              values: ["production"]
  generators:
    - list:
        elements:
          - env: staging
            namespace: ml-staging
            autoSync: "true"
          - env: production
            namespace: ml-production
            autoSync: "false"        # production waits for a human
  template:
    metadata:
      name: 'recommender-{{env}}'
      labels:
        env: '{{env}}'
    spec:
      project: ml-platform
      source:
        repoURL: https://github.com/acme/ml-platform
        targetRevision: main
        path: 'overlays/{{env}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{namespace}}'

Leaving production on manual sync is a deliberate choice rather than timidity. Automated sync everywhere means a merged pull request reaches production without anyone watching, and for a model whose quality you can only assess against live traffic, a person deciding when to press the button is worth the small loss in automation purity.

Where to start

1. Move one model to a repository with the layout above and pin its version by digest, resisting any floating tag.
2. Create the ArgoCD Application with ignoreDifferences on the KServe status fields from the beginning, so you never debug the self heal loop.
3. Fix the readiness probe to hit the model endpoint and confirm a rollout no longer produces 503s during startup.
4. Add the warm up job in sync wave one with a digest verification and a single smoke inference.
5. Write the promotion gate with an evaluation floor and a regression tolerance, and have it open a pull request rather than apply anything.
6. Practice a rollback in staging with a stopwatch, then decide whether to keep the previous revision warm.
7. Only then fan out with an ApplicationSet, keeping production on manual sync until the staging path is boring.

Conclusion

The reason model deployments stay manual long after application deployments are automated is not that GitOps does not fit them. It is that a few details differ enough to break a naive copy of the application pipeline: the artifact lives outside Git, readiness means loaded rather than listening, the controller writes fields your manifest does not own, and quality is not visible in an HTTP status code.

Handle those four and the rest of GitOps works exactly as advertised. Every model change becomes a reviewable diff, every deployment carries its training data hash and code SHA as lineage, and every rollback is a revert your whole team already knows how to perform. Pick your least critical model this week and move it, because the pipeline is much easier to build before it is urgent.

Ready to Master GitOps and Model Serving?

This pipeline sits on two skills that are worth having independently. The GitOps with ArgoCD course on KodeKloud covers sync strategies, application health, and declarative delivery, KServe Fundamentals covers serving models on Kubernetes properly, and the KodeKloud playgrounds give you clusters to practice a real rollback on. Start with one today.

FAQs