Declaratively Automating Infrastructure with GitOps — From Deployment to Automated Recovery with Argo CD

Do you remember the days when we applied infrastructure changes by connecting directly to the server via SSH? I did that for a while, but I recall being quite perplexed one day when I discovered that the settings for the staging and production environments had gradually diverged. There was no way to track who made the changes or why. I later found out that a team member had connected directly and modified a setting out of urgency, and that change had been buried for several months. GitOps emerged precisely to solve that problem.

"Isn't this something DevOps or infrastructure personnel talk about?" you might say, but regardless of whether you are a frontend or backend developer, the moment you understand how the deployment pipeline works, your development speed changes. You become able to answer questions on your own, such as why your pull request hasn't been implemented yet or why staging and production behavior differ.

In this article, I will guide you through understanding the four core principles of GitOps and actual workflows, and enabling you to configure an Argo CD-based declarative deployment pipeline yourself. If you are new to Kubernetes, you can follow along more quickly by referring to the short explanations throughout the article.

Key Concepts

The 4 Principles of GitOps

When you first encounter GitOps, it is easy to think, "Isn't it just deploying with Git?" However, there are some more specific principles. These principles, summarized by Alexis Richardson of Weaveworks in 2017, still form the basis of the OpenGitOps specification.

I also initially looked at the chart and thought, "Isn't that just a given in principle?" but it was only after operating with the Pull method that I truly realized how much of a real difference it makes.

Declarative vs. Imperative: The imperative describes a procedure, such as "install Nginx on the server, open port 80, and start the service." The declarative describes "the state where Nginx is running on port 80" and leaves the method of achieving that state to the tools.

Drift: This is a phenomenon where the actual state of the system differs from the state declared in Git. It occurs when someone manually modifies settings using kubectl for an emergency patch, or when settings are changed due to external factors. GitOps operators continuously detect this drift and automatically recover from it.

Why the Pull Method Is Important

Traditional CI/CD was structured so that pipelines pushed directly to the cluster. This meant that the pipeline server had to hold cluster access credentials, and if those credentials were leaked, the entire cluster would be at risk.

When I first saw the Pull method, I wondered, "Why is the cluster pulling directly?" but it was only after witnessing a credential-related incident that I realized how important it is.

GitOps' pull method works in the opposite way. Operators like Argo CD or Flux watch Git inside the cluster, and when changes are detected, the cluster itself synchronizes. There is no need to expose credentials externally.

[기존 Push 방식]
CI 서버 ---(클러스터 자격증명 필요)---> Kubernetes 클러스터
 
[GitOps Pull 방식]
Git 저장소 <---(감시)--- Argo CD (클러스터 내부)
                              |
                         자동 동기화
                              ↓
                      Kubernetes 클러스터

Kubernetes is a platform that operates containers as clusters. It groups multiple servers together to deploy, scale, and recover applications. If you are new to it, you can understand it as an "operating system that treats servers as a single unit."

View the Entire Workflow at a Glance

개발자 PR 작성
    ↓
코드 리뷰 + 머지
    ↓
CI 파이프라인 (GitHub Actions 등)
 - 테스트 실행
 - 컨테이너 이미지 빌드
 - 이미지 레지스트리에 Push
 - 매니페스트 저장소의 이미지 태그 업데이트
    ↓
GitOps 오퍼레이터 (Argo CD / Flux)가 변경 감지
    ↓
클러스터에 자동 배포
    ↓
드리프트 발생 시 → 자동 복구

The most commonly used pattern in practice is the GitHub Actions (CI) + Argo CD (CD) combination. This structure clearly separates the roles of CI and CD, allowing CI to focus solely on building and testing while Argo CD handles deployment. Although Flux is a competing tool that performs a similar role, Argo CD is more frequently chosen due to its intuitive Web UI, multi-cluster support, and the stability stemming from its status as a CNCF graduation project.

Practical Application

Example 1: Argo CD Application Declaration

This is the most basic Argo CD configuration. It declares "which path of which Git repository to deploy to which cluster" through a CRD (Custom Resource Definition — a type defined in addition to the default Kubernetes resources) named Application.

# argocd-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/my-org/my-app-manifests
    targetRevision: v1.2.0   # 실제 운영에서는 특정 태그 또는 커밋 SHA를 지정하는 것을 권장합니다 (HEAD는 안티패턴)
    path: overlays/production        # Kustomize 오버레이 경로
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true       # Git에서 삭제된 리소스는 클러스터에서도 삭제
      selfHeal: true    # 수동 변경이 생기면 Git 상태로 자동 복구
    syncOptions:
      - CreateNamespace=true

The file you created can be applied to the cluster using the command below.

kubectl apply -f argocd-app.yaml

Example 2: Automatically update image tags with GitHub Actions

This is a pattern that automatically updates the image tag in the manifest repository after building a new image in the CI pipeline. This commit becomes the trigger for Argo CD.

# .github/workflows/deploy.yaml
name: Build and Update Manifest
 
on:
  push:
    branches: [main]
 
jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
 
      - name: Log in to container registry
        run: echo "${{ secrets.REGISTRY_PASSWORD }}" | docker login my-registry -u ${{ secrets.REGISTRY_USERNAME }} --password-stdin
 
      - name: Build and push image
        run: |
          IMAGE_TAG=${{ github.sha }}
          docker build -t my-registry/my-app:$IMAGE_TAG .
          docker push my-registry/my-app:$IMAGE_TAG
          echo "IMAGE_TAG=$IMAGE_TAG" >> $GITHUB_ENV
 
      - name: Update manifest repository
        run: |
          git clone https://github.com/my-org/my-app-manifests.git
          cd my-app-manifests/overlays/production
          # kustomize로 이미지 태그 업데이트 (base/kustomization.yaml에 images 블록이 있어야 작동합니다)
          kustomize edit set image my-app=my-registry/my-app:$IMAGE_TAG
          git config user.email "ci@my-org.com"
          git config user.name "CI Bot"
          git commit -am "chore: update image tag to $IMAGE_TAG"
          git push
        env:
          GITHUB_TOKEN: ${{ secrets.MANIFEST_REPO_TOKEN }}
          # MANIFEST_REPO_TOKEN: GitHub → Settings → Developer settings → Personal access tokens에서
          # 매니페스트 저장소에 대한 'repo' 권한으로 발급 후, 앱 코드 저장소의 Secrets에 등록하세요.
          # 기본 GITHUB_TOKEN은 다른 저장소에 push할 수 없어 403 오류가 납니다.

Use proven Actions instead of direct repository cloning: The git clone + git push method is prone to errors because it requires manual handling of token management and git config. Using Actions like peter-evans/create-pull-request or stefanzweifel/git-auto-commit-action allows for a much more concise process, so it is recommended to consider these alternatives when implementing them in a production environment.

Reasons to Separate App Code Repository and Manifest Repository: Separating the two repositories provides a clear deployment history and allows you to manage infrastructure and code change permissions independently. It may feel cumbersome at first, but this separation proves its worth as the team grows.

Example 3: Branching configurations by environment with Kustomize

Kustomize is a tool that branches Kubernetes manifests by environment using an overlay method. It is built into kubectl by default, so it can be used without a separate installation.

manifests/
├── base/
│   ├── deployment.yaml   # 공통 설정
│   ├── service.yaml
│   └── kustomization.yaml
└── overlays/
    ├── staging/
    │   ├── kustomization.yaml   # 스테이징 오버레이
    │   └── replica-patch.yaml  # replicas: 1
    └── production/
        ├── kustomization.yaml   # 프로덕션 오버레이
        └── replica-patch.yaml  # replicas: 3

For the kustomize edit set image command to function correctly, the images block must exist in base/kustomization.yaml beforehand. If you omit this during the initial setup, it is easy to become confused and wonder, "Why aren't the tags changing?"

# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
 
resources:
  - deployment.yaml
  - service.yaml
 
images:
  - name: my-app
    newTag: latest   # CI가 overlays에서 이 값을 덮어씁니다

# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
 
resources:
  - ../../base
 
patches:
  - path: replica-patch.yaml
 
images:
  - name: my-app
    newTag: "abc1234"   # CI가 이 값을 자동으로 업데이트

Pros and Cons Analysis

Advantages

Disadvantages and Precautions

To be honest, the learning curve is the biggest hurdle. Our team also spent the first three weeks just setting it up. However, once you get on track, the convenience is overwhelming, so I think it was a good decision not to give up and sticking with it.

The Most Common Mistakes in Practice

1. Simply committing secrets to an app code repository — Honestly, I also initially thought, "It's a private repository anyway, so it should be fine," but a survey indicates that as of 2025, 61% of organizations have exposed secrets to public repositories. We recommend implementing Sealed Secrets or an External Secrets Operator from the start.
2. Managing both app code and the manifest in a single repository — It is convenient at first, but the pipeline becomes complex as CI triggers and GitOps triggers get mixed up. It is much more convenient in the long run to separate them from the beginning.
3. Attempting a manual hotfix with selfHeal: true enabled — It is disconcerting to experience a situation where you patch manually with kubectl in an emergency, only for Argo CD to revert to the original state a few seconds later. When urgent fixes are needed, it is recommended to first pause automatic synchronization or share the workflow of committing directly to Git with your team in advance.

In Conclusion

I mentioned at the beginning of the post that "there was no way to track who made changes or why," but once you adopt GitOps, the answer to that question is always in the Git history. One line is enough.

GitOps is more than just a technology to automate deployments; it is an operational culture that empowers the entire team to have trust and visibility into changes.

Trying to apply it to your entire infrastructure from the start can be overwhelming. Starting with small steps in the order below will help you get a feel for GitOps.

1. Try installing Argo CD on local minikube — After installing by referring to the Official Argo CD Installation Guide, configure kubectl port-forward svc/argocd-server -n argocd 8080:443, and then check the deployment status directly through the Web UI in localhost:8080, you can quickly get a feel for GitOps.
2. Separating the Kubernetes manifest of a simple app into a separate Git repository — If you move deployment.yaml and service.yaml to the my-app-manifests repository and apply the Kustomize structure (base/, overlays/), you can experience the convenience of branching settings by environment.
3. Connecting GitHub Actions to automatically commit image tags in the manifest repository after building an image — By referring to the workflow in Example 2 above and connecting CI to act as a trigger for Argo CD, a basic GitOps pipeline is completed.

Next Post: We plan to cover practical patterns (Sealed Secrets, External Secrets Operator) for managing multi-cluster environments and securely handling secrets in Argo CD.

Reference Materials

• GitOps in 2025: From Old-School Updates to the Modern Way | CNCF
• Argo CD - Declarative GitOps CD for Kubernetes | Official Documentation
• A Guide to Secrets Management with GitOps and Kubernetes | Red Hat
• Argo CD and Flux use cases | AWS Prescriptive Guidance