Kubernetes-GitOps落地经验
收录于文集
共53篇
1. GitOps 简介
在笔者看来，Kubernetes与GitOps都遵循着声明式的设计哲学。
使用GitOps会带来如下好处：
集中管理的配置清单：依赖于Git仓库作为唯一资源清单源，可以跟踪YAML文件的每一次更改。这意味着 可审计、易回滚恢复、易灾难迁移，易实现发布自动化。
状态可视：随时显示系统状态与声明状态的偏离，便于运维快速发现在Kubernetes中的误修改或恶意修改。
目前有很多开源的GitOps工具，其中ArgoCD领先一步已被众多公司使用。
2. 落地 GitOp 遇到的问题
更新丢失或更新冲突是落地GitOps时必然会面对的问题。
更新丢失问题
3. 分布式锁解决并发问题
笔者在实践中，串行执行Jenkins pipeline最后的Step解决这个问题。
串行更新 Git 资源清单
为了不引入其额外依赖，使用Kubernetes原有的etcd实现分布式锁。
示例代码如下：
main.go
代码块
JavaScript
自动换行
复制代码
package main
import (
"bufio"
"context"
"fmt"
"github.com/coreos/etcd/clientv3"
"github.com/go-git/go-git/v5"
"github.com/go-git/go-git/v5/plumbing/object"
"github.com/go-git/go-git/v5/plumbing/transport/http"
"go.etcd.io/etcd/clientv3/concurrency"
"log"
"os"
"strings"
"time"
)
const (
repoURL = "https://gitee.com/xxxxx/xxxx.git"
gitUsername = "fanlinjie"
gitPassword = "xxxxxxxxxxxxxxxxxxxxxd" // 这里https协议的不能用普通密码，需要在git端生成令牌
localDir = "gitopsdir"
gitCommitMsg = "pipeline commit"
)
func main() {
if len(os.Args) != 4 {
log.Fatal("Usage: go run main.go <appname> <env>")
}
// Fetch command line arguments.
appName := os.Args[1]
env := os.Args[2]
imageAndTag := os.Args[3]
// 创建etcd客户端配置
config := clientv3.Config{
Endpoints: []string{"192.168.0.248:2379"}, // 确保端口正确，etcd默认端口为2379
DialTimeout: 5 * time.Second,
}
// 建立etcd客户端连接
fmt.Println("start create client")
client, err := clientv3.New(config)
if err != nil {
log.Fatal("Failed to connect to etcd:", err)
}
defer client.Close()
// 定义锁的键名
lockKey := "/my-lock"
// 创建一个session
session, err := concurrency.NewSession(client, concurrency.WithTTL(60))
if err != nil {
log.Fatal("Failed to create session:", err)
}
defer session.Close()
// 尝试获取锁，如果获取不到则等待
mutex := concurrency.NewMutex(session, lockKey)
if err := mutex.Lock(context.Background()); err != nil {
log.Fatal("Failed to acquire lock:", err)
}
fmt.Println("Acquired lock at:", time.Now().Format(time.RFC3339))
// 业务操作，等待5秒
// Create dir if it doesn't exist.
if err := os.MkdirAll(localDir, os.ModePerm); err != nil {
log.Fatalf("Error creating directory: %s\n", err)
}
// Clone the git repository. 拉资源清单项目
// runGitCommand("clone", repoURL)
// r, err := git.Clone(memory.NewStorage(), nil, &git.CloneOptions{
// URL: repoURL,
// })
// if err != nil {
// log.Fatalf("git clone failed : %s\n", err)
// }
r, err := git.PlainClone(localDir, false, &git.CloneOptions{
URL: repoURL,
})
log.Println("clone success")
/*
// Change working directory to dir. 进到gitops目录
if err := os.Chdir(localDir); err != nil {
log.Fatalf("Error changing directory: %s\n", err)
}
// Change to repository directory. //进入到这个项目目录
repoName := filepath.Base(repoURL)
if ext := filepath.Ext(repoName); ext == ".git" {
repoName = repoName[:len(repoName)-4]
}
*/
appyamlfiledir := localDir + "/" + appName + "/overlays/" + env + "/"
/*
if err := os.Chdir(appyamlfiledir); err != nil {
log.Fatalf("Error changing directory to repo: %s\n", err)
}
*/
// Navigate to appname directory and append "abc" to env file.
if err = changeYamlFile(appyamlfiledir, imageAndTag); err != nil {
log.Fatalf("changeYamlFile faile , stop pipeline")
return
}
// Add, commit, and push changes to the remote repository
w, err := r.Worktree()
if err != nil {
fmt.Printf("Cannot get worktree: %s\n", err)
return
}
fmt.Println("Running git add .")
_, err = w.Add(".")
if err != nil {
fmt.Printf("Cannot add changes to the worktree: %s\n", err)
return
}
fmt.Println("Running git commit")
_, err = w.Commit(gitCommitMsg, &git.CommitOptions{
Author: &object.Signature{
Name: "pipeline-program",
When: time.Now(),
},
})
if err != nil {
fmt.Printf("Cannot commit: %s\n", err)
return
}
// 推送
fmt.Println("Running git push -u origin master")
err = r.Push(&git.PushOptions{
RemoteName: "origin",
Auth: &http.BasicAuth{
Username: gitUsername,
Password: gitPassword,
},
Progress: os.Stdout,
})
if err != nil {
fmt.Printf("Cannot push: %s\n", err)
return
}
fmt.Println("Successfully added, committed and pushed changes to origin master")
fmt.Println("Changes have been pushed to the repository.")
time.Sleep(5 * time.Second)
// 释放锁
if err := mutex.Unlock(context.Background()); err != nil {
log.Println("Failed to release lock:", err)
} else {
fmt.Println("Lock released at:", time.Now().Format(time.RFC3339))
}
}
func changeYamlFile(filepath, tag string) error {
// 打开custom_patch.yaml文件
f, err := os.Open(filepath + "/custom_patch.yaml")
if err != nil {
fmt.Printf("打开文件出错: %v\n", err)
return err
}
defer f.Close()
// 创建临时文件
tempFile, err := os.Create(filepath + "tmp.yaml")
if err != nil {
fmt.Printf("创建临时文件出错: %v\n", err)
return err
}
defer tempFile.Close()
// 读取原始文件并按行处理
scanner := bufio.NewScanner(f)
for scanner.Scan() {
line := scanner.Text()
if strings.Contains(line, "image") {
// 替换整个行内容
line = " image: " + tag
}
// 将处理后的行写入临时文件
_, err := tempFile.WriteString(line + "\n")
if err != nil {
fmt.Printf("写入临时文件出错: %v\n", err)
return err
}
}
if err := scanner.Err(); err != nil {
fmt.Printf("读取文件出错: %v\n", err)
return err
}
err = tempFile.Sync()
if err != nil {
log.Fatalf("无法同步文件: %v", err)
return err
}
// 关闭文件来确保数据被刷新到磁盘
tempFile.Close()
f.Close()
// 替换原始文件
err = os.Remove(filepath + "custom_patch.yaml")
if err != nil {
log.Printf("删除文件custom_patch.yaml失败: %v", err)
return err
}
err = os.Rename(filepath+"tmp.yaml", filepath+"custom_patch.yaml")
if err != nil {
fmt.Printf("替换原始文件出错: %v\n", err)
return err
}
return nil
}复制成功
go.mod
代码块
JavaScript
自动换行
复制代码
module github.com/fanlinjie/xxxxxx
go 1.19
replace github.com/coreos/bbolt v1.3.4 => go.etcd.io/bbolt v1.3.4
replace google.golang.org/grpc => google.golang.org/grpc v1.26.0
require (
github.com/golang/protobuf v1.5.4 // indirect
go.etcd.io/etcd v3.3.27+incompatible
google.golang.org/protobuf v1.33.0 // indirect
)
require github.com/coreos/etcd v3.3.27+incompatible
require (
dario.cat/mergo v1.0.0 // indirect
github.com/Microsoft/go-winio v0.6.1 // indirect
github.com/ProtonMail/go-crypto v1.0.0 // indirect
github.com/cloudflare/circl v1.3.7 // indirect
github.com/coreos/bbolt v1.3.4 // indirect
github.com/coreos/go-semver v0.3.1 // indirect
github.com/coreos/go-systemd v0.0.0-20191104093116-d3cd4ed1dbcf // indirect
github.com/coreos/pkg v0.0.0-20240122114842-bbd7aa9bf6fb // indirect
github.com/cyphar/filepath-securejoin v0.2.4 // indirect
github.com/dgrijalva/jwt-go v3.2.0+incompatible // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/emirpasic/gods v1.18.1 // indirect
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
github.com/go-git/go-billy/v5 v5.5.0 // indirect
github.com/go-git/go-git/v5 v5.12.0 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/google/btree v1.1.2 // indirect
github.com/google/go-cmp v0.6.0 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 // indirect
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0 // indirect
github.com/grpc-ecosystem/grpc-gateway v1.16.0 // indirect
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
github.com/jonboulle/clockwork v0.4.0 // indirect
github.com/kevinburke/ssh_config v1.2.0 // indirect
github.com/pjbgf/sha1cd v0.3.0 // indirect
github.com/prometheus/client_golang v1.19.1 // indirect
github.com/sergi/go-diff v1.3.2-0.20230802210424-5b0b94c5c0d3 // indirect
github.com/skeema/knownhosts v1.2.2 // indirect
github.com/soheilhy/cmux v0.1.5 // indirect
github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75 // indirect
github.com/xanzy/ssh-agent v0.3.3 // indirect
github.com/xiang90/probing v0.0.0-20221125231312-a49e3df8f510 // indirect
go.uber.org/multierr v1.10.0 // indirect
go.uber.org/zap v1.27.0 // indirect
golang.org/x/crypto v0.23.0 // indirect
golang.org/x/mod v0.12.0 // indirect
golang.org/x/net v0.22.0 // indirect
golang.org/x/sys v0.20.0 // indirect
golang.org/x/text v0.15.0 // indirect
golang.org/x/time v0.5.0 // indirect
golang.org/x/tools v0.13.0 // indirect
google.golang.org/genproto v0.0.0-20200513103714-09dca8ec2884 // indirect
google.golang.org/grpc v1.33.1 // indirect
gopkg.in/warnings.v0 v0.1.2 // indirect
sigs.k8s.io/yaml v1.4.0 // indirect
)
复制成功
执行示例：
执行成功示例
本文为我原创，未经授权禁止转载
cv34490402
分享至
投诉或建议
