方案
+---------------------+
| Jenkins Master |
| - 持久化配置 |<---(hostpath 存储)
| - 自动容灾 |
+----------+----------+|| Jenkins JNLP 通信|
+----------v----------+ +-------------------+
| Jenkins Agent | | Kubernetes Plugin |
| (动态 Pod) |<---->| (自动创建 Agent) |
+---------------------+ +-------------------+^|
+----------+----------+
| 弹性扩容机制 |
| (HPA + Pod 模板) |
+---------------------+
一、环境准备
k8s 集群ip
kubectl get nodes -o wide # 显示各节点的 InternalIP 和 ExternalIP
检查pod外网是否可通
# 创建临时 Pod 测试公网访问
kubectl run net-test --image=alpine --rm -it --restart=Never -- sh# 在 Pod 内部执行（测试 HTTP 访问）
curl -v -4 --connect-timeout 10 http://example.com# 测试 DNS 解析（确认域名解析是否正常）
nslookup google.com# 测试 TCP 连接（如需要）
nc -zv www.baidu.com 80
jenkins镜像：
docker login kubernetes-register.sswang.com:80 -u sswang
docker pull jenkins/jenkins:2.344 # 官方 2.34 最新版 :cite[2]:cite[5]:cite[7]
docker tag jenkins/jenkins:2.344 kubernetes-register.sswang.com:80/jenkins/jenkins:2.344
docker push kubernetes-register.sswang.com:80/jenkins/jenkins:2.344
docker pull jenkins/jenkins:lts-jdk17
docker tag jenkins/jenkins:lts-jdk17 kubernetes-register.sswang.com:80/jenkins/jenkins:lts-jdk17
docker push kubernetes-register.sswang.com:80/jenkins/jenkins:lts-jdk17
- agent镜像
docker pull jenkins/inbound-agent:jdk11
docker tag jenkins/inbound-agent:jdk11 kubernetes-register.sswang.com:80/jenkins/inbound-agent:jdk11
docker push kubernetes-register.sswang.com:80/jenkins/inbound-agent:jdk11
- 创建namespace
kubectl create namespace jenkins
- hostpath存储准备
kubectl label node kubernetes-master jenkins-master=true # node打标签，支持基于标签选择器进行pod调度
mkdir -p /mnt/jenkins-data
chown 1000:1000 /mnt/jenkins-data # Jenkins 默认 UID 1000 deck1
chmod -R 755 /mnt/jenkins-data
- 版本兼容性说明
二、jenkins master部署
1. jenkins-master.yaml
apiVersion: apps/v1
kind: Deployment
metadata:name: jenkins-masternamespace: jenkins
spec:replicas: 1 # 副本数量selector:matchLabels:app: jenkins-masterstrategy:type: Recreatetemplate:metadata:labels:app: jenkins-masterspec:# 绑定到专用节点（可选）nodeSelector:jenkins-master: "true"securityContext:fsGroup: 1000containers:- name: jenkinsimage: kubernetes-register.sswang.com:80/jenkins/jenkins:lts-jdk17env:- name: JAVA_OPTSvalue: "-Djenkins.install.runSetupWizard=false -Dorg.apache.commons.jelly.tags.fmt.timeZone=Asia/Shanghai"ports:- containerPort: 8080- containerPort: 50000volumeMounts:- name: jenkins-homemountPath: /var/jenkins_homelivenessProbe:httpGet:path: /loginport: 8080initialDelaySeconds: 120periodSeconds: 10readinessProbe:httpGet:path: /loginport: 8080initialDelaySeconds: 30resources:limits:memory: "2Gi"cpu: "1"requests:memory: "1Gi"cpu: "500m"volumes:- name: jenkins-homehostPath:path: /mnt/jenkins-datatype: DirectoryOrCreate
2. jenkins-service.yml
nodeport 对外暴露jenkis服务，能够访问集群内pod。
apiVersion: v1
kind: Service
metadata:name: jenkins-servicenamespace: jenkins
spec:selector:app: jenkins-mastertype: NodePort # 永久对外暴露服务ports:- name: httpport: 8080targetPort: 8080- name: agentport: 50000targetPort: 50000
kubectl get svc -n jenkins jenkins-service
其中，32251为外部访问所使用的端口，ip为宿主机ip或者虚拟机的ip地址。
Jenkins 服务地址确认
# 查看 Jenkins Service 名称
kubectl get svc -n jenkins# 验证内部 DNS 解析（在任意 Pod 内）
kubectl run test-dns --image=busybox -it --rm --restart=Never -- \nslookup jenkins-service.jenkins.svc.cluster.local# [svc].[namespace].svc.cluster.local
3. jenkins-rbac.yml
创建相应的RBAC权限，以便Jenkins Master能够创建和管理Agent Pod
- ServiceAccount：在jenkins命名空间中创建一个名为jenkins的服务账户。
- ClusterRole：定义了一组权限规则，允许对pods、pods/exec、pods/log等资源的操作，以及获取secrets的权限。
- ClusterRoleBinding：将ClusterRole绑定到ServiceAccount，使得jenkins服务账户具有ClusterRole中定义的权限。
apiVersion: v1
kind: ServiceAccount
metadata:name: jenkins # jenkins serviceaccountnamespace: jenkins
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:name: jenkins-agent-role
rules:
- apiGroups: [""]resources: ["pods", "pods/exec", "pods/log"]verbs: ["create", "delete", "get", "list", "watch", "patch"]
- apiGroups: [""]resources: ["secrets"]verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: jenkins-agent-role-binding
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: jenkins-agent-role
subjects:
- kind: ServiceAccountname: jenkinsnamespace: jenkins
kubectl get serviceaccount -n jenkins
如何让某个访问作为serviceaccount 为jenkins的账户？
apiVersion: v1
kind: Secret
metadata:name: jenkins-tokennamespace: jenkinsannotations:kubernetes.io/service-account.name: jenkins
type: kubernetes.io/service-account-token
至此就能通过secret访问serviceaccount，进而访问apiserver拥有相应的pod操作权限。
在这里插入图片描述
三、动态Agent配置与kubernetes插件
1. 插件初始化
https://mirrors.huaweicloud.com/jenkins/updates/update-center.json
名称: kubernetes
Kubernetes 地址: https://kubernetes.default.svc.cluster.local
命名空间: jenkins
凭据: jenkins-service-account (选择创建的 ServiceAccount)
Jenkins 地址: http://jenkins-service.jenkins.svc.cluster.local:8080
凭据获取，要求k8s集群配置：serviceaccount账户作为 集群操作权限的承接者，secret绑定account，用于apiserver鉴权，rbac用于将serviceaccount绑定role。
TOKEN=$(kubectl get secret jenkins-token -n jenkins -o jsonpath='{.data.token}' | base64 --decode)
echo $TOKEN
服务证书key获取：
CA_CRT=$(kubectl get secret jenkins-token -n jenkins -o jsonpath='{.data.ca\.crt}' | base64 -d)
echo "$CA_CRT"
# 验证 ServiceAccount 权限
kubectl auth can-i create pods \--as system:serviceaccount:jenkins:jenkins \-n jenkins
# 应返回 yes# 验证列表权限
kubectl auth can-i list pods \--as system:serviceaccount:jenkins:jenkins \-n jenkins
# 应返回 yes
# [命名空间][serviceaccount]
2. pod 模板
名称: jnlp-agent
标签: jnlp-agent
容器模板:
- 名称: jnlp镜像: kubernetes-register.sswang.com:80/jenkins/inbound-agent:jdk11参数: '${computer.jnlpmac} ${computer.name}'资源限制:CPU: 500m内存: 512Mi资源请求:CPU: 200m内存: 256Mi
podTemplate:name: jenkins-agentlabel: jenkins-agentserviceAccount: jenkins
# imagePullSecrets:
# - name: docker-registry-credcontainers:- name: jnlpimage: kubernetes-register.sswang.com:80/jenkins/inbound-agent:jdk11alwaysPullImage: falseresourceLimitCpu: "1"resourceLimitMemory: "2Gi"args: '$(JENKINS_SECRET) $(JENKINS_NAME)'
)
)
