Gitlab-Runner安装

文章目录

helm方式安装在K8S上
参考
- gitlab CI/CD 文件变量
- 缓存服务器
- - K8S部署
- docker镜像
- - maven
  - docker
  - - 安装docker buildx
  - minio
  - node
  - helm
  - kubectl
  - sonar-scanner-cli
- 问题
- - 清除cache
  - helm执行时无权限
- 下载镜像失败
- - 下载gitlab-runner镜像失败
- Gitlab-ci中使用
- - java
  - 前端

helm方式安装在K8S上

1、下载charts

helm pull gitlab/gitlab-runner
tar -zxvf gitlab-runner-0.27.0.tgz#解压后内容：CHANGELOG.mdChart.yaml    #CONTRIBUTING.mdLICENSEMakefileNOTICEREADME.mdtemplates    #values.yaml  #

2、修改 values.yaml，templates 等资源

values.yaml

gitlabUrl: https://gitlab.example.com/  #修改为gitlab地址
runnerRegistrationToken: ""             #修改为gitlab runner token，可从 /admin/runners 查看
rbac:create: trueclusterWideAccess: trueserviceAccountName: gitlab-runner-gitlab-runner
runners:tags: ""  serviceAccountName: gitlab-runner-gitlab-runner

templates/configmap.yaml

主要用于maven，docker绑定本地目录，修改 entrypoint key。增加 config.toml 配置。

    #以下一段是增加的内容cat >>/home/gitlab-runner/.gitlab-runner/config.toml <<EOF[[runners.kubernetes.volumes.host_path]]name = "maven"mount_path = "/root/.m2"read_only = falsehost_path = "/root/.m2"[[runners.kubernetes.volumes.host_path]]name = "docker"mount_path = "/var/run/docker.sock"read_only = truehost_path = "/var/run/docker.sock"EOF# Start the runnerexec /entrypoint run --user=gitlab-runner \--working-directory=/home/gitlab-runner

新的方式可以通过values.yaml 的 runners 段设置属性。不能同时以上面和下面2种方式，不然会重复。

runners:config: |[[runners]][runners.kubernetes]image = "ubuntu:16.04"[[runners.kubernetes.volumes.host_path]]name = "maven"mount_path = "/root/.m2"read_only = falsehost_path = "/root/.m2"[[runners.kubernetes.volumes.host_path]]name = "docker"mount_path = "/var/run/docker.sock"read_only = truehost_path = "/var/run/docker.sock"

_cache.tpl

里面CACHE_S3_INSECURE 参数是固定值，导致 values 配置无效。

{{-       if .Values.runners.cache.s3CacheInsecure }}
- name: CACHE_S3_INSECUREvalue: "true"
{{-       end }}{{ default "" .Values.runners.cache.s3BucketLocation | quote }}#-----   修改为：- name: CACHE_S3_INSECUREvalue: {{ default "true" .Values.runners.cache.s3CacheInsecure | quote }}

3、添加 helm 仓库

helm repo add gitlab https://charts.gitlab.io

4、创建namespace、等资源

kubectl create ns gitlab

---
apiVersion: v1
data:accesskey: bWluaW8=     #base64 编码secretkey:     #base64 编码
kind: Secret
metadata:name: minio-secrets
type: Opaque

5、启动 gitlab-runner

# 安装仓库中的chart
$ helm install   gitlab-runner   --namespace gitlab    -f values.yaml gitlab/gitlab-runner  
#安装本地的chart
helm install   gitlab-runner  ./   --namespace gitlab#更新配置--通过本地chart更新helm upgrade --install   gitlab-runner    ./gitlab-runner  --namespace gitlab #卸载
helm uninstall gitlab-runner --namespace gitlab

如果没有修改gitlabUrl，则会提示更新配置

#############################################################################################
## WARNING: You did not specify an gitlabUrl in your 'helm install' call.                  ##
#############################################################################################This deployment will be incomplete until you provide the URL that your
GitLab instance is reachable at:helm upgrade gitlab-runner \--set gitlabUrl=http://gitlab.your-domain.com,runnerRegistrationToken=your-registration-token \gitlab/gitlab-runner#也可以使用命令：helm upgrade

参考

安装：https://docs.gitlab.com/runner/install/

https://docs.gitlab.com/runner/

执行器参数：https://docs.gitlab.com/runner/executors/kubernetes.html

cache secret ： https://blog.csdn.net/xichenguan/article/details/101436883

gitlab runner配置（toml配置项）：https://docs.gitlab.com/runner/configuration/advanced-configuration.html

gitlab CI/CD 文件变量

新的版本支持，比较旧的不支持。

但是可以通过base64 编解码来实现

echo $(cat ~/.kube/config | base64) | tr -d " "

deploy_k8s_job:image: registry.cn-hangzhou.aliyuncs.com/haoshuwei24/kubectl:1.16.6stage: deploy_k8stags:- k8s-runnerscript:- mkdir -p /etc/deploy- echo $kube_config |base64 -d > $KUBECONFIG- sed -i "s/IMAGE_TAG/$CI_PIPELINE_ID/g" deployment.yaml- cat deployment.yaml- kubectl apply -f deployment.yaml

缓存服务器

使用minio作为缓存服务器。配置如下：

  cache:## General settings## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration and https://docs.gitlab.com/runner/install/kubernetes.html#using-cache-with-configuration-templatecacheType: s3cachePath: "gitlab_runner"cacheShared: true## S3 settings## DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration and https://docs.gitlab.com/runner/install/kubernetes.html#using-cache-with-configuration-templates3ServerAddress: s3.amazonaws.coms3BucketName: "gitlabrunner"          #Minio buckets3BucketLocation:        #minio时区。s3CacheInsecure: false   #是否在不安全模式。true：使用http；false使用https，不设置则默认为false。## S3 the name of the secret.secretName: minio-secrets     #minio 对应的secret

**注意：**很多博客或者什么资料，把s3CacheInsecure解释为是否使用https，正确的解释应该是是否在不安全模式。意思刚好相反。

最终的文件内容可以在/home/gitlabrunner/.gitlabrunner/config.toml 文件查看。值为false时不会出现在config.toml中。

以上方式是废弃的方式，新的方式采用template。对应的template为_cache.yaml

runners:config: |[[runners]][runners.kubernetes]image = "ubuntu:16.04"[runners.cache]Type = "s3"Path = "gitlab_runner"Shared = true[runners.cache.s3]ServerAddress = "s3.amazonaws.com"BucketName = "gitlabrunner"BucketLocation = "eu-west-1"Insecure = true#AccessKey = "access"   #SecretKey = "secret123456"cache:secretName: minio-secrets

以上使用到了一个secret。通过以下语句创建secret 或者通过yaml创建。

kubectl create secret generic minio \
--from-literal=accesskey="access" \
--from-literal=secretkey="secret123456" -n gitlab

参考：https://docs.gitlab.com/runner/install/kubernetes.html#using-cache-with-configuration-template

K8S部署

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:name: miniofinalizers:- kubernetes.io/pvc-protection
spec:accessModes:- ReadWriteOnceresources:requests:storage: 5GistorageClassName: rook-cephfsvolumeMode: Filesystem---
apiVersion: v1
kind: Service
metadata:labels:app: minioname: miniospec:ports:- name: 9000-tcpport: 9000protocol: TCPtargetPort: 9000selector:app: miniosessionAffinity: Nonetype: ClusterIP---
apiVersion: apps/v1
kind: Deployment
metadata:labels:app: minioname: miniospec:replicas: 1revisionHistoryLimit: 10selector:matchLabels:app: miniotemplate:metadata:labels:app: miniospec:containers:- image: minio/minio:RELEASE.2019-02-26T19-51-46ZimagePullPolicy: Alwaysenv:- name: MINIO_ACCESS_KEYvalue: minio- name: MINIO_SECRET_KEY  value: sssscommand:- minio- server- /dataname: minioports:- containerPort: 9000protocol: TCPterminationMessagePath: /dev/termination-logterminationMessagePolicy: FilevolumeMounts:- mountPath: /dataname: volume-datadnsPolicy: ClusterFirstrestartPolicy: AlwaysschedulerName: default-schedulerimagePullSecrets:- name: harbor-key   #注意docker 仓库 keysecurityContext: {}terminationGracePeriodSeconds: 30volumes:- name: volume-datapersistentVolumeClaim:claimName: minio

kubectl apply -f minio.single.yaml -n gitlab

docker镜像

maven

maven:3.6.3-openjdk-8： https://registry.hub.docker.com/_/maven

maven的setting.xml 可以通过configmap解决，（没验证过）

        [[runners.kubernetes.volumes.config_map]]name = "gitlab-runner-maven"mount_path = "/usr/share/maven/configmap/"

也可以通过mount path解决（见前面内容）

[[runners.kubernetes.volumes.host_path]]

docker

docker ：https://registry.hub.docker.com/_/docker 。版本：（20.10.2）

需要在/root/.docker/config.json 中增加auth 凭据。

FROM docker
MAINTAINER lihz
ADD  config.json  /root/.docker/config.json

config.json

主要是增加访问凭据

{"auths": {"192.168.1.X": {"auth": "?????????????"},"docker-registry-default.cloud.com": {"auth": "YWRtaW46TEpWUUhYX2g3MGFabGYtUlJLdDc1RlBmRW5LeFRXXXXXXXXXXX"}},"experimental": true
}

安装docker buildx

如果需要支持多平台打包，则需要安装docker buildx (github.com/docker/buildx v0.10.5 86bdced)，下载

wget -O docker-buildx  https://github.com/docker/buildx/releases/download/v0.10.5/buildx-v0.10.5.linux-amd64
mkdir -p /usr/libexec/docker/cli-plugins/docker-buildx
mv docker-buildx /usr/libexec/docker/cli-plugins/docker-buildx
chmod +x /usr/libexec/docker/cli-plugins/docker-buildxdocker buildx version

docker客户端需要开启实验室功能

$ cat ~/.docker/config.json
{"experimental": "enabled"
}# 确认实验室性能开启。
$ docker version

构造docker 打包的镜像，包含buildx

FROM docker:20.10.2
MAINTAINER lihz
ADD  config.json  /root/.docker/config.json
RUN mkdir -p /usr/libexec/docker/cli-plugins/  && mkdir -p /etc/docker
COPY docker-buildx /usr/libexec/docker/cli-plugins/docker-buildx
COPY daemon.json buildkitd.toml  /etc/docker/
RUN chmod +x /usr/libexec/docker/cli-plugins/docker-buildx
ENV IMAGE_BUILDKIT=192.168.1.X/GROUP/buildkit:buildx-stable-1

buildkitd.toml

debug = true
# insecure-entitlements allows insecure entitlements, disabled by default.
insecure-entitlements = [ "network.host", "security.insecure" ]# 如果不加这些，就会默认使用https请求。
# optionally mirror configuration can be done by defining it as a registry.
[registry."192.168.1.XX"]http = trueinsecure = true

打包

minio

minio/minio:RELEASE.2019-02-26T19-51-46Z : https://registry.hub.docker.com/r/minio/minio

node

node:14.7.0 : https://registry.hub.docker.com/_/node

FROM node:14.7.0
RUN npm config set registry https://registry.npm.taobao.org

helm

alpine/helm:3.5.0：https://registry.hub.docker.com/r/alpine/helm

Dockerfile：

From 192.168.1.X/GROUP/helm:3.5.0
#增加K8S的凭据
ADD config /etc/deploy/config

config：

K8S的凭据

apiVersion: v1
clusters:
- cluster:certificate-authority-data: ........server: https://lb.kubesphere.local:6443name: cluster.local
contexts:
- context:cluster: cluster.localnamespace: demouser: kubernetes-adminname: ctx-demo
- context:cluster: cluster.localuser: kubernetes-adminname: kubernetes-admin@cluster.local
current-context: ctx-demo
kind: Config
preferences: {}
users:
- name: kubernetes-adminuser:client-certificate-data: ..........client-key-data: ..........

kubectl

将业务镜像部署到k8s上

sonar-scanner-cli

用于扫描前端代码。参考：https://docs.sonarqube.org/latest/analysis/scan/sonarscanner/

sonarsource/sonar-scanner-cli:4.6：https://registry.hub.docker.com/r/sonarsource/sonar-scanner-cli

Dockerfile：

From sonarsource/sonar-scanner-cli:4.6
#登录凭据
ENV SONAR_HOST_URL=http://192.168.1.XXX:9000 SONAR_LOGIN=a34d8e475e19faa108404fec82cd058493XXXXXX
ENTRYPOINT [""]

绑定目录：

docker run --rm -v $PWD:/usr/src

问题

https://docs.gitlab.com/ee/ci/docker/using_docker_build.html

清除cache

cache是没有过期时间的，而且每一次新的push触发的pipeline，都会重新生成cache，重新生成的cache的名字为“-”，其中num是随着push数量递增的。如果不去清除cache，cache会永久保留在Runner上，日积月累会填满存储空间的，因此最好隔一段时间进行一次清除，清除方法请参考https://docs.gitlab.com/ee/ci/caching/#clearing-the-cache，或者使用clear_volumes.sh 这个简单脚本来处理它, 清除cache的原理是将相关的volume移除，当然，docker也有自带的清除命令，推荐将docker system prune -f --volumes加入到定时任务中。

helm执行时无权限

Executing "step_script" stage of the job script
$ sed -i "s/IMAGE_TAG/$DOCKER_TAG/g;s/CI_PROJECT_NAME/$CI_PROJECT_NAME/g;s/SVC_PORT/${SVC_PORT}/g;" ${MODULE_NAME}/src/main/charts/values.yaml
$ sed -i "s/CI_PROJECT_NAME/$CI_PROJECT_NAME/g" ${MODULE_NAME}/src/main/charts/Chart.yaml
$ helm upgrade --install $CI_PROJECT_NAME ${MODULE_NAME}/src/main/charts -n $K8S_NS
Release "sample" does not exist. Installing it now.
Error: rendered manifests contain a resource that already exists. Unable to continue with install: could not get information about the resource: deployments.apps "sample" is forbidden: User "system:serviceaccount:gitlab:gitlab-runner-gitlab-runner" cannot get resource "deployments" in API group "apps" in the namespace "release"
ERROR: Job failed: command terminated with exit code 1

是由于 gitlab runner的权限问题

执行以下语句：

kubectl create clusterrolebinding gitlab-cluster-admin --clusterrole=cluster-admin --group=system:serviceaccounts

下载镜像失败

Job failed (system failure): prepare environment: image pull failed

临时解决方法，在K8S节点 docker pull <IMAGE> 把镜像下载下来

根本性解决：

打开以下选项，并设置docker仓库的secret。

## Specifying ImagePullSecrets on a Pod （设置在gitlab-runner中）
## Kubernetes supports specifying container image registry keys on a Pod.
## ref: https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
##
imagePullSecrets:- name: "harbor-key"## For RBAC support:
rbac:create: true## Specify one or more imagePullSecrets used for pulling the runner image#### ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account##imagePullSecrets: ["harbor-key"]## Configuration for the Pods that the runner launches for each new job
##
runners:## Specify one or more imagePullSecrets  （用于拉取image）#### ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#### DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configurationimagePullSecrets: ["harbor-key"]## Run all containers with the privileged flag enabled## This will allow the docker:dind image to run if you need to run Docker## commands. Please read the docs before turning this on:## ref: https://docs.gitlab.com/runner/executors/kubernetes.html#using-dockerdind#### DEPRECATED: See https://docs.gitlab.com/runner/install/kubernetes.html#additional-configuration#privileged: true

下载gitlab-runner镜像失败

在K8S部署环境中，会下载以下镜像，可能会导致失败，最好重新tag在本地仓库

# helm配置（helpers.tpl中）为：
printf "192.168.1.X/GROUP/gitlab-runner:alpine-%s" $appVersion
#tag为：
192.168.1.x/GROUP/gitlab-runner:alpine-v13.8.0# 最后一部分是 CI_RUNNER_VERSION，对应的版本的 sha256ID，参考：https://gitlab.com/gitlab-org/gitlab-runner/-/tags?sort=updated_desc&search=13.8.0
gitlab/gitlab-runner-helper:x86_64-775dd39d

docker tag gitlab/gitlab-runner-helper:x86_64-775dd39d   192.168.1.X/GROUP/gitlab-runner-helper:x86_64-775dd39d
docker push 192.168.1.X/GROUP/gitlab-runner-helper:x86_64-775dd39d

修改配置：

      [[runners]][runners.kubernetes]image = "ubuntu:22.04"# 由上文可知helper_image = "192.168.1.X/GROUP/gitlab-runner-helper:x86_64-775dd39d"

Gitlab-ci中使用

java

variables:DOCKER_TAG: "3.0.0-RELEASE"MODULE_NAME: "project-biz"SONAR_PROJECT_KEY: "project"stages:- package- docker_buildmvn_build_job:image: ${DEPOSITORY}/mavenstage: packagescript:- mvn clean verify sonar:sonar -DskipTests -DskipDocker -Dsonar.projectVersion=master -Dsonar.projectKey=$SONAR_PROJECT_KEY  -Dsonar.host.url=${SONAR_URL} -Dsonar.login=${SONAR_TOKEN}- mvn deploy -B -DskipTests -DskipDockerartifacts:paths:- ${MODULE_NAME}/target/*.jaronly:- master- /^.*-dev$/when: manualmvn_build_release_job:image: ${DEPOSITORY}/mavenstage: packagescript:- mvn deploy -B -DskipTests -DskipDockerartifacts:paths:- ${MODULE_NAME}/target/*.jaronly:- /^.*-RELEASE$/- /^.*-release/- /^.*-hotfix$/docker_build_release_job:image: ${DEPOSITORY}/dockerstage: docker_buildscript:- cp ${MODULE_NAME}/target/*.jar ${MODULE_NAME}/src/main/docker- docker build -t ${DEPOSITORY}/${MODULE_NAME}:${DOCKER_TAG} ${MODULE_NAME}/src/main/docker- docker push ${DEPOSITORY}/${MODULE_NAME}:${DOCKER_TAG}only:- /^.*-RELEASE$/- /^.*-release/

前端

variables:DOCKER_TAG: "dev"MODULE_NAME: "biz-web"stages:- package- docker_build- deploy npm_build_job:image: maven:3.6.3-openjdk-8stage: packagecache:paths:- node_modules/artifacts:paths:- distscript:- npm install- npm run buildonly:- master- /^.*-dev$/when: manual  docker_build_job:image: dockerstage: docker_buildscript:- docker build -t ${DEPOSITORY}/${MODULE_NAME}:${DOCKER_TAG} ./dependencies:- npm_build_jobonly:- master- /^.*-dev$/when: manualdocker_build_release_job:image: dockerstage: docker_buildscript:- docker build -t ${DEPOSITORY}/${MODULE_NAME}:${DOCKER_TAG} ./dependencies:- npm_build_job  only:- /^.*-RELEASE$/- /^.*-release/- /^.*-hotfix$/