由于禁止的秘密访问规则,无法在 gitlab-runner 中使用 helm upgrade
Posted
技术标签:
【中文标题】由于禁止的秘密访问规则,无法在 gitlab-runner 中使用 helm upgrade【英文标题】:Cannot use helm upgrade in a gitlab-runner due forbidden secret access rules 【发布时间】:2020-07-16 21:09:08 【问题描述】:我想在 gitlab 管道中的 kubernetes 上的 gitlab-runner 上使用执行 helm。
我的 gitlab.ci.yaml:
# Deployment step
deploy:
stage: deploy
image: alpine/helm:latest
script:
- helm --namespace gitlab upgrade initial ./iot/
tags:
- k8s
- dev
到目前为止我做了什么:
-
使用 helm 在我的 kubernetes 上安装了 gitlab-runner (https://docs.gitlab.com/runner/install/kubernetes.html)
我的价值观.yaml:
image: gitlab/gitlab-runner:alpine-v11.6.0
imagePullPolicy: IfNotPresent
gitlabUrl: https://gitlab.com/
runnerRegistrationToken: "mytoken"
unregisterRunners: true
terminationGracePeriodSeconds: 3600
concurrent: 10
checkInterval: 30
## For RBAC support:
rbac:
create: true
## Define specific rbac permissions.
# resources: ["pods", "pods/exec", "secrets"]
# verbs: ["get", "list", "watch", "create", "patch", "delete"]
## Run the gitlab-bastion container with the ability to deploy/manage containers of jobs cluster-wide or only within namespace
clusterWideAccess: false
metrics:
enabled: true
## Configuration for the Pods that that the runner launches for each new job
##
runners:
## Default container image to use for builds when none is specified
##
image: ubuntu:16.04
locked: false
tags: "k8s,dev"
privileged: true
namespace: gitlab
pollTimeout: 180
outputLimit: 4096
cache:
## Build Container specific configuration
##
builds:
# cpuLimit: 200m memoryLimit: 256Mi cpuRequests: 100m memoryRequests: 128Mi
## Service Container specific configuration
##
services:
# cpuLimit: 200m memoryLimit: 256Mi cpuRequests: 100m memoryRequests: 128Mi
## Helper Container specific configuration
##
helpers:
securityContext:
fsGroup: 65533
runAsUser: 100
## Configure resource requests and limits ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
affinity:
nodeSelector:
tolerations: []
envVars:
name: RUNNER_EXECUTOR
value: kubernetes
## list of hosts and IPs that will be injected into the pod's hosts file
hostAliases: []
podAnnotations:
podLabels:
-
gitlab-runner 与 gitlab.com 成功连接
但在执行部署步骤时,我在 gitlab 上收到以下消息:
Error: UPGRADE FAILED: query: failed to query with labels: secrets is forbidden: User "system:serviceaccount:gitlab:default" cannot list resource "secrets" in API group "" in the namespace "gitlab"
我检查了我的 RBAC ClusterRules,它们都默认设置为动词和资源上的通配符,但我也尝试设置所需的权限:
resources: ["pods", "pods/exec", "secrets"]
verbs: ["get", "list", "watch", "create", "patch", "delete"]
没有任何效果:-( 我什么时候做错了?
【问题讨论】:
部署在错误的命名空间?似乎有一个命名空间默认值和一个名为 gitlab。 仍然无法正常工作:-( 我遇到了同样的错误。如果您碰巧找到了解决方案,请发布更新! 【参考方案1】:这是我解决此问题的步骤:
首先,为您的 Gitlab Runner 创建服务帐户和自定义角色:
kubectl create sa sa-runner -n gitlab
kubectl create role sa-runner-role -n gitlab --verb=get,list,watch,create,delete,patch --resource=pods,secret,pods/exec
kubectl create rolebinding sa-runner-rolebinding -n gitlab --role=sa-runner-role --serviceaccount=gitlab:sa-runner
kubectl create clusterrole deploy-default --verb=get,list,watch,patch --resource=deployment
kubectl create clusterrolebinding deploy-default-binding --clusterrole=deploy-default --serviceaccount=gitlab:default
然后,将 serviceAccountName: sa-runner 添加到您的 values.yaml 文件中
## Use the following Kubernetes Service Account name if RBAC is disabled in this Helm chart (see rbac.create)
serviceAccountName: "gitlab-sa"
你已经完成了,别忘了在你的 Helm 中进行升级
【讨论】:
【参考方案2】:我通过将 runners.serviceAccountName 设置为我在命名空间中创建的具有所有权限的 serviceAccount 解决了同样的问题。
【讨论】:
很高兴听到您修复了它。但是你能分享你的修复吗?有代码吗?【参考方案3】:我希望我已经找到了解决这个问题的决定。尝试像这样创建集群角色绑定
kubectl create clusterrolebinding gitlab-cluster-admin --clusterrole=cluster-admin --group=system:serviceaccounts
【讨论】:
不安全!!!这允许任何应用程序对您的集群具有完全访问权限,并且还授予对 Secrets 具有读取权限(或创建任何 pod)的任何用户对您的集群的完全访问权限。 @XPLOT1ON 那你有什么建议?我们并不都是 rbac 专家以上是关于由于禁止的秘密访问规则,无法在 gitlab-runner 中使用 helm upgrade的主要内容,如果未能解决你的问题,请参考以下文章
如何将访问密钥和秘密密钥 shell 变量传递给 Redshift 复制命令