이전 글에서 Terraform으로 AWS EKS에 Kubernetes 클러스터 프로비저닝하는 방법을 살펴봤는데 여기서는 kubectl로 클러스터에 접근하려면 EKS 클러스터를 만든 AWS 사용자만 가능했다. 개인 계정이라서 혼자 사용하긴 하지만 Terraform을 실행하는 AWS 계정을 따로 쓰고 있어서 로컬에서 사용하는 AWS 계정으로도 접근하기 위해 권한을 부여해야 했다.
aws-auth ConfigMap
찾아보니 EKS 클러스터에 권한을 부여하려면 aws-auth ConfigMap에 사용자나 역할을 지정해야 한다는 걸 알게 되었다. 일단 ConfigMap을 조회해 보자.
$ kubectl -n kube-system get cm
NAME DATA AGE
aws-auth 1 10d
coredns 1 10d
cp-vpc-resource-controller 0 10d
eks-certificates-controller 0 10d
extension-apiserver-authentication 6 10d
kube-proxy 1 10d
kube-proxy-config 1 10d
kube-root-ca.crt 1 10d
kube-system 네임스페이스에서 ConfigMap을 확인해 보면 aws-auth라는 ConfigMap이 있는 걸 확인할 수 있다.
$ kubectl -n kube-system get cm/aws-auth -o yaml
apiVersion: v1
data:
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::12345678:role/default_node_group-eks-node-group-202207261234600000004
username: system:node:{{EC2PrivateDNSName}}
kind: ConfigMap
metadata:
creationTimestamp: "2022-07-26T18:55:41Z"
name: aws-auth
namespace: kube-system
resourceVersion: "988"
uid: bc75e1be-4857-4927-89e4-0eb51d6822e4
이 ConfigMap의 내용을 살펴보면 mapRoles에 하나만 지정되어 있는데 정확히 뭔지는 모르지만 이름으로 봤을 때 노드 그룹에 대한 IAM 역할이 지정된 것으로 보인다. 지금 kubectl을 사용하고 있는 사용자는 여기에 지정되어 있지 않은데 AWS 문서를 보면 자동으로 system:masters 권한이 부여된다는 것을 알 수 있었다.
Amazon EKS 클러스터를 생성할 경우 클러스터를 생성하는 AWS Identity and Access Management(IAM) 엔터티 사용자 또는 역할(예: 페더레이션 사용자)에는 Amazon EKS 제어 영역의 클러스터 역할 기반 액세스 제어(RBAC) 구성에 system:masters 권한이 자동으로 부여됩니다.
Role/ClusterRole
위처럼 mapRoles IAM 역할을 지정하고 groups에 Kubernetes의 RoleBinding와 ClusterRoleBinding를 지정해서 권한을 부여할 수 있다. RoleBinding/ClusterRoleBinding를 살펴보기 전에 먼저 Role/ClusterRole을 조회해 보자.(좀 길지만 일단 결과를 다 적었다.)
$ kubectl get roles -A
NAMESPACE NAME CREATED AT
kube-public system:controller:bootstrap-signer 2022-07-26T18:52:02Z
kube-system eks-vpc-resource-controller-role 2022-07-26T18:52:13Z
kube-system eks:addon-manager 2022-07-26T18:52:10Z
kube-system eks:authenticator 2022-07-26T18:52:05Z
kube-system eks:certificate-controller 2022-07-26T18:52:05Z
kube-system eks:fargate-manager 2022-07-26T18:52:10Z
kube-system eks:k8s-metrics 2022-07-26T18:52:05Z
kube-system eks:node-manager 2022-07-26T18:52:09Z
kube-system extension-apiserver-authentication-reader 2022-07-26T18:52:02Z
kube-system system::leader-locking-kube-controller-manager 2022-07-26T18:52:02Z
kube-system system::leader-locking-kube-scheduler 2022-07-26T18:52:02Z
kube-system system:controller:bootstrap-signer 2022-07-26T18:52:02Z
kube-system system:controller:cloud-provider 2022-07-26T18:52:02Z
kube-system system:controller:token-cleaner 2022-07-26T18:52:02Z
kube-system vpc-resource-controller-leader-election-role 2022-07-26T18:52:13Z
Role은 네임스페이스 내에서 권한을 부여하므로 네임스페이스가 같이 출력된다.
$ kubectl get clusterroles
NAME CREATED AT
admin 2022-07-26T18:52:01Z
aws-node 2022-07-26T18:52:09Z
cluster-admin 2022-07-26T18:52:01Z
edit 2022-07-26T18:52:01Z
eks:addon-manager 2022-07-26T18:52:09Z
eks:certificate-controller-approver 2022-07-26T18:52:05Z
eks:certificate-controller-signer 2022-07-26T18:52:05Z
eks:cloud-controller-manager 2022-07-26T18:52:05Z
eks:cloud-provider-extraction-migration 2022-07-26T18:52:06Z
eks:cluster-event-watcher 2022-07-26T18:52:05Z
eks:fargate-manager 2022-07-26T18:52:10Z
eks:fargate-scheduler 2022-07-26T18:52:05Z
eks:k8s-metrics 2022-07-26T18:52:05Z
eks:node-bootstrapper 2022-07-26T18:52:10Z
eks:node-manager 2022-07-26T18:52:09Z
eks:nodewatcher 2022-07-26T18:52:05Z
eks:pod-identity-mutating-webhook 2022-07-26T18:52:05Z
eks:podsecuritypolicy:privileged 2022-07-26T18:52:10Z
eks:tagging-controller 2022-07-26T18:52:06Z
system:aggregate-to-admin 2022-07-26T18:52:01Z
system:aggregate-to-edit 2022-07-26T18:52:01Z
system:aggregate-to-view 2022-07-26T18:52:01Z
system:auth-delegator 2022-07-26T18:52:01Z
system:basic-user 2022-07-26T18:52:01Z
system:certificates.k8s.io:certificatesigningrequests:nodeclient 2022-07-26T18:52:01Z
system:certificates.k8s.io:certificatesigningrequests:selfnodeclient 2022-07-26T18:52:01Z
system:certificates.k8s.io:kube-apiserver-client-approver 2022-07-26T18:52:01Z
system:certificates.k8s.io:kube-apiserver-client-kubelet-approver 2022-07-26T18:52:01Z
system:certificates.k8s.io:kubelet-serving-approver 2022-07-26T18:52:01Z
system:certificates.k8s.io:legacy-unknown-approver 2022-07-26T18:52:01Z
system:controller:attachdetach-controller 2022-07-26T18:52:01Z
system:controller:certificate-controller 2022-07-26T18:52:01Z
system:controller:clusterrole-aggregation-controller 2022-07-26T18:52:01Z
system:controller:cronjob-controller 2022-07-26T18:52:01Z
system:controller:daemon-set-controller 2022-07-26T18:52:01Z
system:controller:deployment-controller 2022-07-26T18:52:01Z
system:controller:disruption-controller 2022-07-26T18:52:01Z
system:controller:endpoint-controller 2022-07-26T18:52:01Z
system:controller:endpointslice-controller 2022-07-26T18:52:01Z
system:controller:endpointslicemirroring-controller 2022-07-26T18:52:01Z
system:controller:ephemeral-volume-controller 2022-07-26T18:52:01Z
system:controller:expand-controller 2022-07-26T18:52:01Z
system:controller:generic-garbage-collector 2022-07-26T18:52:01Z
system:controller:horizontal-pod-autoscaler 2022-07-26T18:52:01Z
system:controller:job-controller 2022-07-26T18:52:01Z
system:controller:namespace-controller 2022-07-26T18:52:01Z
system:controller:node-controller 2022-07-26T18:52:01Z
system:controller:persistent-volume-binder 2022-07-26T18:52:01Z
system:controller:pod-garbage-collector 2022-07-26T18:52:01Z
system:controller:pv-protection-controller 2022-07-26T18:52:01Z
system:controller:pvc-protection-controller 2022-07-26T18:52:01Z
system:controller:replicaset-controller 2022-07-26T18:52:01Z
system:controller:replication-controller 2022-07-26T18:52:01Z
system:controller:resourcequota-controller 2022-07-26T18:52:01Z
system:controller:root-ca-cert-publisher 2022-07-26T18:52:01Z
system:controller:route-controller 2022-07-26T18:52:01Z
system:controller:service-account-controller 2022-07-26T18:52:01Z
system:controller:service-controller 2022-07-26T18:52:01Z
system:controller:statefulset-controller 2022-07-26T18:52:01Z
system:controller:ttl-after-finished-controller 2022-07-26T18:52:01Z
system:controller:ttl-controller 2022-07-26T18:52:01Z
system:coredns 2022-07-26T18:52:09Z
system:discovery 2022-07-26T18:52:01Z
system:heapster 2022-07-26T18:52:01Z
system:kube-aggregator 2022-07-26T18:52:01Z
system:kube-controller-manager 2022-07-26T18:52:01Z
system:kube-dns 2022-07-26T18:52:01Z
system:kube-scheduler 2022-07-26T18:52:01Z
system:kubelet-api-admin 2022-07-26T18:52:01Z
system:monitoring 2022-07-26T18:52:01Z
system:node 2022-07-26T18:52:01Z
system:node-bootstrapper 2022-07-26T18:52:01Z
system:node-problem-detector 2022-07-26T18:52:01Z
system:node-proxier 2022-07-26T18:52:01Z
system:persistent-volume-provisioner 2022-07-26T18:52:01Z
system:public-info-viewer 2022-07-26T18:52:01Z
system:service-account-issuer-discovery 2022-07-26T18:52:01Z
system:volume-scheduler 2022-07-26T18:52:01Z
view 2022-07-26T18:52:01Z
vpc-resource-controller-role 2022-07-26T18:52:13Z
ClusterRole은 네임스페이스에 국한된 역할이고 앞에 system: 접두사가 붙은 것은 클러스터 컨트롤 플레인에서 직접 제어하는 Role을 의미한다.
system:masters, system:bootstrappers, system:nodes
하지만 앞에서 보았던 system:masters, system:bootstrappers, system:nodes은 찾을 수가 없었다. 여기가 아닌가 싶었지만 Kubernetes 소스코드에 하드코딩된 well-known 그룹이었다.
// well-known user and group names
const (
SystemPrivilegedGroup = "system:masters"
NodesGroup = "system:nodes"
MonitoringGroup = "system:monitoring"
AllUnauthenticated = "system:unauthenticated"
AllAuthenticated = "system:authenticated"
Anonymous = "system:anonymous"
APIServerUser = "system:apiserver"
// core kubernetes process identities
KubeProxy = "system:kube-proxy"
KubeControllerManager = "system:kube-controller-manager"
KubeScheduler = "system:kube-scheduler"
)
Role/ClusterRole의 권한
다시 Role/ClusterRole로 돌아가 보자. 여기서는 클러스터를 관리할 사용자를 추가할 예정이므로 ClusterRole만 봐도 된다. 이미 만들어 진 것 중에 사용할 수 있을 걸로 보이는 건 admin, cluster-admin 등이 있었다.
이 권한을 조회해 보자.
$ kubectl describe clusterrole admin
Name: admin
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
rolebindings.rbac.authorization.k8s.io [] [] [create delete deletecollection get list patch update watch]
roles.rbac.authorization.k8s.io [] [] [create delete deletecollection get list patch update watch]
configmaps [] [] [create delete deletecollection patch update get list watch]
events [] [] [create delete deletecollection patch update get list watch]
persistentvolumeclaims [] [] [create delete deletecollection patch update get list watch]
pods [] [] [create delete deletecollection patch update get list watch]
replicationcontrollers/scale [] [] [create delete deletecollection patch update get list watch]
replicationcontrollers [] [] [create delete deletecollection patch update get list watch]
services [] [] [create delete deletecollection patch update get list watch]
daemonsets.apps [] [] [create delete deletecollection patch update get list watch]
deployments.apps/scale [] [] [create delete deletecollection patch update get list watch]
deployments.apps [] [] [create delete deletecollection patch update get list watch]
replicasets.apps/scale [] [] [create delete deletecollection patch update get list watch]
replicasets.apps [] [] [create delete deletecollection patch update get list watch]
statefulsets.apps/scale [] [] [create delete deletecollection patch update get list watch]
statefulsets.apps [] [] [create delete deletecollection patch update get list watch]
horizontalpodautoscalers.autoscaling [] [] [create delete deletecollection patch update get list watch]
cronjobs.batch [] [] [create delete deletecollection patch update get list watch]
jobs.batch [] [] [create delete deletecollection patch update get list watch]
daemonsets.extensions [] [] [create delete deletecollection patch update get list watch]
deployments.extensions/scale [] [] [create delete deletecollection patch update get list watch]
deployments.extensions [] [] [create delete deletecollection patch update get list watch]
ingresses.extensions [] [] [create delete deletecollection patch update get list watch]
networkpolicies.extensions [] [] [create delete deletecollection patch update get list watch]
replicasets.extensions/scale [] [] [create delete deletecollection patch update get list watch]
replicasets.extensions [] [] [create delete deletecollection patch update get list watch]
replicationcontrollers.extensions/scale [] [] [create delete deletecollection patch update get list watch]
ingresses.networking.k8s.io [] [] [create delete deletecollection patch update get list watch]
networkpolicies.networking.k8s.io [] [] [create delete deletecollection patch update get list watch]
poddisruptionbudgets.policy [] [] [create delete deletecollection patch update get list watch]
deployments.apps/rollback [] [] [create delete deletecollection patch update]
deployments.extensions/rollback [] [] [create delete deletecollection patch update]
localsubjectaccessreviews.authorization.k8s.io [] [] [create]
pods/attach [] [] [get list watch create delete deletecollection patch update]
pods/exec [] [] [get list watch create delete deletecollection patch update]
pods/portforward [] [] [get list watch create delete deletecollection patch update]
pods/proxy [] [] [get list watch create delete deletecollection patch update]
secrets [] [] [get list watch create delete deletecollection patch update]
services/proxy [] [] [get list watch create delete deletecollection patch update]
bindings [] [] [get list watch]
endpoints [] [] [get list watch]
limitranges [] [] [get list watch]
namespaces/status [] [] [get list watch]
namespaces [] [] [get list watch]
persistentvolumeclaims/status [] [] [get list watch]
pods/log [] [] [get list watch]
pods/status [] [] [get list watch]
replicationcontrollers/status [] [] [get list watch]
resourcequotas/status [] [] [get list watch]
resourcequotas [] [] [get list watch]
services/status [] [] [get list watch]
controllerrevisions.apps [] [] [get list watch]
daemonsets.apps/status [] [] [get list watch]
deployments.apps/status [] [] [get list watch]
replicasets.apps/status [] [] [get list watch]
statefulsets.apps/status [] [] [get list watch]
horizontalpodautoscalers.autoscaling/status [] [] [get list watch]
cronjobs.batch/status [] [] [get list watch]
jobs.batch/status [] [] [get list watch]
endpointslices.discovery.k8s.io [] [] [get list watch]
daemonsets.extensions/status [] [] [get list watch]
deployments.extensions/status [] [] [get list watch]
ingresses.extensions/status [] [] [get list watch]
replicasets.extensions/status [] [] [get list watch]
ingresses.networking.k8s.io/status [] [] [get list watch]
poddisruptionbudgets.policy/status [] [] [get list watch]
serviceaccounts [] [] [impersonate create delete deletecollection patch update get list watch]
접근할 수 있는 리소스 목록이 있고 리소스에 따라서 Verbs로 읽기만 가능한지 쓰기, 삭제도 가능한지가 나열되어 있다.
$ kubectl describe clusterrole cluster-admin
Name: cluster-admin
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
PolicyRule:
Resources Non-Resource URLs Resource Names Verbs
--------- ----------------- -------------- -----
*.* [] [] [*]
[*] [] [*]
cluster-admin ClusterRole의 권한을 보면 와일드카드로 표시된 것을 알 수 있다. 필요한 권한 그룹을 만들어서 쓰면 되겠지만 일단 RBAC을 파악하는 게 이 글의 목적은 아니므로 넘어가자.
RoleBinding/ClusterRoleBinding
IAM 사용자에서는 RoleBinding이나 ClusterRoleBinding를 써야 하므로 이 둘을 살펴봐야 하는데 여기서는 클러스터 권한을 지정할 것이므로 ClusterRoleBinding만 살펴보자.(RoleBinding은 kubectl get rolebindings -A로 조회해 볼 수 있다.)
$ kubectl get clusterrolebindings
NAME ROLE AGE
aws-node ClusterRole/aws-node 10d
cluster-admin ClusterRole/cluster-admin 10d
eks:addon-manager ClusterRole/eks:addon-manager 10d
eks:certificate-controller ClusterRole/system:controller:certificate-controller 10d
eks:certificate-controller-approver ClusterRole/eks:certificate-controller-approver 10d
eks:certificate-controller-signer ClusterRole/eks:certificate-controller-signer 10d
eks:cloud-controller-manager ClusterRole/eks:cloud-controller-manager 10d
eks:cloud-provider-extraction-migration ClusterRole/eks:cloud-provider-extraction-migration 10d
eks:cluster-event-watcher ClusterRole/eks:cluster-event-watcher 10d
eks:fargate-manager ClusterRole/eks:fargate-manager 10d
eks:fargate-scheduler ClusterRole/eks:fargate-scheduler 10d
eks:k8s-metrics ClusterRole/eks:k8s-metrics 10d
eks:kube-proxy ClusterRole/system:node-proxier 10d
eks:kube-proxy-fargate ClusterRole/system:node-proxier 10d
eks:kube-proxy-windows ClusterRole/system:node-proxier 10d
eks:node-bootstrapper ClusterRole/eks:node-bootstrapper 10d
eks:node-manager ClusterRole/eks:node-manager 10d
eks:nodewatcher ClusterRole/eks:nodewatcher 10d
eks:pod-identity-mutating-webhook ClusterRole/eks:pod-identity-mutating-webhook 10d
eks:podsecuritypolicy:authenticated ClusterRole/eks:podsecuritypolicy:privileged 10d
eks:tagging-controller ClusterRole/eks:tagging-controller 10d
system:basic-user ClusterRole/system:basic-user 10d
system:controller:attachdetach-controller ClusterRole/system:controller:attachdetach-controller 10d
system:controller:certificate-controller ClusterRole/system:controller:certificate-controller 10d
system:controller:clusterrole-aggregation-controller ClusterRole/system:controller:clusterrole-aggregation-controller 10d
system:controller:cronjob-controller ClusterRole/system:controller:cronjob-controller 10d
system:controller:daemon-set-controller ClusterRole/system:controller:daemon-set-controller 10d
system:controller:deployment-controller ClusterRole/system:controller:deployment-controller 10d
system:controller:disruption-controller ClusterRole/system:controller:disruption-controller 10d
system:controller:endpoint-controller ClusterRole/system:controller:endpoint-controller 10d
system:controller:endpointslice-controller ClusterRole/system:controller:endpointslice-controller 10d
system:controller:endpointslicemirroring-controller ClusterRole/system:controller:endpointslicemirroring-controller 10d
system:controller:ephemeral-volume-controller ClusterRole/system:controller:ephemeral-volume-controller 10d
system:controller:expand-controller ClusterRole/system:controller:expand-controller 10d
system:controller:generic-garbage-collector ClusterRole/system:controller:generic-garbage-collector 10d
system:controller:horizontal-pod-autoscaler ClusterRole/system:controller:horizontal-pod-autoscaler 10d
system:controller:job-controller ClusterRole/system:controller:job-controller 10d
system:controller:namespace-controller ClusterRole/system:controller:namespace-controller 10d
system:controller:node-controller ClusterRole/system:controller:node-controller 10d
system:controller:persistent-volume-binder ClusterRole/system:controller:persistent-volume-binder 10d
system:controller:pod-garbage-collector ClusterRole/system:controller:pod-garbage-collector 10d
system:controller:pv-protection-controller ClusterRole/system:controller:pv-protection-controller 10d
system:controller:pvc-protection-controller ClusterRole/system:controller:pvc-protection-controller 10d
system:controller:replicaset-controller ClusterRole/system:controller:replicaset-controller 10d
system:controller:replication-controller ClusterRole/system:controller:replication-controller 10d
system:controller:resourcequota-controller ClusterRole/system:controller:resourcequota-controller 10d
system:controller:root-ca-cert-publisher ClusterRole/system:controller:root-ca-cert-publisher 10d
system:controller:route-controller ClusterRole/system:controller:route-controller 10d
system:controller:service-account-controller ClusterRole/system:controller:service-account-controller 10d
system:controller:service-controller ClusterRole/system:controller:service-controller 10d
system:controller:statefulset-controller ClusterRole/system:controller:statefulset-controller 10d
system:controller:ttl-after-finished-controller ClusterRole/system:controller:ttl-after-finished-controller 10d
system:controller:ttl-controller ClusterRole/system:controller:ttl-controller 10d
system:coredns ClusterRole/system:coredns 10d
system:discovery ClusterRole/system:discovery 10d
system:kube-controller-manager ClusterRole/system:kube-controller-manager 10d
system:kube-dns ClusterRole/system:kube-dns 10d
system:kube-scheduler ClusterRole/system:kube-scheduler 10d
system:monitoring ClusterRole/system:monitoring 10d
system:node ClusterRole/system:node 10d
system:node-proxier ClusterRole/system:node-proxier 10d
system:public-info-viewer ClusterRole/system:public-info-viewer 10d
system:service-account-issuer-discovery ClusterRole/system:service-account-issuer-discovery 10d
system:volume-scheduler ClusterRole/system:volume-scheduler 10d
vpc-resource-controller-rolebinding ClusterRole/vpc-resource-controller-role 10d
ClusterRoleBinding의 이름이 있고 바인딩 된 ClusterRole이 있다. 다행히 ClusterRole/cluster-admin에 바인딩 된 cluster-admin도 있는걸 볼 수 있다.
$ kubectl describe clusterrolebinding cluster-admin
Name: cluster-admin
Labels: kubernetes.io/bootstrapping=rbac-defaults
Annotations: rbac.authorization.kubernetes.io/autoupdate: true
Role:
Kind: ClusterRole
Name: cluster-admin
Subjects:
Kind Name Namespace
---- ---- ---------
Group system:masters
내용을 조회하니 여기도 system:master 그룹이 표시되어 있는지 모르겠지만 정확한 의미를 RBAC 공부하면서 봐야겠다.
IAM 사용자 추가
이제 약간의 지식을 더 얻었으니 사용자를 추가해보자. aws-auth ConfigMap을 수정해보자.
$ kubectl edit configmap aws-auth --namespace kube-system
configmap/aws-auth edited
직접 붙어서 수정했는데 아래처럼 mapUsers 부분을 추가해서 IAM 사용자를 연결했고 앞에서 본 system:masters를 그룹으로 지정했다. 사용자는 mapUsers에 지정하고 역할은 mapRoles에 지정하면 된다.
apiVersion: v1
data:
mapRoles: |
- groups:
- system:bootstrappers
- system:nodes
rolearn: arn:aws:iam::1234567890:role/default_node_group-eks-node-group-20220726184621476500000004
username: system:node:{{EC2PrivateDNSName}}
mapUsers: |
- groups:
- system:masters
userarn: arn:aws:iam::1234567890:user/outsider
username: outsider
kind: ConfigMap
metadata:
creationTimestamp: "2022-07-26T18:55:41Z"
name: aws-auth
namespace: kube-system
resourceVersion: "2569874"
uid: bc75e1be-4857-4927-89e4-0eb51d6822e
이제 로컬의 kubeconfig에서 AWS의 사용자를 클러스터를 생성한 사용자가 아닌 위에서 지정한 outsider 사용자로 지정해서 kubectl을 사용하면 전처럼 Unauthorized 오류가 나지 않고 잘 접근되는 것을 알 수 있다.
$ kubectl get ns
NAME STATUS AGE
default Active 10d
kube-node-lease Active 10d
kube-public Active 10d
kube-system Active 10d
기분으로는 그룹에 cluster-admin을 지정해도 되었는데 cluster-admin만 지정하면 Unauthorized는 안 나지만 권한이 없다고 Error from server (Forbidden): pods is forbidden: User "outsider" cannot list resource "pods" in API group "" in the namespace "default"같은 오류가 발생한다.
RBAC은 차차 공부하기로 하고 일단 사용자를 추가하는 방법은 알게 되었다. 이걸 Terraform으로 관리하고 싶은데 방법을 더 찾아봐야겠다.
Comments