Introduction

In this article we will explore how RBAC (Role-Based Access Control) works in Kubernetes, not just from the kubectl perspective, but by diving deep into the actual HTTP API calls that happen behind the scenes. We’ll see what kubectl is really doing when you create roles, role bindings, and check permissions.

This post aims to demistify the Kubernetes API and also give you a better understanding of how to interact with it by using RBAC. This not only will give us a very good understanding of how Kubernetes APIs work, but it will also open the possibility to build any tool or process that you need using the Kubernetes APIs.

If you read my previous article on Kubernetes authentication and authorization, you know that authentication is about proving who you are, while authorization is about what you’re allowed to do. RBAC is Kubernetes’ primary authorization mechanism, and understanding how it works at the API level will make you much more effective at debugging permission issues.

RBAC Basics:

RBAC in Kubernetes consists of three main components:

• Roles/ClusterRoles: Define what actions can be performed on which resources
• Subjects: Users, groups, or service accounts that need permissions
• RoleBindings/ClusterRoleBindings: Connect roles to subjects

The key difference between Role/RoleBinding and ClusterRole/ClusterRoleBinding is scope:

• Role/RoleBinding: Namespaced (permissions within a specific namespace)
• ClusterRole/ClusterRoleBinding: Cluster-wide (permissions across all namespaces or cluster-scoped resources)

So we are going to create some RBAC resources, test them with kubectl, and then see exactly what HTTP requests are being made to the Kubernetes API server. This will give you a much deeper understanding of how RBAC actually works.

Let’s get to it

Let’s start by setting up our testing environment. I’ll create a namespace, some roles, and users, then show you both the kubectl commands and the equivalent curl calls, either use kubectl or curl as both will be equivalent.

Setup our environment:

First, we need to find out the public ip of our API server:

kubectl get ep -A
NAMESPACE     NAME         ENDPOINTS                                               AGE
default       kubernetes   172.19.0.2:6443                                         78m
kube-system   kube-dns     10.244.0.3:53,10.244.0.4:53,10.244.0.3:53 + 3 more...   78m

if you instead use kubectl proxy, that will use your current credentials and will sort-of bypass the authentication when using curl, so stick to hitting the URL directly for all the examples to work as expected.

First, let’s create a namespace for our experiments:

kubectl create namespace rbac-demo

Now let’s see what this actually does at the API level. Enable kubectl verbose mode to see the HTTP calls:

kubectl create namespace rbac-demo -v=8

You’ll see output like this (truncated for readability):

I0110 10:30:15.123456 POST https://127.0.0.1:6443/api/v1/namespaces
I0110 10:30:15.123456 Request Body: {"apiVersion":"v1","kind":"Namespace","metadata":{"name":"rbac-demo"}}
I0110 10:30:15.123456 Response Status: 201 Created

Now let’s do the same thing with curl to understand the raw API call:

# Get your cluster info
kubectl cluster-info

# Get your token (this will vary based on your setup)
TOKEN=$(kubectl get secret $(kubectl get serviceaccount default -o jsonpath='{.secrets[0].name}') -o jsonpath='{.data.token}' | base64 -d)

# Make the API call
curl -k -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -X POST \
  https://172.19.0.2:6443/api/v1/namespaces \
  -d '{
    "apiVersion": "v1",
    "kind": "Namespace", 
    "metadata": {
      "name": "rbac-demo-curl"
    }
  }'

The response will be a JSON representation of the created namespace. This is exactly what kubectl does behind the scenes!

Creating RBAC Resources

Now let’s create a Role that allows reading pods in our namespace:

kubectl create role pod-reader \
  --namespace=rbac-demo \
  --verb=get,list,watch \
  --resource=pods

Let’s see the actual YAML that was created:

kubectl get role pod-reader -n rbac-demo -o yaml

The output should look like this:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: rbac-demo
  resourceVersion: "12345"
rules:
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
  - list
  - watch

Now let’s make the same call with curl to see the raw HTTP request:

# First, let's see what kubectl would send
kubectl create role pod-reader-curl \
  --namespace=rbac-demo \
  --verb=get,list,watch \
  --resource=pods \
  --dry-run=client -o json

This shows us the exact JSON payload. Now let’s send it via curl:

curl -k -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -X POST \
  https://172.19.0.2:6443/apis/rbac.authorization.k8s.io/v1/namespaces/rbac-demo/roles \
  -d '{
    "apiVersion": "rbac.authorization.k8s.io/v1",
    "kind": "Role",
    "metadata": {
      "name": "pod-reader-curl",
      "namespace": "rbac-demo"
    },
    "rules": [
      {
        "apiGroups": [""],
        "resources": ["pods"],
        "verbs": ["get", "list", "watch"]
      }
    ]
  }'

Notice the API path: /apis/rbac.authorization.k8s.io/v1/namespaces/rbac-demo/roles. This tells us:

• We’re using the RBAC API group (rbac.authorization.k8s.io)
• Version v1
• It’s namespaced (includes /namespaces/rbac-demo)
• We’re working with roles

Creating a RoleBinding

Now let’s create a RoleBinding to give a user the pod-reader role:

kubectl create rolebinding pod-reader-binding \
  --namespace=rbac-demo \
  --role=pod-reader \
  --user=john.doe@example.com

NOTE: creating users can be a bit tricky as we need a certificate and so on, you can read more here, there are plenty of examples in how to create one manually out there, this is also a bit different with cloud offerings, but the Kubernetes mechanics are basically the same.

Let’s inspect what was created:

kubectl get rolebinding pod-reader-binding -n rbac-demo -o yaml

The output shows:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-reader-binding
  namespace: rbac-demo
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: pod-reader
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: john.doe@example.com

Now the curl equivalent:

curl -k -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -X POST \
  https://172.19.0.2:6443/apis/rbac.authorization.k8s.io/v1/namespaces/rbac-demo/rolebindings \
  -d '{
    "apiVersion": "rbac.authorization.k8s.io/v1",
    "kind": "RoleBinding",
    "metadata": {
      "name": "pod-reader-binding-curl",
      "namespace": "rbac-demo"
    },
    "roleRef": {
      "apiGroup": "rbac.authorization.k8s.io",
      "kind": "Role",
      "name": "pod-reader"
    },
    "subjects": [
      {
        "apiGroup": "rbac.authorization.k8s.io",
        "kind": "User",
        "name": "[email protected]"
      }
    ]
  }'

Checking Permissions

Now let’s test permissions. Kubernetes provides a handy API for this - the SubjectAccessReview:

kubectl auth can-i get pods --namespace=rbac-demo --as=john.doe@example.com

This should return yes since we just gave [email protected] the pod-reader role. But what’s happening behind the scenes? Let’s use curl to make the same check:

curl -k -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -X POST \
  https://172.19.0.2:6443/apis/authorization.k8s.io/v1/subjectaccessreviews \
  -d '{
    "apiVersion": "authorization.k8s.io/v1",
    "kind": "SubjectAccessReview",
    "spec": {
      "resourceAttributes": {
        "namespace": "rbac-demo",
        "verb": "get",
        "resource": "pods"
      },
      "user": "[email protected]"
    }
  }'

The response will look like:

{
  "apiVersion": "authorization.k8s.io/v1",
  "kind": "SubjectAccessReview",
  "metadata": {
  },
  "spec": {
    "resourceAttributes": {
      "namespace": "rbac-demo",
      "resource": "pods",
      "verb": "get"
    },
    "user": "[email protected]"
  },
  "status": {
    "allowed": true,
    "reason": "RBAC: allowed by RoleBinding \"pod-reader-binding/rbac-demo\" of Role \"pod-reader\" to User \"[email protected]\""
  }
}

This is incredibly useful! The response not only tells us if the action is allowed ("allowed": true) but also explains exactly why (reason field).

Let’s test a permission that should be denied:

curl -k -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -X POST \
  https://172.19.0.2:6443/apis/authorization.k8s.io/v1/subjectaccessreviews \
  -d '{
    "apiVersion": "authorization.k8s.io/v1",
    "kind": "SubjectAccessReview",
    "spec": {
      "resourceAttributes": {
        "namespace": "rbac-demo",
        "verb": "delete",
        "resource": "pods"
      },
      "user": "[email protected]"
    }
  }'

The response will show:

{
  "status": {
    "allowed": false,
    "reason": "RBAC: access denied"
  }
}

ClusterRole and ClusterRoleBinding Example

Let’s create a ClusterRole that can read nodes (a cluster-scoped resource):

kubectl create clusterrole node-reader --verb=get,list,watch --resource=nodes

The curl equivalent (notice no namespace in the URL since it’s cluster-scoped):

curl -k -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -X POST \
  https://172.19.0.2:6443/apis/rbac.authorization.k8s.io/v1/clusterroles \
  -d '{
    "apiVersion": "rbac.authorization.k8s.io/v1",
    "kind": "ClusterRole",
    "metadata": {
      "name": "node-reader-curl"
    },
    "rules": [
      {
        "apiGroups": [""],
        "resources": ["nodes"],
        "verbs": ["get", "list", "watch"]
      }
    ]
  }'

Now bind it to a user:

curl -k -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -X POST \
  https://172.19.0.2:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings \
  -d '{
    "apiVersion": "rbac.authorization.k8s.io/v1",
    "kind": "ClusterRoleBinding",
    "metadata": {
      "name": "node-reader-binding-curl"
    },
    "roleRef": {
      "apiGroup": "rbac.authorization.k8s.io",
      "kind": "ClusterRole",
      "name": "node-reader-curl"
    },
    "subjects": [
      {
        "apiGroup": "rbac.authorization.k8s.io",
        "kind": "User",
        "name": "[email protected]"
      }
    ]
  }'

Debugging RBAC Issues

One of the most powerful features for debugging RBAC is the ability to check permissions for any user. Let’s create a comprehensive script to audit permissions:

#!/bin/bash
# rbac-check.sh

USER=$1
NAMESPACE=${2:-"default"}

if [ -z "$USER" ]; then
  echo "Usage: $0 <user> [namespace]"
  exit 1
fi

echo "Checking permissions for user: $USER in namespace: $NAMESPACE"
echo "=================================================="

# Common resources to check
RESOURCES=("pods" "services" "deployments" "configmaps" "secrets")
VERBS=("get" "list" "watch" "create" "update" "patch" "delete")

for resource in "${RESOURCES[@]}"; do
  echo "Resource: $resource"
  for verb in "${VERBS[@]}"; do
    result=$(kubectl auth can-i $verb $resource --namespace=$NAMESPACE --as=$USER)
    printf "  %-8s: %s\n" "$verb" "$result"
  done
  echo ""
done

Advanced RBAC Features

Let’s explore some advanced RBAC features using both kubectl and curl:

Resource Names: You can restrict access to specific named resources:

curl -k -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -X POST \
  https://172.19.0.2:6443/apis/rbac.authorization.k8s.io/v1/namespaces/rbac-demo/roles \
  -d '{
    "apiVersion": "rbac.authorization.k8s.io/v1",
    "kind": "Role",
    "metadata": {
      "name": "specific-pod-reader",
      "namespace": "rbac-demo"
    },
    "rules": [
      {
        "apiGroups": [""],
        "resources": ["pods"],
        "resourceNames": ["my-specific-pod"],
        "verbs": ["get", "list", "watch"]
      }
    ]
  }'

API Groups: Different resources belong to different API groups:

# Core API group (empty string) - pods, services, etc.
# apps API group - deployments, replicasets, etc.
# extensions API group - ingresses, etc.

curl -k -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -X POST \
  https://172.19.0.2:6443/apis/rbac.authorization.k8s.io/v1/namespaces/rbac-demo/roles \
  -d '{
    "apiVersion": "rbac.authorization.k8s.io/v1",
    "kind": "Role",
    "metadata": {
      "name": "deployment-manager",
      "namespace": "rbac-demo"
    },
    "rules": [
      {
        "apiGroups": ["apps"],
        "resources": ["deployments"],
        "verbs": ["get", "list", "watch", "create", "update", "patch", "delete"]
      }
    ]
  }'

Inspecting Existing RBAC

To understand what permissions exist in your cluster, you can query the API directly:

# List all roles in a namespace
curl -k -H "Authorization: Bearer $TOKEN" \
  https://172.19.0.2:6443/apis/rbac.authorization.k8s.io/v1/namespaces/rbac-demo/roles

# List all rolebindings in a namespace
curl -k -H "Authorization: Bearer $TOKEN" \
  https://172.19.0.2:6443/apis/rbac.authorization.k8s.io/v1/namespaces/rbac-demo/rolebindings

# List all clusterroles
curl -k -H "Authorization: Bearer $TOKEN" \
  https://172.19.0.2:6443/apis/rbac.authorization.k8s.io/v1/clusterroles

# List all clusterrolebindings
curl -k -H "Authorization: Bearer $TOKEN" \
  https://172.19.0.2:6443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings

You can also use jq to filter and format the output:

# Get all rolebindings and show which users have which roles
curl -s -k -H "Authorization: Bearer $TOKEN" \
  https://172.19.0.2:6443/apis/rbac.authorization.k8s.io/v1/namespaces/rbac-demo/rolebindings | \
  jq -r '.items[] | "\(.metadata.name): \(.subjects[]?.name) -> \(.roleRef.name)"'

Testing with a Real User

Let’s create a service account and test our RBAC rules:

kubectl create serviceaccount test-user -n rbac-demo

# Bind our pod-reader role to this service account
kubectl create rolebinding test-user-binding \
  --namespace=rbac-demo \
  --role=pod-reader \
  --serviceaccount=rbac-demo:test-user

# Special secret to generate a token for the service account
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: test-user-sa-secret
  namespace: rbac-demo
  annotations:
    kubernetes.io/service-account.name: test-user
type: kubernetes.io/service-account-token
EOF

Now let’s get the service account token and test permissions:

# Get the service account token
SA_TOKEN=$(kubectl get secret test-user-sa-secret -n rbac-demo -o jsonpath='{.data.token}' | base64 -d)

# Test if we can list pods using the service account token
curl -k -H "Authorization: Bearer $SA_TOKEN" \
  https://172.19.0.2:6443/api/v1/namespaces/rbac-demo/pods

This should work since we gave the service account the pod-reader role. Now let’s try something it shouldn’t be able to do:

# Try to create a pod (should fail)
curl -k -H "Authorization: Bearer $SA_TOKEN" \
  -H "Content-Type: application/json" \
  -X POST \
  https://172.19.0.2:6443/api/v1/namespaces/rbac-demo/pods \
  -d '{
    "apiVersion": "v1",
    "kind": "Pod",
    "metadata": {
      "name": "test-pod"
    },
    "spec": {
      "containers": [
        {
          "name": "test",
          "image": "nginx"
        }
      ]
    }
  }'

This should return a 403 Forbidden error with a message like:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "pods is forbidden: User \"system:serviceaccount:rbac-demo:test-user\" cannot create resource \"pods\" in API group \"\" in the namespace \"rbac-demo\"",
  "reason": "Forbidden",
  "details": {
    "kind": "pods"
  },
  "code": 403
}

Perfect! The RBAC is working as expected.

Common RBAC Patterns

Here are some common RBAC patterns you’ll encounter:

Read-only access to everything in a namespace:

curl -k -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -X POST \
  https://172.19.0.2:6443/apis/rbac.authorization.k8s.io/v1/namespaces/rbac-demo/roles \
  -d '{
    "apiVersion": "rbac.authorization.k8s.io/v1",
    "kind": "Role",
    "metadata": {
      "name": "namespace-reader",
      "namespace": "rbac-demo"
    },
    "rules": [
      {
        "apiGroups": ["*"],
        "resources": ["*"],
        "verbs": ["get", "list", "watch"]
      }
    ]
  }'

Access to create and manage deployments:

curl -k -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -X POST \
  https://172.19.0.2:6443/apis/rbac.authorization.k8s.io/v1/namespaces/rbac-demo/roles \
  -d '{
    "apiVersion": "rbac.authorization.k8s.io/v1",
    "kind": "Role",
    "metadata": {
      "name": "deployment-manager",
      "namespace": "rbac-demo"
    },
    "rules": [
      {
        "apiGroups": ["apps"],
        "resources": ["deployments"],
        "verbs": ["*"]
      },
      {
        "apiGroups": [""],
        "resources": ["pods"],
        "verbs": ["get", "list", "watch"]
      }
    ]
  }'

Troubleshooting RBAC

When RBAC isn’t working as expected, here’s a systematic approach to debug:

1. Check if the user/service account exists:

   kubectl get serviceaccount test-user -n rbac-demo

2. Check what roles are bound to the user:

   kubectl get rolebindings -n rbac-demo -o wide
   kubectl get clusterrolebindings -o wide

3. Use SubjectAccessReview to test specific permissions:

   kubectl auth can-i create pods --namespace=rbac-demo --as=system:serviceaccount:rbac-demo:test-user

4. Check the exact error message from the API: The error messages are usually very specific about what’s missing.

5. Verify the role rules:

   kubectl describe role pod-reader -n rbac-demo

Clean up

Always remember to clean up your testing resources:

kubectl delete namespace rbac-demo

Or with curl:

curl -k -H "Authorization: Bearer $TOKEN" \
  -X DELETE \
  https://172.19.0.2:6443/api/v1/namespaces/rbac-demo

Errata

If you spot any error or have any suggestion, please send me a message so it gets fixed.

You can read more about RBAC here and about the Kubernetes API here.