Introduction

In this article we will explore how to debug distroless containers in Kubernetes when your application container has no shell, no package manager, and basically no debugging tools whatsoever. If you’ve ever tried to kubectl exec into a distroless container only to be greeted with “executable file not found in $PATH: /bin/sh”, you know the pain.

Distroless containers are amazing for security and size - they contain only your application and its runtime dependencies, nothing else. No shell, no package managers, no debugging tools. They’re perfect for production… until something goes wrong and you need to poke around inside.

But fear not! Kubernetes has a solution: ephemeral containers via kubectl debug. We’ll not only see how to use this feature, but also how to manually set up a user environment and access the main container’s filesystem through the /proc/1/root trick.

What are Distroless Containers?

Before we dive into debugging, let’s quickly understand what we’re dealing with. Distroless containers, popularized by Google, contain only:

• Your application binary
• Runtime dependencies (libraries, certificates, timezone data)
• A minimal user setup (usually just root or a dedicated user)

What they DON’T contain:

• Package managers (apt, yum, apk)
• Shells (bash, sh, zsh)
• Debugging tools (ps, netstat, curl, wget)
• Text editors (vi, nano)
• Pretty much anything that makes debugging easy

This is fantastic for security (smaller attack surface) and performance (smaller images), but terrible when you need to debug a running container.

The Problem

Let’s say you have a Go application running in a distroless container and it’s behaving strangely. Your natural instinct is:

kubectl exec -it my-pod -- /bin/sh

But you’re greeted with:

OCI runtime exec failed: exec failed: unable to start container process: 
exec: "/bin/sh": executable file not found in $PATH: unknown

Even trying different shells won’t help:

kubectl exec -it my-pod -- bash
kubectl exec -it my-pod -- /bin/bash
# Same error, different shell

So what now? This is where kubectl debug comes to the rescue.

Enter kubectl debug

Kubernetes 1.18+ introduced ephemeral containers, and kubectl debug makes them easy to use. Think of it as attaching a debugging sidecar to your running pod temporarily.

Here’s the basic syntax:

kubectl debug -it my-pod --image=ubuntu --target=my-container

This command:

• Creates an ephemeral container using the ubuntu image
• Attaches to it interactively (-it)
• Shares the process namespace with the target container

But there’s a catch - even with this setup, you still can’t directly access your application’s filesystem. That’s where the /proc/1/root magic comes in.

The /proc/1/root Trick

In Linux, /proc/1/root is a symbolic link to the root filesystem of process ID 1. When containers share a process namespace (which kubectl debug does by default), you can access the main container’s filesystem through this path.

Here’s the full debugging workflow:

Step 1: Create the Debug Container

kubectl debug -it my-pod --image=ubuntu --target=my-container --share-processes

You’ll be dropped into a shell in the Ubuntu container. The --share-processes flag ensures you can see all processes from both containers.

Step 2: Verify Process Sharing

ps aux

You should see both your application process (PID 1) and the shell processes from the debug container. If your app is running as PID 1, you’re good to go.

Step 3: Access the Main Container’s Filesystem

ls /proc/1/root/

This shows you the filesystem of your distroless container! You can now navigate and inspect files:

# Check your application binary
ls -la /proc/1/root/app

# Look at configuration files
cat /proc/1/root/etc/ssl/certs/ca-certificates.crt

# Check environment variables
cat /proc/1/environ | tr '\0' '\n'

# Examine the working directory
ls -la /proc/1/root/app/

Step 4: Creating a Proper User Environment

Sometimes you might want to work more comfortably. Here’s how to set up a proper user environment in your debug container:

# Update package list and install useful tools
apt update && apt install -y curl wget netstat-nat procps tree

# Create a user (optional, but good practice)
useradd -m -s /bin/bash debuguser
usermod -aG sudo debuguser

# Switch to the new user
su - debuguser

Now you have a full debugging environment with all the tools you need, while still being able to access your distroless container’s filesystem.

Advanced Debugging Techniques

Once you have access, here are some powerful debugging techniques:

Network Debugging:

# Check what your app is listening on
netstat -tlnp

# Test connectivity from the debug container
curl http://localhost:8080/health

# Check DNS resolution
nslookup your-service

File System Investigation:

# Check disk usage
du -sh /proc/1/root/*

# Find recently modified files
find /proc/1/root/ -type f -mtime -1

# Search for configuration files
find /proc/1/root/ -name "*.conf" -o -name "*.yaml" -o -name "*.json"

Process Analysis:

# Check what files your app has open
lsof -p 1

# Monitor system calls (if strace is available)
strace -p 1 -f

# Check memory usage
cat /proc/1/status | grep -E "(VmSize|VmRSS|VmPeak)"

Real-World Example

Let me show you a complete example. Let’s say you have a Go application in a distroless container that’s failing health checks:

# First, identify the problematic pod
kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
my-app-7d4b8c8f5-xyz42   1/1     Running   5          2h

# Create debug container
kubectl debug -it my-app-7d4b8c8f5-xyz42 --image=ubuntu --target=my-app

# Inside the debug container:
apt update && apt install -y curl procps

# Check if the app is responding
curl http://localhost:8080/health
# Connection refused - aha!

# Check what the app is actually listening on
netstat -tlnp
# Shows it's listening on 0.0.0.0:3000, not 8080

# Check the app's environment variables for clues
cat /proc/1/environ | tr '\0' '\n' | grep PORT
# PORT=3000

# Test the correct port
curl http://localhost:3000/health
# {"status": "ok"} - there's our problem!

The issue was a misconfigured health check endpoint - the service was configured to check port 8080, but the app was listening on 3000.

When kubectl debug Isn’t Available

If you’re running an older Kubernetes version (< 1.18) or your cluster doesn’t support ephemeral containers, you have a few alternatives:

Option 1: Add a debug sidecar to your pod spec

apiVersion: v1
kind: Pod
spec:
  shareProcessNamespace: true
  containers:
  - name: app
    image: gcr.io/distroless/java:11
    # your app config
  - name: debug
    image: ubuntu
    command: ["/bin/sleep", "infinity"]
    stdin: true
    tty: true

Option 2: Use kubectl cp to get files out

# Copy files from the container to investigate locally
kubectl cp my-pod:/app/config.json ./config.json
kubectl cp my-pod:/var/log/ ./logs/

Troubleshooting Common Issues

Debug container can’t see main container processes: Make sure you’re using --share-processes or that shareProcessNamespace: true is set in your pod spec.

Permission denied accessing /proc/1/root: This can happen if your debug container doesn’t have sufficient privileges. Try:

ls -hal /prot/1

To determine the UID/GID and create an user with these values to be able to read/write.

Main container isn’t PID 1: If your app isn’t running as PID 1, find the correct process:

ps aux | grep your-app-name
# Use the correct PID instead of 1
ls /proc/PID/root/

Security Considerations

While kubectl debug is incredibly useful, keep these security considerations in mind:

• Debug containers can access sensitive information from the main container
• They run with the same service account permissions
• Logs from debug containers might contain sensitive data
• Always clean up debug containers when done (they’re ephemeral by default)

Best Practices

Here are some best practices I’ve learned over the years:

• Use minimal debug images: Start with alpine or ubuntu, add tools as needed
• Document your debugging process: Save useful commands for your team
• Create debugging runbooks: Common issues and their investigation steps
• Use labels: Tag your debug containers for easy identification
• Set resource limits: Debug containers can consume cluster resources too

Creating a Debug Container Template

You might want to create a pre-configured debug image for your team:

FROM ubuntu:22.04

RUN apt update && apt install -y \
    curl \
    wget \
    netcat \
    netstat-nat \
    procps \
    strace \
    tcpdump \
    tree \
    jq \
    && rm -rf /var/lib/apt/lists/*

# Add any custom debugging tools your team needs
COPY debug-scripts/ /usr/local/bin/

CMD ["/bin/bash"]

Build and push this to your registry, then use it for debugging:

kubectl debug -it my-pod --image=your-registry/debug-toolkit:latest

Conclusion

Debugging distroless containers doesn’t have to be a nightmare. With kubectl debug and the /proc/1/root technique, you can investigate issues in even the most minimal containers. The key is understanding that you’re not trying to add tools to the distroless container - you’re bringing your own toolbox and accessing the container’s filesystem from the outside.

This approach gives you the security benefits of distroless containers in production while maintaining the ability to debug when things go wrong. It’s the best of both worlds - secure, minimal containers with full debugging capabilities when you need them.

Remember, the goal isn’t to avoid issues entirely (though that would be nice), but to be able to quickly identify and resolve them when they inevitably occur. With these techniques in your toolkit, distroless containers become much less scary to debug.

Hope you found this useful and enjoyed reading it, until next time!