Troubleshooting Kubernetes and Docker with a SuperContainer

Super Powers

The Target

The final step is identifying a problematic container to troubleshoot and then firing up a SuperContainer on the same host as that misbehaving container. Without further ado, here's the clever part.

The target container is shown below in Figure  1 using the docker ps command. Note that the container is an Nginx container, which I have explicitly named nginx. In Kubernetes, the name would be a pod name such as nginx-kv152x. You should first find out which minion (or node) your pod is running on using a command such as:

$ kubectl get nodes

In Figure 1, you can see the target container has the name nginx, on the right-hand side, and is exposing the HTTP port (TCP port 80) from the host via TCP port 80 on the container under PORTS.

Figure 1: I'd better take a closer look at the problematic web server container, named nginx.

The run-time parameters, which make the SuperContainer so useful, are based on launching the SuperContainer within the same process namespace and network namespace as the target container.

To achieve this kernel-trickery, you need to fire up the SuperContainer with the following command line switches and also provide the CAP_SYS_ADMIN capability. If you're not familiar with the CAP_SYS_ADMIN capability, it is a powerful mechanism for opening up system permissions. However, it is not quite as daunting as running a container in privileged mode, which also circumvents a kernel's cgroup resource limitations. Look online for more information on kernel capabilities [3].

The capabilities man page warns about the power of SYS_ADMIN access by saying "Don't choose CAP_SYS_ADMIN if you can possibly avoid it!" A long overloaded list shows the complications associated with running a process at this capability level.

Back to the all-important command line, I run the SuperContainer as follows:

$ docker run --rm -it --pid=container:nginx   --net=container:nginx --cap-add sys_admin   chrisbinnie/supercontainer

Take a closer look at this clever command line. I am running the container with --rm, so it is destroyed afterwards (so attackers can't take advantage of it), and I'd also recommend deleting the image after using it on a host. I use the --pid switch to enter the same process namespace as the container name nginx. In exactly the same way, I use --net to access its network stack. Then I finally add the tricksy SYS_ADMIN.

If you struggle to get access to your applications from inside the target container, you can add other capabilities and, as a last resort, use --privileged mode. For example for running the strace package as a syscall debugging tool (syscalls are made whenever a process needs to request a service from the kernel), you should change your custom Dockerfile as follows:

CMD ["strace", "-p", "1"]

You would also need to add the CAP_SYS_PTRACE capability.

If you're interested in more detail about CAP_SYS_PTRACE, the kernel man page lists the following system resource access:

  • Use ptrace(2) to trace arbitrary processes.
  • Apply get_robust_list(2) to arbitrary processes.
  • Use process_vm_writev(2) to transfer data to or from the memory of arbitrary processes.
  • Use kcmp(2) to inspect processes.

Super Power

Having executed the run command you have spawned an interactive terminal offering access to your SuperContainer.

Figure 2 shows the SuperContainer running with an arbitrary hashed name (c7b61b1a8b82).

Figure 2: I have entered the SuperContainer using run as the superuser root.

Super Process Tables

Be warned that what follows might take a moment to get used to; it is clever, simple, and yet a little surprising. If you have ever used a chroot in the past, it isn't quite as discombobulating.

A reminder that the plan is to access another container from within the SuperContainer by using the same kernel namespaces as the target.

Figure 3 shows a standard command to check the process table. But… look! The Nginx web server is definitely not running inside the SuperContainer, so I know for certain that I'm accessing the target container, named nginx.

Figure 3: Running the ps -ef command inside the SuperContainer shows the process table of another container.

I can identify which processes are running within the container, under which user, and with which PID and parameters, as shown in Figure 3.

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy ADMIN Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

comments powered by Disqus
Subscribe to our ADMIN Newsletters
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs



Support Our Work

ADMIN content is made possible with support from readers like you. Please consider contributing when you've found an article to be beneficial.

Learn More”>
	</a>

<hr>		    
			</div>
		    		</div>

		<div class=