Friday, December 9, 2016

Backdoors for becoming root in a Docker container.

In my last post, the main issue I looked at was whether you can trust what a Docker-formatted image says about the user it will run as. What we found was that if the ‘USER’ statement is used in a Dockefile, but is set to a name, you have no idea what UNIX user ID the application in the container will run as. This is because the name could be mapped to any user ID by the UNIX passwd file.

Setting up the UNIX passwd file such that a user name other than ‘root’ also mapped to the UID of 0 provided a backdoor to becoming root in the running container. By requiring that an integer UID be used with the ‘USER’ statement in a Dockerfile, we can inspect the image metadata and decide not to run the image if ‘USER’ wasn’t a non zero integer value.

Is this enough to protect us though? Are there other backdoors for becoming ‘root' in a Docker container. The answer to that is that there is, and this post will look at some of these ways.

Creating a setuid executable

The primary path for switching from a non privileged user to the ‘root’ user on a UNIX system is a setuid executable. This is an executable that has been blessed in such a way that instead of running as the user that ran it, it runs as the user who is the owner of the executable. Such setuid executables will also work inside of a Docker container.

To illustrate how a setuid executable works, lets look at the UNIX utility called ‘id', which is normally used to display information about what user and group the invoking process runs as. If run normally in our Docker container, we might see:

$ id
uid=1001(app) gid=1001(app) groups=1001(app)

Lets create a setuid version of the executable which is owned by ‘root' and bundle that in our image.

FROM centos:centos7
RUN groupadd --gid 1001 app
RUN useradd --uid 1001 --gid app --home /app app
RUN cp /usr/bin/id /usr/bin/id-setuid-root
RUN chmod 4711 /usr/bin/id-setuid-root
WORKDIR /app
USER 1001

Running the original version of ‘id’ and the setuid version, we get:

$ id
uid=1001(app) gid=1001(app) groups=1001(app)
$ id-setuid-root
uid=1001(app) gid=1001(app) euid=0(root) groups=1001(app)

As can be seen, the result is that although the real user ID is the same, the effective user ID is that of the ‘root’ user. This means that by using a setuid executable, we gain the rights to run something as if we are ‘root’, or at least very close to being ‘root’. I say very close to being ‘root’ as it is only the effective user ID which is ‘root’ and not the real user ID. In most cases it doesn’t matter, but it hardly matters anyway, as we could also switch our real identify to the ‘root’ user relatively easily from a custom setuid executable of our own.

Running programs as 'root'

In the above example we took an existing executable and made it setuid as the ‘root’ user. We can’t go and do this for every executable we want to run as ‘root’, so what do we do if we want to run an arbitrary executable as ‘root’?

You might think that is simple. All we need to do is make a copy of ‘/bin/bash’ and make it setuid as the ‘root’ user. If we can then run that, we can become ‘root’ and run any program we want as the ‘root’ user.

So this time to create the image we use:

FROM centos:centos7
RUN groupadd --gid 1001 app
RUN useradd --uid 1001 --gid app --home /app app
RUN cp /bin/bash /bin/bash-setuid-root
RUN chmod 4711 /bin/bash-setuid-root
WORKDIR /app
USER 1001

Running our setuid version of bash though, we don’t get what we expected.

bash-4.2$ id
uid=1001(app) gid=1001(app) groups=1001(app)
bash-4.2$ bash-setuid-root
bash-setuid-root-4.2$ id
uid=1001(app) gid=1001(app) groups=1001(app)

This doesn’t work because modern implementations of shells have checks builtin which look for the specific case of where they are executed with an effective user ID of ‘root’, but a non ‘root’ real user ID. In this case, just to make it harder to use this sort of backdoor, they will revert back to running as the real user ID for the effective user ID.

Since this doesn’t work, lets look at how we would achieve the same thing if we weren’t trying to use a backdoor.

The first method we would normally use to execute a command as the ‘root’ user when we are not a privileged user, is to use the ‘sudo’ command. An alternative is to use the ‘su’ command to login as the ‘root’ user.

Because in a Docker image we can install and configure anything we want, there is no reason why we can’t just set these up and use them.

FROM centos:centos7
RUN yum install -y sudo
RUN groupadd --gid 1001 app
RUN useradd --uid 1001 --gid app --home /app app
# Allow anyone in group 'app' to use 'sudo' without a password.
RUN echo '%app ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers
# Set the password for the 'root' user to be an empty string.
RUN echo 'root:' | chpasswd
WORKDIR /app
USER 1001

With this we can very easily get an interactive shell as the ‘root’ user using ’sudo’, not even requiring a password.

$ id
uid=1001(app) gid=1001(app) groups=1001(app)
$ sudo -s
# id
uid=0(root) gid=0(root) groups=0(root)

We can also just login as the ‘root’ user, supplying our empty password.

$ id
uid=1001(app) gid=1001(app) groups=1001(app)
$ su root
Password:
# id
uid=0(root) gid=0(root) groups=0(root)

In both cases the real user ID is that of the ‘root’ user and not just the effective user ID.

So we didn’t even need to fiddle with a backdoor, we can just use the existing features of the operating system. We just need to install the ‘sudo’ package and configure it, or set the ‘root’ password. As it happens, both these mechanisms rely on a setuid executable, but combine it with configuration to guard against who can access them. It is a simple matter though to enable that access given that during the build of a Docker image you can change anything.

You can’t completely block 'root'

You might be thinking at this point that if we can become the ‘root’ user in these ways, what is the point then of using a check on what ‘USER’ specified for the image in the first place. Someone can always set it as a non ‘root’ user, using an integer UID to avoid any restriction on using the image, but then use a custom built backdoor marked as a setuid executable, or using existing system tools such as ‘sudo’, or ‘su’.

What is important to understand is that good security is based on having many layers. You don’t rely on just a single security measure to protect your system. Each extra layer you can add, acts as an obstacle to someone reaching their end goal. Not allowing images to run that don’t set ‘USER’ to a non zero integer ID, would be just one step you can take in a overall security plan.

So it isn’t a waste of time just yet. This is because, although there are ways of becoming the ‘root’ user even if ‘USER’ did not originally declare the container should run as ‘root’, we can still control what the ‘root’ user is actually able to do. This is achieved using Linux capabilities, and is the next layer of defence you should employ.

In the next blog post I will look at Linux capabilities and how to use Docker to restrict what someone could do even if they become the ‘root’ user.

Thursday, December 1, 2016

What USER should you use to run Docker images.

If you follow this blog and my rants on Twitter you will know that I often complain about the prevalence of Docker-formatted container images that will only work if run as the root user, even though there is no technical reason to run them as root. With more and more organisations moving towards containers and using these images in production, some at least are realising that running them as root is probably not a good idea after all. As such, organisations are for their own images at least, starting to create basic guidelines for their developers to follow around what user an image should run as.

A typical example of the most basic guidelines you can find are:

  1. Create a new UNIX group called ‘app’ with a group ID (gid) of 1001.
  2. Create a new UNIX account with user name ‘app’ with a user ID (uid) of 1001, with it being a member of the group ‘app’ and where the home directory of this user is the directory ‘/app’.
  3. Put all your application source code under the ‘/app’ directory.
  4. Set the working directory for any application run to the ‘/app’ directory.
  5. Set the user that the image will run as to the ‘app’ user.

All looks good, and better than running as the root user you might be thinking. Unfortunately there are still a number of problems with these guidelines, as well as things that are missing.

In this blog post I am going to look at the last guideline in that list, and issues around how you specify what user an image should run as. In subsequent posts I will pull apart the other guidelines. At the end of the posts I will summarise what I believe are a better set of basic guidelines around setting up a Docker-formatted container image.

Skeleton for a Dockerfile

Following the above guidelines, the skeleton for the Dockerfile would look like:

FROM centos:centos7
RUN groupadd --gid 1001 app
RUN useradd --uid 1001 --gid app --home /app app
COPY . /app
WORKDIR /app
USER app

If we build and then run this image and have it start up an interactive shell, we can validate that the command we have run is running as the user ‘app’ and that we are in the correct directory.

$ docker run -it --rm best-practices

[app@ca172749cd4f ~]$ id
uid=1001(app) gid=1001(app) groups=1001(app)

[app@ca172749cd4f ~]$ pwd
/app

[app@ca172749cd4f ~]$ ls -las
total 24
4 drwx------ 2 app app 4096 Dec 1 03:15 .
4 drwxr-xr-x 27 root root 4096 Dec 1 03:15 ..
4 -rw-r--r-- 1 app app 18 Aug 2 16:00 .bash_logout
4 -rw-r--r-- 1 app app 193 Aug 2 16:00 .bash_profile
4 -rw-r--r-- 1 app app 231 Aug 2 16:00 .bashrc
4 -rw-r--r-- 1 root root 136 Dec 1 03:15 Dockerfile

[app@ca172749cd4f ~]$ exit

Seems simple enough, so why is this a problem?

Who do you think you can trust?

The problem is the ‘USER’ statement added to the Dockerfile. This is what declares what user the container should run as.

We can see that this was the last statement in the Dockerfile, and so this should be what user is used when the image is run. That this is the case, can be validated by inspecting the meta data of the Docker-formatted image using the ‘docker inspect’ command:

$ docker inspect --format='{{.Config.User}}' best-practices
app

This means that you could verify that an image satisfies the guideline that it runs as the ‘app’ user and not as the root user, before actually running it.

The problem is this doesn’t actually guarantee anything. This is because the value associated with the ‘.Config.User’ setting is a name. You cannot tell what UNIX user ID this really maps to inside of the container when run.

To illustrate the problem, consider the changed Dockerfile as follows:

FROM centos:centos7
RUN groupadd --gid 1001 app
RUN useradd --uid 1001 --gid app --home /app app
RUN sed -i -e 's/1001/0/g' /etc/passwd
COPY . /app
WORKDIR /app
USER app

Validating what ‘docker inspect’ says about what user the image will run as we still get the ‘app’ user:

$ docker inspect --format='{{.Config.User}}' best-practices
app

When we do run the actual image though, that isn’t the case in practice.

$ docker run -it --rm best-practices

[root@71150399f77f app]# id
uid=0(root) gid=0(root) groups=0(root)

So although the Dockerfile specified ‘USER app’ and ‘docker inspect’ also indicated that the image will run as ‘app’, the command actually ran as the root user.

This is because when using a name for ‘USER’, it still needs to be mapped to an actual UNIX user ID by the UNIX passwd file. As shown, we can remap the user name to the user ID ‘0’, meaning it still runs as the root user. We did this surreptitiously to indicate the problem of whether you can actually trust what you see. In this case the command to modify the passwd file was in the Dockerfile and in plain site, but it could also have been buried deep inside some script file or program that had been copied into the Dockerfile and then run during the building of the image.

Why does this matter?

If this is your own system you are running on for your own personal use, then you may not care. It is though a problem in a corporate setting, or if you are running a multi tenant hosting environment where you are allowing Docker-formatted images from potentially untrusted sources. In this case you want to be sure you aren’t going to run an image which actually runs as root. As we have seen, even if ‘USER’ is set in the Dockerfile to be a user other than ‘root’ it doesn’t mean it still isn’t running as root.

Verifying the user is not root

How then can we be confident that a Docker-formatted image we have been supplied isn’t going to run as root? As we have seen we obviously can’t trust ‘USER’ when it is set to a name, we have to reject any such image and not allow it to be run.

The solution is not to use a name, but the actual UNIX user ID with the ‘USER’ statement. What we therefore would require is that the Dockerfile be written as:

FROM centos:centos7
RUN groupadd --gid 1001 app
RUN useradd --uid 1001 --gid app --home /app app
RUN sed -i -e 's/1001/0/g' /etc/passwd
COPY . /app
WORKDIR /app
USER 1001

 Inspect the image now using ‘docker inspect’ and we get:

$ docker inspect --format='{{.Config.User}}' best-practices
1001

Run the image and we get:

$ docker run -it --rm best-practices
bash-4.2$ id
uid=1001 gid=0(root) groups=0(root)
bash-4.2$ ls -las
total 36
4 drwx------ 2 1001 app 4096 Dec 1 05:10 .
4 drwxr-xr-x 28 root root 4096 Dec 1 05:12 ..
4 -rw-r--r-- 1 1001 app 18 Aug 2 16:00 .bash_logout
4 -rw-r--r-- 1 1001 app 193 Aug 2 16:00 .bash_profile
4 -rw-r--r-- 1 1001 app 231 Aug 2 16:00 .bashrc
4 -rw-r--r-- 1 root root 177 Dec 1 05:10 Dockerfile

Thus by requiring that ‘USER’ be set to a UNIX user ID, we are able to guarantee that it will run as the user it says it is. Even if the supplier of the image had still fiddled with the passwd file it wouldn’t matter, they can’t change the fact it will run as that user ID.

Recommended Guidelines

What then is a better guideline about what user a Docker-formatted container should be run? I would suggest the following.

Do not run a Docker-formatted container image as the root user. Always override in the Dockerfile what user the image will run as. This should be done by adding the ‘USER’ statement in the Dockerfile. The value of the ‘USER’ statement must be the integer UNIX user ID of the UNIX account you want any application to run as inside of the container. It should not be the user name for the UNIX account.

In addition to that guideline for the author of any Docker-formatted container image, I would also add the following guideline for anyone building a system on top of the Docker service for running images.

Where it is intended not to allow images to run as the root user, but you want to allow an image to run as the user it specifies, reject any Docker-formatted container image that you can't verify what UNIX user ID it will run as. Use ‘docker inspect’ to determine the user it should run as. Reject the image and do not run it if the user setting specified in the image meta data, is not an integer value greater than 0.

Already you will find some orchestration systems for managing containers using the Docker runtime implement this latter recommendation in certain configurations. One such example is Kubernetes and systems based around it, such as OpenShift. Because of the growth of interest in Kubernetes, especially for enterprise usage and for hosting services, adhering to the first guideline is also the first step in ensuring you will be able to deploy your images to these systems when they are set up in a secure way.

In a followup post I will look at some more aspects around what user an image should run as, whether that be the choice of the developer of the image, or whether it is a user enforced by the hosting service.