7

Several evenings, I already try to get an LXC installation working with user-space containers. Since it's going to be a new server, I don't care (yet) too much about distribution and release, so I tried Debian 7+8 and Ubuntu 14.04 and 15.04 as host and Ubuntu 15.04 and Debian 8 as guest. All of them have their problems. The summary:

Debian 7.8: lxc-create: This command has to be run as root (both guests)

Debian 8.0: lxc-create: Operation not permitted (both guests)

Ubuntu 14.04 w/ Ubuntu 15.04 guest: lxc-start: call to cgmanager_create_sync failed: invalid request / The container failed to start.

Ubuntu 14.04 w/ Debian 8 guest: lxc-create: ERROR: Couldn't find a matching image.

Ubuntu 15.04: lxc-start: call to cgmanager_move_pid_sync failed: invalid request / The container failed to start. (both guests)

I followed the tutorial https://help.ubuntu.com/lts/serverguide/lxc.html and https://linuxcontainers.org/lxc/getting-started/, and they don't look like it's so complicated.

I wrote a script to make the test reproducable (to be run as root on a freshly booted live cd). Can anyone tell me what's wrong with it?

#!/bin/sh


# need to be run as root

set -x

echo "==== SYSTEM INFO & INSTALL ===="

lsb_release -a

uname -a

apt-get update
apt-get install -y lxc
apt-get clean

lxc-checkconfig
ifconfig
brctl show

adduser testuser

cat /etc/subuid /etc/subgid

cat >/etc/lxc/lxc-usernet <<.e
# USERNAME TYPE BRIDGE COUNT
testuser veth lxcbr0 2
.e

sudo -u testuser -i mkdir -p .config/lxc

sudo -u testuser -i tee .config/lxc/default.conf <<.e
lxc.id_map = u 0 $(grep testuser /etc/subuid | cut -d: -f2) 65536
lxc.id_map = g 0 $(grep testuser /etc/subgid | cut -d: -f2) 65536

lxc.network.type = veth
lxc.network.link = lxcbr0
lxc.network.flags = up
lxc.network.hwaddr = 00:16:3e:xx:xx:xx
.e
cat /home/testuser/.config/lxc/default.conf

echo "==== TEST UBUNTU VIVID GUEST ===="

sudo -u testuser -i lxc-create -t download -n vivid1 -- -d ubuntu -r utopic -a amd64
#You just created an Ubuntu container (release=trusty, arch=amd64, variant=default)

#sudo -u testuser -i lxc-start -n vivid1 -d
#read press_enter_key_when_quit

if [ $? -eq 0 ]; then
  rm /tmp/log
  sudo -u testuser -i lxc-start -n vivid1 -l debug --logfile /tmp/log
  cat /tmp/log
fi

echo "==== TEST DEBIAN JESSIE GUEST ===="

sudo -u testuser -i lxc-create -t download -n jessie1 -- -d debian -r jessie -a amd64
#You just created an Ubuntu container (release=trusty, arch=amd64, variant=default)

#sudo -u testuser -i lxc-start -n jessie1 -d
#read press_enter_key_when_quit

if [ $? -eq 0 ]; then
  rm /tmp/log
  sudo -u testuser -i lxc-start -n jessie1 -l debug --logfile /tmp/log
  cat /tmp/log
fi

echo "==== END OF SCRIPT ===="

On pastebin, I posted the console output from all these machines:

Update

I tried further with Ubuntu 15.04 as host. I found the bug report https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1413927 Which has a similar but not same problem. But journalctl revealed problems with cgmanager:

May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:do_create_main: pid 17417 (uid 1000 gid 1000) may not create under /run/cgmanager/fs/blkio/user.slice/user-999.slice/session-c1.scope
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:do_create_main: pid 17417 (uid 1000 gid 1000) may not create under /run/cgmanager/fs/cpu/user.slice/user-999.slice/session-c1.scope
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:do_create_main: pid 17417 (uid 1000 gid 1000) may not create under /run/cgmanager/fs/cpuset/user.slice/user-999.slice/session-c1.scope
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:do_create_main: pid 17417 (uid 1000 gid 1000) may not create under /run/cgmanager/fs/devices/user.slice/user-999.slice/session-c1.scope
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:do_create_main: pid 17417 (uid 1000 gid 1000) may not create under /run/cgmanager/fs/freezer/user.slice/user-999.slice/session-c1.scope
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:do_create_main: pid 17417 (uid 1000 gid 1000) may not create under /run/cgmanager/fs/hugetlb/user.slice/user-999.slice/session-c1.scope
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:do_create_main: pid 17417 (uid 1000 gid 1000) may not create under /run/cgmanager/fs/memory/user.slice/user-999.slice/session-c1.scope
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:do_create_main: pid 17417 (uid 1000 gid 1000) may not create under /run/cgmanager/fs/net_cls/user.slice/user-999.slice/session-c1.scope
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:do_create_main: pid 17417 (uid 1000 gid 1000) may not create under /run/cgmanager/fs/perf_event/user.slice/user-999.slice/session-c1.scope
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:do_create_main: pid 17417 (uid 1000 gid 1000) may not create under /run/cgmanager/fs/none,name=systemd/user.slice/user-999.slice/session-c1.scope
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager: Invalid path /run/cgmanager/fs/blkio/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:per_ctrl_move_pid_main: Invalid path /run/cgmanager/fs/blkio/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager: Invalid path /run/cgmanager/fs/cpu/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:per_ctrl_move_pid_main: Invalid path /run/cgmanager/fs/cpu/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager: Invalid path /run/cgmanager/fs/cpuset/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:per_ctrl_move_pid_main: Invalid path /run/cgmanager/fs/cpuset/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager: Invalid path /run/cgmanager/fs/devices/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:per_ctrl_move_pid_main: Invalid path /run/cgmanager/fs/devices/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager: Invalid path /run/cgmanager/fs/freezer/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:per_ctrl_move_pid_main: Invalid path /run/cgmanager/fs/freezer/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager: Invalid path /run/cgmanager/fs/hugetlb/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:per_ctrl_move_pid_main: Invalid path /run/cgmanager/fs/hugetlb/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager: Invalid path /run/cgmanager/fs/memory/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:per_ctrl_move_pid_main: Invalid path /run/cgmanager/fs/memory/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager: Invalid path /run/cgmanager/fs/net_cls/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:per_ctrl_move_pid_main: Invalid path /run/cgmanager/fs/net_cls/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager: Invalid path /run/cgmanager/fs/perf_event/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:per_ctrl_move_pid_main: Invalid path /run/cgmanager/fs/perf_event/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager: Invalid path /run/cgmanager/fs/none,name=systemd/user.slice/user-999.slice/session-c1.scope/lxc/jessie1
May 20 16:06:10 xubuntu cgmanager[4736]: cgmanager:per_ctrl_move_pid_main: Invalid path /run/cgmanager/fs/none,name=systemd/user.slice/user-999.slice/session-c1.scope/lxc/jessie1

Update 2

I tried more: I installed Ubuntu trusty and vivid on two virtual machines and installed all updates on them. Then I tested both once with the original lxc package and once with the ones from the lxc daily ppa. Result is always the same error as shown above.

Daniel Alder
  • 2,426
  • Received the Tumbleweed badge for this question - that's the opposite of what I wanted.. :-( – Daniel Alder May 22 '15 at 07:18
  • I suspect that there is an issue between cgmanager and systemd. AskUbuntu probably isn't the right place to for this, try filing a bug report on launchpad. – maccam94 May 23 '15 at 00:32
  • From my experience, Launchpad is not an option for this kind of unspecific problems. Normally it takes very long there for a bug to be seen, and very often bugs are closed because someone thinks it's not reproducible. Once it even happened that a because temporary closing of my bug all my attachments with complex screenshots were removed. some bugs are also open now for over 3 years and they are still reproducable in current releases. and the questions part of Launchpad is even worse: it's hard there to get attention. Better would be the Github issues page of LXC – Daniel Alder May 23 '15 at 10:44

1 Answers1

7

Eureka! Still not finished, but I managed a first time to start a container. There are a couple of things which went wrong and I found that there are many people out there having similar problems. Here is a little trouble solving guide:

Disribution / Repositories

Because of the most positive feedback, I decided to start with a Ubuntu 14.04 debootstrapped system. This means, the original system is not bigger than only a few 100M and doesn't contain many packages. I used updates and security package sources and the lxc daily ppa. Here is my /etc/apt/sources.list:

deb http://de.archive.ubuntu.com/ubuntu trusty main
deb http://de.archive.ubuntu.com/ubuntu trusty-updates main
deb http://security.ubuntu.com/ubuntu trusty-security main

deb http://ppa.launchpad.net/ubuntu-lxc/daily/ubuntu trusty main

Installation

In many tutorials, forums and bug reports I found lists of packages which have to be installed. I'm not sure yet which of those are important, but here's the list of what I finally installed (starting from a debootstrapped 14.04 system):

apt-get install bridge-utils cgmanager cloud-image-utils debootstrap distro-info \
  distro-info-data euca2ools fuse libaio1 libapparmor1 libcap2 liblxc1 \
  libpam-systemd librados2 libseccomp2 libselinux1 libselinux1 lxc python3-lxc \
  python-distro-info

Again: this list is probably longer than necessary. Here is what's really important:

  • libpam-systemd: this library is important for the correct cgroup permissions. I didn't see any error message, but without it the file /proc/self/cgroup looked like 8:blkio:/ etc. instead of 8:hugetlb:/user/1000.user/1.session
  • fuse: I read it's important for lxcfs (I would say it's a bug that lxcfs package doesn't depend on it)
  • lxc: the most important package and enough if you only use unprivileged containers
  • cgmanager: I'm still learning about its function. Btw: the lxc PPA currently provides both packages cgmanager-utils(0.27) and cgmanager(0.30). cgmanager conflicts with cgmanager-utils (<< 0.30-1) which means the two package from the same repo are not compatible (possibly a mistake). I chose cgmanager.

Configuration

I tried a lot of hints from everywhere, so I don't know yet what's important. But roughly I just ran the script from the question. Roughly it's about these files:

  • /etc/subuid
  • /etc/subgid
  • /etc/lxc/lxc-usernet
  • ~/.config/lxc/default.conf

The only additional thing I did was the following, because I found the line in many threads:

chmod +x /home/testuser/.local/share

Login as user

Now the most important thing which I still don't understand but which ruined all of my script-base experiments: Don't login using su or sudo!

I demonstrate:

root@1404-lxc-test:~# tail -1 /proc/self/cgroup
2:blkio:/user/0.user/1.session
# this is expected. I am root.

root@1404-lxc-test:~# sudo -u testuser -i tail -1 /proc/self/cgroup
2:blkio:/user/0.user/1.session
# here I used sudo to switch user

root@1404-lxc-test:~# su - testuser
testuser@1404-lxc-test:~$ tail -1 /proc/self/cgroup
2:blkio:/user/0.user/1.session
# at least here I expected the correct settings

testuser@1404-lxc-test:~$ exit
root@1404-lxc-test:~# exit

$ ssh testuser@1404-lxc-test
testuser@1404-lxc-test's password: 
Welcome to Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-53-generic x86_64)

testuser@1404-lxc-test:~$ tail -1 /proc/self/cgroup
2:blkio:/user/1000.user/2.session
# now it's correct

I still don't know the reason, but it must be connected with libpam-systemd. Obviously, both su and sudo bypass PAM

And finally, don't forget to use the lxc-xxx commands when logged in as user. They will fail if you do it as root (because user-space containers are stored in .local/share/lxc/ instead of /var/lib/lxc/

Troubleshooting

These commands were most useful for me:

  • journalctl (on upstart-based releases): It revealed problems with cgmanager
  • /proc/self/cgroup
  • lxc-start using options -l debug --logfile logfilename.txt

Let me know what else is important. I think the community needs it.

Credits

Thanks to the following Tutorials:

And to these bug reports ans forum threads:

Disclaimer

I wrote this text after the first successful test. But I did so many things that some of these steps are likely not necessary. I will re-check everything on a new system soon.

Update

All the above was done with Ubuntu trusty 14.04 LTS. Now, I just tested Ubuntu wily. So far I can tell:

  • All you need to install on top of a debootstrapped base system: apt-get install lxc bridge-utils
  • No external mirrors are required. Use the lxc provided by Ubuntu
  • I didn't touch more than the following files on my fresh system: /etc/subuid, /etc/subgid, /etc/lxc/lxc-usernet, additionally, i created a user and filled its home
  • Conclusion: much easier, much more stable. For me it's the first release which really works.

(Wily is now beta and will be released on 22nd October 2015)

Daniel Alder
  • 2,426