Cannot run apt update on ubuntu 22 docker image on a CentOS host

Question

Update solution

I had to install docker-engine from the centos extras repository. These were the steps:

First, remove docker

sudo yum remove docker \
                  docker-client \
                  docker-client-latest \
                  docker-common \
                  docker-latest \
                  docker-latest-logrotate \
                  docker-logrotate \
                  docker-engine

Then add the docker-ce repo:

sudo yum install -y yum-utils
sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

Then install docker-ce

sudo yum install docker-ce docker-ce-cli containerd.io docker-compose-plugin

Finally start docker-ce

sudo systemctl start docker

First let me state that the question about updating PUBKEY (How do I fix the GPG error "NO_PUBKEY"?) does not solve this issue as I've mentioned below.

The following is my process:

docker run --rm -it ubuntu:22.04 bash -l

This downloads the ubuntu 22.04 image, starts a container, and drops me into a root shell. This process is the same for any host OS I've tried (MacOS, CentOS, Amazon Linux).

The next command, run inside the root shell, works on MacOS and Amazon Linux but not on Cent OS host:

apt update

On a CentOS host, I get this:

root@b40f3c9766e7:/# apt update
Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB]
Get:2 http://archive.ubuntu.com/ubuntu jammy InRelease [270 kB]
Err:1 http://security.ubuntu.com/ubuntu jammy-security InRelease   
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [109 kB]
Err:2 http://archive.ubuntu.com/ubuntu jammy InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
Err:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [90.7 kB]
Err:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
Reading package lists... Done
W: http://security.ubuntu.com/ubuntu/dists/jammy-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://security.ubuntu.com/ubuntu/dists/jammy-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://security.ubuntu.com/ubuntu jammy-security InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
E: The repository 'http://security.ubuntu.com/ubuntu jammy-security InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: http://archive.ubuntu.com/ubuntu/dists/jammy/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://archive.ubuntu.com/ubuntu/dists/jammy/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://archive.ubuntu.com/ubuntu jammy InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
E: The repository 'http://archive.ubuntu.com/ubuntu jammy InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: http://archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://archive.ubuntu.com/ubuntu/dists/jammy-updates/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://archive.ubuntu.com/ubuntu jammy-updates InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
E: The repository 'http://archive.ubuntu.com/ubuntu jammy-updates InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: http://archive.ubuntu.com/ubuntu/dists/jammy-backports/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: http://archive.ubuntu.com/ubuntu/dists/jammy-backports/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg are ignored as the file is not readable by user '_apt' executing apt-key.
W: GPG error: http://archive.ubuntu.com/ubuntu jammy-backports InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 871920D1991BC93C
E: The repository 'http://archive.ubuntu.com/ubuntu jammy-backports InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb /var/cache/apt/archives/partial/*.deb /var/cache/apt/*.bin || true'
E: Sub-process returned an error code

I tried to add the PUBKEY using instructions from How do I fix the GPG error "NO_PUBKEY"? but I get this error:

E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation

I cannot install any of these because I cannot use apt, so what's the way out of this situation?

Does anyone know why this works on MacOS & Amazon Linux but not on CentOS?

Also, this works on Ubuntu 18.04 and Ubuntu 20.04. It only fails on 22.04.

There may be some issues with the Docker engine for CentOS, Posting this question on Docker forum (https://forums.docker.com) may help. — turbulence, May 12 '22 at 05:22
Silly question, but why upgrade a container? The whole point of the things is that they're more disposable than a VM. Is it not possible to rebuild your containers using 22.04 as the base? — matigo, May 12 '22 at 05:32
@matigo I'm not upgrading a container. I'm running apt update which updates the apt cache. I need this so that I can install other packages like curl for example. — bluesmoon, May 12 '22 at 13:47
It says The key(s) in the keyring ... are ignored as the file is not readable by user '_apt' executing apt-key. Please add the output of ls -al /etc/apt/trusted.gpg.d/ to the post. I can't repro this (the image hash (ID) I have for ubuntu:22.04 is sha256:d2e4e1f511320dfb2d0baff2468fcf0526998b73fe10c8890b4684bb7ef8290f - what's the hash for the image you're using? Maybe it's an old one? — muru, May 12 '22 at 20:41
The issue was that the docker engine was old. On CentOS we need docker-ce. — bluesmoon, May 13 '22 at 12:06
You need Docker >= 20.10.9. This is because newer ubuntu versions use Glibc 2.34 which uses a new system call called clone3, which is not in the allowlist of the default seccomp profile for older docker versions. See https://github.com/moby/moby/pull/42681. credit: https://stackoverflow.com/questions/71941032/why-i-cannot-run-apt-update-inside-a-fresh-ubuntu22-04#answer-72057185 — Arjun Sreedharan, May 17 '22 at 14:55

score 4 · Accepted Answer · edited May 16 '22 at 08:34

EDIT: After looking at this thoroughly I realized what I had done here was to just use the default yum-provided version of docker in @extras.

I removed these:

Removing:
 docker                          x86_64                   2:1.13.1-209.git7d71120.el7.centos                    @extras                    64 M
 docker-client                   x86_64                   2:1.13.1-209.git7d71120.el7.centos                    @extras                    13 M
 docker-common                   x86_64                   2:1.13.1-209.git7d71120.el7.centos                    @extras                   4.4 k

I then installed the correct method from docker https://docs.docker.com/engine/install/centos/

The issue is now entirely resolved. Below is my original delve into this so you can see a bit of my course in looking through this one.

I am seeing the same thing on CentOS 7 based Docker containers that are running literally every single public Ubuntu image - for whatever reason. I just slashed through the various versions and it's something indicating there is some lower level issue / bug between Docker & CentOS 7 when it comes to this routine. What I genuinely do not know. Still digging.

W: http://security.ubuntu.com/ubuntu/dists/jammy-security/InRelease: The key(s) in the keyring /etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg are ignored as the file is not readable by user '_apt' executing apt-key

That's the problematic line. _apt is on all systems that I am seeing though the userID tends to be different.

Broken system:

root@69a789b8893e:/# grep _apt /etc/passwd /etc/shadow /etc/group
/etc/passwd:_apt:x:100:65534::/nonexistent:/usr/sbin/nologin

Working native system:

_apt:x:105:65534::/nonexistent:/usr/sbin/nologin

GID will always be common as 65534 so that makes sense. What doesn't here is that literally every file and directory is correct on perms & ownership. 0755 is always the default within /etc/apt.

root@69a789b8893e:/# find /etc/apt -type d -ls
  1464065      0 drwxr-xr-x   8 root     root          143 Apr 28 12:01 /etc/apt
 68555287      0 drwxr-xr-x   2 root     root          213 Apr 28 12:04 /etc/apt/apt.conf.d
135370920      0 drwxr-xr-x   2 root     root            6 Apr  8 10:22 /etc/apt/auth.conf.d
202835524      0 drwxr-xr-x   2 root     root            6 Apr  8 10:22 /etc/apt/keyrings
  1467146      0 drwxr-xr-x   2 root     root            6 Apr  8 10:22 /etc/apt/preferences.d
 68562845      0 drwxr-xr-x   2 root     root            6 Apr  8 10:22 /etc/apt/sources.list.d
135370943      0 drwxr-xr-x   2 root     root           84 Apr 28 12:04 /etc/apt/trusted.gpg.d

Going to keep looking.

score 0 · Answer 2 · answered Jul 05 '23 at 09:48

I indeed also encountered this issue, but noticed apt-get is working when starting the container with the option --privileged. It looks like the issue is due to a failing keyring. apt-ket list is also not working without running with the --privileged option. However, docker build does not support this option, so I also had to switched to proposed method on: https://docs.docker.com/engine/install/centos

Cannot run apt update on ubuntu 22 docker image on a CentOS host

2 Answers2

Linked