5

I'm trying to run apt-get update and getting this error for gitlab-runner:

# apt-get update
.
.
.
Get:6 https://packages.gitlab.com/runner/gitlab-runner/ubuntu focal InRelease [23.5 kB]
Err:6 https://packages.gitlab.com/runner/gitlab-runner/ubuntu focal InRelease
  The following signatures were invalid: EXPKEYSIG 3F01618A51312F3F GitLab B.V. (package repository signing key) <packages@gitlab.com>
Reading package lists... Done
W: GPG error: https://packages.gitlab.com/runner/gitlab-runner/ubuntu focal InRelease: The following signatures were invalid: EXPKEYSIG 3F01618A51312F3F GitLab B.V. (package repository signing key) <packages@gitlab.com>
E: The repository 'https://packages.gitlab.com/runner/gitlab-runner/ubuntu focal InRelease' is not signed.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
You have new mail in /var/mail/root
#
.
.
.

I've worked through a few posts about this exact same error but nothing suggested solves my issue.

The official GitLab signing-keys guide at https://docs.gitlab.com/omnibus/update/package_signatures.html#package-repository-metadata-signing-keys does not correct the problem.

The above page suggests:

 awk '/deb \[signed-by=/{
       pubkey = $2;
       sub(/\[signed-by=/, "", pubkey);
       sub(/\]$/, "", pubkey);
       print pubkey
     }' /etc/apt/sources.list.d/gitlab_gitlab-?e.list | \
   while read line; do
     curl -s "https://packages.gitlab.com/gpg.key" | gpg --dearmor > $line
   done

or

 apt-key del 3F01618A51312F3F
 curl -s "https://packages.gitlab.com/gpg.key" | apt-key add -
 apt-key list 3F01618A51312F3F

to solve the issue with gitlab-runner having an expired package repository signing key, but the error remains exactly the same:

W: GPG error: https://packages.gitlab.com/runner/gitlab-runner/ubuntu focal InRelease: The following signatures were invalid: EXPKEYSIG 3F01618A51312F3F GitLab B.V. (package repository signing key) <packages@gitlab.com>

I've also tried steps from numerous posts found here and elsewhere to update the repository signing key, but nothing works - error remains the same.

It seems a simple operation - update a repository signing key for an APT repository - but -how- exactly do I do this to enable gitlab-runner to be updateable and maintainable in Ubuntu 20.04.6 LTS again?

This key

https://packages.gitlab.com/gpg.key

is considered invalid by the "apt-get update" command for the gitlab-runner APT repository, no matter what I do.

Any ideas?

Thanks!

(PS - Aware "apt-key" is deprecated, but I just NEED to get gitlab-runner updated / updatable again, don't mind for now that "apt-key" is deprecated. After I get gitlab-runner updated I can spend time learning how to use gpg itself instead of apt-key for future signature expiries.)

Stefan
  • 161
  • Not just gitlab runner, they've broken the main gitlab ppa as well. I have my gtlab instance saying "update ASAP" and "apt update" saying "good luck with that" – kdopen Mar 06 '24 at 23:18

6 Answers6

7

Official answer for this problem (source of the text from the following link):

https://docs.gitlab.com/omnibus/update/package_signatures#update-keys-after-expiry-extension

Update keys after expiry extension

For Debian based distributions:

PackageCloud generally made use of apt-key, which will be deprecated in the future. Manually installed or configured repositories from some distributions are already using the signed-by support within Debian package source lists.

  1. Determine if you’re using apt-key or signed-by functionality:

     grep 'deb \[signed-by=' /etc/apt/sources.list.d/gitlab_gitlab-?e.list

  2. If this grep returns any lines, you're using signed-by functionality. This takes precedence over any apt-key usage. If this grep returns no lines, you're using apt-key functionality.

    For signed-by, the following script (run as root) updates the public keys for GitLab repositories:

    awk '/deb \[signed-by=/{
    pubkey = $2;
    sub(/\[signed-by=/, "", pubkey);
    sub(/\]$/, "", pubkey);
    print pubkey
  }' /etc/apt/sources.list.d/gitlab_gitlab-?e.list | \
while read line; do
  curl -s "https://packages.gitlab.com/gpg.key" | gpg --dearmor > $line
done

  3. For apt-key, the following script (run as root) updates the public keys for GitLab repositories:

    apt-key del 3F01618A51312F3F  
curl -s "https://packages.gitlab.com/gpg.key" | apt-key add -
apt-key list 3F01618A51312F3F
karel
  • 114,770
Tu4n3r
  • 183
  • 4
2

Note tested it in ubuntu jammy.

   wget -qO-  https://packages.gitlab.com/gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/omnibus_gitlab_gpg.key

Move the old sources.list for this package sources And reload with curl command the skript from the install instuctions.

curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash

My sources.list looks like this

cat /etc/apt/sources.list.d/runner_gitlab-runner.list 
# this file was generated by packages.gitlab.com for
# the repository at https://packages.gitlab.com/runner/gitlab-runner

deb [signed-by=/usr/share/keyrings/runner_gitlab-runner-archive-keyring.gpg] https://packages.gitlab.com/runner/gitlab-runner/ubuntu/ jammy main
deb-src [signed-by=/usr/share/keyrings/runner_gitlab-runner-archive-keyring.gpg] https://packages.gitlab.com/runner/gitlab-runner/ubuntu/ jammy main

Do not forget to adapt this to your release.

nobody
  • 5,437
  • Just want to add to answer of @nobody command for Debian 12. wget -qO- https://packages.gitlab.com/gpg.key | gpg --dearmor -o /usr/share/keyrings/gitlab_gitlab-ce-archive-keyring.gpg – Dmitry G Mar 07 '24 at 03:09
2

You can use the same step that is described in the installation script:

curl -fsSL "https://packages.gitlab.com/runner/gitlab-runner/gpgkey" | gpg --dearmor > /usr/share/keyrings/runner_gitlab-runner-archive-keyring.gpg

then

apt update

You can use the same to fix gitlab-ce and gitlab-ee

curl -fsSL "https://packages.gitlab.com/runner/gitlab-runner/gpgkey" | gpg --dearmor > /usr/share/keyrings/gitlab_gitlab-ee-archive-keyring.gpg
curl -fsSL "https://packages.gitlab.com/runner/gitlab-runner/gpgkey" | gpg --dearmor > /usr/share/keyrings/gitlab_gitlab-ce-archive-keyring.gpg
Gonzalo
  • 21
  • 2
2

You can update your GitLab Runner signature by following the instructions below.

1. How to fix this.

The commands to be fixed are as follows

awk '/deb \[signed-by=/{
    pubkey = $2;
    sub(/\[signed-by=/, "", pubkey);
    sub(/\]$/, "", pubkey);
    print pubkey
  }' /etc/apt/sources.list.d/runner_gitlab-runner.list | \
  while read line; do
    curl -s "https://packages.gitlab.com/gpg.key" | gpg --dearmor > $line
  done

2. Differences from the original

/etc/apt/sources.list.d/gitlab_gitlab-?e.list
indicates where the GitLab apt file is located.

But the GitLab Runner apt file is.
/etc/apt/sources.list.d/runner_gitlab-runner.list.

The /etc/apt/sources.list.d/gitlab_gitlab-?e.list file does not exist on the machine running Runner, so the above original command will not create a key signature file.

3. Check to see if the key signature file exists.

Run the above command and see if the following file is created.
/usr/share/keyrings/runner_gitlab-runner-archive-keyring.gpg

ls -la /usr/share/keyrings/runner_gitlab-runner-archive-keyring.gpg
yano TT
  • 36
  • 1
  • Thanks yano TT this seems to flow into Gonzalo's answer above which fixed my issue. Thanks for replying. – Stefan Mar 18 '24 at 09:44
1

To resolve this situation in 2024, especially on old installs, first we need to remove already added gitlab apt gpg key. Run the command:

sudo apt-key del "F640 3F65 44A3 8863 DAA0 B6E0 3F01 618A 5131 2F3F"

and run latest gitlab runner install script fort Debian/Ubuntu/Mint:

curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash && sudo apt update

That's it, now you can do apt upgrade.

More details:

Note that apt-key on Debian 12 is obsolete, not sure about Ubuntu:

root@gitlab-runner:~# apt-key list
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).

So proper way in general should be to put dearmored gpg signature to /etc/apt/trusted.gpg.d, but its not a gitlab case.

If you look at /etc/apt/sources.list.d/runner_gitlab-runner.list file, you will notice gpg key mentioned directly:

# this file was generated by packages.gitlab.com for
# the repository at https://packages.gitlab.com/runner/gitlab-runner

deb [signed-by=/usr/share/keyrings/runner_gitlab-runner-archive-keyring.gpg] https://packages.gitlab.com/runner/gitlab-runner/debian/ bookworm main
deb-src [signed-by=/usr/share/keyrings/runner_gitlab-runner-archive-keyring.gpg] https://packages.gitlab.com/runner/gitlab-runner/debian/ bookworm main

This is the reason, why manually adding gpg key with apt-key does not resolve the issue. Executing install script again, would deploy latest key signature.

Arunas Bart
  • 143
1

The reason these keys expire is for security - as Gitlab employees workstations are very high value targets it is preferred to generate new keys after about 2-4 years - taking into consideration that Gitlab employees workstations are reinstalled with new operating systems every now and then, also in the hopes to eleminate possible malware. Because of this, we all have to update these signing keys every few years. I have been faced with similar archive keyring errors during apt update (not only from Gitlab) in the past and a more detailed explanation of the problem can be found here, which describes why the signature for gitlab expired, and how to fix it, in great detail.

The somewhat cryptic BASH snippet provided by gitlab starting with awk goes through the file /etc/apt/sources.list.d/runner_gitlab-runner.list line by line to search for the string signed-by=, extract the gpg archive keyring file path, downloads the new gitlab signing key and imports it into the extracted archive keyring file path.

The apt-key command, which only manages /etc/apt/trusted.gpg on Debian, is not used in newer installations of Gitlab anymore - as the /etc/apt/sources.list.d/gitlab_gitlab-ce.list in most modern Gitlab installations shows, a dedicated file /usr/share/keyrings/gitlab_gitlab-ce-archive-keyring.gpg is now used for that.

If your /etc/apt/sources.list.d/gitlab_gitlab-ce.list contains the signed-by= line, the following one-liner can be used to update they keyring with the new signing key:

wget -qO- https://packages.gitlab.com/gpg.key | gpg --no-default-keyring --keyring /usr/share/keyrings/gitlab_gitlab-ce-archive-keyring.gpg --import

If your /etc/apt/sources.list.d/gitlab_gitlab-ce.list does NOT contain a signed-by= line, you should preferably use the apt-key method like so:

curl -s https://packages.gitlab.com/gpg.key | apt-key add -

The warning that is shown in this case:

Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).

Can be ignored for the time being. Here is a more detailed explanation of the apt-key warning.

fmueller
  • 11
  • 2