Can't run apt commands: permissions fun with /var/lib/apt/lists/partial

Question

Today I tried running sudo apt-get update on our Ubuntu 22.04 server and got the following:

Hit:1 http://us.archive.ubuntu.com/ubuntu jammy InRelease
Err:1 http://us.archive.ubuntu.com/ubuntu jammy InRelease
  Couldn't create temporary file /tmp/apt.conf.Z3BHzc for passing config to apt-key
Ign:2 http://us.archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:3 https://download.docker.com/linux/ubuntu jammy InRelease
Err:3 https://download.docker.com/linux/ubuntu jammy InRelease
  Couldn't create temporary file /tmp/apt.conf.LzHmDB for passing config to apt-key
Hit:4 http://us.archive.ubuntu.com/ubuntu jammy-backports InRelease
Err:4 http://us.archive.ubuntu.com/ubuntu jammy-backports InRelease
  Couldn't create temporary file /tmp/apt.conf.BI6ryE for passing config to apt-key
Ign:5 http://security.ubuntu.com/ubuntu jammy-security InRelease
Err:6 http://us.archive.ubuntu.com/ubuntu jammy-updates Release
  Could not open file /var/lib/apt/lists/partial/us.archive.ubuntu.com_ubuntu_dists_jammy-updates_Release - open (30: Read-only file system) [IP: 91.189.91.38 80]
Ign:7 https://packages.adoptium.net/artifactory/deb jammy InRelease
Err:8 http://security.ubuntu.com/ubuntu jammy-security Release
  Could not open file /var/lib/apt/lists/partial/security.ubuntu.com_ubuntu_dists_jammy-security_Release - open (30: Read-only file system) [IP: 185.125.190.36 80]
Err:10 https://packages.adoptium.net/artifactory/deb jammy Release
  Could not open file /var/lib/apt/lists/partial/packages.adoptium.net_artifactory_deb_dists_jammy_Release - open (30: Read-only file system) [IP: 146.75.31.42 443]
Ign:9 https://dl.secondarymetabolites.org/repos/deb stretch InRelease
Err:11 https://dl.secondarymetabolites.org/repos/deb stretch Release
  Could not open file /var/lib/apt/lists/partial/dl.secondarymetabolites.org_repos_deb_dists_stretch_Release - open (30: Read-only file system) [IP: 130.226.55.166 443]
Reading package lists... Done
W: chown to _apt:root of directory /var/lib/apt/lists/partial failed - SetupAPTPartialDirectory (30: Read-only file system)
W: chmod 0700 of directory /var/lib/apt/lists/partial failed - SetupAPTPartialDirectory (30: Read-only file system)
W: chown to _apt:root of directory /var/lib/apt/lists/auxfiles failed - SetupAPTPartialDirectory (30: Read-only file system)
W: chmod 0755 of directory /var/lib/apt/lists/auxfiles failed - SetupAPTPartialDirectory (30: Read-only file system)
W: Not using locking for read only lock file /var/lib/apt/lists/lock
W: Problem unlinking the file /var/lib/apt/lists/partial/.apt-acquire-privs-test.TJV3M3 - IsAccessibleBySandboxUser (30: Read-only file system)
W: Problem unlinking the file /var/lib/apt/lists/partial/.apt-acquire-privs-test.M93znf - IsAccessibleBySandboxUser (30: Read-only file system)
W: Problem unlinking the file /var/lib/apt/lists/partial/.apt-acquire-privs-test.1e8ngs - IsAccessibleBySandboxUser (30: Read-only file system)
W: Problem unlinking the file /var/lib/apt/lists/partial/.apt-acquire-privs-test.G9BiRJ - IsAccessibleBySandboxUser (30: Read-only file system)
W: Problem unlinking the file /var/lib/apt/lists/partial/.apt-acquire-privs-test.a4ZV2H - IsAccessibleBySandboxUser (30: Read-only file system)
W: Problem unlinking the file /var/lib/apt/lists/partial/.apt-acquire-privs-test.4b3Qrx - IsAccessibleBySandboxUser (30: Read-only file system)
W: Problem unlinking the file /var/lib/apt/lists/partial/.apt-acquire-privs-test.PeblqD - IsAccessibleBySandboxUser (30: Read-only file system)
W: chown to _apt:root of file /var/lib/apt/lists/partial/dl.secondarymetabolites.org_repos_deb_dists_stretch_InRelease failed - Item::QueueURI (30: Read-only file system)
W: chmod 0600 of file /var/lib/apt/lists/partial/dl.secondarymetabolites.org_repos_deb_dists_stretch_InRelease failed - Item::QueueURI (30: Read-only file system)
W: Problem unlinking the file /var/lib/apt/lists/partial/us.archive.ubuntu.com_ubuntu_dists_jammy_InRelease - PrepareFiles (30: Read-only file system)
W: chown to _apt:root of file /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_jammy_InRelease failed - Item::QueueURI (30: Read-only file system)
W: chmod 0600 of file /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_jammy_InRelease failed - Item::QueueURI (30: Read-only file system)
W: chown to root:root of file /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_jammy_InRelease failed - 400::URIFailure (30: Read-only file system)
W: chmod 0644 of file /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_jammy_InRelease failed - 400::URIFailure (30: Read-only file system)
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://us.archive.ubuntu.com/ubuntu jammy InRelease: Couldn't create temporary file /tmp/apt.conf.Z3BHzc for passing config to apt-key
W: Problem unlinking the file /var/lib/apt/lists/partial/us.archive.ubuntu.com_ubuntu_dists_jammy-updates_InRelease - PrepareFiles (30: Read-only file system)
W: Problem unlinking the file /var/lib/apt/lists/partial/download.docker.com_linux_ubuntu_dists_jammy_InRelease - PrepareFiles (30: Read-only file system)
W: chown to _apt:root of file /var/lib/apt/lists/download.docker.com_linux_ubuntu_dists_jammy_InRelease failed - Item::QueueURI (30: Read-only file system)
W: chmod 0600 of file /var/lib/apt/lists/download.docker.com_linux_ubuntu_dists_jammy_InRelease failed - Item::QueueURI (30: Read-only file system)
W: chown to root:root of file /var/lib/apt/lists/download.docker.com_linux_ubuntu_dists_jammy_InRelease failed - 400::URIFailure (30: Read-only file system)
W: chmod 0644 of file /var/lib/apt/lists/download.docker.com_linux_ubuntu_dists_jammy_InRelease failed - 400::URIFailure (30: Read-only file system)
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://download.docker.com/linux/ubuntu jammy InRelease: Couldn't create temporary file /tmp/apt.conf.LzHmDB for passing config to apt-key
W: Problem unlinking the file /var/lib/apt/lists/partial/us.archive.ubuntu.com_ubuntu_dists_jammy-backports_InRelease - PrepareFiles (30: Read-only file system)
W: chown to _apt:root of file /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_jammy-backports_InRelease failed - Item::QueueURI (30: Read-only file system)
W: chmod 0600 of file /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_jammy-backports_InRelease failed - Item::QueueURI (30: Read-only file system)
W: chown to root:root of file /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_jammy-backports_InRelease failed - 400::URIFailure (30: Read-only file system)
W: chmod 0644 of file /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_jammy-backports_InRelease failed - 400::URIFailure (30: Read-only file system)
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://us.archive.ubuntu.com/ubuntu jammy-backports InRelease: Couldn't create temporary file /tmp/apt.conf.BI6ryE for passing config to apt-key
W: Problem unlinking the file /var/lib/apt/lists/partial/security.ubuntu.com_ubuntu_dists_jammy-security_InRelease - PrepareFiles (30: Read-only file system)
W: Problem unlinking the file /var/lib/apt/lists/partial/us.archive.ubuntu.com_ubuntu_dists_jammy-updates_Release - PrepareFiles (30: Read-only file system)
E: The repository 'http://us.archive.ubuntu.com/ubuntu jammy-updates Release' no longer has a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: Problem unlinking the file /var/lib/apt/lists/partial/packages.adoptium.net_artifactory_deb_dists_jammy_InRelease - PrepareFiles (30: Read-only file system)
W: Problem unlinking the file /var/lib/apt/lists/partial/security.ubuntu.com_ubuntu_dists_jammy-security_Release - PrepareFiles (30: Read-only file system)
E: The repository 'http://security.ubuntu.com/ubuntu jammy-security Release' no longer has a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: chown to _apt:root of file /var/lib/apt/lists/partial/dl.secondarymetabolites.org_repos_deb_dists_stretch_InRelease failed - Item::QueueURI (30: Read-only file system)
W: chmod 0600 of file /var/lib/apt/lists/partial/dl.secondarymetabolites.org_repos_deb_dists_stretch_InRelease failed - Item::QueueURI (30: Read-only file system)
W: Problem unlinking the file /var/lib/apt/lists/partial/packages.adoptium.net_artifactory_deb_dists_jammy_Release - PrepareFiles (30: Read-only file system)
E: The repository 'https://packages.adoptium.net/artifactory/deb jammy Release' no longer has a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: chown to root:root of file /var/lib/apt/lists/partial/dl.secondarymetabolites.org_repos_deb_dists_stretch_InRelease failed - 400::URIFailure (30: Read-only file system)
W: chmod 0644 of file /var/lib/apt/lists/partial/dl.secondarymetabolites.org_repos_deb_dists_stretch_InRelease failed - 400::URIFailure (30: Read-only file system)
W: Problem unlinking the file /var/lib/apt/lists/partial/dl.secondarymetabolites.org_repos_deb_dists_stretch_Release - PrepareFiles (30: Read-only file system)
E: The repository 'http://dl.secondarymetabolites.org/repos/deb stretch Release' does not have a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: Problem unlinking the file /var/cache/apt/pkgcache.bin - RemoveCaches (30: Read-only file system)
W: Problem unlinking the file /var/cache/apt/srcpkgcache.bin - RemoveCaches (30: Read-only file system)

To get it out of the way, /tmp permissions are set correctly, and I certainly have nothing mounted in /tmp (checked multiple ways). One of the last things I did with the system last week was to mount an NFS storage volume to a new folder I created in / called /mnt. I'm not sure if that has anything to do with it. However, last week I was getting weird error messsages around 'dl.secondarymetabolites.org_repos_deb_dists_stretch_Release' whenever I ran anything with apt, which appears more likely to be part of the problem.

The problem seems to be centered around /var/lib/apt/lists/partial. I went to try and delete partial (or the one item I can see in it, which is 'dl.secondarymetabolites.org_repos_deb_dists_stretch_Release') and recreate the directory, but I apparently don't have permissions to do anything with that directory, including delete it.

Running ls -lah /var/lib/apt/lists/partial shows the following:

drwx------ 2 _apt root  20K Nov 12 20:29 .
drwxr-xr-x 4 root root  20K Nov 12 20:29 ..
-rw-r--r-- 1 root root 5.2K Apr 20  2019 dl.secondarymetabolites.org_repos_deb_dists_stretch_InRelease

when I try to remove the file "dl.secondarymetabolites.org_repos_deb_dists_stretch_InRelease" or the directory "options" or move the file around or anything, I get this:

rm: cannot remove 'partial/dl.secondarymetabolites.org_repos_deb_dists_stretch_InRelease': Read-only file system

Any insight into how I can get apt working again? (getting weird errors on all apt commands)