Error with sudo apt-get update; and Termnial won't open: Python update caused this?

Question

I'm in a bind :-( I'm hoping somebody can help me with this.

I have (had) a perfectly good working Ubuntu 18.04 system, with python2.7.17 and python3.6.9. I wanted to upgrade my Python to 3.8, so I followed some online help. (like here: How do I install Python 3.8 in Lubuntu 18.04?, and here: Error when trying sudo apt-get update - Python related? )

I installed Python3.8.0 (from Ubuntu channels, not PPA). I did update-alternatives --config python3 and in a moment of weakness, I also did update-alternatives --config python. Net-net, my python setup caused the Terminals to not open. I un-did these changes. I had to manually setup the link to python2 again using: $ sudo ln -s /usr/bin/python2.7 /usr/bin/python. But the problems remain.

Problems: Currently, the Terminal still won't open, and sudo apt-get update fails. :-(

(I do have a couple of terminals already open... but I'm afraid of rebooting the system, lest the Terminals not open up)

For various reasons, I have to remain on Ubuntu 18.04, and I cannot do a re-install. How do I fix this? Any help much appreciated!! Thank you!

Current setup is below:

avtester@a3computer:~$ update-alternatives --list python
update-alternatives: error: no alternatives for python
avtester@a3computer:~$ update-alternatives --list python3
/usr/bin/python3.6
/usr/bin/python3.8
avtester@a3computer:~$ 
...
avtester@a3computer:~$ ls -lt /usr/bin/python*
lrwxrwxrwx 1 root root      18 Nov 19 09:33 /usr/bin/python -> /usr/bin/python2.7
lrwxrwxrwx 1 root root      25 Nov 18 23:36 /usr/bin/python3 -> /etc/alternatives/python3
-rwxr-xr-x 2 root root 4526456 Oct  8 05:12 /usr/bin/python3.6
lrwxrwxrwx 1 root root      33 Oct  8 05:12 /usr/bin/python3.6-config -> x86_64-linux-gnu-python3.6-config
-rwxr-xr-x 2 root root 4526456 Oct  8 05:12 /usr/bin/python3.6m
lrwxrwxrwx 1 root root      34 Oct  8 05:12 /usr/bin/python3.6m-config -> x86_64-linux-gnu-python3.6m-config
-rwxr-xr-x 1 root root 3628976 Sep 30 06:38 /usr/bin/python2.7
lrwxrwxrwx 1 root root      33 Sep 30 06:38 /usr/bin/python2.7-config -> x86_64-linux-gnu-python2.7-config
-rwxr-xr-x 2 root root 4456208 Apr 17  2020 /usr/bin/python3.5
-rwxr-xr-x 2 root root 4456208 Apr 17  2020 /usr/bin/python3.5m
-rwxr-xr-x 1 root root 5203488 Oct 28  2019 /usr/bin/python3.8
lrwxrwxrwx 1 root root      33 Oct 28  2019 /usr/bin/python3.8-config -> x86_64-linux-gnu-python3.8-config
lrwxrwxrwx 1 root root      16 Oct 25  2018 /usr/bin/python3-config -> python3.6-config
lrwxrwxrwx 1 root root      10 Oct 25  2018 /usr/bin/python3m -> python3.6m
lrwxrwxrwx 1 root root      17 Oct 25  2018 /usr/bin/python3m-config -> python3.6m-config
lrwxrwxrwx 1 root root       9 Apr 16  2018 /usr/bin/python2 -> python2.7
lrwxrwxrwx 1 root root      16 Apr 16  2018 /usr/bin/python2-config -> python2.7-config
lrwxrwxrwx 1 root root      16 Apr 16  2018 /usr/bin/python-config -> python2.7-config
avtester@a3computer:~$ 
avtester@a3computer:~$ python --version
Python 2.7.17
avtester@a3computer:~$ python3 --version
Python 3.8.0
avtester@a3computer:~$

The failure for sudo apt-get update is here:

avtester@a3computer:~$ sudo apt-get update
Hit:1 http://us.archive.ubuntu.com/ubuntu bionic InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease      
Hit:3 http://security.ubuntu.com/ubuntu bionic-security InRelease       
Hit:4 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease
Traceback (most recent call last):                 
  File "/usr/lib/cnf-update-db", line 8, in <module>
    from CommandNotFound.db.creator import DbCreator
  File "/usr/lib/python3/dist-packages/CommandNotFound/db/creator.py", line 11, in <module>
    import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'
Reading package lists... Done
E: Problem executing scripts APT::Update::Post-Invoke-Success 'if /usr/bin/test -w /var/lib/command-not-found/ -a -e /usr/lib/cnf-update-db; then /usr/lib/cnf-update-db > /dev/null; fi'
E: Sub-process returned an error code
avtester@a3computer:~$ 

UPDATE 1 and 2 Following some help (see: https://stackoverflow.com/questions/56218562/how-to-fix-modulenotfounderror-no-module-named-apt-pkg), I remapped my python3 alternatives to original python3.6 instead of the newly installed python3.8, as below:

avtester@a3computer:~$ ls -l /etc/alternatives/python3 
lrwxrwxrwx 1 root root 18 Nov 18 23:36 /etc/alternatives/python3 -> /usr/bin/python3.8
avtester@a3computer:~$ sudo update-alternatives --config python3
[sudo] password for avtester: 
There are 2 choices for the alternative python3 (providing /usr/bin/python3).
Selection    Path                Priority   Status

0            /usr/bin/python3.8   2         auto mode
1            /usr/bin/python3.6   1         manual mode
2            /usr/bin/python3.8   2         manual mode

Press <enter> to keep the current choice[*], or type selection number: 1
update-alternatives: using /usr/bin/python3.6 to provide /usr/bin/python3 (python3) in manual mode
avtester@a3computer:~$ 
avtester@a3computer:~$ ls -l /etc/alternatives/python3 
lrwxrwxrwx 1 root root 18 Nov 19 10:37 /etc/alternatives/python3 -> /usr/bin/python3.6

And now, I can Open Terminals, and perform sudo apt-get install. HOWEVER, I see a very small list of updates? I'm not sure what that means.

avtester@a3computer:~$ sudo apt-get update
[sudo] password for avtester: 
Hit:1 http://us.archive.ubuntu.com/ubuntu bionic InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease      
Hit:3 http://security.ubuntu.com/ubuntu bionic-security InRelease        
Hit:4 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease     
Reading package lists... Done                      
avtester@a3computer:~$ 

What voodoo is this? If one updates the system python3 to a new version, why the does the system croak so badly??

Answer 1

for the record, please see my answer up in Update 1 and 2.

Update 1 and 2:

Following some help (see: https://stackoverflow.com/questions/56218562/how-to-fix-modulenotfounderror-no-module-named-apt-pkg), I remapped my python3 alternatives to original python3.6 instead of the newly installed python3.8, as below:

avtester@a3computer:~$ ls -l /etc/alternatives/python3 
lrwxrwxrwx 1 root root 18 Nov 18 23:36 /etc/alternatives/python3 -> /usr/bin/python3.8
avtester@a3computer:~$ sudo update-alternatives --config python3
[sudo] password for avtester: 
There are 2 choices for the alternative python3 (providing /usr/bin/python3).
Selection    Path                Priority   Status

0            /usr/bin/python3.8   2         auto mode
1            /usr/bin/python3.6   1         manual mode
2            /usr/bin/python3.8   2         manual mode

Press <enter> to keep the current choice[*], or type selection number: 1
update-alternatives: using /usr/bin/python3.6 to provide /usr/bin/python3 (python3) in manual mode
avtester@a3computer:~$ 
avtester@a3computer:~$ ls -l /etc/alternatives/python3 
lrwxrwxrwx 1 root root 18 Nov 19 10:37 /etc/alternatives/python3 -> /usr/bin/python3.6

Now I can open terminals, and perform sudo apt-get install, however I see a very small list of updates? I'm not sure what that means.

avtester@a3computer:~$ sudo apt-get update
[sudo] password for avtester: 
Hit:1 http://us.archive.ubuntu.com/ubuntu bionic InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease      
Hit:3 http://security.ubuntu.com/ubuntu bionic-security InRelease        
Hit:4 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease     
Reading package lists... Done                      
avtester@a3computer:~$ 

Answer 2

Started writing comment but it's already too long... :)

Anyway, Python is quite an essential package, as lots of base OS packages depend on it. Removing or messing up the Python makes other apps essentially lose their dependencies.

And if those apps are essential system apps, and they were written in Python, and your undoing causes them to miss Python modules or new Python version has breaking changes, then hell breaks loose.

As for using Python 3.8, if you really need it (eg. you are doing some development), seems you'd be best off with using it in separate environment (VM, container) or you should at least research using Python virtual environments. Start with this:

pip install virtualenv

As for short list of apt repos, it's just that... Repos it goes through. If you really want, you can add more.

P.S. Glad you managed to fix it already, that was actually quite quick. You should post your own answer with fix you did, for others.

Answer 3

This is a classic example of a compound problem, and it requires a compound solution.

I just went through the same thing, and used your question/answer (which I found searching on the cryptic sh 1: /usr/lib/cnf-update-db: not found error) and some others to figure out the solution.

First, I got myself into this mess with update-alternatives - I was just messing around with it after installing Python 3.7, 3.8 and 3.9. I somehow also munged the mapping for python, which should go to 2.7.

In my case (Ubuntu 18.04) it was the Python3 alternatives that was breaking apt, so I had to

sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.6 1

then

sudo update-alternatives --config python3

to select 3.6.1.

However, before I did that, I tried 20 other things focused on 'fixing package management', since the cryptic sh 1: /usr/lib/cnf-update-db: not found error is often a symptom of some sort of problem with packages. In this vein, on the advice of strangers on the internet I deleted a few different files and directories relating to packages. I might note here that in general, advice from strangers on the internet involving rm -rf anything should be treated very cautiously, but I was not cautious.

Anyway, after solving my update-alternatives problem, apt-get was 'working' again, but not actually working - I got a super short list and nothing would update or install.

What was required is to restore etc/apt/sources.list - from the command line, I just grabbed this from one of my other (working) Ubuntu 18 systems and copied it in. In my case, the file looks like this:

# deb cdrom:[Ubuntu 18.04.1 LTS _Bionic Beaver_ - Release amd64 (20180725)]/ bionic main restricted
See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
newer versions of the distribution.
deb http://ca.archive.ubuntu.com/ubuntu/ bionic main restricted
deb-src http://ca.archive.ubuntu.com/ubuntu/ bionic main restricted
Major bug fix updates produced after the final release of the
distribution.
deb http://ca.archive.ubuntu.com/ubuntu/ bionic-updates main restricted
deb-src http://ca.archive.ubuntu.com/ubuntu/ bionic-updates main restricted
N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
team. Also, please note that software in universe WILL NOT receive any
review or updates from the Ubuntu security team.
deb http://ca.archive.ubuntu.com/ubuntu/ bionic universe
deb-src http://ca.archive.ubuntu.com/ubuntu/ bionic universe
deb http://ca.archive.ubuntu.com/ubuntu/ bionic-updates universe
deb-src http://ca.archive.ubuntu.com/ubuntu/ bionic-updates universe
N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
team, and may not be under a free licence. Please satisfy yourself as to
your rights to use the software. Also, please note that software in
multiverse WILL NOT receive any review or updates from the Ubuntu
security team.
deb http://ca.archive.ubuntu.com/ubuntu/ bionic multiverse
deb-src http://ca.archive.ubuntu.com/ubuntu/ bionic multiverse
deb http://ca.archive.ubuntu.com/ubuntu/ bionic-updates multiverse
deb-src http://ca.archive.ubuntu.com/ubuntu/ bionic-updates multiverse
N.B. software from this repository may not have been tested as
extensively as that contained in the main release, although it includes
newer versions of some applications which may provide useful features.
Also, please note that software in backports WILL NOT receive any review
or updates from the Ubuntu security team.
deb http://ca.archive.ubuntu.com/ubuntu/ bionic-backports main restricted universe multiverse
deb-src http://ca.archive.ubuntu.com/ubuntu/ bionic-backports main restricted universe multiverse
Uncomment the following two lines to add software from Canonical's
'partner' repository.
This software is not part of Ubuntu, but is offered by Canonical and the
respective vendors as a service to Ubuntu users.
deb http://archive.canonical.com/ubuntu bionic partner
deb-src http://archive.canonical.com/ubuntu bionic partner
deb http://security.ubuntu.com/ubuntu bionic-security main restricted
deb-src http://security.ubuntu.com/ubuntu bionic-security main restricted
deb http://security.ubuntu.com/ubuntu bionic-security universe
deb-src http://security.ubuntu.com/ubuntu bionic-security universe
deb http://security.ubuntu.com/ubuntu bionic-security multiverse

Solution was immediate - apt-get working again, Python versions correct, and my alternatives are still available for venvs.

One final lesson I learned here is that when adding additional versions of Python to your system for dev/test, the use of virtual environments is even more crucial. In systems where the venvs all use the same base version of Python it's not an issue, but when you want to work in a different version, you need to ensure you don't add a 'convenience' link so that python3 uses your new version.