2

I have a remote server (on the other side of the country) that has become corrupted in some way. I still have ssh access to it. Is it possible to overwrite the OS partition while it's running?

In the past I've managed to image a new SSD by using dd over ssh, but that computer was booted from a gparted usb stick. The only OS I have access to on that remote server is the one on the hard disk; no removable media is or will be available.

What will overwriting a currently running OS do? Can I load the disk image into a ram drive and run dd on the current disk? Perhaps I could make a second partition and dual boot with gparted or some other tool.

The image I will be writing to the disk has ssh with default credentials already enabled, so once I boot into the new image I should be able gain access.

This is my own hardware running at a remote location. The only access I have is through SSH

Bacon Snot
  • 51
  • what is "corrupted"? have you tried a file system repair first? or possiblly the software that is corrupted. or.... not much to go on .......the second partition idea seems like something I, myself would go with though, or some variation of that – WU-TANG Dec 02 '20 at 23:58
  • There are some dependency issues, missing / corrupt files after an update that are beyond my ability to remedy. My usual go to when my system gets messed up is to do a fresh install. For now my only access is SSH. I also see this as an interesting challenge. – Bacon Snot Dec 03 '20 at 00:20
  • I would not change too much on the original system at first... The problem is, when you reboot, if for whatever reason, the system doesnt come back up, youre locked out. If you are able to make a new partition and dd your new image to it.. then I would use the original system's update-grub to get a new grub menu entry... rather than reinstalling grub (from within your new system) and risking boot/efi problems... Once youve established you can reliable ssh into your new image/partition.. then you can dd another pre-made install for the original partition, but i dont think you can "install". – WU-TANG Dec 03 '20 at 00:30
  • Also the new pre-made install, you may want to consider installing kvm and putting your system in a VM... This would be WAAAAAAYYYYY easier if you had a corrupt VM that needed a remedy... and their are plenty of "remote" tools, simply because everything is remote by nature – WU-TANG Dec 03 '20 at 00:35
  • Also... the risk comes in when you have to change the default system to boot into in /etc/default/grub (which I always get wrong on first try!!!)... if designate the wrong one you could end up in memtest or something and be locked out... You never stated, do you have someone on the other end that, if need be, could make the selection???? – WU-TANG Dec 03 '20 at 00:44
  • update-grub isn't finding the new image. Could it be because both images are basically identical? What would happen if I ran grub-install on /dev/sda2? – Bacon Snot Dec 03 '20 at 04:45
  • @BaconSnot wait.... you "mounted" the new image on sda2??? or copied it to sda2??? I dont understand what youre doing there???? If all youve done was mounted it, then that mount will not exist when at boot! That OS needs to be copied(dd) to sda2 and grub needs to identify it update-grub so that it can recognized and boot into it... all that can be done before the first reboot.... THEN (the risk) you have to work on telling grub (/etc/default/grub -- GRUB_DEFAULT=_) to boot the correct entry... – WU-TANG Dec 03 '20 at 04:46
  • do not run grub install – WU-TANG Dec 03 '20 at 04:48
  • I mounted the partition from the image to /dev/loop0 then dd to /dev/sda2 – Bacon Snot Dec 03 '20 at 04:48
  • what is the image of???? – WU-TANG Dec 03 '20 at 04:50
  • The image is a disk image. I used this https://askubuntu.com/questions/69363/mount-single-partition-from-image-of-entire-disk-device to get just the partition from the image file. I can mount /dev/sda2 and see the root of the new OS – Bacon Snot Dec 03 '20 at 04:53
  • so it's a copy... that is more work and yes you will run into problems because the UUIDs are all the same... You originally said that you had an image all set up. I assumed you were doing like I would have done... at home, created a minimal install of a small linux disto, installed ssh and any tools i needed, and then dd the OS partition over to the remote machine via ssh to sda2.... Then run grub-update to pick up the new partition's OS.... copying the exact partition will cause problems that you definitely dont want to try to fix remotely. – WU-TANG Dec 03 '20 at 04:59
  • @BaconSnot.... ok i think i understand, it's not a copy of your system, it's a copy of a partition from an img??? I tried that before, I could not get it recognized by grub... Maybe it can be done, but I couldn't figure it out... I would just make my own small load, customize it, and send it over.... disregard what i said about the UUIDs and those problems... i thought you copied your own system... – WU-TANG Dec 03 '20 at 05:12
  • I got it to work using os-prober and update-grub – Bacon Snot Dec 05 '20 at 02:05

2 Answers2

3

Here is how I got it to work:

Step 1: Create a new partition. You can use any number of utilities to do this, I used cfdisk since it was available

Step 2: Image that new partition with a live distro that has SSH enabled and default username and password. In my case the old OS was on /dev/sda1 and the new one was on /dev/sda2. The new OS must be able to accept an SSH connection from boot without any prompts, or you will lose contact with your remote computer. It must also have the network properly configured. Either set a proper static IP, subnet mask, gateway, etc; or use DHCP and use your remote router to find the computer's new IP address.

Step 3: Run update-grub to add it to the boot options.

Step 4: View the file /boot/grub/grub.cfg and find the menuentry for the new OS image. It should look something like this: menuentry "KUbuntu 16.04 amd64 desktop (live)" --class windows --class os. What you need is the label in the quotes, in this case: KUbuntu 16.04 amd64 destop (live)

Step 5: Open /etc/default/grub in any editor and look for GRUB_DEFAULT=. Make sure it is set to GRUB_DEFAULT=saved.

Step 6: Run grub-reboot followed by your chosen menu entry in quotes. In this case grub-reboot "KUbuntu 16.04 amd64 desktop (live)"

Step 7: Run sudo reboot to get your remote computer to reboot. If you have done everything right you should be able to SSH into the new system once it boots.

Step 8: From your new operating system, overwrite the image of the original OS. In my case it was on /dev/sda1. This will also overwrite the grub settings on the boot partition causing you to boot into the updated OS on the original partition next time you boot. Again, what ever OS you put on that partition must have SSH enabled, default username and password, and network settings configured already.

Step 9: Reboot and you should be in your newly imaged operating system.

Bacon Snot
  • 51
  • congrats...very nice... i will bookmark this so i can have a checklist. i'd forgot all about network, could pose a real problem without dhcp. A question/remark or two though... Step 3 did you actually have to run os_prober manually? os_prober gets ran when you run update-grub (it's one of the scripts in /etc/grub.d/). Step 4 says "Edit" but I am assuming you mean just view it? If you edit it, it gets overwritten when you run update-grub. I'm not quite sure "it will boot into the last known option", more like it will boot into the "saved entry" based on how youre currently configured – WU-TANG Dec 06 '20 at 07:04
  • 1
    Step 5 I feel ridiculous not knowing all these years, you could use the OS name vice the enumeration. After looking into that, even more ridiculous, I found there is a much more safer approach. GRUB_DEFAULT=saved, and on the command line grub-set-default with grub-reboot can be used to set a 1-time boot selection. And even tabbing through those 2 commands even presents the OS options. I was able to boot into an older kernel 1-time with sudo grub-reboot "Advanced options for Ubuntu>Ubuntu, with Linux 5.4.0-54-generic" But, glad this worked out, I know this would have been hard to fix. – WU-TANG Dec 06 '20 at 07:10
  • I wish I knew about grub-reboot years ago. I updated the steps to reflect your suggestions. I think it's much easier and safer using 'grub-reboot'. I think I had to run os-prober because of my broken system specifically. Probably not necessary for everyone else. – Bacon Snot Dec 07 '20 at 14:20
  • While this is a great answer, how did you create the sda2 partition? Did you already have unpartitioned space available? Because as far as I know you cannot resize a partition while it is in use. – Terrance Dec 12 '20 at 00:32
  • 1
    You are correct, I did have unpartitioned space available. If you are in the same situation but have no unpartitioned space available, my first idea would be to turn swap off and use the swap partition. There are a few more ideas on how to shrink and create partitions here https://askubuntu.com/questions/390769/how-do-i-resize-partitions-using-command-line-without-using-a-gui-on-a-server. – Bacon Snot Dec 14 '20 at 14:23
  • @BaconSnot That is a great idea! I have now shared your answer with https://askubuntu.com/questions/1305659/im-done-with-mint-sorry-mint-and-want-to-move-to-move-to-ubuntu-i-have-ssh-a as I think it will help OP there as well. :) – Terrance Jan 07 '21 at 02:44
0

It is impossible to overwrite your system's installation while running that system.

You need to either have physical access to the server or access to tools provided by your server's host to do the tasks you want to do.

Consult with the hosting provider, their documentation, or the owner of the physical server to find out the method to complete your tasks.

Nmath
  • 12,333