2

In my company we use a USB bootable live stick with Ubuntu (erm... Xubuntu) 18.04 to do job interviews. It has a bunch of tools (three or four IDEs, a couple of database GUI viewers...) and a toy project to ask questions about.

That USB stick has a Casper-RW overlay partition for persistent storage (created with mkusb). Said partition is mostly used to store Docker images and to keep some start-up values that can change a bit depending on which machine the stick is plugged into.

That partition is mounted "manually" in /media/overlay by adding the following line mkdir --parents /media/overlay/ && mount -L casper-rw /media/overlay/ to the file rc.local (thank you @sudodus for answering my previous question).

Now, that casper-rw persistence layer is working great... Maybe too great. We would like to erase the /home/<user> directory where the .bash_history and other changes made by the interviewee have been stored when we shut down the test environment (test environment... meaning: the laptop where the interviewee has been doing stuff) so it's ready for the next candidate.

Thanks to these answers (Unix&Linux, AskUbuntu) I created a systemd service that... kindaaaa does what I want, it's just it seems to do it when the system boots, not when it's going down. Is not a huge deal, is just... it bugs me a little :-D

This is the systemd service I created in /etc/systemd/system/clear-stick.service:

[Unit]
Description=Cleans-up interview sticker
RequiresMountsFor=/media/overlay/
DefaultDependencies=no
Before=shutdown.target reboot.target halt.target umount.target casper.service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/clear_interview_stick.sh
ExecStop=/usr/local/bin/clear_interview_stick.sh
RemainAfterExit=true

[Install]
WantedBy=multi-user.target

I suspect it's only running the ExecStart line on boot, rather than the ExecStop line on shutdown. If I make changes to the /home/ directory, power off the computer and mount the stick somewhere else, the changes are still there. Plus, if I look at the logs in journalctl I only see references to Cleans-up interview sticker surrounded by what seem start sequences of other services.

I'm not 100% sure whether this happens because the .service definition is wrong (very likely) or because I don't fully understand how the casper-rw overlay works and even though the .service is run on shutdown, it can't clear the /media/overlay/home/interviewee directory?.

As I mentioned before, it's not a huge, huge deal, since it seems to be erasing the /home/ directory on boot, but I'd rather do it """right""" and erase it on shutdown.

And just in case it's relevant, this is the content of /usr/local/bin/clear_interview_stick.sh:

#!/bin/bash

had_to_mount="false";
if [[ ! -d "/media/overlay/upper/" ]]; then
    mount -L casper-rw /media/overlay/
    had_to_mount="true";
fi

if [[ -d "/media/overlay/upper/home/interviewee/" ]]; then
    echo "Removing interviewee /home";
    rm -rf "/media/overlay/upper/home/interviewee/";
else
    echo "interviewee /home/ not found"
fi

if [[ -f /root/.bash_history ]]; then
    rm /root/.bash_history;
fi

if [[ "${had_to_mount}" == "true" ]]; then
       umount /media/overlay;
fi
Savir
  • 153
  • 6
  • are you sure your casper is not unmounted before Before=shutdown.target reboot.target halt.target umount.target casper.service. Could you try if using After=final.target instead makes a difference? That one is called BEFORE unmounting takes place but before shutdown cs. – Rinzwind Aug 02 '19 at 12:51
  • To be honest, I'm not sure of anything at all @Rinzwind (thank you for the hint, btw!!) I know nothing about systemd. I will try and let you know. :-) It probably is unmounted, though. That'd explain why I had to add the mount to the bash script. – Savir Aug 02 '19 at 12:54
  • 1
    ahhhhh yes stupid me thinking wtf do you need a conditional mount there :D That is likely the issue. You are too late with your service :) You could also do it after network.target if not before final.target. If so: make it an answer yourself and accept it ;) – Rinzwind Aug 02 '19 at 12:59
  • Gonna try After=network.target (and no Before) But if it works, you sould make the answer and I'll gladly choose it. I mean... you have reputation to spare, but what's right is right :D – Savir Aug 02 '19 at 13:07
  • 1
    I can put more rep into a bounty than you have so go for it ;-) Ill upvote it – Rinzwind Aug 02 '19 at 13:59
  • So, I tried doing the After=network.target, removing all the Before(s) and making ExecStart=/bin/true. That didn't seem to work. Maybe I shouldn't have changed the ExecStart?? (that's what I get for getting creative!!) I'm gonna try substituting the After=network... by Before=final.target. I'll keep you posted (and again: that you so much for all these hints!!) – Savir Aug 02 '19 at 14:06
  • It's done. It's alive. It's aliiiiive :D – Savir Aug 03 '19 at 07:26

1 Answers1

1

FINALLY!! I got it!!!. Thank you so much to @Rinzwind who gave me crucial clues in the comments to the question for this.

(!!) If someone is reading these words, please: don't assume this is the right way of doing it because it probably isn't. I am really new to SystemD and I know mostly... well... absolutely nothing about it.

I got it working using a service that ExecStarts (important, I had been trying a lot just with ExecStop and didn't work) after shutdown.target and that is wanted by final.target.

  • /etc/systemd/system/clear-stick.service:

    [Unit]
Description=Cleans-up interview sticker

DefaultDependencies=no
Before=final.target umount.target
After=shutdown.target

[Service]
Type=oneshot
ExecStart=/usr/local/bin/clear_interview_stick.sh
RemainAfterExit=true

[Install]
WantedBy=final.target

It also helped seeing how the casper-rw overlay did it (I mean: looking at the actual /etc/systemd/system/final.target.wants/casper.service file) and how it uses ExecStart, makes it run after umount (although I wanted my service run before umount)... things like that.

Also, in one of my many tests I removed the line RequiresMountsFor=/media/overlay/ It might still be ok (I didn't try all the possible permutations), but since at the time that mount /media/overlay was being done through /etc/rc.local, (mkusb takes over the /etc/fstab), and what I did works without it, I kept it out.

In the previous paragraph, I mention "but since at the time that mount..." That's because just in case SystemD was not "seeing" a mount that wasn't listed in the /etc/fstab but created through rc.local, I also created a .mount unit file as explained here, to try having SystemD know about it. Plus it felt cleaner to me.

This is how:

  • /etc/systemd/system/media-overlay.mount:

    [Unit]
Description=Mount Overlay in /media/

[Mount]
What=/dev/disk/by-label/casper-rw
Where=/media/overlay
Type=ext4
Options=defaults

[Install]
WantedBy=multi-user.target

That didn't help with the erasing. I'm guessing maybe???? (big guess, big maybe) that's because I was still using ExecStop? I dunno, really. Nonetheless, that .mount file did succeeded in creating the mount, so I kept it.

I'm sure a cleaner and more reliable .service could be written, but since each test took me more than half an hour because I needed to rebuild the bootable USB stick, I decided leaving it like this. (If it compiles, it ships, right?)

It is worth noting that the list-dependencies command (as in systemctl list-dependencies --after final.target) was also really helpful to see what was going on.

Savir
  • 153
  • 6