Hibernating (and then UN-hibernating) Ubuntu?

TreyH · August 3, 2022

I have “Hibernate” set in VM Manager as the shutdown action, and for years it’s worked flawlessly with my Windows 10 VM. I’d always shutdown and started up my Linux VMs manually, though. It was just when my husband had to power the box off to rearrange power cables the other day that I found out it wasn’t working on Ubuntu—Windows still thought its uptime was months, but Linux’s uptime is just since the power cycle.

So testing, I manually did “Hibernate” from the VM popup menu actions, and nothing happened (as far as I can tell from the logs, nothing even registered Ubuntu-side). Poking around the interwebs it seemed like the qemu-guest-agent service was the ticket, so I installed its apt, did

sudo systemctl start qemu-guest-agent

and tried “Hibernate” again. Seemed to work—after about 30 seconds, the machine showed as powered off in VMs and the host disappeared from my network.

But then when I did “Start”, it didn’t resume but booted anew. In syslog, I found (truncating around the moments the VM was off):

Aug  3 18:22:17 ubuntuvm systemd[1]: Started QEMU Guest Agent.
Aug  3 18:22:53 ubuntuvm gnome-shell[837]: Failed to set DPMS: Failed to set connector 37 property 2: Permission denied
Aug  3 18:22:53 ubuntuvm gnome-shell[837]: Screen lock is locked down, not locking
Aug  3 18:22:53 ubuntuvm NetworkManager[655]: <info>  [1659565373.9953] manager: sleep: sleep requested (sleeping: no  enabled: yes)
Aug  3 18:22:53 ubuntuvm NetworkManager[655]: <info>  [1659565373.9954] manager: NetworkManager state is now ASLEEP
Aug  3 18:22:53 ubuntuvm ModemManager[702]: <info>  [sleep-monitor] system is about to suspend
Aug  3 18:22:54 ubuntuvm systemd[1]: Reached target Sleep.
Aug  3 18:22:54 ubuntuvm systemd[1]: Starting Record successful boot for GRUB...
Aug  3 18:22:54 ubuntuvm systemd[1]: Starting Hibernate...
Aug  3 18:22:54 ubuntuvm kernel: [ 4607.208769] PM: Image not found (code -16)
Aug  3 18:22:54 ubuntuvm systemd[1]: grub-common.service: Succeeded.
Aug  3 18:22:54 ubuntuvm systemd[1]: Finished Record successful boot for GRUB.
Aug  3 18:22:54 ubuntuvm systemd[1]: Starting GRUB failed boot detection...
Aug  3 18:22:54 ubuntuvm systemd[1]: grub-initrd-fallback.service: Succeeded.
Aug  3 18:22:54 ubuntuvm systemd[1]: Finished GRUB failed boot detection.
Aug  3 18:22:54 ubuntuvm systemd-sleep[2912]: Suspending system...
Aug  3 18:24:08 ubuntuvm systemd-modules-load[390]: Inserted module 'msr'
Aug  3 18:24:08 ubuntuvm systemd-sysctl[401]: Not setting net/ipv4/conf/all/promote_secondaries (explicit setting exists).
Aug  3 18:24:08 ubuntuvm systemd-sysctl[401]: Not setting net/ipv4/conf/default/promote_secondaries (explicit setting exists).
Aug  3 18:24:08 ubuntuvm systemd[1]: Starting Flush Journal to Persistent Storage...

and looking back, the messages starting with “Inserted module ‘msr’” on were common to the previous reboots. IOW, it looks exactly like a hibernate and system sleep happened, but for whatever reason it wasn’t resumed.

I have no trouble with other virtio features.

Searching here, I’ve found multiple references to https://wiki.unraid.net/Manual/VM_Guest_Support but that page seems to only discuss Windows guests. (The page is organized as if it will discuss other guest OSes, but I can’t see anywhere it does so.)

I’d appreciate any pointers on how to proceed troubleshooting this!

——-

Additional info

This is Ubuntu 20.04.4 LTS. Besides the new package qemu-guest-agent, I already had the qemu-kvm service installed and it isn’t clear to me what it does, but checking its status gives me:

root# systemctl status qemu-kvm
● qemu-kvm.service - QEMU KVM preparation - module, ksm, hugepages
     Loaded: loaded (/lib/systemd/system/qemu-kvm.service; enabled; vendor preset: enabled)
     Active: active (exited) since Wed 2022-08-03 18:24:05 EDT; 55min ago
    Process: 578 ExecStart=/usr/share/qemu/init/qemu-kvm-init start (code=exited, status=0/SUCCESS)
   Main PID: 578 (code=exited, status=0/SUCCESS)
Aug 03 18:24:05 ubuntuvm systemd[1]: Starting QEMU KVM preparation - module, ksm, hugepages...
Aug 03 18:24:05 ubuntuvm systemd[1]: Finished QEMU KVM preparation - module, ksm, hugepages.

If you look above, the timestamp here (18:24:05) is actually just prior to the first post-reboot message appearing in syslog.

And, for completeness:

root# sudo systemctl status qemu-guest-agent
● qemu-guest-agent.service - QEMU Guest Agent
     Loaded: loaded (/lib/systemd/system/qemu-guest-agent.service; static; vendor preset: enabled)
     Active: active (running) since Wed 2022-08-03 18:24:08 EDT; 1h 2min ago
   Main PID: 668 (qemu-ga)
      Tasks: 1 (limit: 4608)
     Memory: 952.0K
     CGroup: /system.slice/qemu-guest-agent.service
             └─668 /usr/sbin/qemu-ga

Aug 03 18:24:08 ubuntuvm systemd[1]: Started QEMU Guest Agent.

ghost82 · August 4, 2022

It should be an issue within the os guest, not an issue with qemu or kvm.

The guest agent in the guest is needed to send some commands from the host to the guest.

There should be no need of the qemu-kvm package in the guest.

It seems that the swap disk cannot be found.

See if this helps with your issue:

https://bbs.archlinux.org/viewtopic.php?id=247036

It's for arch, but you can consider and search as a general linux issue.

Or this:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1962359

Or this:

https://forum.manjaro.org/t/some-clarification-on-hibernate-needed/108659/4

Edited August 4, 2022 by ghost82

TreyH · August 5, 2022

On 8/4/2022 at 1:07 PM, ghost82 said:

See if this helps with your issue:

https://bbs.archlinux.org/viewtopic.php?id=247036

It's for arch, but you can consider and search as a general linux issue.

Or this:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1962359

Or this:

https://forum.manjaro.org/t/some-clarification-on-hibernate-needed/108659/4

I think I’ve got too many cooks spoiling the broth here (with “cooks” meaning “conflicting apt packages”).

Imagining that one or the other of the many managed apt packages promising to solve “hibernation” would just solve the issue, I installed them one by one and tried hibernating, but none got farther. (A possible clue: when I’ve tried using each of those packages’ inside-guest CLI command to hibernate, they each suspended and turned off the VM, but on resume gave different error messages—none that have been particularly revealing—but, when I “Hibernate” from the Unraid VM interface, I consistently get the above “PM: Image not found (code -16)” message.

This was disappointing, especially with the pkg actually called “hibernate”, which goes on at some length in its manpage about how it tries multiple ways until it find one that works.

But reading through those links I haven’t found a solution yet. I have checked the contents of /sys/power/image_size (it seems fine—it is larger than either my total available memory or my currently used memory, but smaller than the size of my swap (at /swap.img), and the swap seems to be working.

I haven’t tried the “nomodeset” kernel parameter, but I’m using VNC graphics on this VM, so I don’t think it’s applicable…

I think my problem may be that my /etc/default/grub has no “resume=” boot command, so I think I’ll give that a try—but for that I need the stable block-device UUID for my swap. And as it’s an img file in the root filesystem, I’m not sure how to locate that.

Edited August 5, 2022 by TreyH

ghost82 · August 6, 2022

And this?

https://askubuntu.com/questions/6769/hibernate-and-resume-from-a-swap-file

First reply may contain some useful info.

TreyH · August 29, 2022

On 8/6/2022 at 1:47 AM, ghost82 said:

And this?

https://askubuntu.com/questions/6769/hibernate-and-resume-from-a-swap-file

First reply may contain some useful info.

Yes, it does, but following its directions exactly on Ubuntu 20.04 LTS under Unraid doesn’t work. It took me some fiddling to find the right combination, so I really hope this helps someone else out later!

These alterations are necessary because most Linux distros have recently (well, gradually over the past 5–10 years, Ubuntu LTS just a few years ago) switched from swap partitions to swap files. If you have an actual swap partition (fairly unlikely these days, but possible) then the steps in the AskUbuntu question above should work better for you.

(Funny, we (well, not me we—but people who were alive then) used swapfiles on Unixes all the way back to the 1970’s (!), switched to swap partitions in the 1990’s, usually abandoned swaps entirely in the next decade, and now we’re back to swap files—I’m not sure why we call them two-word “swap files” now when we used to call them one-word “swapfiles”, but, ya know…)

Anyway… Here’s what to do:

Locate your swap file—almost certainly it’s at “/swap.img” or “/swap”, but you can run “swapon” (not as root, and with no arguments, lest bad things happen!) to find out. (It should also be listed in /etc/fstab.) Run this (replacing “/swap.img” below if necessary), and make a note of the UUID generated (which I’ll call <SWAP_UUID> below). It should be a hexadecimal-dashed value in a 8-4-4-4-12 form like “b34dc0d3-5968-5a32-f4d9-f9e6b5d4afa6”.
```
sudo findmnt -no SOURCE,UUID -T /swap.img
```
Then run
```
sudo swap-offset /swapfile
```
and make a note of the number value (I’ll call it <OFFSET> below).
Save a backup copy of /etc/default/grub (in your home directory or somewhere else it will stick around between reboots, not /tmp!) and then edit it (with sudoedit or your favorite method for editing system files) to modify the “GRUB_CMDLINE_LINUX_DEFAULT” line that should already be there, but is probably currently blank. Insert the UUID from the last step at “<SWAP_UUID>”. (Yes, that does mean there are two equals signs in “resume=UUID=<SWAP_UUID>”, it’s not a typo…) It’s possible you’ve had to edit it before to make booting Linux work with a passthrough device; if so, keep those arguments while adding the “resume” and “resume_offset”.
```
GRUB_CMDLINE_LINUX_DEFAULT="resume=UUID=<SWAP_UUID> resume_offset=<OFFSET> quiet splash"
```
Run
```
sudo update-grub
```
Important: if you run into any trouble below this step and don’t have a working hibernate/resume, come back here and restore /etc/default/grub to its prior state and re-run this command. If your machine refuses to boot after an attempted hibernation, attach to its graphical console at startup time with VNC, and select a safe boot from the startup GRUB menu. [Continued below at bottom¹.]
Install the uswsusp package:
```
sudo apt install uswsusp
```
Configure it with:
```
sudo dpkg-reconfigure -pmedium uswsusp
```
1. Answer “Yes” to “Continue without a valid swap space?”
2. Do not select the /swap.img choice; instead, select the choice corresponding to the UUID above, probably “/dev/disk/by-uuid/<SWAP_UUID>”.
3. Answer “No” to encryption—unless on this Linux VM you’ve done the steps usually associated with getting a Windows gaming VM to work with native GPU/USB, there is no keyboard or graphics attached to the VM at resume time (which is a slightly weird state not really corresponding to traditional BIOS/UEFI startup), so there’s literally no way to enter the password because the VM doesn’t have any connection to the “outside world” when it asks. If you have done those GPU/USB steps and want to enable encryption, you probably have other work to do, and I can’t tell you what will work. But needing encryption on a shutdown VM in a running virtual host is surely quite a niche case…
When you finish the three questions in the uswsusp wizard, it will exit and should give messages showing it’s refreshing your initramfs config itself. But if for some reason it does not, (or, if for your own reasons you know you need to manually modify your initramfs.conf further) run:
```
sudo update-initramfs -u
```
But whether the wizard updates initramfs or you do, ensure it does not make any complaint of “no matching swap device is available”; if it does, you should look at a guide (there are many out there) for removing and recreating your swap file, and then start over with #1 above.
Now test that uswsusp hibernation works:
```
sudo s2disk
```
If you’re logged in via graphical console, you should see the machine shutdown. If you’re logged in via SSH, your connection should quickly go dead. Either way, refreshing the VMS tab of your Unraid console, you should see your VM as “Stopped”.
Start the VM again, just as if you were booting it cold (most easily, by clicking on its icon in the VMS tab and choosing “Start” from the pop-down.
Within seconds, your machine should be back up and you can SSH back in, or attach to its graphical console. Run “uptime” to verify that, as far as the OS is concerned, the VM never rebooted (that is, the uptime value should be back to the last time you rebooted the OS, not just the seconds or minutes it took to get here from clicking “Start” in this step). Any other running processes that don’t die when a network connection is interrupted should continue running normally.
You’ve manually hibernated and resumed now, but unfortunately this still won’t respond to Unraid asking Linux to hibernate itself. To do that, you need to modify systemd. Run:
```
sudo systemctl edit systemd-hibernate.service
```
You’ll have an editor with either a blank file, or a file that wasn’t working before. Clear it and replace it with:
```
[Service]
ExecStart=
ExecStart=/usr/sbin/s2disk 
ExecStartPost=/bin/run-parts -a post /lib/systemd/system-sleep
```
(The two “ExecStart=” lines, the first with an empty right-hand side, is intentional and required.) Save and exit the editor.
You can verify you now have a hibernate systemd function by running
```
systemctl status systemd-hibernate.service
```
It should give output including a mention of an “override.conf”.
Now test the systemd hibernate manually:
```
systemctl hibernate
```
It will likely ask you for your password several times. (Don’t worry, when Unraid requests a hibernation, it won’t need your password.) Once again the VM should shutdown, and you should do the same to bring it back up and verify it worked as in step 7.
Now you can test the real thing by selecting “Hibernate” from the VM’s pop-down menu in the Unraid VMS tab, waiting a few seconds, reloading to verify the VM is stopped, selecting “Start” from the VM’s pop-down menu, and one last time checking it worked as in steps 7 and 10. If this doesn’t work (whether by the VM’s failing to respond to the Hibernate, or its shutting down hard instead of hibernating, or appearing to hibernate but not successfully resuming) when everything before this did, make sure you’ve installed the recommended “guest” packages for your Linux distro on the VM. (It is usually necessary to reboot after installing the guest packages and before attempting a hibernate/resume cycle.)

Now, if you wish, you can set the VM to auto-hibernate when Unraid reboots. A timeout of 90 seconds should be more than sufficient (unless you have very large RAM and very slow disk, I suppose…).

——

¹ If after this step, you attempt to boot and

you don’t even get a GRUB safe boot option; or,
if you have no way to access the graphical console in order to interact with GRUB;

and so the machine

endlessly reboots,
hangs at boot, or
shuts itself back down immediately after boot;

you’ll need to boot from a live rescue installer image and restore GRUB.

To do that, you’ll either need to change the Unraid VM settings to include a live rescue installer as the OS Install ISO: option, or you’ll need to create a whole new VM with the live image and with the old disk image available as a primary or secondary disk via 9p or other such means.) And then follow its instructions (or the many guides online, such as this one) to repair a corrupted GRUB config. The Boot-Repair tool for Ubuntu (described here for if you accidentally install Windows on your preexisting Ubuntu machine) is a no muss, no fuss way that works in most situations (but I’ve actually never tested it in an Unraid VM—so you may need to fall back to one of the more manual options described in these guides!).

But…

Much safer and easier: If you have the available disk space, the safest and easiest thing to do is to make a backup copy of your entire VM’s boot disk image; that way, even if GRUB gets corrupted so badly it won’t let you boot into safe mode, you can revert back to the backup image of your Ubuntu installation before you modified it.

I happened to have installed a brand-new SSD just before starting this process, so I made image backups as manual “snapshots” at several points above before eventually landing on the above steps I could do start-to-finish.

Edited August 29, 2022 by TreyH

Darkman13 · November 6, 2022

Thanks a lot for all this info that helped me make the hibernation work on my VM (Zorin OS).

The only different thing I had to do is to resize the swapfile (default was 2GiB) based on the instructions in the provided link :

sudo swapoff /swapfile
sudo dd if=/dev/zero of=/swapfile bs=$(cat /proc/meminfo | awk '/MemTotal/ {print $2}') count=1024 conv=notrunc
sudo mkswap /swapfile
sudo swapon /swapfile

And install package uswsusp in order to have swap-offset available.

After that it works well now, with a strange behavior on my login screen that is in like 800x600 resolution, but after it is ok ^^

Thanks again

Letrab · October 24, 2023

Thanks for the wonderful guide.

For me, hibernating a new Ubuntu 22.04 VM let the VM shut down, if hibernate initiated from Unraid. (With the same logs as in the original post)

Hibernating via the menu in Ubuntu did work, and had the benefit that it showed "Suspended" as the VM state...

Also, running `virsh suspend <vm_id>` on the host worked perfect. So maybe this can be solved in the way how Unraid suspends a VM?

So there should be an optimal solution somehow? At least suspends works now with your guide!

As for Ubuntu 22.04, `uswsusp` is not installable from apt-get anymore...

But you can download it here: http://mirrors.kernel.org/ubuntu/pool/universe/u/uswsusp/uswsusp_1.0+20120915-6.2_amd64.deb (with Chrome it could be possible you have to klick "Keep file" in your Downloads).

And then install with `sudo dpkg -i uswsusp_1.0+20120915-6.2_amd64.deb`

I also had the strange, very blurred, login screen after resuming. But this could be solved by bypassing the login screen after hibernation (run in terminal):

`gsettings set org.gnome.desktop.screensaver ubuntu-lock-on-suspend 'false'`

Edited October 24, 2023 by Letrab

Hibernating (and then UN-hibernating) Ubuntu?

Recommended Posts

TreyH

Link to comment

ghost82

Link to comment

TreyH

Link to comment

ghost82

Link to comment

TreyH

Link to comment

Darkman13

Link to comment

Letrab

Link to comment

Join the conversation