Run unRAID from a hard drive - the easy way.

[thohell]

Hi,

 

Long time reader, first time poster.
 

I recently decided to stop booting from, and writing to, the usb flash drive after two drives has gone bad. There seems to be some information on how to do this, but I could not find any method that was simple, clean, persistent and "set-and-forget".
 

I solved it by overlaying bzroot with a tiny initramfs and wrote up all instructions on how to do this here: https://github.com/thohell/unRAID-bzoverlay

 

This will also work well when running unRAID in a vm where boot from usb may not even be possible. No need to chain load using plop or other hacks. Just build a small boot image using these instructions.

 

Hopefully this information is useful to others.

Edited October 23, 2017 by thohell

[DieFalse]

Does this still work or is there a better way?    I have had my USB die off and the downtime was not fun nor was getting my backup to actually load correctly.

 

 

[thohell]

Hi,

I still run unRAID like this, and has had no problems whatsoever.
 

[m3lvm]

This is very interesting.

One question dose this stop unraid loading in to ram?

[uldise]

56 minutes ago, m3lvm said:

One question dose this stop unraid loading in to ram?

No, unraid is still loaded into RAM.  

[gerard6110]

1 - 2 years ago the USB 3.0 flash of my 2nd unRAID system became read-only. I had read about booting from a 2nd flash, while reading the license key from the original flash. But as it was a first, I just replaced the key. Now on my 3rd system, the USB 3.0 flash has also become read-only; after less than 5 months. No way to fix it, like with diskpart or EaseUS; it just seems a hardware failure (like the first one). So now I'm really interested in this bzoverlay solution.

 

However, I've now spent a number of evenings trying to get it to work (to no avail obviously).

For starters I'm a linux noob, but no problem figuring things out myself.

I'm running the latest stable version 6.8.3.

 

What I have done / tried:

- Prepare a flash drive with a single primary FAT32 partition - not bootable yet; just the partition - each time I tried another method.

 

Method 1. From the server itself,

- command mkfs.vfat ... cannot be found; so a real showstopper.

 

Method 2: From a PC, booting with either archiso or ubuntu server 20.04 from a USB stick

- This seems to work, but the BOOTDISK does not become bootable; in fact during the make_bootable_linux it seems to make the original disk bootable; also some message about tmp/BOOTDISK not found or so (I believe; all of my head).

- The syslinux.cfg on the BOOTDISK is also not patched with bzoverlay

 

Method 3:

- running mkfs.vfat ... on the alternative linux distro and add the unraid_disk directory for the mount point

- then insert it in the server, but than the mount point cannot befound, even using mount -t vfat ..., and/or adding chmod 777 to the /mnt/unraid_disk directory while still on the alternative linux distro.

 

Method 4:

- Like 3, but than mounting the drive using unassigned devices - but does did not help either

 

Also checked whether I could add mkfs.vfat to unRAID itself, but could not find the proper info to do so.

... not sure what else I tried. And yes, being a hobby programmer myself, very carefully typing the instructions or copying and inserting the correct device partition.

 

All above methods first using the original read-only flash with original license key.

Finally I replaced the  flash and thus license key, again to now avail. But it means I now have to hope the new flash does not die within the year ... it is a USB 2.0, which should not be an issue towards speed as with the bzoverlay only the license key would be read from it. The real bootdisk will become a 3.0 USB once this works.

 

One note: it seemed that the bzoverlay solution did work partly while using the orignal read-only flash. At least I have seen bzoverlay being loaded, but still unRAID was complaining that my flash drive was read-only - I also could not modify settings in any way (tried for example to change display settings).

 

I'm afraid that the mkfs.vfat command has been removed and also that may be the make_bootable_linux file has been modified.

As you can understand I'm quite interested in the solution, because if the new BOOTDISK dies it can easily be replaced without key-issues.

 

Would you care to have a look. Thanks.

 

Please let me know what you need from me to solve this. Like the proper messages ...

 

 

[DieFalse]

Having recovered many flash devices, I'm more curious what is causing your drives to become read only.   That's not normally a failure that occurs, and twice means something is doing it from the machine it's plugged into.   Is the USB onboard or add on? What motherboard and what bios version? Any other usb devices plugged in?

[gerard6110]

Twice is indeed not funny, but it was on two different machines. The one of 1-2 years ago is still running fine - it's also on 24/7.

The most recent one became read-only in a brand new machine with Ryzen 3900X and Gigabyte Aorus Master X570 mobo, with latest BIOS F11 - in itself running fine with two GameVMs, each with a Gigbyte NVidia 1660S vidoecard passthrough (occasional 1080p gaming is sufficient for me, also in view of in-house streaming - basically because it can; not because of real need). 64 GB of ram. 2x 1 TB nvme for redundant cache and 2x 4 TB of redundant storage. So yes quite a decent machine. And then a simple flash failing (like with an F1 car with failing spark plug 😉 ). This machine is not running 24/7; let's say 15 hours per week.

- I'll see if I can update my signature.

 

The flash was put in the white USB 3.0 slot (f); from the manual:

 

USB 3.1 Gen 1 Port (White)
The USB 3.1 Gen 1 port supports the USB 3.1 Gen 1 specification and is compatible to the USB 2.0
specification. Use this port for USB devices. Before using Q-Flash Plus (Note), make sure to insert the USB
flash drive into this port first.

 

The orignal flash was a small Patriot Tab 3.0 16 GB metal usb; but running quite hot?! Maybe too small in that it could not get rid of the heat?

I'm sure nothing is unintentionally writing to the flash. 

 

At that time a wired mouse was in the back of the machine, for passthrough using the libvirt plug-in.

 

PS. Maybe the white port has too much power on it (being used for Q-flash)?.

But I could not find anything on that and thus not able to reduce the power on it (in BIOS).

Edited May 9, 2020 by gerard6110
Add info about USB port power

[doron]

3 hours ago, gerard6110 said:

Method 1. From the server itself,

- command mkfs.vfat ... cannot be found; so a real showstopper.

You may wanna try mkfs.fat for that.

 

3 hours ago, gerard6110 said:

 

Method 2: From a PC, booting with either archiso or ubuntu server 20.04 from a USB stick

- This seems to work, but the BOOTDISK does not become bootable; in fact during the make_bootable_linux it seems to make the original disk bootable; also some message about tmp/BOOTDISK not found or so (I believe; all of my head).

The original make_bootable_linux script will try to make the disk with label UNRAID, bootable. So if it's the original flash, it will act unto it.

 

The proposed process from @thohell appears to have a slight bug, which might account for what you're seeing here. The editing of the "make_bootable_linux" script should probably be: 

sed -e "s|UNRAID|BOOTDISK|g" -i /mnt/unraid_disk/make_bootable_linux

instead of what's proposed (note the little "g" in the sed command). 

I have not tested this on a live system - just reading the code - but you may want to try this, and then run make_bootable_linux again.

 

3 hours ago, gerard6110 said:

Method 3:

- running mkfs.vfat ... on the alternative linux distro and add the unraid_disk directory for the mount point

- then insert it in the server, but than the mount point cannot befound, even using mount -t vfat ..., and/or adding chmod 777 to the /mnt/unraid_disk directory while still on the alternative linux distro.

Are you sure you used "/dev/sdX1" - emphasis on the "1" - or just "/dev/sdX"?

 

3 hours ago, gerard6110 said:

Also checked whether I could add mkfs.vfat to unRAID itself, but could not find the proper info to do so.

See above - try mkfs.fat.

 

 

[gerard6110]

Hi Doron,

Thanks for the tips.

Indeed I had seen mkfs.fat instead, but as vfat allows longer filenames I wasn't sure if using mkfs.fat would give (hidden) errors (and thus unexplanable issues lateron). So didn't try that.

 

I also checked the make_bootable_linux script after the sed patch command afterwards, and I still saw UNRAID mentioned (and thus not all instances of "UNRAID" replaced by "BOOTDISK"), but as mentioned being a linux noob, I could not check if that was OK or not. BTW, same with the syslinux patch command, where also not al "/bzroot" instances are replaced by "/bzroot,/bzoverlay". Should all the instances be replaced? If so I can do this manually, but I guess not.

 

I checked the sed command, but could not figure it out fully; like the "-e" option. To me it seemed that this "-e" was there to consider only the first instance per code line.

 

Will try your suggestions and post back.

[gerard6110]

So tried again with method 2 and g flag in the sed command.

Still errors in the make_bootable_linux command, as follows:

 

 

In my case:

sdc is the source USB - called UNRAID

sdd is the new target USB - at this moment called BOOTDISK

Target USB still not bootable.

[doron]

15 minutes ago, gerard6110 said:

So tried again with method 2 and g flag in the sed command.

Still errors in the make_bootable_linux command, as follows:

 

 

In my case:

sdc is the source USB - called UNRAID

sdd is the new target USB - at this moment called BOOTDISK

Target USB still not bootable.

Yes, so much for just reading the code and not testing...

The "make bootable" process has changed since those instructions were written, so that some more adjustments are in order.

Try this, in addition to previous edit (again, just offhand looking at code, no time to test right now):

sed "s|UNRAID|BOOTDISK|g" -i /mnt/unraid_disk/syslinux/make_bootable_linux.sh

Then run make_bootable_linux again. 

 

BTW, you mention "target USB" - I assume you mean target HDD, or else I'm not sure I understand what you are trying to do.

[gerard6110]

OK, will do.

I'm trying to boot from a second target USB, while reading the license key from the orginal flash source USB.

As the original flash source USB will not be written to anymore, possibly becoming read-only will not be an issue anymore.

The main and only reason is therefore to be able to easily exchange that target USB in case that would become corrupted; thus without having to claim a replacement license key, particularly within 12 months from an earlier replacement.

I didn't think USB or HDD would matter. Correct?

[doron]

39 minutes ago, gerard6110 said:

I didn't think USB or HDD would matter. Correct?

Correct. Thanks for clarifying.

[gerard6110]

It worked, but differently - still a lot of thanks to thohell for the bzoverlay and the initial idea.

Note: the make_bootable_linux installer itself does not work!

So I did it using Windows, as follows:

- Using the terminal of the server itself to download the bzoverlay file to /boot, with the wget instruction.

- Backing up the flash drive (and thus already with the bzoverlay).

- Extracting the backup.zip to the new flash labeled BOOTDISK

- Modifying the make_bootable.bat; replacing UNRAID by BOOTDISK (only once for tag)

- Running make_bootable.bat (as admin)

- Modifying syslinux/syslinux.cfg; replacing all instances of /bzroot by /bzroot,/bzoverlay (except /bzroot-gui)

- and then of course, changing the bootsequence in BIOS.

Basically that's it.

Detailed instructions I would have also.

[doron]

7 minutes ago, gerard6110 said:

Note: the make_bootable_linux installer itself does not work!

Great news! Happy to hear it worked for you.

Just out of curiosity - can you share what happened when you tried to run make_bootable_linux after doing both "sed" edits?

[gerard6110]

I guess I was a little bit too quick, because when looking at the logs, the following is repeated every 3 seconds.

 

Quote

kernel: sda: sda1
rc.diskinfo[10775]: SIGHUP received, forcing refresh of disks info.
unassigned.devices: Disk with serial 'Flash_Disk_12345678', mountpoint 'BOOTDISK' is not set to auto mount and will not be mounted.

Unquote

 

Not sure if really a problem, but at least I see the flash lighting up every 3 seconds, so cannot be good for the flash itself (although it can now be replaced quite easily - as I did it twice this evening with the same flash usb - first time to try and 2nd time to check and complete my instructions.

 

As to make_bootable_linux.

I went back to the basics and just tried it without any changes and the USB did not boot, so then I stopped going with the linux method.

I could try it again just for the sake of the messages, but most or all had to do with mounting and unmounting.

[DieFalse]

Use "unassigned devices" plug in and set to auto-mount as "BOOTDISK"

[doron]

16 minutes ago, gerard6110 said:

As to make_bootable_linux.

I went back to the basics and just tried it without any changes and the USB did not boot, so then I stopped going with the linux method.

Yeah that won't work.

16 minutes ago, gerard6110 said:

I could try it again just for the sake of the messages, but most or all had to do with mounting and unmounting.

I just tried it on my prod system (shudder). It seems to be doing the right thing (on the right disk...) although it does spit out one error message re unmounting - I believe this is a bug in the original script, has nothing to do with this exercise - but it does seem to work; I have not tried to boot from it (as I said, my main system), may try later.

If you're in for the game, you may want to try it - the process will be more streamlined.

 

16 minutes ago, gerard6110 said:

 

I guess I was a little bit too quick, because when looking at the logs, the following is repeated every 3 seconds.

 

Quote

kernel: sda: sda1
rc.diskinfo[10775]: SIGHUP received, forcing refresh of disks info.
unassigned.devices: Disk with serial 'Flash_Disk_12345678', mountpoint 'BOOTDISK' is not set to auto mount and will not be mounted.

Unquote

 

Not sure if really a problem, but at least I see the flash lighting up every 3 seconds, so cannot be good for the flash itself (although it can now be replaced quite easily - as I did it twice this evening with the same flash usb - first time to try and 2nd time to check and complete my instructions.

It has to do with the UD plugin. Can't be good that it gets a SIGHUP every 3 seconds - not sure why that'd happen.

 

What if you click on the disk in the UD section in the main UI page (the little plus sign near the hdd icon), and set its mount point to be /boot rather than BOOTDISK ? Does the log flood quiet down?

 

[gerard6110]

@fmp4m: Indeed I did set it to automount, but then I got all kinds of continuous RED errors, so I thought that can't be good and thus deslected the automount again, only leaving the above SIGHUP errors.

[gerard6110]

@doron:

 

make_bootable_linux: This I tried after extracting the backup.zip to the usb and then inserting the usb in an ubuntu 20.04 system and running make_bootable_linux. Then rebooting and it did not boot.

 

SIGHUP messages: yes, it was clear this had to do with the UD plugin. So I already tried setting it to automount, but then I go a lot of RED I/O errors, like:

May 13 20:15:44 Tower kernel: sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
May 13 20:15:44 Tower kernel: sd 0:0:0:0: [sda] tag#0 Sense Key : 0x6 [current]
May 13 20:15:44 Tower kernel: sd 0:0:0:0: [sda] tag#0 ASC=0x28 ASCQ=0x0
May 13 20:15:44 Tower kernel: sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 00 00 02 08 00 00 f0 00
May 13 20:15:44 Tower kernel: print_req_error: I/O error, dev sda, sector 520
May 13 20:15:44 Tower kernel: print_req_error: I/O error, dev sda, sector 760
May 13 20:15:44 Tower rc.diskinfo[10768]: SIGHUP received, forcing refresh of disks info.

 

Note: sectors change everytime and also opcode changes.

 

I also tried to change BOOTDISK to /boot, but then again the same "not set to automount" messages

and tried /boot and automount, and again the opcode and sector messages.

 

So now back to BOOTDISK and not automount (with SIGHUP messages).

Edited May 13, 2020 by gerard6110

[doron]

4 hours ago, gerard6110 said:

So now back to BOOTDISK and not automount (with SIGHUP messages).

This is not a stable state. I wouldn't declare victory just yet.

Now, the SIGHUP messages come from the Preclear Disk plugin (not UD), which I presume you have installed. It seems to be signalling itself whenever it sees a change in the mounted partitions table, or when it times out at 3 sec accessing disk info. Seems like it does the latter, and I can only guess it somehow doesn't get along with one of your USB drives (either the boot drive or the "license" drive). 

 

What happens if you disable the Preclear Disk plugin (just for testing)? Does everything quiet down or do the UD messages keep popping out every 3 seconds?

[gerard6110]

OK, removed the PreClear Disks plugin and rebooted.

No more repeating messages. Also checked the system log; no errors; Yesss! Thanks alot doron.

 

Only some warnings (although there were also warnings even when booting from the original UNRAID flash), as follows:

 

May 14 18:29:31 Tower kernel: ACPI: Early table checksum verification disabled
May 14 18:29:31 Tower kernel: Initramfs unpacking failed: Input was encoded with settings that are not supported by this XZ decoder
May 14 18:29:31 Tower kernel: floppy0: no floppy controllers found
May 14 18:29:31 Tower kernel: random: 7 urandom warning(s) missed due to ratelimiting
May 14 18:29:31 Tower kernel: igb 0000:06:00.0 eth0: mixed HW and IP checksum settings.
May 14 18:29:31 Tower kernel: ccp 0000:0e:00.1: psp initialization failed
May 14 18:29:31 Tower kernel: ACPI Warning: SystemIO range 0x0000000000000B00-0x0000000000000B08 conflicts with OpRegion 0x0000000000000B00-0x0000000000000B0F (\GSA1.SMBI) (20180810/utaddress-204)
May 14 18:29:31 Tower kernel: igb 0000:06:00.0 eth0: mixed HW and IP checksum settings.
May 14 18:29:31 Tower kernel: igb 0000:06:00.0 eth0: mixed HW and IP checksum settings.
May 14 18:29:31 Tower kernel: igb 0000:06:00.0 eth0: mixed HW and IP checksum settings.
May 14 18:29:31 Tower kernel: igb 0000:06:00.0 eth0: mixed HW and IP checksum settings.
May 14 18:29:34 Tower kernel: igb 0000:06:00.0 eth0: mixed HW and IP checksum settings.
May 14 18:29:40 Tower rpc.statd[2191]: Failed to read /var/lib/nfs/state: Success
May 14 18:29:56 Tower kernel: igb 0000:06:00.0 eth0: mixed HW and IP checksum settings

 

A;though everything seems to working fine, anything to worry about? 

From memory I believe ACPI Early ... and random: 7 are no issue. floppy0 also not of course - who still uses floppies nowadays 😃.

It seems to me that initramfs, igb and ACPI Warning are new.

 

Then mounted the BOOTDISK flash, and it was succesfully mounted.

Thus set it to automount and still OK, no warning messages.

Except for the few above warnings things seems fine to me.

 

Edited May 14, 2020 by gerard6110
Added automount message

[doron]

Umm, not sure we're there yet. I'm concerned with this one:

 

3 hours ago, gerard6110 said:

 

May 14 18:29:31 Tower kernel: Initramfs unpacking failed: Input was encoded with settings that are not supported by this XZ decoder

This might explain other stuff too.

Can you paste the output of the command "mount" (before you try the suggested fix below)?

 

Then, would you care to try the attached bzoverlay I made - just put it on your BOOTDISK instead of the previous one, and test again.

Finally, paste the output of "mount" again (after the change).

 

If this fix removes the above error message, you may want to enable the plugin you disabled previously, and see if you get any anomalies.

bzoverlay

[gerard6110]

@doron:

Please note the output of "mount" with thohell's bzoverlay: 

 

proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
tmpfs on /dev/shm type tmpfs (rw)
tmpfs on /var/log type tmpfs (rw,size=128m,mode=0755)
/dev/sdb1 on /boot type vfat (rw,noatime,nodiratime,flush,dmask=77,fmask=177,shortname=mixed)
/boot/bzmodules on /lib/modules type squashfs (ro)
/boot/bzfirmware on /lib/firmware type squashfs (ro)
hugetlbfs on /hugetlbfs type hugetlbfs (rw)
/mnt on /mnt type none (rw,bind)
tmpfs on /mnt/disks type tmpfs (rw,size=1M)
/dev/md1 on /mnt/disk1 type xfs (rw,noatime,nodiratime)
/dev/nvme0n1p1 on /mnt/cache type btrfs (rw,noatime,nodiratime)
shfs on /mnt/user0 type fuse.shfs (rw,nosuid,nodev,noatime,allow_other)
shfs on /mnt/user type fuse.shfs (rw,nosuid,nodev,noatime,allow_other)
/dev/sda1 on /mnt/disks/BOOTDISK type vfat (rw,noatime,nodiratime,nodev,nosuid,umask=000)
/mnt/cache/system/docker/docker.img on /var/lib/docker type btrfs (rw)
/mnt/cache/system/libvirt/libvirt.img on /etc/libvirt type btrfs (rw)

 

After reboot, with new doron bzoverlay: Sorry to tell you, but no full boot ...