docbillnet

It seems the most likely fundamemtal cause would have been one of the plugins i had installed. One of them was to do auto updates, and amy of the plugins could have had code that either deliberately or accidently changed the files that caused the failure. Since it really is not possible to carefully peer review every plugin in the community, it is probably best to only install ones i have reviewed and compiled myself.

It seems part of the problem is the cache. The redirects for HTTP to HTTPS are permanent redirects, meaning your browser will continue to try and use them when you do a reinstall. I bypassed that problem on my final procedure by using incognito mode. But as soon as I tried connecting without using incognito mode, I hit the "cannot log in" problem. This is only truly solved by either clearing the browser cache or re-enabling HTTPS and rebooting the NAS.

Presumably i need to restore my users and passwords. But all the docker services are running so that is not urgent.

I did a complete test of the usb using btrfs scrub of random data. No errors. So then i did a fresh install complete with the vfat formatting. Then to avoid browser cache i did the reconfigure in igncognito mode. This resulted in dialog for installing a license. The net result is everything worked. So it looks like both the bzfirmware and files related to licensing were corrupted. This leaves likely caused interrupted update or hackers. Hackers could overwrite files as sort of root kit, which would likely work until i rebooted. A failed update could leave corrupted files. So either cause fits with what i know. Only i was away, and i don't see any updates that were released in the last couple of days that would somehow auto-install.

Yes. I am thinking the next steo is just to go throygh all the steps for a freah install. I might aa well wait for the usb stick i ordered to arrive tomorrow to avoid that variable. Only three hypothesis make sense: 1. Usb drive failure. 2. Failed update 3. Hackers. The replacememt usb will remove the first possibility. The fresh install removes the second. The third seema unlikely, but i can search for unusual activity with my router.

My unRAID system has been working reliably since April. I left for a short trip to the US on Monday. On Tuesday, I noticed errors when attempting to save to Vaultwarden. I logged into the server in the evening to find that there was an error where none of the Docker containers were running. I tried restarting them, but I kept getting a message that they could not restart. The array looked healthy, but one CPU showed 80% usage, while the rest showed none. Finally, I decided it was best to restart. The system never came back up. Last night when I got home, I connected a monitor and keyboard to see why it would not start. I saw the checksum on the firmware file on the USB was not matching. I ran a memory check and dosfsck. I could find no evidence of failing hardware. So I downloaded Unraid's latest zip file and extracted that onto the USB. It then restarted and came up for the internet, but I could not log in as root. Not being sure what was wrong, I erased the shadow and smbpasswd. This allowed me to set a new password. But afterward, I still could not log in. I tried the password reset twice more, this time erasing the USB and essentially doing a fresh install. I simply cannot log in to my unRAID system to finish restoring it. I ordered a replacement USB. But it is a reputable brand from a reputable source and shows no signs of failure. I don't really have any good guesses as to what could have corrupted the file, what caused the NAS to crash in that manner, nor why I cannot log in with corrected files.

This is not a very well thought out procedure. Imagine you reach step 8 and accidently remove the wrong drive. As soon as you power up it will try to bring up the array again, and it will now detect two failed drives. Your array is now non-recoverable. Steps 1 - 7 are probably good. At that point remove the old drive and install the new drive. You should see when trying to assign the drive if you pulled the wrong one, and if so, since you are still offline you can restore the drive and try and find the correct one. I don't really see the point of power cycling the server at all. That seems redundant. BTW. This is not quite a hypothetical. I had forgotten I moved my parity drive from slot 1 to slot 4 because I found it got better cooling. So when I went to replace the last drive in the array, I accidently pulled the parity drive instead. Unfortunately, I was not dealing with a dead drive. I had not unassigned the drive I intended to remove. But I did not realise my mistake until I restarted the array. So I now get to rebuild my parity, before I can try replacing the correct drive. I'm actually not sure which drive it is now, so I will probably have to try one drive at a time while it is offline until I find the one with the correct serial number.

It is not. Initially I created all the filesystems with the default settings. Which is xfs for disk1, disk2, and disk3, and btrfs for cache. But then I remembered I wanted to try out zfs, so I changed disk1, disk2, and disk3 to zfs. Since zfs is sort of like the next generation xfs filesystem, I didn't expect any issues leaving the cache with the default file system.

This might be exactly the type of issue I was concerned about if I change the file system for my cache drive. Do you might asking what filesystem you are using? What error do you get when you do: rm -rf /mnt/cache/appdata/plex_broken/* ?

How so? It seems if the cache drive has the same settings as the drive in the arrays, then I would never be able to upload a file into the NAS to have it stuck in the cache... My concern is not if I can upload non-UTF-8 files. My concern is that when I just copy a folder it uploads and gets stuck... Meaning I have to manually fix the issue on the NAS instead of the device where I'm uploading from. It is certainly far less work to reform a cache that is routinely close to empty anyways, than to reformat mostly full hard drives. But my concern is there might be a really good reason why the cache defaults to BTRFS. Maybe zfs and xfs are unsuitable for the cache drive? Otherwise, I would expect unraid would have defaulted to use the same file system for the cache drive to avoid this type of issue.

If I am reformatting a drive, is there any reason not to convert the cache drive to zfs instead? It is not the UTF-8 only that is the problem, it is the inconsistency in options that is causing files to get stuck. Out of 20 TB's worth of files I think it is just 6 or so that appear not to be UTF-8, so it makes sense to convert those files. But I would save a lot of hassle of the uploads to the NAS failed, rather than having files fail to move once they are on the NAS. It is an odd default to make the cache drive btrfs without compression.

Looks like this is variation of the same issue I just reported for: Only in my case it was slightly worse. Because I could copy the files the network share, but then they a stuck on the cache drive. I know some filesystems strictly check the character encoding, while others don't. The files were originally generated by Windows users and copied to ext4 fs via sftp. Then I copied that onto my hard drive using btrfs. The cache drive also uses btrfs, but I have the hdd formatted in zfs. So my guess is all these other filesystems allow the windows encoded filenames without error, but zsh does not. But that is only a guess. In my case I'll resolve the issue by putting the folder with the offending filenames into a tar file. But that is hardly a good general solution.

Here a short way to reproduce: root@Higgs:/mnt/cache# base64 --decode <<+ |tar xfj - QlpoOTFBWSZTWUvfjaYAAGp/pMiAAGRAAf+AOARYwW7t3iABAAAAgAgwALm2IiNqGjQAAAADTIRM p6g0AAAAAABJEp6mQ0HqaAaA0A0Mll1eKiUYggomSSLzAkUyhhnammM2MvjrAwkDgycsHLbf3yrn n1Xea6jJwzCDf2kKAzDbYkIDuCkwESFJRO0E0oWE1aAI0ilgCMjNGKcLSwzEgyEhhOSkPJ7eycJ9 wMZTzaXC/IUdcAmBgpWp+gQIJOsJVCKAcn9k3i4lsxCQfxdyRThQkEvfjaY= + root@Higgs:/mnt/cache# cd /mnt/disk1 root@Higgs:/mnt/disk1# base64 --decode <<+ |tar xfj - QlpoOTFBWSZTWUvfjaYAAGp/pMiAAGRAAf+AOARYwW7t3iABAAAAgAgwALm2IiNqGjQAAAADTIRM p6g0AAAAAABJEp6mQ0HqaAaA0A0Mll1eKiUYggomSSLzAkUyhhnammM2MvjrAwkDgycsHLbf3yrn n1Xea6jJwzCDf2kKAzDbYkIDuCkwESFJRO0E0oWE1aAI0ilgCMjNGKcLSwzEgyEhhOSkPJ7eycJ9 wMZTzaXC/IUdcAmBgpWp+gQIJOsJVCKAcn9k3i4lsxCQfxdyRThQkEvfjaY= + tar: ./BAW Weekly Bookings Reconciliation Report \226 APAC Scheduled.XLS.gpg: Cannot open: Invalid or incomplete multibyte or wide character tar: Exiting with failure status due to previous errors This is simply extracting a uuencode tar file to each of the respective mount points. Assuming /mnt/cache is btrfs filesystem and /mnt/disk1 is zfs, I would expect similar results. The tar file is simply one of the files that was causing me issues truncated to 0 size, since it is only the filename matters.

Sample output going directly to the array, where /mnt/higgs/backups is an nfs mount to my NAS running unraid: # sudo rsync -aPs --links /share/Backups/ /mnt/higgs/backups/ root/var/lib/cronsrev/usrpftp.servicesource.com:20/From RedHat/NA/BAW Weekly Bookings Reconciliation Report \#226 NALA - Schedule.2015-03-28-09-05-03.xls.gpg": Input/output error (5) rsync: [generator] recv_generator: failed to stat "/mnt/higgs/backups/briemers2-root/var/lib/cronsrev/usrpftp.servicesource.com:20/From RedHat/NA/BAW Weekly Bookings Reconciliation Report \#226 NALA - Schedule.2015-04-07-10-45-55.xls.gpg": Input/output error (5) rsync: [generator] recv_generator: failed to stat "/mnt/higgs/backups/briemers2-root/var/lib/cronsrev/usrpftp.servicesource.com:20/From RedHat/NA/BAW Weekly Bookings Reconciliation Report \#226 NALA - Schedule.2015-04-11-09-05-28.xls.gpg": Input/output error (5) rsync: [generator] recv_generator: failed to stat "/mnt/higgs/backups/briemers2-root/var/lib/cronsrev/usrpftp.servicesource.com:20/From RedHat/NA/Downloads/BAW Weekly Bookings Reconciliation Report \#226 NALA - Schedule.2014-07-26-09-05-53.xls": Input/output error (5) rsync: [generator] recv_generator: failed to stat "/mnt/higgs/backups/briemers2-root/var/lib/cronsrev/usrpftp.servicesource.com:20/From RedHat/NA/Downloads/BAW Weekly Bookings Reconciliation Report \#226 NALA - Schedule.2014-07-26-09-05-53.xls.gpg": Input/output error (5) rsync: [generator] recv_generator: failed to stat "/mnt/higgs/backups/briemers2-root/var/lib/cronsrev/usrpftp.servicesource.com:20/From RedHat/NA/QA/BAW Weekly Bookings Reconciliation Report \#226 NALA - Schedule.2014-11-15-09-05-30.xls.gpg": Input/output error (5) I will see if I can uuencode the names into a small script, since the translation to post the output changes the character encoding...

I ran into an interesting problem. I have a set of files I can successfully copy to the cache drive, but they cannot the be moved to the array. The issue seems to be when filenames contain unicode characters. I don't know if it is all unicode characters or just some. The difference is the cache is using btrfs, and the array is using zsh. I would normally expect filenames to be stored using UTF8, since that is the defacto Linux standard. But this seems only true for btrfs, not zfs. The impact is if I have a share set to copy to the cache drive, the folder and all the files upload to the cache file but then get stuck. If I login and try manually moving the files I then see the problem is with the character encoding. If I set the files to copy directly to the array. then I simply receive an IO error when attempting to copy the files to the NAS. Normally I would avoid this type of issue in Ubuntu, Fedora, etc, by making sure I had a default local set in LC_ALL like en_CA.UTF-8. When I check in the console, I see there is the environmental variable set LANG=en_US.UTF-8, which should have the same effect, so I don't know why it is not working.