fritzdis

What do you think the chances are the drive will be totally fine? I might just set all the shares to cache yes to clear things off, then to cache no once it's cleared (until I have more time to deal with this). If I had thought of doing that earlier, I think I wouldn't have posted right away (hard not to panic a bit when errors like that show up). So assuming I'm able to move everything without issue, I'll probably mark this solved for now and follow up once I'm better able to address it.

I also forgot that I just installed the My Servers plugin (before the reboot). But I was looking at the log because the flash drive didn't seem to backup immediately (it's fine now). I did not see those errors occurring, and I'm pretty sure I had dockers running at the time that would have triggered the drive issue if it existed before the reboot.

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 12562432 1 loop /lib/modules loop1 7:1 0 20738048 1 loop /lib/firmware loop2 7:2 0 42949672960 0 loop /var/lib/docker sda 8:0 0 4000787029504 0 disk └─sda1 8:1 0 4000785104896 0 part sdb 8:16 1 4012900352 0 disk └─sdb1 8:17 1 4010803200 0 part /boot sdc 8:32 0 960197124096 0 disk └─sdc1 8:33 0 960197091328 0 part sdd 8:48 0 480103981056 0 disk └─sdd1 8:49 0 429496696832 0 part /mnt/cache sde 8:64 0 12000138625024 0 disk └─sde1 8:65 0 12000138575360 0 part That discrepancy for sdd1 definitely stands out. Like I said, I'm basically certain it wasn't an issue before the reboot, but maybe the power cycle on the drive triggered something. sdc is the other drive in the pool. I would kinda like to have a bigger cache, though it would be nice to identify the issue. Happy to do any suggested troubleshooting, but if it's going to be a major pain to figure out, I don't mind just moving on.

Quick summary since this turned into a very detailed post: There's probably a physical issue with one of my cache drives (RAID1), but the SMART info looks OK. I'm hoping someone can chime in on what happened and what I should do next. I shut down my server to install a new drive. When I booted up & started the array, I started getting repeating errors in the log (didn't check the log right away): Dec 11 23:52:12 SF-unRAID kernel: BTRFS error (device sdd1): bdev /dev/sdd1 errs: wr 44138, rd 22, flush 0, corrupt 0, gen 0 Dec 11 23:52:17 SF-unRAID kernel: handle_bad_sector: 23 callbacks suppressed Dec 11 23:52:17 SF-unRAID kernel: attempt to access beyond end of device Dec 11 23:52:17 SF-unRAID kernel: sdd1: rw=34817, want=889003608, limit=838860736 Dec 11 23:52:17 SF-unRAID kernel: btrfs_dev_stat_print_on_error: 23 callbacks suppressed This device is part of a BTRFS RAID1 cache pool. Searching for that error brought me to this thread indicating a drive error. Given the info there, I started a non-correcting BTRFS scrub, which produced this output: Scrub started: Sun Dec 12 00:26:12 2021 Status: finished Duration: 0:03:31 Total to scrub: 205.88GiB Rate: 995.87MiB/s Error summary: read=7228792 Corrected: 0 Uncorrectable: 0 Unverified: 0 The log also showed some new errors: Dec 12 00:28:53 SF-unRAID kernel: BTRFS warning (device sdd1): i/o error at logical 1534910537728 on dev /dev/sdd1, physical 455527899136, root 5, inode 259, offset 6868205568, length 4096, links 1 (path: system/docker/docker.img) Dec 12 00:28:53 SF-unRAID kernel: BTRFS error (device sdd1): bdev /dev/sdd1 errs: wr 1996317, rd 7292557, flush 0, corrupt 0, gen 0 Since everything pointed to a drive issue, I ran short & long SMART self-tests, which completed without error. The SMART attributes look good to me, except for 233: # ATTRIBUTE NAME FLAG VALUE WORST THRESHOLD TYPE UPDATED FAILED RAW VALUE 5 Reallocated sector count 0x0033 100 100 005 Pre-fail Always Never 0 9 Power on hours 0x0032 091 091 000 Old age Always Never 44866 (5y, 1m, 12d, 10h) 12 Power cycle count 0x0032 099 099 000 Old age Always Never 38 175 Program fail count chip 0x0032 100 100 000 Old age Always Never 0 176 Erase fail count chip 0x0032 100 100 000 Old age Always Never 0 177 Wear leveling count 0x0012 094 094 000 Old age Always Never 961 178 Used rsvd block count chip 0x0013 095 095 005 Pre-fail Always Never 472 179 Used rsvd block count tot 0x0012 095 095 000 Old age Always Never 253 180 Unused rsvd block count tot 0x0012 095 095 000 Old age Always Never 10024 181 Program fail count total 0x0032 100 100 000 Old age Always Never 0 182 Erase fail count total 0x0032 100 100 000 Old age Always Never 0 194 Temperature celsius 0x0032 069 049 000 Old age Always Never 31 195 ECC error rate 0x0032 100 100 000 Old age Always Never 0 198 Offline uncorrectable 0x0030 100 100 000 Old age Offline Never 0 199 CRC error count 0x003e 100 100 000 Old age Always Never 0 202 Exception mode status 0x0033 100 100 090 Pre-fail Always Never 0 233 Media wearout indicator 0x0032 001 001 000 Old age Always Never 518009717538 Notably, #177 to #180 don't appear to show an issue, so I'm not sure if #233 is a good indicator of the SSD life or not. I have logs from 2016 & 2019 showing the #233 value already at 001 (much lower raw than current), whereas #177 decreased from 099 to 097 to 094. I'm inclined to think #177 is the standard wearout indicator for this drive. But #178 to #180 are unchanged, so I'm not sure what to think. I viewed the logs very recently, so I'm basically certain this started with the reboot. I did a few things: Move the server slightly Added a drive to the hot-swap bay Replaced the UPS battery (had been putting it off unwisely) Booted with an external drive attached (which now comes up as sda, though I doubt it's relevant) So my questions are: Is this definitely a physical drive issue? If so, I'll plan to replace with a larger SSD to match the 2nd one - is there any issue with that? (I'll look for instructions if that's the case, but if anyone wants to point me in the right direction or offer tips, that would be appreciated) If not, what further diagnostic steps should I take? Should I try the correcting scrub? Why did this start upon reboot? Maybe the SSD went into read-only mode when power cycled (how would I tell?) Presumably, this will fill up my log unless I shut down my dockers, so I may have to reboot. Please let me know if there's any other info I should capture before doing so. sf-unraid-diagnostics-20211212-0145.zip

Unfortunately, I was already using IP addresses. Tried 10.253.0.1 and 192.168.1.xxx with no luck on Windows (for the WebUI). For further testing, I tried accessing the built-in FTP server from Windows and Android. On Windows, I could browse some directories and download very small files, but large directory listings timed out, and downloads of most files did not work. On Android, everything seemed to work OK, including large file downloads. I've now installed BlueStacks on Windows to emulate Android with Wireguard installed. Through BlueStacks, I am able to access the WebUI, Dockers, and download files (speed not great). It's not ideal, but it's better than using my phone. It certainly seems like it's the Windows client or Windows itself doing something weird. I'm thinking playing around with MTU settings might have some impact. I can't do that right now, however, because when I make config changes on Chrome remotely, the interface stays inactive.

I'm having trouble accessing the WebUI through Wireguard on Windows 10. However, I can SSH into the server from the same Windows machine. I can also access the WebUI from an Android phone using the same tunnel/peer configuration on the same remote network (WiFi, not cell connection). I tried disabling the Windows firewall entirely, just in case that was interfering, but I still couldn't access the WebUI. Any thoughts?

EDIT: After more digging, I have a guess at what is happening: The errors from the previous read failures may be permanently stored in the drive's SMART error log. After preclear completion, the plugin gathers a SMART report that presumably includes these previous errors and includes them in the preclear log. It's possible that the plugin tries to determine whether these errors were logged before the preclear run. It could compare the "power on hours" at the start of the preclear to the "power on hours" of the logged errors. In my case, however, both values were 4772, so there may be no way for the plugin to know that the errors preceded the preclear run. Does this seem correct? If so, it seems like an unavoidable and rare quirk, and I'm happy to move on. ORIGINAL POST: preclear_disk_WD-WX11DC57S8AX_94806.txt sf-unraid-diagnostics-20190822-2211.zip

Thanks. I had to hold the power button to get it to shut down, but it didn't indicate an unclean shutdown when I booted back up. I changed the disk format for disk1, brought the array online, formatted the disk, and created the clear-me folder. I've modified the script, replacing the umount $d line with the following: if ! umount $d; then echo -e "\rFailed to unmount disk. Exiting ..." logger -tclear_array_drive "Failed to unmount disk. Exiting ..." exit fi I'm just going by the answer here. Do you think this is sufficient to ensure the disk is unmounted? I guess it would exit if the drive somehow became unmounted during the 60 second wait before the umount command is issued, so it's not bulletproof.

I went ahead and closed the script window, which did bring the Web interface back. I downloaded diagnostics (attached), and then tried to stop the array. That seems to have caused me to lose the interface, but I still have a console session. "top" shows unraidd still working, and the syslog shows the following error repeated many times: Oct 22 15:53:41 SF-unRAID kernel: XFS (md1): xfs_log_force: error -5 returned. Is there a safe way to shut down at this point? sf-unraid-diagnostics-20161022-1349.zip

When running this, the following was displayed: Unmounting Disk 1 ... umount: /mnt/disk1: target is busy It still seems to be clearing. What is the purpose of unmounting? Will I be able to remove the drive when it's done? EDIT: The display says it's going crazy slow: 23768072192 bytes (24 GB, 22 GiB) copied, 35769.9 s, 664 kB/s What would the consequences be of closing the window?

Did you try what I mentioned in my edit? To be clear, this is what I meant by "accessed the container filesystem". So far, the issue has not resurfaced. I might have also done "New Permissions" on the cache drive at some point in the process. If that had not worked for me, my next steps would have been to try this: Disable docker Delete docker.img Delete /mnt/cache/appdata/Crashplan/cache/ if it exists* Enable docker Re-add docker containers using CA's previous apps * Assuming /config is mapped to /mnt/cache/appdata/CrashPlan - adjust as necessary

Edit: I was able to fix this (for now at least). I accessed the container filesystem, copied /usr/local/crashplan/cache to /config/cache, and restarted the container as suggested here. Docker usage immediately dropped about 25%, and Crashplan did not have to re-synchronize anything. Original Post: I think my Crashplan is storing its cache within the docker file rather than in its appdata folder. Following the suggestion here, I found several very big files relating to crashplan: root@SF-unRAID:/# find /var/lib/docker/btrfs -type f -size +125000k -exec ls -lh {} \; | awk '{ print $9 ": " $5 }' /var/lib/docker/btrfs/subvolumes/84cd5cba4749aeacfb6b5fae612de48bd840eff3036bc1cd3cbfce588c81cb96/usr/local/crashplan/cache/42/cpfmf: 538M /var/lib/docker/btrfs/subvolumes/84cd5cba4749aeacfb6b5fae612de48bd840eff3036bc1cd3cbfce588c81cb96/usr/local/crashplan/cache/42/cphdf: 867M /var/lib/docker/btrfs/subvolumes/84cd5cba4749aeacfb6b5fae612de48bd840eff3036bc1cd3cbfce588c81cb96/usr/local/crashplan/cache/42/cpfmfp: 199M /var/lib/docker/btrfs/subvolumes/84cd5cba4749aeacfb6b5fae612de48bd840eff3036bc1cd3cbfce588c81cb96/usr/local/crashplan/cache/42/cpbf0000000000000000000/cpbmf: 2.4G A listing of the crashplan folder shown above returns the following: root@SF-unRAID:/var/lib/docker/btrfs/subvolumes/84cd5cba4749aeacfb6b5fae612de48bd840eff3036bc1cd3cbfce588c81cb96/usr/local/crashplan# ls -l total 103740 -rw-rw-r-- 1 root root 7641919 Aug 31 05:37 app.asar lrwxrwxrwx 1 root root 11 Aug 31 05:36 bin -> /config/bin drwxrwxrwx 1 root root 2486 Sep 1 05:18 cache/ lrwxrwxrwx 1 root root 12 Aug 31 05:36 conf -> /config/conf drwxrwxr-x 1 root root 82 Aug 31 05:37 doc/ -rw-rw-rw- 1 nobody root 443 Aug 31 05:37 install.vars drwxr-xr-x 1 2013 users 234 May 19 2015 jre/ -rw-rw-rw- 1 root root 47099083 Jul 29 2015 jre-7-linux-x64.tgz drwxrwxr-x 1 root root 602 Aug 31 05:37 lang/ drwxrwxr-x 1 root root 2914 Aug 31 05:37 lib/ -rw-rw-r-- 1 root root 11854 Aug 31 05:37 libc42archive.so -rw-rw-r-- 1 root root 13060 Aug 31 05:37 libc42archive64.so -rw-rw-r-- 1 root root 2481278 Aug 31 05:37 libc42core.so -rw-rw-r-- 1 root root 24756 Aug 31 05:37 libjniwrap.so -rw-rw-r-- 1 root root 48344 Aug 31 05:37 libjniwrap64.so -rw-rw-r-- 1 root root 257693 Aug 31 05:37 libjtux.so -rw-rw-r-- 1 root root 287934 Aug 31 05:37 libjtux64.so -rw-rw-r-- 1 root root 403970 Aug 31 05:37 libleveldb.so -rw-rw-r-- 1 root root 427599 Aug 31 05:37 libleveldb64.so -rw-rw-r-- 1 root root 7346 Aug 31 05:37 libmd5.so -rw-rw-r-- 1 root root 8417 Aug 31 05:37 libmd564.so lrwxrwxrwx 1 root root 11 Aug 31 05:36 log -> /config/log drwxrwxr-x 1 root root 308 Aug 31 05:37 skin/ drwxrwxrwx 1 nobody root 322 Aug 31 05:37 upgrade/ -rw-rw-rw- 1 root root 47474227 Aug 31 05:36 upgrade.cpi -rw-rw-rw- 1 root root 0 Aug 31 05:37 ~custom Am I correct that cache/ should point to /config/cache? If so, what would be the best way to fix this? I'd prefer not to lose the cache information, as I believe Crashplan would then have to re-synchronize file information, which takes quite a while for me.

I have a ST5000DM000 that I shucked from an external more than a year ago. I've used it in Windows without noticing any issues since then. Now that I'm moving it to unRAID, I'm having issues: Using the Preclear beta plugin, write speeds would fluctuate from very good (200+ MB/s early, 150+ MB/s mid-way) to poor (30-40 MB/s) to almost nothing with another I/O intensive task running. Since the Preclear beta plugin was just deprecated, I decided to stop the preclear and install the non-beta. Once I clicked stop, my server became unresponsive for a little while. When it came back, the drive in question had dropped out. I've attached the portion of the syslog corresponding to the dropout, along with full diagnostics. Does this look like a hardware problem, a problem with the Preclear beta plugin, or something else? I haven't yet rebooted in order to bring the drive back online. Should I wait a while to allow it to perform any necessary maintenance tasks? syslog.txt sf-unraid-diagnostics-20160826-1919.zip

I'd like to create & activate a Windows 10 VM on my current mobo/cpu then move unRAID & the VM to a new mobo/cpu (which won't be delivered until after the free upgrade deadline). I have some questions about how best to achieve this: [*]Is this possible? If not, ignore the rest. [*]Can I use CPU Host Passthrough, or will that cause Windows to require reactivation after I change hardware? [*]Does it matter that my current CPU does not support VT-d, but my new CPU does? [*]Will I have to adjust any VM settings when I change hardware? [*]Any other gotchas I should be aware of?