tcharron

With no error messages anywhere, I wonder if it's the drive, or if the problem is in the controller/cables/driver... I guess I can remove it and test it in another system.

One of my drives suddenly just slowed down. It is operating at about 10% of what it was doing yesterday. I ran a diskspeed test, and the graph shows the impact. The logs attached will show that I also have a cache drive problem due to a bad cable, but this seems unrelated, as all of my other array drives are affected. Any ideas? tower-diagnostics-20201016-1644.zip

My docker.img was somehow corrupted. I deleted it and restored my dockers using CA, and all seems well.

So checked the cables and the device seems stable now. I was able to rebuild the drive. root@Tower:~# btrfs scrub status /mnt/cache UUID: 100735db-0e88-4450-a406-40f3efdd2bb7 Scrub started: Tue Jul 14 20:31:42 2020 Status: finished Duration: 0:16:16 Total to scrub: 434.15GiB Rate: 455.50MiB/s Error summary: verify=17806 csum=6157411 Corrected: 6175217 Uncorrectable: 0 Unverified: 0 root@Tower:~# Thanks for your help!

Actually.. When I try to get a smart report for sdm, it tells me that the drive is offline! This may be good news! Why doesn't the unraid interface show anywhere that the drive is gone?? I get (from the link you provided) that the error count is wrong, but a red ball on this page would go a long way...

So a run of 'btrfs scrub /mnt/cache' has found about a million uncorrectable errors within 30 seconds. Does this mean that my entire cache drive is toast, and now I need to rebuild it? Given it is a pool, I would have expected to be able to rebuild (much like when array drives fail), should any single drive have errors. Is there any way to do this? Can I just remove sdm and replace it?

tower-diagnostics-20200713-1449.zip

I just saw a huge amount of errors logged. I never got a warning or error from unraid about this, and the web interface shows nothing concerning. This is just a small portion of what I'm seeing flood my logs... Jul 13 14:26:42 Tower kernel: BTRFS error (device sdk1): bdev /dev/sdm1 errs: wr 323772515, rd 238527220, flush 476552, corrupt 0, gen 0 Jul 13 14:26:42 Tower kernel: BTRFS error (device sdk1): bdev /dev/sdm1 errs: wr 323772515, rd 238527221, flush 476552, corrupt 0, gen 0 Jul 13 14:26:42 Tower kernel: BTRFS error (device sdk1): bdev /dev/sdm1 errs: wr 323772515, rd 238527222, flush 476552, corrupt 0, gen 0 Jul 13 14:26:42 Tower kernel: BTRFS error (device sdk1): bdev /dev/sdm1 errs: wr 323772515, rd 238527223, flush 476552, corrupt 0, gen 0 Jul 13 14:26:42 Tower kernel: BTRFS error (device sdk1): bdev /dev/sdm1 errs: wr 323772515, rd 238527224, flush 476552, corrupt 0, gen 0 Jul 13 14:26:43 Tower kernel: BTRFS warning (device sdk1): lost page write due to IO error on /dev/sdm1 Jul 13 14:26:43 Tower kernel: BTRFS warning (device sdk1): lost page write due to IO error on /dev/sdm1 Jul 13 14:26:43 Tower kernel: BTRFS warning (device sdk1): lost page write due to IO error on /dev/sdm1 Jul 13 14:26:43 Tower kernel: BTRFS error (device sdk1): error writing primary super block to device 4 Jul 13 14:26:43 Tower kernel: BTRFS warning (device sdk1): lost page write due to IO error on /dev/sdm1 Jul 13 14:26:43 Tower kernel: BTRFS warning (device sdk1): lost page write due to IO error on /dev/sdm1 Jul 13 14:26:43 Tower kernel: BTRFS warning (device sdk1): lost page write due to IO error on /dev/sdm1 Jul 13 14:26:43 Tower kernel: BTRFS error (device sdk1): error writing primary super block to device 4 Jul 13 14:26:43 Tower kernel: BTRFS warning (device sdk1): lost page write due to IO error on /dev/sdm1 Jul 13 14:26:43 Tower kernel: BTRFS warning (device sdk1): lost page write due to IO error on /dev/sdm1 Jul 13 14:26:43 Tower kernel: BTRFS warning (device sdk1): lost page write due to IO error on /dev/sdm1 Jul 13 14:26:43 Tower kernel: BTRFS error (device sdk1): error writing primary super block to device 4 Jul 13 14:26:43 Tower kernel: BTRFS warning (device sdk1): lost page write due to IO error on /dev/sdm1 Jul 13 14:26:43 Tower kernel: BTRFS error (device sdk1): error writing primary super block to device 4 Jul 13 14:26:43 Tower kernel: BTRFS error (device sdk1): error writing primary super block to device 4 Jul 13 14:26:43 Tower kernel: BTRFS error (device sdk1): error writing primary super block to device 4 Jul 13 14:26:43 Tower kernel: BTRFS error (device sdk1): error writing primary super block to device 4 Jul 13 14:26:43 Tower kernel: BTRFS error (device sdk1): error writing primary super block to device 4 Here is the relevant drive information: DEVICE IDENTIFICATION TEMP. READS WRITES ERRORS FS SIZE USED FREE VIEW Cache KINGSTON_SV300S37A240G_50026B726706EDE1 - 240 GB (sdk) 25 C 41,038,636 27,137,657 0 btrfs 496 GB 210 GB 286 GB Browse /mnt/cache Cache 2 INTEL_SSDSC2KW512G8_PHLA8222024A512DGN - 512 GB (sdm) * 41,038,636 27,137,657 0 Device is part of cache pool Cache 3 KINGSTON_SV300S37A240G_50026B776407138B - 240 GB (sdn) 22 C 36,990,993 20,943,909 0 Device is part of cache pool This seems like a failing (failed?) drive, but I don't know if it is sdk or sdm, or if it is something else entirely. Any ideas?

I have found issues with getting Plex to run well. Sure I could probably get it going, but Serviio was just too familiar to me. I still run serviio on my unraid. I put this into /boot/config/setup-riftbit.sh docker pull riftbit/serviio docker run --name serviio_r -d --network host -p 23423:23423/tcp -p 23424:23424/tcp -p 8895:8895/tcp -p 1900:1900/udp -v /etc/localtime:/etc/localtime:ro -v /mnt/myapps/Serviio-riftbit/library:/opt/serviio/library:rw -v /mnt/user/unraid/Videos/Movies:/media/Movies:rw -v /mnt/user/unraid/Videos/Movies_Viewed:/media/Movies_Viewed:rw -v /mnt/user/unraid/Videos/Kids:/media/Kids:rw -v /mnt/user/unraid/videos_tv:/media/videos_tv:ro riftbit/serviio Then run it: /boot/config/setup-riftbit.sh The only downside is that it can't be configured from the docker page (even though it shows up there). If you ever want to change the setup, just destroy the container, edit the file, and run it again. Update: Here's a small script that will destroy the old container, update the image and then restart it: docker stop serviio_r docker rm serviio_r docker pull riftbit/serviio docker run --name serviio_r -d --network host -p 23423:23423/tcp -p 23424:23424/tcp -p 8895:8895/tcp -p 1900:1900/udp -v /etc/localtime:/etc/localtime:ro -v /mnt/myapps/Serviio-riftbit/library:/opt/serviio/library:rw -v /mnt/user/unraid/Videos/Movies:/media/Movies:rw -v /mnt/user/unraid/Videos/Movies_Viewed:/media/Movies_Viewed:rw -v /mnt/user/unraid/Videos/Kids:/media/Kids:rw -v /mnt/user/unraid/videos_tv:/media/videos_tv:ro riftbit/serviio

I got Serviio 2.0 working on my unraid (with my Sony Bravia TV). I based all of this on the riftbit serviio container. That has been updated 9 months ago (not sure if it will be supported beyond that, but it is 2.0). That container is described here (you don't need anything from this page -- it's just an fyi). https://registry.hub.docker.com/r/riftbit/serviio 1) Create a directory on your server that can be used to store library data. I chose to use /mnt/myapps/Serviio-riftbit. You should create a directory on your server before continuing. 2) Run the following commands. Replace the reference to the directory from step 1 with whatever directory you created. For each "-v" section below, map each of your media directories (I hope this part is self evident). The format is "-v [source directory]:[docker directory]:rw". You can probably use :ro at the end, but I haven't tried that to see if it causes any errors. I use RW since serviio can be configured to download metadata and subtitles etc docker pull riftbit/serviio docker run --name serviio_r -d --network host -p 23423:23423/tcp -p 23424:23424/tcp -p 8895:8895/tcp -p 1900:1900/udp -v /etc/localtime:/etc/localtime:ro -v /mnt/myapps/Serviio-riftbit/library:/opt/serviio/library:rw -v /mnt/user/unraid/Videos/Movies:/media/Movies:rw -v /mnt/user/unraid/Videos/Movies_Viewed:/media/Movies_Viewed:rw -v /mnt/user/unraid/Videos/Kids:/media/Kids:rw -v /mnt/user/unraid/videos_tv:/media/videos_tv:ro riftbit/serviio 3) Go to the docker page within your server. You will see a new "serviio_r" docker there. Disable any other serviio instances, or anything that uses conflicting ports (8895, 23424,23423) stop and restart the "serviio_r" docker 4) Navigate your browser to "[your unraid server]:23423/console 5) Configure serviio. In particular, add each of the media paths to the shared folders under the "Library" section. Note that the docker can not be configured or updated within the Unraid docker page. It can only be stopped/started/deleted (or the console accessed) from that page. If you want to make a change (for example, adding a new media directory), edit your command from step 2 (the first line is not needed again), delete the "serviio_r" docker (via the unraid Docker page), edit your script from step 2, and run it again. My experience is that Serviio configuration settings survive when I do this (although I have only tried it once or twice). I hope that this helps others! I'm not really intending to support this, but will try to do my best to help if I can. If I had more time and energy, the next logical step is to make this all configurable from within Unraid, without requiring the script. Tim

If you log in to your account at https://www.crashplanpro.com/app/#/console/ , what does it say? I wonder if somehow you have ended up with recent backups being somehow identified as a new device.

I am not trying to back up a network share. I am trying to back up local drives TO a network share. So effectively there is no way to use CrashPlan to back up local machines to my unraid server, even if I pay for CrashPlan.

I just installed CrashPlan Small Business on a windows machine, and this docker on my unraid box. For the life of me, I can't see how to establish a backup of the windows box to the unraid box. The only destination options I can see are "CrashPlan PRO Online" and "Add Local Destination". When I use the local destination option, I get an error if I select a network drive. I know that the peer to peer support has disappeared, but can't I at least back up to a network destination now? By the way, The OS in the logs reports as: /mnt/user/appdata/CrashPlanPRO/log/app.log:OS = Linux (4.15.0-60-generic, amd64) grep: /mnt/user/appdata/CrashPlanPRO/log/service.log: No such file or directory

I just had a cache drive fail. My former configuration was 3 240G SSD drives. That showed 360G of usable space After the failure of one drive, it still showed 360G usable. I figured that was ok, as the data was all there (but no longer protected). I replaced the bad drive with a 512G drive. I expected this to increase the capacity from 360G to 480G. This expectation was confirmed with the calculator at http://carfax.org.uk/btrfs-usage/. That calculator told me that 32G would go unused. This is presumably since everything written to the 2 240G drives would be duplicated on the 512G, with using 480G (and hence 32G left over). In fact, unraid reports that I now have an available cache pool of 496G. This doesn't make sense to me. What am I missing?

Yeah... as soon as I saw it I knew what was going on. I think that the entire idea of having things auto-installed from my flash drive via auto_install and a line in my go file was probably something that I added 5 years ago or so. It worked well for a very long time, and then was was long forgotten but survived the various upgrades along the way. My point in sharing it here was just to help anyone else figure it out, as I found a few threads with similar problems. The generallly accepted solution was to do a clean install to a new USB key -- neither of which is required with the above knowledge.

Old topic, but it just bit me. For me, the problem was gcc-4.8.2-x86_64-1.txz.auto_install, which was located in my /boot/packages directory. A line in my go file was installing it, and it was the file preventing my machine from booting. I was able to verify this by just trying to install it manually after my system finally booted properly... Verifying package gcc-4.8.2-x86_64-1.txz. Installing package gcc-4.8.2-x86_64-1.txz: PACKAGE DESCRIPTION: # gcc (Base GCC package with C support) # # GCC is the GNU Compiler Collection. # # This package contains those parts of the compiler collection needed to # compile C code. Other packages add C++, Fortran, Objective-C, and # Java support to the compiler core. # Executing install script for gcc-4.8.2-x86_64-1.txz. Package gcc-4.8.2-x86_64-1.txz installed. Verifying package glibc-2.17-x86_64-7.txz. Installing package glibc-2.17-x86_64-7.txz: PACKAGE DESCRIPTION: # glibc (GNU C libraries) # # This package contains the GNU C libraries and header files. The GNU # C library was written originally by Roland McGrath, and is currently # maintained by Ulrich Drepper. Some parts of the library were # contributed or worked on by other people. # # You'll need this package to compile programs. # Executing install script for glibc-2.17-x86_64-7.txz. cp: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by cp) /sbin/ldconfig: Cannot lstat ld-2.17.so: No such file or directory /bin/bash: line 56: /usr/bin/basename: No such file or directory /bin/bash: line 56: /usr/bin/rm: No such file or directory /bin/bash: line 57: /usr/bin/basename: No such file or directory /bin/bash: line 57: /usr/bin/cp: No such file or directory /bin/bash: line 58: /usr/bin/basename: No such file or directory /bin/bash: line 59: /usr/bin/rm: No such file or directory ...

fyi, this approach worked -- my server was rebuilt from parity and I didn't lose any data. The only strange/worrisome moment was the two array shutdowns that I had to do -- each of them took about 10 minutes to complete. I think that the delay was related to some kind of network share issue related to the temporary mount I had created on the problem server which was allowing me to access data on my other unRaid server. For some reason, that mount became unusable when the array was taken offline. the web gui became unresponsive, and all filesystem commands were very slow (including lsof, mount, df). In any event, I'm a happy camper this morning. Oh - A scan of the USB key did not find any errors, so I am not sure what caused the 0 byte super.dat file. In any event, I made sure to take a copy of the super.dat file now that it's all working well! Thanks!

Well, I can't guarantee that I don't have some device on my network which will send a file to unraid when it's up. If that happens, then unraid could try to write to my stale/bad drive. I think I'll do a partial clear to ensure that the drive can't be written to.

I don't see how unraid will know which of my data disks is the 'bad' one... since it is stale but has a valid filesystem on it. The moment I assign all drives including parity and bring the array up, won't it somehow write to the parity drive (as soon as any process anywhere writes to the array), and therefore destroy the data I need to recover the bad drive? (for clarity, I'm referring to the moment between steps 3 and 4 above). More specifically, if any data gets written to the bad/stale drive, then the parity bits of those corresponding writes will get written to the parity drive... which means I won't be able to restore the drive properly. Maybe I should first start a preclear on the stale drive (and cancel after a minute or two) -- that should ensure that the old/stale filesystem is not recognized by unraid -- which will prevent unraid from trying to write to it, which would destroy the related parity drive data. ??

I don't think I did that. I'm pretty sure that when I momentarily added the drive to the array it was the old data drive #1 in slot #1. I'm in the process of copying that data to my second server. However, the old disk is stale. My goal here is to get what I can off of the drive in case my rebuild from parity has some kind of problem. I ended up using a command line mount statement to mount it as read-only. I am in the process of copying this to my second unraid server. Even though it's stale, it's better than nothing. Once this drive's contents are copied off of it, am I required to preclear it? I really just want to start the rebuild -- preclearing it seems like a waste of time. Well, maybe a preclear will find a problem, but given my issue had been a cable problem I don't see much value in wasting a day or two doing a preclear -- i'd rather start the rebuild from parity. If it finds an error, i won't be any worse off. But if it doesn't find an error, I will have avoided a slow pre-clear.

Thanks for taking the time Johnny! I'm running 6.1.7 so this sounds like it could work. The other thread seems like the exact same problem I had. You said I do know which is the parity drive, but I may have broken things anyway... When I looked at the empty array the first time, I momentarily thought I'd reassign drives. I assigned the first data drive to the correct drive. When it didn't show up with the correct FS/size/used/free data (and I noticed this after about 3 seconds), I immediately changed it back to 'unassigned' and came to the forums. Hopefully that lapse in judgement didn't cause anything to be written to the drive. (?) I don't think that I have a spare drive matching the size of the one that was noted as failed (4TB). What I do have is a second unRaid server that has 10TB of free storage. The failed drive was probably only about 25% used. How can I mount that drive in read-only mode so that I can copy it's contents off the drive (and then use the drive as if it were a new one)? It was formatted as XFS.

Well, I figured I can get the data off the drives (even the one that failed, as I suspect it was just a bad cable). The problem is that the disk failure occurred a while ago (maybe 10 days or so), so the 'bad' drive will have stale data. I guess I'll email LimeTech and see if there's any workarounds. Super.dat is quite the weak point. It was clearly corrupted 10 days ago, but the array went on blindly just waiting for a shutdown/reboot. It would have been nice if it detected the imminent problem somewhere in that 10 days.

I don't think it exists, as I upgraded a drive from 3tb to 4tb about 10 days ago -- the day that the super.dat 0 byte file was created. I also replaced a failed drive. I haven't taken a backup since then. I do have a screen print of the array configuration from just before I did all this.

So, I had a bad disk. I let it run as simulated from the parity for a while. Then, I shut down and brought the system back up. It was at this point that I had an empty array. The /boot/super.dat file is 0 bytes and is dated from almost two weeks ago. WHat I want to do is to use the good disks and the parity drive to rebuild the missing data drive. I know which drive is parity. How do I do this?

ip link # ip link 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT link/ipip 0.0.0.0 brd 0.0.0.0 3: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN mode DEFAULT link/gre 0.0.0.0 brd 0.0.0.04: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN mode DEFAU LT qlen 1000 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff5: ip_vti0@NONE: <NOARP> mtu 1364 qdisc noop state DOWN mode DEFAULT link/ipip 0.0.0.0 brd 0.0.0.0 6: rename6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000 link/ether 10:fe:ed:02:06:6d brd ff:ff:ff:ff:ff:ff 7: rename7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT qlen 1000 link/ether 10:fe:ed:02:06:6d brd ff:ff:ff:ff:ff:ff 8: eth2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP mode DEFAULT qlen 1000 link/ether 10:fe:ed:02:06:6d brd ff:ff:ff:ff:ff:ff 10: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT link/ether 10:fe:ed:02:06:6d brd ff:ff:ff:ff:ff:ff 11: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT link/ether a2:46:fb:b0:86:c7 brd ff:ff:ff:ff:ff:ff 12: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT link/ether 52:54:00:0b:15:fd brd ff:ff:ff:ff:ff:ff 13: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN mode DEFAULT qlen 500 link/ether 52:54:00:0b:15:fd brd ff:ff:ff:ff:ff:ff 15: veth559b2d4@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT link/ether a2:46:fb:b0:86:c7 brd ff:ff:ff:ff:ff:ff 16: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UNKNOWN mode DEFAULT qlen 500 link/ether fe:54:00:e5:36:e4 brd ff:ff:ff:ff:ff:ff and ethtool... root@Tower:~# ethtool rename6 Settings for rename6: Supported ports: [ TP MII ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full Supported pause frame use: No Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full Advertised pause frame use: Symmetric Receive-only Advertised auto-negotiation: Yes Link partner advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full Link partner advertised pause frame use: Symmetric Link partner advertised auto-negotiation: Yes Speed: 1000Mb/s Duplex: Full Port: MII PHYAD: 0 Transceiver: internal Auto-negotiation: on Supports Wake-on: pumbg Wake-on: g Current message level: 0x00000033 (51) drv probe ifdown ifup Link detected: no root@Tower:~# ethtool rename7 Settings for rename7: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supported pause frame use: Symmetric Receive-only Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: Yes Speed: Unknown! Duplex: Unknown! (255) Port: Twisted Pair PHYAD: 0 Transceiver: internal Auto-negotiation: on MDI-X: Unknown Current message level: 0x000060e4 (24804) link ifup rx_err tx_err hw wol Link detected: no I'm flummoxed..