joshstrange

Ok, I'll look into stopping the mover then using mv in screen. Ok, I don't want to waste anyone's time, I was just seeing if it turned on a light in anyone else's head. Like I said I think it will finish by the morning and I've only ever needed to do this once.

I followed all the instructions here to replace my cache drive. I went from 1x250GB SSD to 2x1TB SSD. The mover took overnight to move everything off the old drive and I put the new drives in this morning (~8am EST, over 8 hours as of my posting). I started the mover and it's process has been terribly slow, as in after 8 hours the cache drive reports 11.9GB of data on it. To be fair it's moving my appdata folder which contains my plex config (which has a ton of tiny files) and right now my best guess is that this is causing the issues but I want to make sure. The odd thing is mover seems to hang. Here are some example from /var/log/syslog: (Note the timestamps, you can see multiple minute gaps) Nov 26 15:25:59 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/e3bdfba0e3d3daab01d9e855d98fe5e66530fe9.bundle/Contents/Subtitles.xml Nov 26 15:26:00 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/e3bdfba0e3d3daab01d9e855d98fe5e66530fe9.bundle/Contents/GoP-0.xml Nov 26 15:28:33 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/ca69d866762df5930206a3b0ba4930ca925e8c2.bundle/Contents/Thumbnails/thumb1.jpg Nov 26 15:28:33 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/ca69d866762df5930206a3b0ba4930ca925e8c2.bundle/Contents/GoP-0.xml Nov 26 15:28:33 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/ca69d866762df5930206a3b0ba4930ca925e8c2.bundle/Contents/Subtitles.xml Nov 26 15:28:34 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/c40d8c94d68ee0f39b93ae9b5f483cfa9251c37.bundle/Contents/Thumbnails/thumb1.jpg Nov 26 15:30:26 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/c40d8c94d68ee0f39b93ae9b5f483cfa9251c37.bundle/Contents/GoP-0.xml Nov 26 15:30:26 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/c40d8c94d68ee0f39b93ae9b5f483cfa9251c37.bundle/Contents/Subtitles.xml Nov 26 15:30:26 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/76aceb7500df142ab69670a8976ea1ddd421a6e.bundle/Contents/Thumbnails/thumb1.jpg Nov 26 15:30:27 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/76aceb7500df142ab69670a8976ea1ddd421a6e.bundle/Contents/GoP-0.xml Nov 26 15:30:32 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/76aceb7500df142ab69670a8976ea1ddd421a6e.bundle/Contents/Subtitles.xml Nov 26 15:33:34 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/2ad72506324b98ab0320884a3d09ad543490aad.bundle/Contents/Thumbnails/thumb1.jpg Nov 26 15:33:34 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/2ad72506324b98ab0320884a3d09ad543490aad.bundle/Contents/GoP-0.xml Nov 26 15:33:35 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/2ad72506324b98ab0320884a3d09ad543490aad.bundle/Contents/Subtitles.xml Nov 26 15:33:35 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/30c418f92066540ac190d94ddb0da37d83f1d54.bundle/Contents/Thumbnails/thumb1.jpg Nov 26 15:33:37 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/30c418f92066540ac190d94ddb0da37d83f1d54.bundle/Contents/GoP-0.xml Nov 26 15:36:05 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/30c418f92066540ac190d94ddb0da37d83f1d54.bundle/Contents/Subtitles.xml Nov 26 15:36:05 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/f01ea2e455bb1896e411bb568babf67a0379039.bundle/Contents/Thumbnails/thumb1.jpg Nov 26 15:36:06 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/f01ea2e455bb1896e411bb568babf67a0379039.bundle/Contents/GoP-0.xml Nov 26 15:36:07 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/f01ea2e455bb1896e411bb568babf67a0379039.bundle/Contents/Subtitles.xml Nov 26 15:36:17 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/0cc916c70ff86ceabec3e7dc0135751f3acaeae.bundle/Contents/Subtitle Contributions/com.plexapp.agents.localmedia.xml Nov 26 15:39:21 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/0cc916c70ff86ceabec3e7dc0135751f3acaeae.bundle/Contents/Thumbnails/thumb1.jpg Nov 26 15:39:21 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/0cc916c70ff86ceabec3e7dc0135751f3acaeae.bundle/Contents/GoP-0.xml Nov 26 15:39:21 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/0cc916c70ff86ceabec3e7dc0135751f3acaeae.bundle/Contents/Subtitles.xml Nov 26 15:39:21 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/e1e4d45d549eb0fb7b2fcaf749b67e12f52c84b.bundle/Contents/Subtitle Contributions/com.plexapp.agents.subzero.xml Nov 26 15:39:23 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/e1e4d45d549eb0fb7b2fcaf749b67e12f52c84b.bundle/Contents/Subtitle Contributions/com.plexapp.agents.localmedia.xml Nov 26 15:42:07 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/e1e4d45d549eb0fb7b2fcaf749b67e12f52c84b.bundle/Contents/Thumbnails/thumb1.jpg Nov 26 15:42:07 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/e1e4d45d549eb0fb7b2fcaf749b67e12f52c84b.bundle/Contents/GoP-0.xml Nov 26 15:42:07 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/e1e4d45d549eb0fb7b2fcaf749b67e12f52c84b.bundle/Contents/Subtitles.xml Nov 26 15:42:08 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/809985010e8a1df8000a0a6f764d6df294c5c9f.bundle/Contents/Subtitle Contributions/com.plexapp.agents.subzero.xml Nov 26 15:42:10 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/809985010e8a1df8000a0a6f764d6df294c5c9f.bundle/Contents/Subtitle Contributions/com.plexapp.agents.localmedia.xml Nov 26 15:44:54 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/809985010e8a1df8000a0a6f764d6df294c5c9f.bundle/Contents/GoP-0.xml Nov 26 15:44:54 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/809985010e8a1df8000a0a6f764d6df294c5c9f.bundle/Contents/Thumbnails/thumb1.jpg Nov 26 15:44:54 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/809985010e8a1df8000a0a6f764d6df294c5c9f.bundle/Contents/Subtitles.xml Nov 26 15:44:55 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/00ab1d74a1841af6f0a4e7e6d746e516e7a3e8a.bundle/Contents/Subtitle Contributions/com.plexapp.agents.subzero.xml Nov 26 15:44:55 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/00ab1d74a1841af6f0a4e7e6d746e516e7a3e8a.bundle/Contents/Subtitle Contributions/com.plexapp.agents.localmedia.xml Nov 26 15:44:55 Tower move: move: file /mnt/disk2/Config/plex/Plex Media Server/Media/localhost/a/00ab1d74a1841af6f0a4e7e6d746e516e7a3e8a.bundle/Contents/Thumbnails/thumb1.jpg Is this just to be expected? I'm not exactly mr. popular over here with coming up on 24 hours of downtime (covering 2 primetime viewing times) and no known end in sight. I'm guessing I'll just wake up to it completed tomorrow given it took overnight to move off the SSD in the first place.

My current setup is Internet -> Router (Forwarding 80/443) -> Caddy docker on Unraid (Proxy to 81, port nginx is running on) -> Unraid UI My Caddy setup predates the 6.5 emhttp -> nginx switchover and I want to keep it as I use it for many docker containers and like caddy. My problem is that things like plugin installs or docker updates/installs don't "stream", I get a blank popup until it finishes and then all of the content pops in. If I go straight http://IP:81 then it works fine but once I put it behind Caddy it doesn't work. Does anyone know of a Caddy proxy option I need to use (or even a general proxy-related issue I might be running into that I can dig more into).

Are you talking about using the docket tab to stop a container? I’ve tried that and a “docker stop” from the cli

I use docker and have a container that will not stop (the process inside the container is stuck in an uninterruptible sleep state "D"). Is there anyway I can kill the docker VM and restart it without waiting on this container to stop? I can stop the others gracefully (I think) but I'm remote from my server so my normal "fix" for this is a hard reset but I hate doing those to my machine.

Thank you @johnnie.black!

Ok, I have run: btrfs balance start -dusage=5 /mnt/cache And that completed with the message: Done, had to relocate 38 out of 236 chunks For the following values: Data, single: total=196.87GiB, used=98.62GiB System, single: total=4.00MiB, used=48.00KiB Metadata, single: total=2.01GiB, used=861.39MiB GlobalReserve, single: total=157.81MiB, used=0.00B Which value do I need to watch to know when to re-run that?

Crap, I just noticed the /var/log mount is full and so nothing is writing to it anymore. I cleared out space but in the past I've had to restart before logging resumes so I'm going to do that now.

@johnnie.black Here you go tower-diagnostics-20170828-1027.zip

@johnnie.black I tried creating a dummy file (with dd) and copying a file and I got: cp: cannot create regular file 'XXX': No space left on device Both times.

It's a 250GB SSD using btrfs, here is the "df -h" output Filesystem Size Used Avail Use% Mounted on /dev/sdb1 233G 103G 0 100% /mnt/cache Last night I was able to delete a few files on the drive and it jumped back to "40%-ish" use and but this morning I couldn't even get that to work. Is there something that causes df not to report the right usage? Only thing I can add out is I just moved my plex config folder to the cache drive and it has a billion little files in there (Trying to run du on the plex folder takes hours).

@trurl I have DenyHosts setup and I have blocked the attempts. I need them open for the following reasons: 22: This is my jump server into my home network, it is key-only login 80: To redirect to 443 443: To serve up my SSL apps running in docker containers That said this is slightly divergent from my main issue which is writing to the array takes a long time, causes IO blocking, and causes the load average to jump to 40+.

The mover running will also cause my array to lock up. It's like writing to my array is extremely slow. Is there any utility I can use to find the culprit of the slowness?

I assume you are referencing this: Aug 6 13:31:14 Tower sshd[7408]: Received disconnect from 116.31.116.43 port 35126:11: [preauth] Aug 6 13:31:14 Tower sshd[7408]: Disconnected from 116.31.116.43 port 35126 [preauth] Aug 6 13:32:24 Tower sshd[8082]: Received disconnect from 116.31.116.43 port 59440:11: [preauth] Aug 6 13:32:24 Tower sshd[8082]: Disconnected from 116.31.116.43 port 59440 [preauth] Aug 6 13:32:54 Tower sshd[6366]: Received disconnect from 96.29.187.74 port 40130:11: disconnected by user Aug 6 13:32:54 Tower sshd[6366]: Disconnected from 96.29.187.74 port 40130 Aug 6 13:32:55 Tower sshd[6275]: Received disconnect from 96.29.187.74 port 33723:11: disconnected by user Aug 6 13:32:55 Tower sshd[6275]: Disconnected from 96.29.187.74 port 33723 Aug 6 13:33:34 Tower sshd[10060]: Received disconnect from 116.31.116.43 port 25303:11: [preauth] Aug 6 13:33:34 Tower sshd[10060]: Disconnected from 116.31.116.43 port 25303 [preauth] Aug 6 13:34:42 Tower sshd[11499]: Received disconnect from 59.45.175.11 port 50098:11: [preauth] I have pubkey auth-only enabled but yes this is annoying as hell in my syslogs. A while back I looked into something like fail2ban for unraid but didn't find anything. Edit: Also my server is not in a DMZ, just 22, 80, 443 are exposed through the router (which yes, you could argue is close enough to being in a DMZ). Edit 2: I just re-setup DenyHosts, I had it before but I had some issue with it (I can't remember what the problem was). Hopefully that will block the attacker.

I've had this issue for a while and I'm about at my wits end. I'm tried a ton of different things and the problem persists. I first saw issues as my array started to fill up and when NZBGet would unpack a download the load average would spike as a ton of processes would enter state "D" uninterruptible sleep (See picture). It turns out my CPU is sitting there doing nothing, memory is fine, but load average will climb to 20, 30, 40+ and the web UI becomes unresponsive, "docker ps" won't complete, all docker images and VM's just lock and won't do anything. Looking at docker logs for the containers I see messages about "database locked", "query took over 1 seconds", etc (sqlite DB's). I've tried limiting NZBGet's resource allotment but it appears it was just the cannery as even without it running the system can lock up. I have 10x5TB drives in the array and 1x256GB SSD cache drive. The array drives were all sitting at about 95% full but I just deleted about a 1TB of content to get all the drives under 90% as my previous hunch was nearly-full drives were causing slow reads/writes and by clearing out some space I could alleviate the problem. It hasn't. I've done this (or moved data) and I thought that helped fixed this issue but it may have just been my imagination. Pretty much my entire setup is automated so I don't pay much attention to it unless my VM locks up or one of my docker containers stops responding. I will do ANYTHING to try to fix this as it's becoming a time sync. I have 3 unraid boxes at the highest paid tier but this is my only box I run multiple docker containers on as it's my "power house" while the other two are glorified NFS's that are mounted on the main box for Plex to serve up. Here are some stats and I've also attached the diagnostic logs: Model: Custom M/B: ASRock - Z97 Extreme6 CPU: Intel® Core™ i7-4770K CPU @ 3.50GHz HVM: Enabled IOMMU: Disabled Cache: 256 kB, 1024 kB, 8192 kB Memory: 32 GB (max. installable capacity 32 GB) Network: eth0: 1000 Mb/s, full duplex, mtu 1500 eth1: not connected Kernel: Linux 4.9.10-unRAID x86_64 OpenSSL: 1.0.2k tower-diagnostics-20170806-1315.zip

CPU starvation was my thought as well but limiting the nzbget container to only 4 out of 8 cores didn't change anything. IO bandwidth is something that sounds closer to the issue but the unpack all happens on the cache SSD drive so I wouldn't expect that to kill performance on the array. Also I'm not aware of a good tool to track down the IO culprit. iotop comes to mind but I'm not very good with it.

XFS on all my array drives and btrfs on my cache drives

I have a sneaking suspicion this may be due to how full my drives are (11x5TB with 200-500GB free on each) but it does't exactly square with the fact that my downloads share is on the cache drive. When Nzbget is unpacking the downloaded files other applications will freeze up, specifically Plex. I'll be watching something on my Raspberry Pi 2 (RasPlex) and then it will freeze a little after the unrar starts and then become responsive again after it's finished. I'm almost certain I didn't alway have this issue which is why I lean towards a drive space issue and when I last ran into this I had space elsewhere to move stuff off to and I feel like it cleared up for a while. I also thought it might be Nzbget stealing all the CPU so I limited that docker image to only half my cores but that didn't help either and since I'm using the RPi2 I'm pretty much direct playing everything so CPU shouldn't be an issue. I just wanted to see if anyone else has run into this and if there is anything I can do so that Nzbget doesn't halt my server when it's unpacking. Thanks!

Here are the steps I took to fix it: root@Tower:~# hdparm -N /dev/sdc /dev/sdc: max sectors = 5860531055/5860533168, HPA is enabled <--- This is the problem root@Tower:~# hdparm -N p5860533168 /dev/sdc /dev/sdc: setting max visible sectors to 5860533168 (permanent) max sectors = 5860533168/5860533168, HPA is disabled Note that I took the second number from the max sectors and plugged that into the command with a "p" in front of it. After doing this I was able to start the array and then it started a parity-sync.

Just ran the Fix Common Problems plugin and found that 3 of my drives have HPA partitions on them so I'm looking into that being the cause of the issue but I don't know why that would just be happening now.

My array will not start with the error: Stopped. Replacement disk is too small. Start The replacement disk must be as big or bigger than the original. But I have not changed my disks at all. I just upgrade the RAM in this server and it has been working fine for months. While replacing the RAM I accidentally knocked out the power for one of my drives and so when I first started it up 1 drive was missing but after fixing the power it came back however a different drive is showing as "Wrong Drive" (It showed that way with the missing drive as well). I didn't change the drive at all and I didn't want to go mess with anything before I asked here first. I attached my diagnostics and a screenshot of the issue. Lastly I ran lsblk on the server and I see this: root@Tower:~# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 1 3.8G 0 disk ??sda1 8:1 1 3.8G 0 part /boot sdb 8:16 0 2.7T 0 disk ??sdb1 8:17 0 2.7T 0 part sdc 8:32 0 2.7T 0 disk sdd 8:48 0 2.7T 0 disk ??sdd1 8:49 0 2.7T 0 part sde 8:64 0 4.6T 0 disk ??sde1 8:65 0 4.6T 0 part sdf 8:80 0 2.7T 0 disk ??sdf1 8:81 0 2.7T 0 part sdg 8:96 0 4.6T 0 disk ??sdg1 8:97 0 4.6T 0 part Which seems to indicate the partition is missing on this drive tower-diagnostics-20170101-1436.zip

Well the other stick had the same issue. I ended up stealing 2 2GB sticks from an old build and those worked with new build (I think the speed of the ram was the reason they both wouldn't work with the mobo and the fact that the 2 sticks needed to be used the cause of the crashing). My old build got an unplanned upgrade from 6 to to 10 GB and my new build went from a planned 8 to 4 but it's just a fileserver so it will be fine. Thanks again for all the help and pinpointing the memory issue, I was at a loss and didn't even think to check that.

Frank, First off I should have said this earlier: Thank you for your help and quick responses! I will try the other stick now, I did a good job of cleaning the CPU and applying thermal paste and the fan is seated well (I checked the temps and they are all good). If that has the same problems I will try to other mobo.

Well I didn't need 24 hours.... I didn't need 5 minutes, it crashed almost right after the memtest started... I'm not sure where the issue is, the mobo was used and the memory is new. I wonder if the issue is I'm only using 1 of my ram sticks? Also the ram is 1600 but it's running at 1333. Ugh, this is what happens when I don't build from scratch I guess.

Frank1940, I will try the memtst, I had some issues with my mobo. I got a mobo and i5 for $10 from a sale at work but the mobo wasn't ATX so I bought an ATX one off ebay (odd/old socket for ATX) and it wouldn't support (it would get past post) my 2x4GB memory (even though the mobo is rated to for up to 8GB) so I only have 1 stick in it right now.