April 11, 201610 yr My backup was carried out at 02:00 today, the brief loss of network connectivity is where I tried a different network cable
April 11, 201610 yr I have a Docker container that always says "update ready" in the Unraid Docker page. I can update it 6 times in a row and it will still list that. When I do run the update, the logs say that the container is up to date and does not download anything, just re-launches it. The Docker is the official Postgres Docker https://hub.docker.com/_/postgres/ I am running 6.2.0 beta 21 but this has been happening since at least 6.2.0 beta 18 (when I upgraded to the beta). Diagnostics attached aeris-diagnostics-20160411-1055.zip
April 11, 201610 yr I am creating a 2nd server - testing it for now, then to put data on once i trust it. I have a few questions if someone can help. 1- Unraid irc channel says broken - should i just wait for beta 22? At the moment all i need to do is load it up with data and samba share, no vms, at some point i would use docker containers. 2- I precleared 4 seagate 8tb harvested from usb enclosure drives 2 cycles each. After that i made 1 a parity and 3 assigned as data, then i just tried to mount, it appears that there are some errors/warnings, are these errors normal in that the server sees drives aren't formatted and formats them? I see a number of other errors in my syslog as well, is there something wrong? 3- It is doing a parity sync/data rebuild, at 2.2%, i know that a parity sync is normal when creating an array for first time, but is it normal to take so long when there is no actual data? If someone has time to look at my log/answer questions i would appreciate it, before i move forward with transferring data over and enabling my 2nd unraid pro key on this trial. pipe-diagnostics-20160411-1152.zip
April 11, 201610 yr 2- I precleared 4 seagate 8tb harvested from usb enclosure drives 2 cycles each. After that i made 1 a parity and 3 assigned as data, then i just tried to mount, it appears that there are some errors/warnings, are these errors normal in that the server sees drives aren't formatted and formats them? I see a number of other errors in my syslog as well, is there something wrong? It is normal that new drives are seen as unformattted. 3- It is doing a parity sync/data rebuild, at 2.2%, i know that a parity sync is normal when creating an array for first time, but is it normal to take so long when there is no actual data? Parity has no idea what is on the disks and as such is unaware of the data that is on them at the file system level. It just sees the disk as a bunch of sectors that need protecting against failure of anyy disk, so yes on any new system the whole of each disk will be read as part of creating parity.
April 11, 201610 yr 2- I precleared 4 seagate 8tb harvested from usb enclosure drives 2 cycles each. After that i made 1 a parity and 3 assigned as data, then i just tried to mount, it appears that there are some errors/warnings, are these errors normal in that the server sees drives aren't formatted and formats them? I see a number of other errors in my syslog as well, is there something wrong? It is normal that new drives are seen as unformattted. 3- It is doing a parity sync/data rebuild, at 2.2%, i know that a parity sync is normal when creating an array for first time, but is it normal to take so long when there is no actual data? Parity has no idea what is on the disks and as such is unaware of the data that is on them at the file system level. It just sees the disk as a bunch of sectors that need protecting against failure of anyy disk, so yes on any new system the whole of each disk will be read as part of creating parity. would this email/popup be normal - ? Event: unRAID Parity disk error Subject: Warning [PIPE] - Parity disk, parity-sync in progress Description: ST8000AS0002-1NA17Z_Z8408NKD (sdb) Importance: warning
April 11, 201610 yr Yes... that is normal;.. it is telling you there is no partity and that parity is beiing created.
April 11, 201610 yr This morning my GUI was unresponsive, and I had to power off/power on to get it working again. I was able to telnet into the box and get a diag (attached). I'm suspecting (without much data to base it on) that it had something to do with my plex docker. I tried to stop the plex docker from the command line (docker stop plex) and it would just hang the command line until I hit ctrl-c. Diags attached. Would appreciate input as I've found this to be less reliable than I would like. I had to do the power toggle a couple of times yesterday too. unraid-diagnostics-20160411-0912.zip
April 11, 201610 yr I have a Docker container that always says "update ready" in the Unraid Docker page. I can update it 6 times in a row and it will still list that. When I do run the update, the logs say that the container is up to date and does not download anything, just re-launches it. The Docker is the official Postgres Docker https://hub.docker.com/_/postgres/ I am running 6.2.0 beta 21 but this has been happening since at least 6.2.0 beta 18 (when I upgraded to the beta). Diagnostics attached I have the same issue with the "mysql" app. Likely due to the single name/no user repository. https://registry.hub.docker.com/_/mysql/
April 12, 201610 yr I have a Docker container that always says "update ready" in the Unraid Docker page. I can update it 6 times in a row and it will still list that. When I do run the update, the logs say that the container is up to date and does not download anything, just re-launches it. The Docker is the official Postgres Docker https://hub.docker.com/_/postgres/ I am running 6.2.0 beta 21 but this has been happening since at least 6.2.0 beta 18 (when I upgraded to the beta). Diagnostics attached I have the same issue with the "mysql" app. Likely due to the single name/no user repository. https://registry.hub.docker.com/_/mysql/ We plan on fixing the update issue for official Docker images for a future beta.
April 12, 201610 yr I awoke this morning and found that my arrray had been sucessfully backed up overnight @ 2am, also mapped drives were also shown on Windows Explorer. I have copied a file into one of the shares, and deleted it Ok, however if I then try and access any further folders & files in any of the other shares, there is a time delay of several seconds before they appear? Please find attached diagnostics file tower-diagnostics-20160412-0625.zip
April 12, 201610 yr I awoke this morning and found that my arrray had been sucessfully backed up overnight @ 2am, also mapped drives were also shown on Windows Explorer. I have copied a file into one of the shares, and deleted it Ok, however if I then try and access any further folders & files in any of the other shares, there is a time delay of several seconds before they appear? Please find attached diagnostics file The Dynamix Cache Dirs plugin will help to alleviate this
April 12, 201610 yr This morning my GUI was unresponsive, and I had to power off/power on to get it working again. I was able to telnet into the box and get a diag (attached). I'm suspecting (without much data to base it on) that it had something to do with my plex docker. I tried to stop the plex docker from the command line (docker stop plex) and it would just hang the command line until I hit ctrl-c. Diags attached. Would appreciate input as I've found this to be less reliable than I would like. I had to do the power toggle a couple of times yesterday too. I'm in the same situation. I've tried disabling all plugins, vm's and dockers but it still doesn't improve the lock ups. The system becomes unresponsive with a load (according to top) of >50. Iotop doesn't show any IO activity and top says the CPU is not busy, yet load remains extremely high. Dmesg doesn't have anything of interest, with the last messages being about spindowns. The system will not shut down once the load gets that high, so I have to resort to powering off. I should also mention that I can connect through telnet while this is happening, but depending on what command I issue the session will lock up. For example a "btrfs fi sh" will never return. I also noticed that any significant concurrent IO will bring on the problem quickly, which made me wonder if perhaps there is some kind of deadlock/race condition happening with the new dual parity code. Totally unsubstantiated (sorry Tom, I'm not trying to point fingers!), just offering my uninformed guess at least. My other thought was maybe BTRFS was dying under the concurrent IO, but then again BTRFS was solid when I transferred the 70+ TB from my ZFS disks onto BTRFS so I could move back to Unraid. I did that using Ubuntu 16.04 Beta which used the 4.4 kernel as well, and had 3 disks copying at the same time (from 3 other disks, not thrashing). Average throughput on the hardware saturated a SATA2 connection and I never had a lockup in the two weeks it took me to move the data. All 27 data drives in the array are formatted with BTRFS and are spread across two Norco 24 bay enclosures using an Intel SAS expander. This setup was reliable using Ubuntu 14.04 and ZoL so I know the hardware is solid. Something just needs to be tweaked a little to make it reliable. Also, FWIW, heavy IO that brings on the lock up was not using any of the drives on the expander. It was done by moving data from one drive to another using mc in an ssh session and having NzbGet working on uncompressing a large 200GB download. Diags are attached and you can PM me if you want me to test anything for you... Thanks to the LimeTech staff and volunteers for all your efforts! unmedia-diagnostics-20160412-0734.zip
April 12, 201610 yr I previously had Cache Dir installed, and the problem was happening. I went back to bare metal to try and remove plugins in my testing. Sometimes instead of a delay, I get a hard lockup of Terracopy / Windows Explorer
April 12, 201610 yr I previously had Cache Dir installed, and the problem was happening. I went back to bare metal to try and remove plugins in my testing. Sometimes instead of a delay, I get a hard lockup of Terracopy / Windows Explorer You might try turning off SMB2&3 on your client PC (assuming it's Win10?) and see if that helps. 6.2 includes an updated Samba, but I had issues with it that seemed to go away when I turned off SMB2&3 on my Win10 machine.
April 12, 201610 yr I previously had Cache Dir installed, and the problem was happening. I went back to bare metal to try and remove plugins in my testing. Sometimes instead of a delay, I get a hard lockup of Terracopy / Windows Explorer Little tip: it helps to quote responses in this thread because there's many conversations going on at once. PM me if you're running docker apps. I have a possible theory but need someone to try it. (and have been trying to justify it to myself). Sent from my LG-D852 using Tapatalk
April 13, 201610 yr Is it possible to have turbo write on a share and not other? Would be awesome Not possible. Sent from my Nexus 6 using Tapatalk
April 13, 201610 yr Did a little more testing related to the high load/unresponsive server tonight. I re-enabled all the disabled dockers and queued up some downloads. I now have a par2 repair stuck in the download queue that puts enough IO strain on the server to have it lock up within 10 minutes of booting. I've rebooted 3 times to ensure that it will lock up consistently. I then did something daring (or maybe stupid ) to eliminate the possibility of it being the dual parity. I unassigned both my parity drives and rebooted the server to see if it would lock up. The good news is that it locked up with 10 minutes of booting, so I'm now assuming it does not have anything to do with the new/changed dual parity code (sorry for doubting you Tom!). The bad news is I'm more stumped than ever as to what it could be. Below is what top and iotop are reporting at roughly the same time. The server is sitting in an unresponsive state right now, although my previously connected ssh sessions continue to update the top and iotop screens. You'll notice that the top command shows high load and the wa figure indicates its waiting on IO of some sort. But the iotop doesn't show any significant disk use. In fact there is no disk use and if I leave it long enough the drives spin down as per their settings (seen in syslog and dmesg). I'm not enough of a Linux guru to figure out where to look next, so if anyone has suggestions on what next steps could be, please pass them along. Thanks! top: top - 20:53:21 up 42 min, 3 users, load average: 38.53, 38.31, 32.92 Tasks: 1031 total, 2 running, 1029 sleeping, 0 stopped, 0 zombie %Cpu(s): 10.7 us, 9.2 sy, 0.0 ni, 0.0 id, 80.2 wa, 0.0 hi, 0.0 si, 0.0 st KiB Mem : 32989816 total, 17111804 free, 1649940 used, 14228072 buff/cache KiB Swap: 0 total, 0 free, 0 used. 30570024 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 21618 nobody 20 0 513616 11372 3148 R 99.7 0.0 29:55.60 /usr/bin/par2 r /incomplete-d+ 18167 nobody 20 0 432220 162120 37060 S 60.2 0.5 17:19.48 ./Plex Media Server 7785 root 20 0 83092 21380 7160 S 6.9 0.1 2:42.74 /usr/bin/python /usr/sbin/iot+ 8696 root 20 0 25892 4212 2468 R 1.0 0.0 0:22.84 top 292 root 39 19 0 0 0 S 0.3 0.0 0:00.22 [khugepaged] 11685 root 20 0 25772 3880 2368 S 0.3 0.0 0:21.00 top 1 root 20 0 4372 1640 1532 S 0.0 0.0 0:07.00 init 2 root 20 0 0 0 0 S 0.0 0.0 0:00.03 [kthreadd] 3 root 20 0 0 0 0 S 0.0 0.0 0:00.21 [ksoftirqd/0] 5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [kworker/0:0H] 7 root 20 0 0 0 0 S 0.0 0.0 0:01.18 [rcu_preempt] 8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 [rcu_sched] 9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 [rcu_bh] 10 root rt 0 0 0 0 S 0.0 0.0 0:00.01 [migration/0] 11 root rt 0 0 0 0 S 0.0 0.0 0:00.01 [migration/1] 12 root 20 0 0 0 0 S 0.0 0.0 0:00.08 [ksoftirqd/1] 14 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [kworker/1:0H] 15 root rt 0 0 0 0 S 0.0 0.0 0:00.01 [migration/2] 16 root 20 0 0 0 0 S 0.0 0.0 0:00.02 [ksoftirqd/2] 18 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [kworker/2:0H] 19 root rt 0 0 0 0 S 0.0 0.0 0:00.01 [migration/3] 20 root 20 0 0 0 0 S 0.0 0.0 0:00.07 [ksoftirqd/3] 21 root 20 0 0 0 0 S 0.0 0.0 0:00.09 [kworker/3:0] 22 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [kworker/3:0H] 23 root rt 0 0 0 0 S 0.0 0.0 0:00.01 [migration/4] 24 root 20 0 0 0 0 S 0.0 0.0 0:00.06 [ksoftirqd/4] 26 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [kworker/4:0H] 27 root rt 0 0 0 0 S 0.0 0.0 0:00.01 [migration/5] 28 root 20 0 0 0 0 S 0.0 0.0 0:00.02 [ksoftirqd/5] 30 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 [kworker/5:0H] iotop: Total DISK READ : 0.00 B/s | Total DISK WRITE : 0.00 B/s Actual DISK READ: 0.00 B/s | Actual DISK WRITE: 0.00 B/s TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND 4945 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.03 % [kworker/u16:12] 1 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % init 2 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kthreadd] 3 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/0] 5 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/0:0H] 7 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_preempt] 8 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_sched] 9 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [rcu_bh] 10 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/0] 11 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/1] 12 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/1] 14 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/1:0H] 15 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/2] 16 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/2] 18 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/2:0H] 19 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/3] 20 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/3] 21 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/3:0] 22 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/3:0H] 23 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/4] 24 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/4] 26 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/4:0H] 27 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/5] 28 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/5] 30 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/5:0H] 31 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/6] 32 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [ksoftirqd/6] 33 be/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/6:0] 34 be/0 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [kworker/6:0H] 35 rt/4 root 0.00 B/s 0.00 B/s 0.00 % 0.00 % [migration/7] dmesg | tail -n 50: [ 646.361780] eth0: renamed from veth85dfe66 [ 646.370907] docker0: port 4(veth71faf9b) entered forwarding state [ 646.370923] docker0: port 4(veth71faf9b) entered forwarding state [ 646.466434] docker0: port 2(veth415f089) entered forwarding state [ 648.772820] device veth6254b33 entered promiscuous mode [ 648.772972] docker0: port 5(veth6254b33) entered forwarding state [ 648.772988] docker0: port 5(veth6254b33) entered forwarding state [ 648.773860] docker0: port 5(veth6254b33) entered disabled state [ 651.842532] docker0: port 3(vethcda3321) entered forwarding state [ 653.168843] eth0: renamed from veth3c62677 [ 653.173696] docker0: port 5(veth6254b33) entered forwarding state [ 653.173712] docker0: port 5(veth6254b33) entered forwarding state [ 661.378664] docker0: port 4(veth71faf9b) entered forwarding state [ 667.530740] BTRFS info (device loop1): disk space caching is enabled [ 667.530744] BTRFS: has skinny extents [ 668.226770] docker0: port 5(veth6254b33) entered forwarding state [ 668.917968] BTRFS info (device loop1): new size for /dev/loop1 is 1073741824 [ 668.925642] tun: Universal TUN/TAP device driver, 1.6 [ 668.925643] tun: (C) 1999-2004 Max Krasnyansky <[email protected]> [ 670.188941] device virbr0-nic entered promiscuous mode [ 670.303816] virbr0: port 1(virbr0-nic) entered listening state [ 670.303830] virbr0: port 1(virbr0-nic) entered listening state [ 670.326386] virbr0: port 1(virbr0-nic) entered disabled state [ 2500.891496] mdcmd (63): spindown 19 [ 2501.318623] mdcmd (64): spindown 21 [ 2503.464412] mdcmd (65): spindown 9 [ 2503.868055] mdcmd (66): spindown 10 [ 2504.154809] mdcmd (67): spindown 11 [ 2505.158121] mdcmd (68): spindown 14 [ 2505.585249] mdcmd (69): spindown 17 [ 2507.589672] mdcmd (70): spindown 1 [ 2508.026666] mdcmd (71): spindown 3 [ 2509.029710] mdcmd (72): spindown 4 [ 2509.456308] mdcmd (73): spindown 8 [ 2510.460182] mdcmd (74): spindown 12 [ 2510.897515] mdcmd (75): spindown 13 [ 2511.184286] mdcmd (76): spindown 15 [ 2511.471072] mdcmd (77): spindown 16 [ 2511.757819] mdcmd (78): spindown 20 [ 2513.185802] mdcmd (79): spindown 5 [ 2514.473687] mdcmd (80): spindown 6 [ 2518.143213] mdcmd (81): spindown 26 [ 2520.572942] mdcmd (82): spindown 2 [ 2527.584083] mdcmd (83): spindown 22 [ 2533.878293] mdcmd (84): spindown 23 [ 2535.306545] mdcmd (85): spindown 7 [ 2535.593328] mdcmd (86): spindown 18 [ 2536.022367] mdcmd (87): spindown 24 [ 2536.449974] mdcmd (88): spindown 25 [ 2536.736169] mdcmd (89): spindown 27 tail -n 50 /var/log/syslog Apr 12 20:22:19 unmedia root: Starting libvirtd... Apr 12 20:22:19 unmedia kernel: tun: Universal TUN/TAP device driver, 1.6 Apr 12 20:22:19 unmedia kernel: tun: (C) 1999-2004 Max Krasnyansky <[email protected]> Apr 12 20:22:19 unmedia emhttp: nothing to sync Apr 12 20:22:19 unmedia rc.unRAID[18670][18674]: Processing /etc/rc.d/rc.unRAID.d/ start scripts. Apr 12 20:22:20 unmedia kernel: device virbr0-nic entered promiscuous mode Apr 12 20:22:21 unmedia avahi-daemon[12607]: Joining mDNS multicast group on interface virbr0.IPv4 with address 192.168.122.1. Apr 12 20:22:21 unmedia avahi-daemon[12607]: New relevant interface virbr0.IPv4 for mDNS. Apr 12 20:22:21 unmedia avahi-daemon[12607]: Registering new address record for 192.168.122.1 on virbr0.IPv4. Apr 12 20:22:21 unmedia kernel: virbr0: port 1(virbr0-nic) entered listening state Apr 12 20:22:21 unmedia kernel: virbr0: port 1(virbr0-nic) entered listening state Apr 12 20:22:21 unmedia dnsmasq[19079]: started, version 2.75 cachesize 150 Apr 12 20:22:21 unmedia dnsmasq[19079]: compile time options: IPv6 GNU-getopt no-DBus i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-DNSSEC loop-detect inotify Apr 12 20:22:21 unmedia dnsmasq-dhcp[19079]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h Apr 12 20:22:21 unmedia dnsmasq-dhcp[19079]: DHCP, sockets bound exclusively to interface virbr0 Apr 12 20:22:21 unmedia dnsmasq[19079]: reading /etc/resolv.conf Apr 12 20:22:21 unmedia dnsmasq[19079]: using nameserver 192.168.10.1#53 Apr 12 20:22:21 unmedia dnsmasq[19079]: read /etc/hosts - 2 addresses Apr 12 20:22:21 unmedia dnsmasq[19079]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses Apr 12 20:22:21 unmedia dnsmasq-dhcp[19079]: read /var/lib/libvirt/dnsmasq/default.hostsfile Apr 12 20:22:21 unmedia kernel: virbr0: port 1(virbr0-nic) entered disabled state Apr 12 20:38:02 unmedia sshd[24198]: Accepted none for root from 192.168.10.248 port 50879 ssh2 Apr 12 20:52:51 unmedia kernel: mdcmd (63): spindown 19 Apr 12 20:52:52 unmedia kernel: mdcmd (64): spindown 21 Apr 12 20:52:54 unmedia kernel: mdcmd (65): spindown 9 Apr 12 20:52:54 unmedia kernel: mdcmd (66): spindown 10 Apr 12 20:52:54 unmedia kernel: mdcmd (67): spindown 11 Apr 12 20:52:55 unmedia kernel: mdcmd (68): spindown 14 Apr 12 20:52:56 unmedia kernel: mdcmd (69): spindown 17 Apr 12 20:52:58 unmedia kernel: mdcmd (70): spindown 1 Apr 12 20:52:58 unmedia kernel: mdcmd (71): spindown 3 Apr 12 20:52:59 unmedia kernel: mdcmd (72): spindown 4 Apr 12 20:53:00 unmedia kernel: mdcmd (73): spindown 8 Apr 12 20:53:01 unmedia kernel: mdcmd (74): spindown 12 Apr 12 20:53:01 unmedia kernel: mdcmd (75): spindown 13 Apr 12 20:53:01 unmedia kernel: mdcmd (76): spindown 15 Apr 12 20:53:02 unmedia kernel: mdcmd (77): spindown 16 Apr 12 20:53:02 unmedia kernel: mdcmd (78): spindown 20 Apr 12 20:53:03 unmedia kernel: mdcmd (79): spindown 5 Apr 12 20:53:05 unmedia kernel: mdcmd (80): spindown 6 Apr 12 20:53:08 unmedia kernel: mdcmd (81): spindown 26 Apr 12 20:53:11 unmedia kernel: mdcmd (82): spindown 2 Apr 12 20:53:18 unmedia kernel: mdcmd (83): spindown 22 Apr 12 20:53:24 unmedia kernel: mdcmd (84): spindown 23 Apr 12 20:53:26 unmedia kernel: mdcmd (85): spindown 7 Apr 12 20:53:26 unmedia kernel: mdcmd (86): spindown 18 Apr 12 20:53:26 unmedia kernel: mdcmd (87): spindown 24 Apr 12 20:53:27 unmedia kernel: mdcmd (88): spindown 25 Apr 12 20:53:27 unmedia kernel: mdcmd (89): spindown 27 Apr 12 21:01:01 unmedia sshd[27286]: Accepted none for root from 192.168.10.248 port 51560 ssh2
April 13, 201610 yr Did a little more testing related to the high load/unresponsive server tonight. I re-enabled all the disabled dockers and queued up some downloads. I now have a par2 repair stuck in the download queue that puts enough IO strain on the server to have it lock up within 10 minutes of booting. I've rebooted 3 times to ensure that it will lock up consistently. ...[snipped]... tail -n 50 /var/log/syslog Apr 12 20:22:19 unmedia rc.unRAID[18670][18674]: Processing /etc/rc.d/rc.unRAID.d/ start scripts. Apr 12 20:22:20 unmedia kernel: device virbr0-nic entered promiscuous mode Apr 12 20:22:21 unmedia avahi-daemon[12607]: Joining mDNS multicast group on interface virbr0.IPv4 with address 192.168.122.1. Apr 12 20:22:21 unmedia avahi-daemon[12607]: New relevant interface virbr0.IPv4 for mDNS. Apr 12 20:22:21 unmedia avahi-daemon[12607]: Registering new address record for 192.168.122.1 on virbr0.IPv4. Apr 12 20:22:21 unmedia kernel: virbr0: port 1(virbr0-nic) entered listening state Apr 12 20:22:21 unmedia kernel: virbr0: port 1(virbr0-nic) entered listening state Apr 12 20:22:21 unmedia dnsmasq[19079]: started, version 2.75 cachesize 150 Apr 12 20:22:21 unmedia dnsmasq[19079]: compile time options: IPv6 GNU-getopt no-DBus i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-DNSSEC loop-detect inotify Apr 12 20:22:21 unmedia dnsmasq-dhcp[19079]: DHCP, IP range 192.168.122.2 -- 192.168.122.254, lease time 1h Apr 12 20:22:21 unmedia dnsmasq-dhcp[19079]: DHCP, sockets bound exclusively to interface virbr0 Apr 12 20:22:21 unmedia dnsmasq[19079]: reading /etc/resolv.conf Apr 12 20:22:21 unmedia dnsmasq[19079]: using nameserver 192.168.10.1#53 Apr 12 20:22:21 unmedia dnsmasq[19079]: read /etc/hosts - 2 addresses Apr 12 20:22:21 unmedia dnsmasq[19079]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses Apr 12 20:22:21 unmedia dnsmasq-dhcp[19079]: read /var/lib/libvirt/dnsmasq/default.hostsfile Apr 12 20:22:21 unmedia kernel: virbr0: port 1(virbr0-nic) entered disabled state Apr 12 20:38:02 unmedia sshd[24198]: Accepted none for root from 192.168.10.248 port 50879 ssh2 I'm in no way an expert here, but what's curious is that the internal bridge is set up, then disabled. That would leave anything using it hanging. That may not be important though, as nothing has had time to begin using it. Some possible steps to alter what's happening, try removing dnsmasq from the equation (don't know if you can). And try disabling avahi, just to see if anything changes. And perhaps try it without the internal bridging.
April 13, 201610 yr Is it possible to have turbo write on a share and not other? Would be awesome Not possible. Sent from my Nexus 6 using Tapatalk Guess it is time for a future request then Thanks
April 13, 201610 yr What is the disadvantage of Turbo Write? It made a huge difference in my test array with 4 8tb seagate drives. Difference between sustained 16MB to 40MB moving Terabyte of files.
April 13, 201610 yr What is the disadvantage of Turbo Write? It made a huge difference in my test array with 4 8tb seagate drives. Difference between sustained 16MB to 40MB moving Terabyte of files. See search tips in my sig
April 13, 201610 yr What is the disadvantage of Turbo Write? It made a huge difference in my test array with 4 8tb seagate drives. Difference between sustained 16MB to 40MB moving Terabyte of files. All array drives must be spun up for any writes. Normal operation only parity + target drive must be spun up for a write. This may not really be a disadvantage, particularly when it allows you to complete the write much quicker, allowing all drives to spin back down again much sooner than otherwise.
April 13, 201610 yr What is the disadvantage of Turbo Write? It made a huge difference in my test array with 4 8tb seagate drives. Difference between sustained 16MB to 40MB moving Terabyte of files. See here for a discussion of of Turbo Write: http://lime-technology.com/forum/index.php?topic=47839.0 By the way, you have the perfect type of array for its use with only four drives in your array. A slight penalty of increased power usage vs the cost of any parity drive.
April 13, 201610 yr I'm in no way an expert here, but what's curious is that the internal bridge is set up, then disabled. That would leave anything using it hanging. That may not be important though, as nothing has had time to begin using it. Some possible steps to alter what's happening, try removing dnsmasq from the equation (don't know if you can). And try disabling avahi, just to see if anything changes. And perhaps try it without the internal bridging. Thanks for the feedback. I've been waiting for my parity to rebuild since last night before trying any of your suggested changes. However, while the parity was rebuilding I did slowly start enabling dockers and monitored their effect on the load of the system. I went in thinking Plex would be a huge load on the system, but in the end I was able to load all dockers except Sabnzbd. Once the parity rebuilt, I enabled Sabnzbd and the load went through the roof and the web ui stopped responding. Sniffing around while that happened, I noticed that for some reason I was unable to read (ls) the Incomplete folder (on disk 7, not user share) that Sabnzbd was also trying to read from when it hung. After a reboot I started a btrfs scrub on the drive in question to make sure it wasn't an FS error that was causing Sabnzbd to lock up, which would in turn took out the web ui. So while the jury is still out until the scrub is finished (i'm 1TB into a 3 TB drive), it seems that if I don't run the Sabnzbd docker then all is well. My server was stable enough to rebuild dual 4TB parity drives in about 15 hours without Sabnzbd running. Interestingly, as a test I've created an Ubuntu JEOS 14.04 VM and installed Sabnzdbd there. It has been happily running stable for the last hour or so, even with the host server load staying above 15 on the system. So it's almost as though Docker is choking when Sabnzbd would have a lot to process, but KVM keeps chugging along. I would have the thought with less overhead in Docker it would perform better. So, again thanks for the suggestions, but I'm reluctant to try them at the moment as things are somewhat stable, with everything at least working. I still need figure my way through the various combinations of technologies, but in the end it is amazing what this one box is doing. It's: - a 73 TB media server - a 9 TB ZFS RAID5 server with Samba and 3 years and counting of Windows Previous Versions for all my documents, family photos, etc. - a downloading powerhouse (Sabnzbd, Sonarr, FlexGet, etc) - a CrashPlan offsite backup enabler for my 9TB of critical data above, plus selected media (thanks to the 9p passthrough) I've consolidated multiple machines into this thing and thanks to KVM, passthrough, 9p, virtio and all the other Linux and Unraid goodness its running on less electrical power, less equipment and more then adequate performance for home use. I know I sound like an infomercial/fanboy, but it really is awesome when you step back and think about it (and when everything works! lol). So, good job LT/volunteers/open source communities!
Archived
This topic is now archived and is closed to further replies.