tucansam

Turns out unraid has been writing syslogs to my pihole for a while after my last reboot failure, forgot I set that up. After trimming a, 18GB log file, here's the last few hours from when I last interactively logged in, until this morning: Oct 21 16:42:38 ffs2 kernel: eth0: renamed from vethef4de4f Oct 21 16:42:38 ffs2 rc.docker: headphones: started succesfully! Oct 21 16:42:39 ffs2 kernel: eth0: renamed from veth03888e9 Oct 21 16:42:39 ffs2 rc.docker: radarr: started succesfully! Oct 21 16:42:39 ffs2 kernel: eth0: renamed from veth39d4e10 Oct 21 16:42:39 ffs2 rc.docker: sonarr: started succesfully! Oct 21 16:42:40 ffs2 tips.and.tweaks: Tweaks Applied Oct 21 16:42:40 ffs2 unassigned.devices: Mounting 'Auto Mount' Remote Shares... Oct 21 16:42:40 ffs2 sudo: root : PWD=/ ; USER=root ; COMMAND=/bin/bash -c /usr/local/emhttp/plugins/unbalance/unbalance -port 6237 Oct 21 16:42:40 ffs2 kernel: eth0: renamed from veth547d63d Oct 21 16:42:41 ffs2 rc.docker: unifi-controller: started succesfully! Oct 21 16:42:49 ffs2 unassigned.devices: Remote Share '//FFS1/backups' is not set to auto mount. Oct 21 16:50:18 ffs2 unassigned.devices: Warning: shell_exec(/sbin/arp -a 'FFS1.local' 2>&1) took longer than 10s! Oct 21 16:50:47 ffs2 unassigned.devices: Warning: shell_exec(/sbin/arp -a 'FFS1' 2>&1) took longer than 10s! Oct 21 16:51:02 ffs2 unassigned.devices: Warning: shell_exec(/sbin/arp -a 'FFS1' 2>&1) took longer than 10s! Oct 21 16:52:00 ffs2 root: Fix Common Problems Version 2022.10.17 Oct 21 16:54:15 ffs2 kernel: vethef4de4f: renamed from eth0 Oct 21 16:54:24 ffs2 kernel: veth03888e9: renamed from eth0 Oct 21 16:54:36 ffs2 kernel: veth39d4e10: renamed from eth0 Oct 21 17:02:09 ffs2 unassigned.devices: Warning: shell_exec(/sbin/arp -a 'FFS1.local' 2>&1) took longer than 10s! Oct 21 17:05:24 ffs2 unassigned.devices: Warning: shell_exec(/sbin/arp -a 'FFS1.local' 2>&1) took longer than 10s! Oct 21 17:05:39 ffs2 unassigned.devices: Warning: shell_exec(/sbin/arp -a 'FFS1.local' 2>&1) took longer than 10s! Oct 21 17:15:01 ffs2 unassigned.devices: Warning: shell_exec(/sbin/arp -a 'FFS1.local' 2>&1) took longer than 10s! Oct 21 17:16:39 ffs2 unassigned.devices: Warning: shell_exec(/sbin/arp -a 'FFS1.local' 2>&1) took longer than 10s! Oct 21 17:16:53 ffs2 unassigned.devices: Warning: shell_exec(/sbin/arp -a 'FFS1' 2>&1) took longer than 10s! Oct 21 17:18:09 ffs2 unassigned.devices: Warning: shell_exec(/sbin/arp -a 'FFS1.local' 2>&1) took longer than 10s! Oct 21 17:18:38 ffs2 unassigned.devices: Warning: shell_exec(/sbin/arp -a 'FFS1' 2>&1) took longer than 10s! Oct 21 17:19:24 ffs2 unassigned.devices: Warning: shell_exec(/sbin/arp -a 'FFS1.local' 2>&1) took longer than 10s! Oct 21 17:20:52 ffs2 unassigned.devices: Warning: shell_exec(/sbin/arp -a 'FFS1' 2>&1) took longer than 10s! Oct 21 17:22:09 ffs2 unassigned.devices: Warning: shell_exec(/sbin/arp -a 'FFS1.local' 2>&1) took longer than 10s! Oct 21 17:23:43 ffs2 emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog Oct 21 17:26:13 ffs2 unassigned.devices: Warning: shell_exec(/sbin/arp -a 'FFS1.local' 2>&1) took longer than 10s! Oct 21 17:27:42 ffs2 unassigned.devices: Warning: shell_exec(/sbin/arp -a 'FFS1' 2>&1) took longer than 10s! Oct 21 17:32:00 ffs2 unassigned.devices: Warning: shell_exec(/sbin/arp -a 'FFS1' 2>&1) took longer than 10s! Oct 21 17:32:10 ffs2 unassigned.devices: Warning: shell_exec(/sbin/arp -a 'FFS1.local' 2>&1) took longer than 10s! Oct 21 17:33:01 ffs2 unassigned.devices: Warning: shell_exec(/sbin/arp -a 'FFS1.local' 2>&1) took longer than 10s! Oct 21 17:39:14 ffs2 sshd[23216]: Connection from 192.168.0.14 port 52014 on 192.168.0.5 port 22 rdomain "" Oct 21 17:39:15 ffs2 sshd[23216]: Postponed keyboard-interactive for root from 192.168.0.14 port 52014 ssh2 [preauth] Oct 21 17:39:16 ffs2 sshd[23216]: Postponed keyboard-interactive/pam for root from 192.168.0.14 port 52014 ssh2 [preauth] Oct 21 17:39:16 ffs2 sshd[23216]: Accepted keyboard-interactive/pam for root from 192.168.0.14 port 52014 ssh2 Oct 21 17:39:16 ffs2 sshd[23216]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0) Oct 21 17:39:16 ffs2 sshd[23216]: Starting session: shell on pts/0 for root from 192.168.0.14 port 52014 id 0 Oct 21 17:48:47 ffs2 sshd[23216]: Connection closed by 192.168.0.14 port 52014 Oct 21 17:48:47 ffs2 sshd[23216]: Close session: user root from 192.168.0.14 port 52014 id 0 Oct 21 17:48:47 ffs2 sshd[23216]: pam_unix(sshd:session): session closed for user root Oct 21 17:48:47 ffs2 sshd[23216]: Transferred: sent 734128, received 9136 bytes Oct 21 17:48:47 ffs2 sshd[23216]: Closing connection to 192.168.0.14 port 52014 Oct 21 21:14:13 ffs2 ntpd[1687]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized Oct 21 21:14:13 ffs2 rsyslogd: action 'action-2-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2002.0 try https://www.rsyslog.com/e/2359 ] Oct 22 00:00:01 ffs2 kernel: mdcmd (37): check NOCORRECT Oct 22 00:00:01 ffs2 kernel: Oct 22 00:00:01 ffs2 kernel: md: check_array: not started Oct 22 03:10:10 ffs2 ntpd[1704]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized Oct 22 03:10:10 ffs2 rsyslogd: action 'action-2-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2002.0 try https://www.rsyslog.com/e/2359 ] Oct 22 06:59:21 ffs2 webGUI: Successful login user root from 192.168.0.24 Oct 22 07:00:01 ffs2 crond[1725]: exit status 3 from user root /usr/local/sbin/mover &> /dev/null Oct 22 07:06:08 ffs2 ool www[32507]: /usr/local/emhttp/plugins/dynamix/scripts/rsyslog_config Oct 22 07:06:10 ffs2 rsyslogd: [origin software="rsyslogd" swVersion="8.2002.0" x-pid="483" x-info="https://www.rsyslog.com"] start

I restarted the parity check and woke up to the same condition... a rebooted server. I'll enable syslog and let it try again. The only hardware change I've made in 10 years are disk changes, and I've upgraded two parity and one member disk in the last week.

Parity check started, many hours later I checked on it and came to a server with a GUI but the array was not started. Said the parity check had errors, and after checking uptime it had obviously rebooted. Diags attached on the off chance that there's something useful there. ffs2-diagnostics-20221021-1643.zip

New cables for a bunch of drives. syslog is free of errors after a few minutes, speeds are slowly starting to climb.

Current operation started on Fri 14 Oct 2022 07:31:34 PM MST (today) Total size:12 TB Elapsed time:5 minutes Current position:2.35 GB (0.0 %) Estimated speed:410.1 KB/sec Estimated finish:338 days, 3 hours, 53 minutes LOL!!!!!!!!!!! I'm going to swap cables completely and see what gives.

Disk 20 is 11 months old and I see no SMART errors that are of concern, correct me if I'm wrong. I stopped, powered down, re-seated stuff, and started the reconstruction again. Same speeds are happening.

Not even sure what that means. But Disk20 is not being reconstructed, if it matters.

No change. Current operation started on Fri 14 Oct 2022 04:59:38 PM MST (today) Total size: 12 TB Elapsed time: 1 hour, 53 minutes Current position: 68.7 GB (0.6 %) Estimated speed: 12.5 MB/sec Estimated finish: 11 days, 36 minutes

Current operation started on Fri 14 Oct 2022 04:59:38 PM MST (today) Total size:12 TB Elapsed time:3 minutes Current position:618 MB (0.0 %) Estimated speed:2.8 MB/sec Estimated finish:49 days, 7 hours, 47 minutes -- Current operation started on Fri 14 Oct 2022 04:59:38 PM MST (today) Total size: 12 TB Elapsed time: 14 minutes Current position: 10.2 GB (0.1 %) Estimated speed: 7.0 MB/sec Estimated finish: 19 days, 22 hours, 45 minutes Just replaced my two 8TB reds with two 12TBs, and all went swimmingly. Now replaced a 6TB blue member disk with an identical 12TB, and reconstruct is jumping between 2.8MB/s, jumping to 25MB/s, and sometimes 44MB/s, but never exceeding it, ever. Mostly it settles back down to 2-7MB/s. I was hoping after a few minutes, whatever was maybe happening after a reboot to swap in the new disk would finish, and things would pick up. I'll monitor it again in a few hours, but right now its settled at an amazingly slow speed. Diags. ffs2-diagnostics-20221014-1710.zip

I just popped two 12TB refurbs in, I'm swapping out my 8TB Reds. The new refurbs have 17 hours on them. 17 hours. I'll advise what the other two have once parity is sync'd and I start swapping data drives.

Is there a simple command that will reliably test disk performance, similar to iperf3? Looking at ways to improve array performance, and with recent replies to another thread indicating that unraid only performs as fast as the slowest disk (something I learned long ago and then forgot), I'm left with either looking up the MFG performance specs of each individual disk (20 total, huge mix and match), or just running a benchmark of sorts to see which disks I should start replacing one by one. Thanks.

They have a warranty (90 days), and I'm going to be downgrading the number of disks in my array, so there will be a lot of copy/move and parity build/check sequences to blast the disks to death and test their overall health. Right now I have some WD Reds going on six years of continuous spinning. A few years ago I threw away my last Conner and Maxtor IDE 254MB disks from the dark ages of computing -- they would still power up and read/write data, although they were LOUD, I found some old documents on them from when I was a kid. And I've had dozens of 80s and 90s era unix workstations with 1 and 2GB Barracuda's that ran and ran and ran. Hopefully modern disks have a similar lifespan.

Thank you. Is it still considered best practice to place parity disks on a separate (in my case onboard) controller? I'm looking at reconfiguring to have two m.2 SSDs for docker and cache, two 7200rpm parity disks on the MB SATA, and 14 disks spread evenly across two LSI HBAs in x8 slots. Theoretically.

ok. Combo of 5400 and 7200rpm disks, LSI 92xx series, modern gen9 i5-based, and I'm looking for "in general" best practices for maximizing performance. Using rsync to copy from one disk to another, I get 60-80mbs avg, and if I queue up a second copy at the same time, both transfers drop to 9mbs.

In a theoretical system with 16 disks, would it be best to place all 16 on the same 16-port controller, or split them into groups (of 8, or even 4) and run across multiple controllers? From a performance standpoint.

After a couple of years of shucking, and after learning the truth about SMR and realizing more than half my array is built from SMR drives, I just picked up four 12tb refurbs rom serverpartdeals.com -- will arrive mid-week and I will begin swapping out my SMRs. Thank you for the recommendation, they were WAY cheaper than amazon. I'll see how many power-on hours they have when the arrive; hopefully not much. My current 8TB reds, dual parity, have nearly 6y of power on time on them.....

I shut down the server, plugged the disk directly into a SATA port and the power supply, bypassing the USB adapter completely, and it has been mounted and I am now moving data..... USB did not work.

"sdy" is showing under UD's historic devices as the old Disk 15, same serial number.

Encrypted with the same key as everything else in the array, yes. Syslog does have this: Oct 9 09:00:06 ffs2 ntfs-3g[30186]: ntfs_attr_pread_i: ntfs_pread failed: Input/output error Oct 9 09:00:06 ffs2 kernel: Buffer I/O error on dev sdy1, logical block 44, async page read Oct 9 09:00:06 ffs2 ntfs-3g[30186]: Failed to read vcn 0x0: Input/output error Oct 9 09:05:06 ffs2 ntfs-3g[30186]: ntfs_attr_pread_i: ntfs_pread failed: Input/output error Oct 9 09:05:06 ffs2 ntfs-3g[30186]: Failed to read vcn 0x0: Input/output error Oct 9 09:05:06 ffs2 kernel: Buffer I/O error on dev sdy1, logical block 44, async page read Oct 9 09:10:06 ffs2 ntfs-3g[30186]: ntfs_attr_pread_i: ntfs_pread failed: Input/output error Oct 9 09:10:06 ffs2 ntfs-3g[30186]: Failed to read vcn 0x0: Input/output error Oct 9 09:10:06 ffs2 kernel: Buffer I/O error on dev sdy1, logical block 44, async page read Although I have no idea what "sdy1" is, as the old disk comes up under UD as "sdz"

Via USB, unraid sees the disk, but "Mount" is not an option. Only "Format" is an option. All disks in the array are XFS, encrypted, if that matters. I can take other disks (FAT32 etc) and they appear and can be mounted just fine. But my old Disk 15 can only be "formatted" according to Unassigned Devices.

Can I plug it in with USB, and do I have to reboot after I do?

Complete. What is the best way to mount the old disk 15 to pull data? Will unraid see a USB-connect hard disk if I plug it in (without a reboot), or should I use another linux system? The FS is encrypted, if that matters.

Parity is syncing, so far so good, fingers crossed.

Done. Parity is being rebuilt. Assuming all goes well, I will mount the old Disk 15 on another system and copy data back over. There are several (four) disks under "Historical Devices." I have never seen that before.

One last question before I begin. Should I do this with the "new" Disk 15, and then mount the old drive with unassigned devices or on another system for restore, or should I put the old Disk 15 in, run New Config, and when everything is back up, replace it with its replacement disk as originally planned?