July 10, 20232 yr Hello, Due to Google Drive axing their unlimited storage plans, I've decided to build out my media box onto Unraid. I'm an older Unraid user, but this is a new build. Hardware: Motherboard - ASRock - H570M-ITX/ac CPU - Intel® Core™ i5-10400 CPU @ 2.90GHz Memory - 32GB(2x Crucial CT16G4DFS832A.C8FE) HBA - Adaptec ASR-71605 HDD - 4x HGST 12TB SAS - H0H72121CLAR12T0 (4 more on the way) SSD - 3x Intel 800GB SATA - SSDSC2BB80 (5 more on the way) HBA and HDD's are both second hand from ebay. Ran badblocks over disks prior to using them in the array and that passed no problem. The rest of the hardware is repurposed, was working previously. Now to the issue - I have noticed about 5 times in the past week or so that the box freezes and will not recover. Completely dead - Console is unresponsive, nothing is logged to syslog. Have to hard reset to recover. At the moment, the array disks are connected to the HBA, but the cache SSD's are connected to the motherboard's SATA headers. I'm downloading with rclone to cache at around 90MB/s and then transferring to array with mover periodically. From what I've noticed, the freezes typically occur when the mover is running. Under this scenario, the cache is being written to at ~90MB/s and then data is being moved from cache to array at ~200MB/s. One interesting thing I've noticed is that the HBA LEDs are off when it's the box is unresponsive, when they're usually lit up. Another thing is that the HBA firmware was newer than was available on the microchip site(https://storage.microsemi.com/en-us/speed/raid/asr/fw_bios/71605_fw_b32118_zip.php). I have just re-flashed this firmware to see if that makes a difference. I'm thinking it's likely the HBA, but I wasn't sure whether an HBA could completely freeze the kernel in this fashion. Has anyone seen this behaviour and have any ideas before I write it off and grab a 9201-16i or similar. Diagnostics: home-serv-1-diagnostics-20230710-1745.zip arcconf GETCONFIG output: arcconfig.txt Edited July 10, 20232 yr by nayr1991
July 10, 20232 yr Author FWIW, here's the syslog surrounding the hang. My monitoring picked up the hang at 17:07 and I let it sit for ~10mins to see if it wound recover or log something. Jul 10 17:02:45 home-serv-1 move: file: <SNIP> Jul 10 17:03:12 home-serv-1 move: file: <SNIP> Jul 10 17:03:41 home-serv-1 move: file: <SNIP> Jul 10 17:04:09 home-serv-1 move: file: <SNIP> Jul 10 17:04:37 home-serv-1 move: file: <SNIP> Jul 10 17:17:45 home-serv-1 file.activity: Starting File Activity Jul 10 17:17:45 home-serv-1 emhttpd: Starting File Activity... Jul 10 17:17:45 home-serv-1 file.activity: File Activity inotify starting Jul 10 17:17:45 home-serv-1 file.activity: File Activity inotify exiting Jul 10 17:17:46 home-serv-1 rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="5883" x-info="https://www.rsyslog.com"] start Jul 10 17:17:47 home-serv-1 root: Delaying execution of fix common problems scan for 10 minutes Jul 10 17:17:47 home-serv-1 emhttpd: /usr/local/emhttp/plugins/user.scripts/backgroundScript.sh "/tmp/user.scripts/tmpScripts/startup_stuff/script" >/dev/null 2>&1/usr/local/emhttp/plugins/user.scripts/backgroundScript.sh "/tmp/user.scripts/tmpScripts/gdrive_mount/script" >/dev/null 2>&1 Jul 10 17:17:47 home-serv-1 emhttpd: Starting services... Jul 10 17:17:47 home-serv-1 emhttpd: shcmd (63): /etc/rc.d/rc.samba restart Jul 10 17:17:47 home-serv-1 wsdd2[4624]: 'Terminated' signal received. Jul 10 17:17:47 home-serv-1 winbindd[4627]: [2023/07/10 17:17:47.734513, 0] ../../source3/winbindd/winbindd_dual.c:1957(winbindd_sig_term_handler) Jul 10 17:17:47 home-serv-1 winbindd[4627]: Got sig[15] terminate (is_parent=1) Jul 10 17:17:47 home-serv-1 winbindd[4636]: [2023/07/10 17:17:47.734511, 0] ../../source3/winbindd/winbindd_dual.c:1957(winbindd_sig_term_handler) Jul 10 17:17:47 home-serv-1 winbindd[4636]: Got sig[15] terminate (is_parent=0) Jul 10 17:17:47 home-serv-1 wsdd2[4624]: terminating. Jul 10 17:17:47 home-serv-1 winbindd[4750]: [2023/07/10 17:17:47.734672, 0] ../../source3/winbindd/winbindd_dual.c:1957(winbindd_sig_term_handler) Jul 10 17:17:47 home-serv-1 winbindd[4750]: Got sig[15] terminate (is_parent=0) Jul 10 17:17:49 home-serv-1 root: Starting Samba: /usr/sbin/smbd -D Jul 10 17:17:49 home-serv-1 smbd[6055]: [2023/07/10 17:17:49.881243, 0] ../../source3/smbd/server.c:1741(main) Jul 10 17:17:49 home-serv-1 smbd[6055]: smbd version 4.17.7 started. Jul 10 17:17:49 home-serv-1 smbd[6055]: Copyright Andrew Tridgell and the Samba Team 1992-2022 Jul 10 17:17:49 home-serv-1 root: /usr/sbin/wsdd2 -d -4 Jul 10 17:17:49 home-serv-1 wsdd2[6069]: starting. Jul 10 17:17:49 home-serv-1 root: /usr/sbin/winbindd -D
July 10, 20232 yr Community Expert 7 hours ago, nayr1991 said: the cache is being written to at ~90MB/s and then data is being moved from cache to array at ~200MB/s I assume you have mixed up these numbers. 200 for writes to cache might be reasonable. 90 for writes to the array (such as mover) would be much better than expected for array and 200 would not be possible for spinners with parity. Perhaps you are already overflowing to the array when you think you are writing to cache. Mover is intended for idle time. It is impossible to move from cache to slower array as fast as you can write to cache. If you are writing a lot and running mover at the same time it only makes things worse. If you are doing the initial data load, don't cache unless cache has the capacity for the entire load. 7 hours ago, nayr1991 said: syslog surrounding the hang What are those scripts doing? Unrelated, your system share has files on the array.
July 10, 20232 yr Author 6 hours ago, trurl said: I assume you have mixed up these numbers. 200 for writes to cache might be reasonable. 90 for writes to the array (such as mover) would be much better than expected for array and 200 would not be possible for spinners with parity. No, those numbers are correct. I have a bwlimit on my rclone copy, as it's pulling from the internet. I don't want to saturate my gigabit line, so I limit it at 90MB/s The single-disk write performance on the SAS drives that back my array appears to top out at around 240MB/s, but on average I get 200MB/s when moving to array. The share is configured as high-water. Quote Mover is intended for idle time. It is impossible to move from cache to slower array as fast as you can write to cache. If you are writing a lot and running mover at the same time it only makes things worse. If you are doing the initial data load, don't cache unless cache has the capacity for the entire load. I get better performance and lower iowait writing to cache, as it's not sequential(I can only get 90MB/s downloading with 3+ transfers in rclone), and then moving to array. Quote What are those scripts doing? startup_stuff just sets up shell things - bash_history, .vimrc, .bashrc, etc gdrive_mount creates an rclone mount and a mergerfs union, then restarts some docker containers. Both of these are running AFTER the freeze happens, during boot.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.