January 5, 20251 yr Came home from vacation and my wife was encountering issues uploading her camera footage (4k files) to our server which is an array of ZFS mirrors. It's been working fine for over a year now with no issues. To eliminate her PC as the issue, I attempted to copy from my PC to the server, and encountered the same. When writing these large files, the first 2GB or so flies, then the transfer falls on it's face to 0 b/s and after a few minutes fails completely on both Windows 11 and macOS. I was however able to create a small text file in the same share without issue so permissions were not the culprit. I then attempted rebooting the server, only to discover the array takes literal hours (up to 5) to start and fully mount up. This was all yesterday (Friday), and today I spent all day troubleshooting attempting to solve. Performing pv copies from a command line will transfer large files, but at very very slow speeds, Krusader, appears to hang, but maybe it works if I just let it sit long enough (we're talking up to an hour to move a 8 GB video file). Mid-afternoon today (Saturday) I have upgraded to the RC-2 of 7.0 to see if the ZFS upgrades would solve this problem; spoiler alert, they didn't. Attached are my diag zip, syslog, and a screenshot of the Windows error when I try to copy a large file after it sits for 2-3 minutes on 0 byte / sec speed. and I'm hoping for some guidance and help here as I've run out of ideas despite scouring the net and reddit for any ZFS related wisdom out there to help me figure out what's going on. unraid-diagnostics-20250104-1856.zip unraid-syslog-20250104-1755.zip
January 5, 20251 yr Community Expert when was the last time you had a smart test / scrub done to the pools?
January 5, 20251 yr Community Expert ok reviewing the diag file... zfs shows good. Quote ivy ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 sdg1 ONLINE 0 0 0 sdh1 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 sdi1 ONLINE 0 0 0 sdj1 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 sdk1 ONLINE 0 0 0 sdl1 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 sdm ONLINE 0 0 0 sdn1 ONLINE 0 0 0 mirror-4 ONLINE 0 0 0 sde1 ONLINE 0 0 0 sdd1 ONLINE 0 0 0 mirror-5 ONLINE 0 0 0 sdc1 ONLINE 0 0 0 sdf1 ONLINE 0 0 0 since your now on beta 7 rc2. for ech pool it is fully upgraded now? zpool status zpool upgrade pool name ? zpool upgrade cache zpool upgrade ivy With crusader. taht have been some incidents with ZFS and docker. Make sure you are using a xfs docker image: Per release notes you may want the storage driver to be overlay2. https://docs.unraid.net/unraid-os/release-notes/7.0.0/#known-issues are your disk cmr or smr or mixed. as a 5 hour startup seems werid but on par for smr disks. I'm reviewing the syslog and other stuf in diag atm...
January 5, 20251 yr Community Expert as it seems you have got the bug: Please make sure any plugins are also up to date.
January 5, 20251 yr Community Expert as i'm not seeing anything out right int he logs. was looking for: Errors or warnings related to ZFS modules. Disk I/O errors or signs of hardware failure. Network-related bottlenecks (if the storage is accessed over a network). Configuration issues affecting ZFS performance. Observations: ZFS Loaded: The logs confirm that the ZFS module (v2.2.7-1) is loaded correctly, and the pool is initialized. ZFS Operations: A scrub operation was completed successfully, with no data errors (errors: No known data errors). The ZFS pool's mount options include atime=off and compression=on. Error Messages: Network Errors (Nginx): Several errors about "limiting requests" related to excessive request rates by the client (e.g., 192.168.1.172). This could indicate a potential network bottleneck or misconfiguration when clients attempt to access the server. Time Sync Issues: kernel reports TIME_ERROR: Clock Unsynchronized indicates the system clock might not be properly synchronized, potentially impacting time-sensitive operations. No Disk I/O Errors Detected: The logs don’t show any direct disk errors or ZFS-related panic messages, suggesting no immediate hardware failures. beta 7 has a known ntp time sync issues. st manualy or only set the 1st one. Other generic test: Monitor Resource Usage: During file transfers, monitor CPU, memory, and disk I/O using tools like htop, iotop, or zpool iostat. Adjust ZFS Settings: Test with compression=off to see if performance improves for large file transfers. If the sync property is set to standard, consider temporarily changing it to disabled for testing (only if you have a UPS). Synchronize Time: Fix time synchronization issues by configuring NTP (Network Time Protocol). Check for Software Updates: Ensure Unraid and its ZFS module are updated to the latest stable versions. Run Disk Diagnostics: Use smartctl to check the health of your drives. Test with Different File Systems: If feasible, set up a test array using XFS to determine if the issue persists. This can help identify if ZFS is the culprit. Adjust Sync Settings: Temporarily set sync=disabled on your ZFS pool to see if there's an improvement in performance. Monitor the system closely to ensure stability. Maintain Adequate Free Space: Ensure your ZFS pool has sufficient free space. If it's nearly full, delete unnecessary files or expand the pool. Conduct Hardware Checks: Run diagnostics on your server's hardware to rule out any physical issues. Verify Network Integrity: Examine your network setup for any potential issues that could be affecting file transfers. I'm not sure if you edited any of zfs settings... ?Sync Settings: The sync property in ZFS can influence performance. Setting sync=disabled has been reported to improve performance by almost 10 times in some cases. However, this setting can risk data loss in the event of a power failure. If you have a reliable UPS, this risk is mitigated. Adjust ZFS Settings: ?compression zfs issue. Test with compression=off to see if performance improves for large file transfers. If the sync property is set to standard, consider temporarily changing it to disabled for testing (only if you have a UPS). Command to Disable Sync zfs set sync=disabled cache zfs set sync=disabled ivy Verify the Changes zfs get sync cache zfs get sync ivy example output: NAME PROPERTY VALUE SOURCE cache sync disabled local ivy sync disabled local Command to Disable Compression zfs set compression=off cache zfs set compression=off ivy Verify: zfs get compression cache zfs get compression ivy example output: NAME PROPERTY VALUE SOURCE cache compression off local ivy compression off local
January 5, 20251 yr Community Expert Solution There may be an issue with one of the disks, try running the diskspeed docker test on all disks.
January 5, 20251 yr Author Thanks team, it looks like a bad disk has been located. It took running an inplace rebalance script to get it to completely fault out. Once it did, and offlined in the array, speeds a blisteringly fast now. Replacement disk has been ordered. That said, anyone know of a better way to sniff out a bad disk rather than having to wait on a symptom like this to show up?
January 6, 20251 yr Community Expert Not really general troubleshooting. Usually you get smart error that indicate to check the disk. Disk due to have a MTF (Mean time to failure) usually 3-5 years. You seemed to be asymptomatic until the HDD gave up the ghost... Other disk general troubleshooting: SMART Monitoring: Self-Monitoring, Analysis, and Reporting Technology (SMART) is built into most modern hard drives and SSDs. It provides indicators of drive health, such as reallocated sectors, pending sectors, and overall health status. Regularly checking SMART data can help identify drives that are starting to fail. Regular Disk Scans: Performing routine disk scans can detect bad sectors and file system errors. Tools like chkdsk on Windows or fsck on Linux can be scheduled to run periodically. Performance Monitoring: Keep an eye on disk performance metrics, such as read/write speeds and access times. A noticeable decline in performance can be an early sign of disk issues. System Logs: Regularly review system logs for disk-related errors or warnings. Operating systems often log disk I/O errors, which can indicate failing hardware. Dedicated Monitoring Tools: Consider using dedicated disk monitoring tools that provide real-time alerts and detailed health reports. For example, Unraid users can utilize the "Dynamix System Temperature" plugin to monitor disk temperatures, as overheating can lead to disk failure. Regular Backups: While not a detection method, maintaining regular backups ensures data safety in case of sudden disk failure. Edited January 6, 20251 yr by bmartino1
January 6, 20251 yr Author The plot thickens. Apparently I cannot copy anything to the array now over SMB. I can however make copies of files over SSH. I incorrectly assumed things would return to normal while this 1 bad disk was disabled, but that is not the case. Anyone have ideas on how to get things functional again while I wait on the replacement disk to arrive?
January 6, 20251 yr Community Expert 8 hours ago, cpsmith516 said: I cannot copy anything to the array now over SMB Do you mean you cannot access the share, or it just fails to copy?
January 6, 20251 yr Author I can access the share, when I copy a file from a windows device it sits on 0% and fails after some minutes with a try again button. I can copy files locally on device using ssh from one folder to another.
January 6, 20251 yr Community Expert you may need to zfs offline the disk via terminal and stop start the array. zpool offline <pool_name> <device> whats the output of zpool status don't stop the unraid arary. turn of any and all dockers, vms, lxc... temp bring down the broken pool htat lost the disk. zpool offline mypool sda example output. pool: mypool state: DEGRADED scan: none requested Unraid is fickle with ZFS when a disk i missing as zfs doen't understnad the pool array when a disk is missing. It is better to have that pool offline until the new disk arives for a resliver operation.
January 6, 20251 yr Community Expert That's weird, post new diags after a transfer attempt, but suspect that there may not be anything logged.
January 7, 20251 yr Author 15 hours ago, bmartino1 said: you may need to zfs offline the disk via terminal and stop start the array. zpool offline <pool_name> <device> whats the output of zpool status don't stop the unraid arary. turn of any and all dockers, vms, lxc... temp bring down the broken pool htat lost the disk. zpool offline mypool sda example output. pool: mypool state: DEGRADED scan: none requested Unraid is fickle with ZFS when a disk i missing as zfs doen't understnad the pool array when a disk is missing. It is better to have that pool offline until the new disk arives for a resliver operation. Yep, already set the disk to offline before making the post actually. Way ahead of you there. Still same result with 0% transferred. Only thing I haven't done yet is reboot the server or stop/start the array since setting the disk to offline; and that's due to the rebalance script still running against one of the folders. It should be done overnight, which will have all my vdevs on an even keel now. I'll attempt a reboot tomorrow and see if I'm able to copy things over afterward. In the mean time a new disk is on the way but won't be here until Saturday. The wife is starting to get antsy not being able to backup her YouTube content and having it all sit on her SSD's.
January 7, 20251 yr Community Expert are you using a HBA? this sounds like storage contoler failure or irq lockups. double check cables.
January 11, 20251 yr Author Ok new disk has been added to the array, resilvered, and pool is online again. However, Unraid does not seem to be in love with this new disk. It shows up as "new disk" on the Main page, instead of green online and healthy. Another side effect I'm now having, not sure if it's related at all or what's going on is I cannot reliably access over the DNS name. What I mean by this is, I can navigate shares, I can create small text files, but opening up a video file or copying large files like ISOs doesn't work; but if I do it over the IP address it works fine. This just started popping up after the issues with slow copy and subsequent failing disk which has now been replaced. I probably need to start a fresh thread for this new and separate issue, but just wanted to update the original thread with this information in case anyone that has already commented has seen something like this before and knows where to look to fix these two new issues.
January 11, 20251 yr Community Expert 42 minutes ago, cpsmith516 said: Unraid does not seem to be in love with this new disk. Try re-importing the pool, should resolve that, are you one v6.12 or v7? Also, in case you missed, the pool has corrupt data. You should delete/restore those files from a backup.
January 11, 20251 yr Author I am on 7.0 stable release as of this morning; was on RC2 prior. I did see the files, and have already taken action on them; nothing critical there. Will try export and import (again) but I have done this already multiple times as part of reboots and so far nothing has changed, especially the ability to write large files out to shares via DNS name.
January 11, 20251 yr Community Expert To reimport the pool: on main click on that pool and then "remove pool" back on main, create a new pool with the same name and number of slots assign the pool device(s), leave the filesystem set to auto start the array to import the pool
January 11, 20251 yr Author 13 minutes ago, JorgeB said: Try re-importing the pool, should resolve that, are you one v6.12 or v7? Also, in case you missed, the pool has corrupt data. You should delete/restore those files from a backup. Same issue after export import. Disk still showing blue square.
January 11, 20251 yr Author 7 minutes ago, JorgeB said: To reimport the pool: on main click on that pool and then "remove pool" back on main, create a new pool with the same name and number of slots assign the pool device(s), leave the filesystem set to auto start the array to import the pool did that, now we're all green. copying over DNS name still hosed though. diags attached unraid-diagnostics-20250111-1110.zip
January 11, 20251 yr Author Will make new thread for visibility. Have figured out the bug/cause of the dns copy issue. Thank you all for your help in getting this figured out.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.