dev_guy

Thanks for that! I assumed that functionality required a network server to log the data to (as is one of the options). I didn't realize you could have a local syslog file on a cache SSD. I will absolutely be enabling that on my Unraid servers immediately.

I very much wish Unraid had persistent logs so I'd have a record of all my drives that have been disabled yet are perfectly fine. But I understand why Unraid doesn't want to trash the USB thumb drive or spin up array drives to make log entries. It would be nice on systems with SSD cache, where the System folder is located on the Cache, if Unraid could store persistent logs there instead of only in RAM. It also would protect the logs in the event of power loss. That's perhaps another feature request for a future release? The above is in fact an issue with Unraid alternatives that don't use a dedicated drive for the OS. The drive spin down feature on a Qnap, for example, is basically worthless as periodic OS writes are constantly spinning up the array. But OpenMediaVault, for example, puts the OS on its own drive, typically a small SSD, and that works great. It does eat up a SATA port (or m.2 slot) but with quality small SSDs starting around $25 it's a great solution. And any Unraid system that has a cache drive could benefit from having the logs saved there. I certainly will be saving the logs next time this happens to me before powering the server off to remove, test, and potentially backup, the "failed" drive. The issue remains, and @JonathanM even agrees, Unraid is known for disabling perfectly good drives while no other NAS or server OS I've had experience with is anywhere near as prone to doing. Something is different with Unraid and causing this problem. I wish I had the logs to know more.

@JonathanM Thanks for helping explain things further. I'm not familiar enough with what say a Linux mdadm RAID 5 does if there's an I/O error but it appears to be far more resilient of transient SATA issues than Unraid. I agree there's likely a glitch somewhere up stream of the healthy drive being disabled but those glitches, including even a single CRC error, are enough to have a drive be disabled when arguably it should not be. The retry issue is easily managed. Unraid already keeps error counters and could simply issue a warning and wait to disable a drive until the retry count grows above some threshold. That would prevent retries from ever impacting performance. It's obvious the SATA data link is not robust enough to solely rely on given Unraid's parity scheme. This is especially true for the sort of inexpensive DIY-grade builds Unraid encourages using whatever old hardware someone has laying around. I want to stress in many cases there's nothing significantly wrong with the drive, cable, controller, motherboard, etc. The same disabled drive can end up working just fine on the same cable, and same SATA port, after going through all the hassle of getting Unraid to re-adopt it. I guess this is just another significant downside (another being write performance) to Unraid's parity scheme. And, like I said, all things are a trade off. I appreciate the education and would encourage the powers at be to consider a simple retry scheme which should greatly reduce all the issues around perfectly healthy drives being kicked out of the array. Such a scheme has zero downside and would save a lot of headaches.

I'm sorry but this is complete nonsense. Given that disk I/O errors are extremely rare unless a drive is seriously failing (not what we're discussing here) it will make zero difference to Unraid's performance to have a few retries, that can happen in a fraction of a second, to help prevent drives being disabled for no good reason in the extremely rare instance there's an I/O error. I'm also suggesting this all often starts with READ errors which, as you explained, results in Unraid trying to then write to the drive that it couldn't even read from. That just seems wrong especially given the write attempt may corrupt the parity of the entire array creating an even bigger problem. But, regardless of if it's a read or write error, a few retries are going to make zero difference in real world performance unless something is seriously wrong. And my whole point is Unraid disables drives when there is literally NOTHING wrong with the drive.

The read should be re-tried multiple times before any attempt is made to write. If the read passes on a retry I don't see how the parity would be corrupted? It would be better to log the read error and issue a warning. Unraid should arguably NOT try to write to a parity protected drive it can't read as that could indeed corrupt the parity as you suggest. I have various disk diagnostic software capable of a variety of tests with some granularity as to the options. It's normal for something like a surface scan test to have, by default, 3 retries for any read or write errors. That's on top of whatever the drive might be doing internally. Things like simple vibration, say your dog bumps into your server, can cause a single I/O operation to fail. Unraid should be more robust than that. You are correct modern drives have a lot going on. But they also log errors in their SMART data. And Unraid seems unique in how often it disables perfectly good drives with, at most, a few CRC errors and no other errors with not even a single uncorrectable read error. And the same drive will pass even extensive worst case diagnostics. If the devs think the current Unraid scheme is best that's fine I get everything is a trade off. But it doesn't change the fact Unraid creates headaches by disabling perfectly good drives. It forces the user to take worrying and usually time consuming action to address the non-issue with the server being in limp mode for an extended period with a higher risk of data loss. It's not an issue I've ever had with OpenMediaVault (which, like other OS options, uses simple well proven mdadm RAID), FreeNAS (using well proven ZFS), Qnap, Synology, AsusStor, etc. All of these NAS/servers have their own issues but kicking perfectly good drives out of their arrays isn't one of them. We also need to accept XFS is a thirty year old ancient file system that Unraid layers additional code, and Fuse, on top of which is arguably a recipe for problems compared to say mdadm. I get there are a lot of Unraid loyalists on this forum and the level of support is great. But it would be nice if more would acknowledge areas that could be improved instead of defending the status quo. The screenshots show an example of one of my Unraid servers. Disk 2 is perfectly fine. It has exactly 1 CRC error and that's it. It passes extensive diagnostics outside of the server without a single issue. Yet Unraid disabled it. Here are screenshots and the SMART report for the perfectly healthy drive Unraid disabled. I don't have a log of the actual event as the server has since been rebooted so I could remove the presumably "bad" drive for testing and possible data recovery. Unraid unfortunately doesn't keep persistent logs due to the USB thumb drive based operating system. But the drive turned out to have nothing wrong with it. Does anyone want to explain how it was acceptable for Unraid to disable this perfectly functioning drive with all the hassle and worry that creates? WL6000GSA6457_WOL240371821-20221114-1039 2.txt

Thanks for the helpful explanation. That might explain a lot about why Unraid drives get disabled when, in fact, there's nothing wrong with them besides perhaps a very brief unfortunately timed SATA/CRC issue. A single failed write, to me, is not cause to condemn a drive but perhaps issue a warning and try again. Unraid is the only time I've used XFS. All my other servers have used Ext4, Btrfs, and ZFS. And Unraid, of course, is also using a somewhat unique parity scheme.

The LSI controllers need heat sinks as most are an old design and generally capable of doing onboard hardware RAID with the right firmware. I have two of them and they waste power given the servers they're in are generally powered 7/24. You can get a modern SATA controller that use the same ASMedia family chip that's used on many motherboards such as the ASM1064. These run far cooler, are far more power efficient, and work just fine with Unraid. There have been issues Jmicron chips and Chinese "unknown" chips. But you can get something like an Syba or or IO Crest controller that uses an ASM series chip and they're only a few dollars more than the no-name Chinese junk. So the size of the heatsink isn't the sole indicator of the suitability of a SATA card and I'd argue most more modest builds are likely better off with a more modern, smaller, more power efficient controller.

I believe the post linked below is an example but not sure there's the log to definitively prove it. I will admit I'm mostly going by anecdotal data here from various users on the forum and my own personal experience. I have had multiple drives disabled where the only SMART issue are a few CRC errors. The drives test perfectly outside of Unraid. I'm aware if Unraid thinks the file system is corrupted on a drive it may disable it, which could be the case in the post linked below, but I've also had disabled drives pass the XFS file system check. I've copied the entire contents of nearly full disabled drives as a precaution using Linux without a single read error. Many here have reported similar experiences where they have a disabled drive that has nothing wrong with it. It could be the CRC error occurred during a write which should normally just cause a RETRY rather than a hard ERROR. I have an LSI SATA controller with high quality cables. I do think a backplane design with NO cables is a better solution but one relatively few seem to be using with Unraid. But, weirdly, in all my years of working with PCs, other DIY NAS builds using other operating systems, commercial NAS products, etc, I've never had any CRC errors that I even know of let alone having them be a problem. Nor have I ever had a drive be disabled that wasn't obviously defective. There seems to be something about Unraid that possibly even creates CRC errors and causes it to disable perfectly good drives. I will certainly capture the log and diagnostic if I have it happen again. But here's an example of the sort of thing I'm talking about:

I have multiple experiences to the contrary as supported by many posts here where there are zero write errors but Unraid still disables a drive. I agree Unraid should not disable a drive unless a write fails, but that's not how Unraid works in reality. For example there are many examples where Unraid disables a drive during a parity rebuild where there are zero writes to any drive except the parity drive.

For some reason CRC errors are an ongoing issue with Unraid. I've never had any of my other NAS builds, servers, PC's, etc. have issues with CRC errors. It's only been Unraid. By default Unraid will forever flag a drive with even a single CRC error as being a problem drive unless you tell Unraid to ignore CRC errors. But, even worse, Unraid seems to remove drives from the array for a single CRC error which is just wrong and can often be harmful. Everyone agrees CRC errors are generally minor such as someone bumped the cable causing a one time tiny harmless glitch that's automatically corrected. But, in some cases, Unraid wants to forever condemn that drive from the array which is just wrong when that same drive will pass 24 hours of extensive testing with zero errors. To put it simply I've had many drives kicked out of Unraid arrays for even a single CRC error but none of them have been shown to have any actual problems. It's something that needs to be addressed by Unraid.

For anyone else looking to mount an XFS drive on Windows I have some updated info in this post: XFS on Windows

UPDATE: Linux Reader by DiskInternals as an alternative to the non-working Linux File System by Paragon. I went ahead and paid for Linux Reader and can say it sort of works with some important limitations and issues. As I suspected, it is opening an XFS drive as a virtual file system which inhibits performance. The performance with a large number of small file operations, like indexing the drive, is about half the speed of a natively mounted drive. The performance copying large files, however, is good enough to saturate a typical USB 3.0 external drive dock (i.e. around 120 MB/sec). I've yet to test a natively SATA connected drive. Worse, at least for my needs, it will apparently only let you "mount" one drive at a time. It opens a dialog box when you mount a drive under a drive letter (which is the only way to make it available to other Windows applications) and you can't close that window, or do anything else with Linux Reader, without un-mounting the drive first. I don't see any way to mount multiple drives at once. As I mentioned in my first post, it's clearly intended for data recovery where that's less of a limitation. At least with Ext4 drives, Paragon lets you mount multiple drives with no issues. It would be awesome if I could do the same with Unraid XFS drives but, so far, Paragon can't even see the drives (see my previous post). While natively within Linux Reader it has the physical drive size correct, the "mounted" volume in Windows is reporting I have 12 TB of data (around 8.5 TB in reality) on my 10 TB XFS drive. And the actual drive size is reported as 20 TB. So I'm not sure where the disconnect is but it doesn't seem to inhibit access to the files. With an Ext4 10 TB drive with less than 3 TB of data, the drive reports correct size info in Windows with the Paragon driver but it reports as a 9TB drive that's completely full with 9TB of data under Linux Reader. So it's not just an XFS thing. I doubt any low level file operations are going to work though the virtual file system which might also be why sizes are reported wrong. So, for example, a lot of backup software likely won't work. If that was your goal, I'd suggest booting RescueZilla from a thumb drive which should be able to make a compressed image of an XFS Unraid drive. And, as others have suggested, you can do a lot booting a suitable "live" Linux distro like Ubuntu or Mint, or Parted Magic's USB drive, or other Linux-based USB "rescue" drives. But you can't use your favorite Windows tools with any of those. Linux Reader at least somewhat works where Paragon's option (at least for me) hard fails. But I'd much rather have Paragon work like it does with Ext4. But, if you're desperate, I'd suggest trying Paragon's free demo first and, if it doesn't work, you'll have to pay for Linux Reader and cross your fingers working with one drive at a time. In general, I'd suggest if you ever want to work with one or more drives in both Windows and Linux, to choose Ext4 if at all possible as XFS is a purple unicorn on Windows. XFS is an ancient file system from Silicon Graphics which finally went bankrupt in 2009 and XFS dates back to 1993 making it roughly 30 years old. While Ext4 was first released in 2008 and is around 15 years newer than XFS. The two offer similar features and performance so I see little reason to use XFS where you can avoid it. We also need a more SSD friendly file system. Unraid doesn't even officially support the use of SSDs in the array and older file systems tend to be hard on SSDs and have unnecessary overhead. The other credible, but more expensive, option for those looking for just data recovery might be UFS Explorer which comes in a basic home version starting around seventy bucks. They have a RAID version which can possibly handle things like an Unraid BTRFS RAID1 mirrored cache pool even in Windows.

@itimpi posted a few times he was unable to get Paragon's Linux File System to read XFS drives from Unraid. He submitted a support ticket with Paragon but I've not seen any posts regarding the final outcome? There are a few posts here saying it works with Unraid XFS drives and around an equal number saying it did not work. In my case Paragon can't even see an Unraid XFS drive let alone mount it. Interestingly, I took a spare otherwise identical drive, formatted it with parted to XFS, and the Paragon utility mounts it just fine. Same model drive, same USB dock, same everything. There seems to be something unique to Unraid's use of XFS that's preventing it working with Paragon. I looked at both drives with several disk utilities from the Linux command line and I can't see any differences yet one will mount in Windows and the other won't. If anyone knows the secret to getting Paragon to mount Unraid XFS drives, please let me know? Or, for that matter, if anyone has found another utility/driver that works for read-only XFS in Windows? In my case this is for data copying, de-duplication, and re-organizing. There are simply better tools for Windows than Linux for those tasks otherwise I'd just use Linux. I am aware I could run Linux in a VM and share XFS drives back to Windows but I expect the performance would be poor and I'm working with 36 TB of data. I've looked into other utilities. The most promising is Linux Reader from Disk Internals. The demo version will let me "mount" an Unraid XFS drive, and can display the files, but not do anything with them which is presumably a demo limitation. It's mainly designed for data recovery but might still work if the paid version (thirty bucks) actually works. It's appears to open an XFS drive as a virtual drive much like you might with an ISO. It does not appear to be the sort of native OS file system mount that Paragon offers. I'd prefer the Paragon solution as it presents Linux drives as native Windows drives just as if they were NTFS (and works great with Ext4). If I decide to spend the money for Linux Reader I'll report back how well it actually works. I haven't found anything else that's free, or has a free demo, that will mount an XFS drive in Windows. Any help would be greatly appreciated!

I was also generally happy with Unraid back in the 6.8.x days. But things have really changed. Can you really argue 6.11.2 should have ever been released? Is anyone at Unraid even humble about the serious mistake? Where's the participation from Unraid, versus users, in this forum over even critical issues? We're basically all on our own.

To be honest I'm really getting discouraged around trusting my data to Unraid. I updated to 6.11.1 because the SMB performance in 6.10.x was absolutely horrible compared to earlier versions. Then 6.11.2 has a severe bug that seems inexcusable and may cause data loss. And, after 4 years of many reporting the issue and even YouTube videos made on the subject, the USB installer still hangs for a great many of us at "syncing file system." And the list goes on with lots of issues that should have never been released or been fixed long ago. Unraid isn't open source and we're at the mercy of the devs. We pay for one or more licenses yet the overall experience is we're all on our own for support as if it's open source. I haven't seen anyone from Unraid participating in 99% of the threads here. Given the many recent issues I have to wonder if Unraid shouldn't just be moved to open source? We're not getting the sort of support, or quality, one can usually expect from paid software.

I have to also ask why this hasn't been fixed after 4 years? I just tried to re-install Unraid on the same flash drive for a clean 6.11.3 install and it got stuck, as always, until I renamed the drive WHILE the installer is still running which is completely counter-intuitive. Honestly, 4 years of forcing people to use workarounds for this? If I were new to Unraid it might put me off ever even trying it. I wonder how many would be paying users gave up when the installer hung which does nothing to inspire confidence in Unraid.

I would suggest using a NAS that officially supports sleep. You are trying to make Unraid do something it was never intended to do. Popular commercial NAS systems from companies like Qnap, Synology, etc, have native support for sleep because, unlike Unraid, they control the hardware, BIOS, etc. You can use a Wake On LAN packet to wake a Qnap or the power button on the front. You can also set up a power schedule for it to automatically sleep and wake one or more times a day if you like. Unraid doesn’t have that kind of functionality because Unraid is hardware agnostic and, as you have found out, some motherboards don’t support S3 sleep in a compatible way with Unraid. Yes Unraid could add a way to disable parity checks for unclean shutdowns but I have to agree with everyone else you simply shouldn’t be doing unclean shutdowns in the first place with any file server not just Unraid. Any decent file server is going to want to run a file system/parity check on a dirty shutdown. If you care about your data, you should buy even an inexpensive UPS. They start around $40 US for decent ones with USB support to prevent dirty shutdowns. I understand your view if the server is already in the sleep state the data on the drives cannot change even if the power is turned off. But the server has to do a cold boot from an unknown state. You asked for a solution to your problem and it sounds like a UPS is a solution, and is considered wise for any file server/NAS, especially if your power is interrupted as frequently as you suggest. Further many do not realize SSDs can be badly corrupted from power interruption even when they’re not being written to at the time as SSD’s are often performing housekeeping write tasks like TRIM on their own. Most only have, at best, partial power fail protection in their firmware. Any system, not just servers, with data you care about should be on a UPS or, for laptops, have at least a partially charged battery especially if you have known power problems. What you want is analogous to when the Low Oil Check Engine Light in your car comes on. You want a way to just turn the light off, never check the oil, and just keep driving. No car maker is going to offer that “feature” as you could easily damage your engine.

Thanks for the answers! I found info on replacing/upgrading a single drive cache, that's obvious enough, but nothing promising on rebuilding a RAID1 cache with a failed drive. In fact, the advice I did find in this forum suggested moving everything off the remaining working drive to the array, replacing the failed drive, formatting the modfied RAID1 cache pair, and moving the data back. I don't believe this is in the Unraid Manual but probably should be as it's not at all obvious from the GUI what to do. And thanks also for the magic command line option to get a Btrfs RAID1 drive to mount outside of Unraid. I still need to test the performance of Btrfs vs XFS on NVMe SSDs for Docker/VM use. I've seen benchmarks, like the one attached, showing Btrfs taking around twice as long as XFS for the same file operations. I don't know if that's the case with the current versions and within Unraid but is certainly something to consider. Some of my use cases include full text searches, de-duplication, and other file intensive operations running locally on the server.

I've seen several posts where people have had a cache pool drive fail that was in a "RAID-1" two slot mirrored configuration and have still had issues including having to start over with a new cache pool. I've searched but I can't find anywhere that documents how to replace a failed cache drive in a 2 drive pool and have it automatically rebuild the mirrored pair? This doesn't seem to be a GUI feature of Unraid? I do understand one cache drive can fail and you hopefully still have your data on the other drive. But it's my understanding you still have to manually repair the cache? Further many argue btrfs has many performance and stability issues (I'm happy to provide references). It's also less natively compatible for recovering data from drives outside of Unraid compared to a single XFS drive. I tried to mount a healthy btrfs mirrored Unraid cache drive in Linux and it wouldn't mount. There may be some magic obscure incantations to mount such a drive but it wasn't obvious to me. Linux insisted it was corrupt. It may have to do with Unraid not using the official RAID-1 format for cache pools? So all of the above leads me to wonder if many systems wouldn't be better off with single drive XFS cache pools that are regularly backed up to the array (or elsewhere)? I get that if you're using the cache for critical applications where you can't afford to lose even a few hours of data the downsides of btrfs, non-standard RAID, and dual drives might be worth it. But even then if a cache drive fails it seems you still have a significant problem on your hands? Am I missing something here or is the above more or less correct?

Thanks @ChatNoir. The System share was also set up when the main array was fully intact while the shares flagged as unprotected were set up after the main array was missing a drive. While I've been using Unraid for several years, have two licenses on two servers, I'm only recently starting to make several changes and exploring some new things. I appreciate all the great people here who are very willing to help even when people like me make mistakes. It's one of the best aspects of Unraid.

Thank you @itimpi @Kilrahand @JorgeB as you are correct and have hopefully provided the answer. It doesn't explain why the System share, which is also set to "Yes" is green and protected, but I agree I should have chosen "Only" if I want the shares to stay on the cache. I have more testing to do.

Delete it how? I don't see any obvious things to delete in my array? And why was a folder created in my array in the first place when this share was created to ONLY USE CACHE from the very start? This seriously seems like a problem that needs to be addressed? It's easily re-created. This isn't some fringe weird issue. It's something anyone can recreate in a few easy steps. Unraid seems to be broken if it's writing things to the array for newly created shares that are supposed to only use cache. Can you explain how I have this wrong?

Clicking on compute shows SIX BYTES on the array. Again, there seems to be no reason for this? Why did Unraid write 6 bytes of information to the array for a newly created share that's set to "Yes" or cache only? How can someone prevent Unraid from doing what it's not supposed to do?

The shares flagged as "unprotected" have no data on the array. They were created recently and have always been set to "Yes" for cache only hence have never had data on the array unless there's some bug in Unraid that writes data to the array even when you create a share that's supposed to always use a cache pool? This really seems like a bug?

The main array currently has a missing disk. But, if you'll note, the shares are set to "Yes" meaning they shouldn't care about the Unraid array as data will never be stored there. And the "System" share is happy with the situation despite being set to "Yes" on the same cache. This very much seems like a bug in the latest version of Unraid?