I have an unRAID server I recently upgraded to include an HBA (a Broadcom 9500-8i, whose firmware/BIOS/catalog I upgraded to the latest versions) and a handful more NVMe SSDs. This server has been running flawlessly for the past few weeks with a RAIDZ-2 pool (5x 6TB WD Enterprise HDDs) connected to the HBA + 2x ZFS RAID-1 pool (2x WD Red 2TB SATA SSDs) + 2x ZFS RAID-1 pool (2x Corsair 1TB NVMe SSDs - unRAID caching for the HDDs pool's share).

I decided to switch to using an unRAID Array to save power, so I bought 3x 12TB HDDs (2x Seagate Enterprise Capacity, 1x WD HC520) and my plan was to:

1. Pre-clear the drives (OK!)

2. Create an array with dual-parity + 1 data HDD formatted as XFS (OK!)
3. Copy the data from the RAIDZ-2 share via the GUI (Crash randomly during that, machine goes completely unresponsive!)

4. Stop the array

5. Pre-clear 1x12TB drive I have and 4 of the other 6TB drives

6. Add those to the array

7. Have 48TB of dual-parity-protected storage space

 

However, whenever those HDDs are set up as an unRAID array (at least dual-parity), I get a random crash - whether or not they're being used at all! (I once left the machine be, I was not even doing anything, and it froze) - and the machine goes completely unresponsive. The symptoms so far are:

1. Nothing weird in the screen, o dmesg and syslog, as far as I can tell

2. Unresponsive to input via USB

3. Network becomes unavailable - won't even respond to ping

4. As far as I can tell, the system is completely dead.

 

But if I add those 3 HDDs as another ZFS RAIDZ-1 pool, it works without fault!
Same HBA, same HDDs, same cables, nothing changed - but I'm using ZFS pools instead of an unRAID Array.

 

Note: Docker and VMs are both disabled. I do have some plugins, but nothing too crazy, I think. The only thing I think could really cause this issue would be running powertop (still from NerdTools), but it doesn't make sense why it won't crash with ZFS only, but will with the array.

Note 2: memtest86+ passed 6 times, so I assume my RAM is fine

 

I have not tried safe mode for a long time yet, as I'm hoping someone can help me figure that out. 

ps.: I'm using turbo writes because the first copy is to the single disk + the 2 parity disks and it would be too slow in the normal mode.

nas-diagnostics-20250426-1237.zip nas-diagnostics-20250426-1536.zip

Edited June 28, 20251 yr by andrebrait

Update: I have created a new ZFS Pool using the new 3 drives in RAIDZ1 mode, copied all 4TB of data I currently have over (simple copy and paste via the folder GUI), and scrubbed both pools afterwards - with zero errors found.

SMART data looks good on all drives, old and new, and no UDMA errors or anything after the upgrade - years ago I had a bad cable so some HDDs have some UDMA errors, but they stopped increasing after I switches drivers.

So I think this rules out the issue being connected to cables, any power-savings tuning or a bad HBA.

Update 2: It seems it has no problem with the exact same setup, but using btrfs instead of XFS

Update 3: now it works with XFS and I have no idea why

If it keeps crashing, enable the syslog server and post that after a crash., also a good idea to run memtest.

5 minutes ago, JorgeB said:

If it keeps crashing, enable the syslog server and post that after a crash., also a good idea to run memtest.

Memtest86+ passed 6 consecutive times and there are no mentions of anything odd in syslog. Also, it seems the syslog server itself goes down with the entire machine.

I tried plugging my KVM and, on the terminal, just tail -F /var/log/syslog and also dmesg -wH and there was nothing odd with either.

 

Now it seems to be working with XFS. The only difference is that I named the share `array` instead of `data-array`, and I formatted the drives after trying btrfs.
I'll try a clean reboot and array with the same share name to see if I can reproduce the crash.

Are there other debug-like things I can do? I'll try enabling logging to the boot drive for a while as well.

 

 

12 minutes ago, andrebrait said:

Also, it seems the syslog server itself goes down with the entire machine.

It does (not sure what else you would expect), but what it has logged up to then survives a reboot whereas the standard syslog starts afresh after a reboot.

1 minute ago, itimpi said:

It does (not sure what else you would expect), but what it has logged up to then survives a reboot whereas the standard syslog starts afresh after a reboot.

I know, but there isn't anything weird in it. Let me reproduce the crash and grab it for you, but my point was: it's probably crashing before it even has the chance to log anything. I'll try enabling logging to the USB drive.

Well, it completely went away.

I'm not sure if there was some weird interaction between the disks being pre-cleared or something, but it did not crash anymore, despite me trying my best to reproduce the issue.

Well, after some reboots and some more testing and trying to reproduce the issue, it seems it simply doesn't happen anymore.

 

After more than 10 times reproducing it before reporting.

 

Right now, I'm fairly convinced I stumbled upon a bug that only happens if the drives are all pre-cleared and you start an array from scratch with two parity drives and (?) one XFS (?) data drive + turbo writes (?)

Edited April 27, 20251 yr by andrebrait

@itimpi and @JorgeB I have been fighting this again for about a week and I am 100% able to reproduce it. Additionally, one of my HDDs started to exhibit an increasing number of reallocated sectors, so I RMA'd it and the fact I now need to restore the array gave me some more insight into this.

The machine freezes before it even writes anything to syslog or dmesg. I also have a NanoKVM connected to the machine and I ran tail -F /var/log/{syslog,dmesg} but the crash is so severe even the output to the screen contains absolutely nothing about any issue.

Things that do not affect the crash:

1. Plugins. Crash is 100% reproducible in safe mode.

2. RAM. Memtest passes multiple times and the server is rock solid until the array comes into play.

3. Read operations. As long as only read operations are performed, including parity checks and scrubbing, it's all fine. I did get one crash during a parity check, but I never got one again.

4. Docker, VMs, network activity, system load, etc. I tried multiple combinations of everything I could think of, but even with everything here disabled, I still get the crash.

Things that cause it to crash:

1. Always: trying to restore the array with the new HDD I received

2. Often: Mover and rsync'ing from the ZFS cache pool to /mnt/user0/. It's guaranteed to crash after a hundred gigabytes or so.

3. Less often / not at all sometimes: rsync'ing from the ZFS cache pool directly to one of the disks (via /mnt/diskX) and/or moving data between the disks using the unbalance plugin.

I'm really lost here because no logs contain anything useful and the syslog server (and the entire machine, apparently) dies before anything (assuming the crash produces logs) happens, but one thing is clear: the server is absolutely rock solid unless the parity-protected array is in the mix.

Is there anything I can enable to produce additional logs?

Edited June 27, 20251 yr by andrebrait

12 hours ago, andrebrait said:

Is there anything I can enable to produce additional logs?

Nope, but based on the description still sounds like a hardware issue, do you have a different board/CPU combo you could try? Also, memtest is only definitive if it finds errors, if you have multiple sticks, try using the server with just one, if the same try with a different one, that will basically rule out bad RAM.

@JorgeB I'll be damned.

It's running fine with one stick of RAM, and now I think I understand why: I think I mixed up the modules the last time I re-assembled this computer and they have different characteristics. Perhaps there is some incompatibility between them. One kit is single rank and the other kit is dual rank. There's some combination of the two kits that apparently allows it to work correctly under some circumstances (e.g. single rank in channel A and dual-rank in channel B, but not the other way around).

I'll try re-adding a second matching stick later and seeing what happens. If that works, I'll see if I can find an exact match to one of the kits and order that.

They're the same model of RAM and even use Micro dies, but one uses 16 modules and the other one just 8 modules.

I had highly doubted that could be the problem because the server is absolutely rock-solid until write operations start happening to the parity-protected array and it made no sense that my memory configuration would be unstable, but not manifest itself in any other way in months of uptime. No errors while doing massive operations between pools and other shenanigans, all stress tests passing with flying colors, but a total freeze by simply writing something to the array.

Anyway, I'll wait for the data rebuild to finish and try and debug this better.

In any case, I'll mark this as resolved.

Edited June 28, 20251 yr by andrebrait