Not sure what else to say, unraid was working perfectly fine and then started getting bunch of errors in log (attached) that crashes it.  Haven't made any significant changes.

thoughts?

 

 

ds9-syslog-20250226-0403.zip

btrfs is detecting data corruption on both pool devices, and there are multiple segfaults logged, start by running memtest.

OK memtest reported one error....3 times?  See below.  Does memtest ever actually finish or stop, it's been running for 24 hours (and curiously stopped at one point with a big "PASS" written on screen where I had to hit space and then it just continued?

 

Thoughts?

I have 4 sticks of 8GB ram in here, 2 of which are brand new and the other 2 are a couple of years old.

 

It's also weird that system was working perfectly fine for a while and then not.  I've never run memtest before but presumably this error would've been there originally.

 

 

 

 

1 hour ago, darwinsbeard said:

OK memtest reported one error....3 times? 

A single error is a problem, you must replace the affected stick(s).

how do i know which stick(s) gave the error?

Remove one of them and retest until you find the problem.

ok...so memtest doesn't report which stick gives the error?  That seems odd.

 

 

OK, so I don't think it was memory....removed the older sticks and ran memtest and it passed.  Was running well for about a day, then earlier I upgraded to 7.0.1 and seemed to be working ok, but while playing a video it just stopped and now the syslog is full of btrfs errors.  See attached.

Prior to the update, I also don't think memtest had anything to do with it but Unassigned Devices did, if I didn't mount my usb UD it would be ok, but once i mounted it within a few hours erros would occur.

 

any thoughts?  this is starting to get very frustrating

 

ds9-syslog-20250304-0547.zip

There's an issue with the pool filesystem:

Mar  4 00:31:54 ds9 kernel: BTRFS critical (device sdg1: state EA): unable to find ref byte nr 1566436642816 parent 0 root 5 owner 109102 offset 484413440 slot 9
Mar  4 00:31:54 ds9 kernel: BTRFS error (device sdg1: state EA): failed to run delayed ref for logical 1566436642816 num_bytes 151552 type 178 action 2 ref_mod 1: -2

 

With btrfs I recommend backup and reformat, but if it keeps happening, there may be an underlying hardware issue

Quote

With btrfs I recommend backup and reformat

Is there a documentation / process for doing the backup and reformat?  Is it just copying the pool data to some place (like the array), reformat the pool, copy back?  I wouldn't want to have to redo all my dockers etc.

Quote

there may be an underlying hardware issue

While I'm skeptical of underlying hardware issues as all of this hardware was working perfectly fine for years (in windows and Unraid), is there a process for simply removing SSD drives from the pool?  I have pool of 4 SSD drives all of different sizes and I could easily just remove the smallest one but is there a procedure to do so?  Do I just stop the array, remove the disk from the pool, shutdown, unplug the drive and then reboot?

 

Thanks

19 minutes ago, darwinsbeard said:

Is it just copying the pool data to some place (like the array), reformat the pool, copy back?

Yep.

 

 

OK, somehow I feel like that is going to explode in my face and cause far more issues, but I'll try it if it keeps happening.

 

well, it was running fine for about a day, but then another set of errors happend, see attached.  THese errors seem different than the last time, so not sure what is going on if every day it's a different set of problems.  NOte that the last set of errors occurred just now as I logged in.

 

Mar 5 04:40:02 ds9 move: mover: finished Mar 5 07:27:05 ds9 kernel: NVRM: GPU at PCI:0000:06:00: GPU-aebc4fe3-4feb-68d8-58f9-b13064ff03ea Mar 5 07:27:05 ds9 kernel: NVRM: Xid (PCI:0000:06:00): 140, An uncorrectable ECC error detected (possible firmware handling failure) DRAM:1227137661, LTC:0, MMU:0, PCIE:0 Mar 5 07:27:05 ds9 kernel: NVRM: GPU 0000:06:00.0: RmInitAdapter failed! (0x62:0x40:2521) Mar 5 07:27:05 ds9 kernel: NVRM: GPU 0000:06:00.0: rm_init_adapter failed, device minor number 0 Mar 5 07:27:09 ds9 kernel: NVRM: GPU 0000:06:00.0: RmInitAdapter failed! (0x62:0x40:2521)

ds9-syslog-20250305-1227.zip

Mar  5 03:42:55 ds9 kernel: SQUASHFS error: Failed to read block 0x2a0f0fc: -5
Mar  5 03:42:55 ds9 kernel: SQUASHFS error: xz decompression failed, data probably corrupt
Mar  5 03:42:55 ds9 kernel: SQUASHFS error: Failed to read block 0x2a0f0fc: -5
Mar  5 03:42:55 ds9 kernel: SQUASHFS error: xz decompression failed, data probably corrupt
Mar  5 03:42:55 ds9 kernel: SQUASHFS error: Failed to read block 0x2a0f0fc: -5

 

These typically mean a flash drive problem, though it could also be bad RAM for example.

The ram passed memtest so shouldn't be that.

so is next step to replace my flash drive now????  Is there a recommended or officially supported brand?

This is getting nowhere fast, all dockers were stopped and when I logged into the GUI it worked for 2 seconds but trying to get to syslog failed and entire GUI stopped working. Plugged in a monitor to the machine and this is what is shown:

I've also enabled the syslog server and and it's supposed to be writing to "appdata" but I don't see any files there, but attached are the diagnostics from this morning.

 

Again to recap:

This machine was working perfectly fine for my 30 day trial period, then bought Lifetime

The machine has also worked flawlessly with Windows and DrivePool for years

Memtest is PASS

 

 

ds9-diagnostics-20250305-0735.zip

And now getting these errors, it seems different all the time

 

copying this here trying to pin down this behavior

• updated the nvidia driver to open source version and (so far) haven't seen above issue
• reboooted and no issues, UNTIL i started jellyfin from a client and immediately noticed the below and jellyfin docker stopped

can a container cause this level of system instability?  isn't that the whole point of containerization to avoid?

 

2 hours ago, darwinsbeard said:

can a container cause this level of system instability?

It's been known to happen before, could also the GPU/driver.

Always something new....now getting these.

 

 

 

Edited March 7, 20251 yr by darwinsbeard

Enable the syslog server and post that after a crash, there may be more info there, typically what matters most is the first call trace.

Thanks @JorgeB I had actually done that a while ago, but for some reason it never logged anything to my "LOcal syslog folder" which I selected my cache appdata share, see below.  But when i look at /mnt/user/appdata I don't see anything other than the folders for my docker containers (see below).  I now just enabled it to mirror to my flash, let's see if that does anything.

 

 

 

 

 

With those settings the syslog file will be going to the ‘logs’ folder on the flash drive.    You would only get files put into ‘appdata’ if you have put your server’s address into the Remote Server field.

Thanks @itimpi now I see it in /appdata. 

Suggest modifying the help documentation to make it clearer that if logging to appdata, the IP of the unraid server needs to be put in the "Remote" IP address.  Wasn't super clear (at least to me!)

4 hours ago, darwinsbeard said:

Suggest modifying the help documentation

That is already mentioned in the docs.