Log full, Metadata coirruption

[Jeff in Indy]

My system ran for a year without any major hitches, now it seems the problems are non-stop...

 

After my recent debacle, I finally got the system back on track and rebuilding the Parity discs.

 

Today (2 days later), I was told my log was full and needed to be rebooted.  I dumped the diagnostics and rebooted the machine.

 

Question #1 - Why is my log full after 2 days?

 

After running diagnostics, I rebooted the machine.  I was told that discs 2 and 4 had errors on them in Fix Problems.

 

I went into Maintenance mode, and ran xfs_repair on Disc 2, first checking then fixing.  Ran it again, had to fix it again.

 

Question #2:  Is this normal?

 

Ran it on Disc 4, no errors.  Rebooted to see if errors appear on the server monitor, and I was told again that Disc 4 had issues, and to run xfs_repair (again).  I did, it found no errors, but I told it to fix it anyway.  Rebooted again, the error has not reappeared.

 

Question #3:  Is this showing a larger issue?  I'm guessing not, but I really want to get everything fixed.

 

The server is up and restarting the parity rebuild.  I DO have an error on the server monitor:

 

BTRFS:  error (device loop2) in btrfs_finish_ordered_io:2736:  errno=-5 IO failure

 

Question #4:  What should I do with this?  How do I check it?

 

I'm including the initial log full diagnostic.  I have a clean diag after I rebooted the first time, as well as screenshots of the server monitor each time I rebooted, if any of that would help.

 

kaokulk-diagnostics-20220524-0953.zip

[JorgeB]

7 minutes ago, Jeff in Indy said:

Question #1 - Why is my log full after 2 days?

It's full of ATA errors for disks 2 and 4, could be a power/connection problem, could be the controllers, both are connected to Marvell controllers, they are not recommended but if they have been working OK with the same Unraid release unlikely that they would start acting up now, they can if you updated to a new release recently.

 

Also a good idea to first run a couple of memtest passes, some issues suggest possible RAM problem.

 

 

[Jeff in Indy]

Still on 0th pass on memory test, 134 errors 18 minutes into it.  Well, crap.

 

Ordered new memory, going to pick it up this afternoon.

 

Thanks for the advice.

 

I've tried replacing data cable on #2, so IO guess I'll try to see if I can find another power cable to use.

[Jeff in Indy]

One problem solved, another arises.

 

Memory replaced, ran the Mem test, 1 complete pass, no errors (I would run it longer, but I need it up and running by morning).

I was looking at the 2 drives that kept messing up (2 and 4), and noticed that those 2 drives were using the same SATA power cord (!!!!!)  So I replaced it with another cable.  I'm hoping that helps with the ATA errors.

 

It's up and running, and Parity is starting to rebuild.  However, I've got 2 problems, I believe related:

1)  The server monitor is showing the following error: 

     BTRFS: error (device loop2) in btrfs_finish_ordered_io:2736: errno=-5 IO failure

2)  I can't get Docker to start

 

I have attached a diagnostic dump, after booting I turned off Docker, then turned it back on.  It didn't start, but I got the BTRFS message again.  I have a diag dump from initial boot, if you need it.

 

kaokulk-diagnostics-20220524-1931 Restarting Docker.zip

[Jeff in Indy]

Update:  I moved the docker.img file, and restarted Docker.  Then reinstalled the apps, all that seems to work fine.

 

If anyone notices anything else in the file, please let me know.

[trurl]

According to those last diagnostics, your docker.img failed to mount. I assum you have since fixed that.

 

Your system share has files on the array.