NetApp DS4486 - Having issues after update to Unraid 6.12.4 from 6.10.x

[live4soccer7]

I am unable to access these disks in the sense that I can't view data. It shows they are full. There are no temp stats. If I click on the log for a disk it shows this and fills up as fast as "humanly possible". It shows they are there and still in the pools I put them in. It shows they are spun up, etc....

 

Here is one pool I created, so you can see.

 

 

 

Nov 29 09:12:15 Tower kernel: BTRFS error (device sdbh1): bdev /dev/sdbn1 errs: wr 5840, rd 112653, flush 0, corrupt 0, gen 0

 

I have 48 of these disks and they are all acting the same. Again, no issues prior to the update. They were viewable, writable, providing smart data etc... prior to the update.

 

Any ideas at all would be greatly appreciated and helpful.

Edited December 2, 2023 by live4soccer7

[JorgeB]

Please post the diagnostics.

[live4soccer7]

@JorgeB Thank you for taking a look. You were there when I had a couple questions regarding this setup 2-3 yrs ago (the netapp shelf and the stats).

 

 

Edited December 2, 2023 by live4soccer7

[JorgeB]

The syslog already rotated so can't see the start of the problem, please reboot and post new diags after array start or after the problem starts again.

[live4soccer7]

Thank you. I will get it rebooted today and report back.

[live4soccer7]

While going to reboot this morning I noticed 6.12.5 was available. I applied the update and rebooted. It seems things are back to normal.

[live4soccer7]

The issue is resolved.

[JorgeB]

If it happens again try getting the diags as soon as possible to see the beginning of the problem.

[live4soccer7]

Will do. Thanks.

[live4soccer7]

@JorgeB It seems to have cropped back up again. The entire shelf offline with the same errors as the original post in this thread. I'll have to reboot and try and catch it when it happens, which can be quite tough since it takes a while and doesn't notify me.

[JorgeB]

You can enable the syslog server.

[live4soccer7]

Thanks. I'll take a look.

[live4soccer7]

I am also getting these on sonarr/radarr imports. This is also on the normal array and not on the disk shelf.

 

[v3.0.8.1507] System.IO.IOException: Input/output error at System.IO.FileSystem.LinkOrCopyFile (System.String sourceFullPath, System.String destFullPath) [0x000b9] in <533173d24dae460899d2b10975534bb0>:0 at System.IO.FileSystem.MoveFile (System.String sourceFullPath, System.String destFullPath) [0x0003a] in <533173d24dae460899d2b10975534bb0>:0 at System.IO.File.Move (System.String sourceFileName, System.String destFileName) [0x00083] in <533173d24dae460899d2b10975534bb0>:0

 

I went into mc and I am unable to create a folder. It gives me an I/O errors there as well.

 

Definitely something wrong. I'll reboot this evening. It takes a while to reboot and bring all services back online when you're running a lot.

 

 

Edited December 19, 2023 by live4soccer7

[live4soccer7]

These logs won't contain the disk shelf root issue from when it started, however it may have clues as to why it is impossible to take the array offline. 

 

 

tower-diagnostics-20231219-1542.zip

[live4soccer7]

I have enabled syslog services to log locally on a cache drive. I'm not sure if it takes a reboot, but I didn't not see any in the folder I selected.

[live4soccer7]

after rebooting the radarr and sonarr files (on the unraid array) transferred without issue automatically. I have attached the syslog that is from right after I did a reboot. I set to log to flash since logging to cache didn't work. All of the disks from the diskshelf are, again, recognized after the reboot. These disks are not a part of the unraid array, but are separate pools.

 

everything appears to working just fine right now, but I'm sure that will change. I'll check disk logs once or twice a day and update the latest syslog once I notice the issue comes back.

syslog

Edited December 20, 2023 by live4soccer7

[live4soccer7]

I probably won't have more logs for at least a few days. We have a planned power outage here and I'll have to take the server offline for a bit, restarting this process.

[live4soccer7]

I have just shutdown the system before our planned outage today. The disks were all working properly and the array was able to be stopped / taken offline, while when there were disks issues (after a few days or a week) I was unable to take the array offline with the Stop array button. I'll post up new logs after a fresh reboot since it will log to the flash as soon as possible. That was only setup after it was booted last time.

[live4soccer7]

Syslog after rebooting with it starting w syslog service active.

 

syslog (2)

[live4soccer7]

I have lost connection to MOST of the disks now, but the disk errors aren't showing up yet in the disk logs. Here is the current syslog. I believe the lost connection started yesterday or late the day before, which would be the 6th/7th.

 

 

syslog (3)

[JorgeB]

It would be strange losing connection with the disks without anything logged, but I do not see anything relevant, where do you see the issue?

[live4soccer7]

We lost power again, so everything rebooted. I hadn't gotten the actual error yet. I was thinking it was a preliminary issue to the primary one originally posted. I would try and spin up the disks and it they would just not spin up. If I access the shares/disks/pools on a pc then it actually does spin them up. I have set them now to never spin down as I don't want them to spin down.