One data disk has red cross, all disks showing again in unassigned devices, most data folders blank

[extremeaudio]

As subject suggests, one disk has a red cross. Other disks and parity disk are green balls. 

 

There is a read check on at the moment.

 

All the data disks etc are also being listed under Unassigned Devices.

 

When I access some data folder over SMB, most folders are empty and no files show.

 

Diagnostics attached.

akserver-diagnostics-20220604-2114.zip

[JorgeB]

Jun  4 20:25:00 AKserver kernel: ata5.00: link offline, clearing class 1 to NONE
Jun  4 20:25:00 AKserver kernel: ata5.00: disabled
...
Jun  4 21:00:51 AKserver kernel: ata6.00: link offline, clearing class 1 to NONE
Jun  4 21:00:51 AKserver kernel: ata6.00: disabled
...
Jun  4 21:05:00 AKserver kernel: ata3.00: link offline, clearing class 1 to NONE
Jun  4 21:05:00 AKserver kernel: ata3.00: disabled
...
Jun  4 21:08:44 AKserver kernel: ata4: SATA link down (SStatus 0 SControl 300)
Jun  4 21:08:44 AKserver kernel: ata4.00: disabled

 

These four disks are dropping offline, check power, could also be a controller issue since all four are the same Adaptec controller.

[extremeaudio]

I removed and cleaned and reseated the SATA controller and everything seems better than before in the dashboard. Still have one disk with red X. File system is reiserfs. 

 

Is this the process to follow - https://wiki.unraid.net/Manual/Storage_Management#Checking_a_File_System

 

 

[itimpi]

6 minutes ago, extremeaudio said:

Is this the process to follow - https://wiki.unraid.net/Manual/Storage_Management#Checking_a_File_System

 

At this point Unraid should be emulating the disabled disk.  Using that process will check and/or fix any corruption on the emulated drive (and will do no harm if not needed).

 

To clear the disable state you need to rebuild the drive as described here in the documentation.   Since the rebuild basically makes the physical drive match the emulated drive it is a good idea to check that the emulated drive does in fact have the content you expect - if not I would suggest posting back here for advice on the best way to proceed.

[extremeaudio]

Uh oh, same errors repeat. All disks have billions of writes and errors and are showing in the unassigned devices again. What should I do? Even my cache disk somehow unassigned itself, not sure if its related to this issue.

akserver-diagnostics.zip

[JorgeB]

Same issue, again all 4 devices on the Adaptec controller.

[extremeaudio]

I stopped the disk check, and followed what you said. Which was to disable the disk, then re enable it and let it rebuild from parity. This is after swapping out the SATA controller for a new one. Now all disks are green. But a folder to which I was copying etc when all this occurred is empty.

Edited June 12, 2022 by extremeaudio

[JorgeB]

Post new diags, there might be some filesystem corruption, some data loss might also have occurred due to the previous issues.

[extremeaudio]

diagnostics attached.akserver-diagnostics-20220612-1445.zip

[JorgeB]

Jun 12 13:44:11 AKserver kernel: REISERFS error (device md1): vs-5150 search_by_key: invalid format found in block 1159198912. Fsck?

 

Check filesystem on disk1.

 

P.S. also a good idea to convert all existing reiserfs disks to XFS, reiser is not recommended for a long time.

[extremeaudio]

Not sure how to change the disk filesystem. Can you please link to a guide if any. Thanks for the help.

[extremeaudio]

check disk results as below

 

reiserfsck 3.6.27 Will read-only check consistency of the filesystem on /dev/md1 Will put log info to 'stdout' ########### reiserfsck --check started at Sun Jun 12 17:54:51 2022 ########### Replaying journal: Trans replayed: mountid 852, transid 184450, desc 2187, len 1, commit 2189, next trans offset 2172 Replaying journal: | | 0.4% 1 trans Trans replayed: mountid 852, transid 184451, desc 2190, len 1, commit 2192, next trans offset 2175 Trans replayed: mountid 852, transid 184452, desc 2193, len 1, commit 2195, next trans offset 2178 Trans replayed: mountid 852, transid 184453, desc 2196, len 1, commit 2198, next trans offset 2181 Trans replayed: mountid 852, transid 184454, desc 2199, len 1, commit 2201, next trans offset 2184 Replaying journal: |= / 2.1% 5 trans Trans replayed: mountid 852, transid 184455, desc 2202, len 1, commit 2204, next trans offset 2187 Replaying journal: Done. Reiserfs journal '/dev/md1' in blocks [18..8211]: 6 transactions replayed Checking internal tree.. finished Comparing bitmaps..Bad nodes were found, Semantic pass skipped 4 found corruptions can be fixed only when running with --rebuild-tree ########### reiserfsck finished at Sun Jun 12 20:31:52 2022 ########### block 1159198912: The number of items (8) is incorrect, should be (3) the problem in the internal node occured (1159198912), whole subtree is skipped block 1689905582: The level of the node (16304) is not correct, (1) expected the problem in the internal node occured (1689905582), whole subtree is skipped block 1689985600: The level of the node (47940) is not correct, (1) expected the problem in the internal node occured (1689985600), whole subtree is skipped block 1690160786: The level of the node (57731) is not correct, (1) expected the problem in the internal node occured (1690160786), whole subtree is skipped vpf-10640: The on-disk and the correct bitmaps differs.

 

[ChatNoir]

7 hours ago, extremeaudio said:

Not sure how to change the disk filesystem. Can you please link to a guide if any. Thanks for the help.

 

[JorgeB]

15 hours ago, extremeaudio said:

corruptions can be fixed only when running with --rebuild-tree

Run it again with the --rebuild-tree flag, note that it will take a few hours.

[extremeaudio]

So the reiserfsck --rebuild-tree completed after 4-5 excruciating days on my celeron tower and finally finished with the log as here - https://pastebin.com/QBc31GXy

 

I stopped the array and restarted it. And I think all my data is well. Thanks for all the help, everyone!

 

[extremeaudio]

While everything seems ok, the flash drive has trillions of writes to it. And its showing in the unassigned devices section too.

 

Any precautions to take? Somehow unable to download diagnostics too. It just says starting services and then nothing happens.

[JorgeB]

8 minutes ago, extremeaudio said:

the flash drive has trillions of writes to it. And its showing in the unassigned devices section too

That suggests it dropped offline, try another USB port, ideally USB 2.0.

[extremeaudio]

Very thankful for the excellent, precise help.