cyberstyx

The rebuild finished successfully with 0 errors. I also did a sample check on file structure, all files are there and all are working as should. I have also attached the diagnostics file if you think you want to have a look. I will start tackling the libvirt.img corruption issue probably tomorrow, I have some unraid OS backups if needed and the config of the VMs has not changed in a long time. I will also check the suggestion about the proper usage of SSD in unraid as suggested. Thank you all again for your help, especially JorgeB. Having 4+ hardware fails one after the other (one PSU and various SATA cables) and many errors due to that was something tackled only by experts. Christos. tower-diagnostics-20231002-1857.zip

Hello JorgeB, Replaced SATA cable, did a filesystem check -n on the disk and restarted the rebuild.

I stopped the operation, after 195,362,860 writes, Disk9 was giving Errors. I have attached the diagnostics file. While rebuilt was running, shared folders where not working properly. The configuration for them was there (in Shares tab), I could see the shared folders over the network but they were empty. When I checked a share folder from the console I got "/bin/ls: reading directory '.': Input/output error". Disk contents from /mnt disks were there. When I stopped rebuilding, the share folder contents where visible again. I started the Array in Maintenance Mode so I could do a file system check on Disk 9 (with flag -n). I got this: Phase 1 - find and verify superblock... superblock read failed, offset 0, size 524288, ag 0, rval -1 fatal error -- Input/output error I 've stopped for further instructions now. tower-diagnostics-20231001-0005.zip

After a week's delay from the shop to send me the bought PSU, I got the replacement a few days ago and had time today to remove the server PC from its installed location and swap the PSU. I followed the suggested steps again to make a new config and start the array and then replace the disk, the array is now being rebuilt. Hopefully it will finish tomorrow morning without any new surprises, and I will share my news then.

Thank you for that info. I will read more about this and ask you again when I restore the system

Since Disk1 was on MB controller and Disk5 was on PCI controller, it is probably a PSU issue. Will come back to this as soon as I get a new PSU and replace all power cabling. Thanks for you help JorgeB, have a nice Sunday.

Unfortunately the files are inaccessible. Rebuild finished without errors, file structure and files are there, but again the video files are unplayable and the text files are NULL characters. Performing and ls - lsR on /mnt/disk2 gives sometimes results like: ? d????????? ? ? ? ? ? HowTo/ /bin/ls: cannot open directory './User Christos/RaspberryPi3/HowTo': Structure needs cleaning With our without a "structure cleaning" message all files are not rebuild (sampled randomly in different directories around 15 files). After the rebuild finished and after waiting for a few minutes to check for any status changes in the GUI, when I tried to open the first file from Disk2 to check for its contents, Disk9 got disabled. You know better, but to me it looks like Parity cannot be used anymore to rebuild Disk 2. What concerns me more now is the stability of my system: the constant disablement of different, even newly added disks and if any newly added content in the array will be there for future access. I don't mind re-downloading some of the contents I like from Disk2, but the system should operate as designed. Even though the system is old (old CPU, old MB, old RAM, old PSU), it is operating as should, and the disks are new and newish. The system, due to power consumption, is usually OFF, and I turn it on for home work or media access. It is not on 24/7 therefore the disks are not on all the time. Let me know what you think, if you want me to format Disk2, add some new content on it to see if everything is ok, use previous procedure to bring Disk9 online, and if there are any checks that could be made on hardware and OS configuration side to make some good assumptions about the stability of the whole system. But at this point I am not going to invest more cash on it for buying hardware for testing as I have other expense priorities. tower-diagnostics-20230917-1130.zip

Check finished Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... agf_freeblks 8620987, counted 8620468 in ag 0 sb_fdblocks 498026383, counted 498027136 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 4 - agno = 2 - agno = 5 - agno = 3 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (1:2050985) is ahead of log (1:1850812). Format log to cycle 4. done Driver mounted, all other drives appear ok so far, rebuild started. Speed is slow, it will take ~20 hours to complete (speed will drop to 50MB/s later on) Total size:6 TB Elapsed time:5 minutes Current position:24.7 GB (0.4 %) Estimated speed:90.0 MB/sec Estimated finish: 18 hours, 17 minutes Will report back as soon as I have news, hopefully tomorrow with a successful completion. Thank you JorgeB.

I stopped - started the Array in Maintenance mode. File system type for Disk 2 is "auto" and there is no Check button. The other disks have all options available. tower-diagnostics-20230916-1429.zip

Logs attached. tower-diagnostics-20230916-1344.zip

You mean by creating a new config etc. Will start that procedure in 30' as soon as I am back from my chores.

Unfortunately the rebuilt got paused due to Disk 8 going disable. Disk 1 was a SATA HDD on the motherboard, Disk 8 is an SSD disk (for VMs and containers) on an expansion PCI card. The operation got paused at 5.5%, which was probably after 2-3 hours and while I was sleeping. In the morning I saw the Pause status and I resumed it - later on I saw on you post you wanted me to cancel and post new diagnostics. Resume had the same behavior as my initial post here - it was rebuilding with a speed of 3.9GB/s and finished in 26', not a normal speed of 50-100MB/s and 18+ hours. All contents on Disk 2 are said to be there, but they are actually empty / NULL files. I opened a few movie files, they cannot be played, I opened a few text files, they are NULL characters. The rebuilt was not completed successfully. I proceeded doing the same procedure as with Disk 1, shut the PC down, checked the power cable, replaced the SATA cable. Turned it back on, the array was down, all disks were present, when I started the array everything is mounted but Disk 8 is disabled. During BIOS post all disks are reported present (6 disks on M/B , 2 PCI expansions each with 2 disks). Disk 8 (SSD) SMART checks are completed without errors but I cannot see the SMART log. On the monitor of the physical unraid server I see messages every 5-10" for md2 about metadata I/O error, metadata corruption and that I should unmount and xfx_repair it. There is nothing on the web GUI that would suggest something is wrong about Disk2, only that Disk8 is disabled. On the Main page there is constant Reads and Writes without anyone accessing the server - there are no errors. The current situation is the GUI suggested that everything is ok with Disk 8 been disabled (but active and healthy). But Disk 2 is not successfully rebuilt and has some sort of corruption which is not shown on the GUI (it shows mounted, active, healthy with 4TB of data). tower-diagnostics-20230916-1050.zip

Normally it should have been a basic operation that I have done 20+ times in the past 7 years, the multiple errors put me off that 's why I did ask for help rather than try to further troubleshoot it on my own. The help was invaluable. Yes, parity is not a backup, actual work files are backed up on an external SSD and on cloud storage. Thank you again.

Everything looks great now, disk is been rebuilt but will take a long time to complete, as expected from large and slow disks. You can consider this issue solved. Thank you a lot for you help!

Changed the SATA cable and re-sat the power cable, Disk 1 looks back online and I can browse its contents. There is a SMART health error from its UDMA CRC error count which is expected and not to worry I guess? Should I stop the array, add the installed disk as Disk 2 and start the rebuild? tower-diagnostics-20230915-2113.zip