cyberstyx

Hello @JorgeB, If I can add something to this, I didn't find any relevant information with the search I did to avoid double posting. As I installed this card to IT mode a few days ago, from my searching I found this among other info: Flashing firmware and BIOS on LSI SAS HBAs. sas2flash and sas3flash executables are used to upgrade the firmware to a newer version. sas2flsh and sas3flsh executables are used to downgrade firmware and change modes between IR and IT. I could only find the sas3flsh file needed for 9308 as a DOS .exe file. All other files are available for DOS / Linux / UEFI shell. I did find a ZIP file at Broadcom stating that it had the sas3flsh file for Linux, but when I opened it it had a sas3flash file in it and I didn't try it on unraid to see if they mixed up the filename. TIP: If anyone tries to make a bootable USB in DOS, you may want to prepare it as an MS-DOS bootable and not as a FreeDOS one; The FreeDOS one kept cutting the VGA signal to my monitor after 30" to 1' and putting it on power-saving without any means to get picture back expect from hard resetting, where as the MS-DOS one didn't. The timing of flashing a card and getting a black screen 3" after you run a command can be confusing.

Coming back to this issue, I ordered a "10Gtek ‎LSI-3008-8I HBA Card" from Amazon for 94€ plus 2x "Internal HD Mini SAS (SFF-8643 Host) to 4 x SATA (Target) Hard Drive Cable" for 14€ each, which arrived last week. I flashed the card to IT mode on Saturday and took the expected ~15hours to rebuild a 6TB 5200rpm disk, it finished rebuilding yesterday night. The problem is now resolved, the array is up and running again. Looks like the MARVEL based cards are not suitable for unraid 6x and onwards I guess. Thanks again for your help @itimpi, @JorgeB and @trurl. PS: Flashing a card from IR mode (raid functionality) to IT mode (JBOD functionality) is mostly a pain as there is generic information around that may not suite the specific HBA card you may have and you may need to adapt. There 's quite some investigation you will have to do from generic guides and the manufacturer's info, and then you will have to self-improvise. Broadcom's info on this (who bought Avago who bought LSI) is for the lols, with very good generic info, guides and tools that don't work anymore (building a USB bootable with their suite of tools on it) and scattered to no existing links to the actual individual files you need.

I will then look for a suitable replacement card before retrying with the already given walkthrough. I will close this post, thank you for your help @itimpi, @JorgeB, @trurl

I checked its status today, the operation paused after 12 hours and at 24%, Disk 3 went to Disabled status. I will need to cancel the paused operation and try something new. I am attaching the latest info. I found out this thread Recommended controllers for Unraid suggesting to a void the MARVEL chipset and giving other viable options. Do you think the Digitus cards (chipset MARVELL 88SE9230) are giving all the issues? The two SATA expansion cards were bought in 2016 and worked fine, but then it was an older version of unraid too. tower-diagnostics-20250320-1026.zip

I replaced one cable that was old, the others had new cables (installed ~3 years ago). The PSU was also replaced ~3 years ago. Disks 3, 7 and 9 where on the same PCI SATA controller, I moved them all on the second PCI SATA controller that had Disk 8 connected on it and started over the rebuild. The controllers are 2x Digitus PCI SATA with 4 SATA connectors each, which I have them installed for many years now. There are no more free SATA connectors available except the non-used controller that had 3, 7 & 9 on it.

Yes, Disk 5 was an older issue (3 years ago?) that was tackled. I returned home, checked the progress, it looks like there are other issues now. Disk 3 stopped writing, Disks 7 & 9 (which also reported issues when I was trying to mount Disk 3 with the problematic FS but were ok after each array restart), are now showing errors while participating in the rebuild. Should I stop the rebuild and check Disks 7 & 9 FS status too? tower-diagnostics-20250319-2108.zip

The array started with Disk 3 enabled and is currently rebuilding. I will let you know as soon as it finishes (it will take many hours).

Hello @itimpi, I did a "Check Filesystem Status" without the "-n", got the following as a result: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 7 - agno = 3 - agno = 5 - agno = 2 - agno = 6 - agno = 4 - agno = 8 - agno = 9 - agno = 10 - agno = 11 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... done After the results I started the array in Normal mode, Disk 3 is still disabled (with an "X"). Docker has some issues from a few years ago when I had this and some Windows VMs running on an SSD that I eventually replaced with an HDD. I will need to properly set it up again at a later day. tower-diagnostics-20250319-1208.zip

Hello all, I booted up by unraid server last week to find Disk 3 disabled. Upon stopping the array, removing the disk, rebooting the server, mounting again the missing disk and rebooting, the sever reports Disk 3 as "Unmountable: Wrong or missing file system", but now Disk 7 and 9, although mounted and running show the same info. After rebooting the server again, Disk 3 is again disabled and Disk 7 & 9 are again mounted and running without the previous error. At the time I can't re-mount Disk 3 and try to rebuild it. I 've uploaded the diagnostics when the server starts, as well as the diagnostics after trying to re-mount Disk 3 and getting some FS errors for Disks 7 & 9 too. Any help would be much appreciated. tower-diagnostics-20250319-0923.zip tower-diagnostics-20250319-0942.zip

The rebuild finished successfully with 0 errors. I also did a sample check on file structure, all files are there and all are working as should. I have also attached the diagnostics file if you think you want to have a look. I will start tackling the libvirt.img corruption issue probably tomorrow, I have some unraid OS backups if needed and the config of the VMs has not changed in a long time. I will also check the suggestion about the proper usage of SSD in unraid as suggested. Thank you all again for your help, especially JorgeB. Having 4+ hardware fails one after the other (one PSU and various SATA cables) and many errors due to that was something tackled only by experts. Christos. tower-diagnostics-20231002-1857.zip

Hello JorgeB, Replaced SATA cable, did a filesystem check -n on the disk and restarted the rebuild.

I stopped the operation, after 195,362,860 writes, Disk9 was giving Errors. I have attached the diagnostics file. While rebuilt was running, shared folders where not working properly. The configuration for them was there (in Shares tab), I could see the shared folders over the network but they were empty. When I checked a share folder from the console I got "/bin/ls: reading directory '.': Input/output error". Disk contents from /mnt disks were there. When I stopped rebuilding, the share folder contents where visible again. I started the Array in Maintenance Mode so I could do a file system check on Disk 9 (with flag -n). I got this: Phase 1 - find and verify superblock... superblock read failed, offset 0, size 524288, ag 0, rval -1 fatal error -- Input/output error I 've stopped for further instructions now. tower-diagnostics-20231001-0005.zip

After a week's delay from the shop to send me the bought PSU, I got the replacement a few days ago and had time today to remove the server PC from its installed location and swap the PSU. I followed the suggested steps again to make a new config and start the array and then replace the disk, the array is now being rebuilt. Hopefully it will finish tomorrow morning without any new surprises, and I will share my news then.

Thank you for that info. I will read more about this and ask you again when I restore the system

Since Disk1 was on MB controller and Disk5 was on PCI controller, it is probably a PSU issue. Will come back to this as soon as I get a new PSU and replace all power cabling. Thanks for you help JorgeB, have a nice Sunday.

Unfortunately the files are inaccessible. Rebuild finished without errors, file structure and files are there, but again the video files are unplayable and the text files are NULL characters. Performing and ls - lsR on /mnt/disk2 gives sometimes results like: ? d????????? ? ? ? ? ? HowTo/ /bin/ls: cannot open directory './User Christos/RaspberryPi3/HowTo': Structure needs cleaning With our without a "structure cleaning" message all files are not rebuild (sampled randomly in different directories around 15 files). After the rebuild finished and after waiting for a few minutes to check for any status changes in the GUI, when I tried to open the first file from Disk2 to check for its contents, Disk9 got disabled. You know better, but to me it looks like Parity cannot be used anymore to rebuild Disk 2. What concerns me more now is the stability of my system: the constant disablement of different, even newly added disks and if any newly added content in the array will be there for future access. I don't mind re-downloading some of the contents I like from Disk2, but the system should operate as designed. Even though the system is old (old CPU, old MB, old RAM, old PSU), it is operating as should, and the disks are new and newish. The system, due to power consumption, is usually OFF, and I turn it on for home work or media access. It is not on 24/7 therefore the disks are not on all the time. Let me know what you think, if you want me to format Disk2, add some new content on it to see if everything is ok, use previous procedure to bring Disk9 online, and if there are any checks that could be made on hardware and OS configuration side to make some good assumptions about the stability of the whole system. But at this point I am not going to invest more cash on it for buying hardware for testing as I have other expense priorities. tower-diagnostics-20230917-1130.zip

Check finished Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... agf_freeblks 8620987, counted 8620468 in ag 0 sb_fdblocks 498026383, counted 498027136 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 4 - agno = 2 - agno = 5 - agno = 3 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (1:2050985) is ahead of log (1:1850812). Format log to cycle 4. done Driver mounted, all other drives appear ok so far, rebuild started. Speed is slow, it will take ~20 hours to complete (speed will drop to 50MB/s later on) Total size:6 TB Elapsed time:5 minutes Current position:24.7 GB (0.4 %) Estimated speed:90.0 MB/sec Estimated finish: 18 hours, 17 minutes Will report back as soon as I have news, hopefully tomorrow with a successful completion. Thank you JorgeB.

I stopped - started the Array in Maintenance mode. File system type for Disk 2 is "auto" and there is no Check button. The other disks have all options available. tower-diagnostics-20230916-1429.zip

Logs attached. tower-diagnostics-20230916-1344.zip

You mean by creating a new config etc. Will start that procedure in 30' as soon as I am back from my chores.

Unfortunately the rebuilt got paused due to Disk 8 going disable. Disk 1 was a SATA HDD on the motherboard, Disk 8 is an SSD disk (for VMs and containers) on an expansion PCI card. The operation got paused at 5.5%, which was probably after 2-3 hours and while I was sleeping. In the morning I saw the Pause status and I resumed it - later on I saw on you post you wanted me to cancel and post new diagnostics. Resume had the same behavior as my initial post here - it was rebuilding with a speed of 3.9GB/s and finished in 26', not a normal speed of 50-100MB/s and 18+ hours. All contents on Disk 2 are said to be there, but they are actually empty / NULL files. I opened a few movie files, they cannot be played, I opened a few text files, they are NULL characters. The rebuilt was not completed successfully. I proceeded doing the same procedure as with Disk 1, shut the PC down, checked the power cable, replaced the SATA cable. Turned it back on, the array was down, all disks were present, when I started the array everything is mounted but Disk 8 is disabled. During BIOS post all disks are reported present (6 disks on M/B , 2 PCI expansions each with 2 disks). Disk 8 (SSD) SMART checks are completed without errors but I cannot see the SMART log. On the monitor of the physical unraid server I see messages every 5-10" for md2 about metadata I/O error, metadata corruption and that I should unmount and xfx_repair it. There is nothing on the web GUI that would suggest something is wrong about Disk2, only that Disk8 is disabled. On the Main page there is constant Reads and Writes without anyone accessing the server - there are no errors. The current situation is the GUI suggested that everything is ok with Disk 8 been disabled (but active and healthy). But Disk 2 is not successfully rebuilt and has some sort of corruption which is not shown on the GUI (it shows mounted, active, healthy with 4TB of data). tower-diagnostics-20230916-1050.zip

Normally it should have been a basic operation that I have done 20+ times in the past 7 years, the multiple errors put me off that 's why I did ask for help rather than try to further troubleshoot it on my own. The help was invaluable. Yes, parity is not a backup, actual work files are backed up on an external SSD and on cloud storage. Thank you again.

Everything looks great now, disk is been rebuilt but will take a long time to complete, as expected from large and slow disks. You can consider this issue solved. Thank you a lot for you help!

Changed the SATA cable and re-sat the power cable, Disk 1 looks back online and I can browse its contents. There is a SMART health error from its UDMA CRC error count which is expected and not to worry I guess? Should I stop the array, add the installed disk as Disk 2 and start the rebuild? tower-diagnostics-20230915-2113.zip

The main page got refreshed after some time, for Disk 1 it shows "Unmountable: Wrong or no file system"