loungelizard

Sorry. Just tried check xfs with the -n option - so not trying any changes. My important data is safe in multiple locations 😂 but if at all possible I would like to retain my other data. It can be reorganized but it's a pain in the ass..

ok understood, it will take me some time to organize. So assuming changing my PSU or cables does not bring any solution, what would be an attempt to bring back the data? I was planning to replace my hardware anyway... So would it be wise to start with a fresh array (knowing the original of my replacement disk may still have issues or corrupt data), plugging in the data drives in the new configuration and just ask to rebuild the parity disk? Question is, what are the right steps to look into in an attempt to save the data on the disks, if the simple changes to the same hardware do not bring any solutions? cheers, Florian

Thanks for your continuous support. all HBA are fixed and cables are fixed. At the moment, I am unable to try with a different PSU. As I mentioned initially, all I did was try to rebuild on a new disk, the entire setup has been running flawlessly for over a year so far. When I reboot in safe mode (shutdown works), and try to run xfs-check on the disks in problem, i get errors like this in the first run, a second run is not possible due to i/o errors. Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 4851860573 no . entry for directory 4851860573 no .. entry for directory 4851860573 problem with directory contents in inode 4851860573 would have cleared inode 4851860573 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 4851860604 no . entry for directory 4851860604 no .. entry for directory 4851860604 problem with directory contents in inode 4851860604 would have cleared inode 4851860604 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5124536740 no . entry for directory 5124536740 no .. entry for directory 5124536740 problem with directory contents in inode 5124536740 would have cleared inode 5124536740 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5125040217 no . entry for directory 5125040217 no .. entry for directory 5125040217 problem with directory contents in inode 5125040217 would have cleared inode 5125040217 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5125040233 no . entry for directory 5125040233 no .. entry for directory 5125040233 problem with directory contents in inode 5125040233 would have cleared inode 5125040233 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5125040249 no . entry for directory 5125040249 no .. entry for directory 5125040249 problem with directory contents in inode 5125040249 would have cleared inode 5125040249 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5125666493 no . entry for directory 5125666493 no .. entry for directory 5125666493 problem with directory contents in inode 5125666493 would have cleared inode 5125666493 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5126194197 no . entry for directory 5126194197 no .. entry for directory 5126194197 problem with directory contents in inode 5126194197 would have cleared inode 5126194197 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5126194211 no . entry for directory 5126194211 no .. entry for directory 5126194211 problem with directory contents in inode 5126194211 would have cleared inode 5126194211 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5126194233 no . entry for directory 5126194233 no .. entry for directory 5126194233 problem with directory contents in inode 5126194233 would have cleared inode 5126194233 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5126814403 no . entry for directory 5126814403 no .. entry for directory 5126814403 problem with directory contents in inode 5126814403 would have cleared inode 5126814403 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5126814418 no . entry for directory 5126814418 no .. entry for directory 5126814418 problem with directory contents in inode 5126814418 would have cleared inode 5126814418 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5126814432 no . entry for directory 5126814432 no .. entry for directory 5126814432 problem with directory contents in inode 5126814432 would have cleared inode 5126814432 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5127416517 no . entry for directory 5127416517 no .. entry for directory 5127416517 problem with directory contents in inode 5127416517 would have cleared inode 5127416517 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5127416533 no . entry for directory 5127416533 no .. entry for directory 5127416533 problem with directory contents in inode 5127416533 would have cleared inode 5127416533 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5127416546 no . entry for directory 5127416546 no .. entry for directory 5127416546 problem with directory contents in inode 5127416546 would have cleared inode 5127416546 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5127416567 no . entry for directory 5127416567 no .. entry for directory 5127416567 problem with directory contents in inode 5127416567 would have cleared inode 5127416567 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5128291725 no . entry for directory 5128291725 no .. entry for directory 5128291725 problem with directory contents in inode 5128291725 would have cleared inode 5128291725 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5128291740 no . entry for directory 5128291740 no .. entry for directory 5128291740 problem with directory contents in inode 5128291740 would have cleared inode 5128291740 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5128291752 no . entry for directory 5128291752 no .. entry for directory 5128291752 problem with directory contents in inode 5128291752 would have cleared inode 5128291752 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5128291764 xfs_repair: read failed: Input/output error can't read block 1 for directory inode 5128291764 xfs_repair: read failed: Input/output error can't read block 2 for directory inode 5128291764 no . entry for directory 5128291764 no .. entry for directory 5128291764 problem with directory contents in inode 5128291764 would have cleared inode 5128291764 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5128770108 no . entry for directory 5128770108 no .. entry for directory 5128770108 problem with directory contents in inode 5128770108 would have cleared inode 5128770108 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5131898445 xfs_repair: read failed: Input/output error can't read block 1 for directory inode 5131898445 no . entry for directory 5131898445 no .. entry for directory 5131898445 problem with directory contents in inode 5131898445 would have cleared inode 5131898445 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5140237856 no . entry for directory 5140237856 no .. entry for directory 5140237856 problem with directory contents in inode 5140237856 would have cleared inode 5140237856 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5145803332 no . entry for directory 5145803332 no .. entry for directory 5145803332 problem with directory contents in inode 5145803332 would have cleared inode 5145803332 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5145803368 no . entry for directory 5145803368 no .. entry for directory 5145803368 problem with directory contents in inode 5145803368 would have cleared inode 5145803368 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5146272820 no . entry for directory 5146272820 no .. entry for directory 5146272820 problem with directory contents in inode 5146272820 would have cleared inode 5146272820 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5146288900 no . entry for directory 5146288900 no .. entry for directory 5146288900 problem with directory contents in inode 5146288900 would have cleared inode 5146288900 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5146288916 no . entry for directory 5146288916 no .. entry for directory 5146288916 problem with directory contents in inode 5146288916 would have cleared inode 5146288916 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5146288926 no . entry for directory 5146288926 no .. entry for directory 5146288926 problem with directory contents in inode 5146288926 would have cleared inode 5146288926 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5146288946 no . entry for directory 5146288946 no .. entry for directory 5146288946 problem with directory contents in inode 5146288946 would have cleared inode 5146288946 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5147106944 no . entry for directory 5147106944 no .. entry for directory 5147106944 problem with directory contents in inode 5147106944 would have cleared inode 5147106944 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5147106954 no . entry for directory 5147106954 no .. entry for directory 5147106954 problem with directory contents in inode 5147106954 would have cleared inode 5147106954 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5147106971 no . entry for directory 5147106971 no .. entry for directory 5147106971 problem with directory contents in inode 5147106971 would have cleared inode 5147106971 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5147106984 no . entry for directory 5147106984 no .. entry for directory 5147106984 problem with directory contents in inode 5147106984 would have cleared inode 5147106984 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5148528769 no . entry for directory 5148528769 no .. entry for directory 5148528769 problem with directory contents in inode 5148528769 would have cleared inode 5148528769 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5148528788 no . entry for directory 5148528788 no .. entry for directory 5148528788 problem with directory contents in inode 5148528788 would have cleared inode 5148528788 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5148528796 no . entry for directory 5148528796 no .. entry for directory 5148528796 problem with directory contents in inode 5148528796 would have cleared inode 5148528796 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5149320007 no . entry for directory 5149320007 no .. entry for directory 5149320007 problem with directory contents in inode 5149320007 would have cleared inode 5149320007 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5149320041 no . entry for directory 5149320041 no .. entry for directory 5149320041 problem with directory contents in inode 5149320041 would have cleared inode 5149320041 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5150606547 no . entry for directory 5150606547 no .. entry for directory 5150606547 problem with directory contents in inode 5150606547 would have cleared inode 5150606547 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5150606574 no . entry for directory 5150606574 no .. entry for directory 5150606574 problem with directory contents in inode 5150606574 would have cleared inode 5150606574 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5151341396 no . entry for directory 5151341396 no .. entry for directory 5151341396 problem with directory contents in inode 5151341396 would have cleared inode 5151341396 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5151341437 no . entry for directory 5151341437 no .. entry for directory 5151341437 problem with directory contents in inode 5151341437 would have cleared inode 5151341437 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5152195535 no . entry for directory 5152195535 no .. entry for directory 5152195535 problem with directory contents in inode 5152195535 would have cleared inode 5152195535 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5152195547 no . entry for directory 5152195547 no .. entry for directory 5152195547 problem with directory contents in inode 5152195547 would have cleared inode 5152195547 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5152195561 no . entry for directory 5152195561 no .. entry for directory 5152195561 problem with directory contents in inode 5152195561 would have cleared inode 5152195561 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5152195577 no . entry for directory 5152195577 no .. entry for directory 5152195577 problem with directory contents in inode 5152195577 would have cleared inode 5152195577 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5152962569 no . entry for directory 5152962569 no .. entry for directory 5152962569 problem with directory contents in inode 5152962569 would have cleared inode 5152962569 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5152962595 no . entry for directory 5152962595 no .. entry for directory 5152962595 problem with directory contents in inode 5152962595 would have cleared inode 5152962595 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5152962609 no . entry for directory 5152962609 no .. entry for directory 5152962609 problem with directory contents in inode 5152962609 would have cleared inode 5152962609 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5154055966 no . entry for directory 5154055966 no .. entry for directory 5154055966 problem with directory contents in inode 5154055966 would have cleared inode 5154055966 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5154055985 no . entry for directory 5154055985 no .. entry for directory 5154055985 problem with directory contents in inode 5154055985 would have cleared inode 5154055985 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5156226251 no . entry for directory 5156226251 no .. entry for directory 5156226251 problem with directory contents in inode 5156226251 would have cleared inode 5156226251 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5156226263 no . entry for directory 5156226263 no .. entry for directory 5156226263 problem with directory contents in inode 5156226263 would have cleared inode 5156226263 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5156226283 no . entry for directory 5156226283 no .. entry for directory 5156226283 problem with directory contents in inode 5156226283 would have cleared inode 5156226283 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5156226298 no . entry for directory 5156226298 no .. entry for directory 5156226298 problem with directory contents in inode 5156226298 would have cleared inode 5156226298 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5156939327 no . entry for directory 5156939327 no .. entry for directory 5156939327 problem with directory contents in inode 5156939327 would have cleared inode 5156939327 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5158171721 no . entry for directory 5158171721 no .. entry for directory 5158171721 problem with directory contents in inode 5158171721 would have cleared inode 5158171721 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5158171735 no . entry for directory 5158171735 no .. entry for directory 5158171735 problem with directory contents in inode 5158171735 would have cleared inode 5158171735 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5158171753 no . entry for directory 5158171753 no .. entry for directory 5158171753 problem with directory contents in inode 5158171753 would have cleared inode 5158171753 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5158659713 no . entry for directory 5158659713 no .. entry for directory 5158659713 problem with directory contents in inode 5158659713 would have cleared inode 5158659713 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5158792846 no . entry for directory 5158792846 no .. entry for directory 5158792846 problem with directory contents in inode 5158792846 would have cleared inode 5158792846 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5158792875 no . entry for directory 5158792875 no .. entry for directory 5158792875 problem with directory contents in inode 5158792875 would have cleared inode 5158792875 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5158792889 no . entry for directory 5158792889 no .. entry for directory 5158792889 problem with directory contents in inode 5158792889 would have cleared inode 5158792889 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5159967305 no . entry for directory 5159967305 no .. entry for directory 5159967305 problem with directory contents in inode 5159967305 would have cleared inode 5159967305 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5159967327 no . entry for directory 5159967327 no .. entry for directory 5159967327 problem with directory contents in inode 5159967327 would have cleared inode 5159967327 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5159967340 no . entry for directory 5159967340 no .. entry for directory 5159967340 problem with directory contents in inode 5159967340 would have cleared inode 5159967340 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5219165389 no . entry for directory 5219165389 no .. entry for directory 5219165389 problem with directory contents in inode 5219165389 would have cleared inode 5219165389 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5219165406 no . entry for directory 5219165406 no .. entry for directory 5219165406 problem with directory contents in inode 5219165406 would have cleared inode 5219165406 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5219165427 no . entry for directory 5219165427 no .. entry for directory 5219165427 problem with directory contents in inode 5219165427 would have cleared inode 5219165427 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5256723536 no . entry for directory 5256723536 no .. entry for directory 5256723536 problem with directory contents in inode 5256723536 would have cleared inode 5256723536 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5256723563 no . entry for directory 5256723563 no .. entry for directory 5256723563 problem with directory contents in inode 5256723563 would have cleared inode 5256723563 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5256725537 no . entry for directory 5256725537 no .. entry for directory 5256725537 problem with directory contents in inode 5256725537 would have cleared inode 5256725537 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5256859987 no . entry for directory 5256859987 no .. entry for directory 5256859987 problem with directory contents in inode 5256859987 would have cleared inode 5256859987 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5256859988 no . entry for directory 5256859988 no .. entry for directory 5256859988 problem with directory contents in inode 5256859988 would have cleared inode 5256859988 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5256859994 no . entry for directory 5256859994 no .. entry for directory 5256859994 problem with directory contents in inode 5256859994 would have cleared inode 5256859994 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5256907254 no . entry for directory 5256907254 no .. entry for directory 5256907254 problem with directory contents in inode 5256907254 would have cleared inode 5256907254 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5257158026 no . entry for directory 5257158026 no .. entry for directory 5257158026 problem with directory contents in inode 5257158026 would have cleared inode 5257158026 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5257176623 no . entry for directory 5257176623 no .. entry for directory 5257176623 problem with directory contents in inode 5257176623 would have cleared inode 5257176623 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5344999122 no . entry for directory 5344999122 no .. entry for directory 5344999122 problem with directory contents in inode 5344999122 would have cleared inode 5344999122 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5344999148 no . entry for directory 5344999148 no .. entry for directory 5344999148 problem with directory contents in inode 5344999148 would have cleared inode 5344999148 xfs_repair: read failed: Input/output error can't read block 0 for directory inode 5344999158 no . entry for directory 5344999158 no .. entry for directory 5344999158 problem with directory contents in inode 5344999158 would have cleared inode 5344999158 - agno = 3 xfs_repair: read failed: Input/output error cannot read inode 6442451072, disk block 6442451048, cnt 32 does that give any pointers as to what I can attempt in order to fix the broken disks?

Thanks, question is how to proceed now. I am back in the state where the array is in maintenance mode, i am unable to download any new diagnostics or stop the array again. All that is now showing is the read errors on disks 1,2,5 Aug 27 07:29:02 bigpool kernel: Buffer I/O error on dev md2, logical block 3906469856, async page read Aug 27 07:29:02 bigpool kernel: Buffer I/O error on dev md5, logical block 3906469856, async page read Aug 27 07:29:02 bigpool kernel: Buffer I/O error on dev md1, logical block 3906469856, async page read I have taken out the old disk1 from the array to not drain any additional power, that is the same setup that has been running for months now. I'm uncertain how to even try to diagnose or try to correct the problem at this stage... any pointers?

sorry, I was on business trip the last days and couldn't continue checking. I was able to shutdown the server properly last time and now retrieved new diagnostics today as well as one from last Sunday before the unclean shutdown. Again any help appreciated. Thanks for your help and patience. bigpool-diagnostics-20220826-0659.zip bigpool-diagnostics-20220821-0718.zip

tried from the command-line, but disks were no longer found... Could not get diagnostics from webUI to deliver me data, couldn't tell where the process got stuck.

Thanks, I am aware of the problems it has caused and the most super critical data is safely stored in multiple other locations, so if it's all gone in the end it will be painful but not life threating However the issue I am having is that at the moment I am completely stuck as to which command to run to at least unmount the arrays again to even try to recover any data or repair any superblock, if that is even the issue... For the last 2 hours it just kept writing the I/O-error to the log every minute, doesn't shut down or pretty much does anything else.... anyone else with some ideas as to how to continue?

Hi guys, I am trying to explain what happened and am desperately seeking your advice. This is the setup in question: sdg is my parity sdf disk 1 sdi disk 2 sdb disk 4 sdj disk 5 sdh is my old disk 1 After my disk1 repeatedly got disabled, I purchasd a new replacement disk. After pre-clearing I started the rebuild of the missing disk. After a day I went back to check that it had only rebuild 0.7% and the system seemed stuck. The log show some read errors on disk 5 (which so far did not cause any problems) but the system was not responding. Halting, rebooting, nothing worked, also not downloading diagnostics. So after trying to figure out the stuck process for some time I gave up and maybe did an unclever thing by forcing a power down. The system then came back up with, after I started the array it stated 4 disks unmountable. I switched back to maintenance mode, tried to get xfs-repair to show me what may be an issue, but it stopped with superblock errors on all disks in question. The Log file is now showing these messages... 1 - 2 - 5 all in succession, ever minute. Aug 21 18:19:01 bigpool kernel: Buffer I/O error on dev md1, logical block 3906469856, async page read Aug 21 18:20:01 bigpool kernel: Buffer I/O error on dev md2, logical block 3906469856, async page read Aug 21 18:20:01 bigpool kernel: Buffer I/O error on dev md5, logical block 3906469856, async page read Aug 21 18:20:01 bigpool kernel: Buffer I/O error on dev md1, logical block 3906469856, async page read Now I am stuck again as in what I can even try to do. Trying to run xfs-repair give me this, on all disks 1,2,5 Phase 1 - find and verify superblock... superblock read failed, offset 0, size 524288, ag 0, rval -1 fatal error -- Input/output error Now I am "somewhat stuck in a maintenance mode" but it doesn't respond, shut down, stop the array or anything. Since I am unable to download any diagnostics, what can I do manually so some of you has an idea how to recover the disks and data on it? cheers, Florian

I just want to add my experience as I have been struggling for a week to get my unraid system up and running. My setup is a mini tower with an 8 bay backplane, everything I purchased was new. My 16TB seagate ironwolf was acting as parity, other disks as data storage. Everything seemed to be working fine, i could not see any particular entries in the log files. However, my system was behaving erratic. As soon as I tried to build parity, the throughput decreased to 30/40 kbps thus lasting 2-3 years to complete. Other trouble also occured like drives not showing up in the bios, not being able to connect anymore, formatting issues, dockers unable to download etc. I was scratching my head and started to take out components of the system one by one. Since the disk was new and causing issues, I was focussing too much on this as well as SATA adressing topics. It took me a week to identify the root cause, which also left me puzzled. After I disconnected my backplane and hooked up the disks manually with sata cables, everything seemed to be working fine. The backplane is connected with 4SATA to SFF-8087 cables, power supply comes from a SATA cable going to a MOLEX-splitter for 2 backplane power MOLEX adapters. Changing the SFF-8087 cable also didn't solve the issue, so I reached for molex power supply and noticed that the two red cables from the Y-splitter had come off. After fixing this, the system went up and running as it should. So the 5V power supply of the backplane was off, but since the disks spun I never expected this to be the trouble. I'm now a happy unraid user and wanted to share this issue, in case somebody else experiences issues like that.

All I can say is, I'm having the exact same problem. Plex, Radarr etc. working fine, sonarr won't be able to connect even though it's started. Setting is exactly matching the radarr setting.... Any ideas?