ok

tower-diagnostics-20220806-1644.zip

I still think there are connection problems with disk9, do you have another SATA cable?

Aug  6 16:24:12 Tower kernel: ata9.00: ATA-10: OOS12000G,             00007ELF, OOS1, max UDMA/133
...
Aug  6 16:45:03 Tower kernel: ata9: SATA link up 3.0 Gbps (SStatus 123 SControl 320)
Aug  6 16:45:03 Tower kernel: ata9.00: configured for UDMA/133
Aug  6 16:45:09 Tower kernel: ata9.00: exception Emask 0x10 SAct 0x4000 SErr 0x190002 action 0xe frozen
Aug  6 16:45:09 Tower kernel: ata9.00: irq_stat 0x80400000, PHY RDY changed
Aug  6 16:45:09 Tower kernel: ata9: SError: { RecovComm PHYRdyChg 10B8B Dispar }
Aug  6 16:45:09 Tower kernel: ata9.00: failed command: READ FPDMA QUEUED
Aug  6 16:45:09 Tower kernel: ata9.00: cmd 60/20:70:18:01:b1/00:00:b1:00:00/40 tag 14 ncq dma 16384 in
Aug  6 16:45:09 Tower kernel:         res 40/00:70:18:01:b1/00:00:b1:00:00/40 Emask 0x10 (ATA bus error)
Aug  6 16:45:09 Tower kernel: ata9.00: status: { DRDY }
Aug  6 16:45:09 Tower kernel: ata9: hard resetting link

 

yes.

Shutdown, replace cable on disk9, restart and post new diagnostics. Be careful of all other connections.

I am going to have to go out for a few hours. Not sure if I will get back to this before tomorrow.

 

Thread has grown a bit, so let me summarize for anybody else that might want to step in.

 

All disks mountable including emulated disk1, but some syslog entries from earlier diagnostics showed filesystem problems with disks 1, 5, 9.

 

And as I often do, I'll tag the expert on all things storage, @JorgeB, may be very late in that timezone but maybe earlier than I will be tomorrow.

 

 

OK, cable changed

Thank you for all your help. I will monitor this for additional support.

tower-diagnostics-20220806-1712.zip

I will make a small note here... While 100ish days to compete a rebuild is on the extreme... I wonder what the numbers will be now that you have shutdown dockers... read/writes have a huge impact on parity, but too this extreme slow speed, seems very unusual... 

 

what I would suggest is KISS... Keep It Simple Stupid... means that your rig seems a little complex given the notes in this thread... 

 

Disconnect/disable unused/required hardware  for unraid to function... and try a parity check while monitoring system usage/load... there is a bottleneck that is preventing disk1 rebuild and limiting the variables is a good thing to do... 

 

But do please note... I am not an expert on unraid(been a user for 10+ years) but I would suggest limiting activity on the system and ensuring that IO is good would be a good first step... thus the parity check would be a good benchmark for the rebuild

 

Thank you! I will see what it shows with the dockers shut down.

12 hours ago, StewLoft said:

OK, cable changed

Still ATA errors with disk9, try swapping cables with a different disk, one that is connected to the onboard SATA.

 

re: fs corruption, those should all be fixed, recommend running xfs_repair (without -n) on all array disks.

OK, I swapped the cable for disc9 to come off the MB. Disc9 is showing unmountable upon restart.

tower-diagnostics-20220807-1058.zip

I stopped the Array, went Maintenance Mode, ran filesystem check (-n) on disc9. I restarted Array and disc9 is now mounted. This button appeared "Parity-Sync/Data-Rebuild" as an option for the first time.

 

 

 

XFS filesystem check Aug 8 1115am.docx

tower-diagnostics-20220807-1122.zip

Just a note about using the forum. Please attach plain text in future.

 

If it is very large, such as syslog, zip it. Most attachments except screenshots should be plain text or zipped plain text. Diagnostics is zipped plain text except for one file (config/super.dat) in the zip.

 

There is nothing in most of these that will benefit from word processing software. Often word processing software will make things less usable.

 

We should not have to install additional software to help you.

 

3 hours ago, StewLoft said:

I stopped the Array, went Maintenance Mode, ran filesystem check (-n) on disc9. I restarted Array and disc9 is now mounted. This button appeared "Parity-Sync/Data-Rebuild" as an option for the first time.

Not sure what you mean here. (-n) nomodify will not have actually changed anything about disk9.

 

Post a screenshot of Array Operation.

 

Haven't looked at any of these diagnostics from today yet.

 

18 hours ago, StewLoft said:

I will see what it shows with the dockers shut down.

 

Unless a lot of reading or writing is happening on the array, dockers shouldn't have much impact on parity build. In your case, all of your assigned disks are array disks, which is not ideal for docker/VM setup, we can deal with that later.

 

In any case, can't remember if I mentioned it earlier in thread, but you should disable Docker in Settings until everything is resolved.

28 minutes ago, trurl said:

Unless a lot of reading or writing is happening on the array, dockers shouldn't have much impact on parity build

If a lot of reading or writing is happening on the array that will impact parity build whether dockers are involved or not.

28 minutes ago, trurl said:

In your case, all of your assigned disks are array disks

This means some open files on your array since the Docker service has docker.img open, and any running containers will have their appdata open.

 

28 minutes ago, trurl said:

can't remember if I mentioned it earlier in thread

On 8/6/2022 at 2:59 PM, trurl said:

Disable Docker in Settings and leave it disabled until we get things fixed.

  

And better if there is no writing to the array until we get things fixed.

 

In case it gets missed in those long posts

29 minutes ago, trurl said:

Post a screenshot of Array Operation.

 

4 hours ago, StewLoft said:

Disc9 is showing unmountable upon restart.

Not only that, but emulated disk1 was also unmountable. Up until this point

22 hours ago, trurl said:

All disks mountable including emulated disk1

 

Then

4 hours ago, StewLoft said:

I restarted Array and disc9 is now mounted

even though you didn't really do anything. No word about whether emulated disk1 mounts.

 

The diagnostics you posted after that was without the array started, so not immediately obvious if anything mounts or not.

 

But digging into syslog, despite what you said

4 hours ago, StewLoft said:

disc9 is now mounted

emulated disk1 never mounted and also disk9 never mounted.

 

So, we need new diagnostics with the array started, and screenshot of Array Devices.

 

More in next post about continuing connection problems

 

Connection problems on disk9 and also on the unassigned (former parity, to be disk1) disk (sdh, U9NR)

 

Both of these disks are on the motherboard

Dockers are still off from when you instructed yesterday. 

tower-diagnostics-20220807-1637.zip

We usually recommend repairing unmountable filesystem before rebuilding. I don't remember telling you to assign anything as disk1 yet. You are currently rebuilding an unmountable filesystem to the disk1 replacement. I guess it doesn't matter since that replacement disk didn't have anything on it anyway. Would have been more of a problem if attempting to rebuild onto the original disk, because original disk might be useful if repair doesn't go well.

 

Disk9 and emulated disk1 both unmountable.

 

Still having connection problems, replacement disk1, disk9, sdg, which according to previous diagnostics was original disk1. I don't see SMART report for sdg now.

 

Might as well stop rebuild, it can't be going well.

 

Fix hardware problems before attempting anything else.

 

Since we are having filesystem problems, including on some disks that still mount, I wonder if you don't have RAM issues.

 

First hardware problem you should address is confirming no RAM issues by running memtest. You shouldn't even attempt to run a computer if RAM isn't perfect. Everything goes through RAM, the OS and other executable code, your data, everything. CPU can't do anything with anything until it is loaded into RAM.

5 minutes ago, trurl said:

Still having connection problems, replacement disk1, disk9, sdg, which according to previous diagnostics was original disk1. I don't see SMART report for sdg now.

replacement disk1 on 1st Marvell, disk9 and sdg on motherboard