running this now, its just doing the same searching for secondary block.  i'll let it run awhile but it seems to be failing.

If it's not finding the superblock it won't work, I still find strange that disk13 mounts and xfs_repair can't find the superblock, it doesn't make sense, either way you best option is doing a new config without both those disks, do a parity sync, then try to recover any data from old disk4, try xfs_repair again on old disk13 when it's out of the array but it shouldn't change anything.

on disk13 like half the directories say structure needs cleaning, the others seem to work.  some of the files that do work seem corrupted though.  I still have both of the original failed disk4 and 13, plus the rebuilt 13 that has the errors.  I can still get the failed disk4 to be recognized but not sure if the orig 13 will show up at all.  

 

what's the proper way to do the new config?  I just don't want to end up formatting the rest of the array on accident.

It's rebuilding parity now and back up w/o 13 & 4.  what would be the proper way to try to recover anything off those?  can I just put them in unused slots and try to copy off them back to the array?

7 minutes ago, jhanda said:

can I just put them in unused slots and try to copy off them back to the array?

 

No, that would clear them deleting all data, you can use the Unassigned Devices plugin or temporarily unassign your cache drive and assign the old disks one at a time to the cache slot.

thanks man.  you've been alot of help in dealing with this.  really appreciate it.

Was able to recover everything off disk4, the corrupt disk 13 seems to be almost total loss.  disk 4 seems to work ok with the new controllers.  maybe the whole thing was caused by the controller not the drive.

Better than nothing, very possible the corruption on disk13 was controller related.

Is there any reason these controllers would make the array alot slower than the old ones?

Is there any reason these controllers would make the array alot slower than the old ones?

Refresh my memory since thread is too large to reread, what controllers did you have and have now?

PERC H310's

the original ones were Supermicro AOC-SASLP-MV8

These should be faster during check/rebuilds and writes with turbo write enable, same for normal array utilization including writes without turbo write, when you say the array is a lot slower, it's doing what?

Everything, but its most noticable doing multiple things at once.  It's a media server for the whole house.  Previously, I could stream movies to 2 tv's at the same time without issue and still be able to do things on some vm's that were residing there without affecting them.  Now it's not possible.  If I have 1 movie playing and do anything else, copy pictures to the server etc, it'll make the movie skip.  Copying files to/from the server is much slower.  Even copying files from one share to another share from the command line is slower.  eg. cp /mnt/user/share1/file /mnt/user/share2/.  If i copy files from one share to another, I can't do anything else at all.  I used to be able to do that and still watch a movie.

 

I've had some strange errors pop up too lately, I think its unrelated to the performance because this just started recently and seems to affect one of the VM's.  it always crashes the VM when it happens, but the system seems fine.

 

bunch of this on the console:

Message from syslogd@Hoser at Dec 13 04:03:59 ...

 kernel:flags: 0x200000000000014(referenced|dirty)

 

Message from syslogd@Hoser at Dec 13 04:03:59 ...

 kernel:page:ffffea00044f0000 count:0 mapcount:0 mapping:          (null) index:0x1

 

and this in the logs:

Dec 13 04:03:58 Hoser kernel: WARNING: CPU: 0 PID: 6268 at arch/x86/kvm/mmu.c:614 mmu_spte_clear_track_bits+0xd0/0xd8 [kvm]
Dec 13 04:03:58 Hoser kernel: Modules linked in: xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables vhost_net tun vhost macvtap macvlan ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat md_mod bonding e1000e igb ptp pps_core i2c_algo_bit x86_pkg_temp_thermal coretemp kvm_intel i2c_i801 kvm i2c_smbus ahci libahci mpt3sas raid_class scsi_transport_sas i2c_core video backlight [last unloaded: pps_core]
Dec 13 04:03:58 Hoser kernel: CPU: 0 PID: 6268 Comm: worker Tainted: G        W       4.9.30-unRAID #1
Dec 13 04:03:58 Hoser kernel: Hardware name: Supermicro X10SAE/X10SAE, BIOS 1.00 05/03/2013
Dec 13 04:03:58 Hoser kernel: ffffc9000d0339b0 ffffffff813a4a1b 0000000000000000 ffffffffa01b2e28
Dec 13 04:03:58 Hoser kernel: ffffc9000d0339f0 ffffffff8104d0d9 000002660000002b 0000000013c00ff7
Dec 13 04:03:58 Hoser kernel: 0000000000013c00 ffff8803a9f8c000 0000000000000000 00000000000008b8
Dec 13 04:03:58 Hoser kernel: Call Trace:
Dec 13 04:03:58 Hoser kernel: [<ffffffff813a4a1b>] dump_stack+0x61/0x7e
Dec 13 04:03:58 Hoser kernel: [<ffffffff8104d0d9>] __warn+0xb8/0xd3
Dec 13 04:03:58 Hoser kernel: [<ffffffff8104d1a1>] warn_slowpath_null+0x18/0x1a
Dec 13 04:03:58 Hoser kernel: [<ffffffffa018ed17>] mmu_spte_clear_track_bits+0xd0/0xd8 [kvm]
Dec 13 04:03:58 Hoser kernel: [<ffffffffa018ed5f>] drop_spte+0x15/0x7b [kvm]
Dec 13 04:03:58 Hoser kernel: [<ffffffffa0191f5e>] mmu_page_zap_pte+0x55/0x9b [kvm]
Dec 13 04:03:58 Hoser kernel: [<ffffffffa0192c97>] kvm_mmu_prepare_zap_page+0x42/0x1f5 [kvm]
Dec 13 04:03:58 Hoser kernel: [<ffffffffa0195f59>] kvm_mmu_invalidate_zap_all_pages+0x6d/0xb2 [kvm]
Dec 13 04:03:58 Hoser kernel: [<ffffffffa018c402>] kvm_arch_flush_shadow_all+0x9/0xb [kvm]
Dec 13 04:03:58 Hoser kernel: [<ffffffffa017d2ad>] kvm_mmu_notifier_release+0x37/0x50 [kvm]
Dec 13 04:03:58 Hoser kernel: [<ffffffff81104f77>] __mmu_notifier_release+0x66/0xd5
Dec 13 04:03:58 Hoser kernel: [<ffffffff810f210b>] exit_mmap+0x20/0x10c
Dec 13 04:03:58 Hoser kernel: [<ffffffff8110c273>] ? kfree+0xdf/0xfa
Dec 13 04:03:58 Hoser kernel: [<ffffffff8110c273>] ? kfree+0xdf/0xfa
Dec 13 04:03:58 Hoser kernel: [<ffffffff810fa125>] ? __vunmap+0x93/0x99
Dec 13 04:03:58 Hoser kernel: [<ffffffff810fa1b3>] ? vfree+0x60/0x63
Dec 13 04:03:58 Hoser kernel: [<ffffffff811173e3>] ? __khugepaged_exit+0x9c/0x119
Dec 13 04:03:58 Hoser kernel: [<ffffffff8104aa4c>] mmput+0x48/0xec
Dec 13 04:03:58 Hoser kernel: [<ffffffffa0039a0b>] vhost_dev_cleanup+0x351/0x361 [vhost]
Dec 13 04:03:58 Hoser kernel: [<ffffffffa005a8ce>] vhost_net_release+0x3a/0xf5 [vhost_net]
Dec 13 04:03:58 Hoser kernel: [<ffffffff81122d49>] __fput+0xed/0x19f
Dec 13 04:03:58 Hoser kernel: [<ffffffff81122e27>] ____fput+0x9/0xb
Dec 13 04:03:58 Hoser kernel: [<ffffffff81062609>] task_work_run+0x6a/0x7d
Dec 13 04:03:58 Hoser kernel: [<ffffffff810501f2>] do_exit+0x3ae/0x86a
Dec 13 04:03:58 Hoser kernel: [<ffffffff81050716>] do_group_exit+0x3c/0x98
Dec 13 04:03:58 Hoser kernel: [<ffffffff810580c6>] get_signal+0x47e/0x4b0
Dec 13 04:03:58 Hoser kernel: [<ffffffff8101e3e6>] do_signal+0x23/0x4a5
Dec 13 04:03:58 Hoser kernel: [<ffffffff81002b09>] exit_to_usermode_loop+0x3a/0x81
Dec 13 04:03:58 Hoser kernel: [<ffffffff81002c61>] syscall_return_slowpath+0x44/0x47
Dec 13 04:03:58 Hoser kernel: [<ffffffff8167f5c4>] entry_SYSCALL_64_fastpath+0xa7/0xa9
Dec 13 04:03:58 Hoser kernel: ---[ end trace 9b507da535079fbf ]---

6 minutes ago, jhanda said:

Previously, I could stream movies to 2 tv's at the same time without issue and still be able to do things on some vm's that were residing there without affecting them.  Now it's not possible.

Very unlikely that is related to the controllers, if anything they would be faster in that situation, but most likely unchanged.

Well thats good if the H310's should be better, I can keep looking for something else I have setup incorrectly.  I added a 2nd parity drive, would that slow it down?  Thats the only other thing I added.  Maybe need to open it up and see what controller each parity drive is plugged into?

Are the VMs on the array? For best performance they should be on an SSD, cache or unassigned.

They are on the cache drive.  I still have the same performance issues with the VM's powered off though.  Movie playing on the TV, copy photos to the array and it starts skipping.

Yeah, but like I said very unlike those issues are controller related, but if you have the old ones easy to test.