Jump to content

[6.10.0-rc3] Drives are having problems and showing errors. Adaptec Raid Controller Problem?

aronmal

By aronmal
in General Support

Recommended Posts

aronmal Rookie

aronmal

Posted
Posted (edited)

I was doing some configuration on my firewall which is running in a VM on my Unraid server. I noticed how some threads where 100% load and overall load was over 60% which was not normal for my 16 core Xeon. I waited, the CPU load was going down at some point but increased again. I restarted the server. After the server back I noticed the array troubeling to start up. partiy 2 and disk 4 where disabled (from 6 HDDs and I have 2 cache SSDs). I don't think the Firewall VM had anything to do with it. I will post logs and stuff I collected during this situation below. I now get errors when starting the array or rebuilding it. I don't thing this is due to bad drives or corruped data from my point of view and more being errors with my Adaptec Raid Controller somehow as first guess. I need my server back and up running, save as soon as possible. So if anyone can have a look at the logs and diagnostics, many thanks.

Edited by aronmal
Link to comment
aronmal Rookie

aronmal

Posted
  • Author
Posted (edited) 
Apr 25 18:32:57 Tower kernel: ? md_thread+0x103/0x12a [md_mod]
Apr 25 18:32:57 Tower kernel: ? rmw5_write_data+0x17d/0x17d [md_mod]
Apr 25 18:32:57 Tower kernel: md_thread+0x103/0x12a [md_mod]
Apr 25 18:32:57 Tower kernel: ? init_wait_entry+0x29/0x29
Apr 25 18:32:57 Tower kernel: ? md_seq_show+0x6c8/0x6c8 [md_mod]
Apr 25 18:32:57 Tower kernel: kthread+0xde/0xe3
Apr 25 18:32:57 Tower kernel: ? set_kthread_struct+0x32/0x32
Apr 25 18:32:57 Tower kernel: ret_from_fork+0x22/0x30
Apr 25 18:32:57 Tower kernel: </TASK>
Apr 25 18:32:57 Tower kernel: ---[ end trace 7c70e023e1918754 ]---
Apr 25 18:32:57 Tower kernel: DMAR: ERROR: DMA PTE for vPFN 0xac679 already set (to 160047801 not 1a7026803)
Apr 25 18:32:57 Tower kernel: ------------[ cut here ]------------
Apr 25 18:32:57 Tower kernel: WARNING: CPU: 9 PID: 12007 at drivers/iommu/intel/iommu.c:2387 __domain_mapping+0x2e5/0x390
Apr 25 18:32:57 Tower kernel: Modules linked in: xfs md_mod nvidia(PO) m88rs6000t m88ds3103 i2c_mux regmap_i2c smipcie dvb_core mc ip6table_filter ip6_tables iptable_filter ip_tables x_tables ipmi_ssif ast drm_vram_helper drm_ttm_helper ttm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm drm_kms_helper crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd drm rapl intel_cstate intel_uncore igb aacraid backlight i2c_i801 agpgart acpi_ipmi i2c_algo_bit syscopyarea i2c_smbus sysfillrect ahci sysimgblt i2c_core input_leds fb_sys_fops led_class libahci intel_pch_thermal wmi ipmi_si acpi_power_meter acpi_pad button
Apr 25 18:32:57 Tower kernel: CPU: 9 PID: 12007 Comm: unraidd0 Tainted: P        W  O      5.15.27-Unraid #1
Apr 25 18:32:57 Tower kernel: Hardware name: bluechip bluechip SERVERline/X11SPL-F, BIOS 3.5 05/19/2021
Apr 25 18:32:57 Tower kernel: RIP: 0010:__domain_mapping+0x2e5/0x390
Apr 25 18:32:57 Tower kernel: Code: 2b 48 8b 4c 24 08 48 89 c2 4c 89 e6 48 c7 c7 d4 c7 ef 81 e8 b7 95 2c 00 8b 05 de 3c c3 00 85 c0 74 08 ff c8 89 05 d2 3c c3 00 <0f> 0b 8b 74 24 38 b8 34 00 00 00 8d 0c f6 83 e9 09 39 c1 0f 4f c8
Apr 25 18:32:57 Tower kernel: RSP: 0018:ffffc900038b7810 EFLAGS: 00010246
Apr 25 18:32:57 Tower kernel: RAX: 0000000000000000 RBX: ffff88819f1d33c8 RCX: 0000000000000000
Apr 25 18:32:57 Tower kernel: RDX: 0000000000000000 RSI: ffff88903fc5c550 RDI: ffff88903fc5c550
Apr 25 18:32:57 Tower kernel: RBP: ffff88810e7b8000 R08: ffff88907ffa70a8 R09: 0000000000000001
Apr 25 18:32:57 Tower kernel: R10: 00000000ffffffff R11: ffff888154cf9ab0 R12: 00000000000ac679
Apr 25 18:32:57 Tower kernel: R13: ffff88819f1d33c8 R14: 0000000000001000 R15: 00000000ac679000
Apr 25 18:32:57 Tower kernel: FS:  0000000000000000(0000) GS:ffff88903fc40000(0000) knlGS:0000000000000000
Apr 25 18:32:57 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 25 18:32:57 Tower kernel: CR2: 000014c46c267ff8 CR3: 000000000200a003 CR4: 00000000007706e0
Apr 25 18:32:57 Tower kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 25 18:32:57 Tower kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Apr 25 18:32:57 Tower kernel: PKRU: 55555554
Apr 25 18:32:57 Tower kernel: Call Trace:
Apr 25 18:32:57 Tower kernel: <TASK>
Apr 25 18:32:57 Tower kernel: intel_iommu_map_pages+0xf3/0x102
Apr 25 18:32:57 Tower kernel: __iommu_map+0x138/0x211
Apr 25 18:32:57 Tower kernel: __iommu_map_sg+0x8c/0x110
Apr 25 18:32:57 Tower kernel: iommu_dma_map_sg+0x245/0x3e3
Apr 25 18:32:57 Tower kernel: __dma_map_sg_attrs+0x63/0x95
Apr 25 18:32:57 Tower kernel: dma_map_sg_attrs+0xa/0x12
Apr 25 18:32:57 Tower kernel: scsi_dma_map+0x34/0x3e
Apr 25 18:32:57 Tower kernel: aac_build_sg+0x26/0xb8 [aacraid]
Apr 25 18:32:57 Tower kernel: aac_scsi_32+0x23/0xa0 [aacraid]
Apr 25 18:32:57 Tower kernel: aac_scsi_cmd+0x467/0xf94 [aacraid]
Apr 25 18:32:57 Tower kernel: ? sd_init_command+0x572/0x934
Apr 25 18:32:57 Tower kernel: ? wbt_issue+0x16/0x3c
Apr 25 18:32:57 Tower kernel: aac_queuecommand+0x17/0x21 [aacraid]
Apr 25 18:32:57 Tower kernel: scsi_queue_rq+0x57d/0x6db
Apr 25 18:32:57 Tower kernel: blk_mq_dispatch_rq_list+0x2a7/0x4da
Apr 25 18:32:57 Tower kernel: __blk_mq_do_dispatch_sched+0x23d/0x281
Apr 25 18:32:57 Tower kernel: __blk_mq_sched_dispatch_requests+0xd5/0x129
Apr 25 18:32:57 Tower kernel: blk_mq_sched_dispatch_requests+0x2f/0x52
Apr 25 18:32:57 Tower kernel: __blk_mq_run_hw_queue+0x50/0x76
Apr 25 18:32:57 Tower kernel: __blk_mq_delay_run_hw_queue+0x4d/0x108
Apr 25 18:32:57 Tower kernel: blk_mq_sched_insert_requests+0xa2/0xd9
Apr 25 18:32:57 Tower kernel: blk_mq_flush_plug_list+0xfb/0x12c
Apr 25 18:32:57 Tower kernel: blk_finish_plug+0x1f/0x2c
Apr 25 18:32:57 Tower kernel: unraidd+0x11ed/0x1237 [md_mod]
Apr 25 18:32:57 Tower kernel: ? md_thread+0x103/0x12a [md_mod]
Apr 25 18:32:57 Tower kernel: ? rmw5_write_data+0x17d/0x17d [md_mod]
Apr 25 18:32:57 Tower kernel: md_thread+0x103/0x12a [md_mod]
Apr 25 18:32:57 Tower kernel: ? init_wait_entry+0x29/0x29
Apr 25 18:32:57 Tower kernel: ? md_seq_show+0x6c8/0x6c8 [md_mod]
Apr 25 18:32:57 Tower kernel: kthread+0xde/0xe3
Apr 25 18:32:57 Tower kernel: ? set_kthread_struct+0x32/0x32
Apr 25 18:32:57 Tower kernel: ret_from_fork+0x22/0x30
Apr 25 18:32:57 Tower kernel: </TASK>
Apr 25 18:32:57 Tower kernel: ---[ end trace 7c70e023e1918755 ]---
Apr 25 18:32:57 Tower kernel: DMAR: ERROR: DMA PTE for vPFN 0xac67a already set (to 10195c801 not 13698d803)
Apr 25 18:32:57 Tower kernel: ------------[ cut here ]------------
Apr 25 18:32:57 Tower kernel: WARNING: CPU: 9 PID: 12007 at drivers/iommu/intel/iommu.c:2387 __domain_mapping+0x2e5/0x390
Apr 25 18:32:57 Tower kernel: Modules linked in: xfs md_mod nvidia(PO) m88rs6000t m88ds3103 i2c_mux regmap_i2c smipcie dvb_core mc ip6table_filter ip6_tables iptable_filter ip_tables x_tables ipmi_ssif ast drm_vram_helper drm_ttm_helper ttm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm drm_kms_helper crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd drm rapl intel_cstate intel_uncore igb aacraid backlight i2c_i801 agpgart acpi_ipmi i2c_algo_bit syscopyarea i2c_smbus sysfillrect ahci sysimgblt i2c_core input_leds fb_sys_fops led_class libahci intel_pch_thermal wmi ipmi_si acpi_power_meter acpi_pad button
Apr 25 18:32:57 Tower kernel: CPU: 9 PID: 12007 Comm: unraidd0 Tainted: P        W  O      5.15.27-Unraid #1
Apr 25 18:32:57 Tower kernel: Hardware name: bluechip bluechip SERVERline/X11SPL-F, BIOS 3.5 05/19/2021
Apr 25 18:32:57 Tower kernel: RIP: 0010:__domain_mapping+0x2e5/0x390
Apr 25 18:32:57 Tower kernel: Code: 2b 48 8b 4c 24 08 48 89 c2 4c 89 e6 48 c7 c7 d4 c7 ef 81 e8 b7 95 2c 00 8b 05 de 3c c3 00 85 c0 74 08 ff c8 89 05 d2 3c c3 00 <0f> 0b 8b 74 24 38 b8 34 00 00 00 8d 0c f6 83 e9 09 39 c1 0f 4f c8
Apr 25 18:32:57 Tower kernel: RSP: 0018:ffffc900038b7810 EFLAGS: 00010246
Apr 25 18:32:57 Tower kernel: RAX: 0000000000000000 RBX: ffff88819f1d33d0 RCX: 0000000000000003
Apr 25 18:32:57 Tower kernel: RDX: 0000000000000000 RSI: ffffc900038b76a0 RDI: 0000000000000001
Apr 25 18:32:57 Tower kernel: RBP: ffff88810e7b8000 R08: ffff88907ffa70a8 R09: 0000000000000001
Apr 25 18:32:57 Tower kernel: R10: 00000000ffffffff R11: ffff888154cf94f0 R12: 00000000000ac67a
Apr 25 18:32:57 Tower kernel: R13: ffff88819f1d33d0 R14: 0000000000001000 R15: 00000000ac67a000
Apr 25 18:32:57 Tower kernel: FS:  0000000000000000(0000) GS:ffff88903fc40000(0000) knlGS:0000000000000000
Apr 25 18:32:57 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 25 18:32:57 Tower kernel: CR2: 000014c46c267ff8 CR3: 000000000200a003 CR4: 00000000007706e0
Apr 25 18:32:57 Tower kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 25 18:32:57 Tower kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Apr 25 18:32:57 Tower kernel: PKRU: 55555554
Apr 25 18:32:57 Tower kernel: Call Trace:
Apr 25 18:32:57 Tower kernel: <TASK>
Apr 25 18:32:57 Tower kernel: intel_iommu_map_pages+0xf3/0x102
Apr 25 18:32:57 Tower kernel: __iommu_map+0x138/0x211
Apr 25 18:32:57 Tower kernel: __iommu_map_sg+0x8c/0x110
Apr 25 18:32:57 Tower kernel: iommu_dma_map_sg+0x245/0x3e3
Apr 25 18:32:57 Tower kernel: __dma_map_sg_attrs+0x63/0x95
Apr 25 18:32:57 Tower kernel: dma_map_sg_attrs+0xa/0x12
Apr 25 18:32:57 Tower kernel: scsi_dma_map+0x34/0x3e
Apr 25 18:32:57 Tower kernel: aac_build_sg+0x26/0xb8 [aacraid]
Apr 25 18:32:57 Tower kernel: aac_scsi_32+0x23/0xa0 [aacraid]
Apr 25 18:32:57 Tower kernel: aac_scsi_cmd+0x467/0xf94 [aacraid]
Apr 25 18:32:57 Tower kernel: ? sd_init_command+0x572/0x934
Apr 25 18:32:57 Tower kernel: ? wbt_issue+0x16/0x3c
Apr 25 18:32:57 Tower kernel: aac_queuecommand+0x17/0x21 [aacraid]
Apr 25 18:32:57 Tower kernel: scsi_queue_rq+0x57d/0x6db
Apr 25 18:32:57 Tower kernel: blk_mq_dispatch_rq_list+0x2a7/0x4da
Apr 25 18:32:57 Tower kernel: __blk_mq_do_dispatch_sched+0x23d/0x281
Apr 25 18:32:57 Tower kernel: __blk_mq_sched_dispatch_requests+0xd5/0x129
Apr 25 18:32:57 Tower kernel: blk_mq_sched_dispatch_requests+0x2f/0x52
Apr 25 18:32:57 Tower kernel: __blk_mq_run_hw_queue+0x50/0x76
Apr 25 18:32:57 Tower kernel: __blk_mq_delay_run_hw_queue+0x4d/0x108
Apr 25 18:32:57 Tower kernel: blk_mq_sched_insert_requests+0xa2/0xd9
Apr 25 18:32:57 Tower kernel: blk_mq_flush_plug_list+0xfb/0x12c
Apr 25 18:32:57 Tower kernel: blk_finish_plug+0x1f/0x2c
Apr 25 18:32:57 Tower kernel: unraidd+0x11ed/0x1237 [md_mod]
Apr 25 18:32:57 Tower kernel: ? md_thread+0x103/0x12a [md_mod]
Apr 25 18:32:57 Tower kernel: ? rmw5_write_data+0x17d/0x17d [md_mod]
Apr 25 18:32:57 Tower kernel: md_thread+0x103/0x12a [md_mod]
Apr 25 18:32:57 Tower kernel: ? init_wait_entry+0x29/0x29
Apr 25 18:32:57 Tower kernel: ? md_seq_show+0x6c8/0x6c8 [md_mod]
Apr 25 18:32:57 Tower kernel: kthread+0xde/0xe3
Apr 25 18:32:57 Tower kernel: ? set_kthread_struct+0x32/0x32
Apr 25 18:32:57 Tower kernel: ret_from_fork+0x22/0x30
Apr 25 18:32:57 Tower kernel: </TASK>
Apr 25 18:32:57 Tower kernel: ---[ end trace 7c70e023e1918756 ]---
Apr 25 18:32:57 Tower kernel: DMAR: ERROR: DMA PTE for vPFN 0xac67b already set (to 19f096801 not 156fae803)
Apr 25 18:32:57 Tower kernel: ------------[ cut here ]------------
Apr 25 18:32:57 Tower kernel: WARNING: CPU: 9 PID: 12007 at drivers/iommu/intel/iommu.c:2387 __domain_mapping+0x2e5/0x390
Apr 25 18:32:57 Tower kernel: Modules linked in: xfs md_mod nvidia(PO) m88rs6000t m88ds3103 i2c_mux regmap_i2c smipcie dvb_core mc ip6table_filter ip6_tables iptable_filter ip_tables x_tables ipmi_ssif ast drm_vram_helper drm_ttm_helper ttm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm drm_kms_helper crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd drm rapl intel_cstate intel_uncore igb aacraid backlight i2c_i801 agpgart acpi_ipmi i2c_algo_bit syscopyarea i2c_smbus sysfillrect ahci sysimgblt i2c_core input_leds fb_sys_fops led_class libahci intel_pch_thermal wmi ipmi_si acpi_power_meter acpi_pad button
Apr 25 18:32:57 Tower kernel: CPU: 9 PID: 12007 Comm: unraidd0 Tainted: P        W  O      5.15.27-Unraid #1
Apr 25 18:32:57 Tower kernel: Hardware name: bluechip bluechip SERVERline/X11SPL-F, BIOS 3.5 05/19/2021
Apr 25 18:32:57 Tower kernel: RIP: 0010:__domain_mapping+0x2e5/0x390
Apr 25 18:32:57 Tower kernel: Code: 2b 48 8b 4c 24 08 48 89 c2 4c 89 e6 48 c7 c7 d4 c7 ef 81 e8 b7 95 2c 00 8b 05 de 3c c3 00 85 c0 74 08 ff c8 89 05 d2 3c c3 00 <0f> 0b 8b 74 24 38 b8 34 00 00 00 8d 0c f6 83 e9 09 39 c1 0f 4f c8
Apr 25 18:32:57 Tower kernel: RSP: 0018:ffffc900038b7810 EFLAGS: 00010246
Apr 25 18:32:57 Tower kernel: RAX: 0000000000000000 RBX: ffff88819f1d33d8 RCX: 0000000000000000
Apr 25 18:32:57 Tower kernel: RDX: 0000000000000000 RSI: ffff88903fc5c550 RDI: ffff88903fc5c550
Apr 25 18:32:57 Tower kernel: RBP: ffff88810e7b8000 R08: ffff88907ffa70a8 R09: 0000000000000001
Apr 25 18:32:57 Tower kernel: R10: 00000000ffffffff R11: ffff888154cf8910 R12: 00000000000ac67b
Apr 25 18:32:57 Tower kernel: R13: ffff88819f1d33d8 R14: 0000000000001000 R15: 00000000ac67b000
Apr 25 18:32:57 Tower kernel: FS:  0000000000000000(0000) GS:ffff88903fc40000(0000) knlGS:0000000000000000
Apr 25 18:32:57 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Apr 25 18:32:57 Tower kernel: CR2: 000014c46c267ff8 CR3: 000000000200a003 CR4: 00000000007706e0
Apr 25 18:32:57 Tower kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Apr 25 18:32:57 Tower kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Apr 25 18:32:57 Tower kernel: PKRU: 55555554
Apr 25 18:32:57 Tower kernel: Call Trace:
Apr 25 18:32:57 Tower kernel: <TASK>
Apr 25 18:32:57 Tower kernel: intel_iommu_map_pages+0xf3/0x102
Apr 25 18:32:57 Tower kernel: __iommu_map+0x138/0x211
Apr 25 18:32:57 Tower kernel: __iommu_map_sg+0x8c/0x110
Apr 25 18:32:57 Tower kernel: iommu_dma_map_sg+0x245/0x3e3
Apr 25 18:32:57 Tower kernel: __dma_map_sg_attrs+0x63/0x95
Apr 25 18:32:57 Tower kernel: dma_map_sg_attrs+0xa/0x12
Apr 25 18:32:57 Tower kernel: scsi_dma_map+0x34/0x3e
Apr 25 18:32:57 Tower kernel: aac_build_sg+0x26/0xb8 [aacraid]
Apr 25 18:32:57 Tower kernel: aac_scsi_32+0x23/0xa0 [aacraid]
Apr 25 18:32:57 Tower kernel: aac_scsi_cmd+0x467/0xf94 [aacraid]
Apr 25 18:32:57 Tower kernel: ? sd_init_command+0x572/0x934
Apr 25 18:32:57 Tower kernel: ? wbt_issue+0x16/0x3c
Apr 25 18:32:57 Tower kernel: aac_queuecommand+0x17/0x21 [aacraid]
Apr 25 18:32:57 Tower kernel: scsi_queue_rq+0x57d/0x6db
Apr 25 18:32:57 Tower kernel: blk_mq_dispatch_rq_list+0x2a7/0x4da
Apr 25 18:32:57 Tower kernel: __blk_mq_do_dispatch_sched+0x23d/0x281
Apr 25 18:32:57 Tower kernel: __blk_mq_sched_dispatch_requests+0xd5/0x129
Apr 25 18:32:57 Tower kernel: blk_mq_sched_dispatch_requests+0x2f/0x52
Apr 25 18:32:57 Tower kernel: __blk_mq_run_hw_queue+0x50/0x76
Apr 25 18:32:57 Tower kernel: __blk_mq_delay_run_hw_queue+0x4d/0x108
Apr 25 18:32:57 Tower kernel: blk_mq_sched_insert_requests+0xa2/0xd9
Apr 25 18:32:57 Tower kernel: blk_mq_flush_plug_list+0xfb/0x12c
Apr 25 18:32:57 Tower kernel: blk_finish_plug+0x1f/0x2c
Apr 25 18:32:57 Tower kernel: unraidd+0x11ed/0x1237 [md_mod]
Apr 25 18:32:57 Tower kernel: ? md_thread+0x103/0x12a [md_mod]
Apr 25 18:32:57 Tower kernel: ? rmw5_write_data+0x17d/0x17d [md_mod]
Apr 25 18:32:57 Tower kernel: md_thread+0x103/0x12a [md_mod]
Apr 25 18:32:57 Tower kernel: ? init_wait_entry+0x29/0x29
Apr 25 18:32:57 Tower kernel: ? md_seq_show+0x6c8/0x6c8 [md_mod]
Apr 25 18:32:57 Tower kernel: kthread+0xde/0xe3
Apr 25 18:32:57 Tower kernel: ? set_kthread_struct+0x32/0x32
Apr 25 18:32:57 Tower kernel: ret_from_fork+0x22/0x30
Apr 25 18:32:57 Tower kernel: </TASK>
Apr 25 18:32:57 Tower kernel: ---[ end trace 7c70e023e1918757 ]---
Apr 25 18:32:57 Tower kernel: DMAR: DRHD: handling fault status reg 2
Apr 25 18:32:57 Tower kernel: DMAR: [DMA Write NO_PASID] Request device [66:00.0] fault addr 0xac670000 [fault reason 0x05] PTE Write access is not set
Apr 25 18:32:57 Tower kernel: DMAR: [DMA Write NO_PASID] Request device [66:00.0] fault addr 0xac671000 [fault reason 0x05] PTE Write access is not set
Apr 25 18:32:57 Tower kernel: DMAR: DRHD: handling fault status reg 200
Apr 25 18:32:57 Tower kernel: DMAR: DRHD: handling fault status reg 202
Apr 25 18:32:57 Tower kernel: DMAR: [DMA Write NO_PASID] Request device [66:00.0] fault addr 0xac672000 [fault reason 0x05] PTE Write access is not set
Apr 25 18:32:57 Tower kernel: DMAR: [DMA Write NO_PASID] Request device [66:00.0] fault addr 0xac673000 [fault reason 0x05] PTE Write access is not set
Apr 25 18:32:57 Tower kernel: DMAR: [DMA Write NO_PASID] Request device [66:00.0] fault addr 0xac673000 [fault reason 0x05] PTE Write access is not set
Apr 25 18:33:07 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:33:07 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:33:24 Tower sshd[16066]: Connection from 10.1.0.193 port 35322 on 10.1.0.15 port 22 rdomain ""
Apr 25 18:33:24 Tower sshd[16066]: Failed publickey for root from 10.1.0.193 port 35322 ssh2: RSA SHA256:60UN6V0VaNRuoWiAuF7EhnvmNT4BgJVi7Px/pyOTtoA
Apr 25 18:33:24 Tower sshd[16066]: Postponed keyboard-interactive for root from 10.1.0.193 port 35322 ssh2 [preauth]
Apr 25 18:33:27 Tower sshd[16066]: Postponed keyboard-interactive/pam for root from 10.1.0.193 port 35322 ssh2 [preauth]
Apr 25 18:33:27 Tower sshd[16066]: Accepted keyboard-interactive/pam for root from 10.1.0.193 port 35322 ssh2
Apr 25 18:33:27 Tower sshd[16066]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Apr 25 18:33:27 Tower elogind-daemon[1456]: New session c1 of user root.
Apr 25 18:33:27 Tower sshd[16066]: Starting session: shell on pts/0 for root from 10.1.0.193 port 35322 id 0
Apr 25 18:33:37 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:33:37 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:33:37 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:33:37 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:33:37 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:33:37 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:33:37 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:33:37 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:33:37 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:33:37 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:33:37 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:33:37 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:33:37 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:33:37 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:33:37 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:33:37 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:33:37 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:33:37 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:33:37 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:33:37 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:33:37 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:33:37 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:33:37 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:33:37 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:33:37 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:33:37 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:33:37 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:33:37 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:33:37 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:33:37 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:33:37 Tower kernel: aacraid: Host bus reset request. SCSI hang ?
Apr 25 18:33:37 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: midlevel-0
Apr 25 18:33:37 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: lowlevel-0
Apr 25 18:33:37 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: error handler-14
Apr 25 18:33:37 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: firmware-0
Apr 25 18:33:37 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: kernel-0
Apr 25 18:33:37 Tower kernel: aacraid 0000:66:00.0: Controller reset type is 3
Apr 25 18:33:37 Tower kernel: aacraid 0000:66:00.0: Issuing IOP reset
Apr 25 18:33:49 Tower kernel: mdcmd (37): nocheck Pause
Apr 25 18:34:47 Tower kernel: aacraid 0000:66:00.0: IOP reset failed
Apr 25 18:34:47 Tower kernel: aacraid 0000:66:00.0: ARC Reset attempt failed
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: [sdi] tag#391 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=185s
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: [sdi] tag#391 CDB: opcode=0x2a 2a 00 00 20 3a 20 00 00 20 00
Apr 25 18:34:47 Tower kernel: print_req_error: 58 callbacks suppressed
Apr 25 18:34:47 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2112032 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 0
Apr 25 18:34:47 Tower kernel: btrfs_dev_stat_print_on_error: 83 callbacks suppressed
Apr 25 18:34:47 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 151, rd 14, flush 6, corrupt 0, gen 0
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: [sdi] tag#392 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=185s
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: [sdi] tag#392 CDB: opcode=0x2a 2a 00 00 20 3a 60 00 00 40 00
Apr 25 18:34:47 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2112096 op 0x1:(WRITE) flags 0x1800 phys_seg 8 prio class 0
Apr 25 18:34:47 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 152, rd 14, flush 6, corrupt 0, gen 0
Apr 25 18:34:47 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 153, rd 14, flush 6, corrupt 0, gen 0
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: [sdi] tag#393 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=185s
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: [sdi] tag#393 CDB: opcode=0x2a 2a 00 00 20 39 a0 00 00 60 00
Apr 25 18:34:47 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2111904 op 0x1:(WRITE) flags 0x1800 phys_seg 12 prio class 0
Apr 25 18:34:47 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 154, rd 14, flush 6, corrupt 0, gen 0
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: [sdi] tag#394 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=185s
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: [sdi] tag#394 CDB: opcode=0x2a 2a 00 00 20 3c 00 00 01 60 00
Apr 25 18:34:47 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2112512 op 0x1:(WRITE) flags 0x1800 phys_seg 44 prio class 0
Apr 25 18:34:47 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 155, rd 14, flush 6, corrupt 0, gen 0
Apr 25 18:34:47 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 156, rd 14, flush 6, corrupt 0, gen 0
Apr 25 18:34:47 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 157, rd 14, flush 6, corrupt 0, gen 0
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: [sdi] tag#395 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=185s
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: [sdi] tag#395 CDB: opcode=0x2a 2a 00 00 20 37 a0 00 00 40 00
Apr 25 18:34:47 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2111392 op 0x1:(WRITE) flags 0x1800 phys_seg 8 prio class 0
Apr 25 18:34:47 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 158, rd 14, flush 6, corrupt 0, gen 0
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: [sdi] tag#396 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=185s
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: [sdi] tag#396 CDB: opcode=0x2a 2a 00 00 20 38 00 00 00 20 00
Apr 25 18:34:47 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2111488 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 0
Apr 25 18:34:47 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 159, rd 14, flush 6, corrupt 0, gen 0
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: [sdi] tag#397 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=185s
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: [sdi] tag#397 CDB: opcode=0x2a 2a 00 00 20 3b c0 00 00 20 00
Apr 25 18:34:47 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2112448 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 0
Apr 25 18:34:47 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 160, rd 14, flush 6, corrupt 0, gen 0
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: [sdi] tag#398 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=185s
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: [sdi] tag#398 CDB: opcode=0x2a 2a 00 00 20 3e 00 00 00 20 00
Apr 25 18:34:47 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2113024 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 0
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: [sdi] tag#399 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=185s
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: [sdi] tag#399 CDB: opcode=0x2a 2a 00 00 20 3e 40 00 00 20 00
Apr 25 18:34:47 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2113088 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 0
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: [sdi] tag#400 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=185s
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: [sdi] tag#400 CDB: opcode=0x2a 2a 00 00 20 3f 40 00 00 20 00
Apr 25 18:34:47 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2113344 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 0
Apr 25 18:34:47 Tower kernel: sd 1:1:7:0: rejecting I/O to offline device
Apr 25 18:34:47 Tower kernel: BTRFS warning (device sdh1): lost page write due to IO error on /dev/sdi1 (-5)
Apr 25 18:34:47 Tower kernel: BTRFS warning (device sdh1): lost page write due to IO error on /dev/sdi1 (-5)
Apr 25 18:34:47 Tower kernel: BTRFS warning (device sdh1): lost page write due to IO error on /dev/sdi1 (-5)

Apr 25 18:03:42 Tower emhttpd: shcmd (123): mkdir -p /mnt/cache
Apr 25 18:03:42 Tower emhttpd: /mnt/cache uuid: dfc0c8b4-bb27-44d2-b15e-006bbee148b4
Apr 25 18:03:42 Tower emhttpd: /mnt/cache found: 2
Apr 25 18:03:42 Tower emhttpd: /mnt/cache extra: 0
Apr 25 18:03:42 Tower emhttpd: /mnt/cache missing: 0
Apr 25 18:03:43 Tower emhttpd: /mnt/cache Label: none  uuid: dfc0c8b4-bb27-44d2-b15e-006bbee148b4
Apr 25 18:03:43 Tower emhttpd: /mnt/cache Total devices 2 FS bytes used 1.17TiB
Apr 25 18:03:43 Tower emhttpd: /mnt/cache devid    1 size 1.82TiB used 1.37TiB path /dev/sdh1
Apr 25 18:03:43 Tower emhttpd: /mnt/cache devid    2 size 1.82TiB used 1.37TiB path /dev/sdi1
Apr 25 18:03:43 Tower emhttpd: shcmd (124): mount -t btrfs -o noatime,space_cache=v2,discard=async -U dfc0c8b4-bb27-44d2-b15e-006bbee148b4 /mnt/cache
Apr 25 18:03:43 Tower kernel: BTRFS info (device sdh1): flagging fs with big metadata feature
Apr 25 18:03:43 Tower kernel: BTRFS info (device sdh1): turning on async discard
Apr 25 18:03:43 Tower kernel: BTRFS info (device sdh1): using free space tree
Apr 25 18:03:43 Tower kernel: BTRFS info (device sdh1): has skinny extents
Apr 25 18:03:43 Tower kernel: BTRFS info (device sdh1): bdev /dev/sdh1 errs: wr 0, rd 0, flush 0, corrupt 16, gen 0
Apr 25 18:03:43 Tower kernel: BTRFS info (device sdh1): bdev /dev/sdi1 errs: wr 71, rd 0, flush 6, corrupt 0, gen 0
Apr 25 18:03:43 Tower sshd[9267]: Connection from 10.1.0.193 port 35320 on 10.1.0.15 port 22 rdomain ""
Apr 25 18:03:43 Tower sshd[9267]: Failed publickey for root from 10.1.0.193 port 35320 ssh2: RSA SHA256:60UN6V0VaNRuoWiAuF7EhnvmNT4BgJVi7Px/pyOTtoA
Apr 25 18:03:43 Tower sshd[9267]: Postponed keyboard-interactive for root from 10.1.0.193 port 35320 ssh2 [preauth]
Apr 25 18:03:43 Tower kernel: BTRFS info (device sdh1): enabling ssd optimizations
Apr 25 18:03:43 Tower kernel: BTRFS info (device sdh1): checking UUID tree
Apr 25 18:03:44 Tower emhttpd: shcmd (125): sync
Apr 25 18:03:44 Tower kernel: aacraid: Host bus reset request. SCSI hang ?
Apr 25 18:03:44 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: midlevel-1
Apr 25 18:03:44 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: lowlevel-0
Apr 25 18:03:44 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: error handler-0
Apr 25 18:03:44 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: firmware-0
Apr 25 18:03:44 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: kernel-0
Apr 25 18:03:44 Tower kernel: aacraid 0000:66:00.0: Controller reset type is 3
Apr 25 18:03:44 Tower kernel: aacraid 0000:66:00.0: Issuing IOP reset
Apr 25 18:03:47 Tower sshd[9267]: Postponed keyboard-interactive/pam for root from 10.1.0.193 port 35320 ssh2 [preauth]
Apr 25 18:03:47 Tower sshd[9267]: Accepted keyboard-interactive/pam for root from 10.1.0.193 port 35320 ssh2
Apr 25 18:03:47 Tower sshd[9267]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Apr 25 18:03:47 Tower elogind-daemon[1457]: New session c1 of user root.
Apr 25 18:03:47 Tower sshd[9267]: Starting session: shell on pts/0 for root from 10.1.0.193 port 35320 id 0
Apr 25 18:04:20 Tower kernel: aacraid 0000:66:00.0: IOP reset succeeded
Apr 25 18:04:20 Tower kernel: aacraid: Comm Interface type2 enabled
Apr 25 18:04:29 Tower kernel: aacraid 0000:66:00.0: Scheduling bus rescan
Apr 25 18:05:09 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:05:09 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:05:40 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:05:40 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:05:40 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:05:40 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:05:40 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:05:40 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:05:40 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:05:40 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:05:40 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:05:40 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:05:40 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:05:40 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:05:40 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:05:40 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:05:40 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:05:40 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:05:40 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:05:40 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:05:40 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:05:40 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:05:40 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:05:40 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:05:40 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:05:40 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:05:40 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:05:40 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:05:40 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:05:40 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:05:40 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:05:40 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:05:40 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:05:40 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:05:40 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:05:40 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:05:40 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:05:40 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:05:40 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:05:40 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:05:40 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:05:40 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:05:40 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:05:40 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:05:40 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:05:40 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:05:40 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:05:40 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:05:40 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:05:40 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:05:40 Tower kernel: aacraid: Host adapter abort request.
Apr 25 18:05:40 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 18:05:40 Tower kernel: aacraid: Host bus reset request. SCSI hang ?
Apr 25 18:05:40 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: midlevel-0
Apr 25 18:05:40 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: lowlevel-0
Apr 25 18:05:40 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: error handler-24
Apr 25 18:05:40 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: firmware-0
Apr 25 18:05:40 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: kernel-0
Apr 25 18:05:40 Tower kernel: aacraid 0000:66:00.0: Controller reset type is 3
Apr 25 18:05:40 Tower kernel: aacraid 0000:66:00.0: Issuing IOP reset
Apr 25 18:06:38 Tower nginx: 2022/04/25 18:06:38 [error] 5206#5206: *597 upstream timed out (110: Connection timed out) while reading upstream, client: 10.1.0.193, server: , request: "POST /update.htm HTTP/1.1", upstream: "http://unix:/var/run/emhttpd.socket:/update.htm", host: "10.1.0.15", referrer: "http://10.1.0.15/Main"
Apr 25 18:06:50 Tower kernel: aacraid 0000:66:00.0: IOP reset failed
Apr 25 18:06:50 Tower kernel: aacraid 0000:66:00.0: ARC Reset attempt failed
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: Device offlined - not ready after error recovery
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: [sdi] tag#326 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=130s
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: [sdi] tag#326 CDB: opcode=0x2a 2a 00 00 20 10 e0 00 01 80 00
Apr 25 18:06:50 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2101472 op 0x1:(WRITE) flags 0x1800 phys_seg 47 prio class 0
Apr 25 18:06:50 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 72, rd 0, flush 6, corrupt 0, gen 0
Apr 25 18:06:50 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 73, rd 0, flush 6, corrupt 0, gen 0
Apr 25 18:06:50 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 74, rd 0, flush 6, corrupt 0, gen 0
Apr 25 18:06:50 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 75, rd 0, flush 6, corrupt 0, gen 0
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: [sdi] tag#328 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=130s
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: [sdi] tag#328 CDB: opcode=0x2a 2a 00 00 20 12 80 00 02 00 00
Apr 25 18:06:50 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2101888 op 0x1:(WRITE) flags 0x1800 phys_seg 64 prio class 0
Apr 25 18:06:50 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 76, rd 0, flush 6, corrupt 0, gen 0
Apr 25 18:06:50 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 77, rd 0, flush 6, corrupt 0, gen 0
Apr 25 18:06:50 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 78, rd 0, flush 6, corrupt 0, gen 0
Apr 25 18:06:50 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 79, rd 0, flush 6, corrupt 0, gen 0
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: [sdi] tag#342 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=130s
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: [sdi] tag#342 CDB: opcode=0x2a 2a 00 00 20 14 a0 00 00 40 00
Apr 25 18:06:50 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2102432 op 0x1:(WRITE) flags 0x1800 phys_seg 8 prio class 0
Apr 25 18:06:50 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 80, rd 0, flush 6, corrupt 0, gen 0
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: [sdi] tag#343 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=130s
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: [sdi] tag#343 CDB: opcode=0x2a 2a 00 00 20 15 40 00 00 20 00
Apr 25 18:06:50 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2102592 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 0
Apr 25 18:06:50 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 81, rd 0, flush 6, corrupt 0, gen 0
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: [sdi] tag#344 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=130s
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: [sdi] tag#344 CDB: opcode=0x2a 2a 00 00 20 15 a0 00 00 20 00
Apr 25 18:06:50 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2102688 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 0
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: [sdi] tag#345 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=130s
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: [sdi] tag#345 CDB: opcode=0x2a 2a 00 00 20 15 e0 00 00 80 00
Apr 25 18:06:50 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2102752 op 0x1:(WRITE) flags 0x1800 phys_seg 16 prio class 0
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: [sdi] tag#346 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=130s
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: [sdi] tag#346 CDB: opcode=0x2a 2a 00 00 20 16 80 00 00 20 00
Apr 25 18:06:50 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2102912 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 0
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: [sdi] tag#347 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=130s
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: [sdi] tag#347 CDB: opcode=0x2a 2a 00 00 20 17 a0 00 00 20 00
Apr 25 18:06:50 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2103200 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 0
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: [sdi] tag#348 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=130s
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: [sdi] tag#348 CDB: opcode=0x2a 2a 00 00 20 17 e0 00 00 40 00
Apr 25 18:06:50 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2103264 op 0x1:(WRITE) flags 0x1800 phys_seg 8 prio class 0
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: [sdi] tag#349 UNKNOWN(0x2003) Result: hostbyte=0x03 driverbyte=DRIVER_OK cmd_age=130s
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: [sdi] tag#349 CDB: opcode=0x2a 2a 00 00 20 18 40 00 00 40 00
Apr 25 18:06:50 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2103360 op 0x1:(WRITE) flags 0x1800 phys_seg 8 prio class 0
Apr 25 18:06:50 Tower kernel: sd 1:1:7:0: rejecting I/O to offline device
Apr 25 18:06:58 Tower ntpd[1484]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
Apr 25 18:07:50 Tower kernel: aacraid: Host bus reset request. SCSI hang ?
Apr 25 18:07:50 Tower kernel: aacraid 0000:66:00.0: Adapter health - -3
Apr 25 18:07:50 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: midlevel-0
Apr 25 18:07:50 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: lowlevel-0
Apr 25 18:07:50 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: error handler-0
Apr 25 18:07:50 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: firmware-18
Apr 25 18:07:50 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: kernel-0
Apr 25 18:07:50 Tower kernel: aacraid 0000:66:00.0: Controller reset type is 3
Apr 25 18:07:50 Tower kernel: aacraid 0000:66:00.0: Issuing IOP reset

Apr 25 17:27:12 Tower kernel: sd 1:1:7:0: [sdi] tag#809 device offline or changed
Apr 25 17:27:12 Tower kernel: sd 1:1:7:0: [sdi] tag#823 device offline or changed
Apr 25 17:27:12 Tower kernel: sd 1:1:7:0: [sdi] tag#824 device offline or changed
Apr 25 17:27:12 Tower kernel: sd 1:1:7:0: [sdi] tag#770 device offline or changed
Apr 25 17:27:12 Tower kernel: BTRFS warning (device sdh1): lost page write due to IO error on /dev/sdi1 (-5)
Apr 25 17:27:12 Tower kernel: sd 1:1:7:0: [sdi] tag#771 device offline or changed
Apr 25 17:27:12 Tower kernel: BTRFS warning (device sdh1): lost page write due to IO error on /dev/sdi1 (-5)
Apr 25 17:27:12 Tower kernel: sd 1:1:7:0: [sdi] tag#772 device offline or changed
Apr 25 17:27:12 Tower kernel: BTRFS warning (device sdh1): lost page write due to IO error on /dev/sdi1 (-5)
Apr 25 17:27:12 Tower kernel: BTRFS error (device sdh1): error writing primary super block to device 2
Apr 25 17:27:12 Tower kernel: BTRFS info (device sdh1): checking UUID tree
Apr 25 17:27:14 Tower emhttpd: shcmd (46): sync
Apr 25 17:27:17 Tower kernel: aacraid: Host bus reset request. SCSI hang ?
Apr 25 17:27:17 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: midlevel-1
Apr 25 17:27:17 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: lowlevel-0
Apr 25 17:27:17 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: error handler-0
Apr 25 17:27:17 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: firmware-0
Apr 25 17:27:17 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: kernel-0
Apr 25 17:27:17 Tower kernel: aacraid 0000:66:00.0: Controller reset type is 3
Apr 25 17:27:17 Tower kernel: aacraid 0000:66:00.0: Issuing IOP reset
Apr 25 17:27:45 Tower webGUI: Successful login user root from 10.1.0.193
Apr 25 17:27:51 Tower kernel: aacraid 0000:66:00.0: IOP reset succeeded
Apr 25 17:27:51 Tower kernel: aacraid: Comm Interface type2 enabled
Apr 25 17:28:00 Tower kernel: aacraid 0000:66:00.0: Scheduling bus rescan
Apr 25 17:28:41 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:28:41 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:29:11 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:29:11 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:29:11 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:29:11 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:29:11 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:29:11 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:29:11 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:29:11 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:29:11 Tower kernel: aacraid: Host bus reset request. SCSI hang ?
Apr 25 17:29:11 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: midlevel-0
Apr 25 17:29:11 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: lowlevel-0
Apr 25 17:29:11 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: error handler-3
Apr 25 17:29:11 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: firmware-0
Apr 25 17:29:11 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: kernel-0
Apr 25 17:29:11 Tower kernel: aacraid 0000:66:00.0: Controller reset type is 3
Apr 25 17:29:11 Tower kernel: aacraid 0000:66:00.0: Issuing IOP reset
Apr 25 17:29:45 Tower kernel: aacraid 0000:66:00.0: IOP reset succeeded
Apr 25 17:29:45 Tower kernel: aacraid: Comm Interface type2 enabled
Apr 25 17:29:54 Tower kernel: aacraid 0000:66:00.0: Scheduling bus rescan
Apr 25 17:30:02 Tower shutdown[9779]: shutting down for system reboot
Apr 25 17:30:04 Tower kernel: sdi: detected capacity change from 3907029168 to 0
Apr 25 17:30:04 Tower kernel: sd 1:1:7:0: [sdi] tag#404 access beyond end of device
Apr 25 17:30:04 Tower kernel: print_req_error: 333 callbacks suppressed
Apr 25 17:30:04 Tower kernel: blk_update_request: I/O error, dev sdi, sector 7454656 op 0x0:(READ) flags 0x1000 phys_seg 4 prio class 0
Apr 25 17:30:04 Tower kernel: btrfs_dev_stat_print_on_error: 500 callbacks suppressed
Apr 25 17:30:04 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 582, rd 1, flush 6, corrupt 0, gen 0
Apr 25 17:30:04 Tower kernel: aacraid: Host bus reset request. SCSI hang ?
Apr 25 17:30:04 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: midlevel-1
Apr 25 17:30:04 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: lowlevel-0
Apr 25 17:30:04 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: error handler-0
Apr 25 17:30:04 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: firmware-0
Apr 25 17:30:04 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: kernel-0
Apr 25 17:30:04 Tower kernel: aacraid 0000:66:00.0: Controller reset type is 3
Apr 25 17:30:04 Tower kernel: aacraid 0000:66:00.0: Issuing IOP reset
Apr 25 17:30:38 Tower kernel: aacraid 0000:66:00.0: IOP reset succeeded
Apr 25 17:30:38 Tower kernel: aacraid: Comm Interface type2 enabled
Apr 25 17:30:47 Tower kernel: aacraid 0000:66:00.0: Scheduling bus rescan
Apr 25 17:30:57 Tower kernel: sd 1:1:7:0: [sdi] tag#915 access beyond end of device
Apr 25 17:30:57 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2282450784 op 0x1:(WRITE) flags 0x1800 phys_seg 4 prio class 0
Apr 25 17:30:57 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 583, rd 1, flush 6, corrupt 0, gen 0
Apr 25 17:30:57 Tower kernel: sd 1:1:7:0: [sdi] tag#916 access beyond end of device
Apr 25 17:30:57 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2282451136 op 0x1:(WRITE) flags 0x1800 phys_seg 11 prio class 0
Apr 25 17:30:57 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 584, rd 1, flush 6, corrupt 0, gen 0
Apr 25 17:30:57 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 585, rd 1, flush 6, corrupt 0, gen 0
Apr 25 17:30:57 Tower kernel: sd 1:1:7:0: [sdi] tag#917 access beyond end of device
Apr 25 17:30:57 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2282451328 op 0x1:(WRITE) flags 0x1800 phys_seg 12 prio class 0
Apr 25 17:30:57 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 586, rd 1, flush 6, corrupt 0, gen 0
Apr 25 17:30:57 Tower kernel: sd 1:1:7:0: [sdi] tag#918 access beyond end of device
Apr 25 17:30:57 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2282451616 op 0x1:(WRITE) flags 0x1800 phys_seg 16 prio class 0
Apr 25 17:30:57 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 587, rd 1, flush 6, corrupt 0, gen 0
Apr 25 17:30:57 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 588, rd 1, flush 6, corrupt 0, gen 0
Apr 25 17:30:57 Tower kernel: sd 1:1:7:0: [sdi] tag#919 access beyond end of device
Apr 25 17:30:57 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2282451968 op 0x1:(WRITE) flags 0x1800 phys_seg 12 prio class 0
Apr 25 17:30:57 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 589, rd 1, flush 6, corrupt 0, gen 0
Apr 25 17:30:57 Tower kernel: sd 1:1:7:0: [sdi] tag#920 access beyond end of device
Apr 25 17:30:57 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2282452608 op 0x1:(WRITE) flags 0x1800 phys_seg 12 prio class 0
Apr 25 17:30:57 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 590, rd 1, flush 6, corrupt 0, gen 0
Apr 25 17:30:57 Tower kernel: sd 1:1:7:0: [sdi] tag#921 access beyond end of device
Apr 25 17:30:57 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2282453728 op 0x1:(WRITE) flags 0x1800 phys_seg 12 prio class 0
Apr 25 17:30:57 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 591, rd 1, flush 6, corrupt 0, gen 0
Apr 25 17:30:57 Tower kernel: BTRFS error (device sdh1): bdev /dev/sdi1 errs: wr 592, rd 1, flush 6, corrupt 0, gen 0
Apr 25 17:30:57 Tower kernel: sd 1:1:7:0: [sdi] tag#922 access beyond end of device
Apr 25 17:30:57 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2282454080 op 0x1:(WRITE) flags 0x1800 phys_seg 20 prio class 0
Apr 25 17:30:57 Tower kernel: sd 1:1:7:0: [sdi] tag#923 access beyond end of device
Apr 25 17:30:57 Tower kernel: blk_update_request: I/O error, dev sdi, sector 2282455840 op 0x1:(WRITE) flags 0x1800 phys_seg 8 prio class 0
Apr 25 17:30:57 Tower kernel: sd 1:1:7:0: [sdi] tag#924 access beyond end of device
Apr 25 17:30:57 Tower kernel: blk_update_request: I/O error, dev sdi, sector 7454656 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Apr 25 17:30:57 Tower kernel: sd 1:1:7:0: [sdi] tag#925 access beyond end of device
Apr 25 17:30:57 Tower kernel: aacraid: Host bus reset request. SCSI hang ?
Apr 25 17:30:57 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: midlevel-1
Apr 25 17:30:57 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: lowlevel-0
Apr 25 17:30:57 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: error handler-0
Apr 25 17:30:57 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: firmware-0
Apr 25 17:30:57 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: kernel-0
Apr 25 17:30:57 Tower kernel: aacraid 0000:66:00.0: Controller reset type is 3
Apr 25 17:30:57 Tower kernel: aacraid 0000:66:00.0: Issuing IOP reset
Apr 25 17:31:16 Tower ntpd[1484]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized

Apr 25 17:00:45 Tower kernel: sd 1:1:7:0: [sdi] tag#1000 device offline or changed
Apr 25 17:00:45 Tower kernel: BTRFS error (device sdh1): error writing primary super block to device 2
Apr 25 17:00:45 Tower kernel: sd 1:1:7:0: [sdi] tag#48 device offline or changed
Apr 25 17:00:45 Tower kernel: sd 1:1:7:0: [sdi] tag#49 device offline or changed
Apr 25 17:00:45 Tower kernel: sd 1:1:7:0: [sdi] tag#51 device offline or changed
Apr 25 17:00:45 Tower kernel: BTRFS error (device sdh1): error writing primary super block to device 2
Apr 25 17:00:45 Tower kernel: sd 1:1:7:0: [sdi] tag#52 device offline or changed
Apr 25 17:00:45 Tower kernel: sd 1:1:7:0: [sdi] tag#911 device offline or changed
Apr 25 17:00:45 Tower kernel: sd 1:1:7:0: [sdi] tag#912 device offline or changed
Apr 25 17:00:45 Tower kernel: sd 1:1:7:0: [sdi] tag#914 device offline or changed
Apr 25 17:00:45 Tower kernel: BTRFS error (device sdh1): error writing primary super block to device 2
Apr 25 17:00:45 Tower kernel: sd 1:1:7:0: [sdi] tag#56 device offline or changed
Apr 25 17:00:45 Tower kernel: sd 1:1:7:0: [sdi] tag#57 device offline or changed
Apr 25 17:00:45 Tower kernel: sd 1:1:7:0: [sdi] tag#59 device offline or changed
Apr 25 17:00:45 Tower kernel: BTRFS error (device sdh1): error writing primary super block to device 2
Apr 25 17:00:45 Tower kernel: sd 1:1:7:0: [sdi] tag#0 device offline or changed
Apr 25 17:00:45 Tower kernel: sd 1:1:7:0: [sdi] tag#917 device offline or changed
Apr 25 17:00:45 Tower kernel: sd 1:1:7:0: [sdi] tag#918 device offline or changed
Apr 25 17:00:45 Tower kernel: sd 1:1:7:0: [sdi] tag#920 device offline or changed
Apr 25 17:00:45 Tower kernel: BTRFS error (device sdh1): error writing primary super block to device 2
Apr 25 17:00:45 Tower kernel: sd 1:1:7:0: [sdi] tag#2 device offline or changed
Apr 25 17:00:45 Tower kernel: sd 1:1:7:0: [sdi] tag#4 device offline or changed
Apr 25 17:00:45 Tower kernel: sd 1:1:7:0: [sdi] tag#6 device offline or changed
Apr 25 17:00:45 Tower kernel: BTRFS error (device sdh1): error writing primary super block to device 2
Apr 25 17:00:45 Tower kernel: sd 1:1:7:0: [sdi] tag#12 device offline or changed
Apr 25 17:00:45 Tower kernel: sd 1:1:7:0: [sdi] tag#16 device offline or changed
Apr 25 17:00:45 Tower kernel: sd 1:1:7:0: [sdi] tag#13 device offline or changed
Apr 25 17:00:45 Tower kernel: sd 1:1:7:0: [sdi] tag#14 device offline or changed
Apr 25 17:00:45 Tower kernel: sd 1:1:7:0: [sdi] tag#15 device offline or changed
[...]
Apr 25 17:03:39 Tower kernel: sd 1:1:7:0: [sdi] tag#1011 device offline or changed
Apr 25 17:03:39 Tower kernel: sd 1:1:7:0: [sdi] tag#256 device offline or changed
Apr 25 17:03:39 Tower kernel: sd 1:1:7:0: [sdi] tag#1012 device offline or changed
Apr 25 17:03:39 Tower kernel: sd 1:1:7:0: [sdi] tag#1013 device offline or changed
Apr 25 17:03:39 Tower kernel: sd 1:1:7:0: [sdi] tag#1014 device offline or changed
Apr 25 17:03:39 Tower kernel: sd 1:1:7:0: [sdi] tag#1015 device offline or changed
Apr 25 17:03:39 Tower kernel: sd 1:1:7:0: [sdi] tag#962 device offline or changed
Apr 25 17:03:39 Tower kernel: sd 1:1:7:0: [sdi] tag#963 device offline or changed
Apr 25 17:03:39 Tower kernel: sd 1:1:7:0: [sdi] tag#990 device offline or changed
Apr 25 17:03:39 Tower kernel: sd 1:1:7:0: [sdi] tag#991 device offline or changed
Apr 25 17:03:39 Tower kernel: sd 1:1:7:0: [sdi] tag#992 device offline or changed
Apr 25 17:03:41 Tower kernel: sd 1:1:7:0: [sdi] tag#39 device offline or changed
Apr 25 17:03:41 Tower kernel: btrfs_printk: 3143 callbacks suppressed
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fa9a0 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fa9b8 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fa9c0 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fa9c8 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fa9d8 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fa9e0 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fa9e8 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fa9d0 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fa9f0 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fa9f8 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faa08 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faa00 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faa10 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faa18 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faa20 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faa28 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faa30 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faa40 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faa48 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faa38 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faa50 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faa58 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faa60 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faa68 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faa70 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faa78 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faa80 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faa88 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faa90 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faa98 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faaa0 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faaa8 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faab0 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faab8 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faac0 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faac8 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faad0 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faad8 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faaf0 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faae8 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faae0 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faaf8 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fab00 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fab08 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fab10 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fab18 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fab20 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fab28 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fab30 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fab38 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fab40 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fab48 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fab50 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fab58 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fab60 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fab70 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fab68 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fab78 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fab80 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fab88 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fab90 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fab98 len 0 err no 10
Apr 25 17:03:41 Tower kernel: sd 1:1:7:0: [sdi] tag#40 device offline or changed
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faba0 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160faba8 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fabb0 len 0 err no 10
Apr 25 17:03:41 Tower kernel: sd 1:1:7:0: [sdi] tag#41 device offline or changed
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fabb8 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fabc0 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fabc8 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fabd0 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fabd8 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fabe0 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fabe8 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fabf0 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fabf8 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fac08 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fac00 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fac10 len 0 err no 10
Apr 25 17:03:41 Tower kernel: sd 1:1:7:0: [sdi] tag#42 device offline or changed
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fac18 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fac20 len 0 err no 10
Apr 25 17:03:41 Tower kernel: sd 1:1:7:0: [sdi] tag#43 device offline or changed
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fac28 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fac30 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fac40 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fac38 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fac48 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x160fac50 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x1a1c3ee8 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x1a1c3ef0 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x1a1c3f00 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x1a1c3ef8 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x1a1c3f08 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x1a1c3f10 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x1a1c3f18 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x1a1c3f20 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x1a1c3f30 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x1a1c3f28 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x1a1c3f38 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x1a1c3f40 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x1a1c3f48 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x1a1c3f50 len 0 err no 10
Apr 25 17:03:41 Tower kernel: BTRFS warning (device sdh1): direct IO failed ino 2102925 rw 0,0 sector 0x1a1c3f58 len 0 err no 10
Apr 25 17:03:41 Tower kernel: sd 1:1:7:0: [sdi] tag#44 device offline or changed
Apr 25 17:03:41 Tower kernel: sd 1:1:7:0: [sdi] tag#45 device offline or changed
Apr 25 17:03:41 Tower kernel: sd 1:1:7:0: [sdi] tag#47 device offline or changed
Apr 25 17:03:41 Tower kernel: sd 1:1:7:0: [sdi] tag#48 device offline or changed
Apr 25 17:03:41 Tower kernel: sd 1:1:7:0: [sdi] tag#277 device offline or changed
[...]
Apr 25 17:03:41 Tower kernel: sd 1:1:7:0: [sdi] tag#319 device offline or changed
Apr 25 17:03:41 Tower kernel: sd 1:1:7:0: [sdi] tag#256 device offline or changed
Apr 25 17:03:41 Tower kernel: sd 1:1:7:0: [sdi] tag#257 device offline or changed
Apr 25 17:03:41 Tower kernel: sd 1:1:7:0: [sdi] tag#258 device offline or changed
Apr 25 17:03:41 Tower kernel: sd 1:1:7:0: [sdi] tag#259 device offline or changed
Apr 25 17:03:41 Tower kernel: sd 1:1:7:0: [sdi] tag#260 device offline or changed
Apr 25 17:03:41 Tower kernel: sd 1:1:7:0: [sdi] tag#261 device offline or changed
Apr 25 17:03:41 Tower kernel: sd 1:1:7:0: [sdi] tag#262 device offline or changed
Apr 25 17:03:41 Tower kernel: sd 1:1:7:0: [sdi] tag#263 device offline or changed
Apr 25 17:03:41 Tower kernel: sd 1:1:7:0: [sdi] tag#264 device offline or changed
Apr 25 17:03:41 Tower kernel: sd 1:1:7:0: [sdi] tag#265 device offline or changed
Apr 25 17:03:41 Tower kernel: sd 1:1:7:0: [sdi] tag#266 device offline or changed
Apr 25 17:03:41 Tower kernel: sd 1:1:7:0: [sdi] tag#267 device offline or changed
Apr 25 17:03:41 Tower kernel: sd 1:1:7:0: [sdi] tag#268 device offline or changed
Apr 25 17:03:41 Tower kernel: sd 1:1:7:0: [sdi] tag#269 device offline or changed
Apr 25 17:03:41 Tower kernel: sd 1:1:7:0: [sdi] tag#270 device offline or changed
Apr 25 17:04:08 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:08 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:04:39 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:04:39 Tower kernel: aacraid: Host bus reset request. SCSI hang ?
Apr 25 17:04:39 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: midlevel-0
Apr 25 17:04:39 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: lowlevel-0
Apr 25 17:04:39 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: error handler-88
Apr 25 17:04:39 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: firmware-0
Apr 25 17:04:39 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: kernel-0
Apr 25 17:04:39 Tower kernel: aacraid 0000:66:00.0: Controller reset type is 3
Apr 25 17:04:39 Tower kernel: aacraid 0000:66:00.0: Issuing IOP reset
Apr 25 17:04:50 Tower webGUI: Successful login user root from 10.1.0.193
Apr 25 17:05:12 Tower kernel: aacraid 0000:66:00.0: IOP reset succeeded
Apr 25 17:05:12 Tower kernel: aacraid: Comm Interface type2 enabled
Apr 25 17:05:21 Tower kernel: aacraid 0000:66:00.0: Scheduling bus rescan
Apr 25 17:05:56 Tower sshd[26550]: Connection from 10.1.0.193 port 35310 on 10.1.0.15 port 22 rdomain ""
Apr 25 17:05:56 Tower sshd[26550]: Failed publickey for root from 10.1.0.193 port 35310 ssh2: RSA SHA256:60UN6V0VaNRuoWiAuF7EhnvmNT4BgJVi7Px/pyOTtoA
Apr 25 17:05:56 Tower sshd[26550]: Postponed keyboard-interactive for root from 10.1.0.193 port 35310 ssh2 [preauth]
Apr 25 17:05:59 Tower sshd[26550]: Postponed keyboard-interactive/pam for root from 10.1.0.193 port 35310 ssh2 [preauth]
Apr 25 17:05:59 Tower sshd[26550]: Accepted keyboard-interactive/pam for root from 10.1.0.193 port 35310 ssh2
Apr 25 17:05:59 Tower sshd[26550]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Apr 25 17:05:59 Tower elogind-daemon[1452]: New session c2 of user root.
Apr 25 17:05:59 Tower sshd[26550]: Starting session: shell on pts/1 for root from 10.1.0.193 port 35310 id 0
Apr 25 17:06:02 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:06:02 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:06:32 Tower kernel: aacraid: Host adapter abort request.
Apr 25 17:06:32 Tower kernel: aacraid: Outstanding commands on (1,1,7,0):
Apr 25 17:06:32 Tower kernel: aacraid: Host bus reset request. SCSI hang ?
Apr 25 17:06:32 Tower kernel: sdi: detected capacity change from 3907029168 to 0
Apr 25 17:06:32 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: midlevel-0
Apr 25 17:06:32 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: lowlevel-0
Apr 25 17:06:32 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: error handler-88
Apr 25 17:06:32 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: firmware-0
Apr 25 17:06:32 Tower kernel: aacraid 0000:66:00.0: outstanding cmd: kernel-0
Apr 25 17:06:32 Tower kernel: aacraid 0000:66:00.0: Controller reset type is 3
Apr 25 17:06:32 Tower kernel: aacraid 0000:66:00.0: Issuing IOP reset

 

Edited by aronmal
Link to comment
  • aronmal changed the title to Urgent [6.10.0-rc3] Drives are having problems and showing errors. Adaptec Raid Controller Problem?
  • aronmal changed the title to [6.10.0-rc3] Drives are having problems and showing errors. Adaptec Raid Controller Problem?
  • 1 year later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing
×
×
  • Create New...