Hello again, it seems after solving one problem another one pops up

 

This time its my Raid1 SDDs (2x240GB Seagate Ironwolf 510), which are connected via "Cablecc 4X NVME M.2 AHCI to PCIE Express 3.0 Gen3 X16 RAID Card VROC Raid0 Hyper Adapter".  Out of the blue I got errors that one of the drives is missing:

"Cache pool BTRFS missing device(s)". I thought it was the PCIe riser cable which I used, but even with another PCIe to NVME adapter (Sabrent NVMe M.2 SSD to PCIe X16 / X8 / X4 Card with Aluminium Heatsink (EC-PCIE) I still get the same error. 

 

These errors are with the Sabrent adapter and SSD1:

I do get these warnings:

Nov 15 11:14:46 Stormy kernel: ACPI: Early table checksum verification disabled
Nov 15 11:14:46 Stormy kernel: pci 0000:24:00.0: BAR 7: failed to assign [mem size 0x00100000 64bit]
Nov 15 11:14:46 Stormy kernel: pci 0000:24:00.0: BAR 10: failed to assign [mem size 0x00100000 64bit]
Nov 15 11:14:46 Stormy kernel: pci 0000:24:00.1: BAR 7: failed to assign [mem size 0x00100000 64bit]
Nov 15 11:14:46 Stormy kernel: pci 0000:24:00.1: BAR 10: failed to assign [mem size 0x00100000 64bit]
Nov 15 11:14:46 Stormy kernel: pci 0000:21:02.0: BAR 14: failed to assign [mem size 0x00500000]
Nov 15 11:14:46 Stormy kernel: pci 0000:24:00.1: BAR 6: failed to assign [mem size 0x00080000 pref]
Nov 15 11:14:46 Stormy kernel: pci 0000:24:00.0: BAR 7: failed to assign [mem size 0x00100000 64bit]
Nov 15 11:14:46 Stormy kernel: pci 0000:24:00.0: BAR 10: failed to assign [mem size 0x00100000 64bit]
Nov 15 11:14:46 Stormy kernel: pci 0000:24:00.1: BAR 7: failed to assign [mem size 0x00100000 64bit]
Nov 15 11:14:46 Stormy kernel: pci 0000:24:00.1: BAR 10: failed to assign [mem size 0x00100000 64bit]
Nov 15 11:14:46 Stormy kernel: pci 0000:20:00.0: BAR 14: failed to assign [mem size 0x02400000]
Nov 15 11:14:46 Stormy kernel: pci 0000:21:06.0: BAR 14: failed to assign [mem size 0x01800000]
Nov 15 11:14:46 Stormy kernel: pci 0000:21:01.0: BAR 14: failed to assign [mem size 0x00100000]
Nov 15 11:14:46 Stormy kernel: pci 0000:21:02.0: BAR 14: failed to assign [mem size 0x00500000]
Nov 15 11:14:46 Stormy kernel: pci 0000:21:04.0: BAR 14: failed to assign [mem size 0x00100000]
Nov 15 11:14:46 Stormy kernel: pci 0000:21:05.0: BAR 14: failed to assign [mem size 0x00100000]
Nov 15 11:14:46 Stormy kernel: pci 0000:21:08.0: BAR 14: failed to assign [mem size 0x00200000]
Nov 15 11:14:46 Stormy kernel: pci 0000:21:09.0: BAR 14: failed to assign [mem size 0x00100000]
Nov 15 11:14:46 Stormy kernel: pci 0000:21:0a.0: BAR 14: failed to assign [mem size 0x00100000]
Nov 15 11:14:46 Stormy kernel: pci 0000:23:00.0: BAR 0: failed to assign [mem size 0x00004000 64bit]
Nov 15 11:14:46 Stormy kernel: pci 0000:24:00.0: BAR 7: failed to assign [mem size 0x00100000 64bit]
Nov 15 11:14:46 Stormy kernel: pci 0000:24:00.0: BAR 10: failed to assign [mem size 0x00100000 64bit]
Nov 15 11:14:46 Stormy kernel: pci 0000:24:00.1: BAR 7: failed to assign [mem size 0x00100000 64bit]
Nov 15 11:14:46 Stormy kernel: pci 0000:24:00.1: BAR 10: failed to assign [mem size 0x00100000 64bit]
Nov 15 11:14:46 Stormy kernel: pci 0000:26:00.0: BAR 0: failed to assign [mem size 0x00080000]
Nov 15 11:14:46 Stormy kernel: pci 0000:26:00.0: BAR 3: failed to assign [mem size 0x00004000]
Nov 15 11:14:46 Stormy kernel: pci 0000:27:00.0: BAR 0: failed to assign [mem size 0x00080000]
Nov 15 11:14:46 Stormy kernel: pci 0000:27:00.0: BAR 3: failed to assign [mem size 0x00004000]
Nov 15 11:14:46 Stormy kernel: pci 0000:28:00.0: BAR 14: failed to assign [mem size 0x01800000]
Nov 15 11:14:46 Stormy kernel: pci 0000:29:00.0: BAR 0: failed to assign [mem size 0x01000000]
Nov 15 11:14:46 Stormy kernel: pci 0000:29:00.0: BAR 1: failed to assign [mem size 0x00020000]
Nov 15 11:14:46 Stormy kernel: pci 0000:2a:00.1: BAR 0: failed to assign [mem size 0x00100000 64bit]
Nov 15 11:14:46 Stormy kernel: pci 0000:2a:00.3: BAR 0: failed to assign [mem size 0x00100000 64bit]
Nov 15 11:14:46 Stormy kernel: pci 0000:2b:00.0: BAR 5: failed to assign [mem size 0x00000800]
Nov 15 11:14:46 Stormy kernel: pci 0000:2c:00.0: BAR 5: failed to assign [mem size 0x00000800]
Nov 15 11:14:46 Stormy kernel: sd 0:0:0:0: [sda] Asking for cache data failed

 

And these errors:

Nov 15 11:16:20 Stormy kernel: BTRFS error (device nvme2n1p1): bad tree block start, want 321237188608 have 545259526
Nov 15 11:16:20 Stormy kernel: BTRFS: error (device nvme2n1p1) in __btrfs_free_extent:3188: errno=-5 IO failure
Nov 15 11:16:20 Stormy kernel: BTRFS: error (device nvme2n1p1) in btrfs_run_delayed_refs:2150: errno=-5 IO failure
Nov 15 11:16:25 Stormy inotifywait[9879]: Couldn't watch /mnt/apps: Input/output error
Nov 15 11:16:25 Stormy kernel: BTRFS error (device nvme2n1p1): bad tree block start, want 321237745664 have 1157409746040655887
Nov 15 11:16:29 Stormy kernel: BTRFS error (device nvme2n1p1): bad tree block start, want 321241186304 have 826882912313
Nov 15 11:16:29 Stormy kernel: BTRFS error (device nvme2n1p1): bad tree block start, want 321241186304 have 826882912313
Nov 15 11:16:29 Stormy kernel: blk_update_request: I/O error, dev loop2, sector 62914432 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Nov 15 11:16:29 Stormy kernel: BTRFS error (device nvme2n1p1): bad tree block start, want 321241186304 have 826882912313
Nov 15 11:16:29 Stormy kernel: blk_update_request: I/O error, dev loop2, sector 62914432 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Nov 15 11:16:29 Stormy kernel: Buffer I/O error on dev loop2, logical block 7864304, async page read
Nov 15 11:16:29 Stormy kernel: BTRFS error (device nvme2n1p1): bad tree block start, want 321241186304 have 826882912313
Nov 15 11:16:29 Stormy kernel: blk_update_request: I/O error, dev loop2, sector 62914432 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Nov 15 11:16:29 Stormy kernel: Buffer I/O error on dev loop2, logical block 7864304, async page read
Nov 15 11:16:29 Stormy kernel: BTRFS error (device nvme2n1p1): bad tree block start, want 321241186304 have 826882912313
Nov 15 11:16:29 Stormy kernel: blk_update_request: I/O error, dev loop2, sector 62914432 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Nov 15 11:16:29 Stormy kernel: Buffer I/O error on dev loop2, logical block 7864304, async page read
Nov 15 11:16:29 Stormy kernel: BTRFS error (device loop2): bad tree block start, want 22020096 have 1117782016
Nov 15 11:16:29 Stormy kernel: BTRFS error (device loop2): bad tree block start, want 22020096 have 1126170624
Nov 15 11:16:29 Stormy kernel: BTRFS error (device loop2): failed to read chunk root
Nov 15 11:16:29 Stormy root: mount: /var/lib/docker: wrong fs type, bad option, bad superblock on /dev/loop2, missing codepage or helper program, or other error.
Nov 15 11:16:29 Stormy kernel: BTRFS error (device loop2): open_ctree failed
Nov 15 11:16:29 Stormy root: mount error

 

I also tried different PCIe slots on my MB, but same behaviour.

 

When I use the same adapter with SSD2. I get these errors (the PCI warnings are the same):

Nov 15 14:01:54 Stormy kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 49, gen 0
Nov 15 14:01:54 Stormy kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 50, gen 0
Nov 15 14:01:54 Stormy kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 51, gen 0
Nov 15 14:01:54 Stormy kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 52, gen 0
Nov 15 14:01:54 Stormy kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 53, gen 0
Nov 15 14:01:54 Stormy kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 54, gen 0
Nov 15 14:06:11 Stormy kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 55, gen 0
Nov 15 14:06:11 Stormy kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 56, gen 0
Nov 15 14:06:11 Stormy kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 57, gen 0
Nov 15 14:06:11 Stormy kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 58, gen 0
Nov 15 14:06:11 Stormy kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 59, gen 0
Nov 15 14:06:11 Stormy kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 60, gen 0

 

So  does this mean that the SSD1 is defective?

If so I hope that SSD2 is still okay. What. should I do next? Format it again and then install Docker etc. from scratch?

 

 

Please post the complete diagnostics, the earlier syslog snippet is irrelevant for this and looks like the device dropped offline, and that is missing.

4 minutes ago, JorgeB said:

Please post the complete diagnostics, the earlier syslog snippet is irrelevant for this and looks like the device dropped offline, and that is missing.

Thanks for the swift reply

 

Here you go:

stormy-diagnostics-20211115-1432.zip

Diags are after rebooting but the btrfs dev stats do suggest that one of the devices dropped offline before, this is not necessarily a device problem, you can redo the pool and if it happens again post new diags before rebooting.

1 hour ago, JorgeB said:

Diags are after rebooting but the btrfs dev stats do suggest that one of the devices dropped offline before, this is not necessarily a device problem, you can redo the pool and if it happens again post new diags before rebooting.

I just did that and it seems fine for now. No mention of a disconnected device! Thanks for the tip  

Any idea how this can be? Did I mess up the config or something? Or could it still be a bad cable? I just dont want it happen again.

Edited November 15, 20214 yr by Sledgehamma