I decided to reboot after noticing that my docker image was unreadable as well as /mnt/user/system and anything below it.

 

I decided to reboot and now my cache pool of two drives won't mount. Brining my array into maintenance mode yielded me this on the read-only filesystem check against my cache pool:
 

[1/7] checking root items [2/7] checking extents [3/7] checking free space tree free space info recorded 1351 extents, counted 1361 There are still entries left in the space cache cache appears valid but isn't 6182821691392 [4/7] checking fs roots [5/7] checking only csums items (without verifying data) [6/7] checking root refs [7/7] checking quota groups skipped (not enabled on this FS) Opening filesystem to check... Checking filesystem on /dev/sdj1 UUID: d5b29014-e26e-4407-a52d-73db030f39ef found 759680454656 bytes used, error(s) found total csum bytes: 236747020 total tree bytes: 1881374720 total fs tree bytes: 1275772928 total extent tree bytes: 314228736 btree space waste bytes: 386624833 file data blocks allocated: 7470265802752 referenced 690431000576

 

Should I run the check without the read-only parameter?

 

I will provide any diagnostics that are asked for, but I will not read megabytes of compressed files trying to censor them for privacy.

Please post the diagnostics.

10 hours ago, JorgeB said:

Please post the diagnostics.

I'll DM you the password for the file, because I really don't feel comfortable putting up info like this publicly.

 

https://drive.google.com/file/d/1Zw4NvfVu0YSxuaKYmTdsy01Xh85U45bR/view?usp=drivesdk

 

If you decide to help me out you have all my thanks and appreciation in advance already.

 

Cheers

Diags should be posted in the forum, they can be anonymized, in this case if you prefer you can post just the syslog part where it shows the cache mount attempt, include a few lines before and after to make sure nothing is missing, that should be enough, but if it's not I'm not gonna keep asking for a file at a time.

Alright, sounds like a good compromise, thank you so much for willing to help on those terms.

 

Here's a snippet of around the mount attempt. If you need more, like you suggested I'll merrily provide more.

 

Jun 20 22:43:20 Ahri kernel: scsi 1:0:6:0: Direct-Access     ATA      TOSHIBA MG08ACA1 0102 PQ: 0 ANSI: 6
Jun 20 22:43:20 Ahri kernel: scsi 1:0:6:0: SATA: handle(0x0010), sas_addr(0x5001438024fdabe7), phy(6), device_name(0x5000039af8d2f414)
Jun 20 22:43:20 Ahri kernel: sd 1:0:0:0: [sdb] Attached SCSI disk
Jun 20 22:43:20 Ahri kernel: scsi 1:0:6:0: enclosure logical id (0x5001438024fdabe0), slot(7)
Jun 20 22:43:20 Ahri kernel: scsi 1:0:6:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Jun 20 22:43:20 Ahri kernel: scsi 1:0:6:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Jun 20 22:43:20 Ahri kernel: sdc: sdc1
Jun 20 22:43:20 Ahri kernel: sd 1:0:1:0: [sdc] Attached SCSI disk
Jun 20 22:43:20 Ahri kernel: sdf: sdf1
Jun 20 22:43:20 Ahri kernel: sd 1:0:4:0: [sdf] Attached SCSI disk
Jun 20 22:43:20 Ahri kernel: sde: sde1
Jun 20 22:43:20 Ahri kernel: sd 1:0:3:0: [sde] Attached SCSI disk
Jun 20 22:43:20 Ahri kernel: sdd: sdd1
Jun 20 22:43:20 Ahri kernel: sd 1:0:2:0: [sdd] Attached SCSI disk
Jun 20 22:43:20 Ahri kernel: sdg: sdg1
Jun 20 22:43:20 Ahri kernel: sd 1:0:5:0: [sdg] Attached SCSI disk
Jun 20 22:43:20 Ahri kernel: sd 1:0:6:0: Attached scsi generic sg7 type 0
Jun 20 22:43:20 Ahri kernel: end_device-1:0:6: add: handle(0x0010), sas_addr(0x5001438024fdabe7)
Jun 20 22:43:20 Ahri kernel: sd 1:0:6:0: [sdh] 27344764928 512-byte logical blocks: (14.0 TB/12.7 TiB)
Jun 20 22:43:20 Ahri kernel: sd 1:0:6:0: [sdh] 4096-byte physical blocks
Jun 20 22:43:20 Ahri kernel: scsi 1:0:7:0: Direct-Access     ATA      TOSHIBA MG08ACA1 0102 PQ: 0 ANSI: 6
Jun 20 22:43:20 Ahri kernel: scsi 1:0:7:0: SATA: handle(0x0011), sas_addr(0x5001438024fdabe8), phy(7), device_name(0x5000039af8d2e434)
Jun 20 22:43:20 Ahri kernel: scsi 1:0:7:0: enclosure logical id (0x5001438024fdabe0), slot(8)
Jun 20 22:43:20 Ahri kernel: scsi 1:0:7:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Jun 20 22:43:20 Ahri kernel: scsi 1:0:7:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Jun 20 22:43:20 Ahri kernel: sd 1:0:6:0: [sdh] Write Protect is off
Jun 20 22:43:20 Ahri kernel: sd 1:0:6:0: [sdh] Mode Sense: 7f 00 00 08
Jun 20 22:43:20 Ahri kernel: sd 1:0:6:0: [sdh] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Jun 20 22:43:20 Ahri kernel: sdh: sdh1
Jun 20 22:43:20 Ahri kernel: sd 1:0:6:0: [sdh] Attached SCSI disk
Jun 20 22:43:20 Ahri kernel: sd 1:0:7:0: Attached scsi generic sg8 type 0
Jun 20 22:43:20 Ahri kernel: end_device-1:0:7: add: handle(0x0011), sas_addr(0x5001438024fdabe8)
Jun 20 22:43:20 Ahri kernel: scsi 1:0:8:0: Direct-Access     ATA      Samsung SSD 860  4B6Q PQ: 0 ANSI: 6
Jun 20 22:43:20 Ahri kernel: scsi 1:0:8:0: SATA: handle(0x0012), sas_addr(0x5001438024fdabed), phy(12), device_name(0x5002538e90804b01)
Jun 20 22:43:20 Ahri kernel: scsi 1:0:8:0: enclosure logical id (0x5001438024fdabe0), slot(13)
Jun 20 22:43:20 Ahri kernel: scsi 1:0:8:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Jun 20 22:43:20 Ahri kernel: scsi 1:0:8:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Jun 20 22:43:20 Ahri kernel: sd 1:0:7:0: [sdi] 27344764928 512-byte logical blocks: (14.0 TB/12.7 TiB)
Jun 20 22:43:20 Ahri kernel: sd 1:0:7:0: [sdi] 4096-byte physical blocks
Jun 20 22:43:20 Ahri kernel: sd 1:0:8:0: Attached scsi generic sg9 type 0
Jun 20 22:43:20 Ahri kernel: end_device-1:0:8: add: handle(0x0012), sas_addr(0x5001438024fdabed)
Jun 20 22:43:20 Ahri kernel: sd 1:0:8:0: [sdj] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
Jun 20 22:43:20 Ahri kernel: sd 1:0:7:0: [sdi] Write Protect is off
Jun 20 22:43:20 Ahri kernel: scsi 1:0:9:0: Direct-Access     ATA      Samsung SSD 860  4B6Q PQ: 0 ANSI: 6
Jun 20 22:43:20 Ahri kernel: scsi 1:0:9:0: SATA: handle(0x0013), sas_addr(0x5001438024fdabee), phy(13), device_name(0x5002538e702b1ef3)
Jun 20 22:43:20 Ahri kernel: scsi 1:0:9:0: enclosure logical id (0x5001438024fdabe0), slot(14)
Jun 20 22:43:20 Ahri kernel: scsi 1:0:9:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
Jun 20 22:43:20 Ahri kernel: scsi 1:0:9:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
Jun 20 22:43:20 Ahri kernel: sd 1:0:9:0: Attached scsi generic sg10 type 0
Jun 20 22:43:20 Ahri kernel: end_device-1:0:9: add: handle(0x0013), sas_addr(0x5001438024fdabee)
Jun 20 22:43:20 Ahri kernel: scsi 1:0:10:0: Enclosure         HP       Gen8 ServBP 12+2 3.30 PQ: 0 ANSI: 5
Jun 20 22:43:20 Ahri kernel: sd 1:0:7:0: [sdi] Mode Sense: 7f 00 00 08
Jun 20 22:43:20 Ahri kernel: sd 1:0:9:0: [sdk] 1953525168 512-byte logical blocks: (1.00 TB/932 GiB)
Jun 20 22:43:20 Ahri kernel: scsi 1:0:10:0: set ignore_delay_remove for handle(0x0014)
Jun 20 22:43:20 Ahri kernel: scsi 1:0:10:0: SES: handle(0x0014), sas_addr(0x5001438024fdabf9), phy(24), device_name(0x0000000000000000)
Jun 20 22:43:20 Ahri kernel: scsi 1:0:10:0: enclosure logical id (0x5001438024fdabe0), slot(0)
Jun 20 22:43:20 Ahri kernel: scsi 1:0:10:0: qdepth(1), tagged(0), scsi_level(6), cmd_que(0)
Jun 20 22:43:20 Ahri kernel: sd 1:0:8:0: [sdj] Write Protect is off
Jun 20 22:43:20 Ahri kernel: sd 1:0:8:0: [sdj] Mode Sense: 7f 00 00 08
Jun 20 22:43:20 Ahri kernel: sd 1:0:9:0: [sdk] Write Protect is off
Jun 20 22:43:20 Ahri kernel: sd 1:0:9:0: [sdk] Mode Sense: 7f 00 00 08
Jun 20 22:43:20 Ahri kernel: sd 1:0:8:0: [sdj] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Jun 20 22:43:20 Ahri kernel: sd 1:0:9:0: [sdk] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Jun 20 22:43:20 Ahri kernel: scsi 1:0:10:0: Attached scsi generic sg11 type 13
Jun 20 22:43:20 Ahri kernel: end_device-1:0:10: add: handle(0x0014), sas_addr(0x5001438024fdabf9)
Jun 20 22:43:20 Ahri kernel: sdj: sdj1
Jun 20 22:43:20 Ahri kernel: sdk: sdk1
Jun 20 22:43:20 Ahri kernel: sd 1:0:8:0: [sdj] Attached SCSI disk
Jun 20 22:43:20 Ahri kernel: sd 1:0:9:0: [sdk] Attached SCSI disk
Jun 20 22:43:20 Ahri kernel: sd 1:0:7:0: [sdi] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Jun 20 22:43:20 Ahri kernel: sdi: sdi1
Jun 20 22:43:20 Ahri kernel: sd 1:0:7:0: [sdi] Attached SCSI disk
Jun 20 22:43:20 Ahri kernel: ata2: SATA link down (SStatus 0 SControl 300)
Jun 20 22:43:20 Ahri kernel: ata3: SATA link down (SStatus 0 SControl 300)
Jun 20 22:43:20 Ahri kernel: ata4: SATA link down (SStatus 0 SControl 300)
Jun 20 22:43:20 Ahri kernel: ata5: SATA link down (SStatus 0 SControl 300)
Jun 20 22:43:20 Ahri kernel: ata6: SATA link down (SStatus 0 SControl 300)
Jun 20 22:43:20 Ahri kernel: mgag200 0000:01:00.1: vgaarb: deactivate vga console
Jun 20 22:43:20 Ahri kernel: Console: switching to colour dummy device 80x25
Jun 20 22:43:20 Ahri kernel: [drm] Initialized mgag200 1.0.0 20110418 for 0000:01:00.1 on minor 0
Jun 20 22:43:20 Ahri kernel: fbcon: mgag200drmfb (fb0) is primary device
Jun 20 22:43:20 Ahri kernel: mgag200 0000:01:00.1: [drm] drm_plane_enable_fb_damage_clips() not called
Jun 20 22:43:20 Ahri kernel: BTRFS: device fsid b8c88925-d8be-4ab9-8ef8-17484fdafe91 devid 1 transid 4309837 /dev/sdd1 scanned by udevd (1083)
Jun 20 22:43:20 Ahri kernel: BTRFS: device fsid 878890c5-81f9-4f8f-8f69-818e2af4175e devid 1 transid 360242 /dev/sdf1 scanned by udevd (1057)
Jun 20 22:43:20 Ahri kernel: BTRFS: device fsid c2923828-e4c3-483e-bc45-ba3fe7ef2b6a devid 1 transid 5985837 /dev/sdb1 scanned by udevd (1038)
Jun 20 22:43:20 Ahri kernel: BTRFS: device fsid 943d58d4-9214-47a8-8d61-01272b03f9ba devid 1 transid 368310 /dev/sdg1 scanned by udevd (1083)
Jun 20 22:43:20 Ahri kernel: BTRFS: device fsid 71dab375-beb9-4327-bea1-6a7ac48dcd5f devid 1 transid 133339 /dev/sde1 scanned by udevd (1038)
Jun 20 22:43:20 Ahri kernel: BTRFS: device fsid 979bf416-df16-4476-91f0-ea4e6c4084b2 devid 1 transid 1796375 /dev/sdc1 scanned by udevd (1057)
Jun 20 22:43:20 Ahri kernel: BTRFS: device fsid d5b29014-e26e-4407-a52d-73db030f39ef devid 2 transid 44263972 /dev/sdk1 scanned by udevd (1057)
Jun 20 22:43:20 Ahri kernel: Console: switching to colour frame buffer device 128x48
Jun 20 22:43:20 Ahri kernel: BTRFS: device fsid d5b29014-e26e-4407-a52d-73db030f39ef devid 1 transid 44263972 /dev/sdj1 scanned by udevd (1083)
Jun 20 22:43:20 Ahri kernel: mgag200 0000:01:00.1: [drm] fb0: mgag200drmfb frame buffer device
Jun 20 22:43:20 Ahri kernel: igb 0000:02:00.3: removed PHC on eth3
Jun 20 22:43:20 Ahri kernel: igb 0000:02:00.2: removed PHC on eth2
Jun 20 22:43:20 Ahri kernel: igb 0000:02:00.1: removed PHC on eth1
Jun 20 22:43:20 Ahri kernel: igb 0000:02:00.0: removed PHC on eth0
Jun 20 22:43:20 Ahri kernel: igb: Intel(R) Gigabit Ethernet Network Driver
Jun 20 22:43:20 Ahri kernel: igb: Copyright (c) 2007-2014 Intel Corporation.
Jun 20 22:43:20 Ahri kernel: igb 0000:02:00.0: added PHC on eth0
Jun 20 22:43:20 Ahri kernel: igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection

 

That's not what I needed, I used your link but reposting here for reference:

 

Jun 20 22:45:27 Ahri  emhttpd: /mnt/cache found: 2
Jun 20 22:45:27 Ahri  emhttpd: /mnt/cache extra: 0
Jun 20 22:45:27 Ahri  emhttpd: /mnt/cache missing: 0
Jun 20 22:45:27 Ahri  emhttpd: shcmd (60): mount -t btrfs -o noatime,space_cache=v2,discard=async -U d5b29014-e26e-4407-a52d-73db030f39ef /mnt/cache
Jun 20 22:45:27 Ahri kernel: BTRFS info (device sdj1): turning on async discard
Jun 20 22:45:27 Ahri kernel: BTRFS info (device sdj1): using free space tree
Jun 20 22:45:27 Ahri kernel: BTRFS info (device sdj1): has skinny extents
Jun 20 22:45:28 Ahri kernel: BTRFS info (device sdj1): enabling ssd optimizations
Jun 20 22:45:28 Ahri kernel: BTRFS info (device sdj1): start tree-log replay
Jun 20 22:45:28 Ahri kernel: BTRFS error (device sdj1): incorrect extent count for 6182821691392; counted 1361, expected 1351
Jun 20 22:45:28 Ahri kernel: BTRFS: error (device sdj1) in btrfs_replay_log:2500: errno=-5 IO failure (Failed to recover log tree)
Jun 20 22:45:28 Ahri kernel: BTRFS error (device sdj1: state E): open_ctree failed
Jun 20 22:45:28 Ahri root: mount: /mnt/cache: can't read superblock on /dev/sdk1.
Jun 20 22:45:28 Ahri root:        dmesg(1) may have more information after failed mount system call.
Jun 20 22:45:28 Ahri  emhttpd: shcmd (60): exit status: 32
Jun 20 22:45:28 Ahri  emhttpd: /mnt/cache mount error: No file system
Jun 20 22:45:28 Ahri  emhttpd: shcmd (61): umount /mnt/cache
Jun 20 22:45:28 Ahri root: umount: /mnt/cache: not mounted.

 

If the log tree is the main issue this may help:

btrfs rescue zero-log /dev/sdj1

Then re-start array and see if it mounts.

Array started, I feel alive again! Here's to hoping every service gets back up flawlessly, but since it mounted without any hiccup right away after that command I'm feeling optimistic.

 

Thank you so very much, you didn't just make my day, you made my week. Truly thank you from the bottom of my heart, @JorgeB!

Edited June 23, 20233 yr by Glassed Silver