Docker 403 errors, cache disk previously disappeared

[Adam Kesher]

hi everyone

 

so i had nvme related issues where my cache drive went missing. it's back, but now docker programs are failing and in cli i tried this

 

system~# cp /boot/bzimage /mnt/cache
cp: cannot create regular file '/mnt/cache/bzimage': Read-only file system

 

Not sure what to do. I assume it's more a docker issue now. Diags included here. Thanks in advance!

zigplex2-diagnostics-20220301-2126.zip

[ChatNoir]

4 hours ago, Adam Kesher said:

so i had nvme related issues where my cache drive went missing. it's back,

There are still issues with your NVME, as well as your docker image. But the second is probably caused by the first.

 

Mar  1 18:17:22 Zigplex2 kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 8017 rw 0,0 sector 0x27324c18 len 4096 err no 10
Mar  1 18:17:22 Zigplex2 kernel: blk_update_request: I/O error, dev loop2, sector 2149120 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Mar  1 18:17:22 Zigplex2 kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 8017 rw 0,0 sector 0x27324c18 len 4096 err no 10
Mar  1 18:17:22 Zigplex2 kernel: blk_update_request: I/O error, dev loop2, sector 2149120 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Mar  1 18:17:22 Zigplex2 kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 8017 rw 0,0 sector 0x27324c18 len 4096 err no 10
Mar  1 18:17:22 Zigplex2 kernel: blk_update_request: I/O error, dev loop2, sector 2149120 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Mar  1 18:17:22 Zigplex2 kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 8017 rw 0,0 sector 0x2731f728 len 4096 err no 10
Mar  1 18:17:22 Zigplex2 kernel: blk_update_request: I/O error, dev loop2, sector 1650232 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Mar  1 18:17:22 Zigplex2 kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 8017 rw 0,0 sector 0x2731f728 len 4096 err no 10
### [PREVIOUS LINE REPEATED 1 TIMES] ###
Mar  1 18:17:22 Zigplex2 kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 8017 rw 0,0 sector 0x27324c18 len 4096 err no 10
### [PREVIOUS LINE REPEATED 5 TIMES] ###
Mar  1 18:17:22 Zigplex2 kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 8017 rw 0,0 sector 0x2731f728 len 4096 err no 10
### [PREVIOUS LINE REPEATED 1 TIMES] ###
Mar  1 18:17:22 Zigplex2 kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 8017 rw 0,0 sector 0x27324c18 len 4096 err no 10
### [PREVIOUS LINE REPEATED 5 TIMES] ###
Mar  1 18:17:22 Zigplex2 kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 8017 rw 0,0 sector 0x2731f728 len 4096 err no 10
Mar  1 18:17:22 Zigplex2 kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 8017 rw 0,0 sector 0x27324c18 len 4096 err no 10
### [PREVIOUS LINE REPEATED 35 TIMES] ###
Mar  1 18:17:22 Zigplex2 kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 8017 rw 0,0 sector 0x27326018 len 4096 err no 10
### [PREVIOUS LINE REPEATED 2 TIMES] ###
Mar  1 18:17:22 Zigplex2 kernel: BTRFS warning (device nvme0n1p1): direct IO failed ino 8017 rw 0,0 sector 0x27324c18 len 4096 err no 10
### [PREVIOUS LINE REPEATED 17 TIMES] ###
Mar  1 18:19:09 Zigplex2 kernel: btrfs_dev_stat_print_on_error: 159 callbacks suppressed
Mar  1 18:19:09 Zigplex2 kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 15, rd 140, flush 0, corrupt 0, gen 0
Mar  1 18:19:09 Zigplex2 kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 15, rd 141, flush 0, corrupt 0, gen 0
Mar  1 18:19:09 Zigplex2 kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 15, rd 142, flush 0, corrupt 0, gen 0
Mar  1 18:19:09 Zigplex2 kernel: BTRFS error (device nvme0n1p1): bdev /dev/nvme0n1p1 errs: wr 15, rd 143, flush 0, corrupt 0, gen 0

 

Also, your pool 'cache' seems OFF. I am no specialist so I'll wait for JorgeB to chime in.

 

Pool: cache
Overall:
    Device size:		 931.51GiB
    Device allocated:		 475.05GiB
    Device unallocated:		 456.47GiB
    Device missing:		 931.51GiB
    Used:			 392.50GiB
    Free (estimated):		 537.54GiB	(min: 537.54GiB)
    Free (statfs, df):		 537.54GiB
    Data ratio:			      1.00
    Metadata ratio:		      1.00
    Global reserve:		 484.45MiB	(used: 32.00KiB)
    Multiple profiles:		        no

                  Data      Metadata System              
Id Path           single    single   single   Unallocated
-- -------------- --------- -------- -------- -----------
 1 /dev/nvme0n1p1 472.01GiB  3.01GiB 32.00MiB  -475.05GiB
-- -------------- --------- -------- -------- -----------
   Total          472.01GiB  3.01GiB 32.00MiB  -475.05GiB
   Used           390.93GiB  1.58GiB 80.00KiB    

 

[Adam Kesher]

maybe i should think about moving this to a regular sata3 internal SSD? i don't know how to migrate everything without losing everything and having it act as if it was the current cache. I'm super green on this. would that be advisable considering my issues? Jorge actually helped me deal with this cache disappearing but that issue occured again recently so maybe with this mobo i shouldn't have my cache on an nvme drive...

 

also should I not be using BTRFS for the cache file system?

Edited March 2, 2022 by Adam Kesher
Additional Q