Hi all,

 

Been having issues with my PiHole docker that escalated to "Docker Service" failed to start errors. I'm not certain if this is limited to the docker image itself or whether it's indicative of something else, hence why I'm posting here first. I've restarted my unRAID box a few times and had things run fine only for things to recur; digging in the logs revealed the following:

 

 

Aug 25 17:18:14 Tower emhttpd: shcmd (1350): /usr/local/sbin/mount_image '/mnt/user/system/docker/docker.img' /var/lib/docker 20
Aug 25 17:18:14 Tower root: truncate: cannot open '/mnt/cache/system/docker/docker.img' for writing: Read-only file system
Aug 25 17:18:14 Tower kernel: BTRFS: device fsid a751297c-8ef2-44d2-b96c-1d34e347ff81 devid 1 transid 1598788 /dev/loop3
Aug 25 17:18:15 Tower kernel: BTRFS info (device loop3): disk space caching is enabled
Aug 25 17:18:15 Tower kernel: BTRFS info (device loop3): has skinny extents
Aug 25 17:18:15 Tower kernel: BTRFS info (device loop3): start tree-log replay
Aug 25 17:18:15 Tower kernel: BTRFS warning (device loop3): log replay required on RO media
Aug 25 17:18:15 Tower root: mount: /var/lib/docker: can't read superblock on /dev/loop3.
Aug 25 17:18:15 Tower kernel: BTRFS error (device loop3): open_ctree failed
Aug 25 17:18:15 Tower root: mount error
Aug 25 17:18:15 Tower emhttpd: shcmd (1350): exit status: 1

 

Is this a hardware failure or something else? I'm not seeing any errors at all on any of my drives or the boot flash drive. I'm not certain where to go from here. Appreciate any help, thank you!

Please attach diagnostics. Tools -> Diagnostics -> attach zip file to your next post.

Diagnostics attached.

tower-diagnostics-20200825-1840.zip

11 minutes ago, pcss said:

Diagnostics attached.

You already rebooted so there isn't any useful info in the syslog. Wait till next time you have issue and extract diagnostic.

 

Also your cache drive is very full. That can cause false out of space error and funky issues especially if you do a lot of write to cache.

Will do, thank you.

Can I manually flush the cache drive somehow beyond having TRIM enabled on it?

25 minutes ago, pcss said:

Will do, thank you.

Can I manually flush the cache drive somehow beyond having TRIM enabled on it?

You have a few shares with Cache = Yes so start with running the mover to clear out data from cache to array. 120GB is very small so I recommend you don't use Cache = Yes.

Then run trim from command line:

fstrim -a

 

An even more drastic measure if you have false out of space error is to run balance (Main -> Cache -> Balance). This will rewrite everything on your SSD so will take some time to run and add to your write cycles.

 

 

 

Hi, original issue recurred today. Diagnostics attached, have not rebooted. Suspected corrupted docker image, deleted it, but now cannot restart docker service.

tower-diagnostics-20200903-2111.zip

Problems with the cache device:

 

Sep  1 06:07:40 Tower kernel: ata1.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6
Sep  1 06:07:40 Tower kernel: ata1.01: failed command: WRITE MULTIPLE EXT
Sep  1 06:07:40 Tower kernel: ata1.01: cmd 39/00:d0:e0:ce:0e/00:02:00:00:00/f0 tag 0 pio 368640 out
Sep  1 06:07:40 Tower kernel:         res 51/84:01:00:00:00/84:00:00:00:00/f0 Emask 0x10 (ATA bus error)
Sep  1 06:07:40 Tower kernel: ata1.01: status: { DRDY ERR }
Sep  1 06:07:40 Tower kernel: ata1.01: error: { ICRC ABRT }
Sep  1 06:07:40 Tower kernel: ata1: soft resetting link
Sep  1 06:07:40 Tower kernel: ata1.01: configured for PIO0
Sep  1 06:07:40 Tower kernel: sd 1:0:1:0: [sdf] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Sep  1 06:07:40 Tower kernel: sd 1:0:1:0: [sdf] tag#0 Sense Key : 0xb [current]
Sep  1 06:07:40 Tower kernel: sd 1:0:1:0: [sdf] tag#0 ASC=0x47 ASCQ=0x0
Sep  1 06:07:40 Tower kernel: sd 1:0:1:0: [sdf] tag#0 CDB: opcode=0x2a 2a 00 00 0e ce e0 00 02 d0 00
Sep  1 06:07:40 Tower kernel: print_req_error: I/O error, dev sdf, sector 970464

 

It's set to IDE mode, set it to AHCI, also check connections.

Thanks, much appreciated!