Cache disk ran full - Docker and VM refuse to start

[dafuzz]

Hello

My cache disk seemingly ran full and then the VM and the Docker stopped working 

I also do not seem to beable to browse /mnt/cache - i get this error when trying to browse the directory through the terminal "/bin/ls: reading directory '.': Input/output error"

 

Attached is the diagnostics, any advice on getting things running again would be appreciated 

tower-diagnostics-20230415-1750.zip

[dafuzz]

Any ideas guys? im dead in the water, i cant even seem to access my shares at the moment

[itimpi]

Not an answer to your current problem, but you should set the Minimum Free Space setting for the pool to avoid it filling up completely which tends to make it misbehave (especially when using btrfs).

 

I 'think' you could get past the current problem by simply unassigning the pool (for the time being) and starting the array but you may want to wait for someone else to chime in (e.g. @JorgeB) to see what they think.  You would then need to take action to try and recover the pool contents and I am not sure what is the best way to go about this.

[JorgeB]

Cache is going read only, first thing you need to try and fix the hardware problems:

 

Apr 15 17:39:47 Tower kernel: ata5.00: status: { DRDY ERR }
Apr 15 17:39:47 Tower kernel: ata5.00: error: { ABRT }
Apr 15 17:39:47 Tower kernel: ata5.00: configured for UDMA/133 (device error ignored)
Apr 15 17:39:47 Tower kernel: sd 6:0:0:0: [sdc] tag#10 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=30s
Apr 15 17:39:47 Tower kernel: sd 6:0:0:0: [sdc] tag#10 Sense Key : 0x5 [current]
Apr 15 17:39:47 Tower kernel: sd 6:0:0:0: [sdc] tag#10 ASC=0x21 ASCQ=0x4
Apr 15 17:39:47 Tower kernel: sd 6:0:0:0: [sdc] tag#10 CDB: opcode=0x2a 2a 00 00 00 2a a0 00 01 40 00
Apr 15 17:39:47 Tower kernel: I/O error, dev sdc, sector 10912 op 0x1:(WRITE) flags 0x1800 phys_seg 24 prio class 0
Apr 15 17:39:47 Tower kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 4, rd 0, flush 0, corrupt 2, gen 0
Apr 15 17:39:47 Tower kernel: BTRFS: error (device sdc1) in btrfs_commit_transaction:2418: errno=-5 IO failure (Error while writing out transaction)
Apr 15 17:39:47 Tower kernel: ata5: EH complete
Apr 15 17:39:47 Tower kernel: BTRFS info (device sdc1: state E): forced readonly

 

Try replacing cables to rule that out.

 

[dafuzz]

thanks, its an m.2 drive. I re-seated it and tried again but those errors persist
im guessing the drive has failed - Will have to look into to replacing it during the week

 

[dafuzz]

Hey guys, thanks for the help with this

 

I would like to possibly run my system without the cache until my replacement disk arrives during the week 

I have backups of the appdata that was on the lost drive

 

Would it be possible to to run the system without the cache until the new drive arrives? What should i do to achieve this. if theres an article i can follow or any assistance would be greatly appreciated here. Thanks

[JorgeB]

Just start the array without cache assigned, obviously any services that were on cache won't work.

[dafuzz]

Can i restore the appdata and get my dockers working?

I assume that when the cache is replaced it will move it there?

[JorgeB]

You can restore the appdata to the array, then recreate the docker image, if docker paths are set to /mnt/user no need to adjust anything else.