EDIT: My dockers are now reporting: IOError: [Errno 28] No space left on device

 

Everything has been working fine for years, until it was. 

 

I tried to create a new folder on my server and was presented with an error about no free space. As there is 3.1Tb free (and the cache has 300Gb free) I decided to reboot.  After coming back up everything appear to be working, until I realised all my dockers failed to start, reporting errors about being read only. Initially, I was able to create a new folder and copy files, but, 1 minute later I'm getting errors about no free space again. 

 

Main log is current spitting out the following....

 

May 3 14:11:42 Tower kernel: print_req_error: I/O error, dev loop2, sector 128
May 3 14:11:42 Tower kernel: print_req_error: I/O error, dev loop2, sector 128
May 3 14:11:42 Tower kernel: BTRFS warning (device loop2): lost page write due to IO error on /dev/loop2
May 3 14:11:42 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 13, rd 0, flush 0, corrupt 0, gen 0
May 3 14:11:42 Tower kernel: BTRFS error (device loop2): error writing primary super block to device 1
May 3 14:11:42 Tower kernel: BTRFS: error (device loop2) in write_all_supers:3848: errno=-5 IO failure (1 errors while writing supers)
May 3 14:11:42 Tower kernel: BTRFS info (device loop2): forced readonly
May 3 14:11:42 Tower kernel: BTRFS: error (device loop2) in btrfs_sync_log:3177: errno=-5 IO failure
May 3 14:11:42 Tower kernel: BTRFS error (device loop2): pending csums is 8192
May 3 14:18:11 Tower dnsmasq-dhcp[15076]: DHCPREQUEST(virbr0) 192.168.122.21 52:54:00:ac:8b:30 
May 3 14:18:11 Tower dnsmasq-dhcp[15076]: DHCPACK(virbr0) 192.168.122.21 52:54:00:ac:8b:30 DESKTOP-PQQ8NRQ
May 3 14:19:28 Tower kernel: loop: Write error at byte offset 2535108608, length 4096.
May 3 14:19:28 Tower kernel: print_req_error: I/O error, dev loop2, sector 4951384
May 3 14:19:28 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 14, rd 0, flush 0, corrupt 0, gen 0
May 3 14:20:34 Tower kernel: loop: Write error at byte offset 2538713088, length 4096.
May 3 14:20:34 Tower kernel: print_req_error: I/O error, dev loop2, sector 4958424
May 3 14:20:34 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 15, rd 0, flush 0, corrupt 0, gen 0
May 3 14:20:35 Tower emhttpd: req (10): cmd=/webGui/scripts/disk_log&arg1=sdf&csrf_token=****************
May 3 14:20:35 Tower emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/disk_log sdf

May 3 14:27:08 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 18, rd 0, flush 0, corrupt 0, gen 0
May 3 14:30:44 Tower kernel: loop: Write error at byte offset 2538721280, length 4096.
May 3 14:30:44 Tower kernel: print_req_error: I/O error, dev loop2, sector 4958424
May 3 14:30:44 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 19, rd 0, flush 0, corrupt 0, gen 0
May 3 14:32:46 Tower kernel: loop: Write error at byte offset 2538717184, length 4096.
May 3 14:32:46 Tower kernel: print_req_error: I/O error, dev loop2, sector 4958424
May 3 14:32:46 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 20, rd 0, flush 0, corrupt 0, gen 0
May 3 14:34:39 Tower kernel: loop: Write error at byte offset 2520858624, length 4096.
May 3 14:34:39 Tower kernel: print_req_error: I/O error, dev loop2, sector 4923552
May 3 14:34:39 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 21, rd 0, flush 0, corrupt 0, gen 0
May 3 14:34:44 Tower kernel: loop: Write error at byte offset 2520858624, length 4096.
May 3 14:34:44 Tower kernel: print_req_error: I/O error, dev loop2, sector 4923552
May 3 14:34:44 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 22, rd 0, flush 0, corrupt 0, gen 0
May 3 14:34:49 Tower kernel: loop: Write error at byte offset 2520858624, length 4096.
May 3 14:34:49 Tower kernel: print_req_error: I/O error, dev loop2, sector 4923552
May 3 14:34:49 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 23, rd 0, flush 0, corrupt 0, gen 0

 

 

 

tower-syslog-20190503-1440.zip

tower-diagnostics-20190503-1714.zip

Edited May 3, 20197 yr by Fpsware

Cache filesystem is fully allocated, see here for fix:

https://lime-technology.com/forums/topic/62230-out-of-space-errors-on-cache-drive/?do=findComment&comment=610551

 

After that and there are still errors with the docker image delete and re-create it.