Server becoming read-only after a while

[XDD128]

Hey y'all, I've been trying to fix my server for a while and just when I thought I've fixed everything, the same problem happened again. I have Nextcloud and plex running most of the time, and after starting the server for maybe a day, I start getting these btrfs loop2 errors in my log:

 

Sep 3 01:11:01 Tower kernel: BTRFS info (device nvme0n1p1): forced readonly
Sep 3 01:11:04 Tower kernel: loop: Write error at byte offset 7234027520, length 4096.
Sep 3 01:11:04 Tower kernel: print_req_error: I/O error, dev loop2, sector 14128960
Sep 3 01:11:04 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
Sep 3 01:11:04 Tower kernel: loop: Write error at byte offset 7234043904, length 4096.
Sep 3 01:11:04 Tower kernel: print_req_error: I/O error, dev loop2, sector 14128992
Sep 3 01:11:04 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
Sep 3 01:11:04 Tower kernel: loop: Write error at byte offset 7502462976, length 4096.
Sep 3 01:11:04 Tower kernel: print_req_error: I/O error, dev loop2, sector 14653248
Sep 3 01:11:04 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0
Sep 3 01:11:04 Tower kernel: loop: Write error at byte offset 7502479360, length 4096.
Sep 3 01:11:04 Tower kernel: print_req_error: I/O error, dev loop2, sector 14653280
Sep 3 01:11:04 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 4, rd 0, flush 0, corrupt 0, gen 0
Sep 3 01:11:04 Tower kernel: loop: Write error at byte offset 4576100352, length 4096.
Sep 3 01:11:04 Tower kernel: print_req_error: I/O error, dev loop2, sector 8937696
Sep 3 01:11:04 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 5, rd 0, flush 0, corrupt 0, gen 0
Sep 3 01:11:04 Tower kernel: loop: Write error at byte offset 4575133696, length 4096.
Sep 3 01:11:04 Tower kernel: print_req_error: I/O error, dev loop2, sector 8935808
Sep 3 01:11:04 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 6, rd 0, flush 0, corrupt 0, gen 0
Sep 3 01:11:04 Tower kernel: loop: Write error at byte offset 4576100352, length 4096.
Sep 3 01:11:04 Tower kernel: print_req_error: I/O error, dev loop2, sector 8937696
Sep 3 01:11:04 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 7, rd 0, flush 0, corrupt 0, gen 0
Sep 3 01:11:04 Tower kernel: loop: Write error at byte offset 4575133696, length 4096.
Sep 3 01:11:04 Tower kernel: print_req_error: I/O error, dev loop2, sector 8935808
Sep 3 01:11:04 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 8, rd 0, flush 0, corrupt 0, gen 0
Sep 3 01:11:04 Tower kernel: loop: Write error at byte offset 4575133696, length 4096.
Sep 3 01:11:04 Tower kernel: print_req_error: I/O error, dev loop2, sector 8935808
Sep 3 01:11:04 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 9, rd 0, flush 0, corrupt 0, gen 0
Sep 3 01:11:04 Tower kernel: loop: Write error at byte offset 4575264768, length 4096.
Sep 3 01:11:04 Tower kernel: print_req_error: I/O error, dev loop2, sector 8936064
Sep 3 01:11:04 Tower kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 10, rd 0, flush 0, corrupt 0, gen 0
Sep 3 01:11:04 Tower kernel: BTRFS: error (device loop2) in btrfs_commit_transaction:2267: errno=-5 IO failure (Error while writing out transaction)
Sep 3 01:11:04 Tower kernel: BTRFS info (device loop2): forced readonly
Sep 3 01:11:04 Tower kernel: BTRFS warning (device loop2): Skipping commit of aborted transaction.
Sep 3 01:11:04 Tower kernel: BTRFS: error (device loop2) in cleanup_transaction:1860: errno=-5 IO failure
Sep 3 01:11:04 Tower kernel: BTRFS info (device loop2): delayed_refs has NO entry
Sep 3 03:57:29 Tower crond[2005]: exit status 1 from user root /usr/local/sbin/mover &> /dev/null
Sep 3 12:57:31 Tower kernel: logitech-hidpp-device 0003:046D:404D.0004: HID++ 4.1 device connected.

and then i can't write anything to the server at all. I'm trying to figure out if this is a hardware issue or something I can fix in software before I buy another cache drive. Im running 4x WD Red 8TB in my array and a Sabrent 1TB SSD for my cache by the way.

[trurl]

Go to Tools - Diagnostics and attach the complete Diagnostics ZIP file to your NEXT post in this thread.

[XDD128]

Here you go

tower-diagnostics-20200903-1609.zip

[Mistershiverz]

Hi @XDD128 sorry to jump in on your thread but i have been getting the exact same issues myself for the last week it seems. Has this also recently started happening for you also?

 

 

[trurl]

docker.img corrupt. You will have to delete and recreate it. Previous Apps on the Apps page will reinstall your dockers just as they were.

 

Some of your appdata is on the array even though cache isn't full. Have you had problems filling cache? While you have docker disabled you can run mover to get the rest of appdata moved back to cache. Mover can't move open files so you will have to do that before installing your dockers again.

 

 

[trurl]

5 minutes ago, Mistershiverz said:

sorry to jump in on your thread

I have replied on your thread.

[XDD128]

I just tried what you said and again, a little after I start writing files in nextcloud, it gives me a 503 server error because it can't write to the config, and the whole thing becomes read only again. Might this have something to do with the cache preferences for my shares?

[trurl]

post new diagnostics

[JorgeB]

There's also corruption in the cache filesystem, backup any data on cache then re-format, restore data and recreate docker image again.

 

P.S. Ryzen with overclocked RAM is known to corrupt data in some cases, btrfs is very sensitive to RAM errors, see here for more info.