BTRFS Errors


Recommended Posts

Slowly this has been getting worse, and it seems like I may have to scrub my cache pool. Can anybody help me with the follow errors. I can seem to do anything in relation to my dockers and they are not responding?

 

Mar 30 14:06:53 eMu-Unraid kernel: loop: Write error at byte offset 844144640, length 4096.
Mar 30 14:06:53 eMu-Unraid kernel: print_req_error: I/O error, dev loop2, sector 1648720
Mar 30 14:06:53 eMu-Unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 152, rd 0, flush 0, corrupt 0, gen 0
Mar 30 14:07:21 eMu-Unraid kernel: resource sanity check: requesting [mem 0x000c0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000c0000-0x000dffff window]
Mar 30 14:07:21 eMu-Unraid kernel: caller _nv000908rm+0x1bf/0x1f0 [nvidia] mapping multiple BARs
Mar 30 14:07:23 eMu-Unraid root: Fix Common Problems: Error: Unable to write to cache
Mar 30 14:07:23 eMu-Unraid root: Fix Common Problems: Error: Unable to write to Docker Image
Mar 30 14:07:23 eMu-Unraid kernel: loop: Write error at byte offset 844144640, length 4096.
Mar 30 14:07:23 eMu-Unraid kernel: print_req_error: I/O error, dev loop2, sector 1648720
Mar 30 14:07:23 eMu-Unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 153, rd 0, flush 0, corrupt 0, gen 0
Mar 30 14:07:24 eMu-Unraid kernel: resource sanity check: requesting [mem 0x000c0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000c0000-0x000dffff window]
Mar 30 14:07:24 eMu-Unraid kernel: caller _nv000908rm+0x1bf/0x1f0 [nvidia] mapping multiple BARs
Mar 30 14:07:52 eMu-Unraid kernel: sd 2:1:13:0: [sdf] tag#638 UNKNOWN(0x2003) Result: hostbyte=0x05 driverbyte=0x00
Mar 30 14:07:52 eMu-Unraid kernel: sd 2:1:13:0: [sdf] tag#638 CDB: opcode=0x28 28 00 74 7a 31 20 00 00 08 00
Mar 30 14:07:52 eMu-Unraid kernel: print_req_error: I/O error, dev sdf, sector 1954165024
Mar 30 14:07:52 eMu-Unraid kernel: md: disk5 read error, sector=1954164960
Mar 30 14:07:54 eMu-Unraid kernel: loop: Write error at byte offset 844144640, length 4096.
Mar 30 14:07:54 eMu-Unraid kernel: print_req_error: I/O error, dev loop2, sector 1648720
Mar 30 14:07:54 eMu-Unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 154, rd 0, flush 0, corrupt 0, gen 0
Mar 30 14:08:01 eMu-Unraid sSMTP[16065]: Creating SSL connection to host
Mar 30 14:08:01 eMu-Unraid sSMTP[16065]: SSL connection using TLS_AES_256_GCM_SHA384
Mar 30 14:08:04 eMu-Unraid sSMTP[16065]: Sent mail for xxxxxxxxxx (221 2.0.0 closing connection p38sm12204726qtf.50 - gsmtp) uid=0 username=root outbytes=785
Mar 30 14:08:14 eMu-Unraid kernel: mdcmd (44): spindown 5
Mar 30 14:08:15 eMu-Unraid root: Fix Common Problems: Warning: Write Cache is disabled on disk5
Mar 30 14:08:25 eMu-Unraid kernel: loop: Write error at byte offset 844144640, length 4096.
Mar 30 14:08:25 eMu-Unraid kernel: print_req_error: I/O error, dev loop2, sector 1648720
Mar 30 14:08:25 eMu-Unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 155, rd 0, flush 0, corrupt 0, gen 0
Mar 30 14:08:53 eMu-Unraid kernel: sd 2:1:13:0: [sdf] tag#196 UNKNOWN(0x2003) Result: hostbyte=0x05 driverbyte=0x00
Mar 30 14:08:53 eMu-Unraid kernel: sd 2:1:13:0: [sdf] tag#196 CDB: opcode=0x2a 2a 00 74 7a 31 20 00 00 08 00
Mar 30 14:08:53 eMu-Unraid kernel: print_req_error: I/O error, dev sdf, sector 1954165024
Mar 30 14:08:53 eMu-Unraid kernel: md: disk5 write error, sector=1954164960
Mar 30 14:08:56 eMu-Unraid kernel: loop: Write error at byte offset 844144640, length 4096.
Mar 30 14:08:56 eMu-Unraid kernel: print_req_error: I/O error, dev loop2, sector 1648720
Mar 30 14:08:56 eMu-Unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 156, rd 0, flush 0, corrupt 0, gen 0
Mar 30 14:09:04 eMu-Unraid sSMTP[16687]: Creating SSL connection to host
Mar 30 14:09:04 eMu-Unraid sSMTP[16687]: SSL connection using TLS_AES_256_GCM_SHA384
Mar 30 14:09:06 eMu-Unraid sSMTP[16687]: Sent mail for [email protected] (221 2.0.0 closing connection h11sm11384241qta.44 - gsmtp) uid=0 username=root outbytes=756
Mar 30 14:09:26 eMu-Unraid kernel: loop: Write error at byte offset 844144640, length 4096.
Mar 30 14:09:26 eMu-Unraid kernel: print_req_error: I/O error, dev loop2, sector 1648720
Mar 30 14:09:26 eMu-Unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 157, rd 0, flush 0, corrupt 0, gen 0
Mar 30 14:09:57 eMu-Unraid kernel: loop: Write error at byte offset 844144640, length 4096.
Mar 30 14:09:57 eMu-Unraid kernel: print_req_error: I/O error, dev loop2, sector 1648720
Mar 30 14:09:57 eMu-Unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 158, rd 0, flush 0, corrupt 0, gen 0
Mar 30 14:10:28 eMu-Unraid kernel: loop: Write error at byte offset 3260116992, length 4096.
Mar 30 14:10:28 eMu-Unraid kernel: print_req_error: I/O error, dev loop2, sector 6367416
Mar 30 14:10:28 eMu-Unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 159, rd 0, flush 0, corrupt 0, gen 0
Mar 30 14:10:58 eMu-Unraid kernel: loop: Write error at byte offset 844144640, length 4096.
Mar 30 14:10:58 eMu-Unraid kernel: print_req_error: I/O error, dev loop2, sector 1648720
Mar 30 14:10:58 eMu-Unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 160, rd 0, flush 0, corrupt 0, gen 0
Mar 30 14:11:12 eMu-Unraid kernel: mdcmd (45): spindown 5
Mar 30 14:11:29 eMu-Unraid kernel: loop: Write error at byte offset 844144640, length 4096.
Mar 30 14:11:29 eMu-Unraid kernel: print_req_error: I/O error, dev loop2, sector 1648720
Mar 30 14:11:29 eMu-Unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 161, rd 0, flush 0, corrupt 0, gen 0
Mar 30 14:12:00 eMu-Unraid kernel: loop: Write error at byte offset 844144640, length 4096.
Mar 30 14:12:00 eMu-Unraid kernel: print_req_error: I/O error, dev loop2, sector 1648720
Mar 30 14:12:00 eMu-Unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 162, rd 0, flush 0, corrupt 0, gen 0
Mar 30 14:12:31 eMu-Unraid kernel: loop: Write error at byte offset 844144640, length 4096.
Mar 30 14:12:31 eMu-Unraid kernel: print_req_error: I/O error, dev loop2, sector 1648720
Mar 30 14:12:31 eMu-Unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 163, rd 0, flush 0, corrupt 0, gen 0
Mar 30 14:12:32 eMu-Unraid kernel: resource sanity check: requesting [mem 0x000c0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000c0000-0x000dffff window]
Mar 30 14:12:32 eMu-Unraid kernel: caller _nv000908rm+0x1bf/0x1f0 [nvidia] mapping multiple BARs
Mar 30 14:12:34 eMu-Unraid kernel: resource sanity check: requesting [mem 0x000c0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000c0000-0x000dffff window]
Mar 30 14:12:34 eMu-Unraid kernel: caller _nv000908rm+0x1bf/0x1f0 [nvidia] mapping multiple BARs
Mar 30 14:12:37 eMu-Unraid kernel: resource sanity check: requesting [mem 0x000c0000-0x000fffff], which spans more than PCI Bus 0000:00 [mem 0x000c0000-0x000dffff window]
Mar 30 14:12:37 eMu-Unraid kernel: caller _nv000908rm+0x1bf/0x1f0 [nvidia] mapping multiple BARs
Mar 30 14:13:01 eMu-Unraid kernel: loop: Write error at byte offset 844144640, length 4096.
Mar 30 14:13:01 eMu-Unraid kernel: print_req_error: I/O error, dev loop2, sector 1648720
Mar 30 14:13:01 eMu-Unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 164, rd 0, flush 0, corrupt 0, gen 0
Mar 30 14:13:32 eMu-Unraid kernel: loop: Write error at byte offset 844144640, length 4096.
Mar 30 14:13:32 eMu-Unraid kernel: print_req_error: I/O error, dev loop2, sector 1648720
Mar 30 14:13:32 eMu-Unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 165, rd 0, flush 0, corrupt 0, gen 0
Mar 30 14:14:03 eMu-Unraid kernel: loop: Write error at byte offset 844144640, length 4096.
Mar 30 14:14:03 eMu-Unraid kernel: print_req_error: I/O error, dev loop2, sector 1648720
Mar 30 14:14:03 eMu-Unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 166, rd 0, flush 0, corrupt 0, gen 0

 

Link to comment

also to add 

 

image.thumb.png.31a4e6b864fd14b26abfb1f40114a5d1.png

 

 

[1/7] checking root items
[2/7] checking extents
ref mismatch on [194688524288 65536] extent item 0, found 1
data backref 194688524288 root 5 owner 11000 offset 24625152 num_refs 0 not found in extent tree
incorrect local backref count on 194688524288 root 5 owner 11000 offset 24625152 found 1 wanted 0 back 0x1062f950
backpointer mismatch on [194688524288 65536]
ref mismatch on [194688557056 65536] extent item 1, found 0
incorrect local backref count on 194688557056 root 5 owner 11000 offset 24625152 found 0 wanted 1 back 0x9344620
backref disk bytenr does not match extent record, bytenr=194688557056, ref bytenr=0
backpointer mismatch on [194688557056 65536]
owner ref check failed [194688557056 65536]
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space cache
[4/7] checking fs roots
root 5 inode 11000 errors 1000, some csum missing
ERROR: errors found in fs roots
Opening filesystem to check...
Checking filesystem on /dev/sdm1
UUID: 544aa70e-af02-4453-8115-da59be8c2580
found 536521904128 bytes used, error(s) found
total csum bytes: 452987428
total tree bytes: 796852224
total fs tree bytes: 163954688
total extent tree bytes: 75792384
btree space waste bytes: 107032372
file data blocks allocated: 3968869117952
 referenced 497898889216

 

Edited by emuhack
Link to comment

I ran a scrub on the pool and it seems to be stable, but when I try to delete a certain folder in appdata, it freaks out and gives these error. I have ordered a 1tb NVME drive for use with my cache to get the sata ssd's out of the system, but would be interested on if i can delete the folder before i migrate to the NVME

 

also I know disk 5 in the array is bad, waiting until the nvme is here to take out and rebuild the array :)

 

as asked diags attached emu-unraid-diagnostics-20200330-2123.zip

Edited by emuhack
Link to comment
6 hours ago, johnnie.black said:

Cache filesystem has issues, best bet is to backup all data, re-format the pool and restore the data.

Yeah I'm going to switch to a nvme this weekend, I have a backup of my appdata folder and will restore the dockers when I install the nvme. I appreciate the help and the time spend on looking at the diags.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.