Jump to content
al_uk

BTRFS corrupt leaf, no csum, cache pool, snapshots

11 posts in this topic Last Reply

Recommended Posts

Just noticed these errors in the log. sdf is one SSD out of 2 in my cache pool.

 

From other threads I think I need to run BTRFS scrub.

 

I have stopped all VMs and docker.

 

Before I did that I tried to take a backup of the cache drive to an unassigned disk, however some files failed with "cannot access source file"

 

I have daily snapshots of my VMs scripted and my VMs are in a subvolume.

 

Trying to copy one of the vm snapshots also resulted in "cannot access source file"

 

Should I run the scrub? or try another way to copy? or reboot, or something else?

 

Diagnostics attached.

 

Thanks!

May 10 16:27:50 Tower kernel: BTRFS critical (device sdf1): corrupt leaf, bad key order: block=4364604653568, root=1, slot=101
May 10 16:27:50 Tower kernel: BTRFS info (device sdf1): no csum found for inode 256620 start 9863168
May 10 16:27:50 Tower kernel: BTRFS warning (device sdf1): csum failed root 5 ino 256620 off 9863168 csum 0x48a1ad1c expected csum 0x00000000 mirror 1
May 10 16:27:50 Tower kernel: BTRFS critical (device sdf1): corrupt leaf, bad key order: block=4364604653568, root=1, slot=101
May 10 16:27:50 Tower kernel: BTRFS critical (device sdf1): corrupt leaf, bad key order: block=4364604653568, root=1, slot=101
May 10 16:27:51 Tower kernel: BTRFS critical (device sdf1): corrupt leaf, bad key order: block=4364604653568, root=1, slot=101
May 10 16:27:51 Tower kernel: BTRFS critical (device sdf1): corrupt leaf, bad key order: block=4364604653568, root=1, slot=101
May 10 16:27:51 Tower kernel: BTRFS critical (device sdf1): corrupt leaf, bad key order: block=4364604653568, root=1, slot=101
May 10 16:27:51 Tower kernel: BTRFS critical (device sdf1): corrupt leaf, bad key order: block=4364604653568, root=1, slot=101
May 10 16:30:01 Tower kernel: BTRFS critical (device sdf1): corrupt leaf, bad key order: block=4364604653568, root=1, slot=101
May 10 16:30:01 Tower kernel: __btrfs_lookup_bio_sums: 6 callbacks suppressed
May 10 16:30:01 Tower kernel: BTRFS info (device sdf1): no csum found for inode 256620 start 9863168
May 10 16:30:01 Tower kernel: btrfs_print_data_csum_error: 6 callbacks suppressed
May 10 16:30:01 Tower kernel: BTRFS warning (device sdf1): csum failed root 5 ino 256620 off 9863168 csum 0x48a1ad1c expected csum 0x00000000 mirror 1
May 10 16:30:01 Tower kernel: BTRFS critical (device sdf1): corrupt leaf, bad key order: block=4364604653568, root=1, slot=101
May 10 16:30:01 Tower kernel: BTRFS info (device sdf1): no csum found for inode 256620 start 9863168
May 10 16:30:01 Tower kernel: BTRFS warning (device sdf1): csum failed root 5 ino 256620 off 9863168 csum 0x48a1ad1c expected csum 0x00000000 mirror 2
May 10 16:32:50 Tower kernel: BTRFS critical (device sdf1): corrupt leaf, bad key order: block=4364604653568, root=1, slot=101
May 10 16:32:50 Tower kernel: BTRFS info (device sdf1): no csum found for inode 256620 start 9863168
May 10 16:32:50 Tower kernel: BTRFS warning (device sdf1): csum failed root 5 ino 256620 off 9863168 csum 0x48a1ad1c expected csum 0x00000000 mirror 1
May 10 16:32:50 Tower kernel: BTRFS critical (device sdf1): corrupt leaf, bad key order: block=4364604653568, root=1, slot=101
May 10 16:32:50 Tower kernel: BTRFS info (device sdf1): no csum found for inode 256620 start 9863168
May 10 16:32:50 Tower kernel: BTRFS warning (device sdf1): csum failed root 5 ino 256620 off 9863168 csum 0x48a1ad1c expected csum 0x00000000 mirror 2
May 10 16:32:50 Tower kernel: BTRFS critical (device sdf1): corrupt leaf, bad key order: block=4364604653568, root=1, slot=101
May 10 16:32:50 Tower kernel: BTRFS info (device sdf1): no csum found for inode 256620 start 9863168
May 10 16:32:50 Tower kernel: BTRFS warning (device sdf1): csum failed root 5 ino 256620 off 9863168 csum 0x48a1ad1c expected csum 0x00000000 mirror 2
May 10 16:32:50 Tower kernel: BTRFS critical (device sdf1): corrupt leaf, bad key order: block=4364604653568, root=1, slot=101
May 10 16:32:50 Tower kernel: BTRFS info (device sdf1): no csum found for inode 256620 start 9863168
May 10 16:32:50 Tower kernel: BTRFS warning (device sdf1): csum failed root 5 ino 256620 off 9863168 csum 0x48a1ad1c expected csum 0x00000000 mirror 1
May 10 16:32:50 Tower kernel: BTRFS critical (device sdf1): corrupt leaf, bad key order: block=4364604653568, root=1, slot=101
May 10 16:32:50 Tower kernel: BTRFS info (device sdf1): no csum found for inode 256620 start 9863168
May 10 16:32:50 Tower kernel: BTRFS warning (device sdf1): csum failed root 5 ino 256620 off 9863168 csum 0x48a1ad1c expected csum 0x00000000 mirror 2
May 10 16:32:50 Tower kernel: BTRFS critical (device sdf1): corrupt leaf, bad key order: block=4364604653568, root=1, slot=101
May 10 16:32:50 Tower kernel: BTRFS info (device sdf1): no csum found for inode 256620 start 9863168
May 10 16:32:50 Tower kernel: BTRFS warning (device sdf1): csum failed root 5 ino 256620 off 9863168 csum 0x48a1ad1c expected csum 0x00000000 mirror 2

 

tower-diagnostics-20180515-2227.zip

Share this post


Link to post

Keen to get comments before I make things worse!

 

Check btrfs file system in readonly mode gives me this.

checking extents
bad key ordering 101 102
bad block 4364604653568
ERROR: errors found in extent allocation tree or chunk allocation
checking free space cache
checking fs roots
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
root 4941 inode 302 errors 500, file extent discount, nbytes wrong
Found file extent holes:
	start: 0, len: 64424509440
root 4941 inode 303 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 6 namelen 3 name Ha2 filetype 2 errors 4, no inode ref
root 4941 inode 305 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 7 namelen 10 name desk-audio filetype 2 errors 4, no inode ref
root 4941 inode 309 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 8 namelen 5 name vDesk filetype 2 errors 4, no inode ref
root 4941 inode 311 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 9 namelen 11 name vDesk-Admin filetype 2 errors 4, no inode ref
root 4941 inode 315 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 12 namelen 12 name Mikrotik CHR filetype 2 errors 4, no inode ref
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
root 4942 inode 302 errors 500, file extent discount, nbytes wrong
Found file extent holes:
	start: 0, len: 64424509440
root 4942 inode 303 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 6 namelen 3 name Ha2 filetype 2 errors 4, no inode ref
root 4942 inode 305 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 7 namelen 10 name desk-audio filetype 2 errors 4, no inode ref
root 4942 inode 309 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 8 namelen 5 name vDesk filetype 2 errors 4, no inode ref
root 4942 inode 311 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 9 namelen 11 name vDesk-Admin filetype 2 errors 4, no inode ref
root 4942 inode 315 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 12 namelen 12 name Mikrotik CHR filetype 2 errors 4, no inode ref
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
root 4943 inode 303 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 6 namelen 3 name Ha2 filetype 2 errors 4, no inode ref
root 4943 inode 305 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 7 namelen 10 name desk-audio filetype 2 errors 4, no inode ref
root 4943 inode 309 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 8 namelen 5 name vDesk filetype 2 errors 4, no inode ref
root 4943 inode 311 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 9 namelen 11 name vDesk-Admin filetype 2 errors 4, no inode ref
root 4943 inode 315 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 12 namelen 12 name Mikrotik CHR filetype 2 errors 4, no inode ref
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
root 4944 inode 303 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 6 namelen 3 name Ha2 filetype 2 errors 4, no inode ref
root 4944 inode 304 errors 500, file extent discount, nbytes wrong
Found file extent holes:
	start: 0, len: 28244508672
root 4944 inode 305 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 7 namelen 10 name desk-audio filetype 2 errors 4, no inode ref
root 4944 inode 309 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 8 namelen 5 name vDesk filetype 2 errors 4, no inode ref
root 4944 inode 311 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 9 namelen 11 name vDesk-Admin filetype 2 errors 4, no inode ref
root 4944 inode 315 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 12 namelen 12 name Mikrotik CHR filetype 2 errors 4, no inode ref
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
root 4945 inode 303 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 6 namelen 3 name Ha2 filetype 2 errors 4, no inode ref
root 4945 inode 304 errors 500, file extent discount, nbytes wrong
Found file extent holes:
	start: 0, len: 28244508672
root 4945 inode 305 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 7 namelen 10 name desk-audio filetype 2 errors 4, no inode ref
root 4945 inode 309 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 8 namelen 5 name vDesk filetype 2 errors 4, no inode ref
root 4945 inode 311 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 9 namelen 11 name vDesk-Admin filetype 2 errors 4, no inode ref
root 4945 inode 315 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 12 namelen 12 name Mikrotik CHR filetype 2 errors 4, no inode ref
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
root 4946 inode 303 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 6 namelen 3 name Ha2 filetype 2 errors 4, no inode ref
root 4946 inode 305 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 7 namelen 10 name desk-audio filetype 2 errors 4, no inode ref
root 4946 inode 309 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 8 namelen 5 name vDesk filetype 2 errors 4, no inode ref
root 4946 inode 311 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 9 namelen 11 name vDesk-Admin filetype 2 errors 4, no inode ref
root 4946 inode 315 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 12 namelen 12 name Mikrotik CHR filetype 2 errors 4, no inode ref
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
root 4947 inode 303 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 6 namelen 3 name Ha2 filetype 2 errors 4, no inode ref
root 4947 inode 305 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 7 namelen 10 name desk-audio filetype 2 errors 4, no inode ref
root 4947 inode 309 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 8 namelen 5 name vDesk filetype 2 errors 4, no inode ref
root 4947 inode 311 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 9 namelen 11 name vDesk-Admin filetype 2 errors 4, no inode ref
root 4947 inode 315 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 12 namelen 12 name Mikrotik CHR filetype 2 errors 4, no inode ref
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
root 4948 inode 303 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 6 namelen 3 name Ha2 filetype 2 errors 4, no inode ref
root 4948 inode 305 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 7 namelen 10 name desk-audio filetype 2 errors 4, no inode ref
root 4948 inode 309 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 8 namelen 5 name vDesk filetype 2 errors 4, no inode ref
root 4948 inode 311 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 9 namelen 11 name vDesk-Admin filetype 2 errors 4, no inode ref
root 4948 inode 315 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 12 namelen 12 name Mikrotik CHR filetype 2 errors 4, no inode ref
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
root 4949 inode 303 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 6 namelen 3 name Ha2 filetype 2 errors 4, no inode ref
root 4949 inode 305 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 7 namelen 10 name desk-audio filetype 2 errors 4, no inode ref
root 4949 inode 309 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 8 namelen 5 name vDesk filetype 2 errors 4, no inode ref
root 4949 inode 311 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 9 namelen 11 name vDesk-Admin filetype 2 errors 4, no inode ref
root 4949 inode 315 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 12 namelen 12 name Mikrotik CHR filetype 2 errors 4, no inode ref
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
root 4950 inode 303 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 6 namelen 3 name Ha2 filetype 2 errors 4, no inode ref
root 4950 inode 305 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 7 namelen 10 name desk-audio filetype 2 errors 4, no inode ref
root 4950 inode 309 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 8 namelen 5 name vDesk filetype 2 errors 4, no inode ref
root 4950 inode 311 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 9 namelen 11 name vDesk-Admin filetype 2 errors 4, no inode ref
root 4950 inode 315 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 12 namelen 12 name Mikrotik CHR filetype 2 errors 4, no inode ref
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
root 4951 inode 302 errors 500, file extent discount, nbytes wrong
Found file extent holes:
	start: 0, len: 64424509440
root 4951 inode 303 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 6 namelen 3 name Ha2 filetype 2 errors 4, no inode ref
root 4951 inode 305 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 7 namelen 10 name desk-audio filetype 2 errors 4, no inode ref
root 4951 inode 309 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 8 namelen 5 name vDesk filetype 2 errors 4, no inode ref
root 4951 inode 311 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 9 namelen 11 name vDesk-Admin filetype 2 errors 4, no inode ref
root 4951 inode 315 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 12 namelen 12 name Mikrotik CHR filetype 2 errors 4, no inode ref
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
root 4952 inode 302 errors 500, file extent discount, nbytes wrong
Found file extent holes:
	start: 0, len: 64424509440
root 4952 inode 303 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 6 namelen 3 name Ha2 filetype 2 errors 4, no inode ref
root 4952 inode 305 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 7 namelen 10 name desk-audio filetype 2 errors 4, no inode ref
root 4952 inode 309 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 8 namelen 5 name vDesk filetype 2 errors 4, no inode ref
root 4952 inode 311 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 9 namelen 11 name vDesk-Admin filetype 2 errors 4, no inode ref
root 4952 inode 315 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 12 namelen 12 name Mikrotik CHR filetype 2 errors 4, no inode ref
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
root 4953 inode 302 errors 500, file extent discount, nbytes wrong
Found file extent holes:
	start: 0, len: 64424509440
root 4953 inode 303 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 6 namelen 3 name Ha2 filetype 2 errors 4, no inode ref
root 4953 inode 305 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 7 namelen 10 name desk-audio filetype 2 errors 4, no inode ref
root 4953 inode 309 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 8 namelen 5 name vDesk filetype 2 errors 4, no inode ref
root 4953 inode 311 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 9 namelen 11 name vDesk-Admin filetype 2 errors 4, no inode ref
root 4953 inode 315 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 12 namelen 12 name Mikrotik CHR filetype 2 errors 4, no inode ref
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
root 4954 inode 303 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 6 namelen 3 name Ha2 filetype 2 errors 4, no inode ref
root 4954 inode 305 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 7 namelen 10 name desk-audio filetype 2 errors 4, no inode ref
root 4954 inode 309 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 8 namelen 5 name vDesk filetype 2 errors 4, no inode ref
root 4954 inode 311 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 9 namelen 11 name vDesk-Admin filetype 2 errors 4, no inode ref
root 4954 inode 315 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 12 namelen 12 name Mikrotik CHR filetype 2 errors 4, no inode ref
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
root 4955 inode 303 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 6 namelen 3 name Ha2 filetype 2 errors 4, no inode ref
root 4955 inode 305 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 7 namelen 10 name desk-audio filetype 2 errors 4, no inode ref
root 4955 inode 309 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 8 namelen 5 name vDesk filetype 2 errors 4, no inode ref
root 4955 inode 311 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 9 namelen 11 name vDesk-Admin filetype 2 errors 4, no inode ref
root 4955 inode 315 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 12 namelen 12 name Mikrotik CHR filetype 2 errors 4, no inode ref
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
bad key ordering 101 102
root 4956 inode 303 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 6 namelen 3 name Ha2 filetype 2 errors 4, no inode ref
root 4956 inode 305 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 7 namelen 10 name desk-audio filetype 2 errors 4, no inode ref
root 4956 inode 309 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 8 namelen 5 name vDesk filetype 2 errors 4, no inode ref
root 4956 inode 311 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 9 namelen 11 name vDesk-Admin filetype 2 errors 4, no inode ref
root 4956 inode 315 errors 2001, no inode item, link count wrong
	unresolved ref dir 256 index 12 namelen 12 name Mikrotik CHR filetype 2 errors 4, no inode ref
ERROR: errors found in fs roots
Checking filesystem on /dev/sde1
UUID: 6110ce92-70ca-4bac-be5d-15972207af53
found 640056541184 bytes used, error(s) found
total csum bytes: 26273224
total tree bytes: 979714048
total fs tree bytes: 353976320
total extent tree bytes: 584417280
btree space waste bytes: 177109926
file data blocks allocated: 1306020298752
 referenced 50241048576

 

Share this post


Link to post

Reboot, post new diags so we can see the start of the syslog, you can then also try to run a correcting scrub.

Share this post


Link to post
Posted (edited)

The scrub aborted itself. Tried twice. Not sure what to do next?

 

scrub status for 6110ce92-70ca-4bac-be5d-15972207af53
	scrub started at Wed May 16 08:23:09 2018 and was aborted after 00:16:54
	total bytes scrubbed: 723.26GiB with 2 errors
	error details: csum=2
	corrected errors: 0, uncorrectable errors: 2, unverified errors: 0

 

Last part of log.

 

y 16 08:09:19 Tower kernel: ------------[ cut here ]------------
May 16 08:09:19 Tower kernel: WARNING: CPU: 1 PID: 18865 at fs/btrfs/scrub.c:2746 scrub_stripe+0xb02/0xd92
May 16 08:09:19 Tower kernel: Modules linked in: xfs nfsd lockd grace sunrpc md_mod ipmi_devintf nct6775 hwmon_vid mlx4_en mlx4_core igb ptp pps_core ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper ast ttm drm_kms_helper cryptd drm intel_cstate intel_uncore intel_rapl_perf i2c_i801 agpgart i2c_algo_bit i2c_core syscopyarea sysfillrect cdc_acm sysimgblt fb_sys_fops ahci libahci wmi acpi_power_meter ipmi_si button [last unloaded: mlx4_core]
May 16 08:09:19 Tower kernel: CPU: 1 PID: 18865 Comm: btrfs Not tainted 4.14.16-unRAID #1
May 16 08:09:19 Tower kernel: Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0b 06/12/2017
May 16 08:09:19 Tower kernel: task: ffff880efec09c00 task.stack: ffffc9000be50000
May 16 08:09:19 Tower kernel: RIP: 0010:scrub_stripe+0xb02/0xd92
May 16 08:09:19 Tower kernel: RSP: 0018:ffffc9000be53a50 EFLAGS: 00010246
May 16 08:09:19 Tower kernel: RAX: 0000000000001000 RBX: ffff880f2d433400 RCX: 0000000000000000
May 16 08:09:19 Tower kernel: RDX: 000000004ceb3901 RSI: ffff8810124db540 RDI: ffff881033e5a300
May 16 08:09:19 Tower kernel: RBP: ffffc9000be53c00 R08: 000060efc08012b0 R09: ffffffff811d3060
May 16 08:09:19 Tower kernel: R10: ffffea00404936c0 R11: 0000000000000000 R12: 0000000000003000
May 16 08:09:19 Tower kernel: R13: 0000041400901000 R14: ffff88103527ed80 R15: 0000000000000000
May 16 08:09:19 Tower kernel: FS:  0000154229703700(0000) GS:ffff88103f240000(0000) knlGS:0000000000000000
May 16 08:09:19 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 16 08:09:19 Tower kernel: CR2: 00001480e7a2bb50 CR3: 0000000f82722003 CR4: 00000000001606e0
May 16 08:09:19 Tower kernel: Call Trace:
May 16 08:09:19 Tower kernel: ? comp_keys+0x26/0x2b
May 16 08:09:19 Tower kernel: ? scrub_chunk+0x108/0x133
May 16 08:09:19 Tower kernel: scrub_chunk+0x108/0x133
May 16 08:09:19 Tower kernel: scrub_enumerate_chunks+0x2f1/0x51b
May 16 08:09:19 Tower kernel: ? wait_woken+0x1a/0x68
May 16 08:09:19 Tower kernel: btrfs_scrub_dev+0x2b9/0x400
May 16 08:09:19 Tower kernel: ? __kmalloc_track_caller+0xd1/0x11c
May 16 08:09:19 Tower kernel: btrfs_ioctl+0x13d4/0x1e00
May 16 08:09:19 Tower kernel: ? kmem_cache_alloc_node+0xd0/0x105
May 16 08:09:19 Tower kernel: ? vfs_ioctl+0x19/0x26
May 16 08:09:19 Tower kernel: ? btrfs_ioctl_get_supported_features+0x1d/0x1d
May 16 08:09:19 Tower kernel: vfs_ioctl+0x19/0x26
May 16 08:09:19 Tower kernel: do_vfs_ioctl+0x506/0x519
May 16 08:09:19 Tower kernel: ? get_task_io_context+0x3c/0x64
May 16 08:09:19 Tower kernel: SyS_ioctl+0x39/0x55
May 16 08:09:19 Tower kernel: entry_SYSCALL_64_fastpath+0x24/0x87
May 16 08:09:19 Tower kernel: RIP: 0033:0x15422980f727
May 16 08:09:19 Tower kernel: RSP: 002b:0000154229702d68 EFLAGS: 00000246
May 16 08:09:19 Tower kernel: Code: ff 89 85 c0 fe ff ff e8 09 e8 3a 00 48 ff 83 a8 02 00 00 4c 01 a3 b8 02 00 00 e9 1f fe ff ff 8b 80 5c 13 00 00 89 85 c0 fe ff ff <0f> ff e9 14 fe ff ff 48 89 df e8 4a ae ff ff 4d 8d 4f 38 49 89 
May 16 08:09:19 Tower kernel: ---[ end trace 4497dd8cdf1a55ba ]---
May 16 08:09:19 Tower kernel: ------------[ cut here ]------------
May 16 08:09:19 Tower kernel: WARNING: CPU: 0 PID: 2245 at fs/btrfs/scrub.c:2044 scrub_bio_end_io_worker+0x237/0x3d4
May 16 08:09:19 Tower kernel: Modules linked in: xfs nfsd lockd grace sunrpc md_mod ipmi_devintf nct6775 hwmon_vid mlx4_en mlx4_core igb ptp pps_core ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper ast ttm drm_kms_helper cryptd drm intel_cstate intel_uncore intel_rapl_perf i2c_i801 agpgart i2c_algo_bit i2c_core syscopyarea sysfillrect cdc_acm sysimgblt fb_sys_fops ahci libahci wmi acpi_power_meter ipmi_si button [last unloaded: mlx4_core]
May 16 08:09:19 Tower kernel: CPU: 0 PID: 2245 Comm: kworker/u24:8 Tainted: G        W       4.14.16-unRAID #1
May 16 08:09:19 Tower kernel: Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0b 06/12/2017
May 16 08:09:19 Tower kernel: Workqueue: btrfs-scrub btrfs_scrub_helper
May 16 08:09:19 Tower kernel: task: ffff8810383fe200 task.stack: ffffc90006704000
May 16 08:09:19 Tower kernel: RIP: 0010:scrub_bio_end_io_worker+0x237/0x3d4
May 16 08:09:19 Tower kernel: RSP: 0018:ffffc90006707df0 EFLAGS: 00010246
May 16 08:09:19 Tower kernel: RAX: 000000ffff880fd8 RBX: ffff880eddd59e00 RCX: 0000000000000001
May 16 08:09:19 Tower kernel: RDX: ffff880edd7c5480 RSI: 0001000000000000 RDI: ffff88103ec03200
May 16 08:09:19 Tower kernel: RBP: ffff880ff4d0d140 R08: 0000000000000001 R09: ffffffff8122d300
May 16 08:09:19 Tower kernel: R10: ffffea003b775600 R11: ffff8810383fe201 R12: ffff880f2d433400
May 16 08:09:19 Tower kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000001
May 16 08:09:19 Tower kernel: FS:  0000000000000000(0000) GS:ffff88103f200000(0000) knlGS:0000000000000000
May 16 08:09:19 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 16 08:09:19 Tower kernel: CR2: 0000152ef3f8f2c0 CR3: 0000000001c0a002 CR4: 00000000001606f0
May 16 08:09:19 Tower kernel: Call Trace:
May 16 08:09:19 Tower kernel: ? set_next_entity+0x123/0x351
May 16 08:09:19 Tower kernel: btrfs_worker_helper+0xc2/0x185
May 16 08:09:19 Tower kernel: process_one_work+0x14c/0x23f
May 16 08:09:19 Tower kernel: ? rescuer_thread+0x258/0x258
May 16 08:09:19 Tower kernel: worker_thread+0x1c3/0x292
May 16 08:09:19 Tower kernel: kthread+0x111/0x119
May 16 08:09:19 Tower kernel: ? kthread_create_on_node+0x3a/0x3a
May 16 08:09:19 Tower kernel: ret_from_fork+0x35/0x40
May 16 08:09:19 Tower kernel: Code: ff 86 e8 02 00 00 48 89 c7 e8 52 fc 3a 00 48 8b 03 45 85 ed be 03 00 00 00 48 8b 78 10 75 05 be 04 00 00 00 e8 9e 95 fd ff eb 02 <0f> ff 48 8b 83 90 00 00 00 44 8b a8 5c 02 00 00 45 85 ed 74 0b 
May 16 08:09:19 Tower kernel: ---[ end trace 4497dd8cdf1a55bb ]---
May 16 08:09:19 Tower kernel: ------------[ cut here ]------------
May 16 08:09:19 Tower kernel: WARNING: CPU: 0 PID: 2245 at fs/btrfs/scrub.c:2044 scrub_bio_end_io_worker+0x237/0x3d4
May 16 08:09:19 Tower kernel: Modules linked in: xfs nfsd lockd grace sunrpc md_mod ipmi_devintf nct6775 hwmon_vid mlx4_en mlx4_core igb ptp pps_core ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper ast ttm drm_kms_helper cryptd drm intel_cstate intel_uncore intel_rapl_perf i2c_i801 agpgart i2c_algo_bit i2c_core syscopyarea sysfillrect cdc_acm sysimgblt fb_sys_fops ahci libahci wmi acpi_power_meter ipmi_si button [last unloaded: mlx4_core]
May 16 08:09:19 Tower kernel: CPU: 0 PID: 2245 Comm: kworker/u24:8 Tainted: G        W       4.14.16-unRAID #1
May 16 08:09:19 Tower kernel: Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0b 06/12/2017
May 16 08:09:19 Tower kernel: Workqueue: btrfs-scrub btrfs_scrub_helper
May 16 08:09:19 Tower kernel: task: ffff8810383fe200 task.stack: ffffc90006704000
May 16 08:09:19 Tower kernel: RIP: 0010:scrub_bio_end_io_worker+0x237/0x3d4
May 16 08:09:19 Tower kernel: RSP: 0018:ffffc90006707df0 EFLAGS: 00010246
May 16 08:09:19 Tower kernel: RAX: 000000ffff880fd8 RBX: ffff880eddd59f00 RCX: 0000000000000001
May 16 08:09:19 Tower kernel: RDX: ffff880edd7c5400 RSI: 0001000000000000 RDI: ffff88103ec03200
May 16 08:09:19 Tower kernel: RBP: ffff880ff4d0d140 R08: 0000000000000001 R09: ffffffff8122d300
May 16 08:09:19 Tower kernel: R10: ffffea003b775600 R11: ffff8810383fe201 R12: ffff880f2d433400
May 16 08:09:19 Tower kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000002
May 16 08:09:19 Tower kernel: FS:  0000000000000000(0000) GS:ffff88103f200000(0000) knlGS:0000000000000000
May 16 08:09:19 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 16 08:09:19 Tower kernel: CR2: 0000152ef3f8f2c0 CR3: 0000000001c0a002 CR4: 00000000001606f0
May 16 08:09:19 Tower kernel: Call Trace:
May 16 08:09:19 Tower kernel: ? set_next_entity+0x123/0x351
May 16 08:09:19 Tower kernel: btrfs_worker_helper+0xc2/0x185
May 16 08:09:19 Tower kernel: process_one_work+0x14c/0x23f
May 16 08:09:19 Tower kernel: ? rescuer_thread+0x258/0x258
May 16 08:09:19 Tower kernel: worker_thread+0x1c3/0x292
May 16 08:09:19 Tower kernel: kthread+0x111/0x119
May 16 08:09:19 Tower kernel: ? kthread_create_on_node+0x3a/0x3a
May 16 08:09:19 Tower kernel: ret_from_fork+0x35/0x40
May 16 08:09:19 Tower kernel: Code: ff 86 e8 02 00 00 48 89 c7 e8 52 fc 3a 00 48 8b 03 45 85 ed be 03 00 00 00 48 8b 78 10 75 05 be 04 00 00 00 e8 9e 95 fd ff eb 02 <0f> ff 48 8b 83 90 00 00 00 44 8b a8 5c 02 00 00 45 85 ed 74 0b 
May 16 08:09:19 Tower kernel: ---[ end trace 4497dd8cdf1a55bc ]---
May 16 08:09:19 Tower kernel: ------------[ cut here ]------------
May 16 08:09:19 Tower kernel: WARNING: CPU: 0 PID: 2245 at fs/btrfs/scrub.c:2044 scrub_bio_end_io_worker+0x237/0x3d4
May 16 08:09:19 Tower kernel: Modules linked in: xfs nfsd lockd grace sunrpc md_mod ipmi_devintf nct6775 hwmon_vid mlx4_en mlx4_core igb ptp pps_core ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper ast ttm drm_kms_helper cryptd drm intel_cstate intel_uncore intel_rapl_perf i2c_i801 agpgart i2c_algo_bit i2c_core syscopyarea sysfillrect cdc_acm sysimgblt fb_sys_fops ahci libahci wmi acpi_power_meter ipmi_si button [last unloaded: mlx4_core]
May 16 08:09:19 Tower kernel: CPU: 0 PID: 2245 Comm: kworker/u24:8 Tainted: G        W       4.14.16-unRAID #1
May 16 08:09:19 Tower kernel: Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0b 06/12/2017
May 16 08:09:19 Tower kernel: Workqueue: btrfs-scrub btrfs_scrub_helper
May 16 08:09:19 Tower kernel: task: ffff8810383fe200 task.stack: ffffc90006704000
May 16 08:09:19 Tower kernel: RIP: 0010:scrub_bio_end_io_worker+0x237/0x3d4
May 16 08:09:19 Tower kernel: RSP: 0018:ffffc90006707df0 EFLAGS: 00010246
May 16 08:09:19 Tower kernel: RAX: 000000ffff880fd8 RBX: ffff880eddd58900 RCX: 0000000000000001
May 16 08:09:19 Tower kernel: RDX: ffff880edd7c5380 RSI: 0001000000000000 RDI: ffff88103ec03200
May 16 08:09:19 Tower kernel: RBP: ffff880ff4d0d140 R08: 0000000000000001 R09: ffffffff8122d300
May 16 08:09:19 Tower kernel: R10: ffffea003b775600 R11: ffff8810383fe201 R12: ffff880f2d433400
May 16 08:09:19 Tower kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000003
May 16 08:09:19 Tower kernel: FS:  0000000000000000(0000) GS:ffff88103f200000(0000) knlGS:0000000000000000
May 16 08:09:19 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 16 08:09:19 Tower kernel: CR2: 0000152ef3f8f2c0 CR3: 0000000001c0a002 CR4: 00000000001606f0
May 16 08:09:19 Tower kernel: Call Trace:
May 16 08:09:19 Tower kernel: ? set_next_entity+0x123/0x351
May 16 08:09:19 Tower kernel: btrfs_worker_helper+0xc2/0x185
May 16 08:09:19 Tower kernel: process_one_work+0x14c/0x23f
May 16 08:09:19 Tower kernel: ? rescuer_thread+0x258/0x258
May 16 08:09:19 Tower kernel: worker_thread+0x1c3/0x292
May 16 08:09:19 Tower kernel: kthread+0x111/0x119
May 16 08:09:19 Tower kernel: ? kthread_create_on_node+0x3a/0x3a
May 16 08:09:19 Tower kernel: ret_from_fork+0x35/0x40
May 16 08:09:19 Tower kernel: Code: ff 86 e8 02 00 00 48 89 c7 e8 52 fc 3a 00 48 8b 03 45 85 ed be 03 00 00 00 48 8b 78 10 75 05 be 04 00 00 00 e8 9e 95 fd ff eb 02 <0f> ff 48 8b 83 90 00 00 00 44 8b a8 5c 02 00 00 45 85 ed 74 0b 
May 16 08:09:19 Tower kernel: ---[ end trace 4497dd8cdf1a55bd ]---
May 16 08:13:00 Tower root: Fix Common Problems Version 2018.04.25
May 16 08:13:12 Tower root: Fix Common Problems: Error: Call Traces found on your server
May 16 08:13:13 Tower kernel: ------------[ cut here ]------------
May 16 08:13:13 Tower kernel: WARNING: CPU: 6 PID: 18866 at fs/btrfs/scrub.c:2746 scrub_stripe+0xb02/0xd92
May 16 08:13:13 Tower kernel: Modules linked in: xfs nfsd lockd grace sunrpc md_mod ipmi_devintf nct6775 hwmon_vid mlx4_en mlx4_core igb ptp pps_core ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper ast ttm drm_kms_helper cryptd drm intel_cstate intel_uncore intel_rapl_perf i2c_i801 agpgart i2c_algo_bit i2c_core syscopyarea sysfillrect cdc_acm sysimgblt fb_sys_fops ahci libahci wmi acpi_power_meter ipmi_si button [last unloaded: mlx4_core]
May 16 08:13:13 Tower kernel: CPU: 6 PID: 18866 Comm: btrfs Tainted: G        W       4.14.16-unRAID #1
May 16 08:13:13 Tower kernel: Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0b 06/12/2017
May 16 08:13:13 Tower kernel: task: ffff880efec0c600 task.stack: ffffc9000be58000
May 16 08:13:13 Tower kernel: RIP: 0010:scrub_stripe+0xb02/0xd92
May 16 08:13:13 Tower kernel: RSP: 0018:ffffc9000be5ba50 EFLAGS: 00010246
May 16 08:13:13 Tower kernel: RAX: 0000000000001000 RBX: ffff880f2d695400 RCX: 0000000000000000
May 16 08:13:13 Tower kernel: RDX: 000000008a92c306 RSI: ffff881013c6e850 RDI: ffff881033e5a300
May 16 08:13:13 Tower kernel: RBP: ffffc9000be5bc00 R08: 000060efc08012b0 R09: ffffffff811d3060
May 16 08:13:13 Tower kernel: R10: ffffea00404f1b80 R11: 0000000000000000 R12: 0000000000003000
May 16 08:13:13 Tower kernel: R13: 0000041400901000 R14: ffff88103527ed80 R15: 0000000000000000
May 16 08:13:13 Tower kernel: FS:  0000154229502700(0000) GS:ffff88103f380000(0000) knlGS:0000000000000000
May 16 08:13:13 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 16 08:13:13 Tower kernel: CR2: 0000000000636000 CR3: 0000000f82722005 CR4: 00000000001606e0
May 16 08:13:13 Tower kernel: Call Trace:
May 16 08:13:13 Tower kernel: ? comp_keys+0x26/0x2b
May 16 08:13:13 Tower kernel: ? scrub_chunk+0x108/0x133
May 16 08:13:13 Tower kernel: scrub_chunk+0x108/0x133
May 16 08:13:13 Tower kernel: scrub_enumerate_chunks+0x2f1/0x51b
May 16 08:13:13 Tower kernel: ? wait_woken+0x1a/0x68
May 16 08:13:13 Tower kernel: btrfs_scrub_dev+0x2b9/0x400
May 16 08:13:13 Tower kernel: ? __kmalloc_track_caller+0xd1/0x11c
May 16 08:13:13 Tower kernel: btrfs_ioctl+0x13d4/0x1e00
May 16 08:13:13 Tower kernel: ? kmem_cache_alloc_node+0xd0/0x105
May 16 08:13:13 Tower kernel: ? vfs_ioctl+0x19/0x26
May 16 08:13:13 Tower kernel: ? btrfs_ioctl_get_supported_features+0x1d/0x1d
May 16 08:13:13 Tower kernel: vfs_ioctl+0x19/0x26
May 16 08:13:13 Tower kernel: do_vfs_ioctl+0x506/0x519
May 16 08:13:13 Tower kernel: ? get_task_io_context+0x3c/0x64
May 16 08:13:13 Tower kernel: SyS_ioctl+0x39/0x55
May 16 08:13:13 Tower kernel: entry_SYSCALL_64_fastpath+0x24/0x87
May 16 08:13:13 Tower kernel: RIP: 0033:0x15422980f727
May 16 08:13:13 Tower kernel: RSP: 002b:0000154229501d68 EFLAGS: 00000246
May 16 08:13:13 Tower kernel: Code: ff 89 85 c0 fe ff ff e8 09 e8 3a 00 48 ff 83 a8 02 00 00 4c 01 a3 b8 02 00 00 e9 1f fe ff ff 8b 80 5c 13 00 00 89 85 c0 fe ff ff <0f> ff e9 14 fe ff ff 48 89 df e8 4a ae ff ff 4d 8d 4f 38 49 89 
May 16 08:13:13 Tower kernel: ---[ end trace 4497dd8cdf1a55be ]---
May 16 08:13:13 Tower kernel: ------------[ cut here ]------------
May 16 08:13:13 Tower kernel: WARNING: CPU: 0 PID: 13024 at fs/btrfs/scrub.c:2044 scrub_bio_end_io_worker+0x237/0x3d4
May 16 08:13:13 Tower kernel: Modules linked in: xfs nfsd lockd grace sunrpc md_mod ipmi_devintf nct6775 hwmon_vid mlx4_en mlx4_core igb ptp pps_core ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper ast ttm drm_kms_helper cryptd drm intel_cstate intel_uncore intel_rapl_perf i2c_i801 agpgart i2c_algo_bit i2c_core syscopyarea sysfillrect cdc_acm sysimgblt fb_sys_fops ahci libahci wmi acpi_power_meter ipmi_si button [last unloaded: mlx4_core]
May 16 08:13:13 Tower kernel: CPU: 0 PID: 13024 Comm: kworker/u24:1 Tainted: G        W       4.14.16-unRAID #1
May 16 08:13:13 Tower kernel: Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0b 06/12/2017
May 16 08:13:13 Tower kernel: Workqueue: btrfs-scrub btrfs_scrub_helper
May 16 08:13:13 Tower kernel: task: ffff881013d2c600 task.stack: ffffc900207b4000
May 16 08:13:13 Tower kernel: RIP: 0010:scrub_bio_end_io_worker+0x237/0x3d4
May 16 08:13:13 Tower kernel: RSP: 0018:ffffc900207b7df0 EFLAGS: 00010246
May 16 08:13:13 Tower kernel: RAX: 000000ffff880fd8 RBX: ffff880f90ee2b00 RCX: 0000000000000001
May 16 08:13:13 Tower kernel: RDX: ffff880fee9d6f80 RSI: 0001000000000000 RDI: ffff88103ec03200
May 16 08:13:13 Tower kernel: RBP: ffff880feca98540 R08: 0000000000000001 R09: ffffffff8122d300
May 16 08:13:13 Tower kernel: R10: ffffea003b35db00 R11: ffff88103f260901 R12: ffff880f2d695400
May 16 08:13:13 Tower kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000001
May 16 08:13:13 Tower kernel: FS:  0000000000000000(0000) GS:ffff88103f200000(0000) knlGS:0000000000000000
May 16 08:13:13 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 16 08:13:13 Tower kernel: CR2: 000014a319ebfb30 CR3: 0000000001c0a002 CR4: 00000000001606f0
May 16 08:13:13 Tower kernel: Call Trace:
May 16 08:13:13 Tower kernel: ? enqueue_task_fair+0x94/0x42c
May 16 08:13:13 Tower kernel: ? resched_curr+0x38/0x4b
May 16 08:13:13 Tower kernel: ? check_preempt_curr+0x4a/0x71
May 16 08:13:13 Tower kernel: btrfs_worker_helper+0xc2/0x185
May 16 08:13:13 Tower kernel: process_one_work+0x14c/0x23f
May 16 08:13:13 Tower kernel: ? rescuer_thread+0x258/0x258
May 16 08:13:13 Tower kernel: worker_thread+0x1c3/0x292
May 16 08:13:13 Tower kernel: kthread+0x111/0x119
May 16 08:13:13 Tower kernel: ? kthread_create_on_node+0x3a/0x3a
May 16 08:13:13 Tower kernel: ? do_group_exit+0x95/0x95
May 16 08:13:13 Tower kernel: ret_from_fork+0x35/0x40
May 16 08:13:13 Tower kernel: Code: ff 86 e8 02 00 00 48 89 c7 e8 52 fc 3a 00 48 8b 03 45 85 ed be 03 00 00 00 48 8b 78 10 75 05 be 04 00 00 00 e8 9e 95 fd ff eb 02 <0f> ff 48 8b 83 90 00 00 00 44 8b a8 5c 02 00 00 45 85 ed 74 0b 
May 16 08:13:13 Tower kernel: ---[ end trace 4497dd8cdf1a55bf ]---
May 16 08:13:13 Tower kernel: ------------[ cut here ]------------
May 16 08:13:13 Tower kernel: WARNING: CPU: 0 PID: 13024 at fs/btrfs/scrub.c:2044 scrub_bio_end_io_worker+0x237/0x3d4
May 16 08:13:13 Tower kernel: Modules linked in: xfs nfsd lockd grace sunrpc md_mod ipmi_devintf nct6775 hwmon_vid mlx4_en mlx4_core igb ptp pps_core ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper ast ttm drm_kms_helper cryptd drm intel_cstate intel_uncore intel_rapl_perf i2c_i801 agpgart i2c_algo_bit i2c_core syscopyarea sysfillrect cdc_acm sysimgblt fb_sys_fops ahci libahci wmi acpi_power_meter ipmi_si button [last unloaded: mlx4_core]
May 16 08:13:13 Tower kernel: CPU: 0 PID: 13024 Comm: kworker/u24:1 Tainted: G        W       4.14.16-unRAID #1
May 16 08:13:13 Tower kernel: Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0b 06/12/2017
May 16 08:13:13 Tower kernel: Workqueue: btrfs-scrub btrfs_scrub_helper
May 16 08:13:13 Tower kernel: task: ffff881013d2c600 task.stack: ffffc900207b4000
May 16 08:13:13 Tower kernel: RIP: 0010:scrub_bio_end_io_worker+0x237/0x3d4
May 16 08:13:13 Tower kernel: RSP: 0018:ffffc900207b7df0 EFLAGS: 00010246
May 16 08:13:13 Tower kernel: RAX: 000000ffff880fd8 RBX: ffff880f90ee3b00 RCX: 0000000000000001
May 16 08:13:13 Tower kernel: RDX: ffff880fee9d6500 RSI: 0001000000000000 RDI: 000000000000001f
May 16 08:13:13 Tower kernel: RBP: ffff880feca98540 R08: 0000000000000001 R09: ffffffff8122d301
May 16 08:13:13 Tower kernel: R10: ffffea003e43b880 R11: ffff88103f260900 R12: ffff880f2d695400
May 16 08:13:13 Tower kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000002
May 16 08:13:13 Tower kernel: FS:  0000000000000000(0000) GS:ffff88103f200000(0000) knlGS:0000000000000000
May 16 08:13:13 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 16 08:13:13 Tower kernel: CR2: 000014a319ebfb30 CR3: 0000000001c0a002 CR4: 00000000001606f0
May 16 08:13:13 Tower kernel: Call Trace:
May 16 08:13:13 Tower kernel: ? enqueue_task_fair+0x94/0x42c
May 16 08:13:13 Tower kernel: ? resched_curr+0x38/0x4b
May 16 08:13:13 Tower kernel: ? check_preempt_curr+0x4a/0x71
May 16 08:13:13 Tower kernel: btrfs_worker_helper+0xc2/0x185
May 16 08:13:13 Tower kernel: process_one_work+0x14c/0x23f
May 16 08:13:13 Tower kernel: ? rescuer_thread+0x258/0x258
May 16 08:13:13 Tower kernel: worker_thread+0x1c3/0x292
May 16 08:13:13 Tower kernel: kthread+0x111/0x119
May 16 08:13:13 Tower kernel: ? kthread_create_on_node+0x3a/0x3a
May 16 08:13:13 Tower kernel: ? do_group_exit+0x95/0x95
May 16 08:13:13 Tower kernel: ret_from_fork+0x35/0x40
May 16 08:13:13 Tower kernel: Code: ff 86 e8 02 00 00 48 89 c7 e8 52 fc 3a 00 48 8b 03 45 85 ed be 03 00 00 00 48 8b 78 10 75 05 be 04 00 00 00 e8 9e 95 fd ff eb 02 <0f> ff 48 8b 83 90 00 00 00 44 8b a8 5c 02 00 00 45 85 ed 74 0b 
May 16 08:13:13 Tower kernel: ---[ end trace 4497dd8cdf1a55c0 ]---
May 16 08:13:13 Tower kernel: ------------[ cut here ]------------
May 16 08:13:13 Tower kernel: WARNING: CPU: 0 PID: 13024 at fs/btrfs/scrub.c:2044 scrub_bio_end_io_worker+0x237/0x3d4
May 16 08:13:13 Tower kernel: Modules linked in: xfs nfsd lockd grace sunrpc md_mod ipmi_devintf nct6775 hwmon_vid mlx4_en mlx4_core igb ptp pps_core ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper ast ttm drm_kms_helper cryptd drm intel_cstate intel_uncore intel_rapl_perf i2c_i801 agpgart i2c_algo_bit i2c_core syscopyarea sysfillrect cdc_acm sysimgblt fb_sys_fops ahci libahci wmi acpi_power_meter ipmi_si button [last unloaded: mlx4_core]
May 16 08:13:13 Tower kernel: CPU: 0 PID: 13024 Comm: kworker/u24:1 Tainted: G        W       4.14.16-unRAID #1
May 16 08:13:13 Tower kernel: Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0b 06/12/2017
May 16 08:13:13 Tower kernel: Workqueue: btrfs-scrub btrfs_scrub_helper
May 16 08:13:13 Tower kernel: task: ffff881013d2c600 task.stack: ffffc900207b4000
May 16 08:13:13 Tower kernel: RIP: 0010:scrub_bio_end_io_worker+0x237/0x3d4
May 16 08:13:13 Tower kernel: RSP: 0018:ffffc900207b7df0 EFLAGS: 00010246
May 16 08:13:13 Tower kernel: RAX: 000000ffff880fd8 RBX: ffff880f90ee3800 RCX: 0000000000000001
May 16 08:13:13 Tower kernel: RDX: ffff880fee9d6a80 RSI: 0001000000000000 RDI: ffff88103ec03200
May 16 08:13:13 Tower kernel: RBP: ffff880feca98540 R08: 0000000000000001 R09: ffffffff8122d300
May 16 08:13:13 Tower kernel: R10: ffffea003e43b880 R11: ffff88103f260901 R12: ffff880f2d695400
May 16 08:13:13 Tower kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000003
May 16 08:13:13 Tower kernel: FS:  0000000000000000(0000) GS:ffff88103f200000(0000) knlGS:0000000000000000
May 16 08:13:13 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 16 08:13:13 Tower kernel: CR2: 000014a319ebfb30 CR3: 0000000001c0a002 CR4: 00000000001606f0
May 16 08:13:13 Tower kernel: Call Trace:
May 16 08:13:13 Tower kernel: ? enqueue_task_fair+0x94/0x42c
May 16 08:13:13 Tower kernel: ? resched_curr+0x38/0x4b
May 16 08:13:13 Tower kernel: ? check_preempt_curr+0x4a/0x71
May 16 08:13:13 Tower kernel: btrfs_worker_helper+0xc2/0x185
May 16 08:13:13 Tower kernel: process_one_work+0x14c/0x23f
May 16 08:13:13 Tower kernel: ? rescuer_thread+0x258/0x258
May 16 08:13:13 Tower kernel: worker_thread+0x1c3/0x292
May 16 08:13:13 Tower kernel: kthread+0x111/0x119
May 16 08:13:13 Tower kernel: ? kthread_create_on_node+0x3a/0x3a
May 16 08:13:13 Tower kernel: ? do_group_exit+0x95/0x95
May 16 08:13:13 Tower kernel: ret_from_fork+0x35/0x40
May 16 08:13:13 Tower kernel: Code: ff 86 e8 02 00 00 48 89 c7 e8 52 fc 3a 00 48 8b 03 45 85 ed be 03 00 00 00 48 8b 78 10 75 05 be 04 00 00 00 e8 9e 95 fd ff eb 02 <0f> ff 48 8b 83 90 00 00 00 44 8b a8 5c 02 00 00 45 85 ed 74 0b 
May 16 08:13:13 Tower kernel: ---[ end trace 4497dd8cdf1a55c1 ]---
May 16 08:13:13 Tower kernel: pcieport 0000:00:1c.7: Enabling MPC IRBNCE
May 16 08:13:13 Tower kernel: pcieport 0000:00:1c.7: Intel PCH root port ACS workaround enabled
May 16 08:13:13 Tower root: Fix Common Problems: Other Warning: Mover logging is enabled
May 16 08:13:13 Tower sSMTP[11479]: Creating SSL connection to host
May 16 08:13:13 Tower sSMTP[11479]: SSL connection using ECDHE-RSA-AES256-GCM-SHA384
May 16 08:13:15 Tower sSMTP[11479]: Sent mail for aa@aa.net.uk (221 b-painless.mh.aa.net.uk closing connection) uid=0 username=root outbytes=653
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4956, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4955, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4954, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4953, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4952, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4951, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4950, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4949, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4948, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4947, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4946, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4945, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4944, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4943, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4942, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4941, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4940, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4939, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4938, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4937, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4936, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4935, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4934, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4933, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4932, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4931, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4930, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4929, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4928, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4927, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4926, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4925, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4924, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4923, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4922, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4883, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4875, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4810, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS error (device sdf1): bdev /dev/sde1 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
May 16 08:14:54 Tower kernel: BTRFS error (device sdf1): unable to fixup (regular) error at logical 4868798496768 on dev /dev/sde1
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4956, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4955, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4954, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4953, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4952, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4951, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4950, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4949, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4948, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4947, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4946, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4945, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4944, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4943, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4942, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4941, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4940, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4939, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4938, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4937, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4936, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4935, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4934, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4933, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4932, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4931, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4930, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4929, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4928, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4927, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4926, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4925, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4924, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4923, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4922, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4883, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4875, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4810, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:14:54 Tower kernel: BTRFS error (device sdf1): bdev /dev/sde1 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
May 16 08:14:54 Tower kernel: BTRFS error (device sdf1): unable to fixup (regular) error at logical 4868798500864 on dev /dev/sde1
May 16 08:15:37 Tower kernel: BTRFS critical (device sdf1): corrupt leaf, bad key order: block=4364604653568, root=1, slot=101
May 16 08:15:37 Tower kernel: BTRFS critical (device sdf1): corrupt leaf, bad key order: block=4364604653568, root=1, slot=101
May 16 08:22:44 Tower kernel: BTRFS critical (device sdf1): corrupt leaf, bad key order: block=4364604653568, root=1, slot=101
May 16 08:22:44 Tower kernel: BTRFS critical (device sdf1): corrupt leaf, bad key order: block=4364604653568, root=1, slot=101
May 16 08:23:09 Tower ool www[18045]: /usr/local/emhttp/plugins/dynamix/scripts/btrfs_scrub 'start' '/mnt/cache' ''
May 16 08:26:05 Tower kernel: ------------[ cut here ]------------
May 16 08:26:05 Tower kernel: WARNING: CPU: 8 PID: 22855 at fs/btrfs/scrub.c:2746 scrub_stripe+0xb02/0xd92
May 16 08:26:05 Tower kernel: Modules linked in: xfs nfsd lockd grace sunrpc md_mod ipmi_devintf nct6775 hwmon_vid mlx4_en mlx4_core igb ptp pps_core ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper ast ttm drm_kms_helper cryptd drm intel_cstate intel_uncore intel_rapl_perf i2c_i801 agpgart i2c_algo_bit i2c_core syscopyarea sysfillrect cdc_acm sysimgblt fb_sys_fops ahci libahci wmi acpi_power_meter ipmi_si button [last unloaded: mlx4_core]
May 16 08:26:05 Tower kernel: CPU: 8 PID: 22855 Comm: btrfs Tainted: G        W       4.14.16-unRAID #1
May 16 08:26:05 Tower kernel: Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0b 06/12/2017
May 16 08:26:05 Tower kernel: task: ffff881038768e00 task.stack: ffffc9000c4d0000
May 16 08:26:05 Tower kernel: RIP: 0010:scrub_stripe+0xb02/0xd92
May 16 08:26:05 Tower kernel: RSP: 0018:ffffc9000c4d3a50 EFLAGS: 00010246
May 16 08:26:05 Tower kernel: RAX: 0000000000001000 RBX: ffff880fed62d800 RCX: 0000000000000000
May 16 08:26:05 Tower kernel: RDX: 00000001c3a65508 RSI: ffff881002583a80 RDI: ffff881033e5a300
May 16 08:26:05 Tower kernel: RBP: ffffc9000c4d3c00 R08: 000060efc08012b0 R09: ffffffff811d3060
May 16 08:26:05 Tower kernel: R10: ffffea00400960c0 R11: 0000000000000000 R12: 0000000000003000
May 16 08:26:05 Tower kernel: R13: 0000041400901000 R14: ffff88103527ed80 R15: 0000000000000000
May 16 08:26:05 Tower kernel: FS:  000014767534a700(0000) GS:ffff88103f400000(0000) knlGS:0000000000000000
May 16 08:26:05 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 16 08:26:05 Tower kernel: CR2: 0000149cfe34f6f8 CR3: 0000000eb620c003 CR4: 00000000001606e0
May 16 08:26:05 Tower kernel: Call Trace:
May 16 08:26:05 Tower kernel: ? comp_keys+0x26/0x2b
May 16 08:26:05 Tower kernel: ? scrub_chunk+0x108/0x133
May 16 08:26:05 Tower kernel: scrub_chunk+0x108/0x133
May 16 08:26:05 Tower kernel: scrub_enumerate_chunks+0x2f1/0x51b
May 16 08:26:05 Tower kernel: ? wait_woken+0x1a/0x68
May 16 08:26:05 Tower kernel: btrfs_scrub_dev+0x2b9/0x400
May 16 08:26:05 Tower kernel: ? __kmalloc_track_caller+0xd1/0x11c
May 16 08:26:05 Tower kernel: btrfs_ioctl+0x13d4/0x1e00
May 16 08:26:05 Tower kernel: ? dequeue_entity+0x49d/0x4c2
May 16 08:26:05 Tower kernel: ? pick_next_task_fair+0x22d/0x3df
May 16 08:26:05 Tower kernel: ? put_prev_entity+0x21/0x2dc
May 16 08:26:05 Tower kernel: ? __update_idle_core+0x14/0x9a
May 16 08:26:05 Tower kernel: ? kmem_cache_alloc_node+0xd0/0x105
May 16 08:26:05 Tower kernel: ? vfs_ioctl+0x19/0x26
May 16 08:26:05 Tower kernel: ? btrfs_ioctl_get_supported_features+0x1d/0x1d
May 16 08:26:05 Tower kernel: vfs_ioctl+0x19/0x26
May 16 08:26:05 Tower kernel: do_vfs_ioctl+0x506/0x519
May 16 08:26:05 Tower kernel: ? get_task_io_context+0x3c/0x64
May 16 08:26:05 Tower kernel: SyS_ioctl+0x39/0x55
May 16 08:26:05 Tower kernel: entry_SYSCALL_64_fastpath+0x24/0x87
May 16 08:26:05 Tower kernel: RIP: 0033:0x147675456727
May 16 08:26:05 Tower kernel: RSP: 002b:0000147675349d68 EFLAGS: 00000246
May 16 08:26:05 Tower kernel: Code: ff 89 85 c0 fe ff ff e8 09 e8 3a 00 48 ff 83 a8 02 00 00 4c 01 a3 b8 02 00 00 e9 1f fe ff ff 8b 80 5c 13 00 00 89 85 c0 fe ff ff <0f> ff e9 14 fe ff ff 48 89 df e8 4a ae ff ff 4d 8d 4f 38 49 89 
May 16 08:26:05 Tower kernel: ---[ end trace 4497dd8cdf1a55c2 ]---
May 16 08:26:05 Tower kernel: ------------[ cut here ]------------
May 16 08:26:05 Tower kernel: WARNING: CPU: 0 PID: 13023 at fs/btrfs/scrub.c:2044 scrub_bio_end_io_worker+0x237/0x3d4
May 16 08:26:05 Tower kernel: Modules linked in: xfs nfsd lockd grace sunrpc md_mod ipmi_devintf nct6775 hwmon_vid mlx4_en mlx4_core igb ptp pps_core ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper ast ttm drm_kms_helper cryptd drm intel_cstate intel_uncore intel_rapl_perf i2c_i801 agpgart i2c_algo_bit i2c_core syscopyarea sysfillrect cdc_acm sysimgblt fb_sys_fops ahci libahci wmi acpi_power_meter ipmi_si button [last unloaded: mlx4_core]
May 16 08:26:05 Tower kernel: CPU: 0 PID: 13023 Comm: kworker/u24:0 Tainted: G        W       4.14.16-unRAID #1
May 16 08:26:05 Tower kernel: Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0b 06/12/2017
May 16 08:26:05 Tower kernel: Workqueue: btrfs-scrub btrfs_scrub_helper
May 16 08:26:05 Tower kernel: task: ffff881013e1d400 task.stack: ffffc90020b6c000
May 16 08:26:05 Tower kernel: RIP: 0010:scrub_bio_end_io_worker+0x237/0x3d4
May 16 08:26:05 Tower kernel: RSP: 0018:ffffc90020b6fdf0 EFLAGS: 00010246
May 16 08:26:05 Tower kernel: RAX: 000000ffff880fd8 RBX: ffff880eb58b4600 RCX: 0000000000000001
May 16 08:26:05 Tower kernel: RDX: ffff880eb5156600 RSI: 0001000000000000 RDI: ffff88103ec03200
May 16 08:26:05 Tower kernel: RBP: ffff880ff0f19940 R08: 0000000000000001 R09: ffffffff8122d300
May 16 08:26:05 Tower kernel: R10: ffffea003ad62d00 R11: ffff88103f3a0901 R12: ffff880fed62d800
May 16 08:26:05 Tower kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000001
May 16 08:26:05 Tower kernel: FS:  0000000000000000(0000) GS:ffff88103f200000(0000) knlGS:0000000000000000
May 16 08:26:05 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 16 08:26:05 Tower kernel: CR2: 00007ffdccf71fb0 CR3: 0000000001c0a004 CR4: 00000000001606f0
May 16 08:26:05 Tower kernel: Call Trace:
May 16 08:26:05 Tower kernel: ? enqueue_task_fair+0x94/0x42c
May 16 08:26:05 Tower kernel: ? resched_curr+0x38/0x4b
May 16 08:26:05 Tower kernel: ? check_preempt_curr+0x4a/0x71
May 16 08:26:05 Tower kernel: btrfs_worker_helper+0xc2/0x185
May 16 08:26:05 Tower kernel: process_one_work+0x14c/0x23f
May 16 08:26:05 Tower kernel: ? rescuer_thread+0x258/0x258
May 16 08:26:05 Tower kernel: worker_thread+0x1c3/0x292
May 16 08:26:05 Tower kernel: kthread+0x111/0x119
May 16 08:26:05 Tower kernel: ? kthread_create_on_node+0x3a/0x3a
May 16 08:26:05 Tower kernel: ? do_group_exit+0x95/0x95
May 16 08:26:05 Tower kernel: ret_from_fork+0x35/0x40
May 16 08:26:05 Tower kernel: Code: ff 86 e8 02 00 00 48 89 c7 e8 52 fc 3a 00 48 8b 03 45 85 ed be 03 00 00 00 48 8b 78 10 75 05 be 04 00 00 00 e8 9e 95 fd ff eb 02 <0f> ff 48 8b 83 90 00 00 00 44 8b a8 5c 02 00 00 45 85 ed 74 0b 
May 16 08:26:05 Tower kernel: ---[ end trace 4497dd8cdf1a55c3 ]---
May 16 08:26:05 Tower kernel: ------------[ cut here ]------------
May 16 08:26:05 Tower kernel: WARNING: CPU: 0 PID: 13023 at fs/btrfs/scrub.c:2044 scrub_bio_end_io_worker+0x237/0x3d4
May 16 08:26:05 Tower kernel: Modules linked in: xfs nfsd lockd grace sunrpc md_mod ipmi_devintf nct6775 hwmon_vid mlx4_en mlx4_core igb ptp pps_core ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper ast ttm drm_kms_helper cryptd drm intel_cstate intel_uncore intel_rapl_perf i2c_i801 agpgart i2c_algo_bit i2c_core syscopyarea sysfillrect cdc_acm sysimgblt fb_sys_fops ahci libahci wmi acpi_power_meter ipmi_si button [last unloaded: mlx4_core]
May 16 08:26:05 Tower kernel: CPU: 0 PID: 13023 Comm: kworker/u24:0 Tainted: G        W       4.14.16-unRAID #1
May 16 08:26:05 Tower kernel: Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0b 06/12/2017
May 16 08:26:05 Tower kernel: Workqueue: btrfs-scrub btrfs_scrub_helper
May 16 08:26:05 Tower kernel: task: ffff881013e1d400 task.stack: ffffc90020b6c000
May 16 08:26:05 Tower kernel: RIP: 0010:scrub_bio_end_io_worker+0x237/0x3d4
May 16 08:26:05 Tower kernel: RSP: 0018:ffffc90020b6fdf0 EFLAGS: 00010246
May 16 08:26:05 Tower kernel: RAX: 000000ffff880fd8 RBX: ffff880eb58b4700 RCX: 0000000000000001
May 16 08:26:05 Tower kernel: RDX: ffff880eb5156680 RSI: 0001000000000000 RDI: ffff88103ec03200
May 16 08:26:05 Tower kernel: RBP: ffff880ff0f19940 R08: 0000000000000001 R09: ffffffff8122d300
May 16 08:26:05 Tower kernel: R10: ffffea003ad62d00 R11: ffff88103f3a0901 R12: ffff880fed62d800
May 16 08:26:05 Tower kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000002
May 16 08:26:05 Tower kernel: FS:  0000000000000000(0000) GS:ffff88103f200000(0000) knlGS:0000000000000000
May 16 08:26:05 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 16 08:26:05 Tower kernel: CR2: 00007ffdccf71fb0 CR3: 0000000001c0a004 CR4: 00000000001606f0
May 16 08:26:05 Tower kernel: Call Trace:
May 16 08:26:05 Tower kernel: ? enqueue_task_fair+0x94/0x42c
May 16 08:26:05 Tower kernel: ? resched_curr+0x38/0x4b
May 16 08:26:05 Tower kernel: ? check_preempt_curr+0x4a/0x71
May 16 08:26:05 Tower kernel: btrfs_worker_helper+0xc2/0x185
May 16 08:26:05 Tower kernel: process_one_work+0x14c/0x23f
May 16 08:26:05 Tower kernel: ? rescuer_thread+0x258/0x258
May 16 08:26:05 Tower kernel: worker_thread+0x1c3/0x292
May 16 08:26:05 Tower kernel: kthread+0x111/0x119
May 16 08:26:05 Tower kernel: ? kthread_create_on_node+0x3a/0x3a
May 16 08:26:05 Tower kernel: ? do_group_exit+0x95/0x95
May 16 08:26:05 Tower kernel: ret_from_fork+0x35/0x40
May 16 08:26:05 Tower kernel: Code: ff 86 e8 02 00 00 48 89 c7 e8 52 fc 3a 00 48 8b 03 45 85 ed be 03 00 00 00 48 8b 78 10 75 05 be 04 00 00 00 e8 9e 95 fd ff eb 02 <0f> ff 48 8b 83 90 00 00 00 44 8b a8 5c 02 00 00 45 85 ed 74 0b 
May 16 08:26:05 Tower kernel: ---[ end trace 4497dd8cdf1a55c4 ]---
May 16 08:26:05 Tower kernel: ------------[ cut here ]------------
May 16 08:26:05 Tower kernel: WARNING: CPU: 0 PID: 13023 at fs/btrfs/scrub.c:2044 scrub_bio_end_io_worker+0x237/0x3d4
May 16 08:26:05 Tower kernel: Modules linked in: xfs nfsd lockd grace sunrpc md_mod ipmi_devintf nct6775 hwmon_vid mlx4_en mlx4_core igb ptp pps_core ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper ast ttm drm_kms_helper cryptd drm intel_cstate intel_uncore intel_rapl_perf i2c_i801 agpgart i2c_algo_bit i2c_core syscopyarea sysfillrect cdc_acm sysimgblt fb_sys_fops ahci libahci wmi acpi_power_meter ipmi_si button [last unloaded: mlx4_core]
May 16 08:26:05 Tower kernel: CPU: 0 PID: 13023 Comm: kworker/u24:0 Tainted: G        W       4.14.16-unRAID #1
May 16 08:26:05 Tower kernel: Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0b 06/12/2017
May 16 08:26:05 Tower kernel: Workqueue: btrfs-scrub btrfs_scrub_helper
May 16 08:26:05 Tower kernel: task: ffff881013e1d400 task.stack: ffffc90020b6c000
May 16 08:26:05 Tower kernel: RIP: 0010:scrub_bio_end_io_worker+0x237/0x3d4
May 16 08:26:05 Tower kernel: RSP: 0018:ffffc90020b6fdf0 EFLAGS: 00010246
May 16 08:26:05 Tower kernel: RAX: 000000ffff880fd8 RBX: ffff880eb58b4800 RCX: 0000000000000001
May 16 08:26:05 Tower kernel: RDX: ffff880eb5156700 RSI: 0001000000000000 RDI: ffff88103ec03200
May 16 08:26:05 Tower kernel: RBP: ffff880ff0f19940 R08: 0000000000000001 R09: ffffffff8122d300
May 16 08:26:05 Tower kernel: R10: ffffea003ad62d00 R11: ffff88103f3a0901 R12: ffff880fed62d800
May 16 08:26:05 Tower kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000003
May 16 08:26:05 Tower kernel: FS:  0000000000000000(0000) GS:ffff88103f200000(0000) knlGS:0000000000000000
May 16 08:26:05 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 16 08:26:05 Tower kernel: CR2: 00007ffdccf71fb0 CR3: 0000000001c0a004 CR4: 00000000001606f0
May 16 08:26:05 Tower kernel: Call Trace:
May 16 08:26:05 Tower kernel: ? enqueue_task_fair+0x94/0x42c
May 16 08:26:05 Tower kernel: ? resched_curr+0x38/0x4b
May 16 08:26:05 Tower kernel: ? check_preempt_curr+0x4a/0x71
May 16 08:26:05 Tower kernel: btrfs_worker_helper+0xc2/0x185
May 16 08:26:05 Tower kernel: process_one_work+0x14c/0x23f
May 16 08:26:05 Tower kernel: ? rescuer_thread+0x258/0x258
May 16 08:26:05 Tower kernel: worker_thread+0x1c3/0x292
May 16 08:26:05 Tower kernel: kthread+0x111/0x119
May 16 08:26:05 Tower kernel: ? kthread_create_on_node+0x3a/0x3a
May 16 08:26:05 Tower kernel: ? do_group_exit+0x95/0x95
May 16 08:26:05 Tower kernel: ret_from_fork+0x35/0x40
May 16 08:26:05 Tower kernel: Code: ff 86 e8 02 00 00 48 89 c7 e8 52 fc 3a 00 48 8b 03 45 85 ed be 03 00 00 00 48 8b 78 10 75 05 be 04 00 00 00 e8 9e 95 fd ff eb 02 <0f> ff 48 8b 83 90 00 00 00 44 8b a8 5c 02 00 00 45 85 ed 74 0b 
May 16 08:26:05 Tower kernel: ---[ end trace 4497dd8cdf1a55c5 ]---
May 16 08:29:45 Tower kernel: mdcmd (50): spindown 1
May 16 08:29:59 Tower kernel: mdcmd (51): spindown 3
May 16 08:30:01 Tower kernel: mdcmd (52): spindown 0
May 16 08:30:01 Tower kernel: mdcmd (53): spindown 2
May 16 08:30:12 Tower kernel: ------------[ cut here ]------------
May 16 08:30:12 Tower kernel: WARNING: CPU: 8 PID: 22856 at fs/btrfs/scrub.c:2746 scrub_stripe+0xb02/0xd92
May 16 08:30:12 Tower kernel: Modules linked in: xfs nfsd lockd grace sunrpc md_mod ipmi_devintf nct6775 hwmon_vid mlx4_en mlx4_core igb ptp pps_core ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper ast ttm drm_kms_helper cryptd drm intel_cstate intel_uncore intel_rapl_perf i2c_i801 agpgart i2c_algo_bit i2c_core syscopyarea sysfillrect cdc_acm sysimgblt fb_sys_fops ahci libahci wmi acpi_power_meter ipmi_si button [last unloaded: mlx4_core]
May 16 08:30:12 Tower kernel: CPU: 8 PID: 22856 Comm: btrfs Tainted: G        W       4.14.16-unRAID #1
May 16 08:30:12 Tower kernel: Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0b 06/12/2017
May 16 08:30:12 Tower kernel: task: ffff88103876e200 task.stack: ffffc9000c930000
May 16 08:30:12 Tower kernel: RIP: 0010:scrub_stripe+0xb02/0xd92
May 16 08:30:12 Tower kernel: RSP: 0018:ffffc9000c933a50 EFLAGS: 00010246
May 16 08:30:12 Tower kernel: RAX: 0000000000001000 RBX: ffff880fe99da800 RCX: 0000000000000000
May 16 08:30:12 Tower kernel: RDX: 000000022bf4f908 RSI: ffff881002583690 RDI: ffff881033e5a300
May 16 08:30:12 Tower kernel: RBP: ffffc9000c933c00 R08: 000060efc08012b0 R09: ffffffff811d3060
May 16 08:30:12 Tower kernel: R10: ffffea00400960c0 R11: 0000000000000000 R12: 0000000000003000
May 16 08:30:12 Tower kernel: R13: 0000041400901000 R14: ffff88103527ed80 R15: 0000000000000000
May 16 08:30:12 Tower kernel: FS:  0000147675149700(0000) GS:ffff88103f400000(0000) knlGS:0000000000000000
May 16 08:30:12 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 16 08:30:12 Tower kernel: CR2: 00000000006517e0 CR3: 0000000eb620c001 CR4: 00000000001606e0
May 16 08:30:12 Tower kernel: Call Trace:
May 16 08:30:12 Tower kernel: ? comp_keys+0x26/0x2b
May 16 08:30:12 Tower kernel: ? scrub_chunk+0x108/0x133
May 16 08:30:12 Tower kernel: scrub_chunk+0x108/0x133
May 16 08:30:12 Tower kernel: scrub_enumerate_chunks+0x2f1/0x51b
May 16 08:30:12 Tower kernel: ? wait_woken+0x1a/0x68
May 16 08:30:12 Tower kernel: btrfs_scrub_dev+0x2b9/0x400
May 16 08:30:12 Tower kernel: ? __kmalloc_track_caller+0xd1/0x11c
May 16 08:30:12 Tower kernel: btrfs_ioctl+0x13d4/0x1e00
May 16 08:30:12 Tower kernel: ? kmem_cache_alloc_node+0xd0/0x105
May 16 08:30:12 Tower kernel: ? vfs_ioctl+0x19/0x26
May 16 08:30:12 Tower kernel: ? btrfs_ioctl_get_supported_features+0x1d/0x1d
May 16 08:30:12 Tower kernel: vfs_ioctl+0x19/0x26
May 16 08:30:12 Tower kernel: do_vfs_ioctl+0x506/0x519
May 16 08:30:12 Tower kernel: ? get_task_io_context+0x3c/0x64
May 16 08:30:12 Tower kernel: SyS_ioctl+0x39/0x55
May 16 08:30:12 Tower kernel: entry_SYSCALL_64_fastpath+0x24/0x87
May 16 08:30:12 Tower kernel: RIP: 0033:0x147675456727
May 16 08:30:12 Tower kernel: RSP: 002b:0000147675148d68 EFLAGS: 00000246
May 16 08:30:12 Tower kernel: Code: ff 89 85 c0 fe ff ff e8 09 e8 3a 00 48 ff 83 a8 02 00 00 4c 01 a3 b8 02 00 00 e9 1f fe ff ff 8b 80 5c 13 00 00 89 85 c0 fe ff ff <0f> ff e9 14 fe ff ff 48 89 df e8 4a ae ff ff 4d 8d 4f 38 49 89 
May 16 08:30:12 Tower kernel: ---[ end trace 4497dd8cdf1a55c6 ]---
May 16 08:30:12 Tower kernel: ------------[ cut here ]------------
May 16 08:30:12 Tower kernel: WARNING: CPU: 8 PID: 79 at fs/btrfs/scrub.c:2044 scrub_bio_end_io_worker+0x237/0x3d4
May 16 08:30:12 Tower kernel: Modules linked in: xfs nfsd lockd grace sunrpc md_mod ipmi_devintf nct6775 hwmon_vid mlx4_en mlx4_core igb ptp pps_core ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper ast ttm drm_kms_helper cryptd drm intel_cstate intel_uncore intel_rapl_perf i2c_i801 agpgart i2c_algo_bit i2c_core syscopyarea sysfillrect cdc_acm sysimgblt fb_sys_fops ahci libahci wmi acpi_power_meter ipmi_si button [last unloaded: mlx4_core]
May 16 08:30:12 Tower kernel: CPU: 8 PID: 79 Comm: kworker/u24:3 Tainted: G        W       4.14.16-unRAID #1
May 16 08:30:12 Tower kernel: Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0b 06/12/2017
May 16 08:30:12 Tower kernel: Workqueue: btrfs-scrub btrfs_scrub_helper
May 16 08:30:12 Tower kernel: task: ffff881038769c00 task.stack: ffffc90006574000
May 16 08:30:12 Tower kernel: RIP: 0010:scrub_bio_end_io_worker+0x237/0x3d4
May 16 08:30:12 Tower kernel: RSP: 0018:ffffc90006577df0 EFLAGS: 00010246
May 16 08:30:12 Tower kernel: RAX: 000000ffff880fd8 RBX: ffff880eb66ee900 RCX: 0000000000000001
May 16 08:30:12 Tower kernel: RDX: ffff880eb51f5380 RSI: 0001000000000000 RDI: 000000000000001f
May 16 08:30:12 Tower kernel: RBP: ffff880fec71f140 R08: 0000000000000001 R09: ffffffff8122d301
May 16 08:30:12 Tower kernel: R10: ffffea003ad9bb80 R11: ffff88103f2a0900 R12: ffff880fe99da800
May 16 08:30:12 Tower kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000001
May 16 08:30:12 Tower kernel: FS:  0000000000000000(0000) GS:ffff88103f400000(0000) knlGS:0000000000000000
May 16 08:30:12 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 16 08:30:12 Tower kernel: CR2: 00000000006517e0 CR3: 0000000001c0a004 CR4: 00000000001606e0
May 16 08:30:12 Tower kernel: Call Trace:
May 16 08:30:12 Tower kernel: ? check_preempt_wakeup+0x12b/0x179
May 16 08:30:12 Tower kernel: ? check_preempt_curr+0x2a/0x71
May 16 08:30:12 Tower kernel: btrfs_worker_helper+0xc2/0x185
May 16 08:30:12 Tower kernel: process_one_work+0x14c/0x23f
May 16 08:30:12 Tower kernel: ? rescuer_thread+0x258/0x258
May 16 08:30:12 Tower kernel: worker_thread+0x1c3/0x292
May 16 08:30:12 Tower kernel: kthread+0x111/0x119
May 16 08:30:12 Tower kernel: ? kthread_create_on_node+0x3a/0x3a
May 16 08:30:12 Tower kernel: ? call_usermodehelper_exec_async+0x116/0x11d
May 16 08:30:12 Tower kernel: ret_from_fork+0x35/0x40
May 16 08:30:12 Tower kernel: Code: ff 86 e8 02 00 00 48 89 c7 e8 52 fc 3a 00 48 8b 03 45 85 ed be 03 00 00 00 48 8b 78 10 75 05 be 04 00 00 00 e8 9e 95 fd ff eb 02 <0f> ff 48 8b 83 90 00 00 00 44 8b a8 5c 02 00 00 45 85 ed 74 0b 
May 16 08:30:12 Tower kernel: ---[ end trace 4497dd8cdf1a55c7 ]---
May 16 08:30:12 Tower kernel: ------------[ cut here ]------------
May 16 08:30:12 Tower kernel: WARNING: CPU: 8 PID: 79 at fs/btrfs/scrub.c:2044 scrub_bio_end_io_worker+0x237/0x3d4
May 16 08:30:12 Tower kernel: Modules linked in: xfs nfsd lockd grace sunrpc md_mod ipmi_devintf nct6775 hwmon_vid mlx4_en mlx4_core igb ptp pps_core ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper ast ttm drm_kms_helper cryptd drm intel_cstate intel_uncore intel_rapl_perf i2c_i801 agpgart i2c_algo_bit i2c_core syscopyarea sysfillrect cdc_acm sysimgblt fb_sys_fops ahci libahci wmi acpi_power_meter ipmi_si button [last unloaded: mlx4_core]
May 16 08:30:12 Tower kernel: CPU: 8 PID: 79 Comm: kworker/u24:3 Tainted: G        W       4.14.16-unRAID #1
May 16 08:30:12 Tower kernel: Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0b 06/12/2017
May 16 08:30:12 Tower kernel: Workqueue: btrfs-scrub btrfs_scrub_helper
May 16 08:30:12 Tower kernel: task: ffff881038769c00 task.stack: ffffc90006574000
May 16 08:30:12 Tower kernel: RIP: 0010:scrub_bio_end_io_worker+0x237/0x3d4
May 16 08:30:12 Tower kernel: RSP: 0018:ffffc90006577df0 EFLAGS: 00010246
May 16 08:30:12 Tower kernel: RAX: 000000ffff880fd8 RBX: ffff880eb66ef900 RCX: 0000000000000001
May 16 08:30:12 Tower kernel: RDX: ffff880eb51f5400 RSI: 0001000000000000 RDI: ffff88103ec03200
May 16 08:30:12 Tower kernel: RBP: ffff880fec71f140 R08: 0000000000000001 R09: ffffffff8122d300
May 16 08:30:12 Tower kernel: R10: ffffea003ad9bb80 R11: ffff88103f2a0901 R12: ffff880fe99da800
May 16 08:30:12 Tower kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000002
May 16 08:30:12 Tower kernel: FS:  0000000000000000(0000) GS:ffff88103f400000(0000) knlGS:0000000000000000
May 16 08:30:12 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 16 08:30:12 Tower kernel: CR2: 00000000006517e0 CR3: 0000000001c0a004 CR4: 00000000001606e0
May 16 08:30:12 Tower kernel: Call Trace:
May 16 08:30:12 Tower kernel: ? check_preempt_wakeup+0x12b/0x179
May 16 08:30:12 Tower kernel: ? check_preempt_curr+0x2a/0x71
May 16 08:30:12 Tower kernel: btrfs_worker_helper+0xc2/0x185
May 16 08:30:12 Tower kernel: process_one_work+0x14c/0x23f
May 16 08:30:12 Tower kernel: ? rescuer_thread+0x258/0x258
May 16 08:30:12 Tower kernel: worker_thread+0x1c3/0x292
May 16 08:30:12 Tower kernel: kthread+0x111/0x119
May 16 08:30:12 Tower kernel: ? kthread_create_on_node+0x3a/0x3a
May 16 08:30:12 Tower kernel: ? call_usermodehelper_exec_async+0x116/0x11d
May 16 08:30:12 Tower kernel: ret_from_fork+0x35/0x40
May 16 08:30:12 Tower kernel: Code: ff 86 e8 02 00 00 48 89 c7 e8 52 fc 3a 00 48 8b 03 45 85 ed be 03 00 00 00 48 8b 78 10 75 05 be 04 00 00 00 e8 9e 95 fd ff eb 02 <0f> ff 48 8b 83 90 00 00 00 44 8b a8 5c 02 00 00 45 85 ed 74 0b 
May 16 08:30:12 Tower kernel: ---[ end trace 4497dd8cdf1a55c8 ]---
May 16 08:30:12 Tower kernel: ------------[ cut here ]------------
May 16 08:30:12 Tower kernel: WARNING: CPU: 8 PID: 79 at fs/btrfs/scrub.c:2044 scrub_bio_end_io_worker+0x237/0x3d4
May 16 08:30:12 Tower kernel: Modules linked in: xfs nfsd lockd grace sunrpc md_mod ipmi_devintf nct6775 hwmon_vid mlx4_en mlx4_core igb ptp pps_core ipmi_ssif sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper ast ttm drm_kms_helper cryptd drm intel_cstate intel_uncore intel_rapl_perf i2c_i801 agpgart i2c_algo_bit i2c_core syscopyarea sysfillrect cdc_acm sysimgblt fb_sys_fops ahci libahci wmi acpi_power_meter ipmi_si button [last unloaded: mlx4_core]
May 16 08:30:12 Tower kernel: CPU: 8 PID: 79 Comm: kworker/u24:3 Tainted: G        W       4.14.16-unRAID #1
May 16 08:30:12 Tower kernel: Hardware name: Supermicro X10SRA-F/X10SRA-F, BIOS 2.0b 06/12/2017
May 16 08:30:12 Tower kernel: Workqueue: btrfs-scrub btrfs_scrub_helper
May 16 08:30:12 Tower kernel: task: ffff881038769c00 task.stack: ffffc90006574000
May 16 08:30:12 Tower kernel: RIP: 0010:scrub_bio_end_io_worker+0x237/0x3d4
May 16 08:30:12 Tower kernel: RSP: 0018:ffffc90006577df0 EFLAGS: 00010246
May 16 08:30:12 Tower kernel: RAX: 000000ffff880fd8 RBX: ffff880eb66ef000 RCX: 0000000000000001
May 16 08:30:12 Tower kernel: RDX: ffff880eb51f5480 RSI: 0001000000000000 RDI: ffff88103ec03200
May 16 08:30:12 Tower kernel: RBP: ffff880fec71f140 R08: 0000000000000001 R09: ffffffff8122d300
May 16 08:30:12 Tower kernel: R10: ffffea003ad9bb80 R11: ffff88103f2a0901 R12: ffff880fe99da800
May 16 08:30:12 Tower kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000003
May 16 08:30:12 Tower kernel: FS:  0000000000000000(0000) GS:ffff88103f400000(0000) knlGS:0000000000000000
May 16 08:30:12 Tower kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 16 08:30:12 Tower kernel: CR2: 00000000006517e0 CR3: 0000000001c0a004 CR4: 00000000001606e0
May 16 08:30:12 Tower kernel: Call Trace:
May 16 08:30:12 Tower kernel: ? check_preempt_wakeup+0x12b/0x179
May 16 08:30:12 Tower kernel: ? check_preempt_curr+0x2a/0x71
May 16 08:30:12 Tower kernel: btrfs_worker_helper+0xc2/0x185
May 16 08:30:12 Tower kernel: process_one_work+0x14c/0x23f
May 16 08:30:12 Tower kernel: ? rescuer_thread+0x258/0x258
May 16 08:30:12 Tower kernel: worker_thread+0x1c3/0x292
May 16 08:30:12 Tower kernel: kthread+0x111/0x119
May 16 08:30:12 Tower kernel: ? kthread_create_on_node+0x3a/0x3a
May 16 08:30:12 Tower kernel: ? call_usermodehelper_exec_async+0x116/0x11d
May 16 08:30:12 Tower kernel: ret_from_fork+0x35/0x40
May 16 08:30:12 Tower kernel: Code: ff 86 e8 02 00 00 48 89 c7 e8 52 fc 3a 00 48 8b 03 45 85 ed be 03 00 00 00 48 8b 78 10 75 05 be 04 00 00 00 e8 9e 95 fd ff eb 02 <0f> ff 48 8b 83 90 00 00 00 44 8b a8 5c 02 00 00 45 85 ed 74 0b 
May 16 08:30:12 Tower kernel: ---[ end trace 4497dd8cdf1a55c9 ]---
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4956, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4955, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4954, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4953, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4952, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4951, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4950, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4949, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4948, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4947, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4946, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4945, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4944, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4943, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4942, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4941, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4940, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4939, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4938, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4937, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4936, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4935, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4934, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4933, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4932, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4931, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4930, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4929, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4928, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4927, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4926, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4925, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4924, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4923, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4922, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4883, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4875, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798496768 on dev /dev/sde1, sector 647604640, root 4810, inode 259, offset 15611043840, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS error (device sdf1): bdev /dev/sde1 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
May 16 08:31:35 Tower kernel: BTRFS error (device sdf1): unable to fixup (regular) error at logical 4868798496768 on dev /dev/sde1
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4956, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4955, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4954, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4953, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4952, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4951, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4950, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4949, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4948, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4947, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4946, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4945, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4944, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4943, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4942, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4941, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4940, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4939, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4938, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4937, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4936, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4935, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4934, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4933, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4932, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4931, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4930, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4929, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4928, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4927, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4926, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4925, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4924, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4923, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4922, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4883, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4875, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS warning (device sdf1): checksum error at logical 4868798500864 on dev /dev/sde1, sector 647604648, root 4810, inode 259, offset 15611047936, length 4096, links 1 (path: vDesk-Scratch/vdisk1.img)
May 16 08:31:35 Tower kernel: BTRFS error (device sdf1): bdev /dev/sde1 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
May 16 08:31:35 Tower kernel: BTRFS error (device sdf1): unable to fixup (regular) error at logical 4868798500864 on dev /dev/sde1
May 16 08:32:18 Tower kernel: BTRFS critical (device sdf1): corrupt leaf, bad key order: block=4364604653568, root=1, slot=101
May 16 08:32:19 Tower kernel: BTRFS critical (device sdf1): corrupt leaf, bad key order: block=4364604653568, root=1, slot=101
May 16 08:40:02 Tower kernel: BTRFS critical (device sdf1): corrupt leaf, bad key order: block=4364604653568, root=1, slot=101
May 16 08:40:03 Tower kernel: BTRFS critical (device sdf1): corrupt leaf, bad key order: block=4364604653568, root=1, slot=101
May 16 08:47:54 Tower emhttpd: req (2): csrf_token=****************&title=System+Log&cmd=%2FwebGui%2Fscripts%2Ftail_log&arg1=syslog
May 16 08:47:54 Tower emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog
May 16 08:48:23 Tower emhttpd: req (3): csrf_token=****************&title=System+Log&cmd=%2FwebGui%2Fscripts%2Ftail_log&arg1=syslog
May 16 08:48:23 Tower emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog
May 16 08:49:16 Tower kernel: pcieport 0000:00:1c.7: Enabling MPC IRBNCE
May 16 08:49:16 Tower kernel: pcieport 0000:00:1c.7: Intel PCH root port ACS workaround enabled
May 16 08:51:08 Tower emhttpd: req (4): csrf_token=****************&title=System+Log&cmd=%2FwebGui%2Fscripts%2Ftail_log&arg1=syslog
May 16 08:51:08 Tower emhttpd: cmd: /usr/local/emhttp/plugins/dynamix/scripts/tail_log syslog

 

Edited by al_uk

Share this post


Link to post

Thanks Johnnie, much appreciated. Could I ask your opinion on a couple of things please? I had an unexplained lockup about 20 days ago, which was the 1st in 3 years, which may have started this off. Everything appeared to start up fine afterwards though.

 

1. The cache mounts ok and my VMs run ok. I only noticed the problem because I happened to look at the unraid log. I have the VMs in a subvolume and I am snapshotting them nightly. The VMs start up ok. If I try and copy the 60GB disk image file to another drive from the command line using cp, then it fails after 16G with a "error reading 'vdisk1.img': Input/output error

" error. Is there anything I can do to save this file? The snapshots have the same issue. At this point it looks like I need to rebuild all my VMs?

 

2. Is there anything I should/could have done differently to detect and resolve this earlier? Snapshotting has not saved me. Cache pooling has not saved me. I may be better off going back to a single XFS cache drive and script everything offline nightly to my other server.

 

Cheers!

 

 

Share this post


Link to post
16 minutes ago, al_uk said:

If I try and copy the 60GB disk image file to another drive from the command line using cp, then it fails after 16G with a "error reading 'vdisk1.img': Input/output error

btrfs  restore linked on the post above will copy the file, including the corruption, looks like checksums don't match on both mirrors, this is likely a hardware problem, like bad RAM, are you using ECC or non-ECC RAM?

Share this post


Link to post
23 minutes ago, al_uk said:

Is there anything I should/could have done differently to detect and resolve this earlier? Snapshotting has not saved me.

Forgot to add, are use just snapshotying or send/receive to another device? If using send /receive you would notice the error when it happened and be able to restore to a previous one.

Share this post


Link to post

I'm using ECC Ram. The drives are in hot swap cages but have not been disturbed for 6 months. I did get BTRFS send running to another drive, but stopped it while I got around to making the script a little better. I do have a snapshot from a couple of months ago.

 

BTRFS restore is running now... It has found a couple of files so far that it was "looping" on. If that completes, then is there anything else I need to do outside the VM, or should I just then do a chkdsk from within the windows VM?

 

If I had detected the problem earlier, do you think it would have helped - would BTRFS scrub have been more likely to fix it?

 

 

Share this post


Link to post
19 minutes ago, al_uk said:

If I had detected the problem earlier, do you think it would have helped - would BTRFS scrub have been more likely to fix it?

Difficult to say, but I do daily send/receive, so if there's a problem it will be noticed immediately, and I can revert to the previous day snapshot.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now