Fix common problems caught a call trace error this morning. I am hoping someone can take a look at my diagnostics and give me an idea of what could be causing it. The systems is only about 3 months old, so I hope it's not a hardware issue.

 

thanks!

Since btrfs is mentioned on the trace run a scrub and correct any errors, if no errors are found do a read-only filesystem check on your cache, if there are filesystem issues don't try to fix them before backing up your cache.

Since btrfs is mentioned on the trace run a scrub and correct any errors, if no errors are found do a read-only filesystem check on your cache, if there are filesystem issues don't try to fix them before backing up your cache.

 

 

Thanks for the help johnnie. I have never run that command before. I assume I just follow the steps outlined below?

 

https://lime-technology.com/wiki/index.php/Check_Disk_Filesystems#btrfs_scrub

 

Not exactly sure how to the follow the syntax...

 

"btrfs scrub start -r" ?

You can use the GUI,  on the main page click on Cache.

The first indication of a problem is:

 

Feb 17 19:38:02 PageRAID kernel: transmission-da[15206]: segfault at 48 ip 0000561fe65cf0cc sp 00002b38859c49e8 error 4 in transmission-daemon[561fe659e000+72000]

 

Later there's another segfault:

 

Feb 18 04:39:31 PageRAID kernel: Plex Media Scan[15977]: segfault at 0 ip          (null) sp 00002ac4d31ff8d8 error 14 in Plex Media Scanner[400000+429000]

Feb 18 04:39:31 PageRAID kernel: Plex Media Scan[15975]: segfault at 2ac518d568ff ip 00002ac4cd8c142c sp 00007ffd27cd6f40 error 4 in libjemalloc.so.1[2ac4cd826000+133000]

 

Next, there's a page allocation problem, that results in the call trace:

 

Feb 18 21:57:06 PageRAID kernel: Threadpool work: page allocation stalls for 11617ms, order:0, mode:0x2400840(GFP_NOFS|__GFP_NOFAIL)

Feb 18 21:57:06 PageRAID kernel: CPU: 2 PID: 18229 Comm: Threadpool work Not tainted 4.9.10-unRAID #1

Feb 18 21:57:06 PageRAID kernel: Hardware name: Supermicro X10SL7-F/X10SL7-F, BIOS 3.0a 12/17/2015

Feb 18 21:57:06 PageRAID kernel: ffffc9001bdd33a0 ffffffff813a353e 0000000000000001 0000000000000000

Feb 18 21:57:06 PageRAID kernel: ffffc9001bdd3430 ffffffff810cb40d 02400840810c9c00 ffffffff8193d200

Feb 18 21:57:06 PageRAID kernel: ffffc9001bdd33c8 0000000000000010 ffffc9001bdd3440 ffffc9001bdd33e0

Feb 18 21:57:06 PageRAID kernel: Call Trace:

Feb 18 21:57:06 PageRAID kernel: [<ffffffff813a353e>] dump_stack+0x61/0x7e

Feb 18 21:57:06 PageRAID kernel: [<ffffffff810cb40d>] warn_alloc+0x102/0x116

Feb 18 21:57:06 PageRAID kernel: [<ffffffff810cb9c3>] __alloc_pages_nodemask+0x541/0xc71

Feb 18 21:57:06 PageRAID kernel: [<ffffffff8106f2ae>] ? __enqueue_entity+0x67/0x69

Feb 18 21:57:06 PageRAID kernel: [<ffffffff81073fb9>] ? put_prev_entity+0x4e/0x5f0

Feb 18 21:57:06 PageRAID kernel: [<ffffffff81076cb8>] ? pick_next_task_fair+0x107/0x3c4

Feb 18 21:57:06 PageRAID kernel: [<ffffffff81102ad2>] alloc_pages_current+0xbe/0xe8

Feb 18 21:57:06 PageRAID kernel: [<ffffffff810c4bff>] __page_cache_alloc+0x89/0x9f

Feb 18 21:57:06 PageRAID kernel: [<ffffffff810c4d53>] pagecache_get_page+0x13e/0x1e6

Feb 18 21:57:06 PageRAID kernel: [<ffffffff8130d37a>] alloc_extent_buffer+0xf7/0x375

Feb 18 21:57:06 PageRAID kernel: [<ffffffff812e7b6c>] btrfs_find_create_tree_block+0x10/0x12

Feb 18 21:57:06 PageRAID kernel: [<ffffffff812e7cc1>] read_tree_block+0x14/0x4c

Feb 18 21:57:06 PageRAID kernel: [<ffffffff812cd0bd>] read_block_for_search.isra.12+0x25b/0x296

Feb 18 21:57:06 PageRAID kernel: [<ffffffff81321fbc>] ? btrfs_clear_lock_blocking_rw+0x79/0xc1

Feb 18 21:57:06 PageRAID kernel: [<ffffffff812cf025>] btrfs_search_slot+0x710/0x803

Feb 18 21:57:06 PageRAID kernel: [<ffffffff812e345f>] btrfs_lookup_csum+0x3a/0x108

Feb 18 21:57:06 PageRAID kernel: [<ffffffff812e3756>] __btrfs_lookup_bio_sums+0x21a/0x451

Feb 18 21:57:06 PageRAID kernel: [<ffffffff812e3c79>] btrfs_lookup_bio_sums+0x11/0x13

Feb 18 21:57:06 PageRAID kernel: [<ffffffff812f13ce>] btrfs_submit_bio_hook+0xcc/0x145

Feb 18 21:57:06 PageRAID kernel: [<ffffffff81307e6c>] submit_one_bio+0x66/0x84

Feb 18 21:57:06 PageRAID kernel: [<ffffffff8130a87f>] submit_extent_page+0x9b/0x224

Feb 18 21:57:06 PageRAID kernel: [<ffffffff8130bdc2>] __do_readpage+0x62f/0x721

Feb 18 21:57:06 PageRAID kernel: [<ffffffff8130b2f5>] ? btrfs_create_repair_bio+0xc6/0xc6

Feb 18 21:57:06 PageRAID kernel: [<ffffffff812f1b9c>] ? inode_tree_add+0x140/0x140

Feb 18 21:57:06 PageRAID kernel: [<ffffffff8130c1d1>] __extent_readpages.constprop.20+0x23e/0x284

Feb 18 21:57:06 PageRAID kernel: [<ffffffff812f1b9c>] ? inode_tree_add+0x140/0x140

Feb 18 21:57:06 PageRAID kernel: [<ffffffff8130cd28>] extent_readpages+0x145/0x1ec

Feb 18 21:57:06 PageRAID kernel: [<ffffffff812f1b9c>] ? inode_tree_add+0x140/0x140

Feb 18 21:57:06 PageRAID kernel: [<ffffffff812f191b>] btrfs_readpages+0x1a/0x1c

Feb 18 21:57:06 PageRAID kernel: [<ffffffff810d0b1e>] __do_page_cache_readahead+0x15d/0x21f

Feb 18 21:57:06 PageRAID kernel: [<ffffffff810c672b>] filemap_fault+0x184/0x458

Feb 18 21:57:06 PageRAID kernel: [<ffffffff810c672b>] ? filemap_fault+0x184/0x458

Feb 18 21:57:06 PageRAID kernel: [<ffffffff810e8d03>] __do_fault+0x68/0xbb

Feb 18 21:57:06 PageRAID kernel: [<ffffffff810edd20>] handle_mm_fault+0x6b1/0xf96

Feb 18 21:57:06 PageRAID kernel: [<ffffffff810421cc>] __do_page_fault+0x24a/0x3ed

Feb 18 21:57:06 PageRAID kernel: [<ffffffff810423b2>] do_page_fault+0x22/0x27

Feb 18 21:57:06 PageRAID kernel: [<ffffffff8167ec98>] page_fault+0x28/0x30

 

So yes, definitely BTRFS issues but I recommend that you at least re-boot before trying to continue. I would run a MemTest before booting back into unRAID.

Forgot to mention, scrub is done with the cache mounted, if you do the check you need to start in maitnance mode, only then will the check button be active.

So yes, definitely BTRFS issues but I recommend that you at least re-boot before trying to continue. I would run a MemTest before booting back into unRAID.

 

He's using ECC Ram, rebooting won't hurt though.

So yes, definitely BTRFS issues but I recommend that you at least re-boot before trying to continue. I would run a MemTest before booting back into unRAID.

 

He's using ECC Ram, rebooting won't hurt though.

 

 

Thanks guys. I did stop the volume and do a full reboot. This seems to have removed the error from Fix Common Problems, but i am not sure if those get wiped on restart regardless.

 

I just finished the scrub of the cache pool and received the follow:

 

scrub status for 0c7e101c-e8ad-4ab9-9057-d898b9ebaddb

scrub started at Sun Feb 19 09:58:07 2017 and finished after 00:10:30

total bytes scrubbed: 535.26GiB with 0 errors

 

Next steps? An FSCK check?

Yes, read-only

 

Yes, read-only

 

 

Command line? Or is there a GUI option for this as well?

GUI, after starting in maintenance mode.

GUI, after starting in maintenance mode.

 

Got it. Need to step out for a bit, so I will run this later and report back.

 

Thanks again for all the help.

 

 

GUI, after starting in maintenance mode.

 

Results of the check in readonly ... looks to be ok I think...

 

checking extents

checking free space cache

checking fs roots

checking csums

checking root refs

Checking filesystem on /dev/sdb1

UUID: 0c7e101c-e8ad-4ab9-9057-d898b9ebaddb

found 287366225920 bytes used err is 0

total csum bytes: 67652988

total tree bytes: 509018112

total fs tree bytes: 347144192

total extent tree bytes: 71073792

btree space waste bytes: 118314197

file data blocks allocated: 2882781945856

referenced 268878364672

It's fine, you're already on latest unRAID release, so don't have anymore suggestions, continue using the server normally to see if the issue re-occurs.

It's fine, you're already on latest unRAID release, so don't have anymore suggestions, continue using the server normally to see if the issue re-occurs.

 

Works for me. Things seem to be running well.

 

Enjoy the rest of the weekend and thanks again.