G Speed Posted November 20, 2019 Share Posted November 20, 2019 Help please So I noticed my cache was filing up... files weren't moving to Disk 2 Drive had 150gb of space left File was 60gb It starts moving and gets stuck at around 16gb then just stops That share is set for 80GB minimum space left I forgot what I was doing, but ended up getting an error on the drive "SMART" 187 - Reported uncorrect - 1 Nov 20 03:55:15 Servo kernel: BTRFS warning (device md2): csum failed root -9 ino 15705 off 302710784 csum 0x2ac15d26 expected csum 0x914939f8 mirror 1 Nov 20 03:55:15 Servo kernel: BTRFS warning (device md2): csum failed root -9 ino 15705 off 303038464 csum 0x2ac15d26 expected csum 0x445bb47b mirror 1 Nov 20 03:55:15 Servo kernel: BTRFS warning (device md2): csum failed root -9 ino 15705 off 302710784 csum 0x2ac15d26 expected csum 0x914939f8 mirror 1 Tried to Balance, but afterwhile it ends at this ^^^ When I check the disk.. I get no errors? btrfs dev stats /mnt/disk2 [/dev/md2].write_io_errs 0 [/dev/md2].read_io_errs 0 [/dev/md2].flush_io_errs 0 [/dev/md2].corruption_errs 0 [/dev/md2].generation_errs 0 Any other questions for me please ask! Thanks! Quote Link to comment
JorgeB Posted November 20, 2019 Share Posted November 20, 2019 Run a scrub on disk2 then grab and post full diagnostics. Quote Link to comment
G Speed Posted November 20, 2019 Author Share Posted November 20, 2019 1 hour ago, johnnie.black said: Run a scrub on disk2 then grab and post full diagnostics. I will get back to you in 12 hours lmao Quote Link to comment
G Speed Posted November 20, 2019 Author Share Posted November 20, 2019 Total to scrub: 6.78TiB Bytes scrubbed: 4.68TiB Rate: 159.87MiB/s Error summary: csum=1470 Corrected: 0 Uncorrectable: 0 Unverified: 0 Some log stuff.. Nov 20 11:59:24 Servo kernel: BTRFS warning (device md2): checksum error at logical 2546538983424 on dev /dev/md2, physical 2546538983424, root 5, inode 364, offset 38792761344, length 4096, links 1 (path: 4K Movies/4K Unwatched/Batman.Vs.Teenage.Mutant.Ninja.Turtles.2019.2160p.Remux.HEVC.DTS-HD.MA.5.1-FraMeSToR.mkv) Nov 20 11:59:24 Servo kernel: BTRFS warning (device md2): checksum error at logical 2546539167744 on dev /dev/md2, physical 2546539167744, root 5, inode 364, offset 38792945664, length 4096, links 1 (path: 4K Movies/4K Unwatched/Batman.Vs.Teenage.Mutant.Ninja.Turtles.2019.2160p.Remux.HEVC.DTS-HD.MA.5.1-FraMeSToR.mkv) Nov 20 11:59:24 Servo kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0 Nov 20 11:59:24 Servo kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0 Nov 20 11:59:24 Servo kernel: BTRFS warning (device md2): checksum error at logical 2546539298816 on dev /dev/md2, physical 2546539298816, root 5, inode 364, offset 38793076736, length 4096, links 1 (path: 4K Movies/4K Unwatched/Batman.Vs.Teenage.Mutant.Ninja.Turtles.2019.2160p.Remux.HEVC.DTS-HD.MA.5.1-FraMeSToR.mkv) Nov 20 11:59:24 Servo kernel: BTRFS warning (device md2): checksum error at logical 2546539036672 on dev /dev/md2, physical 2546539036672, root 5, inode 364, offset 38792814592, length 4096, links 1 (path: 4K Movies/4K Unwatched/Batman.Vs.Teenage.Mutant.Ninja.Turtles.2019.2160p.Remux.HEVC.DTS-HD.MA.5.1-FraMeSToR.mkv) Nov 20 11:59:24 Servo kernel: BTRFS warning (device md2): checksum error at logical 2546539429888 on dev /dev/md2, physical 2546539429888, root 5, inode 364, offset 38793207808, length 4096, links 1 (path: 4K Movies/4K Unwatched/Batman.Vs.Teenage.Mutant.Ninja.Turtles.2019.2160p.Remux.HEVC.DTS-HD.MA.5.1-FraMeSToR.mkv) Nov 20 11:59:24 Servo kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0 Nov 20 11:59:24 Servo kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0 Nov 20 11:59:24 Servo kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 5, gen 0 Nov 20 11:59:24 Servo kernel: BTRFS warning (device md2): checksum error at logical 2546539560960 on dev /dev/md2, physical 2546539560960, root 5, inode 364, offset 38793338880, length 4096, links 1 (path: 4K Movies/4K Unwatched/Batman.Vs.Teenage.Mutant.Ninja.Turtles.2019.2160p.Remux.HEVC.DTS-HD.MA.5.1-FraMeSToR.mkv) Nov 20 11:59:24 Servo kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 6, gen 0 Nov 20 11:59:24 Servo kernel: BTRFS warning (device md2): checksum error at logical 2546538987520 on dev /dev/md2, physical 2546538987520, root 5, inode 364, offset 38792765440, length 4096, links 1 (path: 4K Movies/4K Unwatched/Batman.Vs.Teenage.Mutant.Ninja.Turtles.2019.2160p.Remux.HEVC.DTS-HD.MA.5.1-FraMeSToR.mkv) Nov 20 11:59:24 Servo kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0 Nov 20 11:59:24 Servo kernel: BTRFS warning (device md2): checksum error at logical 2546539171840 on dev /dev/md2, physical 2546539171840, root 5, inode 364, offset 38792949760, length 4096, links 1 (path: 4K Movies/4K Unwatched/Batman.Vs.Teenage.Mutant.Ninja.Turtles.2019.2160p.Remux.HEVC.DTS-HD.MA.5.1-FraMeSToR.mkv) Nov 20 11:59:24 Servo kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 8, gen 0 Nov 20 11:59:24 Servo kernel: BTRFS warning (device md2): checksum error at logical 2546538991616 on dev /dev/md2, physical 2546538991616, root 5, inode 364, offset 38792769536, length 4096, links 1 (path: 4K Movies/4K Unwatched/Batman.Vs.Teenage.Mutant.Ninja.Turtles.2019.2160p.Remux.HEVC.DTS-HD.MA.5.1-FraMeSToR.mkv) Nov 20 11:59:24 Servo kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 9, gen 0 Nov 20 11:59:24 Servo kernel: BTRFS warning (device md2): checksum error at logical 2546539040768 on dev /dev/md2, physical 2546539040768, root 5, inode 364, offset 38792818688, length 4096, links 1 (path: 4K Movies/4K Unwatched/Batman.Vs.Teenage.Mutant.Ninja.Turtles.2019.2160p.Remux.HEVC.DTS-HD.MA.5.1-FraMeSToR.mkv) Nov 20 11:59:24 Servo kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 10, gen 0 Nov 20 13:15:21 Servo kernel: mdcmd (83): spindown 3 Nov 20 14:42:40 Servo kernel: scrub_handle_errored_block: 975 callbacks suppressed Nov 20 14:42:40 Servo kernel: BTRFS warning (device md2): checksum error at logical 4256738496512 on dev /dev/md2, physical 4180502827008, root 5, inode 426, offset 19786149888, length 4096, links 1 (path: 4K Movies/Jack.Reacher.Never.Go.Back.2016.2160p.UHD.BluRay.REMUX.HDR.HEVC.Atmos-EPSiLON.mkv) Nov 20 14:42:40 Servo kernel: btrfs_dev_stat_print_on_error: 976 callbacks suppressed Nov 20 14:42:40 Servo kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 987, gen 0 Nov 20 14:42:40 Servo kernel: BTRFS warning (device md2): checksum error at logical 4256738574336 on dev /dev/md2, physical 4180502904832, root 5, inode 426, offset 19786227712, length 4096, links 1 (path: 4K Movies/Jack.Reacher.Never.Go.Back.2016.2160p.UHD.BluRay.REMUX.HDR.HEVC.Atmos-EPSiLON.mkv) Nov 20 14:42:40 Servo kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 988, gen 0 Nov 20 14:42:40 Servo kernel: BTRFS warning (device md2): checksum error at logical 4256738836480 on dev /dev/md2, physical 4180503166976, root 5, inode 426, offset 19786489856, length 4096, links 1 (path: 4K Movies/Jack.Reacher.Never.Go.Back.2016.2160p.UHD.BluRay.REMUX.HDR.HEVC.Atmos-EPSiLON.mkv) Nov 20 14:42:40 Servo kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 989, gen 0 Nov 20 14:42:40 Servo kernel: BTRFS warning (device md2): checksum error at logical 4256738967552 on dev /dev/md2, physical 4180503298048, root 5, inode 426, offset 19786620928, length 4096, links 1 (path: 4K Movies/Jack.Reacher.Never.Go.Back.2016.2160p.UHD.BluRay.REMUX.HDR.HEVC.Atmos-EPSiLON.mkv) Nov 20 14:42:40 Servo kernel: BTRFS warning (device md2): checksum error at logical 4256738705408 on dev /dev/md2, physical 4180503035904, root 5, inode 426, offset 19786358784, length 4096, links 1 (path: 4K Movies/Jack.Reacher.Never.Go.Back.2016.2160p.UHD.BluRay.REMUX.HDR.HEVC.Atmos-EPSiLON.mkv) Nov 20 14:42:40 Servo kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 990, gen 0 Nov 20 14:42:40 Servo kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 991, gen 0 Nov 20 14:42:40 Servo kernel: BTRFS warning (device md2): checksum error at logical 4256739098624 on dev /dev/md2, physical 4180503429120, root 5, inode 426, offset 19786752000, length 4096, links 1 (path: 4K Movies/Jack.Reacher.Never.Go.Back.2016.2160p.UHD.BluRay.REMUX.HDR.HEVC.Atmos-EPSiLON.mkv) Nov 20 14:42:40 Servo kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 992, gen 0 Nov 20 14:42:40 Servo kernel: BTRFS warning (device md2): checksum error at logical 4256738500608 on dev /dev/md2, physical 4180502831104, root 5, inode 426, offset 19786153984, length 4096, links 1 (path: 4K Movies/Jack.Reacher.Never.Go.Back.2016.2160p.UHD.BluRay.REMUX.HDR.HEVC.Atmos-EPSiLON.mkv) Nov 20 14:42:40 Servo kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 993, gen 0 Nov 20 14:42:40 Servo kernel: BTRFS warning (device md2): checksum error at logical 4256738578432 on dev /dev/md2, physical 4180502908928, root 5, inode 426, offset 19786231808, length 4096, links 1 (path: 4K Movies/Jack.Reacher.Never.Go.Back.2016.2160p.UHD.BluRay.REMUX.HDR.HEVC.Atmos-EPSiLON.mkv) Nov 20 14:42:40 Servo kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 994, gen 0 Nov 20 14:42:40 Servo kernel: BTRFS warning (device md2): checksum error at logical 4256738504704 on dev /dev/md2, physical 4180502835200, root 5, inode 426, offset 19786158080, length 4096, links 1 (path: 4K Movies/Jack.Reacher.Never.Go.Back.2016.2160p.UHD.BluRay.REMUX.HDR.HEVC.Atmos-EPSiLON.mkv) Nov 20 14:42:40 Servo kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 995, gen 0 Nov 20 14:42:40 Servo kernel: BTRFS warning (device md2): checksum error at logical 4256738709504 on dev /dev/md2, physical 4180503040000, root 5, inode 426, offset 19786362880, length 4096, links 1 (path: 4K Movies/Jack.Reacher.Never.Go.Back.2016.2160p.UHD.BluRay.REMUX.HDR.HEVC.Atmos-EPSiLON.mkv) Nov 20 14:42:40 Servo kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 996, gen 0 Quote Link to comment
JorgeB Posted November 21, 2019 Share Posted November 21, 2019 Those files are corrupt and will need replacing, would still like to see full diags when it's done. Quote Link to comment
JorgeB Posted November 21, 2019 Share Posted November 21, 2019 Also, and I was waiting for the diags to confirm, but if you don't have ECC RAM first thing to do now is to run a memtest. Quote Link to comment
G Speed Posted November 21, 2019 Author Share Posted November 21, 2019 5 hours ago, johnnie.black said: Those files are corrupt and will need replacing, would still like to see full diags when it's done. Syslog attached, did you need more? Also Memtest is done Pass syslog.txt Quote Link to comment
JorgeB Posted November 21, 2019 Share Posted November 21, 2019 15 minutes ago, G Speed said: did you need more? Please post full diags: Tools -> Diagnostics Quote Link to comment
JorgeB Posted November 21, 2019 Share Posted November 21, 2019 There are some recent UNC @ LBA errors on disk2, about 3 days ago, I would recommend running a scrub on at least another array disk, if errors are limited to disk2 then it was likely a disk problem, if they appear on other disks it's another hardware issue, most likely bad RAM. Quote Link to comment
G Speed Posted November 21, 2019 Author Share Posted November 21, 2019 46 minutes ago, johnnie.black said: There are some recent UNC @ LBA errors on disk2, about 3 days ago, I would recommend running a scrub on at least another array disk, if errors are limited to disk2 then it was likely a disk problem, if they appear on other disks it's another hardware issue, most likely bad RAM. Yeah, 3 days ago mover was trying to move a file over to disk 2.. and it would just get stuck. On that note.... Is Disk 2 done? Or Do I need to do that format to a diff fs, then back to btfrs? 187 Reported_Uncorrect - doesn't seem serious provided it doesn't change.. Quote Link to comment
JorgeB Posted November 21, 2019 Share Posted November 21, 2019 2 minutes ago, G Speed said: Is Disk 2 done? A single UNC @ LBA error doesn't mean the disk is done, much more worrying would be the disk corrupting data, disks should never return corrupt data, they should give a read error, though it's been known to happen. You don't need to format the disk, just need to replace (or delete) the corrupt files listed on the scrub, then run an extended SMART test and if OK keep monitoring the disk, but any more errors on the near future I would replace. Quote Link to comment
JorgeB Posted November 21, 2019 Share Posted November 21, 2019 And don't forget to scrub at least another disk to make sure problems are limited to disk2. Quote Link to comment
G Speed Posted November 21, 2019 Author Share Posted November 21, 2019 1 hour ago, johnnie.black said: And don't forget to scrub at least another disk to make sure problems are limited to disk2. Okay, I will do one of the smaller drives as it takes less time + it's still on the same card/breakout cable Quote Link to comment
G Speed Posted November 21, 2019 Author Share Posted November 21, 2019 1 hour ago, johnnie.black said: A single UNC @ LBA error doesn't mean the disk is done, much more worrying would be the disk corrupting data, disks should never return corrupt data, they should give a read error, though it's been known to happen. You don't need to format the disk, just need to replace (or delete) the corrupt files listed on the scrub, then run an extended SMART test and if OK keep monitoring the disk, but any more errors on the near future I would replace. What was causing the transfer to stall out though? 64gb file 16.7gb would transfer.. Tried it a few times.. Ended up moving the file to another disk with no problems Quote Link to comment
JorgeB Posted November 22, 2019 Share Posted November 22, 2019 That part is kind of strange, since a checksum error will abort a read, it shouldn't happen on writes, I can't find that operation on the syslog, do you have a syslog showing it? Quote Link to comment
G Speed Posted November 22, 2019 Author Share Posted November 22, 2019 1 hour ago, johnnie.black said: That part is kind of strange, since a checksum error will abort a read, it shouldn't happen on writes, I can't find that operation on the syslog, do you have a syslog showing it? I do not I do recall seeing the log, and nothing was thrown up...? Quote Link to comment
JorgeB Posted November 22, 2019 Share Posted November 22, 2019 There should be something logged, if it happens again save the syslog. Quote Link to comment
G Speed Posted November 22, 2019 Author Share Posted November 22, 2019 4 hours ago, johnnie.black said: There should be something logged, if it happens again save the syslog. i'm 99.1% sure, there was nothing.. I recalled checking after a few times... That was a few times with mover... Then a few times with Krusader Cache Drive -> Disk 2 Anywho, I scaned another disk on the same breakout cable.. all good.. I deleted those files, rescanned Disk 2.. 0 errors.. I will fill up disk 2 again to see if happens again Thanks again for your help! I will post back tomorrow afer I filled the drive Quote Link to comment
G Speed Posted November 24, 2019 Author Share Posted November 24, 2019 (edited) So I removed all the corrupted files, and replaced them with the same... So I was exactly at 158gb again, then tried to move that 64gb file again "I also redownloaded" I'm doing another scrub, to see if any new files etc. messed up I looked at diagnostics, Disk 2 no "new" errors or issues? servo-diagnostics-20191124-1511.zip Edited November 24, 2019 by G Speed Quote Link to comment
JorgeB Posted November 25, 2019 Share Posted November 25, 2019 Scrub finished without errors so everything is fine now, you can also reset disk2 stats with: btrfs dev stats -z /dev/md2 So it's much easier to see if new errors come up in the future. Quote Link to comment
G Speed Posted November 25, 2019 Author Share Posted November 25, 2019 1 hour ago, johnnie.black said: Scrub finished without errors so everything is fine now, you can also reset disk2 stats with: btrfs dev stats -z /dev/md2 So it's much easier to see if new errors come up in the future. It was still running Error summary: csum=231 Corrected: 0 Uncorrectable: 0 Unverified: 0 Diagnostic attached, after it finished Should I scrub with repair enabled? servo-diagnostics-20191125-0844.zip Quote Link to comment
JorgeB Posted November 25, 2019 Share Posted November 25, 2019 21 minutes ago, G Speed said: It was still running Ahh, though it had finished, btrfs can't correct the errors since each disk is an independent filesystem without redundancy, you need to delete or replace the affected files, they are identified on the syslog, but if these are new files you need to find out why data is being corrupted, is corruption limited to this disk? Quote Link to comment
G Speed Posted November 25, 2019 Author Share Posted November 25, 2019 6 hours ago, johnnie.black said: Ahh, though it had finished, btrfs can't correct the errors since each disk is an independent filesystem without redundancy, you need to delete or replace the affected files, they are identified on the syslog, but if these are new files you need to find out why data is being corrupted, is corruption limited to this disk? First things first - how come I can't click SMART on any of my disks? Extended test short everything is greyed out Quote Link to comment
JorgeB Posted November 25, 2019 Share Posted November 25, 2019 4 minutes ago, G Speed said: First things first - how come I can't click SMART on any of my disks? Extended test short everything is greyed out Because or controller is set to raid mode and is using the raid driver, it should be re-flashed to IT mode. Quote Link to comment
G Speed Posted November 25, 2019 Author Share Posted November 25, 2019 8 minutes ago, johnnie.black said: Because or controller is set to raid mode and is using the raid driver, it should be re-flashed to IT mode. It is IT mode? H310 Flashed Even on boot it shows 6 - Non-Raid Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.