okay i need some serious immediate help!!!!


55 posts in this topic Last Reply

Recommended Posts

  • Replies 54
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

That's a hardware problem, most likely a connection issue, try replacing the cables on that disk.

By default metadata uses the DUP profile to be redundant, but with memory corruption both can be corrupt, if it's using dup a scrub (with repair enable) will correct it, if it can't be fixed best to r

crap, it can't be fixed.  but thanks for all the help, you are life saver man.

@JorgeBthanks to u i have successfully restored my disk1 and now im running scrub on disk2 to know which files are corrupt on disk2 so that i can replace them but i noticed that my also getting these types of error in between, are they showing cause of corruption and they will go away once i get rid of those corrupt files or their is something that going wrong too.

Mar 18 16:03:43 Unraid kernel: ata2.00: failed command: READ FPDMA QUEUED
Mar 18 16:03:43 Unraid kernel: ata2.00: cmd 60/20:e8:60:b6:0c/00:00:00:00:00/40 tag 29 ncq dma 16384 in
Mar 18 16:03:43 Unraid kernel:         res 40/00:20:80:b7:0c/00:00:00:00:00/40 Emask 0x50 (ATA bus error)
Mar 18 16:03:43 Unraid kernel: ata2.00: status: { DRDY }
Mar 18 16:03:43 Unraid kernel: ata2.00: failed command: READ FPDMA QUEUED
Mar 18 16:03:43 Unraid kernel: ata2.00: cmd 60/20:f0:80:b6:0c/00:00:00:00:00/40 tag 30 ncq dma 16384 in
Mar 18 16:03:43 Unraid kernel:         res 40/00:20:80:b7:0c/00:00:00:00:00/40 Emask 0x50 (ATA bus error)
Mar 18 16:03:43 Unraid kernel: ata2.00: status: { DRDY }
Mar 18 16:03:43 Unraid kernel: ata2.00: failed command: READ FPDMA QUEUED
Mar 18 16:03:43 Unraid kernel: ata2.00: cmd 60/40:f8:a0:b6:0c/00:00:00:00:00/40 tag 31 ncq dma 32768 in
Mar 18 16:03:43 Unraid kernel:         res 40/00:20:80:b7:0c/00:00:00:00:00/40 Emask 0x50 (ATA bus error)
Mar 18 16:03:43 Unraid kernel: ata2.00: status: { DRDY }
Mar 18 16:03:43 Unraid kernel: ata2: hard resetting link
Mar 18 16:03:49 Unraid kernel: ata2: link is slow to respond, please be patient (ready=0)
Mar 18 16:03:50 Unraid kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Mar 18 16:03:50 Unraid kernel: ata2.00: configured for UDMA/133
Mar 18 16:03:50 Unraid kernel: ata2: EH complete

 

Link to post
1 hour ago, JorgeB said:

That's a hardware problem, most likely a connection issue, try replacing the cables on that disk.

okay, replaced the cable on that disk, now scrub is not showing those errors but i still get these 6 verify error at the very beginning of scrub, these don't say anything about corrupt file, so what should i do to get rid of these.

Mar 18 17:59:44 Unraid kernel: BTRFS warning (device md2): checksum/header error at logical 622215168 on dev /dev/md2, physical 630603776: metadata leaf (level 0) in tree 7
Mar 18 17:59:44 Unraid kernel: BTRFS warning (device md2): checksum/header error at logical 622215168 on dev /dev/md2, physical 630603776: metadata leaf (level 0) in tree 7
Mar 18 17:59:44 Unraid kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 48346, gen 68
Mar 18 17:59:44 Unraid kernel: BTRFS warning (device md2): checksum/header error at logical 622870528 on dev /dev/md2, physical 631259136: metadata leaf (level 0) in tree 7
Mar 18 17:59:44 Unraid kernel: BTRFS warning (device md2): checksum/header error at logical 622870528 on dev /dev/md2, physical 631259136: metadata leaf (level 0) in tree 7
Mar 18 17:59:44 Unraid kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 48346, gen 69
Mar 18 17:59:44 Unraid kernel: BTRFS warning (device md2): checksum/header error at logical 623017984 on dev /dev/md2, physical 631406592: metadata leaf (level 0) in tree 7
Mar 18 17:59:44 Unraid kernel: BTRFS warning (device md2): checksum/header error at logical 623017984 on dev /dev/md2, physical 631406592: metadata leaf (level 0) in tree 7
Mar 18 17:59:44 Unraid kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 48346, gen 70
Mar 18 17:59:49 Unraid kernel: BTRFS warning (device md2): checksum/header error at logical 622215168 on dev /dev/md2, physical 1704345600: metadata leaf (level 0) in tree 7
Mar 18 17:59:49 Unraid kernel: BTRFS warning (device md2): checksum/header error at logical 622215168 on dev /dev/md2, physical 1704345600: metadata leaf (level 0) in tree 7
Mar 18 17:59:49 Unraid kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 48346, gen 71
Mar 18 17:59:49 Unraid kernel: BTRFS warning (device md2): checksum/header error at logical 622870528 on dev /dev/md2, physical 1705000960: metadata leaf (level 0) in tree 7
Mar 18 17:59:49 Unraid kernel: BTRFS warning (device md2): checksum/header error at logical 622870528 on dev /dev/md2, physical 1705000960: metadata leaf (level 0) in tree 7
Mar 18 17:59:49 Unraid kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 48346, gen 72
Mar 18 17:59:49 Unraid kernel: BTRFS warning (device md2): checksum/header error at logical 623017984 on dev /dev/md2, physical 1705148416: metadata leaf (level 0) in tree 7
Mar 18 17:59:49 Unraid kernel: BTRFS warning (device md2): checksum/header error at logical 623017984 on dev /dev/md2, physical 1705148416: metadata leaf (level 0) in tree 7
Mar 18 17:59:49 Unraid kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 48346, gen 73

 

Link to post
38 minutes ago, JorgeB said:

Those are corrupt metadata:

 

is their any way to fix metadata ?? or i will have to do same thing that did with disk1, i.e. backup and restore.

Link to post

By default metadata uses the DUP profile to be redundant, but with memory corruption both can be corrupt, if it's using dup a scrub (with repair enable) will correct it, if it can't be fixed best to reformat.

Link to post
  • Max changed the title to (Solved) okay i need some serious immediate help!!!!
34 minutes ago, JorgeB said:

By default metadata uses the DUP profile to be redundant, but with memory corruption both can be corrupt, if it's using dup a scrub (with repair enable) will correct it, if it can't be fixed best to reformat.

crap, it can't be fixed. 

but thanks for all the help, you are life saver man.

Link to post
  • Max changed the title to okay i need some serious immediate help!!!!

@JorgeB 

Mar 21 04:18:34 Unraid kernel: BTRFS warning (device md2): csum failed root 5 ino 3611 off 39428833280 csum 0x03d6ab02 expected csum 0xe395e029 mirror 1
Mar 21 04:18:34 Unraid kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
Mar 21 04:18:34 Unraid kernel: BTRFS warning (device md2): csum failed root 5 ino 3611 off 39428845568 csum 0x11013b58 expected csum 0xaa1a5420 mirror 1
Mar 21 04:18:34 Unraid kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
Mar 21 04:18:34 Unraid kernel: BTRFS warning (device md2): csum failed root 5 ino 3611 off 39428866048 csum 0xd226d08a expected csum 0x63b2c2c3 mirror 1
Mar 21 04:18:34 Unraid kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
Mar 21 04:18:34 Unraid kernel: BTRFS warning (device md2): csum failed root 5 ino 3611 off 39428886528 csum 0x33980e71 expected csum 0x4c775c09 mirror 1
Mar 21 04:18:34 Unraid kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
Mar 21 04:18:34 Unraid kernel: BTRFS warning (device md2): csum failed root 5 ino 3611 off 39428907008 csum 0x1863e21f expected csum 0x3027eacb mirror 1
Mar 21 04:18:34 Unraid kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 5, gen 0
Mar 21 04:18:34 Unraid kernel: BTRFS warning (device md2): csum failed root 5 ino 3611 off 39428968448 csum 0x337f1b41 expected csum 0x2619b33a mirror 1
Mar 21 04:18:34 Unraid kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 6, gen 0
Mar 21 04:18:34 Unraid kernel: BTRFS warning (device md2): csum failed root 5 ino 3611 off 39429296128 csum 0xc0486401 expected csum 0xa6614d88 mirror 1
Mar 21 04:18:34 Unraid kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0
Mar 21 04:18:34 Unraid kernel: BTRFS warning (device md2): csum failed root 5 ino 3611 off 39429357568 csum 0x7e844d92 expected csum 0xb62cd92e mirror 1
Mar 21 04:18:34 Unraid kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 8, gen 0
Mar 21 04:18:34 Unraid kernel: BTRFS warning (device md2): csum failed root 5 ino 3611 off 39429582848 csum 0x8ae91f99 expected csum 0xf55f5d7c mirror 1
Mar 21 04:18:34 Unraid kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 9, gen 0
Mar 21 04:18:34 Unraid kernel: BTRFS warning (device md2): csum failed root 5 ino 3611 off 39429869568 csum 0x75e706ad expected csum 0xda1d3603 mirror 1
Mar 21 04:18:34 Unraid kernel: BTRFS error (device md2): bdev /dev/md2 errs: wr 0, rd 0, flush 0, corrupt 10, gen 0

looks like it wasn't my ram cause they are back, so far only disk 2 is showing these errors, so i would like to fix this asap.

Any suggestions  ??

Link to post
4 minutes ago, JorgeB said:

If it's not RAM board and CPU would be the next suspects.

ohhhh boy😭.

would it be possible that somehow i managed to copy some corrupted data and thats why its showing up now, cause when was backing up my data their were some which failed to copy so thought they were corrupt as well but they didn't show up as corrupted under scrub.

and is it possible for it to have anything to do with 6.9.1 cause as far as i remember they are were pretty close to each other like less then 24 hours apart.

Link to post

Any new corruptions can only be new, scrub scan all the data and compares checksumns, if they fail later it's new.

3 minutes ago, Max said:

and is it possible for it to have anything to do with 6.9.1

I would say no, but of course can't say it with 100% confidence, about 99.99%.

Link to post
2 hours ago, Max said:

looks like it wasn't my ram cause they are back

This is after swapping old RAM with new one correct? Or just using different settings?

Link to post

@JorgeB hey today when those errors occured i noticed one strange thing which is both today and yesterday when those errors occurred ca backup and restore plugin was creating backup and the only file on entire server (that includes both disk drives and cache drive) was the one it created.

and another thing that i noticed is that both the time, first it showed these errors.

Mar 22 14:52:25 Unraid kernel: ata3.00: exception Emask 0x50 SAct 0x771e SErr 0x4890800 action 0xe frozen
Mar 22 14:52:25 Unraid kernel: ata3.00: irq_stat 0x0c400040, interface fatal error, connection status changed
Mar 22 14:52:25 Unraid kernel: ata3: SError: { HostInt PHYRdyChg 10B8B LinkSeq DevExch }
Mar 22 14:52:25 Unraid kernel: ata3.00: failed command: READ FPDMA QUEUED
Mar 22 14:52:25 Unraid kernel: ata3.00: cmd 60/40:08:f8:40:5f/05:00:e0:00:00/40 tag 1 ncq dma 688128 in
Mar 22 14:52:25 Unraid kernel:         res 40/00:70:f8:01:5f/00:00:e0:00:00/40 Emask 0x50 (ATA bus error)
Mar 22 14:52:25 Unraid kernel: ata3.00: status: { DRDY }
Mar 22 14:52:25 Unraid kernel: ata3.00: failed command: READ FPDMA QUEUED
Mar 22 14:52:25 Unraid kernel: ata3.00: cmd 60/40:10:38:46:5f/05:00:e0:00:00/40 tag 2 ncq dma 688128 in
Mar 22 14:52:25 Unraid kernel:         res 40/00:70:f8:01:5f/00:00:e0:00:00/40 Emask 0x50 (ATA bus error)
Mar 22 14:52:25 Unraid kernel: ata3.00: status: { DRDY }
Mar 22 14:52:25 Unraid kernel: ata3.00: failed command: READ FPDMA QUEUED
Mar 22 14:52:25 Unraid kernel: ata3.00: cmd 60/40:18:78:4b:5f/05:00:e0:00:00/40 tag 3 ncq dma 688128 in
Mar 22 14:52:25 Unraid kernel:         res 40/00:70:f8:01:5f/00:00:e0:00:00/40 Emask 0x50 (ATA bus error)
Mar 22 14:52:25 Unraid kernel: ata3.00: status: { DRDY }
Mar 22 14:52:25 Unraid kernel: ata3.00: failed command: READ FPDMA QUEUED
Mar 22 14:52:25 Unraid kernel: ata3.00: cmd 60/40:20:b8:50:5f/05:00:e0:00:00/40 tag 4 ncq dma 688128 in
Mar 22 14:52:25 Unraid kernel:         res 40/00:70:f8:01:5f/00:00:e0:00:00/40 Emask 0x50 (ATA bus error)
Mar 22 14:52:25 Unraid kernel: ata3.00: status: { DRDY }
Mar 22 14:52:25 Unraid kernel: ata3.00: failed command: READ FPDMA QUEUED
Mar 22 14:52:25 Unraid kernel: ata3.00: cmd 60/08:40:f8:55:5f/00:00:e0:00:00/40 tag 8 ncq dma 4096 in
Mar 22 14:52:25 Unraid kernel:         res 40/00:70:f8:01:5f/00:00:e0:00:00/40 Emask 0x50 (ATA bus error)
Mar 22 14:52:25 Unraid kernel: ata3.00: status: { DRDY }
Mar 22 14:52:25 Unraid kernel: ata3.00: failed command: READ FPDMA QUEUED
Mar 22 14:52:25 Unraid kernel: ata3.00: cmd 60/a0:48:00:56:5f/02:00:e0:00:00/40 tag 9 ncq dma 344064 in
Mar 22 14:52:25 Unraid kernel:         res 40/00:70:f8:01:5f/00:00:e0:00:00/40 Emask 0x50 (ATA bus error)
Mar 22 14:52:25 Unraid kernel: ata3.00: status: { DRDY }
Mar 22 14:52:25 Unraid kernel: ata3.00: failed command: WRITE FPDMA QUEUED
Mar 22 14:52:25 Unraid kernel: ata3.00: cmd 61/40:50:38:f2:5e/05:00:e0:00:00/40 tag 10 ncq dma 688128 out
Mar 22 14:52:25 Unraid kernel:         res 40/00:70:f8:01:5f/00:00:e0:00:00/40 Emask 0x50 (ATA bus error)
Mar 22 14:52:25 Unraid kernel: ata3.00: status: { DRDY }
Mar 22 14:52:25 Unraid kernel: ata3.00: failed command: WRITE FPDMA QUEUED
Mar 22 14:52:25 Unraid kernel: ata3.00: cmd 61/40:60:78:f7:5e/05:00:e0:00:00/40 tag 12 ncq dma 688128 out
Mar 22 14:52:25 Unraid kernel:         res 40/00:70:f8:01:5f/00:00:e0:00:00/40 Emask 0x50 (ATA bus error)
Mar 22 14:52:25 Unraid kernel: ata3.00: status: { DRDY }
Mar 22 14:52:25 Unraid kernel: ata3.00: failed command: WRITE FPDMA QUEUED
Mar 22 14:52:25 Unraid kernel: ata3.00: cmd 61/40:68:b8:fc:5e/05:00:e0:00:00/40 tag 13 ncq dma 688128 out
Mar 22 14:52:25 Unraid kernel:         res 40/00:70:f8:01:5f/00:00:e0:00:00/40 Emask 0x50 (ATA bus error)
Mar 22 14:52:25 Unraid kernel: ata3.00: status: { DRDY }
Mar 22 14:52:25 Unraid kernel: ata3.00: failed command: WRITE FPDMA QUEUED
Mar 22 14:52:25 Unraid kernel: ata3.00: cmd 61/40:70:f8:01:5f/05:00:e0:00:00/40 tag 14 ncq dma 688128 out
Mar 22 14:52:25 Unraid kernel:         res 40/00:70:f8:01:5f/00:00:e0:00:00/40 Emask 0x50 (ATA bus error)
Mar 22 14:52:25 Unraid kernel: ata3.00: status: { DRDY }
Mar 22 14:52:25 Unraid kernel: ata3: hard resetting link
Mar 22 14:52:31 Unraid kernel: ata3: link is slow to respond, please be patient (ready=0)
Mar 22 14:52:32 Unraid kernel: ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Mar 22 14:52:32 Unraid kernel: ata3.00: configured for UDMA/133
Mar 22 14:52:32 Unraid kernel: ata3: EH complete

only then it showed corruption errors, so i thought maybe the sata cable that i replaced on disk2 is also faulty, so i bought new sata cable to check just in case but it is still showing those errors is disk2 only could it be my disk2 going bonkers.

Link to post
29 minutes ago, JorgeB said:

Disks shouldn't corrupt data when there are those type of errors, but if there's something wrong with that disk it's a possibility.

okay so what do u suggest should i claim warranty on disk2  cause so far after replacing ram, only disk 2 is showing data corruption plus those errors. i changed sata port, i have changed sata cable (twice actually) but its still showing errors on only that drive. and i think if it was my board then these would show up randomly on any drive or it would be limited to one of the sata port on board but its not showing up on other drives and its not limited to one port, it is sticking to drive in particular thats disk 2 which would be really weird for board or cpu failure.

Link to post

For now it's only that disk, but before there was corruption on 2 disks, which suggests another issue, unless it was the RAM also, but quite unusual to have two different things corrupting data, do you have another disk you could use in place of that one?

Link to post
34 minutes ago, JorgeB said:

For now it's only that disk, but before there was corruption on 2 disks, which suggests another issue, unless it was the RAM also, 

yeah i did had a thought, that maybe RAM plus disk2 is going bonkers but yeah i know, for two seprate parts to go bad plus to both of them to actaully only corrupt data is quite unusual.

38 minutes ago, JorgeB said:

do you have another disk you could use in place of that one?

i thought of that too but unfortunately i don't have one. would making my backup share disk1 or maybe cache only help testing this as for now corruption is only occurring while backing up.

Link to post

@JorgeB update: so after limiting writes to disk1 and cache everything was going well, no error this time when backup/restore plugin ran but today i accidently shut off my server which resulted in unclean shutdown. so when when i started my array later it ran parity sync check without doing corrections and soon it started reporting many sync errors and it was running at like 4-5 megbabyte/sec and when i checked syslog it was full of these read errors from disk2.

Mar 23 12:10:55 Unraid kernel: blk_update_request: I/O error, dev sdd, sector 738691496 op 0x0:(READ) flags 0x0 phys_seg 85 prio class 0
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691432
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691440
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691448
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691456
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691464
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691472
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691480
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691488
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691496
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691504
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691512
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691520
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691528
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691536
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691544
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691552
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691560
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691568
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691576
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691584
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691592
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691600
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691608
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691616
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691624
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691632
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691640
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691648
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691656
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691664
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691672
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691680
Mar 23 12:10:55 Unraid kernel: md: disk2 read error, sector=738691688

im not showing all cause their were literally too of them plus of course those ata3 errors from earlier were also their.

Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.