Hi you guys,

 

I have a problem - my VM Disk Images are on the Cachepool.

Since yesterday I cannot copy these files to the Array, I get Errors.

 

Nov  9 23:25:39 CHUnraidBIG kernel: BTRFS warning (device sdc1): csum failed root 5 ino 271 off 18753646592 csum 0xcf84ebed expected csum 0x909f2020 mirror 2
Nov  9 23:25:39 CHUnraidBIG kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 20910, gen 0
Nov  9 23:25:39 CHUnraidBIG kernel: BTRFS warning (device sdc1): csum failed root 5 ino 271 off 18753650688 csum 0xa6f629f1 expected csum 0x8188ffff mirror 2
Nov  9 23:25:39 CHUnraidBIG kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 20911, gen 0
Nov  9 23:25:39 CHUnraidBIG kernel: BTRFS warning (device sdc1): csum failed root 5 ino 271 off 18753646592 csum 0xcf84ebed expected csum 0x909f2020 mirror 1
Nov  9 23:25:39 CHUnraidBIG kernel: BTRFS error (device sdc1): bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 20910, gen 0
Nov  9 23:25:39 CHUnraidBIG kernel: BTRFS warning (device sdc1): csum failed root 5 ino 271 off 18753650688 csum 0xa6f629f1 expected csum 0x8188ffff mirror 1
Nov  9 23:25:39 CHUnraidBIG kernel: BTRFS error (device sdc1): bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 20911, gen 0
Nov  9 23:25:39 CHUnraidBIG kernel: BTRFS warning (device sdc1): csum failed root 5 ino 271 off 18753646592 csum 0xcf84ebed expected csum 0x909f2020 mirror 2

 

What shall I do to recover my files ?

 

Diagnostics are attached.

 

Thanks in advance

 

Hanns

chunraidbig-diagnostics-20241109-2343.zip

Btrfs is detecting data corruption on both devices, this is typically bad RAM, but you appear to be running ECC RAM, run a correcting scrub on the pool and post the results.

Nov 10 11:04:02 CHUnraidBIG ool www[45170]: /usr/local/emhttp/plugins/dynamix/scripts/btrfs_scrub 'start' '/mnt/cache' ''
Nov 10 11:04:02 CHUnraidBIG kernel: BTRFS info (device sdc1): scrub: started on devid 1
Nov 10 11:04:02 CHUnraidBIG kernel: BTRFS info (device sdc1): scrub: started on devid 2
Nov 10 11:19:26 CHUnraidBIG kernel: BTRFS warning (device sdc1): checksum error at logical 445373927424 on dev /dev/sdc1, physical 445373927424, root 5, inode 271, offset 18753646592, length 4096, links 1 (path: domains/Windows Server 2019DC/vdisk1.img)
Nov 10 11:19:26 CHUnraidBIG kernel: BTRFS warning (device sdc1): checksum error at logical 445373927424 on dev /dev/sdd1, physical 445352955904, root 5, inode 271, offset 18753646592, length 4096, links 1 (path: domains/Windows Server 2019DC/vdisk1.img)
Nov 10 11:19:26 CHUnraidBIG kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 20924, gen 0
Nov 10 11:19:26 CHUnraidBIG kernel: BTRFS error (device sdc1): bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 20924, gen 0
Nov 10 11:19:26 CHUnraidBIG kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 445373927424 on dev /dev/sdc1
Nov 10 11:19:26 CHUnraidBIG kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 445373927424 on dev /dev/sdd1
Nov 10 11:19:26 CHUnraidBIG kernel: BTRFS warning (device sdc1): checksum error at logical 445373931520 on dev /dev/sdc1, physical 445373931520, root 5, inode 271, offset 18753650688, length 4096, links 1 (path: domains/Windows Server 2019DC/vdisk1.img)
Nov 10 11:19:26 CHUnraidBIG kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 20925, gen 0
Nov 10 11:19:26 CHUnraidBIG kernel: BTRFS warning (device sdc1): checksum error at logical 445373931520 on dev /dev/sdd1, physical 445352960000, root 5, inode 271, offset 18753650688, length 4096, links 1 (path: domains/Windows Server 2019DC/vdisk1.img)
Nov 10 11:19:26 CHUnraidBIG kernel: BTRFS error (device sdc1): bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 20925, gen 0
Nov 10 11:19:26 CHUnraidBIG kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 445373931520 on dev /dev/sdc1
Nov 10 11:19:26 CHUnraidBIG kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 445373931520 on dev /dev/sdd1
Nov 10 11:53:33 CHUnraidBIG kernel: BTRFS warning (device sdc1): checksum error at logical 1475972390912 on dev /dev/sdc1, physical 1475972390912, root 5, inode 272, offset 993067491328, length 4096, links 1 (path: domains/Windows Server 2019DC/vdisk2.img)
Nov 10 11:53:33 CHUnraidBIG kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 20926, gen 0
Nov 10 11:53:33 CHUnraidBIG kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 1475972390912 on dev /dev/sdc1
Nov 10 11:55:01 CHUnraidBIG kernel: BTRFS warning (device sdc1): checksum error at logical 1475972390912 on dev /dev/sdd1, physical 1475951419392, root 5, inode 272, offset 993067491328, length 4096, links 1 (path: domains/Windows Server 2019DC/vdisk2.img)
Nov 10 11:55:01 CHUnraidBIG kernel: BTRFS error (device sdc1): bdev /dev/sdd1 errs: wr 0, rd 0, flush 0, corrupt 20926, gen 0
Nov 10 11:55:01 CHUnraidBIG kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 1475972390912 on dev /dev/sdd1
Nov 10 12:22:39 CHUnraidBIG kernel: BTRFS info (device sdc1): scrub: finished on devid 1 with status: 0
Nov 10 12:25:27 CHUnraidBIG kernel: BTRFS info (device sdc1): scrub: finished on devid 2 with status: 0

After long hours I found out that the Backup did not work either for quite some time 😞
It is very important that I read the contents of that image !

Can I open it regardless of the errors ? I need these files - sadly

 

5 hours ago, Schulmeister said:

After long hours I found out that the Backup did not work either for quite some time 😞
It is very important that I read the contents of that image !

Can I open it regardless of the errors ? I need these files - sadly

 

I was able to copy the contents of the img-file, thanks to this wonderful forum.

 

I figured it out.  I needed to specify the byte offset of where the partition begins.  For anyone who might have the same question in the future, here is what I did.

 

From the unRAID command console, display partition information of the vdisk:

fdisk -l /mnt/disks/Windows/vdisk1.img

 

I was after the values in red.  The output will looks something like this:

[pre]Disk vdisk1.img: 20 GiB, 21474836480 bytes, 41943040 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disklabel type: dos

Disk identifier: 0xda00352d

 

Device      Boot  Start      End  Sectors  Size Id Type

vdisk1.img1 *      2048  206847  204800  100M  7 HPFS/NTFS/exFAT

vdisk1.img2      206848 41940991 41734144 19.9G  7 HPFS/NTFS/exFAT[/pre]

 

To find the offset in bytes, multiply the sector start value by the sector size to get the offset in bytes.  In this case, I wanted to mount vdisk1.img2. 

206848 * 512 = 105906176

 

Final command to mount the vdisk NTFS partition as read-only:

mount -r -t ntfs -o loop,offset=105906176 /mnt/disks/Windows/vdisk1.img /mnt/test

 

*********************

I still need to copy the .img-files regardless of errors - is that possible ?

You should be able to get them with btrfs restore:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=543490

 

For this

1 hour ago, JorgeB said:

You should be able to get them with btrfs restore:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=543490

 

do I need to unmount the pool ?
How do I do that ?

 

Stop the array to unmount the pool.

I just tried something else, and as far as I can tell - it worked.

I copied the files via SMB on another computer and opened the Images with no problems ?

 

So my pressing needs are solved and I thank you very much for you help.

I now am on the task to find out why !!! Why the heck does this happen ? It should not happen, I have ECC Rams, two SSDs (not cheap ones) in a Raid1 and still this errors ?

 

Any ideas ?

 

What shal I do to test my hardware ?

8 hours ago, Schulmeister said:

and opened the Images with no problems ?

They will still be corrupt, it may or not be in an important area.

 

Keep monitoring the pool for more issues, if new corruptions are detected there's still a problem:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=700582

 

Conclusion to this topic.

Dear Friends, I have been struggeling with that issue for some time and it happened a few times.

I thought it was a hardware issue and changed it - same issue after some time.

Now I am back on my old hardware, set up a Proxmox Virtualizer and keep Unraid running as follows:

16 Cores, 32GB RAM, a virtual Disk as cachedrive and Flashdrive (Unraid USB Stick) and HDDs passthrough.

Unraid - with it's genius Harddrive management is my NAS and my Docker machine.

If I need VMs that's where Proxmox has the advantage.

 

That combination runs now without any issues for months and I am very happy.

 

I hope that this helps.