Hey there,

 

Suddenly I can't start my home assistant VM. I was on vacation when my unraid server went offline.

 

I tried replacing the Cache drive and restarting from scratch via backups. Unfortunately with docker or VM's running, I get the below.

 

I tried replacing the SATA cable.

 

May 29 14:56:46 racer5 kernel: BTRFS warning (device sdc1): csum failed root 5 ino 102864 off 191668224 csum 0x8667902b expected csum 0xbc5b4684 mirror 1
May 29 14:56:46 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 474, gen 0
May 29 14:56:46 racer5 kernel: BTRFS warning (device sdc1): csum failed root 5 ino 102864 off 191668224 csum 0x8667902b expected csum 0xbc5b4684 mirror 1
May 29 14:56:46 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 475, gen 0
May 29 14:56:46 racer5 kernel: BTRFS warning (device sdc1): csum failed root 5 ino 102864 off 191668224 csum 0x8667902b expected csum 0xbc5b4684 mirror 1
May 29 14:56:46 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 476, gen 0
May 29 14:56:46 racer5 kernel: BTRFS warning (device sdc1): csum failed root 5 ino 102864 off 191668224 csum 0x8667902b expected csum 0xbc5b4684 mirror 1
May 29 14:56:46 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 477, gen 0
May 29 14:56:46 racer5 kernel: BTRFS warning (device sdc1): csum failed root 5 ino 102864 off 191668224 csum 0x8667902b expected csum 0xbc5b4684 mirror 1
May 29 14:56:46 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 478, gen 0
May 29 14:57:16 racer5 kernel: BTRFS warning (device sdc1): csum failed root 5 ino 102864 off 191668224 csum 0x8667902b expected csum 0xbc5b4684 mirror 1
May 29 14:57:16 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 479, gen 0
May 29 14:57:16 racer5 kernel: BTRFS warning (device sdc1): csum failed root 5 ino 102864 off 191668224 csum 0x8667902b expected csum 0xbc5b4684 mirror 1
May 29 14:57:16 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 480, gen 0
May 29 14:57:16 racer5 kernel: BTRFS warning (device sdc1): csum failed root 5 ino 102864 off 191668224 csum 0x8667902b expected csum 0xbc5b4684 mirror 1
May 29 14:57:16 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 481, gen 0
May 29 14:57:47 racer5 kernel: BTRFS warning (device sdc1): csum failed root 5 ino 102864 off 191668224 csum 0x8667902b expected csum 0xbc5b4684 mirror 1
May 29 14:57:47 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 482, gen 0
May 29 14:57:47 racer5 kernel: BTRFS warning (device sdc1): csum failed root 5 ino 102864 off 191668224 csum 0x8667902b expected csum 0xbc5b4684 mirror 1
May 29 14:57:47 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 483, gen 0
May 29 14:57:47 racer5 kernel: BTRFS warning (device sdc1): csum failed root 5 ino 102864 off 191668224 csum 0x8667902b expected csum 0xbc5b4684 mirror 1
May 29 14:57:47 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 484, gen 0

 

Does anyone have any ideas on how to fix this?

 

btrfs check --readonly /dev/sdc1
Opening filesystem to check...
Checking filesystem on /dev/sdc1
UUID: bd5f1f25-4aa0-4c44-b0ed-cc48da178f60
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space tree
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 63641214976 bytes used, no error found
total csum bytes: 41113636
total tree bytes: 62963712
total fs tree bytes: 17170432
total extent tree bytes: 2179072
btree space waste bytes: 5528641
file data blocks allocated: 105935065088
 referenced 63543685120

 

Smartctl passes fine.

racer5-diagnostics-20240529-1504.zip

Btrfs is detecting data corruption, run a correcting scrub and post the results, also a good idea to run memtest.

Thanks Jorge.

 

Unfortunately they appear to be unfix-able.

 

    UUID:             bd5f1f25-4aa0-4c44-b0ed-cc48da178f60
    Scrub started:    Thu May 30 05:21:15 2024
    Status:           finished
    Duration:         0:01:52
    Total to scrub:   59.33GiB
    Rate:             542.44MiB/s
    Error summary:    csum=329
      Corrected:      0
      Uncorrectable:  329
      Unverified:     0

 

 

 

May 30 05:22:10 racer5 kernel: BTRFS warning (device sdc1): checksum error at logical 73484632064 on dev /dev/sdc1, physical 33764573184, root 5, inode 81409, offset 23402172416, length 4096, links 1 (path: appdata/serge/weights/Mixtral-8X7B-Instruct-v0_1.bin)
May 30 05:22:10 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1026, gen 0
May 30 05:22:10 racer5 kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 73484632064 on dev /dev/sdc1
May 30 05:22:10 racer5 kernel: BTRFS warning (device sdc1): checksum error at logical 73484054528 on dev /dev/sdc1, physical 33763995648, root 5, inode 81409, offset 23401594880, length 4096, links 1 (path: appdata/serge/weights/Mixtral-8X7B-Instruct-v0_1.bin)
May 30 05:22:10 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1027, gen 0
May 30 05:22:10 racer5 kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 73484054528 on dev /dev/sdc1
May 30 05:22:10 racer5 kernel: BTRFS warning (device sdc1): checksum error at logical 73503387648 on dev /dev/sdc1, physical 33783328768, root 5, inode 81409, offset 23420928000, length 4096, links 1 (path: appdata/serge/weights/Mixtral-8X7B-Instruct-v0_1.bin)
May 30 05:22:10 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1028, gen 0
May 30 05:22:10 racer5 kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 73503387648 on dev /dev/sdc1
May 30 05:22:10 racer5 kernel: BTRFS warning (device sdc1): checksum error at logical 73503330304 on dev /dev/sdc1, physical 33783271424, root 5, inode 81409, offset 23420870656, length 4096, links 1 (path: appdata/serge/weights/Mixtral-8X7B-Instruct-v0_1.bin)
May 30 05:22:10 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1029, gen 0
May 30 05:22:10 racer5 kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 73503330304 on dev /dev/sdc1
May 30 05:22:10 racer5 kernel: BTRFS warning (device sdc1): checksum error at logical 73523089408 on dev /dev/sdc1, physical 33803030528, root 5, inode 81409, offset 23440629760, length 4096, links 1 (path: appdata/serge/weights/Mixtral-8X7B-Instruct-v0_1.bin)
May 30 05:22:10 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1030, gen 0
May 30 05:22:10 racer5 kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 73523089408 on dev /dev/sdc1
May 30 05:22:10 racer5 kernel: BTRFS warning (device sdc1): checksum error at logical 73523093504 on dev /dev/sdc1, physical 33803034624, root 5, inode 81409, offset 23440633856, length 4096, links 1 (path: appdata/serge/weights/Mixtral-8X7B-Instruct-v0_1.bin)
May 30 05:22:10 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1031, gen 0
May 30 05:22:10 racer5 kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 73523093504 on dev /dev/sdc1
May 30 05:22:16 racer5 kernel: BTRFS warning (device sdc1): checksum error at logical 76412858368 on dev /dev/sdc1, physical 36692799488, root 5, inode 81409, offset 26313261056, length 4096, links 1 (path: appdata/serge/weights/Mixtral-8X7B-Instruct-v0_1.bin)
May 30 05:22:16 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1032, gen 0
May 30 05:22:16 racer5 kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 76412858368 on dev /dev/sdc1
May 30 05:22:16 racer5 kernel: BTRFS warning (device sdc1): checksum error at logical 76483960832 on dev /dev/sdc1, physical 36763901952, root 5, inode 81409, offset 26384363520, length 4096, links 1 (path: appdata/serge/weights/Mixtral-8X7B-Instruct-v0_1.bin)
May 30 05:22:16 racer5 kernel: BTRFS warning (device sdc1): checksum error at logical 76483854336 on dev /dev/sdc1, physical 36763795456, root 5, inode 81409, offset 26384257024, length 4096, links 1 (path: appdata/serge/weights/Mixtral-8X7B-Instruct-v0_1.bin)
May 30 05:22:16 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1033, gen 0
May 30 05:22:16 racer5 kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 76483960832 on dev /dev/sdc1
May 30 05:22:16 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1034, gen 0
May 30 05:22:16 racer5 kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 76483854336 on dev /dev/sdc1
May 30 05:22:16 racer5 kernel: BTRFS warning (device sdc1): checksum error at logical 76591443968 on dev /dev/sdc1, physical 36871385088, root 5, inode 84302, offset 16424960, length 4096, links 1 (path: appdata/Plex-Media-Server/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db-2024-05-14)
May 30 05:22:16 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1035, gen 0
May 30 05:22:16 racer5 kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 76591443968 on dev /dev/sdc1
May 30 05:22:16 racer5 kernel: BTRFS warning (device sdc1): checksum error at logical 76719296512 on dev /dev/sdc1, physical 36999237632, root 5, inode 84306, offset 1011712, length 4096, links 1 (path: appdata/Plex-Media-Server/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db-2024-05-20)
May 30 05:22:16 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1036, gen 0
May 30 05:22:16 racer5 kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 76719296512 on dev /dev/sdc1
May 30 05:22:16 racer5 kernel: BTRFS warning (device sdc1): checksum error at logical 76877938688 on dev /dev/sdc1, physical 37157879808, root 5, inode 84310, offset 14839808, length 4096, links 1 (path: appdata/Plex-Media-Server/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.blobs.db)
May 30 05:22:16 racer5 kernel: BTRFS warning (device sdc1): checksum error at logical 76877897728 on dev /dev/sdc1, physical 37157838848, root 5, inode 84310, offset 14798848, length 4096, links 1 (path: appdata/Plex-Media-Server/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.blobs.db)
May 30 05:22:16 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1037, gen 0
May 30 05:22:16 racer5 kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 76877938688 on dev /dev/sdc1
May 30 05:22:16 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1038, gen 0
May 30 05:22:16 racer5 kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 76877897728 on dev /dev/sdc1
May 30 05:22:16 racer5 kernel: BTRFS warning (device sdc1): checksum error at logical 76877967360 on dev /dev/sdc1, physical 37157908480, root 5, inode 84310, offset 14868480, length 4096, links 1 (path: appdata/Plex-Media-Server/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.blobs.db)
May 30 05:22:16 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1039, gen 0
May 30 05:22:16 racer5 kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 76877967360 on dev /dev/sdc1
May 30 05:22:19 racer5 kernel: BTRFS warning (device sdc1): checksum error at logical 78149033984 on dev /dev/sdc1, physical 38428975104, root 5, inode 86827, offset 1560576, length 4096, links 1 (path: appdata/Plex-Media-Server/Library/Application Support/Plex Media Server/Cache/PhotoTranscoder/ab/ab9666ff6b34977202cab44b05fe1728d65303e0.jpg)
May 30 05:22:19 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1040, gen 0
May 30 05:22:19 racer5 kernel: BTRFS warning (device sdc1): checksum error at logical 78149079040 on dev /dev/sdc1, physical 38429020160, root 5, inode 86827, offset 1605632, length 4096, links 1 (path: appdata/Plex-Media-Server/Library/Application Support/Plex Media Server/Cache/PhotoTranscoder/ab/ab9666ff6b34977202cab44b05fe1728d65303e0.jpg)
May 30 05:22:19 racer5 kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 78149033984 on dev /dev/sdc1
May 30 05:22:19 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1041, gen 0
May 30 05:22:19 racer5 kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 78149079040 on dev /dev/sdc1
May 30 05:23:00 racer5 kernel: scrub_handle_errored_block: 33 callbacks suppressed
May 30 05:23:00 racer5 kernel: BTRFS warning (device sdc1): checksum error at logical 101532372992 on dev /dev/sdc1, physical 61812314112, root 5, inode 102864, offset 1014145024, length 4096, links 1 (path: domains/Home Assistant/haos_ova-12.3.qcow2)
May 30 05:23:00 racer5 kernel: btrfs_dev_stat_inc_and_print: 33 callbacks suppressed
May 30 05:23:00 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1075, gen 0
May 30 05:23:00 racer5 kernel: scrub_handle_errored_block: 33 callbacks suppressed
May 30 05:23:00 racer5 kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 101532372992 on dev /dev/sdc1
May 30 05:23:00 racer5 kernel: BTRFS warning (device sdc1): checksum error at logical 101549453312 on dev /dev/sdc1, physical 61829394432, root 5, inode 102864, offset 1031225344, length 4096, links 1 (path: domains/Home Assistant/haos_ova-12.3.qcow2)
May 30 05:23:00 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1076, gen 0
May 30 05:23:00 racer5 kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 101549453312 on dev /dev/sdc1
May 30 05:23:00 racer5 kernel: BTRFS warning (device sdc1): checksum error at logical 101549461504 on dev /dev/sdc1, physical 61829402624, root 5, inode 102864, offset 1031233536, length 4096, links 1 (path: domains/Home Assistant/haos_ova-12.3.qcow2)
May 30 05:23:00 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1077, gen 0
May 30 05:23:00 racer5 kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 101549461504 on dev /dev/sdc1
May 30 05:23:00 racer5 kernel: BTRFS warning (device sdc1): checksum error at logical 101852925952 on dev /dev/sdc1, physical 62132867072, root 5, inode 102864, offset 858279936, length 4096, links 1 (path: domains/Home Assistant/haos_ova-12.3.qcow2)
May 30 05:23:00 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1078, gen 0
May 30 05:23:00 racer5 kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 101852925952 on dev /dev/sdc1
May 30 05:23:00 racer5 kernel: BTRFS warning (device sdc1): checksum error at logical 101852930048 on dev /dev/sdc1, physical 62132871168, root 5, inode 102864, offset 858284032, length 4096, links 1 (path: domains/Home Assistant/haos_ova-12.3.qcow2)
May 30 05:23:00 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1079, gen 0
May 30 05:23:00 racer5 kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 101852930048 on dev /dev/sdc1
May 30 05:23:00 racer5 kernel: BTRFS warning (device sdc1): checksum error at logical 101901234176 on dev /dev/sdc1, physical 62181175296, root 5, inode 102864, offset 906588160, length 4096, links 1 (path: domains/Home Assistant/haos_ova-12.3.qcow2)
May 30 05:23:00 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1080, gen 0
May 30 05:23:00 racer5 kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 101901234176 on dev /dev/sdc1
May 30 05:23:00 racer5 kernel: BTRFS warning (device sdc1): checksum error at logical 101947916288 on dev /dev/sdc1, physical 62227857408, root 5, inode 102864, offset 953270272, length 4096, links 1 (path: domains/Home Assistant/haos_ova-12.3.qcow2)
May 30 05:23:00 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1081, gen 0
May 30 05:23:00 racer5 kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 101947916288 on dev /dev/sdc1
May 30 05:23:00 racer5 kernel: BTRFS warning (device sdc1): checksum error at logical 101947920384 on dev /dev/sdc1, physical 62227861504, root 5, inode 102864, offset 953274368, length 4096, links 1 (path: domains/Home Assistant/haos_ova-12.3.qcow2)
May 30 05:23:00 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1082, gen 0
May 30 05:23:00 racer5 kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 101947920384 on dev /dev/sdc1
May 30 05:23:01 racer5 kernel: BTRFS warning (device sdc1): checksum error at logical 102041509888 on dev /dev/sdc1, physical 62321451008, root 5, inode 102864, offset 1170399232, length 4096, links 1 (path: domains/Home Assistant/haos_ova-12.3.qcow2)
May 30 05:23:01 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1083, gen 0
May 30 05:23:01 racer5 kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 102041509888 on dev /dev/sdc1
May 30 05:23:01 racer5 kernel: BTRFS warning (device sdc1): checksum error at logical 102041513984 on dev /dev/sdc1, physical 62321455104, root 5, inode 102864, offset 1170403328, length 4096, links 1 (path: domains/Home Assistant/haos_ova-12.3.qcow2)
May 30 05:23:01 racer5 kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 0, rd 0, flush 0, corrupt 1084, gen 0
May 30 05:23:01 racer5 kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 102041513984 on dev /dev/sdc1

 

I can't run a memtest at present. I'll have to look into that later when I can throw a GPU in here.

You will need to delete/restore the corrupt files listed in the syslog, then run another scrub to confirm 0 errors, but I would recommend memtest first.