hermy65

@jorgeb Thanks for all of the assistance here, hopefully I am finally past all of these problems!!

@jorgeb Ok, i deleted libvrt.img and then stopped/started and the vm page works and one of my vms came back so i should be good with that one. Were the BTRFS errors all pointing at the libvirt issue or was that something else i need to take care of?

@JorgeB Got the cache pool re-formatted, etc. Just noticed some more BTRFS errors, attached are my diagnostics storage-diagnostics-20240423-1113.zip Also, not sure if related but after rebuilding my cache pool my VMs tab no longer works? It doesnt load any of the vms i had or let me create vms, just a blank page.

I had FolderView installed for a while and it was working great. I had to rebuild my cache pool today and my docker.img and now my folders are gone. If i click on the Add Folder button it does nothing. Any ideas on what to do here?

@JorgeB thanks. Any reason to replace sdc1? Since that's the one that keeps having errors?

@JorgeB does that mean I should like format the cache drives and start over or how does a person recreate the filesystem?

@JorgeB woke up this morning to some more BTRFS errors, ive attached a new set of diagnostics. storage-diagnostics-20240420-0758.zip

@JorgeB deleted those files, re-ran a repair scrub and it came back clean. Also moved docker to ipvlan. Anything else that i need to do/watch for? Appreciate the help as always!

@JorgeB looks like its a couple pieces of plex and jellyfin metadata unless tthis is the wrong area to look at: Apr 19 08:49:34 Storage kernel: BTRFS warning (device sdc1): checksum error at logical 19965161472 on dev /dev/sdc1, physical 10267930624, root 5, inode 686348049, offset 4096, length 4096, links 1 (path: appdata/jellyfin/data/metadata/People/E/Eamon Sheehan/folder.jpg) Apr 19 08:49:34 Storage kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 335783, rd 191311, flush 1, corrupt 29971, gen 0 Apr 19 08:49:34 Storage kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 19965161472 on dev /dev/sdc1 Apr 19 08:49:34 Storage kernel: BTRFS warning (device sdc1): checksum error at logical 19965161472 on dev /dev/sdb1, physical 9206771712, root 5, inode 686348049, offset 4096, length 4096, links 1 (path: appdata/jellyfin/data/metadata/People/E/Eamon Sheehan/folder.jpg) Apr 19 08:49:34 Storage kernel: BTRFS error (device sdc1): bdev /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 6, gen 0 Apr 19 08:49:34 Storage kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 19965161472 on dev /dev/sdb1 Apr 19 08:49:34 Storage kernel: BTRFS warning (device sdc1): checksum error at logical 19965165568 on dev /dev/sdc1, physical 10267934720, root 5, inode 686348049, offset 8192, length 4096, links 1 (path: appdata/jellyfin/data/metadata/People/E/Eamon Sheehan/folder.jpg) Apr 19 08:49:34 Storage kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 335783, rd 191311, flush 1, corrupt 29972, gen 0 Apr 19 08:49:34 Storage kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 19965165568 on dev /dev/sdc1 Apr 19 08:49:34 Storage kernel: BTRFS warning (device sdc1): checksum error at logical 19965165568 on dev /dev/sdb1, physical 9206775808, root 5, inode 686348049, offset 8192, length 4096, links 1 (path: appdata/jellyfin/data/metadata/People/E/Eamon Sheehan/folder.jpg) Apr 19 08:49:34 Storage kernel: BTRFS error (device sdc1): bdev /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0 Apr 19 08:49:34 Storage kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 19965165568 on dev /dev/sdb1 Apr 19 08:48:46 Storage kernel: BTRFS warning (device sdc1): checksum error at logical 2367467520 on dev /dev/sdc1, physical 1293725696, root 5, inode 812538915, offset 20480, length 4096, links 1 (path: appdata/PlexMediaServer/Library/Application Support/Plex Media Server/Media/localhost/8/4b1112dba0e382f5a87080425e1a7ac0d711dec.bundle/Contents/GoP-0.xml) Apr 19 08:48:46 Storage kernel: BTRFS warning (device sdc1): checksum error at logical 2367467520 on dev /dev/sdb1, physical 199012352, root 5, inode 812538915, offset 20480, length 4096, links 1 (path: appdata/PlexMediaServer/Library/Application Support/Plex Media Server/Media/localhost/8/4b1112dba0e382f5a87080425e1a7ac0d711dec.bundle/Contents/GoP-0.xml) Apr 19 08:48:46 Storage kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 335783, rd 191311, flush 1, corrupt 29969, gen 0 Apr 19 08:48:46 Storage kernel: BTRFS error (device sdc1): bdev /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0 Apr 19 08:48:46 Storage kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 2367467520 on dev /dev/sdc1 Apr 19 08:48:46 Storage kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 2367467520 on dev /dev/sdb1 Apr 19 08:48:46 Storage kernel: BTRFS warning (device sdc1): checksum error at logical 2367471616 on dev /dev/sdc1, physical 1293729792, root 5, inode 812538915, offset 24576, length 4096, links 1 (path: appdata/PlexMediaServer/Library/Application Support/Plex Media Server/Media/localhost/8/4b1112dba0e382f5a87080425e1a7ac0d711dec.bundle/Contents/GoP-0.xml) Apr 19 08:48:46 Storage kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 335783, rd 191311, flush 1, corrupt 29970, gen 0 Apr 19 08:48:46 Storage kernel: BTRFS warning (device sdc1): checksum error at logical 2367471616 on dev /dev/sdb1, physical 199016448, root 5, inode 812538915, offset 24576, length 4096, links 1 (path: appdata/PlexMediaServer/Library/Application Support/Plex Media Server/Media/localhost/8/4b1112dba0e382f5a87080425e1a7ac0d711dec.bundle/Contents/GoP-0.xml) Apr 19 08:48:46 Storage kernel: BTRFS error (device sdc1): bdev /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 5, gen 0

@JorgeB Here are the results from the correcting scrub UUID: bbc56f07-1a5f-4d7b-b019-a515d7eb35aa Scrub started: Fri Apr 19 08:48:42 2024 Status: finished Duration: 0:39:21 Total to scrub: 1.26TiB Rate: 563.20MiB/s Error summary: csum=8 Corrected: 0 Uncorrectable: 8 Unverified: 0

@JorgeB I ran the command in terminal, received this response: Clearing log on /dev/sdc1, previous log_root 4906986635264, level 0 Stopped then restarted the array and it looks like my cache pool has come back online, here are the new diagnostics. storage-diagnostics-20240419-0838.zip

@JorgeB here are fresh diagnostics after a reboot. One thing i noticed is that now my entire cache pool says unmountable when it was working fine last night. Any ideas? storage-diagnostics-20240418-1501.zip

I had an SSD go bad in my cache pool last week so I swapped a new drive in and I think got everything fixed. Today I'm getting BTRFS errors and I'm not sure exactly what to do. Diagnostics attached storage-diagnostics-20240418-1054.zip

Yes, without the -n sorry. Appreciate the help!

@JorgeB Thanks, this is what the -n gave me Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - agno = 13 bad CRC for inode 27918115649 bad CRC for inode 27918115649, will rewrite cleared inode 27918115649 - agno = 14 - agno = 15 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 2 - agno = 8 - agno = 13 - agno = 4 - agno = 5 - agno = 7 - agno = 6 - agno = 9 - agno = 10 - agno = 12 - agno = 3 - agno = 14 - agno = 15 - agno = 11 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... done

Woke up this morning and my machine was completely unresponsive, monitor showed some sort of XFS errors but i didnt see a disk number on the screen. Managed to get in via putty and kick of diagnostics before rebooting. Not seeing any errors on the dashboard, no idea what happened or how to fix/prevent it. Diagnostics attached Thanks in advance! storage-diagnostics-20240415-0859.zip

@trurl ok so if i move it to cache and update the path i should be good to go, correct? Thanks again!

@trurl Interesting, checked the system/libvrt folder on cache and its empty, the one on disk17 has the .img in it and it was last modified after i started the array when you told me to. Is there a reason that .img file would be on disk17 and being used when the setting is as you posted above?

@trurl appreciate the help! The lost+found folders have been taken care of, appreciate the alert on that. As for the system folder, it looks like it has my libvrt.img in it for some reason, i assume i can move that to cache and then update the location in VM settings or will that cause an issue?

@trurl updated diagnostics are attached! storage-diagnostics-20240228-0945.zip

@trurl without -n, i see no mention of -L in this output. Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 bad CRC for inode 87062 inode identifier 16149622513293918207 mismatch on inode 87062 bad CRC for inode 87062, will rewrite inode identifier 16149622513293918207 mismatch on inode 87062 cleared inode 87062 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 1 - agno = 3 - agno = 4 - agno = 7 - agno = 6 - agno = 5 entry "11 - The Amity Affliction - Stairway to Hell.mp3" at block 0 offset 736 in directory inode 87051 references free inode 87062 clearing inode number in entry at offset 736... Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... bad hash table for directory inode 87051 (no data entry): rebuilding rebuilding directory inode 87051 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... done

@trurl here is the output: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 bad CRC for inode 87062 inode identifier 16149622513293918207 mismatch on inode 87062 bad CRC for inode 87062, would rewrite inode identifier 16149622513293918207 mismatch on inode 87062 would have cleared inode 87062 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 4 - agno = 2 - agno = 3 - agno = 5 - agno = 6 - agno = 7 - agno = 1 entry "11 - The Amity Affliction - Stairway to Hell.mp3" at block 0 offset 736 in directory inode 87051 references free inode 87062 would clear inode number in entry at offset 736... bad CRC for inode 87062, would rewrite inode identifier 16149622513293918207 mismatch on inode 87062 would have cleared inode 87062 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... entry "11 - The Amity Affliction - Stairway to Hell.mp3" in directory inode 87051 points to free inode 87062, would junk entry bad hash table for directory inode 87051 (no data entry): would rebuild would rebuild directory inode 87051 - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... No modify flag set, skipping filesystem flush and exiting.

Popped open my syslog this morning and its full of this but it doesnt list what disk its detecting the corruption on so i have no idea what to do. None of the disks show errors on the dashboard either. Diagnostics are attached. Feb 28 08:55:51 Storage kernel: XFS (md18p1): Metadata corruption detected at xfs_dinode_verify+0xa0/0x732 [xfs], inode 0x15416 dinode Feb 28 08:55:51 Storage kernel: XFS (md18p1): Unmount and run xfs_repair Feb 28 08:55:51 Storage kernel: XFS (md18p1): First 128 bytes of corrupted metadata buffer: Feb 28 08:55:51 Storage kernel: 00000000: 49 4e 81 ff 03 02 00 00 00 00 00 63 00 00 00 64 IN.........c...d Feb 28 08:55:51 Storage kernel: 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 ................ Feb 28 08:55:51 Storage kernel: 00000020: 5c e8 88 37 21 21 f6 d0 5c e8 88 12 1e 16 0a a8 \..7!!..\....... Feb 28 08:55:51 Storage kernel: 00000030: 5c e8 93 98 08 a2 4f 36 00 00 00 00 00 b7 90 e7 \.....O6........ Feb 28 08:55:51 Storage kernel: 00000040: 00 00 00 00 00 00 0b 7a 00 00 00 00 00 00 00 01 .......z........ Feb 28 08:55:51 Storage kernel: 00000050: 00 00 18 01 00 00 00 00 00 00 00 00 d5 4c 13 f1 .............L.. Feb 28 08:55:51 Storage kernel: 00000060: ff ff ff ff fa 0a 06 c8 00 00 00 00 00 00 00 0d ................ Feb 28 08:55:51 Storage kernel: 00000070: 00 00 00 01 00 18 ff 66 00 00 00 00 00 00 00 00 .......f........ Feb 28 08:55:51 Storage kernel: XFS (md18p1): Metadata corruption detected at xfs_dinode_verify+0xa0/0x732 [xfs], inode 0x15416 dinode Feb 28 08:55:51 Storage kernel: XFS (md18p1): Unmount and run xfs_repair Feb 28 08:55:51 Storage kernel: XFS (md18p1): First 128 bytes of corrupted metadata buffer: Feb 28 08:55:51 Storage kernel: 00000000: 49 4e 81 ff 03 02 00 00 00 00 00 63 00 00 00 64 IN.........c...d Feb 28 08:55:51 Storage kernel: 00000010: 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 ................ Feb 28 08:55:51 Storage kernel: 00000020: 5c e8 88 37 21 21 f6 d0 5c e8 88 12 1e 16 0a a8 \..7!!..\....... Feb 28 08:55:51 Storage kernel: 00000030: 5c e8 93 98 08 a2 4f 36 00 00 00 00 00 b7 90 e7 \.....O6........ Feb 28 08:55:51 Storage kernel: 00000040: 00 00 00 00 00 00 0b 7a 00 00 00 00 00 00 00 01 .......z........ Feb 28 08:55:51 Storage kernel: 00000050: 00 00 18 01 00 00 00 00 00 00 00 00 d5 4c 13 f1 .............L.. Feb 28 08:55:51 Storage kernel: 00000060: ff ff ff ff fa 0a 06 c8 00 00 00 00 00 00 00 0d ................ Feb 28 08:55:51 Storage kernel: 00000070: 00 00 00 01 00 18 ff 66 00 00 00 00 00 00 00 00 .......f........ storage-diagnostics-20240228-0851.zip

@JorgeB @itimpi The array started fine and Disk 1 is back and operational. Disk 8 though is still marked as disabled and emulated . I assume my next step is to force a rebuild on that disk? storage-diagnostics-20231207-1045.zip

@JorgeB Done - logs are below, what should i do next? disk 1 with -L did this: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used. - scan filesystem freespace and inode maps... clearing needsrepair flag and regenerating metadata sb_icount 23808, counted 132032 sb_ifree 2334, counted 35228 sb_fdblocks 10484039, counted 33519164 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 2 - agno = 9 - agno = 3 - agno = 1 - agno = 6 - agno = 7 - agno = 5 - agno = 8 - agno = 4 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (3:1193189) is ahead of log (1:2). Format log to cycle 6. done Disk 8 did: Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... ALERT: The filesystem has valuable metadata changes in a log which is being destroyed because the -L option was used. - scan filesystem freespace and inode maps... clearing needsrepair flag and regenerating metadata sb_icount 61696, counted 312768 sb_ifree 9362, counted 47709 sb_fdblocks 47064461, counted 92818335 - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - agno = 13 - agno = 14 - agno = 15 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 7 - agno = 12 - agno = 5 - agno = 3 - agno = 6 - agno = 8 - agno = 2 - agno = 9 - agno = 11 - agno = 10 - agno = 13 - agno = 14 - agno = 15 - agno = 4 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... Maximum metadata LSN (7:742200) is ahead of log (1:2). Format log to cycle 10. done