hermy65

I upgraded to 7.2.2 last week from 7.1 and anything related to docker is painfully slow. I do have a large number of containers so my docker image is larger than an average user. Diagnostics are attached. I did try to clean up the build cache a little the other day to see if that would help but it did not. storage-diagnostics-20251222-0936.zip

yeah, i have the remote syslog server enabled, thats how i got the syslog chunk i uploaded 30 minutes ago. Was there anything in that piece that gave you any insight?

@JorgeB Im not sure what the segfault message means, does that mean its a memory issue or a cpu issue since it references cpu 36? I forgot i had setup remote syslog streaming to another unraid box. Also, it happened again and i had to hard power it so i grabbed a large chunk of the syslog and attached it here so maybe that will help! syslog.txt

Over the last 2 weeks or so there have been 2-3 times where i would go to use one of my self-hosted services to find they didnt respond. Try to access the unraid gui and it wouldnt respond, couldnt ssh in, nothing. I end up having to hard power it off and then things are fine again for a while but since i cant ssh in i cant get diagnostics. This morning i had a similar issue but after 5+ minutes it finally loaded and i was able to grab diagnostics. I have zero idea if this is a related issue or not but i was hoping the diagnostics would point you guys in the direction of potential issues maybe. Thanks! storage-diagnostics-20251118-0743.zip

@JorgeB I created a new docker image, re-installed everything and now today im having issues where containers just stop working or have other issues. Syslog shows btrfs issues of some sort. Diagnostics attached. storage-diagnostics-20250512-2032.zip

Woke up this morning to all of my containers not working and a Docker Failed to Start message on the docker tab. Pulled diagnostics before rebooting and they are attached below. Any insight into what is happening here or what needs to be done to correc it? storage-diagnostics-20250509-0643.zip

@JorgeB that's a bummer. Anything I should look into in the meantime before it happens again?

This morning i found a ton of errors in AppdataBackup saying that my shares didnt exist so it failed. Thought that was strange, went into the SHares tab and sure enough it said No Exportable Shares. I ssh'd in and /mnt/user was not accessible but /mnt/user0 was and i could see all of my shares under it as well as my data. I rebooted my server and now all of my shares are available. Can someone tell me what happened and how to fix it/prevent it? Pre-reboot diagnostics attached storage-diagnostics-20250330-0902.zip

I have a pihole on my network but that's always been there and I haven't had issues. Did something change in 7.0 that would be getting caught by pihole?

I finally pulled the trigger and updated to 7 and now i get no drives in my gui and no "data" on the dashboard. On the first reboot after update i did see that info, then i rebooted and now i get nothing. I was able to start/stop my array but no drive info anywhere. Any idea what i need to do here? Ive tried differnet browsers, mobile, incognito, etc. Ive also noticed if i try to update a plugin or anything now i dont get any data in the plugin popup either.

@jorgeb Thanks for all of the assistance here, hopefully I am finally past all of these problems!!

@jorgeb Ok, i deleted libvrt.img and then stopped/started and the vm page works and one of my vms came back so i should be good with that one. Were the BTRFS errors all pointing at the libvirt issue or was that something else i need to take care of?

@JorgeB Got the cache pool re-formatted, etc. Just noticed some more BTRFS errors, attached are my diagnostics storage-diagnostics-20240423-1113.zip Also, not sure if related but after rebuilding my cache pool my VMs tab no longer works? It doesnt load any of the vms i had or let me create vms, just a blank page.

I had FolderView installed for a while and it was working great. I had to rebuild my cache pool today and my docker.img and now my folders are gone. If i click on the Add Folder button it does nothing. Any ideas on what to do here?

@JorgeB thanks. Any reason to replace sdc1? Since that's the one that keeps having errors?

@JorgeB does that mean I should like format the cache drives and start over or how does a person recreate the filesystem?

@JorgeB woke up this morning to some more BTRFS errors, ive attached a new set of diagnostics. storage-diagnostics-20240420-0758.zip

@JorgeB deleted those files, re-ran a repair scrub and it came back clean. Also moved docker to ipvlan. Anything else that i need to do/watch for? Appreciate the help as always!

@JorgeB looks like its a couple pieces of plex and jellyfin metadata unless tthis is the wrong area to look at: Apr 19 08:49:34 Storage kernel: BTRFS warning (device sdc1): checksum error at logical 19965161472 on dev /dev/sdc1, physical 10267930624, root 5, inode 686348049, offset 4096, length 4096, links 1 (path: appdata/jellyfin/data/metadata/People/E/Eamon Sheehan/folder.jpg) Apr 19 08:49:34 Storage kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 335783, rd 191311, flush 1, corrupt 29971, gen 0 Apr 19 08:49:34 Storage kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 19965161472 on dev /dev/sdc1 Apr 19 08:49:34 Storage kernel: BTRFS warning (device sdc1): checksum error at logical 19965161472 on dev /dev/sdb1, physical 9206771712, root 5, inode 686348049, offset 4096, length 4096, links 1 (path: appdata/jellyfin/data/metadata/People/E/Eamon Sheehan/folder.jpg) Apr 19 08:49:34 Storage kernel: BTRFS error (device sdc1): bdev /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 6, gen 0 Apr 19 08:49:34 Storage kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 19965161472 on dev /dev/sdb1 Apr 19 08:49:34 Storage kernel: BTRFS warning (device sdc1): checksum error at logical 19965165568 on dev /dev/sdc1, physical 10267934720, root 5, inode 686348049, offset 8192, length 4096, links 1 (path: appdata/jellyfin/data/metadata/People/E/Eamon Sheehan/folder.jpg) Apr 19 08:49:34 Storage kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 335783, rd 191311, flush 1, corrupt 29972, gen 0 Apr 19 08:49:34 Storage kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 19965165568 on dev /dev/sdc1 Apr 19 08:49:34 Storage kernel: BTRFS warning (device sdc1): checksum error at logical 19965165568 on dev /dev/sdb1, physical 9206775808, root 5, inode 686348049, offset 8192, length 4096, links 1 (path: appdata/jellyfin/data/metadata/People/E/Eamon Sheehan/folder.jpg) Apr 19 08:49:34 Storage kernel: BTRFS error (device sdc1): bdev /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0 Apr 19 08:49:34 Storage kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 19965165568 on dev /dev/sdb1 Apr 19 08:48:46 Storage kernel: BTRFS warning (device sdc1): checksum error at logical 2367467520 on dev /dev/sdc1, physical 1293725696, root 5, inode 812538915, offset 20480, length 4096, links 1 (path: appdata/PlexMediaServer/Library/Application Support/Plex Media Server/Media/localhost/8/4b1112dba0e382f5a87080425e1a7ac0d711dec.bundle/Contents/GoP-0.xml) Apr 19 08:48:46 Storage kernel: BTRFS warning (device sdc1): checksum error at logical 2367467520 on dev /dev/sdb1, physical 199012352, root 5, inode 812538915, offset 20480, length 4096, links 1 (path: appdata/PlexMediaServer/Library/Application Support/Plex Media Server/Media/localhost/8/4b1112dba0e382f5a87080425e1a7ac0d711dec.bundle/Contents/GoP-0.xml) Apr 19 08:48:46 Storage kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 335783, rd 191311, flush 1, corrupt 29969, gen 0 Apr 19 08:48:46 Storage kernel: BTRFS error (device sdc1): bdev /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0 Apr 19 08:48:46 Storage kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 2367467520 on dev /dev/sdc1 Apr 19 08:48:46 Storage kernel: BTRFS error (device sdc1): unable to fixup (regular) error at logical 2367467520 on dev /dev/sdb1 Apr 19 08:48:46 Storage kernel: BTRFS warning (device sdc1): checksum error at logical 2367471616 on dev /dev/sdc1, physical 1293729792, root 5, inode 812538915, offset 24576, length 4096, links 1 (path: appdata/PlexMediaServer/Library/Application Support/Plex Media Server/Media/localhost/8/4b1112dba0e382f5a87080425e1a7ac0d711dec.bundle/Contents/GoP-0.xml) Apr 19 08:48:46 Storage kernel: BTRFS error (device sdc1): bdev /dev/sdc1 errs: wr 335783, rd 191311, flush 1, corrupt 29970, gen 0 Apr 19 08:48:46 Storage kernel: BTRFS warning (device sdc1): checksum error at logical 2367471616 on dev /dev/sdb1, physical 199016448, root 5, inode 812538915, offset 24576, length 4096, links 1 (path: appdata/PlexMediaServer/Library/Application Support/Plex Media Server/Media/localhost/8/4b1112dba0e382f5a87080425e1a7ac0d711dec.bundle/Contents/GoP-0.xml) Apr 19 08:48:46 Storage kernel: BTRFS error (device sdc1): bdev /dev/sdb1 errs: wr 0, rd 0, flush 0, corrupt 5, gen 0

@JorgeB Here are the results from the correcting scrub UUID: bbc56f07-1a5f-4d7b-b019-a515d7eb35aa Scrub started: Fri Apr 19 08:48:42 2024 Status: finished Duration: 0:39:21 Total to scrub: 1.26TiB Rate: 563.20MiB/s Error summary: csum=8 Corrected: 0 Uncorrectable: 8 Unverified: 0

@JorgeB I ran the command in terminal, received this response: Clearing log on /dev/sdc1, previous log_root 4906986635264, level 0 Stopped then restarted the array and it looks like my cache pool has come back online, here are the new diagnostics. storage-diagnostics-20240419-0838.zip

@JorgeB here are fresh diagnostics after a reboot. One thing i noticed is that now my entire cache pool says unmountable when it was working fine last night. Any ideas? storage-diagnostics-20240418-1501.zip

I had an SSD go bad in my cache pool last week so I swapped a new drive in and I think got everything fixed. Today I'm getting BTRFS errors and I'm not sure exactly what to do. Diagnostics attached storage-diagnostics-20240418-1054.zip

Yes, without the -n sorry. Appreciate the help!

@JorgeB Thanks, this is what the -n gave me Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan and clear agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - agno = 10 - agno = 11 - agno = 12 - agno = 13 bad CRC for inode 27918115649 bad CRC for inode 27918115649, will rewrite cleared inode 27918115649 - agno = 14 - agno = 15 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 2 - agno = 8 - agno = 13 - agno = 4 - agno = 5 - agno = 7 - agno = 6 - agno = 9 - agno = 10 - agno = 12 - agno = 3 - agno = 14 - agno = 15 - agno = 11 Phase 5 - rebuild AG headers and trees... - reset superblock... Phase 6 - check inode connectivity... - resetting contents of realtime bitmap and summary inodes - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify and correct link counts... done