andyd

I get that. What I'm not clear on is if the data needs to be backed up from the array enitrely and then deleted. Then format the failed drive. But based on what you mentioned, it sounds like I don't have to delete fully. 1. I backed up what's on there which wasn't a lot anyway 2. Will turn off array 3. Format the drive 4. Add back the data 5. turn on the array

But what does backing up what you can mean here though? There is media backed by parity. Are you saying that data is potentially problematic as well? That I should back up all data on the drive and delete the data from the array?

We have talked about this in other posts as I tried to deal with data corruption from bad RAM. Figured a separate post makes sense since it's now the drive in the array and not the cache / docker setup. Status below. Scrubbing does nothing. There is only one zfs drive in the pool. This drive was meant to only be used for appdata snapshots root@HomeServer:/mnt/cache/system/libvirt# zpool status -v pool: apps state: ONLINE config: NAME STATE READ WRITE CKSUM apps ONLINE 0 0 0 sdc1 ONLINE 0 0 0 errors: No known data errors pool: cache state: ONLINE config: NAME STATE READ WRITE CKSUM cache ONLINE 0 0 0 nvme0n1p1 ONLINE 0 0 0 errors: No known data errors pool: disk3 state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A scan: scrub repaired 0B in 02:11:03 with 0 errors on Tue May 21 20:36:29 2024 config: NAME STATE READ WRITE CKSUM disk3 ONLINE 0 0 0 md3p1 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: disk3/movies:<0x18e>

Due to bad ram, the drive has some corruption that scrubbing does not help with. I want to get this drive back to a good state but I am not sure how to that while not losing any media on that drive. The parity check had no errors. It's just zfs pool status showing corruption. What steps should I follow?

Makes sense. I upgraded and things seem to be ok. As for disk 3, yeah I know. I did have bad RAM which I have removed from system. I was reformatting the cache drive for corruption in the docker image and some other folders. That's resolved now. I will start another thread about the drive as I'm not sure how to handle that. Thanks!

Yeah not sure what's going on but I did upgrade and the message went away. After setting up dockers again, things seem to be working ok? 🤷‍♂️ Diagnostics attached homeserver-diagnostics-20240524-1011.zip

What do you mean by same release? 1. i had stopped the array 2. Cleared the cache drive 3. Formatted the drive with zfs but yeah I’ll fully upgrade.

I recently formatted the cache drive because of permanent errors. I am in the middle of addressing that and noticed this message. It mentions "zpool upgrade" can be executed. Is this something we should do?

Thanks! I'll give that a try

I am trying to address some corruption on the cache pool. I am thinking of... 1. Disabling docker 2. Backing up appdata to an external drive 3. Formatting the drive 4. Putting back appdata 5. Enabling docker Is that enough? It looks like the docker image is in the system folder on this drive - safe to assume that when docker is enabled, the container images will be re downloaded? There are other folders I believe on this cache drive like downloads - just recreate the same folders?

I just noticed the cache drive is also problematic. I'll back up and try to reformat But main concern is the array and how to proceed

I’m confused. Are you saying the entire array is a mess even though there is only one zfs drive in the pool? The parity itself has no errors. If there is no zfs file system after the drive is removed, why would the file system still be an issue? Why would the parity check clear if there are issues with data in general? “Backing up and reformatting” - did you mean the entire array? Or the zfs drive? As for the other question - I’m wondering more about the zfs snapshots that are currently being done for app data backups

Yeah that didn’t do anything. So this drive.. 1. Part of the array 2. Is used for backups of docker appdata is there a need to backup anything? Thinking of treating the drive as failed of disconnecting the drive and letting the array rebuild without it but honestly not sure what the best process is. Also, I can ask in the other thread but how will zfs master and the current backup process work in this case? Like is it smart enough to know there are no backups and start from 0?

Ok got it. So looks like I did have bad ram. Now that isn't an issue after isolating the problem stick I still see errors... zpool status -v pool: apps state: ONLINE config: NAME STATE READ WRITE CKSUM apps ONLINE 0 0 0 sdc1 ONLINE 0 0 0 errors: No known data errors pool: cache state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A config: NAME STATE READ WRITE CKSUM cache ONLINE 0 0 0 nvme0n1p1 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: /mnt/cache/system/docker/docker.img cache/downloads:<0x62e> pool: disk3 state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A scan: scrub repaired 0B in 02:06:06 with 0 errors on Tue May 14 18:00:32 2024 config: NAME STATE READ WRITE CKSUM disk3 ONLINE 0 0 0 md3p1 ONLINE 0 0 0 errors: Permanent errors have been detected in the following files: disk3/movies:<0x18e> Still not clear how to proceed. At least before it was clear on what the file was. No idea what it's pointing to now.

I had a few crashes when trying to use a docker. Things have been ok since then but I assume that's what lead to some corruption? Otherwise, why memtest? If I have issues with ram, it would lead to the error above?

I'm seeing the below which I'm not sure how to handle. I'll get 1 data error any time something gets added to the disk now that related to zfs backup since I use the disk in the array as well Best way to handle? I am thinking of disabling backups (possible?) and reformating the drive to see if that helps. For now, I don't care about backups pool: disk3 state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A scan: scrub repaired 0B in 02:06:06 with 0 errors on Tue May 14 18:00:32 2024 config: NAME STATE READ WRITE CKSUM disk3 ONLINE 0 0 0 /dev/md3p1 ONLINE 0 0 164 errors: 1 data errors, use '-v' for a list

attaching homeserver-diagnostics-20240517-2136.zip

I’m seeing the below. I already deleted the file that was being reported as a data error so i no longer see that error. But the first part I’m not sure how to interpret. Do I need to reformat this drive? 1. I have data that I assume can be rebuilt from parity 2. App data backup I don’t need since dockers work One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P scan: scrub repaired 0B in 02:06:06 with 0 errors on Tue May 14 18:00:32 2024

Ok thanks. I’ll give that a try!!

I used to have the vm pinned to specific cores. Not sure if that’s the difference? Or I just have to recreate the vm because of the swap in underlying cpu? just in case - amd 3700x to intel 12600k

scrubbing wouldn't help? waiting for parity check to be done to run that

How can I verify if zfs is corrupt?

Attached was the last error. I was able to access unraid for a little bit but docker was down. Some CPU cores were at 100%. I wasn't able to check what docker might have been causing issues since "docker stats" also wasn't working. I see macvlan listed in "modules linked in" - could this be the reason? I enabled macvlan today because I wanted to test it out. Disabled now after forcing shut down of server. error.txt

Does enabling it mean that there will automatically be something else on the network with the same ip address and different mac? I would have expected this to potentially happen if I change the network to some docker container but not just automatically appearing.

Yeah I’ll try that but since the issues don’t manifest till a few days later I’m thinking it makes sense to ensure setup is correct. should I be changing the port? Network doesn’t matter?