Hi folks

 

I wonder what's your opinion regarding a potentially very dangerous ZFS bug (described  here )

 

I've run the test myself and few of my pools are affected, I'm now running the test on my main NAS (It will take hours if not days to complete).

 

ZFS scrub seems unable to detect any corruption at all (hence the 'silent' bug).

 

What's your thoughts? Any remediation? I hate the idea to scrap my pool(s) and go back to array...

 

Cheers!

1 hour ago, Netfab said:

I've run the test myself

If you mean using the zdb -y command, see this:

 

1 minute ago, JorgeB said:

If you mean using the zdb -y command, see this:

 

 

No, the author is quite clear about that in his post:

 

"Summary:

• In ZFS, metaslabs keep track of free space on disk where data can be written.
• A bug is causing metaslabs to become corrupted.
• When deleting a file or snapshot where corruption has occured, ZFS causes a kernel panic. The pool can then only be imported in read-only mode.
• Scrubs do not fix the corruption.
• The only way to fix the pool is to recreate it from backup.
• Edit: in a previous version of this post, I suggested running zdb -y on pools to detect metaslab corruption. However, I’ve since learned that zdb should not be used to detect corruption, even if it crashes. I’m sorry for any undue alarm that this may have caused.

The issue is logged and well known by the OpenZFS team to a point, that a recovery tool was requested back in 2022 but as for today, nothing has been developed in that regard.

You can read more here, here and here

 

It affect OpenZFS both on Linux and FreeBSD "Randomly (not at boot or pool import) system shows panic messages (see below) on the console. The system seemed to be operational. Upon trying to reboot I was unable to get a clean shutdown. Had several Failed unmounting messages with references to /root and /var/log. Along with a failed to start Journal Service. Eventually it hung at a systemd-shutdown[1]: Syncing filesystems and block devices - timed out, issuing SIGKILL to PID ... Had to kill the power after waiting ~20 minutes."

 

Corruption of metaslab means that the the kernel will not know where to allocated space for new data...

2 hours ago, Netfab said:

I've run the test myself

So what was this test?

 

I've never heard of this bug being a general problem.

1 hour ago, JorgeB said:

I've never heard of this bug being a general problem.

 

Ok not sure how that is relevant/important though 🤔 (google will tell you that it's widely discussed and a real issue).

 

You can execute: zdb -bccsv -y pool_name (this will likely not crash -or at least, it didn't when I did it-; just be aware that it's extremely heavy on CPU and IO and will take long time to complete)

 

I wonder if all the people suddenly complaining that they are unable to write or mount/umount their pools are facing this issue...

 

Like mentioned, that zdb -y test appears to be irrelevant.

 

It does look like there's an issue, but since I've never run into it myself, nor have I seen anyone in the forum with that problem, or even in the TrueNAS forums, and while I don't read every thread there like here, if that was a big issue, I would expect to have seen one or more big threads about it, so not really worried for now, but something that I will keep an eye out for.

 

 

 

 

9 minutes ago, JorgeB said:

Like mentioned, that zdb -y test appears to be irrelevant.

 

It does look like there's an issue, but since I've never run into it myself, nor have I seen anyone in the forum with that problem, or even in the TrueNAS forums, and while I don't read every thread there like here, if that was a big issue, I would expect to have seen one or more big threads about it, so not really worried for now, but something that I will keep an eye out for.

 

 

 

 

 

Ok, here's some example from the TrueNAS forum, here, here and here

 

The fact that the response are getting nowhere doesn't mean that it's not a problem (this bug is going on since few years now and because it's silent in nature, people just tend to believe that this is some kind of hardware failure while it's not). Again I'm not pushing for an answer, I'm just asking if anyone else had a similar experience that didn't find an appropriate answer.

 

For what concerns me, I'm seriously thinking to ditch ZFS and go back to Array+XFS as for me, stability is paramount.

 

3 minutes ago, Netfab said:

The fact that the response are getting nowhere doesn't mean that it's not a problem

 

17 minutes ago, JorgeB said:

It does look like there's an issue