On 11/28/2018 at 6:05 AM, JorgeB said:

How can I monitor a btrfs pool for read/write errors?

 

As some may have noticed the GUI errors column for the cache pool is just for show, at least for now, as the error counter remains at zero even when there are some, I already asked and hope LT will use the info from btrfs dev stats in the near future, but for now, anyone using a btrfs cache or unassigned redundant pool should regularly monitor it for errors since it's fairly common for a device to drop offline, usually from a cable/connection issue, since there's redundancy the user keeps working without noticing and when the device comes back online on the next reboot it will be out of sync, a scrub can usually fix it (though note that any NOCOW shares can't be checked or fixed, and worse than that, if you bring online an out of sync device it can easy corrupt the data on the remaining good devices, since btrfs can read from the out of sync device without knowing it contains out of sync/invalid data), but it's good for the user to know there's a problem as soon as possible so it can be corrected.

 

Any btrfs device or pool can be checked for errors read/write with btrfs dev stats command, e.g.:

btrfs dev stats /mnt/cache

It will output something like this:

[/dev/sdd1].write_io_errs    0
[/dev/sdd1].read_io_errs     0
[/dev/sdd1].flush_io_errs    0
[/dev/sdd1].corruption_errs  0
[/dev/sdd1].generation_errs  0
[/dev/sde1].write_io_errs    0
[/dev/sde1].read_io_errs     0
[/dev/sde1].flush_io_errs    0
[/dev/sde1].corruption_errs  0
[/dev/sde1].generation_errs  0

All values should always be zero, and to avoid surprises they can be monitored with a script using Squid's great User Scripts plugin, just create a script with the contents below, adjust path and pool name if needed to monitor e.g. an unassigned pool, and I recommend scheduling it to run hourly, if there are any errors you'll get a system notification on the GUI and/or push/email if so configured.

#!/bin/bash
if mountpoint -q /mnt/cache; then
btrfs dev stats -c /mnt/cache
if [[ $? -ne 0 ]]; then /usr/local/emhttp/webGui/scripts/notify -i warning -s "ERRORS on cache pool"; fi
fi

If you get notified you can then check with the dev stats command which device is having issues and take the appropriate steps to fix them, most times when there are read/write errors, especially with SSDs, it's a cable issue, so start by replacing the cables, then and since the stats are for the lifetime of the filesystem, i.e., they don't reset with a reboot, force a reset of the stats with:

btrfs dev stats -z /mnt/cache

Finally run a scrub, make sure there are no uncorrectable errors and keep working normally, any more issues you'll get a new notification.

 

P.S. you can also monitor a single btrfs device or a non redundant pool, but for those any dropped device is usually quickly apparent.

 

Thanks to @golli53for a script improvement so errors are not reported if the pool is not mounted.

 

I downloaded the script app you recommended.  Named it, put in a description, copied the script from the above (see attachments). 

 

When I click on the "run script" I get the below window with the done button.    This is so cool.  How complicated would it be to have the script not only warn you about the error but then ask you to click Yes to run the scrub with the error box checked, then run a balance, then clear the stats, else click no to abort?  I've never written scripts before and found this very exciting.  Kind of like when I first got an apple II with no drive/cassette and just typed stuff from magazine articles. 🙂

Please don't ask questions in the FAQ thread, there's a FAQ feedback thread or just create a new thread.

 

6 hours ago, tr3bjockey said:

How complicated would it be to have the script not only warn you about the error but then ask you to click Yes to run the scrub with the error box checked, then run a balance, then clear the stats, else click no to abort?

Probably not that complicated, but I know almost nothing about scripts, feel free to post here if you or anyone else can make one.