So I believe I spoke too soon... this problem is persisting. I'm only able to fix it by selecting 'DETECT' under the PWM fan setting and then choosing apply. The value in the settings doesn't change when I re-detect it, but it seems to update the setting in the running app.
What I noticed, is that the value in the settings doesn't match what's in the config file. Settings shows 'hwmon4' while the config file shows 'hwmon2'. It's as if the settings are not being pushed to the startup config file, although the settings page shows the correct values after a reboot.
I tried to modify the config to match what's in the settings page, but then it returns a permissions denied error when starting.
Any ideas?