iarp Posted August 21, 2023 Share Posted August 21, 2023 (edited) I upgraded about a week ago, I'm already setup with dual nic and docker is on its own so I wasn't worried about the macvlan issue. I decided to redo my cache drive as zfs and lost my docker in doing so. Reconfigured and reinstalled everything and I'm back in business. That was two days ago now. I've just noticed that the reads on flash are super high, like in the millions and almost a constant 1.5MB/s with jumps to 5MB/s. I went into terminal and started inotifywait --timefmt %c --format '%T %_e %w %f' -mr /boot and the following is the output Sun Aug 20 13:23:01 2023 ACCESS /boot/ bzfirmware Sun Aug 20 13:23:01 2023 ACCESS /boot/ bzfirmware Above repeats 80 more times... Sun Aug 20 13:23:01 2023 ACCESS /boot/ bzfirmware Sun Aug 20 13:23:01 2023 ACCESS /boot/ bzfirmware Sun Aug 20 13:23:01 2023 ACCESS /boot/ bzfirmware Sun Aug 20 13:23:01 2023 ACCESS /boot/config/plugins/dynamix/ dynamix.cfg Sun Aug 20 13:23:01 2023 ACCESS /boot/config/ docker.cfg Sun Aug 20 13:23:01 2023 ACCESS /boot/config/plugins/dynamix/ monitor.ini Sun Aug 20 13:23:02 2023 ACCESS /boot/config/plugins/dynamix.my.servers/ myservers.cfg Sun Aug 20 13:23:02 2023 ACCESS_ISDIR /boot/ config Sun Aug 20 13:23:02 2023 ACCESS_ISDIR /boot/config/ Sun Aug 20 13:23:02 2023 ACCESS /boot/config/ Pro.key Sun Aug 20 13:23:02 2023 ACCESS /boot/config/ Pro1.key Sun Aug 20 13:23:02 2023 ACCESS_ISDIR /boot/ config Sun Aug 20 13:23:02 2023 ACCESS_ISDIR /boot/config/ Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ CommunityApplicationsAppdataBackup.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ FRS.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ appdata.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ archived.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ backup.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ clonezilla.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ development.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ docker.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ documents.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ downloads.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ family.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ games.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ gitea.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ iansdocs.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ music.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ recordings.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ system.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ temp_data.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ tvshows.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ veeam.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ video.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ webserver_media.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ youtube.cfg ^C That bzfirmware followed by reading of the share config files is over and over and over again every two seconds. I stopped docker, still happening. I uninstalled all my plugins and restarted, still happening. The Open Files plugin shows nothing using any of the files listed above. I stopped docker again and stopped the array, its still reading from flash constantly. So i decided to just start killing processes within htop that could've been related to anything to do with the boot drive. Killing emhttpd stopped the excessive readings. On a fresh boot or restart, the ACCESS /boot/config/shares/* and ACCESS config/pro.key are checked every second. But the bzfirmware is not constantly read. It's usually a few hours after startup that this issue starts. Edited August 21, 2023 by iarp Quote Link to comment
iarp Posted August 22, 2023 Author Share Posted August 22, 2023 Excessive reading and writing of a usb stick isn't an issue? Quote Link to comment
limetech Posted August 22, 2023 Share Posted August 22, 2023 1 minute ago, iarp said: Excessive reading and writing of a usb stick isn't an issue? Some access is normal, but should not be accessing bzfirmware at all. We will try to reproduce.. Quote Link to comment
JoeyP91 Posted August 26, 2023 Share Posted August 26, 2023 I am having the same issue, with constant reads to /boot/config/shares* and /boot/config/plugins* I upgraded to 6.12.3 and also have done a lot of reconfiguring of the array as I moved most shares to a new ZFS pool. In logs, I also see this: Aug 26 12:09:57 unraid kernel: BTRFS warning (device loop2): csum failed root 5 ino 7162730 off 11677696 csum 0x42345997 expected csum 0x9a877341 mirror 1 Aug 26 12:09:57 unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 11964, gen 0 Quote Link to comment
JoeyP91 Posted August 26, 2023 Share Posted August 26, 2023 (edited) Okay, I think the /dev/loop2 issue is docker image corruption based on this: /dev/loop2 40G 22G 18G 56% /var/lib/docker Are the two issues related? Edited August 26, 2023 by JoeyP91 Quote Link to comment
JorgeB Posted August 26, 2023 Share Posted August 26, 2023 7 hours ago, JoeyP91 said: Are the two issues related? Possibly not, but you should recreate the docker image. 1 Quote Link to comment
healthy-slat5620 Posted September 30, 2023 Share Posted September 30, 2023 I am also experiencing this on Unraid 6.12.3. Running inotifywait shows hundreds of access events over a few seconds for /boot/bzfirmware, and this applies to other files under the /boot directory as well: $ inotifywait --timefmt %c --format '%T %_e %w %f' -m /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware For me this is actually causing other applications to hang, as the CPU utilization caps out due to the amount of iowait on the system. I use Netdata and have attached screenshots showing the CPU util (note the iowait) as well as the i/o for sda, which is my boot USB. Quote Link to comment
trurl Posted September 30, 2023 Share Posted September 30, 2023 Attach Diagnostics to your NEXT post in this thread Quote Link to comment
healthy-slat5620 Posted September 30, 2023 Share Posted September 30, 2023 starlight-diagnostics-20230930-2220.zip Quote Link to comment
trurl Posted October 1, 2023 Share Posted October 1, 2023 22 hours ago, healthy-slat5620 said: Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware That file should only be accessed during boot, or possibly during upgrade. Did you attempt an upgrade but didn't reboot? Quote Link to comment
4Ply Posted October 1, 2023 Share Posted October 1, 2023 1 hour ago, trurl said: That file should only be accessed during boot, or possibly during upgrade. Did you attempt an upgrade but didn't reboot? No, I had this issue on an older version of Unraid (I'm not sure which but I think it was 6.12.2), then upgraded and still saw this issue weeks later. I did reboot after upgrading. I upgraded to 6.12.4 (and rebooted) yesterday, but it's still happening: $ inotifywatch -r -t 30 /boot Establishing watches... Finished establishing watches, now collecting statistics. total access close_nowrite open filename 2088 696 696 696 /boot/config/shares/ 310 124 93 93 /boot/config/ 187 119 34 34 /boot/ 78 26 26 26 /boot/config/plugins/dynamix/ 40 20 10 10 /boot/config/plugins/dynamix/notifications/agents/ 40 20 10 10 /boot/config/plugins/dynamix/notifications/ 2 0 1 1 /boot/config/plugins/dynamix.my.servers/ I haven't been able to catch which process is causing this using lsof or watching /proc, as the files seem to be opened and closed very, very quickly. (P.S. I am healthy-slat5620, my previous posts were accidentally from a new account as I logged in using Google instead of email) Quote Link to comment
Canaan Posted October 7, 2023 Share Posted October 7, 2023 Hi there, I have the same issue here on 6.12.4. Constant read on /boot caused the webui very slow and almost freeze. The reading speed shows 10MB to 50MB/s on boot U-flash. It happened after I click into apps page, with no apps from loaded successfully from the community app market. But I am not sure it's the trigger or conincidence. I also found dockerd high cpu from top command. inotifywatch -r -t 30 /boot Establishing watches... Finished establishing watches, now collecting statistics. total access close_nowrite open filename 38497 38435 31 31 /boot/ 970 324 323 323 /boot/config/shares/ 272 112 80 80 /boot/config/ 234 80 77 77 /boot/config/plugins/dynamix/ 104 52 26 26 /boot/config/plugins/ 37 12 11 14 /boot/config/plugins/nut/ 32 16 8 8 /boot/config/plugins/dynamix/notifications/ 28 14 7 7 /boot/config/plugins/dynamix/notifications/agents/ 27 9 9 9 /boot/.git/ 20 10 5 5 /boot/config/plugins/user.scripts/scripts/ 16 8 4 4 /boot/config/plugins/dynamix.file.integrity/ 12 6 3 3 /boot/config/plugins/dockerMan/ 12 6 3 3 /boot/config/plugins/NerdPack/packages/ 10 4 3 3 /boot/config/plugins/dynamix.my.servers/ 9 3 3 3 /boot/.git/refs/heads/ 8 4 2 2 /boot/EFI/ 8 4 2 2 /boot/config/plugins/fix.common.problems/ 8 4 2 2 /boot/config/plugins/user.scripts/ 8 4 2 2 /boot/config/plugins/NerdPack/ 8 4 2 2 /boot/config/plugins/compose.manager/ 8 4 2 2 /boot/config/plugins/unassigned.devices-plus/ 8 4 2 2 /boot/config/plugins/theme.engine/ 8 4 2 2 /boot/config/plugins/corefreq/packages/ 8 4 2 2 /boot/config/plugins/corefreq/ 8 4 2 2 /boot/config/wireguard/ 8 4 2 2 /boot/config/ssh/ 8 4 2 2 /boot/config/ssl/ 6 2 2 2 /boot/.git/info/ 4 2 1 1 /boot/EFI/boot/ 4 2 1 1 /boot/config/plugins/dockerMan/templates/ 4 2 1 1 /boot/config/plugins/dockerMan/templates-user/ 4 2 1 1 /boot/config/plugins/dynamix/users/ 4 2 1 1 /boot/config/plugins/unassigned.devices/ 4 2 1 1 /boot/config/plugins/community.applications/ 4 2 1 1 /boot/config/plugins/fix.common.problems/scripts/ 4 2 1 1 /boot/config/plugins/composerize/ 4 2 1 1 /boot/config/plugins/file.activity/ 4 2 1 1 /boot/config/plugins/user.scripts/scripts/delete.ds_store/ 4 2 1 1 /boot/config/plugins/user.scripts/scripts/delete_dangling_images/ 4 2 1 1 /boot/config/plugins/user.scripts/scripts/viewDockerLogSize/ 4 2 1 1 /boot/config/plugins/user.scripts/scripts/nextcloud_cron/ 4 2 1 1 /boot/config/plugins/dynamix.s3.sleep/ 4 2 1 1 /boot/config/plugins/dynamix.file.manager/ 4 2 1 1 /boot/config/plugins/dynamix.system.temp/ 4 2 1 1 /boot/config/plugins/dynamix.system.stats/ 4 2 1 1 /boot/config/plugins/NerdPack/packages/6.10/ 4 2 1 1 /boot/config/plugins/NerdPack/packages/6.11/ 4 2 1 1 /boot/config/plugins/compose.manager/projects/ 4 2 1 1 /boot/config/plugins/unassigned.devices-plus/packages/ 4 2 1 1 /boot/config/plugins/theme.engine/themes/ 4 2 1 1 /boot/config/plugins/dynamix.cache.dirs/ 4 2 1 1 /boot/config/plugins/unbalance/ 4 2 1 1 /boot/config/plugins/corefreq/packages/6.1.49/ 4 2 1 1 /boot/config/plugins/NerdTools/ 4 2 1 1 /boot/config/plugins/nut/ups/ 4 2 1 1 /boot/config/plugins/dynamix.file.integrity/logs/ 4 2 1 1 /boot/config/plugins/dynamix.file.integrity/export/ 4 2 1 1 /boot/config/plugins/dynamix.file.integrity/saved/ 4 2 1 1 /boot/config/plugins/tips.and.tweaks/ 4 2 1 1 /boot/config/plugins/appdata.backup/ 4 2 1 1 /boot/config/wireguard/peers/ 4 2 1 1 /boot/config/ssh/root/ 4 2 1 1 /boot/config/modprobe.d/ 4 2 1 1 /boot/config/pools/ 4 2 1 1 /boot/config/ssl/certs/ 4 2 1 1 /boot/config/plugins-removed/ 4 2 1 1 /boot/syslinux/ 4 2 1 1 /boot/.git/refs/ 4 2 1 1 /boot/.git/objects/pack/ 4 0 2 2 /boot/.git/objects/22/ 4 2 1 1 /boot/.git/objects/ 2 0 1 1 /boot/.git/objects/d0/ syslog-192.168.31.10.log Quote Link to comment
Canaan Posted October 7, 2023 Share Posted October 7, 2023 tower-diagnostics-20231007-1129.zip diagnose attached Quote Link to comment
Canaan Posted October 15, 2023 Share Posted October 15, 2023 Anyone could help me on this? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.