August 21, 20232 yr I upgraded about a week ago, I'm already setup with dual nic and docker is on its own so I wasn't worried about the macvlan issue. I decided to redo my cache drive as zfs and lost my docker in doing so. Reconfigured and reinstalled everything and I'm back in business. That was two days ago now. I've just noticed that the reads on flash are super high, like in the millions and almost a constant 1.5MB/s with jumps to 5MB/s. I went into terminal and started inotifywait --timefmt %c --format '%T %_e %w %f' -mr /boot and the following is the output Sun Aug 20 13:23:01 2023 ACCESS /boot/ bzfirmware Sun Aug 20 13:23:01 2023 ACCESS /boot/ bzfirmware Above repeats 80 more times... Sun Aug 20 13:23:01 2023 ACCESS /boot/ bzfirmware Sun Aug 20 13:23:01 2023 ACCESS /boot/ bzfirmware Sun Aug 20 13:23:01 2023 ACCESS /boot/ bzfirmware Sun Aug 20 13:23:01 2023 ACCESS /boot/config/plugins/dynamix/ dynamix.cfg Sun Aug 20 13:23:01 2023 ACCESS /boot/config/ docker.cfg Sun Aug 20 13:23:01 2023 ACCESS /boot/config/plugins/dynamix/ monitor.ini Sun Aug 20 13:23:02 2023 ACCESS /boot/config/plugins/dynamix.my.servers/ myservers.cfg Sun Aug 20 13:23:02 2023 ACCESS_ISDIR /boot/ config Sun Aug 20 13:23:02 2023 ACCESS_ISDIR /boot/config/ Sun Aug 20 13:23:02 2023 ACCESS /boot/config/ Pro.key Sun Aug 20 13:23:02 2023 ACCESS /boot/config/ Pro1.key Sun Aug 20 13:23:02 2023 ACCESS_ISDIR /boot/ config Sun Aug 20 13:23:02 2023 ACCESS_ISDIR /boot/config/ Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ CommunityApplicationsAppdataBackup.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ FRS.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ appdata.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ archived.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ backup.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ clonezilla.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ development.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ docker.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ documents.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ downloads.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ family.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ games.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ gitea.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ iansdocs.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ music.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ recordings.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ system.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ temp_data.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ tvshows.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ veeam.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ video.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ webserver_media.cfg Sun Aug 20 13:23:02 2023 ACCESS /boot/config/shares/ youtube.cfg ^C That bzfirmware followed by reading of the share config files is over and over and over again every two seconds. I stopped docker, still happening. I uninstalled all my plugins and restarted, still happening. The Open Files plugin shows nothing using any of the files listed above. I stopped docker again and stopped the array, its still reading from flash constantly. So i decided to just start killing processes within htop that could've been related to anything to do with the boot drive. Killing emhttpd stopped the excessive readings. On a fresh boot or restart, the ACCESS /boot/config/shares/* and ACCESS config/pro.key are checked every second. But the bzfirmware is not constantly read. It's usually a few hours after startup that this issue starts. Edited August 21, 20232 yr by iarp
August 22, 20232 yr 1 minute ago, iarp said: Excessive reading and writing of a usb stick isn't an issue? Some access is normal, but should not be accessing bzfirmware at all. We will try to reproduce..
August 26, 20232 yr I am having the same issue, with constant reads to /boot/config/shares* and /boot/config/plugins* I upgraded to 6.12.3 and also have done a lot of reconfiguring of the array as I moved most shares to a new ZFS pool. In logs, I also see this: Aug 26 12:09:57 unraid kernel: BTRFS warning (device loop2): csum failed root 5 ino 7162730 off 11677696 csum 0x42345997 expected csum 0x9a877341 mirror 1 Aug 26 12:09:57 unraid kernel: BTRFS error (device loop2): bdev /dev/loop2 errs: wr 0, rd 0, flush 0, corrupt 11964, gen 0
August 26, 20232 yr Okay, I think the /dev/loop2 issue is docker image corruption based on this: /dev/loop2 40G 22G 18G 56% /var/lib/docker Are the two issues related? Edited August 26, 20232 yr by JoeyP91
August 26, 20232 yr Community Expert 7 hours ago, JoeyP91 said: Are the two issues related? Possibly not, but you should recreate the docker image.
September 30, 20232 yr I am also experiencing this on Unraid 6.12.3. Running inotifywait shows hundreds of access events over a few seconds for /boot/bzfirmware, and this applies to other files under the /boot directory as well: $ inotifywait --timefmt %c --format '%T %_e %w %f' -m /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware For me this is actually causing other applications to hang, as the CPU utilization caps out due to the amount of iowait on the system. I use Netdata and have attached screenshots showing the CPU util (note the iowait) as well as the i/o for sda, which is my boot USB.
October 1, 20232 yr Community Expert 22 hours ago, healthy-slat5620 said: Sat Sep 30 21:17:43 2023 ACCESS /boot/bzfirmware That file should only be accessed during boot, or possibly during upgrade. Did you attempt an upgrade but didn't reboot?
October 1, 20232 yr 1 hour ago, trurl said: That file should only be accessed during boot, or possibly during upgrade. Did you attempt an upgrade but didn't reboot? No, I had this issue on an older version of Unraid (I'm not sure which but I think it was 6.12.2), then upgraded and still saw this issue weeks later. I did reboot after upgrading. I upgraded to 6.12.4 (and rebooted) yesterday, but it's still happening: $ inotifywatch -r -t 30 /boot Establishing watches... Finished establishing watches, now collecting statistics. total access close_nowrite open filename 2088 696 696 696 /boot/config/shares/ 310 124 93 93 /boot/config/ 187 119 34 34 /boot/ 78 26 26 26 /boot/config/plugins/dynamix/ 40 20 10 10 /boot/config/plugins/dynamix/notifications/agents/ 40 20 10 10 /boot/config/plugins/dynamix/notifications/ 2 0 1 1 /boot/config/plugins/dynamix.my.servers/ I haven't been able to catch which process is causing this using lsof or watching /proc, as the files seem to be opened and closed very, very quickly. (P.S. I am healthy-slat5620, my previous posts were accidentally from a new account as I logged in using Google instead of email)
October 7, 20232 yr Hi there, I have the same issue here on 6.12.4. Constant read on /boot caused the webui very slow and almost freeze. The reading speed shows 10MB to 50MB/s on boot U-flash. It happened after I click into apps page, with no apps from loaded successfully from the community app market. But I am not sure it's the trigger or conincidence. I also found dockerd high cpu from top command. inotifywatch -r -t 30 /boot Establishing watches... Finished establishing watches, now collecting statistics. total access close_nowrite open filename 38497 38435 31 31 /boot/ 970 324 323 323 /boot/config/shares/ 272 112 80 80 /boot/config/ 234 80 77 77 /boot/config/plugins/dynamix/ 104 52 26 26 /boot/config/plugins/ 37 12 11 14 /boot/config/plugins/nut/ 32 16 8 8 /boot/config/plugins/dynamix/notifications/ 28 14 7 7 /boot/config/plugins/dynamix/notifications/agents/ 27 9 9 9 /boot/.git/ 20 10 5 5 /boot/config/plugins/user.scripts/scripts/ 16 8 4 4 /boot/config/plugins/dynamix.file.integrity/ 12 6 3 3 /boot/config/plugins/dockerMan/ 12 6 3 3 /boot/config/plugins/NerdPack/packages/ 10 4 3 3 /boot/config/plugins/dynamix.my.servers/ 9 3 3 3 /boot/.git/refs/heads/ 8 4 2 2 /boot/EFI/ 8 4 2 2 /boot/config/plugins/fix.common.problems/ 8 4 2 2 /boot/config/plugins/user.scripts/ 8 4 2 2 /boot/config/plugins/NerdPack/ 8 4 2 2 /boot/config/plugins/compose.manager/ 8 4 2 2 /boot/config/plugins/unassigned.devices-plus/ 8 4 2 2 /boot/config/plugins/theme.engine/ 8 4 2 2 /boot/config/plugins/corefreq/packages/ 8 4 2 2 /boot/config/plugins/corefreq/ 8 4 2 2 /boot/config/wireguard/ 8 4 2 2 /boot/config/ssh/ 8 4 2 2 /boot/config/ssl/ 6 2 2 2 /boot/.git/info/ 4 2 1 1 /boot/EFI/boot/ 4 2 1 1 /boot/config/plugins/dockerMan/templates/ 4 2 1 1 /boot/config/plugins/dockerMan/templates-user/ 4 2 1 1 /boot/config/plugins/dynamix/users/ 4 2 1 1 /boot/config/plugins/unassigned.devices/ 4 2 1 1 /boot/config/plugins/community.applications/ 4 2 1 1 /boot/config/plugins/fix.common.problems/scripts/ 4 2 1 1 /boot/config/plugins/composerize/ 4 2 1 1 /boot/config/plugins/file.activity/ 4 2 1 1 /boot/config/plugins/user.scripts/scripts/delete.ds_store/ 4 2 1 1 /boot/config/plugins/user.scripts/scripts/delete_dangling_images/ 4 2 1 1 /boot/config/plugins/user.scripts/scripts/viewDockerLogSize/ 4 2 1 1 /boot/config/plugins/user.scripts/scripts/nextcloud_cron/ 4 2 1 1 /boot/config/plugins/dynamix.s3.sleep/ 4 2 1 1 /boot/config/plugins/dynamix.file.manager/ 4 2 1 1 /boot/config/plugins/dynamix.system.temp/ 4 2 1 1 /boot/config/plugins/dynamix.system.stats/ 4 2 1 1 /boot/config/plugins/NerdPack/packages/6.10/ 4 2 1 1 /boot/config/plugins/NerdPack/packages/6.11/ 4 2 1 1 /boot/config/plugins/compose.manager/projects/ 4 2 1 1 /boot/config/plugins/unassigned.devices-plus/packages/ 4 2 1 1 /boot/config/plugins/theme.engine/themes/ 4 2 1 1 /boot/config/plugins/dynamix.cache.dirs/ 4 2 1 1 /boot/config/plugins/unbalance/ 4 2 1 1 /boot/config/plugins/corefreq/packages/6.1.49/ 4 2 1 1 /boot/config/plugins/NerdTools/ 4 2 1 1 /boot/config/plugins/nut/ups/ 4 2 1 1 /boot/config/plugins/dynamix.file.integrity/logs/ 4 2 1 1 /boot/config/plugins/dynamix.file.integrity/export/ 4 2 1 1 /boot/config/plugins/dynamix.file.integrity/saved/ 4 2 1 1 /boot/config/plugins/tips.and.tweaks/ 4 2 1 1 /boot/config/plugins/appdata.backup/ 4 2 1 1 /boot/config/wireguard/peers/ 4 2 1 1 /boot/config/ssh/root/ 4 2 1 1 /boot/config/modprobe.d/ 4 2 1 1 /boot/config/pools/ 4 2 1 1 /boot/config/ssl/certs/ 4 2 1 1 /boot/config/plugins-removed/ 4 2 1 1 /boot/syslinux/ 4 2 1 1 /boot/.git/refs/ 4 2 1 1 /boot/.git/objects/pack/ 4 0 2 2 /boot/.git/objects/22/ 4 2 1 1 /boot/.git/objects/ 2 0 1 1 /boot/.git/objects/d0/ syslog-192.168.31.10.log
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.