Hello,

 

This is my first post here, as until now i was always able to find the solution in other people's threads. This time seems to be different.

 

I am suffering from exactly the same issue described in these 2 other threads:

 

I am currently on unraid 6.12.10, but have had this issue since i upgraded from 6.11.5 to 6.12.2 some time ago. This thing becomes an issue only when i have high disk I/O, for example scanning some files or in the current moment, reading from multiple disks.

 

Using the same command as in the other topics:

inotifywait --timefmt %c --format '%T %_e %w %f' -mr /boot

I have some access to disk share configs and then a lot of access to /boot/bzfirmware (Thu May  9 10:04:27 2024 ACCESS /boot/ bzfirmware)

 

At the moment the server only has 16 hours uptime and 500k reads from flash. Fortunately there are no writes done by this issue.

 

Has anyone figured out what is causing this behaviour? I am at a loss trying to figure this one out..

 

Attached diagnostics.

 

Thank you in advance !

titan-diagnostics-20240509_1257.zip

Edited May 9, 20242 yr by deienache
grammar

So what i did was try to eliminate zfs from the equation, replaced the three m.2 zfs cache drives with a single m.2 drive using xfs, no difference...

Anyone has any clues where to start troubleshooting ?

 

Thanks for any tips

Edited May 12, 20242 yr by deienache

Ok, nevermind, thread can be closed/archived, whatever needs to happen. I downgraded to 6.11.5 and all problems from the last months went away, the one described above, not being able to open another gui page if there was any plugin update/docker update going on and a few other little things. In my personal opinion ( i know others might not agree ) 6.11.5 is the last true version of unraid. The rest is just a rushed out the door mess...

 

Sad to see limetech went downhill with 6.12, then the horrible april fools joke in community apps messing with MY piece of software thay I BOUGHT, not rented, not subscribed...anyway, discussion for another day.

Will have to go somewhere else, as this piece of software is clearly no longer a safe home for my data...

 

Anyway, rant is over, have a great day everyone !

 

Hi,

I would still like to keep this thread open.

 

I am experiencing the exact same issue as you are describing. I also found the posts you mentioned.

When the server is crashing/lagging I see lot's of "access" to /boot/bzfirmware. (using `inotifywait --timefmt %c --format '%T %_e %w %f' -mr /boot`)
(When the server is running fine (no gui opened) all files in boot/config/shares/ gets read every second, I have no idea if this is normal behaviour.)

 

TL;DR:

I think this is being caused by either SabNZBD and/or the -arr suite. Whenever a file is being downloaded the `Mon May 13 15:42:37 2024 ACCESS /boot/ bzfirmware` logs popup like crazy.

I have also installed Synchting, just like you. I didn't have time to properly debug this, but maybe it's related as well.

Can you relate to this -arr suite conclusion? Is there possibly anything happing with your files when your crash occures?  @deienache

Or are you moving/uploading/reading any big files on your array/cache?

 

Long story:
 

Symptoms
When crashing my flash drive is getting a lot of reads(just as you describe). The GUI is getting slower and slower each second.

I even noticed a 464 MB/s reads speed.

 

 

I have tried some troubleshooting but it's really hard to do since the whole server slowsdown (All docker-containers not responding, GUI not loading and ssh really really slow), VM's however keep running fine without any problems and the internet(using the same ethernet port as unraid) keeps working for the VM's. This indicates some kind of issue with the unraid OS.
(@deienache I assume your Nextcloud VM also kept running without any problems?)

 

History

The first time I noticed this kind of slowdown was on April 5th 2024. (Unraid version 6.12.6. I upgraded to 6.12.10 on 2024 April 23)

I didn't get the change to properly debug the problem until a few days ago.

Since April the 5th Unraid slowed down/crashed about +-4 times, a quick unraid reboot fixed the issue for a few days/weeks.

(These crashes are noticed quickly since unraid also runs my AdGuard docker container, which causes DNS issues when unraid is crashing)

 

Investigation

To investigate this issue I wanted to install some monitoring (Grafana, prometheus, cadvisor, prometheus-node-exporter), I started doing this 2 days ago.

After I installed the `prometheus_node_exporter.plg` plugin 2 days ago (saterday evening, 11-may-2024) I noticed the slowdown starting. I removed the prometheus_node_exporter.plg file and rebooted shortly after and the server ran fine the next day(sunday).
I this morning(monday) reinstalled the prometheus_node_exporter.plg plugin and in about one hour later the slowdown started again.

I rebooted and now it is running fine for a few hours. (sadly unable to create a diagnostics export of this crash)

The prometheus_node_exporter.plg could be a coincidence since both times I used the unraid-GUI heavily(mostly updating containers and browsing apps) while I didn't use the unraid-GUI this sunday.

This however does not explain the previous slowdowns last month, those were completely random, I wasn't using the unraid-GUI at all. 

 

This however got me thinking, maybe it is plugin or docker related?

 

When rebooting or installing a docker-container I see a lot of reads to the /boot/bzfirmware file.
After the crash of 2 days ago I ran `# docker ps`. This showed 3 containers unhealthy (idk why, they are healthy normally).

Maybe some containers get into a docker boot loop causing the /boot/bzfirmware to be read, which cascades to more reboots and reads?

 

Network data

Everytime the crash occurs the network data/second is peaking. This seems to be happing everytime. I am unsure if it maybe is the omada-controller container not logging the correct data because the server is lagging. Or maybe the docker network being overloaded causing the unraid OS to hang. (I still think it is weird the VM's network keep working without any problem and are accessable but non of the unraid-os/container resources are accessable)

(This is network data from my omada-switch, which get's logged by the omada-controller which is running on unraid(docker). The data-gap is a reboot of unraid)

 

 

This got met thinking again. Why is there a high spike in network right before it crashes.

At first I thought is was either the docker network overloading, or a container going haywire. (I have an mDNS relay container to relay 'pings' to other vlan's)

 

Possiblities

Now I finally found something, most of the crash-dates can be matched with SabNZBD downloading a file. (This is also the case for the crashes of today and saterday evening)

After Sab downloads a file, it is moved by Radarr, which is then import by Jellyfin and encoded by Tdarr.

My current theorie is that these crashes are related to (large) files being moved or modified.

In your diagnostics I can see you also have Tdarr installed. Is it possible this is causing the issue? Is anything happing with your files when your Unraid server crashes? If so I can investigatie furthing into Tdarr. @deienache

 

Ending

I don't have an anonymized diagnostics of the crash. I do however have 2 "personal" diagnostics (If needed a can share certain files, but I am not too keen on sharing the full non-anonymized diagnostics)

 

 

I currently have no idea how to fully trigger the crash. I tried (re)installing some more plugins and docker apps to see if that causes any problems. But the server is still running fine for now.

The closed I have come to imitating the crash is triggering a file download for SabNZBD. This causes a low of `ACCESS /boot/ bzfirmware` logs which is similar behaviour when crashing. I do have anonymized logs after this is happing.

 

 

I hope this will help to find the issue we are both facing.

Your answers to my questions are appreciated!

 

(ps. I don't think downgrading should be the final solution, this will causes problems in the long term if it issue isn't fixed in a future update)

 

 

 

Hi @onno204,

 

Indeed, it looks like you did way more research in this issue than i did.

 

Personally my problems started a few months back, i think February, not exactly sure, when i upgraded first to 6.12.2. At that point i had neither Tdarr ( the only -arr ever installed on this server ), neither Grafana or Prometheus, these 2 were installed to try to debug the slowness.

I agree with you that -arr suite can cause this issue, but this is only because of the high I/O they require.

 

I have personally never used SabNZBD, so my issue is definetely not coming from there.

 

In my testing, what i did:

-A full appdata/docker/vm backup and deleted everything, disabled docker, disabled vm. Issues persisted when doing high I/O operations.

-After this, decided to uninstall ALL plugins, was left with a "as clean as it gets without new install" server. Issues persisted when doing high I/O operations.

-Disable bond0 and removed the extra NIC had an old Intel PRO 1000/PT set up in Transmit Load Balancing. Left the server only on eth0 (Realtek, but meh, consumer board). Issues persisted when doing high I/O operations.

-Since zfs is a new implementation, replaced my 3 m.2 nvme cache drives with a single xfs formatted m.2 nvme drive. Issues persisted when doing high I/O operations.

-Recreated unraid usb with fresh 6.12.10 install using USB Creator tool and only replaced config file. Issues persisted when doing high I/O operations.

 

Considering all this, i decided to rollback to 6.11.5. Installed a fresh copy with unraid usb creator, replaced config folder with the old. Everything went smooth and issues is gone, now with all previous plugins and dockers running. I don't have community apps working, but that's not an issue for me at this time.

 

The only thing that changed was the unraid version, so it's clearly an issue in the OS. How and why is not everyone affected by it beats me, but it may have something to do with kernel drivers for different hardware platforms. I'm not that technical unfortunately.

 

I agree with you that rollback is not the best option, but i personally decided to not continue with unraid. I have an order for a 730xd and some drives coming which will be running normal raidz2 using another solution. This is not because i have not loved unraid, i absolutely did and considering there are few people encountering this issue, i am positive it will keep growing. It is after all a perfect solution for a home server. However my use case has changed since i deployed this box and now data integrity and iops are way more important. Not a flaw of unraid, it's just not built for that.

 

I am unable to keep testing for this as i mentioned before, i already went to 6.11.5. What i did not test is high I/O usage on direct disk share bypassing the /mnt/user shares as i didn't have the time. Might pe worth looking into.

 

I hope you will be able to find the issue !

 

Edited May 14, 20242 yr by deienache
Forgot troubleshooting step.

I had a similar issue and solved it with a RAM upgrade. More details here: 

 

Btw I still have read accesses to the bzfirmware file but there is no utilization shown in the web interface or in netdata. Here is a example how it looks just filtered for bzfirmware with inotifywait --timefmt %c --format '%T %_e %w %f' -mr /boot | grep bzfirmware

Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:24 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:27 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:28 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:28 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:28 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:28 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:28 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:28 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:28 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:28 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:28 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:28 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:28 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:28 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:28 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:28 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:28 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:28 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:28 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:28 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:28 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:28 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:28 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:28 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:28 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:28 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:32 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:32 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:32 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:32 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:37 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:37 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:37 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:37 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:37 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:37 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:37 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:37 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:37 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:37 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:37 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:37 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:37 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:37 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:37 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:37 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:37 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:37 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:37 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:37 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:43 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:43 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:43 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:43 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:43 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:51 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:52 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:52 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:52 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:52 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:52 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:52 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:52 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:52 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:52 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:52 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:52 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:52 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:52 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:52 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:52 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:52 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:52 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:52 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:52 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:52 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:52 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:45:52 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:46:01 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:46:01 2024 ACCESS /boot/ bzfirmware
Fri Aug  9 12:46:01 2024 ACCESS /boot/ bzfirmware

 

Edited August 9, 20241 yr by nivrix
Shared more information

I have the same problem where the boot disk often has high IO, causing me to not be able to connect to SSH or open GUI. I have found several similar problem posts in the forum, but none of them have provided a solution. One of them said it was solved by adding memory, but at least it's not suitable for me. I already have 32GB of memory. Based on my observation, it is highly likely that the occurrence of high IO on the boot disk is also due to a certain disk being high IO. By the way, my system version is 6.12.9. (Why haven't I upgraded to version 6.12.14? Because after upgrading to 6.12.14, the system will lose contact directly after a few hours, suspected to be a kernel crash.)

18 hours ago, RWDai said:

I have the same problem where the boot disk often has high IO

There was a recent report that it may be related to memory, pressure, I intend to try and reproduce it when I have some time, but see if it's similar to what you see:

 

https://forums.unraid.net/bug-reports/stable-releases/61213-unraid-having-troubles-with-memory-pressure-and-keeping-the-os-in-ram-r3355/

23 hours ago, JorgeB said:

There was a recent report that it may be related to memory, pressure, I intend to try and reproduce it when I have some time, but see if it's similar to what you see:
最近有报道说可能和记忆、压力有关，打算有时间尝试重现一下，不过看看是不是和你看到的类似：

 

 

Thank you for mentioning the bug post, it seems to be the same issue I encountered. The command echo 3 > /proc/sys/vm/drop_caches appears to work for me.

Process flow: iostat showed that sda (the boot drive) was near 100% utilization, about 98%. Using inotifywait --timefmt %c --format '%T %_e %w %f' -mr /boot, I observed a large number of ACCESS logs for /boot/bzfirmware, and the speed was very high. After running the drop_caches command, the sda utilization dropped to around 1%, and the ACCESS logs in inotifywait for bzfirmware significantly reduced. The load also slightly decreased.

By the way, I have 32GB of RAM, with 13GB used by ZFS, 8GB by virtual machines, and some space taken by Docker. Before executing the command, the free -h command showed about 3.3GB free. After running it, around 4.8GB of free memory was available. A few minutes later, the free memory dropped back to around 3.5GB.

I hope this helps with the investigation! If you need any additional information, feel free to contact me. I should be able to reproduce this in a few days.