What is writing to my Unraid USB drive [Solved]

[54lzy]

I recently had an "unclean shutdown." When I booted one disk was disabled so I removed and added to the array to fix. I am rebuilding my array. I noticed a lot of writes on my USB (6000 in 14 hours) so I rebooted my system. Bad move because now the rebuild has restarted. Now I am 2 hours into a boot with 1000 writes.

 

I am wary because I had a USB drive fail me this month or last after tens of thousands or writes to it. Can anyone tell what is writing to my USB? I have uploaded my diagnostics. I am interested in knowing what to be looking at too (so I can solve this myself next time).

tower-diagnostics-20210626-1402.zip

Edited June 30, 2021 by 54lzy

[Squid]

Do you have the syslog server enabled (Settings - Syslog Server) and set to mirror to the flash drive?

[54lzy]

1 hour ago, Squid said:

Do you have the syslog server enabled (Settings - Syslog Server) and set to mirror to the flash drive?

It is not enabled and it is not set to mirror to the flash (it is set to Disabled, and mirror setting says No).

[trurl]

Some plugin? My Servers or Docker Folder?

[remotevisitor]

I would do an ‘ls -ltR /boot’ and look at the output to see what files have recently been changed to see if that gives any idea what might be writing to the flash drive

[54lzy]

46 minutes ago, remotevisitor said:

I would do an ‘ls -ltR /boot’ and look at the output to see what files have recently been changed to see if that gives any idea what might be writing to the flash drive

Does it make sense that the most recent write was 2 hours ago, but there are about 500 more writes since then? Thanks for the suggestion.

[remotevisitor]

Does seam a bit strange .

 

how about trying ‘lsof +D /boot’ to see what has any open files on the flash.

[54lzy]

11 hours ago, remotevisitor said:

Does seam a bit strange .

 

how about trying ‘lsof +D /boot’ to see what has any open files on the flash.

Nothing displayed so I am assuming the answer is no. I am up to 5629 writes now. so in the 19 hours I posted, an additional 4000+ writes.  Actually 5634, it just keeps creeping up and up. I just looked at the output of the ls -ltR /boot command again and there was nothing in the last couple minutes. Is there some other folder I can look at or is the USB limited to the /boot folder?

 

edit: could this be a bug? I am very sensitive to this because I lost a USB drive this month and don't want to lose a second drive within 30 days.

Edited June 27, 2021 by 54lzy

[remotevisitor]

I would try turning off the docker and VM service and see if it stops.    Then at least you know it is something to do with them.

 

if it still continues, try booting in Safe mode from the Unraid boot menu, which will not load any plugins.   See if it now stops.

 

if at any stage it stops it will then be a case of working through enabling your plugins/dockers/VMs to find which one is the cause.

 

this number of writes to the flash drive is certainly not normal.

[54lzy]

2 hours ago, remotevisitor said:

I would try turning off the docker and VM service and see if it stops.    Then at least you know it is something to do with them.

 

if it still continues, try booting in Safe mode from the Unraid boot menu, which will not load any plugins.   See if it now stops.

 

if at any stage it stops it will then be a case of working through enabling your plugins/dockers/VMs to find which one is the cause.

 

this number of writes to the flash drive is certainly not normal.

I disabled the VM service. I left docker service on but stopped all containers. In 1 hour 45 min I went from 6938 to 7500 writes. I’m now disabling the docker service to see if that makes a difference. 
 

One other recent change (maybe this matters maybe it doesn’t), other than my rebuild was that I implemented a cache array. Right now it is just system, appdata, and domains on there.

 

i will update in an hour or two now that I have disabled docker service to see what the write situation is. 
 

edit: 7777 writes. 

Edited June 27, 2021 by 54lzy

[54lzy]

I am at 8194 writes after disabling the docker service. So about 400 in 1 hour. I don't think that fixed the USB writes. Thus I do not think it is the VM or Docker services causing the massive amount of writes.

[Squid]

uninstall preclear plugin followed by myservers plugin.  They're you're most likely culprits

[54lzy]

9 minutes ago, Squid said:

uninstall preclear plugin followed by myservers plugin.  They're you're most likely culprits

Thanks for the response. I uninstalled MyServers and the Unassigned Drives plugin (no longer need so might as well rule one more thing out). It doesn't look like I have preclear installed as a plugin, I think I used a docker instead. Because docker is not currently enabled I have not uninstalled that but I do not think that should matter. I will uninstall once I restart docker just in case.

 

I am at 8759 writes. I keep listing them here for my own record.

[54lzy]

3 hours ago, 54lzy said:

Thanks for the response. I uninstalled MyServers and the Unassigned Drives plugin (no longer need so might as well rule one more thing out). It doesn't look like I have preclear installed as a plugin, I think I used a docker instead. Because docker is not currently enabled I have not uninstalled that but I do not think that should matter. I will uninstall once I restart docker just in case.

 

I am at 8759 writes. I keep listing them here for my own record.

I’m up to 9434 writes after removing the My Servers and preclear plugin/docker. So still writing about 200 an hour.

[trurl]

10 hours ago, remotevisitor said:

try booting in Safe mode from the Unraid boot menu

 

[54lzy]

11 hours ago, trurl said:

 

Alright - safe mode has stopped the writes! I suspect it might be the cache backup plugin only because that is new. My questions are:

 

1. Is the only difference in safe mode the fact that plugins don't run?

2. Assuming 1 is yes, knowing that it is probably plugins causing my problems, is there a better way to pinpoint or should I just disable all plugins and slowly re-enable?

[JonathanM]

3 minutes ago, 54lzy said:

should I just disable all plugins and slowly re-enable?

Use the 50% rule. Disable half, test, disable half of the offending group, repeat. Should only take a small number of cycles to hone in that way.

[54lzy]

Deleting CA Backup/Restore Appdata, Docker Folder seems to have stopped the writes after rebooting in regular (not safe) mode.

 

Next I plan to reinstall MyServers after a couple hours of stability. I suspect that CA Backup/Restore Appdata is my issue because that is the newest plugin. Maybe I will try and duplicate via rsync or something instead. Leaving this open until I can confirm over a longer time period, but we may have just solved this!

[ALERT]

what's the end of the story?

[54lzy]

8 hours ago, ALERT said:

what's the end of the story?

It was the myservers plug-in and it’s a problem that’s been fixed. What wound up helping me was I uninstalled all plugins and slowly added them back in. Same could be said for docker but that wasn’t causing the issue.