SSD extremely excessive writes - How to detect?

[johnsanc]

Apologies if this was asked before - but is there a way to pinpoint exactly what is causing excessive writing to my two 1TB SDDs in my cache pool? I have had these drives less than one year and they are already nearing 500 TBW - which is insane!!!

 

I have a feeling it's a docker that is writing a ton of temp files or something. This is a huge issue as I have burned through the useful life of these drives in less than 1/4th the warranty period. I didn't even notice the wear until I checked the smart reports today. 

 

Any help getting to the bottom of this is appreciated, as well as any tips for preventing this excessive usage.

(This would be a great thing to include as a test in Fix Common Problems)

[Squid]

Netdata / cAdvisor should help pinpoint it.

 

Or via a command prompt,

iotop

[johnsanc]

I tried using iotop and I noticed similar results as these two threads below

Initially I saw a ton of writes coming from shfs (100-200mb/s steady). I disabled my Nextcloud docker and it immediately reduced the disk writes. I turned Nextcloud back on and things seem relatively normal... I see loop2 now has the most writes but its not crazy like before. Its at about 2GB in 15 minutes.

 

Is there a bug with Nextcloud or something? I checked my access logs and no one was logged in and there were no transfers.

 

Here you can see a near constant write of 250mb/s until I restarted Nextcloud just a few moments ago near the end of the timeline.

 

 

Whatever this issue is it doesn't happen immediately. According to the disk graph it started after about 7 hours of uptime. I guess I'll check on this tomorrow morning to see if the disk usage is excessively high again.

 

 

Edited November 17, 2019 by johnsanc

[BRiT]

Others have identified using BTRFS with Encyption as a common cause of excessive writes.

[johnsanc]

I use BTRFS on my cache drives, but no encryption.

[johnsanc]

One thing I noticed is that my docker config is set to /mnt/user/appdata instead of /mnt/cache/appdata

Could that be causing an issue?


Sent from my iPhone using Tapatalk

[johnsanc]

OK - So I woke up this morning and lo and behold it was writing like crazy again. Shut off Nextcloud and it stopped completely. No other changes were made. The mover started at 3:30 AM so I assume that is the first bump of writes, but then it kicks up at about 4:20 AM.

 

Does anyone have any idea what Nextcloud is doing in the background that might be causing this? I assume I cannot be the only person this is happening to.

 

Edit: This appears to be oc_filecache updates... but still not sure why it seems to perpetually update once it starts.

 

Edited November 17, 2019 by johnsanc

[johnsanc]

I think I figured this out - and its totally user error and not being careful and mounting something as external storage that I shouldn't have.

Basically there where two issues:

 

1) Nextcloud does NOT index symlinks properly. I had literally millions of rows in the oc_filecache that were junk looped paths due to symlinks

2) Something that was mounted was symlinked to system paths where files change basically non-stop, which causes constant modified date updates

 

I ended up having well over 400 million rows. Instead of cleaning it up I just truncated the entire oc_filecache table and starting fresh with correct external storage mounts. At least 1 TB 860 EVO drives are on sale for $105 at Best Buy for black friday...