I am hoping someone can help point out something wrong in my diagnostics to help me solve this issue. While downloading through sab and deluge as well as sometimes when not, all dockers and web interfaces as well as plex hang for a period of time for what seems like no reason at all. I am sitting and watching my cpu/ram/storage/network usage all at the same time and cpu never goes above 10-15%, network sits around 80-120Mbps, storage up and down from no usage to ~250MBps, and ram sits at 6.8gb free, 78.5gb cached, and 9gb used. I understand that the file transfers from the downloads are what is most likely causing the issue but I feel like my server should be able to handle it without hanging repeatedly. Would it also have something to do with my plex and appdata folders all being stored on the cache where the downloads are put initially? 

Thanks for any help. 

tower-diagnostics-20171023-1231.zip

Cache filesystem is fully allocated, see if this helps, first run a balance:

 

btrfs balance start -dusage=95 /mnt/cache

 

Then run a manual trim:

 

fstrim -v /mnt/cache

 

See if there's any difference

 

2 minutes ago, johnnie.black said:

Cache filesystem is fully allocated, see if this helps, first run a balance:

 

 

By that do you mean how my ram is ~78GB cache or just the whole SSD? To me it sort of seems like SAB is downloading directly to my ram. The ram cache will fill up and it usually just stays full and occasionally when downloading a large file (>30gb) the cache section in the ram will empty out and it looks like the file will start downloading into it. 

 

I started that first command and it has been sitting on it for about 5 minutes now. Is balancing a lengthy process? 

I mean your cache SSD, I assume downloads go there first.

2 minutes ago, johnnie.black said:

I mean your cache SSD, I assume downloads go there first.

Okay. So should the balancing command still be running or should it be done by now? 

Also for the cache SSD I am using the Dynamix SSD Trim plugin to use its mover and trim functions which are scheduled to run daily. 

Just now, Chandler said:

So should the balancing command still be running or should it be done by now? 

 

It will take some time since you filesystem was fully allocated

 

1 minute ago, Chandler said:

Also for the cache SSD I am using the Dynamix SSD Trim plugin to use its mover and trim functions which are scheduled to run daily. 

 

There's a bug with btrfs that in most cases only the unallocated space is trimmed, since you cache was fully allocated nothing was trimmed, e.g., from your syslog:

 

Oct 23 00:10:20 Tower sSMTP[11912]: Authorization failed (535 5.7.8  https://support.google.com/mail/?p=BadCredentials f33sm3579910otj.6 - gsmtp)
Oct 23 05:01:02 Tower root: /etc/libvirt: 1022.5 MiB (1072185344 bytes) trimmed
Oct 23 05:01:02 Tower root: /var/lib/docker: 11.8 GiB (12604219392 bytes) trimmed
Oct 23 05:40:01 Tower root: mover started

Only docker and libvrt images were trimmed.

2 minutes ago, johnnie.black said:

 

There's a bug with btrfs that in most cases only the unallocated space is trimmed, since you cache was fully allocated nothing was trimmed, e.g., from your syslog:

 

Only docker and libvrt images were trimmed.

 

Oh I see. Would it be a good idea to setup a script to balance the SSD every so often then? 

1 minute ago, Chandler said:

Would it be a good idea to setup a script to balance the SSD every so often then? 

 

Yes, once a week it's usually enough, if run frequently you can also use a lower data usage, this is what I use:

 

btrfs balance start -dusage=75 /mnt/cache

 

 

5 minutes ago, johnnie.black said:

 

Yes, once a week it's usually enough, if run frequently you can also use a lower data usage, this is what I use:

 

btrfs balance start -dusage=75 /mnt/cache

 

 

Alright, cool. Well I'll let this commands finish and set that up. Hopefully it solves the issue, it keeps coming and going every week or so. Thanks for your help. 

3 hours ago, johnnie.black said:

 

Then run a manual trim:

 

fstrim -v /mnt/cache

 

See if there's any difference

 

 

When running the trim command it tells me that the discard operation is not supported. Any idea? 

It's connected on an LSI that doesn't support trim, that means it's never been trimmed, connect it on one of the Intel onboard ports.

30 minutes ago, johnnie.black said:

It's connected on an LSI that doesn't support trim, that means it's never been trimmed, connect it on one of the Intel onboard ports.

That did it. Luckily I had a molex to sata power adapter  184GB were trimmed. 

 

Hopefully this solves it! Thanks. 

Well it seems my problem persists -- unless my cache filesystem is already fully allocated again. Here's another diagnostics. Also, where can I find where it says the filesystem is fully allocated? 

tower-diagnostics-20171023-2309.zip

3 hours ago, Chandler said:

unless my cache filesystem is already fully allocated again

 

It is, but that doesn't make sense unless the balance didn't complete, it usually takes weeks to get like that, you can see an explanation of how to check allocation here:

 

 

8 hours ago, johnnie.black said:

 

It is, but that doesn't make sense unless the balance didn't complete, it usually takes weeks to get like that, you can see an explanation of how to check allocation here:

 

Well this time it has clearly worked. 

And a lot more space was trimmed than last time. 

 

So hopefully this was the issue! Thanks for your help.