Missing over 100 GB on cache drive

MagicSG1 · October 3, 2017

I've noticed that on my 250GB Samsung Evo cache drive that 104GB is used at all times. There is nothing that I can see that has to be moved to the array. I've used the likes of cadvisor to see if there were rogue log files building up etc but it doesn't show anything out of the ordinary. I've also ssh ed into the server and used various commands to list biggest files on the server etc but again nothing seems to be showing up. Is there something blatantly obvious that I'm missing? I am only running five docker containers which are as follows:

Sabnzbd
Sonarr
Couchpotato
EmbyServer
cadvisor

Here is a attachment of my container mappings? Maybe I've mapped something incorrectly?

If anyone can help me solve this it would be very much appreciated.

FreeMan · October 3, 2017

The only thing I see is that CP, SAB & Sonarr all point to /mnt/cache/Downloads are you sure they're not busy filling up the disk with downloaded files?

Delarius · October 3, 2017

Hi,

There's a few things I'd look at to check the issue. I'm not sure what commands you have run, but I'd start by ssh root@servername and login.

Then I'd look at the file sizes specifically for the cache drive.

First I'd see what you get when you type:

df -h /mnt/cache

This should give you info about the disk in standard numbers (-h human readable)

I don't use a cache drive but I'd expect it will show you that you've got a 250GB(ish) drive and however much is used/free.

Next to figure out what is using this data (provided the drive shows up as expected above.)

du -h --max-depth 1 /mnt/cache

This should list the size of the directories on that drive - hopefully this will get you closer.

If you note a specific directory which looks strangely large you can use that same command on that directory, so for example:

du -h --max-depth 1 /mnt/cache/appdata (or whichever folder seems to be the issue.)

Obviously, you can adjust the --max-depth number to suit your needs.

Hopefully this will help you narrow down where your drive space is going.

Del

Edited October 3, 2017 by Delarius

MagicSG1 · October 4, 2017

@DelariusIve run the command and I still dont see anything overly out of the ordinary. Here is the output from two variations of the command. Would you be able to have a look at this for me?

du -h --max-depth 1 /mnt/cache

4.6G /mnt/cache/appdata

1.3G /mnt/cache/Downloads

16G /mnt/cache

du -h --max-depth 3 /mnt/cache

896K /mnt/cache/appdata/sabnzbd/admin

0 /mnt/cache/appdata/sabnzbd/logs

916K /mnt/cache/appdata/sabnzbd

1.2M /mnt/cache/appdata/cache/images

1.4M /mnt/cache/appdata/cache/python

3.2M /mnt/cache/appdata/cache

0 /mnt/cache/appdata/backup

0 /mnt/cache/appdata/logs

80K /mnt/cache/appdata/database/_indexes

1.9M /mnt/cache/appdata/database

4.0K /mnt/cache/appdata/custom_plugins

264K /mnt/cache/appdata/db_backup

79M /mnt/cache/appdata/sonarr/MediaCover

8.2M /mnt/cache/appdata/sonarr/Backups

2.5M /mnt/cache/appdata/sonarr/logs

111M /mnt/cache/appdata/sonarr

12M /mnt/cache/appdata/couchpotato/database

166M /mnt/cache/appdata/couchpotato/cache

4.0K /mnt/cache/appdata/couchpotato/custom_plugins

7.1M /mnt/cache/appdata/couchpotato/db_backup

200K /mnt/cache/appdata/couchpotato/logs

184M /mnt/cache/appdata/couchpotato

4.0K /mnt/cache/appdata/scripts/nzbget

4.0K /mnt/cache/appdata/scripts/sabnzbd

16K /mnt/cache/appdata/scripts

5.3M /mnt/cache/appdata/EmbyServer/logs

2.0G /mnt/cache/appdata/EmbyServer/cache

456K /mnt/cache/appdata/EmbyServer/config

80K /mnt/cache/appdata/EmbyServer/localization

154M /mnt/cache/appdata/EmbyServer/data

4.0K /mnt/cache/appdata/EmbyServer/ssl

88M /mnt/cache/appdata/EmbyServer/ffmpeg

16K /mnt/cache/appdata/EmbyServer/root

2.1G /mnt/cache/appdata/EmbyServer/metadata

1.2M /mnt/cache/appdata/EmbyServer/plugins

0 /mnt/cache/appdata/EmbyServer/transcoding-temp

116K /mnt/cache/appdata/EmbyServer/fonts

0 /mnt/cache/appdata/EmbyServer/sync

4.3G /mnt/cache/appdata/EmbyServer

4.6G /mnt/cache/appdata

12K /mnt/cache/Downloads/downloads/incomplete

1.1G /mnt/cache/Downloads/downloads/complete

1.2G /mnt/cache/Downloads/downloads

0 /mnt/cache/Downloads/tmp/transcoding-temp

0 /mnt/cache/Downloads/tmp

1.3G /mnt/cache/Downloads

16G /mnt/cache

tdallen · October 4, 2017

Hi, post the df command too, so we can see the total size of the drive.

MagicSG1 · October 4, 2017

Hi, @tdallenthanks for taking a look.

df -h /mnt/cache

Filesystem Size Used Avail Use% Mounted on

/dev/sdg1 233G 181G 52G 78% /mnt/cache

At the moment there is a 52gb linux iso() downloading to the cache drive.

Delarius · October 4, 2017

That all looks good.

I think there’s something else that’s marked as deleted but isn’t.

Could you try:

lsof | grep /mnt/cache

With luck that’ll show all files open on that drive - see if that helps.

You could also check for files that are tagged as deleted,

lsof | grep -i deleted

If that doesn’t help, try fsck on that drive? I think you can also check btrfs from the GUI under disk management when array is in maintenance mode.

Edited October 5, 2017 by Delarius
Fixing words

MagicSG1 · October 5, 2017

@Delarius Ok these commands have returned new information, I just don't know how to understand it.

lsof | grep /mnt/cache

shfs 7214 root 4w REG 0,31 37253693440 13180300 /mnt/cache/Movies/The Martian (2015)/.fuse_hidden01208dd30001671b (deleted)

shfs 7214 7215 root 4w REG 0,31 37253693440 13180300 /mnt/cache/Movies/The Martian (2015)/.fuse_hidden01208dd30001671b (deleted)

shfs 7214 14828 root 4w REG 0,31 37253693440 13180300 /mnt/cache/Movies/The Martian (2015)/.fuse_hidden01208dd30001671b (deleted)

shfs 7214 16042 root 4w REG 0,31 37253693440 13180300 /mnt/cache/Movies/The Martian (2015)/.fuse_hidden01208dd30001671b (deleted)

shfs 7214 16043 root 4w REG 0,31 37253693440 13180300 /mnt/cache/Movies/The Martian (2015)/.fuse_hidden01208dd30001671b (deleted)

shfs 7214 16044 root 4w REG 0,31 37253693440 13180300 /mnt/cache/Movies/The Martian (2015)/.fuse_hidden01208dd30001671b (deleted)

shfs 7214 16045 root 4w REG 0,31 37253693440 13180300 /mnt/cache/Movies/The Martian (2015)/.fuse_hidden01208dd30001671b (deleted)

shfs 7214 16051 root 4w REG 0,31 37253693440 13180300 /mnt/cache/Movies/The Martian (2015)/.fuse_hidden01208dd30001671b (deleted)

shfs 7214 16053 root 4w REG 0,31 37253693440 13180300 /mnt/cache/Movies/The Martian (2015)/.fuse_hidden01208dd30001671b (deleted)

shfs 7214 16058 root 4w REG 0,31 37253693440 13180300 /mnt/cache/Movies/The Martian (2015)/.fuse_hidden01208dd30001671b (deleted)

shfs 7214 16059 root 4w REG 0,31 37253693440 13180300 /mnt/cache/Movies/The Martian (2015)/.fuse_hidden01208dd30001671b (deleted)

shfs 7214 25121 root 4w REG 0,31 37253693440 13180300 /mnt/cache/Movies/The Martian (2015)/.fuse_hidden01208dd30001671b (deleted)

smbd 21690 root cwd DIR 0,31 70 256 /mnt/cache

smbd 21690 root 31r DIR 0,31 70 256 /mnt/cache

lsof | grep -i deleted

superviso 2766 root 5w REG 0,31 4172632 13433540 /config/supervisord.log (deleted)