Drive Error Red X

[GTP]

49 minutes ago, trurl said:

What are your plans for that disk, a new data slot?

 

appdata, domains, system shares have files on the array. This will impact performance of dockers / VMs due to slower parity, and will keep array disks spunup since dockers / VMs will have open files on these.

 

Your docker.img has 150G allocated, and is using 26G of that. I normally recommend only 20G as being more than enough. I have 17 dockers and they are using less than half of 20G. Are you sure you don't have some app misconfigured so it is writing to a path that isn't mapped? Common mistakes are specifying paths in the app using different upper/lower case than mapped container paths.

 

Which dockers do you run?

 

Do you have custom docker networks such as with letsencrypt / SWAG?

@trurl Yes, I was thinking I should format it as a new data disk. What do you think?

 

I have switched appdata/domains/system to "Prefer"

 

As far as the docker allocation I think I did that because I run Plex and I was worried about it running out of room to do conversions and all the metadata it stores on the library. I am fine with lowering it if you think I should. I don't remember how to lower the amount allocated to the docker image though so if you could help me with that I would appreciate it. Also I am sure I made all the mistakes when I configured it so any mistakes I made I will correct if I can identify them.

 

binhex-plexpass

binhex-preclear

Darkstat

deezloaderrrmx(not working right now)

duckdns

FlexTV

gaps

Grafana

grocy

HOOBS

Influxdb

Let's Encrypt

krusader

mysql

ombi

Synclounge(not working right now)

tautulli

telegraf

upstatsboard

Varken

xteve

 

Yes, I use Let's encrypt for my reverse proxy for OMBI I believe. 

 

I hope this answers your questions please feel free to ask more if it will help.

 

Thank you,

 

-GTP

 

 

Edited September 18, 2020 by GTP
question

[trurl]

29 minutes ago, GTP said:

I have switched appdata/domains/system to "Prefer"

That is the correct setting for those, but it won't be enough to get them moved to cache since mover can't move open files. We will deal with that after some more information.

 

24 minutes ago, GTP said:

run Plex and I was worried about it running out of room to do conversions and all the metadata it stares on the library.

Plex metadata (library) is in appdata assuming you have it configured correctly. Plex doesn't normally write a lot of data unless you use DVR or transcode. Those paths would need to be in mapped storage of course.

 

 

You have some dockers I don't use, and I use a different Plex docker.

 

Go to Docker page, click Container Size at the bottom, and post a screenshot of the result.

 

[GTP]

18 minutes ago, trurl said:

That is the correct setting for those, but it won't be enough to get them moved to cache since mover can't move open files. We will deal with that after some more information.

 

Plex metadata (library) is in appdata assuming you have it configured correctly. Plex doesn't normally write a lot of data unless you use DVR or transcode. Those paths would need to be in mapped storage of course.

 

 

You have some dockers I don't use, and I use a different Plex docker.

 

Go to Docker page, click Container Size at the bottom, and post a screenshot of the result.

 

I have attached the screenshots for the container and my plex container. I also have been playing around with xteve which is a DVR.

 

Edited September 18, 2020 by GTP
Clarification

[trurl]

Screenshots of Add / Edit Container page seldom show all the information. See this very first link at the Docker FAQ for a better way:

 

When I do Container Size I get a grand total at the bottom but I don't see it with yours. Doesn't matter but that doesn't always tell us everything anyway.

 

Maybe you need more than 20G but I am more concerned that it not grow. As mentioned you are already using 26G. I use less than 10G of 20G with my 17 dockers.

 

The reason I asked about custom docker networks is because you will have to recreate when you recreate docker.img at a smaller size.

 

What do you get from the command line with this?

docker network ls

 

[GTP]

4 minutes ago, trurl said:

Screenshots of Add / Edit Container page seldom show all the information. See this very first link at the Docker FAQ for a better way:

 

When I do Container Size I get a grand total at the bottom but I don't see it with yours. Doesn't matter but that doesn't always tell us everything anyway.

 

Maybe you need more than 20G but I am more concerned that it not grow. As mentioned you are already using 26G. I use less than 10G of 20G with my 17 dockers.

 

The reason I asked about custom docker networks is because you will have to recreate when you recreate docker.img at a smaller size.

 

What do you get from the command line with this?

docker network ls

 

Screenshot attached reading the FAQ

[trurl]

1 hour ago, GTP said:

I was thinking I should format it as a new data disk. What do you think?

With the large number of disks you have you really should consider dual parity.

[GTP]

Just now, trurl said:

With the large number of disks you have you really should consider dual parity.

I was considering that but do I need another 12 TB disk for that? Or can I use the 8 and 12 together?

[GTP]

I found the attached information in the FAQ. Which option do you recommend I do?

[trurl]

2 hours ago, GTP said:

can I use the 8 and 12 together?

You can but no data drive can be larger than any parity drive.

 

2 hours ago, GTP said:

Which option do you recommend I do?

The simplest is to just use the default, transcode into appdata, which will be on cache after we get things moved. That is good enough for starters and we can discuss other options after other things are done.

[trurl]

3 hours ago, GTP said:

Screenshot attached reading the FAQ

Are you actually using all of these?

 

 

Probably already asked somewhere on this thread, but do you have any VMs?

[GTP]

17 hours ago, trurl said:

Are you actually using all of these?

 

 

Probably already asked somewhere on this thread, but do you have any VMs?

Br0 -unknknown

bridge - I think so

docker-configs - I don't even know what this one does

hassio - I don't think I am anymore if its for the hassio container because I deleted that. I changed my methodology and I do run a Hassio VM and that could be the bridge to the internet.

host - I think you need that one right

influxdb - this one I can say for sure "yes" I use Grafana dashboards and Varken

none - no idea what that is

proxynet - yes for reverse proxy

proxynet1 - yes for reverse proxy

 

Yes, I run a Hassio VM instance

 

I am sorry I don't know more I feel like I should but I was mostly just following SpaceInvadersOne's directions to install most of those containers

 

Best,

 

-GTP

 

[GTP]

I set those shares to prefer and started the mover. Its been going for a while now and my cache drive is getting pretty full is that ok? Below is a screenshot of the shares and the cache drive.

 

 

[itimpi]

Is your cache drive large enough to hold the contents of the shares set to Prefer? 

[trurl]

14 hours ago, GTP said:

I set those shares to prefer and started the mover. Its been going for a while now and my cache drive is getting pretty full is that ok? Below is a screenshot of the shares and the cache drive.

Unless you also disabled docker and VM service, it won't be able to move everything, since it can't move open files. But that is unrelated to filling cache.

 

Probably should have made sure everything that didn't need to be on cache had already been moved off first before trying to move things to cache.

 

Sorry for the delay following up. Other life going on.

 

Post new diagnostics.

[GTP]

No problem at all although the drive has just kept filling up and I tried changing the setting back but it just keeps going the opposite direction.

Diagnostics attached. I have a 1tb cache and it looks like I can't set all those shares I should set to "prefer"

 

Best,

 

-GTP

 

tower-diagnostics-20200920-1002.zip

Edited September 20, 2020 by GTP

[trurl]

Set appdata, domains, system to cache-only for now. That will get them ignored by mover until we can make room for them.

 

You have some other shares with files on cache. You can see which shares have files on cache by going to Main - Cache Devices, click on the folder icon at the far right of Cache under View to see which folders (shares) are on cache.

 

Set each of those shares to cache-yes and run mover. Wait for it to complete then post new diagnostics.

[trurl]

Just noticed your cache is completely full and your log is full of mover messages. If mover is still running try this

Mover logging can sometimes be useful if you need to debug something specific, but I don't think there is anything we don't already understand, at least for now. Go to Settings - Scheduler - Mover Settings and disable mover logging.

 

You can reboot to clear log space, or you could just do this to get rid of these older logs

rm /var/log/syslog.*

 

[GTP]

25 minutes ago, trurl said:

Set appdata, domains, system to cache-only for now. That will get them ignored by mover until we can make room for them.

 

You have some other shares with files on cache. You can see which shares have files on cache by going to Main - Cache Devices, click on the folder icon at the far right of Cache under View to see which folders (shares) are on cache.

 

Set each of those shares to cache-yes and run mover. Wait for it to complete then post new diagnostics.

Cache is completely full I have posted two attached screenshots that should help thank you . I followed your instructions regarding stop and "only" 

-GTP

[trurl]

17 minutes ago, GTP said:

I followed your instructions regarding stop and "only" 

Also do this

35 minutes ago, trurl said:

Go to Settings - Scheduler - Mover Settings and disable mover logging.

 

You can reboot to clear log space, or you could just do this to get rid of these older logs

rm /var/log/syslog.*

 

Mover ignores cache-no and cache-only shares. It moves cache-yes shares from cache to array (we will get to cache-prefer later). As you can see, you have some cache-no shares that have files on cache. To get those moved from cache, set each of them to cache-yes.

 

Then run mover, wait for it to complete, post new diagnostics.

[GTP]

6 hours ago, trurl said:

Also do this

 

Mover ignores cache-no and cache-only shares. It moves cache-yes shares from cache to array (we will get to cache-prefer later). As you can see, you have some cache-no shares that have files on cache. To get those moved from cache, set each of them to cache-yes.

 

Then run mover, wait for it to complete, post new diagnostics.

@trurl I followed your instructions and I moved everything off of cache that I could. Two directories seem to be sticking, one directory with one movie in movies. and one directory with one tv episode in TV Shows. Other than that it only has the files it should. What should we do next? Diagnostics have been posted below and a picture of the cache contents.

 

Best,

 

-GTP

 

tower-diagnostics-20200920-2010.zip

[trurl]

Are you sure something doesn't have those files open? Go to Settings - Docker and disable dockers, then try to move them again.

[trurl]

Looks like DVR TV Shows has all its files on cache.

 

Go to Shares - User Shares, click Compute All. Wait for the result. If it doesn't appear after a few minutes, refresh the page.

 

Then post a screenshot of the results.

[GTP]

I have attached the screenshots from the compute all below sorry that there are 12. I ran mover again several times with docker disabled but those files in those directories just will not move. Its just a move and a couple episodes of a tv show so if we had to delete it it would not be a big deal but its weird that they are not moving. I have seen warnings about the particular movie thats stuck in the cache in the logs though. I don't know what to do but I have moved all I can and the Cache is now sitting at 335gb used. Let me know what you want me to do next.

 

Thank you,

 

-GTP

[trurl]

Lets go ahead with getting appdata, domains, system moved to cache. I'm going to bed now but I will go ahead and tell you what I have in mind for next. You can go ahead with it if you want or you can wait till tomorrow if you don't want to go without dockers or VMs until then.

 

Set appdata, domains, system shares to cache-prefer.

Go to Settings - VM Manager and disable VMs. Leave them disabled

Go to Settings - Docker and disable Dockers. On that same page, delete docker.img. Change its size to 20G. Leave them disabled

Run mover and wait for it to complete then post new diagnostics.

[GTP]

1 minute ago, trurl said:

Lets go ahead with getting appdata, domains, system moved to cache. I'm going to bed now but I will go ahead and tell you what I have in mind for next. You can go ahead with it if you want or you can wait till tomorrow if you don't want to go without dockers or VMs until then.

 

Set appdata, domains, system shares to cache-prefer.

Go to Settings - VM Manager and disable VMs. Leave them disabled

Go to Settings - Docker and disable Dockers. On that same page, delete docker.img. Change its size to 20G. Leave them disabled

Run mover and wait for it to complete then post new diagnostics.

I can absolutely wait, I understand its late here as well. I will do the following tomorrow but I do have one question before I delete the docker.img file will that delete my Plex metedata or settings?

 

Night,

 

-GTP