Things are running excellent, but file distribution looks weird to me

[HomerJ]

Hi fellow unRAIDers! Thanks to many of you, over the last month of so I've managed to setup a system which is running pretty much faultless. I don't have heaps of storage, but I am moving closer and closer every day towards filling up my 4 array disks. At present I don't have a parity drive, but this will change very soon. 

 

My question....

 

If I were to tell you that 95 percent or more of the storage taken up above is by my media share which spans across all 4 disks, would you, like me, think that how the data has been spread across the drives is a little lopsided? I do remember though that when the array was starting to fill up, that it did so on one drive first until it was around half full before moving over to another drive, so the above might be totally normal, but it just seems weird to me that one drive is almost full while there is another that barely has anything on it.

 

Another thing that might be related..... I noticed last night that drive 4 didn't have the little green dot and wasn't displaying is temperature. I had it do a quick Smarttest and when it had concluded(no errors found) it had changed to green and was again displaying the temperature. I doubt it's related because I'm certain I would of noticed no green dot, but maybe there has been an issue with the drive since day one that unRAID has always known about...

 

Anyways, it would be much appreciated if I could get some feedback on the screenshot above, as well as any suggestions on the issue with the drive not showing its temperature.

 

Thanks again everyone! 

[itimpi]

How files are distributed depends on how your shares are set up.   You are likely to get better informed feedback if you attach your system’s diagnostics zip file to your next post in this thread so we can see details of your system.

 

it is normal for a drive to not display temperature if it is spun down.

[HomerJ]

52 minutes ago, itimpi said:

How files are distributed depends on how your shares are set up.   You are likely to get better informed feedback if you attach your system’s diagnostics zip file to your next post in this thread so we can see details of your system.

 

it is normal for a drive to not display temperature if it is spun down.

G'day, thanks heaps for your reply!

 

Attached is my diagnostics zip. 

 

I have thought about using the unbalance app, but I was thinking that since the app was originally made many years ago, that by now the people at Limetech may have made changes to unRAID that possibly made the unbalance app obsolete by now, but if the consensus is that the app is still useful, I'll start using it. In the end, everything seems to be functioning fine, it's just that it seems a bit weird to me lol.

 

And thanks for that info on the hard drive temperature issue I thought I had! 

 

 

server-diagnostics-20220818-1704.zip

[itimpi]

Not sure quite what is going on, but there appear to be some strange results in the diagnostics.   I would suggest rebooting, making sure the array is started and then post new diagnostics.

 

Before you do there is some tidying up that might help with interpreting the diagnostics.

• Remove the nerdpack plugin as I believe it is incompatible with 6.11,x Unraid releases.
• Make sure that all the files on the flash drive under/config/shares correspond to shares you actually have and are not legacy from previous activities.  You can delete any superfluous ones.
• I would avoid using the drive include/exclude options on any share if you want to include all drives (the default) as this avoids the chance of forgetting later to change them when adding a new drive.   Also only use EITHER the Include OR Exclude entries - not both.
• I noticed a share with include drives set but Use Cache set to Only for the Use cache setting - in such a case the Include drives makes no sense and is redundant.

[HomerJ]

Thank you for sharing your thoughts as well as your suggestions. 

 

I've just completed some housekeeping, rebooted and then let the system run for a couple minutes before downloading the diagnostics zip, which I've attached to this post. 

 

Getting it all running was a little bumpy, which may reflect in how it's been set up and could show up in the diagnostics zip. One of my children has severe anxiety and his own computer died a month or so ago, and at the time I was just starting to get my unRAID up and running on the trial licence. The licence ended up expiring and I had no idea if or when I'd get my computer back, so I ended up buying new components and building myself a new one. I'm very far from being a computer expert, so by the time I had the new computer built, I'd forgotten a lot of what I had learned when I originally setup unRAID on my old computer, so I made a few mistakes. Hopefully some of them have been rectified now, but if any more are found in the attachment, please let me know!

 

Thanks again!! 

server-diagnostics-20220818-2109.zip

Edited August 18, 2022 by HomerJ
Spelling correction

[HomerJ]

1 hour ago, itimpi said:

Not sure quite what is going on, but there appear to be some strange results in the diagnostics.   I would suggest rebooting, making sure the array is started and then post new diagnostics.

 

• I noticed a share with include drives set but Use Cache set to Only for the Use cache setting - in such a case the Include drives makes no sense and is redundant.

I think the above relates to what I've attached below in the screenshot. This drive was being used for something else originally, which is why it has disks included. As you can see though, they aren't able to be changed without making other changes first. I'm hoping unRAID knows why and ignores this. 

[trurl]

Why do you have 50G libvirt? I've never seen any reason for more than default 1G

 

And 50G is probably twice what you need for docker.img

 

Something still wrong though. Looks like user shares are working, but most of your shares don't exist, including system share, which should have been created by Docker / VM Manager

 

Post a screenshot of User Shares page

 

 

[HomerJ]

1 hour ago, trurl said:

Why do you have 50G libvirt? I've never seen any reason for more than default 1G

 

I don't know why I do.... I just tried finding where I can change it, but I lucked out.

 

1 hour ago, trurl said:

And 50G is probably twice what you need for docker.img

 

It was originally set at 25GB, but at one time I had used very close to that amount so i changed it. I figured that at the time I had heaps of space, so letting it have that amount wouldn't be a big deal. If you think I should have it at a lower amount, I'll make the change. 

 

1 hour ago, trurl said:

 

Something still wrong though. Looks like user shares are working, but most of your shares don't exist, including system share, which should have been created by Docker / VM Manager

 

Post a screenshot of User Shares page

I remember having issues and I ended up making a bit of a mess of the shares. I do though have a system share.

 

Below is a screenshot of my shares.

 

 

I appreciate the time taken to help me. Cheers! 

Edited August 18, 2022 by HomerJ
Spelling correction

[trurl]

Your diagnostics shows only these shares with files:

appdata                           shareUseCache="prefer"  Exists on t-------------e
domains                           shareUseCache="yes"     Exists on t-------------e
t---------g                       shareUseCache="only"    Exists on t-------------e

 

Click Compute All button at bottom of User Shares page. If you don't get complete results after a few minutes, refresh page. Then post screenshot.

[HomerJ]

12 minutes ago, trurl said:

Your diagnostics shows only these shares with files:

appdata                           shareUseCache="prefer"  Exists on t-------------e
domains                           shareUseCache="yes"     Exists on t-------------e
t---------g                       shareUseCache="only"    Exists on t-------------e

 

Click Compute All button at bottom of User Shares page. If you don't get complete results after a few minutes, refresh page. Then post screenshot.

Great minds think alike? Lol I had just done what you suggested above and came here to share it, to find you asking for it!

 

Yes I just looked at the diagnostics zip myself to see what I could learn, and your right; most of the shares are saying that there is no data! Check out these screenshots below:

 

 

 

The one directly above is my main share: data. It's the one which holds almost everything, and yet this says I it's not using any of the drives. 

Edited August 18, 2022 by HomerJ
Forgot a word

[trurl]

Reboot in SAFE mode and see if diagnostics say the shares have files.

[HomerJ]

Do I have the option of starting fresh without losing any data on the drives? 

 

When I was setting unRAID up on the new pc, the were a few times when I had to go into the TOOLS menu so I could select the NEW CONFIG. Maybe that's got me where I am... 

[HomerJ]

1 minute ago, trurl said:

Reboot in SAFE mode and see if diagnostics say the shares have files.

Ok will do that now. 

[trurl]

What do you get from command line with this?

ls -lah /mnt/user

 

[HomerJ]

23 minutes ago, trurl said:

Reboot in SAFE mode and see if diagnostics say the shares have files.

Done. Here is the diagnostics zip. I haven't looked at it yet but will do when I finish this post. 

 

server-diagnostics-20220819-0044.zip

 

20 minutes ago, trurl said:

What do you get from command line with this?

ls -lah /mnt/user

 

This is what I get:

 

 

I also don't across a another weird issue about a week ago where I'm not able to update done docker containers. For example I just tried, again, to update Jacket while the server was still in safe mode, but I wasn't successful. This is the message I get:

FYI all containers have been installed via the Community Apps plugin. 

[trurl]

12 minutes ago, HomerJ said:

This is the message I get

Your docker templates are on flash at /boot/config/plugins/dockerMan/templates-user. Without its template you can't work with a container from the webUI.

 

Those diagnostics show the same as the others for shares. I haven't tried 6.11rc yet but I don't think anybody has reported any issues like this with diagnostics.

 

What do you get from command line with this?

ls -lah /boot/config/shares

 

 

 

[HomerJ]

16 minutes ago, trurl said:

Your docker templates are on flash at /boot/config/plugins/dockerMan/templates-user. Without its template you can't work with a container from the webUI.

 

Those diagnostics show the same as the others for shares. I haven't tried 6.11rc yet but I don't think anybody has reported any issues like this with diagnostics.

 

What do you get from command line with this?

ls -lah /boot/config/shares

 

 

 

This is what I get(I'm still in safe mode):

 

 

By the way, one of the drives which was in the process of being filled up by my data share which officially doesn't exist, has completely filled up. I just get the message saying it was at 100%.

 

Maybe my best option is to recreate my unRAID server.... Am I able to do this without losing any files on the drives? 

 

 

[trurl]

It is easy to do a new install and keep your data, but you would lose any settings you have made in the webUI. Seems drastic to me since things are working well and no evidence except odd diagnostic output that anything is wrong.

 

I guess you could rollback to 6.10.3 and see what happens on that version.

[HomerJ]

Again I have to say thank you so much for all your assistance! I promise you that it's very much appreciated!!

 

What concerns me is with these errors being in place now, that I will end up having serious issues in the future, but by that time things might be in such a big mess that I end up at a point where I will lose data. 

 

Rather than you spending too much more in helping me, would you happen to have a link handy that will help me reinstall unRAID so I have a freshly running system without losing any data?

 

By the way, I hope the above is clear without too many mistakes; it's currently just after 2am here and I should be sleeping lol

[HomerJ]

I've found a couple support threads which explain the couple steps required to achieve a new install of unRAID without losing data, but there are a couple things that lead me to be concerned that I could just end up transferring the old issues to the new install, or losing shares. 

 

I understand what to do in regards to disk assignment. If I don't physically remove any drives, there is no reason for them to not line up, so that's good. What I'm concerned about is this; I'm assuming (because I can't remember) that the new install create a couple shares at the onset. How do I ensure that these shares line up with what's been laid down on my drives, like the SYSTEM and the APPDATA shares? And when I created my own personal shares, how do they 'attach' themselves to what's already been laid down?

 

One support thread that I read suggested copying over to the USB the file that details the shares, but I don't feel doing this would be in my best interest as it's the shares that are the issue in my current setup. 

 

Keen to hear the thoughts of others. I'm planning to action this in the next few hours, but I'm hesitant due to the above reasons and don't want to be making such a big change without knowing for certain that all will be fine. 

 

I also have another question regarding the creation of the USB stick; seeing that once it boots it's all in the computer's memory, can I create the USB drive on my unRAID computer using a VM, transfer the files over that need to be restored, and then reboot it? The response I'll probably get will be no, but I thought I'd ask.

 

It's times like these that I wish I paid more attention at school lol

Edited August 18, 2022 by HomerJ
Spelling correction

[trurl]

You should make a flash backup if you don't have a current one. That will allow you to recover things from your current configuration if you want. All of your configuration is in the config folder.

 

One thing you will want to keep is the license .key file. If you make a new install on the same flash drive, your license will still work. If you make a new install on another flash drive, you will have to transfer the license, but you still need the original license to help with that.

 

2 hours ago, HomerJ said:

I understand what to do in regards to disk assignment. If I don't physically remove any drives, there is no reason for them to not line up, so that's good.

It isn't a matter of the physical placement of drives, since Unraid has no way to know that. Unraid identifies disks by their serial number, and uses that when assigning disks to the correct slot. You won't have anything else about your configuration such as include/exclude for any user shares, so which particular data disk is assigned to which particular data slot isn't important, The only thing that really matters as far as keeping your data is don't assign any data disk to any parity slot or it will be overwritten with parity.

 

Your drive assignments are in super.dat and the pools folder in config. If you keep those you won't have to worry about assigning disks again.

 

2 hours ago, HomerJ said:

ensure that these shares line up with what's been laid down on my drives, like the SYSTEM and the APPDATA shares? And when I created my own personal shares, how do they 'attach' themselves

User shares are simply the combined top level folders on array and pools. If the top level folders exist, you automatically have a user share named for the top level folder, and all top level folders on all array and pools with the same name are in the same user share. If the user share has a corresponding .cfg file in shares folder of config, that is applied. If not, it has default settings.

 

As far as creating flash in VM. Do you not have another computer you can use?

 

Ideally you will want it on the same flash so you don't have to transfer license. And of course, you can't create it on the current flash while Unraid is running.

[HomerJ]

14 hours ago, trurl said:

You should make a flash backup if you don't have a current one. That will allow you to recover things from your current configuration if you want. All of your configuration is in the config folder.

 

One thing you will want to keep is the license .key file. If you make a new install on the same flash drive, your license will still work. If you make a new install on another flash drive, you will have to transfer the license, but you still need the original license to help with that.

 

It isn't a matter of the physical placement of drives, since Unraid has no way to know that. Unraid identifies disks by their serial number, and uses that when assigning disks to the correct slot. You won't have anything else about your configuration such as include/exclude for any user shares, so which particular data disk is assigned to which particular data slot isn't important, The only thing that really matters as far as keeping your data is don't assign any data disk to any parity slot or it will be overwritten with parity.

 

Your drive assignments are in super.dat and the pools folder in config. If you keep those you won't have to worry about assigning disks again.

 

User shares are simply the combined top level folders on array and pools. If the top level folders exist, you automatically have a user share named for the top level folder, and all top level folders on all array and pools with the same name are in the same user share. If the user share has a corresponding .cfg file in shares folder of config, that is applied. If not, it has default settings.

 

As far as creating flash in VM. Do you not have another computer you can use?

 

Ideally you will want it on the same flash so you don't have to transfer license. And of course, you can't create it on the current flash while Unraid is running.

Thank you for your post. 

 

I've decided to basically start again. The way I originally set up my shares and how iset up their allocation methods was something I was never entirely happy with. I currently have one particular share which is very large, so I'll be able to break it apart and then better manage the allocation methods to better suit each time system in the shares. I think this will help resolve my original issue which was one drive filling up much sooner than the others. 

 

Thanks again for your help. Your thoughts and ideas have helped me better understand how unRAID works and has encouraged me to read up more to better understand not just what to do, but why.

[trurl]

2 hours ago, HomerJ said:

allocation methods

default highwater is usually best. It is a good compromise, eventually using all disks, but without constantly switching disks simply because one disk temporarily has more free than another. Most Free is actually least efficient. Fillup might make sense for archiving purposes but not for shares/disks frequently written.

[HomerJ]

16 hours ago, trurl said:

default highwater is usually best. It is a good compromise, eventually using all disks, but without constantly switching disks simply because one disk temporarily has more free than another. Most Free is actually least efficient. Fillup might make sense for archiving purposes but not for shares/disks frequently written.

Thanks.

 

I'll also be changing the folder split too because I think this is what contributed greatly to my original issue. 

[trurl]

3 hours ago, HomerJ said:

folder split

I usually don't bother with that setting and let it split anything. All it does is prevent a slight delay spinning up another disk if the "next" file isn't on the same disk, such as when playing TV series or music albums.