[SOLVED] Unraid filling disk to 100%

[RIDGID]

 I have an array of 16 drives (1 parity) + cache, for some reason recently disk13 is getting filled to 100%. MY shares are setup as high-water with 100gb min free space and not to use the cache drive. The array has over 14TBs of free space still, but disk 13 is getting overfilled. I have deleted and/or moved files from the disk manually but it fills right back up when new data is coming in. 

 

Syslog is attached, any help would be greatly appreciated.

 

(Solved)

 

definer5-diagnostics-20191129-1834.zip

Edited December 26, 2019 by RIDGID

[JorgeB]

What share? Allocation method takes precedence over split level.

[RIDGID]

Data comes into the download share and later gets moved the the media share, both are allocated "High-water", split level "Automatically split any directory as required"

[JorgeB]

You appear to have a media and a Media share, this can cause issues.

[RIDGID]

I mean I don't think so, here's a screen of my shares. Could you elaborate? 

[JorgeB]

Just now, RIDGID said:

Could you elaborate? 

Check your flash drive (config/shares), names are partially anonimized, but there is a m----a share, and another called M---a.

[RIDGID]

Hmm, you're right. in /boot/config/shares there is a Media.cfg (but not a media.cfg) while in user shares, as in my screen shot, the share is named "media". 

 

I would note that I haven't edited any of my shares main shares in over a year and this problem only started very recently. 

[JorgeB]

Check all your disks, if media and Media exists on any of them, as a top folder.

[trurl]

30 minutes ago, johnnie.black said:

What share? Allocation method takes precedence over split level.

Other way around. If allocation method had precedence then split level would have no meaning.

 

11 minutes ago, RIDGID said:

Hmm, you're right. in /boot/config/shares there is a Media.cfg (but not a media.cfg) while in user shares, as in my screen shot, the share is named "media". 

 

I would note that I haven't edited any of my shares main shares in over a year and this problem only started very recently. 

According to your diagnostics, your media(1).cfg has settings. This is likely the Media.cfg file you are seeing. But there is no share by that name (no files). The media share itself has default settings, and that is the share that has the files. You might try deleting Media.cfg, reboot and make settings for the media share to see if that clears up the confusion.

[JorgeB]

1 minute ago, trurl said:

47 minutes ago, johnnie.black said:

What share? Allocation method takes precedence over split level.

Other way around. If allocation method had precedence then split level would have no meaning.

d'oh, that's what I meant.

[RIDGID]

19 minutes ago, johnnie.black said:

Check all your disks, if media and Media exists on any of them, as a top folder.

No Media folder on any disk.

 

12 minutes ago, trurl said:

According to your diagnostics, your media(1).cfg has settings. This is likely the Media.cfg file you are seeing. But there is no share by that name (no files). The media share itself has default settings, and that is the share that has the files. You might try deleting Media.cfg, reboot and make settings for the media share to see if that clears up the confusion.

I am not seeing "media(1).cfg" anywhere. Diagnostics file has a M---a.cfg and m---a.cfg file, but my only share in /mnt/user is "media" and in /boot/config/shares I only have Media.cfg (which must correspond to my actual media share).  I am hesitant to delete the .cfg as the majority of my files are in that share.

[trurl]

1 minute ago, RIDGID said:

No Media folder on any disk.

 

I am not seeing "media(1).cfg" anywhere. Diagnostics file has a M---a.cfg and m---a.cfg file, but my only share in /mnt/user is "media" and in /boot/config/shares I only have Media.cfg (which must correspond to my actual media share).  I am hesitant to delete the .cfg as the majority of my files are in that share.

Actually, what I saw in your diagnostics was m___a(1).cfg since it was anonymized. Possibly if you look at your not anonymized diagnostics then you see it as Media.cfg

 

Deleting that .cfg will not have any effect on your files. It will only make the user share have default settings. Then you can make the settings again. Take a screenshot if you need to.

[RIDGID]

Uhh so I deleted to .cfg, stopped my containers, and went to stop the array before reboot. It hung for a few minutes on "retry unmounting user shares" and now is at "Stopped. Unclean shutdown detected." which will parity check if I bring it back online. 

 

 

[RIDGID]

My last restart (about a week ago) was unclean--I had lost webui access and had to issue a 'reboot' command via ssh--but the parity check after restarting was clean and everything has been more or less fine since.

[RIDGID]

Okay, so I restarted and it was no longer considered unclean (idk what happened) and the array started fine without needing a full parity check. the 'media' share was re-setup so now /boot/config/shares has media.cfg and /mnt/user has a 'media' folder

[trurl]

Here is how "unclean shutdown" works.

 

You must stop the array before rebooting or shutting down. If you shutdown or reboot from the webUI  or command line then Unraid will attempt to stop the array before rebooting or shutting down. I think if you briefly push the power button Unraid also recognizes that as a command to shutdown and tries to stop the array first. If you pull the plug, or hold the power button or press the reset switch if you have one, then Unraid doesn't have a chance to stop the array, so you get an unclean shutdown and parity check.

 

Unraid stores the started/stopped status on flash. If it successfully stops the array then it will try to write that stopped status to flash. If for some reason it can't successfully write the stopped status, then when it boots it will think the array wasn't stopped and will begin a parity check due to unclean shutdown.

 

Let us know if your user share problem is fixed now.

 

 

[RIDGID]

Thanks, I am going to usethe unbalance plugin to spread the files out and see if incoming data is still being sent exclusively to disk13. Will update when its finished

[RIDGID]

Hey, problem seems to be fixed, data is going to multiple disks and none have gone past the high-water mark. Thank you so much, I absolutely would never have figured that out on my own.

[RIDGID]

I guess I spoke too soon... Same problem different disk. This time disk 7 blew right past the high-water mark and now has 20kb free. I will move the data to another disk and see if it gets overfilled again, here is a fresh diagnostics if anyone has any thoughts.

definer5-diagnostics-20191203-0321.zip

[trurl]

This is a little long with some details about how some of this stuff works "under the hood". If something I have said is unclear ask for clarification.

 

Have a look at the diagnostics you posted. Diagnostics are just text files in folders and can be easily read. Some of these are actually the same .cfg files that exist on your flash drive. In the shares folder are the .cfg files for each share. These contain the settings from the webUI of your shares.

 

User shares are simply the aggregate across all disks of the top level folders with the same name. Any folder at the top level of array or cache drive is automatically a user share, whether you actually created it as a user share or not. If you don't make settings for a user share, it has default settings.

 

The share .cfg files in your diagnostics contain several "duplicates" with the (1) appended. They are named this way due to DOS compatible filenames where it doesn't distinguish between upper and lower case.

 

If you take a look at both of the files in these "duplicate" pairs, you will see one has settings but doesn't exist on any disk, and the other has default settings and does exist on disks.

 

The ones with default settings don't actually correspond to anything on your flash drive, because you have never made any settings for them. And those are the ones that actually have files and actually exist in your user shares.

 

The ones that don't exist on any disk do correspond to files on your flash drive because you made settings for them, but they don't actually exist in your user shares.

 

Probably at some point you accidentally created the shares without any settings by specifying the wrong case in some path in a docker mapping or something.

 

I don't really see how this could cause the symptoms you are having, since default share settings are highwater, split any, 0 minimum, and cache-no.

 

But, there is definitely some confusion with the way your shares are setup, so clean up all of these just as you did before with your media share and see what happens.

 

[RIDGID]

Sorry for the late reply. Everything you said is quite clear and I appreciate the lengthy response. 

 

Looking through my diagnostics I still no not see any true "duplicates" that have been appended with a (1). There were three CFGs which were capitalized showing "# Share exists on no drives" that correspond with non-capitalized versions that do exist. I've deleted these configs and they were recreated automatically (with no capitalization) and default share settings.

 

I then rebooted the server, moved files around so no disks were over there high-water mark, and then let downloading continue. Disk 7 filled right back up to 20.5KB free and downloading halted. 

 

On 12/3/2019 at 8:55 AM, trurl said:

I don't really see how this could cause the symptoms you are having, since default share settings are highwater, split any, 0 minimum, and cache-no.

 

This is my sentiment as well; I was quite surprised changing a share config initially seemed to fix the problem. I am thinking the issue has more to do with how ruTorrent and radarr and/or sonarr are handing files and less to do with share configuration. I will probably try and setup a different download client (or duplicate ruTorrent docker) and see if the problem persists. 

 

I am going to toss a final diagnostics on here, I think my share settings should be fine, but if there are still .cfg duplicates with a (1) could you let me know where you are seeing them? 

definer5-diagnostics-20191205-1831.zip

[trurl]

17 minutes ago, RIDGID said:

Looking through my diagnostics I still no not see any true "duplicates" that have been appended with a (1).

Probably you were not looking at the same diagnostics you posted. Maybe you were looking at new diagnostics you had collected without anonymizing them. If you actually download those you posted earlier you will see what I was talking about with the (1), but in the end the effect is the same.

19 minutes ago, RIDGID said:

There were three CFGs which were capitalized showing "# Share exists on no drives" that correspond with non-capitalized versions that do exist.

I think if you don't anonymize you get to see them as they actually exist in the system, with upper and lower case.

20 minutes ago, RIDGID said:

I've deleted these configs and they were recreated automatically (with no capitalization) and default share settings.

They were recreated as default shares because they were top level folders on cache or array that didn't have a corresponding .cfg file.

 

Don't really know what was going on. Your latest diagnostics don't seem to have any duplicates.

 

What do you get from the command line with this?

ls -lah /mnt/user

 

[trurl]

27 minutes ago, RIDGID said:

I am thinking the issue has more to do with how ruTorrent and radarr and/or sonarr are handing files and less to do with share configuration.

Linux is case-sensitive so /mnt/user/Media is a different path than /mnt/user/media.

 

Using the wrong upper/lower case in a docker mapping is a very common way to get shares with default settings accidentally created.

[RIDGID]

31 minutes ago, trurl said:

Linux is case-sensitive so /mnt/user/Media is a different path than /mnt/user/media.

 

Using the wrong upper/lower case in a docker mapping is a very common way to get shares with default settings accidentally created.

Yeah, almost certainly what happened but I think we got that cleared up. The problem is, the default settings have the same high-water mark and split level as any share I may have incidentally created, but something is still breaking the 'rules' and overfilling disks. 

 

I really do not know how unraid, behind the scenes, handles filling multiple disks and what happens as they approach their high-water mark. My best guess as to what is happening in my case is something along the lines of ruTorrent recieved a bunch of files, checked for free space, then allocated them to a location; since torrents often download simultaneously and some of them approach the size of the min free space limit, somehow it caused the too much data to be written to a disk. Alternatively Radarr may have decided to copy certain files instead of hardlinking them resulting in double the space usage. 

 

I realize both dockers should only see the share and are not 'aware' that there even are multiple disks, but I can't come up with much else especially given that downloading is force to stop when a given disk gets 100% filled.

 

[trurl]

I don't use that application, but some have experienced a similar issue using other applications. I have had it happen to me using rsync to my backup server, for example, and there are other reports of it happening with Krusader.

 

What happens in the case of rsync and Krusader, the disk is still below highwater mark, and rsync, for example, has a lot of folders and files to write. It starts by creating all of the folders in advance, and then begins filling them with files. The folders were written to a disk with plenty of room, and the files got written to those folders, with the result that they all tried to go to the same disk.

 

In my case with rsync I saw what was happening and I also noticed that it was proceeding in alphabetical order. So I just started at the other end of the alphabet and started moving the empty, pre-created folders to other disks so when it came time to fill them that is where the files went.