Hi!

 

Since a week or so back I've had this error reported in Netdata:

BTRFS allocated space utilization.

 

Since the only drives I have that uses BTRFS is the cache-drives (my main storage NVME, my media cache disk and my NVR cache disk) I've isolated these three. I don't, however, know how to decipher which of these three that it concerns. I also dont know what the problem is and how to ammend it?

 

I am sorry if this question is asked in the wrong part of this forum - fairly new here but eager to learn.

 

Cheers!

Kalle

You are likely to get more informed feedback if you attach your system's diagnostics zip file to your next post in this thread.  
It is always a good idea to do this to allow us to see the current state of your system and so we can see logs.

11 minutes ago, itimpi said:

You are likely to get more informed feedback if you attach your system's diagnostics zip file to your next post in this thread.  
It is always a good idea to do this to allow us to see the current state of your system and so we can see logs.

Thank you kindly for the advise. Please find my diagnostics attached

ash-diagnostics-20250216-0958.zip

I see that you have run a balance on the 'nvr' pool and that could well have been the cause of the warning and the balance rectified the issue.

 

FYI:  The docker.img file also uses btrfs internally so you can occasionally get this type of warnings for that.

 

Other potential issues:

• Docker.img file is 100GB - any reason it is that large?   If you find it was running out of space then it probably means that you have a docker container incorrectly configured to write internally to the image instead of to an externally mapped volume
• You have part of 'appdata' on the main array and the share configured so that mover will ignore it.   You normally want all of this share all on a pool for best performance.

appdata                           shareUseCache="only"    # Share exists on main, cache

21 hours ago, itimpi said:

I see that you have run a balance on the 'nvr' pool and that could well have been the cause of the warning and the balance rectified the issue.

 

FYI:  The docker.img file also uses btrfs internally so you can occasionally get this type of warnings for that.

 

Other potential issues:

• Docker.img file is 100GB - any reason it is that large?   If you find it was running out of space then it probably means that you have a docker container incorrectly configured to write internally to the image instead of to an externally mapped volume
• You have part of 'appdata' on the main array and the share configured so that mover will ignore it.   You normally want all of this share all on a pool for best performance.

appdata                           shareUseCache="only"    # Share exists on main, cache

Yes I was currently running a balance on all cache pools but this was initiated after the warning occured - to possibly amend the problem

 

Thank you for the info regarding docker.img - this seems to be the culprint. Is there anyway I can investigate further what caused the issue?

 

Regarding other issues:

- The Docker.img is quite big I agree. It doesnt seem to grow anymore but I've tried reviewing all Dockers (50ish or so) to see if anyone is writing internally instead of to a set path outside the Docker. Could this be found in some automatic manner? I've failed to find anything myself.

- The appdata to array is a great find! I'll review and see if I can make some changes.

Cheers!
Kalle

49 minutes ago, raeserfisk said:

Thank you for the info regarding docker.img - this seems to be the culprint. Is there anyway I can investigate further what caused the issue?

There was no suggestion in the diagnostics that I could see about corruption in the docker.img file.   It is mounted at /dev/loop3 and so any corruption messages would refer to this device.

 

The commonest cause of BTRFS corruption in the docker.img file is issues with RAM.    If this does happen then after rectifying any RAM related issue you can recreate the docker.img file and then restore your containers using Apps->Previous Apps.

Based on the capacity, the warning is based on the docker image, post the output from:

 

btrfs fi usage -T /var/lib/docker

 

Absolutley! 

 

Overall:
    Device size:                 100.00GiB
    Device allocated:             99.91GiB
    Device unallocated:           93.00MiB
    Device missing:                  0.00B
    Device slack:                    0.00B
    Used:                         51.93GiB
    Free (estimated):             46.19GiB      (min: 46.14GiB)
    Free (statfs, df):            46.19GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              139.89MiB      (used: 0.00B)
    Multiple profiles:                  no

              Data     Metadata System                              
Id Path       single   DUP      DUP      Unallocated Total     Slack
-- ---------- -------- -------- -------- ----------- --------- -----
 1 /dev/loop2 94.85GiB  5.00GiB 64.00MiB    93.00MiB 100.00GiB     -
-- ---------- -------- -------- -------- ----------- --------- -----
   Total      94.85GiB  2.50GiB 32.00MiB    93.00MiB 100.00GiB 0.00B
   Used       48.75GiB  1.59GiB 16.00KiB

The image is almost fully allocated, this can be a problem sometimes, if it needs to create a new metadata chuck for example, you can balance the filesystem, but having such a large docker image, and the fact that it was at least once close to being full, may indicate a container misconfiguration.

 

btrfs balance start /var/lib/docker

 

When done, post once more:

 

btrfs fi usage -T /var/lib/docker

 

23 minutes ago, JorgeB said:

The image is almost fully allocated, this can be a problem sometimes, if it needs to create a new metadata chuck for example, you can balance the filesystem, but having such a large docker image, and the fact that it was at least once close to being full, may indicate a container misconfiguration.

 

btrfs balance start /var/lib/docker

 

When done, post once more:

 

btrfs fi usage -T /var/lib/docker

 

Done!

 

Here's the updated version

Overall:
    Device size:                 100.00GiB
    Device allocated:             54.06GiB
    Device unallocated:           45.94GiB
    Device missing:                  0.00B
    Device slack:                    0.00B
    Used:                         51.92GiB
    Free (estimated):             47.19GiB      (min: 24.22GiB)
    Free (statfs, df):            47.19GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              136.19MiB      (used: 0.00B)
    Multiple profiles:                  no

              Data     Metadata System                              
Id Path       single   DUP      DUP      Unallocated Total     Slack
-- ---------- -------- -------- -------- ----------- --------- -----
 1 /dev/loop2 50.00GiB  4.00GiB 64.00MiB    45.94GiB 100.00GiB     -
-- ---------- -------- -------- -------- ----------- --------- -----
   Total      50.00GiB  2.00GiB 32.00MiB    45.94GiB 100.00GiB 0.00B
   Used       48.75GiB  1.59GiB 16.00KiB   

 

---

 

It seems there's now a big difference in allocated space. Should I keep an eye out here for increasing allocation? And what could be the reason if it starts increasing?

Edited February 17, 20251 yr by raeserfisk

19 minutes ago, raeserfisk said:

And what could be the reason if it starts increasing?

As was previously mentioned this normally means that you have a container writing to a location that is not mapped to storage external to the container.

As far as I know it's perfectly normal, as btrfs does copy on writes when something is changed a new block is allocated to store it, the old location is freed internally but not deallocated. So after a while it's normal for the entire medium to be allocated and it'll remain so unless you run a balance which forces it to reorganize/consolidate everything at which point it'll deallocate what it doesn't need.

 

Here's my cache, 100% allocated too, has been for years probably. 

 

Netdata makes a warning for it but it's silent by default and is just not meaningful.

 

 

 

 

 

 

2 hours ago, raeserfisk said:

Should I keep an eye out here for increasing allocation?

Typically, it's not needed, and the docker image should never get close to max space used, but keep an eye on it, a fully allocated btrfs is not necessarily a problem, but it can be if a new metadata chunk needs to be created, and there's no unallocated space available

@JorgeB, @Kilrah & @itimpi: Thank you so much for all your efforts. I'll keep an eye out if the actual usage starts increasing and I'll relax on the warning from Netdata. 

Just bumping this as I too am running Netdata and had the same warning which was the same Docker vidsk problem.

And a balance sorted it out.. I'm confident it was down to me installing around 12 containers in a day and messing around (now all tidied up!), I've double checked all containers and can't find any misconfiguration now and will keep an eye on things.

I wonder if it's something that would be useful adding to the Unraid UI? i.e. if BTRFS allocation being that high is a potential sign of an issue, why not inform users?

2 hours ago, Snubbers said:

if BTRFS allocation being that high is a potential sign of an issue, why not inform users?

Because it only netdata's debatable point of view that it is...