log constantly filling - leads to 100% full cache and corrupt docker

[DoesItHimself]

Unfortunately I've hit a bad streak of problems lately. I had a few previous posts in the past month as well after a few years of zero issues with my server. I've had this happen a handful of times now but chalked it up to other things previously, but I've slowly ruled things out and have now gotten to this point. My log consistently gets to a full status and seems to be causing my cache drive to fill up and corrupt my docker causing me to have to rebuild it over and over.

 

I captured diagnostics after it filled (posted) before I restart so it can clear out and get some space. I had to take run xfs_repair on a drive (disk 3) that went unmountable about a week ago, and I also re-ran parity after that and it found 30k+ errors but according to what I see it repaired that. So as far as I see everything should be healthy, but this log issue is persisting and causing issues over and over.

 

I'm starting to think I am going to have to completely wipe everything and rebuild, but I've got 15TB of data across the array and I'd REALLY prefer to not have to work a method to back that up, as I currently don't have any way to back it up due to a long distance move.

zunraid-diagnostics-20210401-2100.zip

[ChatNoir]

Your log is being spammed by errors.

 

Mar 31 17:09:28 zUnraid shfs: share cache full
Mar 31 17:09:32 zUnraid kernel: BTRFS error (device loop2): parent transid verify failed on 236830720 wanted 26617 found 26510
### [PREVIOUS LINE REPEATED 357 TIMES] ###

 

Mar 31 17:43:21 zUnraid kernel: blk_update_request: critical space allocation error, dev loop2, sector 3931264 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0
Mar 31 17:43:21 zUnraid kernel: BTRFS error (device loop2): parent transid verify failed on 236830720 wanted 26617 found 26510
### [PREVIOUS LINE REPEATED 155 TIMES] ###

 

Your cache seems to be full and your docker image is throwing errors (probably because it cannot write).

 

You should first fix that, are you sure all your shares are properly set ?

 

to transfer from cache to Array = cache : Yes

to transfer from Array to cache = cache : Prefer

[DoesItHimself]

Edit: Did more digging and I think I figured out the actual issue, however I'm not entirely sure how to fix it or why it is an issue.

 

It appears my cache drive somehow got added to the list of drives included in my 'Media' share and now is being including in the disks to fill when new files are brought in. The only issue I'm having is how to remove it. I checked my Media share and it only has disk 1-5 checked which are my 5 normal disks. I can't find anywhere that shows my cache as part of that share drive, however if I add a file to the cache it populates to the Media share when I browse via rootshare -> user -> Media to see the files within the share.

Edited April 2, 2021 by DoesItHimself

[Hoopster]

3 hours ago, DoesItHimself said:

It appears my cache drive somehow got added to the list of drives included in my 'Media' share and now is being including in the disks to fill when new files are brought in

You have the media share set to cache:prefer.  This tells unRAID to write new files to this share on the cache drive if it exists and has space (prefer the cache drive as the share storage location).  If it has no space, it will write files to the array. 

 

Not only are all new files for the Media share currently going to cache but they are never moved to the array by the Mover because of the prefer setting.  In fact, running Mover will result in your cache drive filling up again as it will move files from the Media share off the array and to the cache drive as long as there is free space.  Even if you manually free up space on the cache drive, as soon as you write anything to the Media share or run the Mover, it is going straight to cache and staying there.

 

You should set the share to cache:yes if you want files written there temporarily and moved to the array (disks 1-5) by the Mover.

 

# Generated settings:
shareComment=""
shareInclude="disk1,disk2,disk3,disk4,disk5"
shareExclude=""
shareUseCache="prefer"
shareCachePool="cache"
shareCOW="auto"
# Share exists on cache,disk1,disk2,disk3,disk4

 

Edited April 2, 2021 by Hoopster

[trurl]

You can toggle Help for the whole webUI by clicking Help (?) in the main menu bar. You can also toggle help for a specific setting by clicking on its label. Take a look at the help for the Use cache setting.

 

Also. study this:

 

 

[trurl]

After you take care of moving your Media off cache, you will have to reboot to clear the log space.

 

Post new diagnostics after you have taken care of all that so we can see if you still have things that need attention.