(SOLVED) Cache full, mover not moving

[TBoneStaek]

Running 6.8.3

 

Symptoms possibly unrelated: Have 500GB SSD cache pool which typically stays at 208GB free (another issue I was beginning to question as to why it's that full). Everytime I'd run autobackup, cache would fill almost all the way but then mover would kick in and cache would reset to 208GB free. Lately though, cache would fill and seemingly crash. Today I began receiving BTRS loop errors, needed to stop the array and reboot via CLI as the webgui wouldn't respond. After reboot Mover wouldn't run and cache remains fill with only 1.9GB free. Attached are current logs after reboot. I also have logs prior to reboot if needed.

 

EDIT: reuploaded logs

 

Thanks! crom-diagnostics-20201014-1056.zip

Edited October 14, 2020 by TBoneStaek
reattached logs

[JorgeB]

Can't download diags.

[TBoneStaek]

8 minutes ago, JorgeB said:

Can't download diags.

Thanks, try now!

[JorgeB]

What share is using cache that you want moved?

 

You also appear to have shares downloads and Downloads, a setting for both exists on flash and it will only work for one, delete the one that shouldn't be there, also make sure only one of those exists on all the disks, Linux is case sensitive.

[TBoneStaek]

The share /backups has my most recent appdatabackup which takes up nearly half of my SSD (as nearly half of my SSD is full from appdata). So the most recent backup from today needs to be moved. to the array to free up cache.

 

Within my webgui, I only see the share Downloads. No second lower-cased downloads. Any other place to look for this share?

 

Thanks!

 

Just saw you said check on all the disks. Hang on... Thanks!

Edited October 14, 2020 by TBoneStaek
clarification

[TBoneStaek]

Very odd, when I go to my /boot/config/shares on my Flash, I only see a downloads.cfg. There's no Downloads.cfg.

[JorgeB]

@trurlmight know better why both appear on the diags, don't remember the exact circumstances that cause that.

 

For the mover issue, please enable mover logging, run the mover and post new diags.

[TBoneStaek]

Mover log, duh. I swear, don't get dad-brain. I just can't think like I used to.

 

So, looks like it's trying to write to disk2 which I recognize as being less than 50GB left.

 

The share, backups, though is set top autosplit as required so I don't understand why it won't go to another disk. crom-diagnostics-20201014-1148.zip

[TBoneStaek]

Ugh, now I see it, it's not the backups, it's the appdata that's stuck. I do have that set only split the top level.

[TBoneStaek]

But appdata shouldn't be pushing to the array... okay, now I'm really confused.

[JorgeB]

1 minute ago, TBoneStaek said:

But appdata shouldn't be pushing to the array... okay, now I'm really confused.

It's not, it's trying to move some data from disk2 to cache.

[TBoneStaek]

Yup, just figured that out. But, why won't it move that last backup to the array to make room? That latest backup is still on the cache causing it to remain full.

Edited October 14, 2020 by TBoneStaek
clarification

[TBoneStaek]

I can move it using unbalance... I guess.

[JorgeB]

16 minutes ago, TBoneStaek said:

why won't it move that last backup to the array to make room?

I guess the mover goes in alphabetical order, and appdata comes first.

[TBoneStaek]

Just now, JorgeB said:

I guess the mover goes in alphabetical order, and appdata comes first.

Laaaaaaaaaaaaaaaaaaaaaaaaaame. Guess I'm renaming /backups to /_backups after I run unbalance lol

[TBoneStaek]

Now if you could fix my lightscribe DVD burner that also decided to go up in the last few days, I'll be just at a par 8.

[trurl]

8 hours ago, TBoneStaek said:

Very odd, when I go to my /boot/config/shares on my Flash, I only see a downloads.cfg. There's no Downloads.cfg.

Linux is case-sensitive, so downloads and Downloads are different shares.

 

The downloads share does not exist, but it apparently did at one time, and you made settings for it, so that is why downloads.cfg is on flash.

 

User shares are the top level folders on cache and array. If a top level folder exists, a user share with the same name as that top level folder exists. People often accidentally create user shares by specifying a top level path in a docker mapping or something.

 

You created a share (top level folder) named Downloads, but you have never made any settings for it, so there's no Downloads.cfg. That share has default settings (cache-no, split any, include all, highwater, 0 minimum).