Deleted file from Cache can't reclaim space

[darbronnoco]

Hi everyone, I started a file copy from my mac to unraid but I think the copy must have errored out at some point.  When I tried to delete the directory on my cache drive I received an error that the "database" was still in use(The file being copied was an aperture library and inside of the library there was a directory called database).  While I rebooted my laptop and unraid to make sure the files were no longer thinking they were in use but I still received the error.  I logged in with telnet and rm /mnt/cach/Directory_to_delete and I was able to remove the directory.  The problem is I could not recover the space.  from what I have seen I can't recover the space if the file is still open. So how on earth can I recover the space... this file took up about 95% of my cache drive. 

 

I know I should upgrade to 5.0 final but I wanted to make sure my system was in proper working condition before I moved forward. 

 

 

I am running 5.0-rc8a. My cache drive is 96.0GB and 85.5 GB used and 10.5 GB free.  I should have around 80GB free.

 

Thanks!

[darbronnoco]

Any ideas?  My cache drive is full until I can sort this one out.

[dgaschk]

What does "ls -la /mnt/cache" show?

 

[darbronnoco]

Thanks-

 

root@Tower:~# ls -la /mnt/cache

total 12

drwxrwx---  7 nobody users  264 2013-11-01 21:10 ./

drwxr-xr-x 11 root  root    0 2013-11-01 23:17 ../

drwxrwx---  3 nobody users  112 2012-12-23 16:40 .TemporaryItems/

-rw-rw----  1 nobody users 4096 2012-12-23 16:40 ._.TemporaryItems

-rw-rw----  1 nobody users 4096 2012-12-23 16:40 ._.apdisk

-rw-rw----  1 nobody users  291 2012-12-23 16:40 .apdisk

drwxrwx--- 11 nobody users  424 2013-01-31 23:09 TowerApps/

drwxrwx---  5 nobody users  200 2013-01-31 21:01 appdata/

 

[Influencer]

Have you rebooted since you actually removed the file? Either that or you'll have to us lsof or fuser to find what PID has that file open (not sure how that'll work since the file is gone) and either kill the PID or the filehandle.

[darbronnoco]

I have rebooted but I can give it another shot. Otherwise how do I use those other methods? 

[dgaschk]

What does "du -ah --max-depth=1 /mnt/cache" show?

[darbronnoco]

4.0K    /mnt/cache/.apdisk

4.0K    /mnt/cache/._.TemporaryItems

4.0K    /mnt/cache/._.apdisk

4.0K    /mnt/cache/.TemporaryItems

318M    /mnt/cache/appdata

 

[darbronnoco]

I ran    "fuser /mnt/cache"  and it returned nothing. 

 

root@Tower:~# fuser -v ./

 

                    USER        PID ACCESS COMMAND

./:                  root      1156 ..c.. bash

                    root      25786 ..c.. bash

 

 

 

 

 

[darbronnoco]

Just did another total shutdown with no luck.

[mikejp]

Will this do you any good...

 

http://lime-technology.com/forum/index.php?topic=27691.0;topicseen

[darbronnoco]

I would have to backup the rest of my apps on the cache drive.  I can do this as a last resort. 

[dgaschk]

See Check Disk Filesystems in my sig.

[darbronnoco]

Ok my cache disk is labeled sdf.  I stopped the array before I ran to command to make sure that all the programs were closed out.  The output is below and it requested I run --rebuild-sb so I wanted to double check before running anything. 

 

root@Tower:~# umount /dev/sdf

umount: /dev/sdf: not mounted

root@Tower:~# reiserfsck --check /dev/sdf

reiserfsck 3.6.21 (2009 www.namesys.com)

 

*************************************************************

** If you are using the latest reiserfsprogs and  it fails **

** please  email bug reports to [email protected], **

** providing  as  much  information  as  possible --  your **

** hardware,  kernel,  patches,  settings,  all reiserfsck **

** messages  (including version),  the reiserfsck logfile, **

** check  the  syslog file  for  any  related information. **

** If you would like advice on using this program, support **

** is available  for $25 at  www.namesys.com/support.html. **

*************************************************************

 

Will read-only check consistency of the filesystem on /dev/sdf

Will put log info to 'stdout'

 

Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes

 

reiserfs_open: the reiserfs superblock cannot be found on /dev/sdf.

Failed to open the filesystem.

 

If the partition table has not been changed, and the partition is

valid  and  it really  contains  a reiserfs  partition,  then the

superblock  is corrupted and you need to run this utility with

--rebuild-sb.

 

 

 

 

I also noticed that format is unknown where all the data drives are MBRL unaligned.  Not sure if that matters for the cache drive or not. 

 

[cache] => Array

                (

                    [idx] => 24

                    [name] => cache

                    [device] => sdf

                    [id] => KINGSTON_SVP100S296G_21RY101MY5SK

                    => 93778744

                    [status] => DISK_OK

                    [temp] => 24

                    [numReads] => 38438

                    [numWrites] => 154510

                    [numErrors] => 0

                    [format] => unknown

                    => green-on

                    [fsStatus] => Mounted

                    [spindownDelay] =>

                    [spinupGroup] =>

                    [deviceSb] => sdf1

                    [idSb] => KINGSTON_SVP100S296G_21RY101MY5SK

                    [sizeSb] => 93778744

                    [fsFree] => 10444540

                    [fsSize] => 78148320

                    [fsType] => reiserfs

[darbronnoco]

for what its worth all my applications are running properly on my cache drive. 

[Joe L.]

Ok my cache disk is labeled sdf.  I stopped the array before I ran to command to make sure that all the programs were closed out.  The output is below and it requested I run --rebuild-sb so I wanted to double check before running anything. 

 

root@Tower:~# umount /dev/sdf

umount: /dev/sdf: not mounted

root@Tower:~# reiserfsck --check /dev/sdf

reiserfsck 3.6.21 (2009 www.namesys.com)

 

*************************************************************

** If you are using the latest reiserfsprogs and  it fails **

** please  email bug reports to [email protected], **

** providing  as  much  information  as  possible --  your **

** hardware,  kernel,  patches,  settings,  all reiserfsck **

** messages  (including version),  the reiserfsck logfile, **

** check  the  syslog file  for  any  related information. **

** If you would like advice on using this program, support **

** is available  for $25 at  www.namesys.com/support.html. **

*************************************************************

 

Will read-only check consistency of the filesystem on /dev/sdf

Will put log info to 'stdout'

 

Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes

 

reiserfs_open: the reiserfs superblock cannot be found on /dev/sdf.

Failed to open the filesystem.

 

If the partition table has not been changed, and the partition is

valid  and  it really  contains  a reiserfs  partition,  then the

superblock  is corrupted and you need to run this utility with

--rebuild-sb.

 

 

 

 

I also noticed that format is unknown where all the data drives are MBRL unaligned.  Not sure if that matters for the cache drive or not. 

 

[cache] => Array

                (

                    [idx] => 24

                    [name] => cache

                    [device] => sdf

                    [id] => KINGSTON_SVP100S296G_21RY101MY5SK

                    => 93778744

                    [status] => DISK_OK

                    [temp] => 24

                    [numReads] => 38438

                    [numWrites] => 154510

                    [numErrors] => 0

                    [format] => unknown

                    => green-on

                    [fsStatus] => Mounted

                    [spindownDelay] =>

                    [spinupGroup] =>

                    [deviceSb] => sdf1

                    [idSb] => KINGSTON_SVP100S296G_21RY101MY5SK

                    [sizeSb] => 93778744

                    [fsFree] => 10444540

                    [fsSize] => 78148320

                    [fsType] => reiserfs

On the cache drive you need to check the FIRST partition, not the raw drive.

 

So, use

reiserfsck --check /dev/sdf1

 

(note the trailing "1" indicating the device fr the first partition)

[darbronnoco]

Thanks sorry I missed that!  Here is the output

 

reiserfsck --check started at Sun Nov  3 14:56:54 2013

###########

Replaying journal: Done.

Reiserfs journal '/dev/sdf1' in blocks [18..8211]: 0 transactions replayed

Checking internal tree.. finished                               

Comparing bitmaps..finished

Checking Semantic tree:

finished                                                                     

No corruptions found

There are on the filesystem:

Leaves 36021

Internal nodes 245

Directories 74557

Other files 131880

Data block pointers 16881478 (0 of them are zero)

Safe links 0

###########

reiserfsck finished at Sun Nov  3 14:57:14 2013

 

[itimpi]

You should be using /dev/sdf1 (and not /dev/sdf) with reiserfsck.

[darbronnoco]

You should be using /dev/sdf1 (and not /dev/sdf) with reiserfsck.

 

Thanks Joe pointed that out.  I have my output from the properly ran command above. Everything seems to be ok on the scan.

[darbronnoco]

Still trying to work this issue out.  If anyone has any other ideas before Influencer and I have a chance to talk I would be willing to give it a try. 

[DaleWilliams]

Something similar happened to me.

(Not sure any of this helps you, but I've been following your saga for clues to the problem I encountered.)

 

I was cleaning up a lot of old files...

I had two large file copies simultaneously in operation from my Mac to unRAID.

I had lots of empty space on unRAID, and I use a cache drive.

Either copy operation would fit nicely on unRAID, and both together on unRAID would still have left 30% empty space.

What I forgot about, was the cache drive's size.

Turns out that while either copy would work, the two together wouldn't fit on the smaller cache drive.

Obviously, the Mac looked ahead and saw that each copy would have plenty of space.

 

The Mac copy operations appears to first create a '.dot' filename for each file that is to be copied.

So to copy 'FOO.BAR', and 'TWO.BAR', , the Mac first creates all the 'dot' files '.FOO.BAR', '.TWO.BAR', etc. on the destination.

Once all the .dot files and folders are setup, Then the actual files and folders begin to copy.

As each actual file completes, its 'dot' file is deleted.

 

When the two file copies (running simultaneously) filled the cache drive, both COPY commands stopped with 'Disk Full'.

This left behind many many invisible dot files on the cache drive, and some of them were very large, and many were undeletable from the finder.

 

In the end, I had to use Midnight Commander (the builtin MC command from the console) to find and delete all the invisible file trash...I don't understand 'why' but some of the trash files were pretty huge.

There was also something about the way the Permissions were set, that required that the directory of .dot files be traversed by hand...deleting files, then the folder that held the files, etc.  Took a long time.

 

 

[darbronnoco]

Something similar happened to me.

(Not sure any of this helps you, but I've been following your saga for clues to the problem I encountered.)

 

I was cleaning up a lot of old files...

I had two large file copies simultaneously in operation from my Mac to unRAID.

I had lots of empty space on unRAID, and I use a cache drive.

Either copy operation would fit nicely on unRAID, and both together on unRAID would still have left 30% empty space.

What I forgot about, was the cache drive's size.

Turns out that while either copy would work, the two together wouldn't fit on the smaller cache drive.

Obviously, the Mac looked ahead and saw that each copy would have plenty of space.

 

The Mac copy operations appears to first create a '.dot' filename for each file that is to be copied.

So to copy 'FOO.BAR', and 'TWO.BAR', , the Mac first creates all the 'dot' files '.FOO.BAR', '.TWO.BAR', etc. on the destination.

Once all the .dot files and folders are setup, Then the actual files and folders begin to copy.

As each actual file completes, its 'dot' file is deleted.

 

When the two file copies (running simultaneously) filled the cache drive, both COPY commands stopped with 'Disk Full'.

This left behind many many invisible dot files on the cache drive, and some of them were very large, and many were undeletable from the finder.

 

In the end, I had to use Midnight Commander (the builtin MC command from the console) to find and delete all the invisible file trash...I don't understand 'why' but some of the trash files were pretty huge.

There was also something about the way the Permissions were set, that required that the directory of .dot files be traversed by hand...deleting files, then the folder that held the files, etc.  Took a long time.

 

That sounds exactly like what happened to me!  I was copying 2 large files over.  I'm thinking maybe my cache drive free space % isn't set correctly because I thought it would start going directly to the array if the cache was filled.  I will try digging around to see if I can find those .dot files.  Thanks for sharing.

[darbronnoco]

Well crap...

 

there were the only .files and I killed them but it didn't see to change the space any.

 

? ._.TemporaryItems                                ?  4096?Dec 23  2012??*mkmbr                                            ?  7565?Sep 25  2012?

 

? ._.apdisk                                        ?  4096?Dec 23  2012??@powerdown                                        ?    25?Sep 25  2012?

 

? .apdisk                                          ?    291?Dec 23  2012??@samba     

 

 

 

 

[DaleWilliams]

Yes.

I also thought that when the cache drive filled, future writes would go straight to the array.

(I was certain I'd read that somewhere.)

 

Sorry that didn't help.

[darbronnoco]

Well Influencer sorted out the issue.  Turns out Plex Sync content was not getting deleted.  There must be some pled related issue that caused the synced content to balloon. 

 

Thanks everyone for your help.