mover fails to move some files [efforts to resolve resulted in missing files!!]


Recommended Posts

I'm not sure why this is happening, or when it started, but my cache disk is filling up, so I started looking into what's on the cache drive.  There's a lot that should have been moved, but is still there.  I manually started the mover, but things remain that should not.

 

I see the 'errors' in the syslog (attached), but I'm not sure why there's "No space left on device".  Is it related to the fact that it's under "user0", which I'm not sure I understand (why is there another user?)

 

Ideas?

syslog.zip

Link to comment

User0 exists to facilitate the mover functionality. I would suggest ensuring there is enough free space on the destination drive, and to remember that is chosen based on split, allocation method and disk inclusion/exclusion.

 

Sent from a mobile device, sorry for any typos.

 

 

Link to comment

That they may have been 'moved' but not deleted from the cache drive seems the most likely, but I'm not sure how to confirm, nor how to fix this situation.

 

I would have to look at 10 different drives to see if a file on the cache is also on one of the other drives, which seems a lot of work to test a theory.  I'd be willing to spend the time to do it, but even if I confirm this is the case, how would I fix this situation?  I haven't looked at all folders, but in the few I have checked, there are dozens of files still on the cache that I've checked, so I assume there are hundreds of files 'stuck' on the cache drive.  I don't want to try moving them one at a time, so how would I move them in bulk?

Link to comment

Thanks.  I tried opening midnight commander, then moving whole folders from the cache to a user share, and that seemed to work.  I assumed it would not, since normally sending to a user share just puts the files on the cache drive, but when I finished, I had over 400GB of free space on the cache drive again.  Not sure how it got jacked up in the first place, nor why mover wasn't working, but I suspect it was duplicate files causing the issue.

 

I don't have unmenu installed, so couldn't use that to check for duplicates.  It would be a great addition to the core unRAID functionality though.

 

Anyway, I'm calling this fixed now.  Thanks for all the help.

Link to comment

Mover *SHOULD* have been deleting things...calling this 'fixed' may be premature.  :-\

 

You might run Reiserfsck --check on your cache drive.

And run the permissions fix mentioned in the v.5 install instructions.

 

Agreed, but *my* issues are solved, even if mover may yet be broken.  The lack of response from LimeTech tells me they are focused on other things right now, so if I call it solved or not is probably irrelevant at this point.

 

How do I run Reiserfsck on the cache drive?  What is the exact command, and do I run it from Putty?

 

thanks

Link to comment

Putty should do fine. I use Telnet from my Mac.

See this link:

http://lime-technology.com/wiki/index.php?title=Check_Disk_Filesystems

Watch the cautions in Red...but basically, no harm will come if you ONLY '--check'

 

with the ARRAY in MAINTENANCE MODE:

'Reiserfsck --check /dev/sdX1'  where 'X' is the cache drive identifier.

(on the webpage at the bottom, in 'Additional Comments' there's a discussion of the cache drive. Don't forget the digit '1' for the cache partition number) The 'X' can be figured out from your syslog.

 

Because Cache drive is slightly different than the data drives, you also might want to read THIS thread:

http://lime-technology.com/forum/index.php?topic=15942.msg147860#msg147860

where Joe L. explains it better.

 

 

Link to comment

I would NOT move or even copy files from a disk share to a user share! It is a recipee for data loss!! If you did it you should check that the files arrived on an array disk.

 

I copied from cache to user share, I'm not sure if this is the same thing.  Regardless, it's done, so if I lost files, so be it.  Maybe this thread will catch the attention of LimeTech, and they can/will investigate the issue with mover for future users.

Link to comment

If the cache is in the share I think it is the same thing. Removing a disk from the share does NOT fix the problem.

 

Please let me know if this happened to you. I have been trying to raise awareness on this issue with Limetech and a real user losing data would get more attention than the theoretical.

Link to comment

If the cache is in the share I think it is the same thing. Removing a disk from the share does NOT fix the problem.

 

Please let me know if this happened to you. I have been trying to raise awareness on this issue with Limetech and a real user losing data would get more attention than the theoretical.

 

I must admit that I don't really follow you here.  I have no idea if 'the cache is in the share'.  For me user shares are shares I manually setup, and the cache drive is an unRAID construct.

 

I also don't know what you mean by 'removing a disk from the share' in this context.  I have no disks assigned, nor specifically removed from any of my user shares, and I did not change that during my testing, or as part of the moving I did in Midnight Commander.

 

With all that said, yes, I lost files.  I had a movie that I had recently finished downloading, which is no longer to be found anywhere on my unRAID server.  I had hundreds of files, maybe thousands that were 'stuck' on the cache drive that mover would not move, and no (easy) way to figure out what those files were.

 

Now that I realize what has happened, I'm not at all happy about it.  If you can bring this to the attention of LimeTech, I'd appreciate it, and will be happy to help them test/find a resolution.

 

Thanks.

Link to comment

Let me at least explain what happened so hopefully all will understand. This is confusing, so please ask questions. Do not glance through anything as all of this is important to understanding.

 

What are user shares?

 

A user share is a construct by which the files and subdirectories in root directories on ALL of the disks in an array are logically combined to form a logical mounted filesystem exported as a network share. The thing that combines them is called FUSE I believe.

 

If a directory on a disk contains a subdirectory of the name of the user share, it participates in the user share. You have no control over this. The user share settings DO NOT impact whether a disk participates in a user share. The only way to remove a disk from a user share is to literally rename the subdirectory so that it does not have the same name as the user share. You may have to stop and start the array afterwards - I am not sure. You may even have trouble renaming it - I've never tried as I have never been a user share fan. But I repeat, if the root of any drive has a subdirectory with the same name as the user share, its files are combined with the other files on other disks to form the user share.

 

So what does the setting for user shares mean?

 

The setting affects what disks NEW files are copied to if they are written to the share. So lets say you have a share called "Movies", and the "Movies" folder exists on the disks 1, 2, and 3. But the setting says only disk3, then new files will be written to disk3. So if you copy "NEWMOVE.ISO" to the user share, it will go on disk3. But the files from disk1, disk2, and disk3 are all in the Movies share.

 

Now what happens if you do not create a new file, what happens if you are overwriting an existing file. Let's say that file lives on disk2. Well the file on disk2 will be overwritten. I do not believe it is written to the cache even if there is a cache disk but I may be mistaken. Perhaps you can confirm. (see ramblings below after reading the rest)

 

Ok, so how does this result in losing my data?

 

Ok, so let's say you try to copy or move a file from disk share "disk2" to the user share "Movies". The user share thinks you are overwriting an existing file. If that file were coming from a Windows box or some other directory, it would work just fine and overwrite the file just like you want. But coming from you disk2, unRAID will try to copy the file from disk2 to disk2. The operating system will usually have special protections to not let you copy a file on top of itself, but the user share somehow bypasses that protection. The net result is that the file is truncated. And if you were moving the file, it is then deleted from disk2, which also deletes it from the user share. Poof.

 

You copied files from cache to the user share, and since the files were on the cache drive, they were copied overtop of themselves. Poof.

 

I'm not trying to be cute. But this is what happens. Users should never copy files from a disk share to a user share. You either copy from a disk share to a disk share, or from a user share to a user share. But don't mix them.

 

Ramblings

 

If a cache disk changes the user share behavior, and instead of overwriting a file it stages it to the cache disk and plans to overwrite it via the mover, then a cache disk would protect you if copying from a disk share, except cache, to a user share. Again, I am not sure it works this way and a test would need to be performed to see. It sort of makes sense to me, given the very few situations that this has occurred, that this might be the behavior. But you were copying files from the cache disk - so they would go right back to the cache disk. And if they originated in the subdirectory for the user share, you would lose data as described.

Link to comment

Bjp999,

So following your logic, wouldn't this work:

1. Files are 'stuck' on the cache drive and may or may not have been written to user shares.

2. "movies" share is defined and is a top level folder on disk1, disk2, and disk3.

3. Using MC (or whatever) copy the top level folder contents FROM 'Cache/movies' into "DISKx Share/movies" (where x is 1,2,or 3)

 

This is a specific copy from a specific physical disk (cache) to another specific physical disk.

 

The problem is that the same file might end up in disk1/movies as well as disk2/movies, but that's usually better than losing a file. And there are tools to detect duplicate files.

 

Link to comment

Also if copying to a "user0" share folder is fine.  That is how mover does it.  I do that all the time with MC on unRAID.  What doesn't work is a "user" share folder since "user" share folder's include the cache drive but "user0" share folders do not.

 

So if I copy from "/mnt/cache/ATSC/Movies" to "/mnt/user0/ATSC/Movies" it works fine.

 

But if I copy from "/mnt/cache/ATSC/Movies" to "/mnt/user/ATSC/Movies" it would delete the files as bjp999 points out.

 

Link to comment

Also if copying to a "user0" share folder is fine.  That is how mover does it.  I do that all the time with MC on unRAID.  What doesn't work is a "user" share folder since "user" share folder's include the cache drive but "user0" share folders do not.

 

So if I copy from "/mnt/cache/ATSC/Movies" to "/mnt/user0/ATSC/Movies" it works fine.

 

But if I copy from "/mnt/cache/ATSC/Movies" to "/mnt/user/ATSC/Movies" it would delete the files as bjp999 points out.

 

It may be ok to move from the cache drive to the user0 share.  But it is NOT ok to copy from another disk in the protected array to a user or user0 share.

 

But I really suggest, when copying or moving files from one place to another on your array, that you stick with disk shares. It takes all of the danger out of it.

 

I contacted Limetech on this issue and this is on the list of high priority issues to address.

Link to comment

Also if copying to a "user0" share folder is fine.  That is how mover does it.  I do that all the time with MC on unRAID.  What doesn't work is a "user" share folder since "user" share folder's include the cache drive but "user0" share folders do not.

 

So if I copy from "/mnt/cache/ATSC/Movies" to "/mnt/user0/ATSC/Movies" it works fine.

 

But if I copy from "/mnt/cache/ATSC/Movies" to "/mnt/user/ATSC/Movies" it would delete the files as bjp999 points out.

 

It may be ok to move from the cache drive to the user0 share.  But it is NOT ok to copy from another disk in the protected array to a user or user0 share.

Correct.  Sorry I should have pointed that out.
Link to comment

I just wanted to update the thread with a couple of updates.

 

1. I had originally created the share "download" as a share that uses cache, but not exclusively, then later changed it to use cache only.  However, in the meantime, mover had moved some files to the download directory on 2 of my disks, which caused unRAID to be 'confused' and was including the 2 disks in the size calculation of the share called "download" and also caused mover to try to move files from this (now) cache only share to the array.  removing the folders from the 2 disks resolved this issue.  however, I still had problems with mover not working, and giving 'no room available' error messages, which I resolved by

 

2. Somehow, at some point in the fairly recent history, my user share called "video" had gotten changed from use cache: yes to use cache: only, which was preventing it from actually moving the files.  I finally realized this issue, then changed it back to "yes", then mover started working again.  I ended up with a couple of duplicate files, which i moved with Midnight Commander from the cache disk to the individual disk, and now it seems that I finally have a 'clean' cache disk.

 

Thanks for everyone's help, and hopefully my (self-induced) issue has raised awareness with LimeTech of a potential problem, and leads to a better product for everyone.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.