Can I make a simple request for the next release (or help changing it on my own)? I want the mover script to preserve timestamps when moving the files, a la: cp -r --no-preserve=mode --preserve=timestamps

Can I make a simple request for the next release (or help changing it on my own)? I want the mover script to preserve timestamps when moving the files, a la: cp -r --no-preserve=mode --preserve=timestamps

I guess the question is why?

 

and rsync is used in the mover, not cp.

I always assumed the timestamps were preserved. It seems the only "right" way to do it to me. Is there some gotcha I am not considering?

 

i think they are preserved because if they weren't XBMC would see the movie again and add another copy (before/after mover).

i think they are preserved because if they weren't XBMC would see the movie again and add another copy (before/after mover).

 

Luckily XBMC uses a CRC of the path as the unique identifier so whilst it might be detected as changed it wont be added twice.

I guess the question is why?

 

I use time stamps for organizational purposes, and it gets confusing when some data looks newer than others

 

Example: If I put a folder of content on a cache enable share right now, and then a second folder 12 hours from now, when I look at my share tomorrow, after the script has run, it will look like they were created moments apart,. Plus the first folder I created might actually appear as the most recently created. Now multiple that by 50 folders, or however much stuff I've been working with, and you can see how easily it is to get lost

 

and rsync is used in the mover, not cp.

 

I'm aware of that, I actually looked into the script before posting, I just wanted to provide an example of what I was looking for. Trying to be thorough

 

I always assumed the timestamps were preserved. It seems the only "right" way to do it to me. Is there some gotcha I am not considering?

 

I guess I should have clarified the request is for folder timestamps. File timestamps are kept just fine

As far as I am concerned the mover should not modify any timestamps in any way. If it does then I suspect it is a bug. I can think of no logical reason why it would be otherwise.

As far as I am concerned the mover should not modify any timestamps in any way. If it does then I suspect it is a bug. I can think of no logical reason why it would be otherwise.

 

Must be a bug then, unless there's a conflict with the switches that are currently in place (rsync -i -dIWRpEAXogtO --numeric-ids --inplace). My mover is set to run at 3:40AM in the Web GUI, and I just checked the folders it moved over this morning. The first thing it moved has a folder creation date of 3:40AM, the next one 3:43AM, and it keeps going from there

 

I also had a folder on my cache drive from earlier this morning, went and manually clicked to invoke the mover, and sure enough, it updated the timestamp of the folder to 11:01AM

Yes, this is a problem.  In the 5.0 release, I added the -O option to rsync command in the mover script, but this was wrong.  What this does is cause all directories that get moved to have their mtime's changed to the time of move.  I will remove this option in 5.0.4, but this will not solve this reported bug.

 

The issue is that when an object is moved from a containing-directory on the cache drive, to an array disk that where the containing-directory already exists, the mtime of the containing-directory on the array disk will get updated, and I'm not sure how to prevent that...

The issue is that when an object is moved from a containing-directory on the cache drive, to an array disk that where the containing-directory already exists, the mtime of the containing-directory on the array disk will get updated, and I'm not sure how to prevent that...

 

The mover is working on a file by file basis rather than a directory basis.

This is why the directory mtimes on the destination get updated.

 

We would either need to cache the directory mtime or change the mover to do rsync based on the directory name. The issue then becomes, how to check if the file is busy.

 

I suppose you could do the rsync based on each file, then the directory, however once the file is moved, the source directory has been updated, so this would not work. I'll try to think of other ideas.

 

for Share in */ ; do
  if [ -d "/mnt/user0/$Share" ]; then
    echo "moving $Share"
    find "./$Share" -depth \( \( -type f ! -exec fuser -s {} \; \) -o \( -type d -empty \) \) -print \
      -exec rsync -i -dIWRpEAXogtO --numeric-ids --inplace {} /mnt/user0/ \; -delete
  else
    echo "skipping $Share"
  fi
done

The issue is that when an object is moved from a containing-directory on the cache drive, to an array disk that where the containing-directory already exists, the mtime of the containing-directory on the array disk will get updated, and I'm not sure how to prevent that...

 

The mover is working on a file by file basis rather than a directory basis.

This is why the directory mtimes on the destination get updated.

 

We would either need to cache the directory mtime or change the mover to do rsync based on the directory name. The issue then becomes, how to check if the file is busy.

 

I suppose you could do the rsync based on each file, then the directory, however once the file is moved, the source directory has been updated, so this would not work. I'll try to think of other ideas.

 

for Share in */ ; do
  if [ -d "/mnt/user0/$Share" ]; then
    echo "moving $Share"
    find "./$Share" -depth \( \( -type f ! -exec fuser -s {} \; \) -o \( -type d -empty \) \) -print \
      -exec rsync -i -dIWRpEAXogtO --numeric-ids --inplace {} /mnt/user0/ \; -delete
  else
    echo "skipping $Share"
  fi
done

 

Yeah, it's a pretty complex problem.  Have to do this:

 

for each file moved

  capture the mtime of the file and each component in the files path

  copy the file from cache to array [creating parent dirs as necessary]

  set the mtime of the new file

  set the mtime of each component of the files path on the array to the mtime captured above

  delete the file on the cache [this will cause files immediate parent directory mtime to change]

  set the mtime of the files parent directory back to what it was before

endfor

 

Might be easier to just write a program to do this.

Might be easier to just write a program to do this.

 

I thought the same thing. Only I might write a wrapper for rsync since I'm a lil lazy.

 

 

Cache the directory mtime, do the rsync, utime() the directory to the cached value.

 

There's also FTW64() which could traverse a tree.

Also consider... While I know this is an issue which can be programmed.

This might be one of those topics that the community could assist with.

 

I bet people would be more inclined to prioritize 64bit unRAID then this. I could be wrong.

 

 

Also consider... While I know this is an issue which can be programmed.

This might be one of those topics that the community could assist with.

 

I bet people would be more inclined to prioritize 64bit unRAID then this. I could be wrong.

Agreed, which is why I haven't done it.  Also the copy has to preserve ownership, mode and extended attributes, which is why rsync was used (cp doesn't preserve xattrs).

Any updates to this?

 

I'm thinking about switching back to cache drive to resolve the "out of disk space" errors I'm getting when running low on most of my drives

Any updates to this?

 

I'm thinking about switching back to cache drive to resolve the "out of disk space" errors I'm getting when running low on most of my drives

 

This can only be fixed with a custom mover program that patches directory mtimes.  There's hope for this to get in to 6.x because we have other functions to be implemented in a mover program.

 

On 12/6/2013 at 11:45 AM, limetech said:

The issue is that when an object is moved from a containing-directory on the cache drive, to an array disk that where the containing-directory already exists, the mtime of the containing-directory on the array disk will get updated, and I'm not sure how to prevent that...

 

Does this include the root of a share?  It seems like that will always exist.  I have tried a few times to copy data from an old computer to an EMPTY cache enabled unRAID share. I have managed to preserve the directory timestamps while copied to the cache, but every time the mover executes, the directory timestamps reflect the time the mover executed.

 

On 6/5/2015 at 2:03 PM, limetech said:

This can only be fixed with a custom mover program that patches directory mtimes.  There's hope for this to get in to 6.x because we have other functions to be implemented in a mover program.

 

Any update to integrating in 6.x?

11 minutes ago, dboonthego said:

Any update to integrating in 6.x?

 

I was just looking for this topic, and here you are posting in a 2-year old thread...

This related to:

 

 

26 minutes ago, limetech said:

I was just looking for this topic, and here you are posting in a 2-year old thread...

 

Good to know.  Thank you.

I was reading the change notes for 6.4.0 and jumped through the roof when I read there is a fix for this.   A big thank you!

 

Quote

Improved shfs/mover (-rc1)

 

The LimeTech user share file system (shfs) has been improved in two areas.  First, we now make use of FUSE read_buf/write_buf methods.  This should result in significant throughput increases.  Second, the mover script/move program no longer uses rsync to move files/directories between the cache pool and the parity array.  Instead the move program invokes a new shfs ioctl() call.  This should result in complete preservation of all metadata including atime and mtime.

 

1 hour ago, dboonthego said:

I was reading the change notes for 6.4.0 and jumped through the roof when I read there is a fix for this.   A big thank you!

 

Where have you been, man?

Been waiting patiently