Unmover appears to work very slowly...

[jeffreywhunter]

Is there a way to determine the file transfer rate for Mover?  I have about 280gb data on the cache disk that I wanted to move, clicked move now in settings>scheduler, it says 'moving', but its been running for nearly an hour and very slow progress...

 

Slow Mover progress from syslog.

http://my.jetscreenshot.com/12412/20150506-gywg-307kb

 

Main display showing drive status

http://my.jetscreenshot.com/12412/20150506-b6kx-161kb

 

Red arrow shows 279gb of files on the cache, still there - even after many refreshes over an hour - and not reduced even though a few files transfered (see first screenshot).  Blue arrow shows where 2gb of files were deleted and space freed.

 

Does the mover take a long time to run?  Is there some place where I can see how fast it is actually moving files (mb/s?)?  Does mover really copy, then delete the files...i.e. after they have ALL been moved, then they are deleted from the cache?

 

Finally, is there a link somewhere that fully discusses how mover works?

[trurl]

root@unSERVER:~# which mover
/usr/local/sbin/mover

root@unSERVER:~# cat /usr/local/sbin/mover
#!/bin/bash

# This is the 'mover' script used for moving files from the cache disk to the
# main array.  It is typically invoked via cron.

# After checking if it's valid for this script to run, we check each of the top-level
# directories (shares) on the cache disk: if the share is marked "cache-only" then we
# skip this directory; otherwise, we use the 'find' command to process the objects (files
# and directories) of that directory, moving them to the array.

# The script is set up so that hidden directories (i.e., directory names beginning with a '.'
# character) at the topmost level of the cache drive are also not moved.  This behavior can be
# turned off by uncommenting the following line:
# shopt -s dotglob

# Files at the top level of the cache disk are never moved to the array.

# The 'find' command generates a list of all files and directories on the cache disk.
# For each file, if the file is not "in use" by any process (as detected by 'fuser' command),
# then the file is copied to the array, and upon success, deleted from the cache disk.
# For each directory, if the directory is empty, then the directory is created on the array,
# and upon success, deleted from the cache disk.

# For each file or directory, we use 'rsync' to copy the file or directory to the array.
# We specify the proper options to rsync so that files and directories get copied to the
# array while preserving ownership, permissions, access times, and extended attributes (this
# is why we use rsync: a simple mv command will not preserve all metadata properly).

# If an error occurs in copying (or overwriting) a file from the cache disk to the array, the
# file on the array, if present, is deleted and the operation continues on to the next file.

# Only run script if cache disk enabled and in use
if [ ! -d /mnt/cache -o ! -d /mnt/user0 ]; then
  exit 0
fi

# If a previous invokation of this script is already running, exit
if [ -f /var/run/mover.pid ]; then
  if ps h `cat /var/run/mover.pid` | grep mover ; then
      echo "mover already running"
      exit 0
  fi
fi
echo $$ >/var/run/mover.pid
echo "mover started"

cd /mnt/cache
shopt -s nullglob
for Share in */ ; do
  if grep -qs 'shareUseCache="only"' "/boot/config/shares/${Share%/}.cfg" ; then
    echo "skipping \"${Share%/}\""
  else
    echo "moving \"${Share%/}\""
    find "./$Share" -depth \( \( -type f ! -exec fuser -s {} \; \) -o \( -type d -empty \) \) -print \
      \( -exec rsync -i -dIWRpEAXogt --numeric-ids --inplace {} /mnt/user0/ \; -delete \) -o \( -type f -exec rm -f /mnt/user0/{} \; \)
  fi
done

rm /var/run/mover.pid
echo "mover finished"

I don't completely understand all of that, but I think it is looping through the top level folders on cache that are not set to cache-only, and moving them to /mnt/user0 (which is the user shares excluding cache) by using rsync with delete.

 

Constantly refreshing the GUI can affect performance.

 

Mover writes to the array using the usual unRAID realtime parity method; i.e., read data to be overwritten, read corresponding parity, calculate parity change, write data, write parity. I think 30-40 mb/s is typical.

[garycase]

Agree ... a rate of about 30-40MB/s is typical for writes to the protected array.

 

Not sure why you're seeing such slow speeds (if you've been running for an hour, that's EXCRUCIATINGLY slow with only 17MB copied (that much should have been done in the first second !!).

 

One interesting thing I noticed that I haven't seen before:  You have your entire array set to use BTRFS.  I'd have expected the array drives would be XFS (or Reiser).    Don't know if that's an issue or not ... but you might want to try switching to XFS while you still only have a modest amount of data you'll need to recopy (you'll lose everything on the drives if you switch to a different file system).

 

[jeffreywhunter]

Hmmm, I thought BTRFS was the latest and best file system?  I made that choice based on Tom's post regarding BTRFS has more attention in the community and will soon overtake XFS. 

 

And you are right, I'm at the point where I can change my FS without too much pain.  Which FS should I use?  BTRFS, XFS or ReiserFS?

 

Use case - Media, Documents, backups & some dockers - Plex, Cloudfile storage, etc.  I want snappy speeds for file copies, etc...

[jonp]

 

 

Hmmm, I thought BTRFS was the latest and best file system?  I made that choice based on Tom's post regarding BTRFS has more attention in the community and will soon overtake XFS. 

 

And you are right, I'm at the point where I can change my FS without too much pain.  Which FS should I use?  BTRFS, XFS or ReiserFS?

 

Use case - Media, Documents, backups & some dockers - Plex, Cloudfile storage, etc.  I want snappy speeds for file copies, etc...

 

Btrfs is on its way to becoming THE filesystem of choice for Linux, but it still has some development ongoing. For array devices or raid1 pooling, I've not really had any issues. From a feature standpoint, btrfs supports full checksumming of data and subvolumes/snapshots using copy on write.

 

I'd be surprised if a switch to XFS solved your performance issue. It seems to me like something else is going on. Then again, if its simple enough for you to switch one disk to XFS and try, why not?

[garycase]

If JonP has already tried using btrfs for array disks, then that's likely not the issue ... but it certainly wouldn't hurt to try it.

 

I think most folks are configuring v6 setups using btrfs for the cache (so they can set up fault tolerant cache pools); and XFS for their array disks.

[jonp]

I agree with Gary's assertation of what most folks are doing, and think that XFS is a very solid filesystem.  For those that don't have a need for checksumming at the filesystem level, or those that skeptical about the reliability of BTRFS, XFS is a "tried and true" filesystem that is ideal for array devices.  In fact, I remember a blog post I wrote early on when additional file system support was added to unRAID where we stated exactly that.

[Helmonder]

 

 

Hmmm, I thought BTRFS was the latest and best file system?  I made that choice based on Tom's post regarding BTRFS has more attention in the community and will soon overtake XFS. 

 

And you are right, I'm at the point where I can change my FS without too much pain.  Which FS should I use?  BTRFS, XFS or ReiserFS?

 

Use case - Media, Documents, backups & some dockers - Plex, Cloudfile storage, etc.  I want snappy speeds for file copies, etc...

 

Btrfs is on its way to becoming THE filesystem of choice for Linux, but it still has some development ongoing. For array devices or raid1 pooling, I've not really had any issues. From a feature standpoint, btrfs supports full checksumming of data and subvolumes/snapshots using copy on write.

 

I'd be surprised if a switch to XFS solved your performance issue. It seems to me like something else is going on. Then again, if its simple enough for you to switch one disk to XFS and try, why not?

 

If checksumming will be included in unraid at some moment, that would be the moment for me to move to btrfs.. would be an awsome addition (if there would not be an effect on writing speed).

[jonp]

Its already in BTRFS on unraid. The command to validate data integrity is also already there (its referred to as scrub).

[jonp]

I should also note a planned feature to add to the scheduler will be periodic btrfs scrubs. Also, btrfs is the only filesystem I know that also let's you do a file system integrity check live, while data is in use and the array is running.

 

[Helmonder]

Its already in BTRFS on unraid. The command to validate data integrity is also already there (its referred to as scrub).

 

Great !  I have a 6tb allmost redy with its 3 pass preclear, I will make it BTRFS !

[Helmonder]

Done... copying over my last 2TB red.. I will keep the 6TB my only btrfs in the mix at the moment.. See how it behaves..