The ability to see how fast the mover was copying, like a parity rebuild.


Recommended Posts

For what purpose?    You already know that writes to the protected array are MUCH slower than writes to a cache drive ... that's the whole idea of using a cache.    The mover is going to write at essentially the same speed the writes would have been at if they hadn't been cached ... which will, as NAS noted, be appreciably slower for a lot of small files vs. a few larger files.    But the speed shouldn't matter => that's why it's designed to run overnight (although you can, of course, schedule it for other times, or simply force it to run whenever you want).

 

Link to comment

For those asking why this feature would be nice.

 

This feature would be nice for knowing how much work your cache disk is doing. If I know the mover is running and it's transferring at a certain rate. Perhaps I would wait to make a copy. But if it were going pretty slowly and I know that I have some bandwidth free for another copy to it.

 

Maybe this idea isn't a winner, but I would like this feature.

 

Thanks!

Link to comment

Mover is just rsync right?  So this request is for adding the --progress option to the rsync command, and then capturing its output and putting it into the GUI in a usable form.  Technically, this number will be the speed of constructing the current single file at the destination, but should be close to the transfer speed.  Trying to keep the GUI updated for each file will probably impact transfer performance, especially when transferring many small files, but since this is the Mover that is not important.

Link to comment

Some food for thought.

 

FWIW, the mover writes it's pid in /var/run/mover.pid.

If the gui was active it could grep lines from syslog matching the pid if it were tagged properly.

 

The --progress option has useful information, but since it's designed for a terminal using \r instead of \n, it would not work well being piped into syslog via logger.

In addition --progress tells you how many files and where you are in that list but that only works when doing one rsync to move a bunch of files.

Currently the mover does a find and executes rsync many times, 1 rsync per 1 file.

 

Changing mover to log to it's own log file and tailing that via the gui might work, but it's not necessary, The mover just needs to be tagged properly in the syslog with -tmover[pid]

 

Also not all that difficult with a co-process in the mover depending on the bash version.

 

I think the real gotcha for telling where you are in the list is the mover would need to be changed to capture how many files will be affected.

Then provide some kind of indication of where it was in that list.

 

I.E. a /bin/find capturing the file list, counting it, or iterating through it.

It's much more complex then the current bash script that's in place using find and rsync.

Link to comment

Agreed.

 

On the other hand, I've seen so many awful progress indicators, that I suspect most people's expectations are very low.  We could create a rudimentary indicator by dividing 100 by the number of shares.  So with an example of 5 shares, some with files to move, the indicator might jump to 20%, hang there awhile, then jump to 60%, hold briefly, then be done.  Almost useful.  But with that done, in future updates the developer could begin step-wise refinement, adding stepping updates to the number by either file counting or file sizes.  By file count might be easier, but using file sizes would give a more accurate progress percentage.

 

I recall one progress bar, that would stick at 0% for about 15 seconds, then jump to 73% and hang there for 2 minutes, then jump to 84% and hang there for a minute, then jump to 100% and hang there for 5 minutes.  I don't know why the programmer bothered.

Link to comment

Another idea might be to let the mover do it's own thing with a co-process monitoring disk usage.

When the mover ends it can send a signal to the co-process or remove the co-process's pidfile thus signaling it should exit.

Or perhaps, (and again I'm thinking out loud so people get ideas) add another -exec line to the find  :o  that calls du -hs on the share.

In these cases, using du the progress would not be going up, but would be going down as files are moved from the cache share.

You would at least know how much data is left. If you know your array writes generally get from 60MB/s to 30MB/s you can guesstimate duration.

 

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.