Jump to content
nickp85

[6.4rc20] - Mover and IMG files

6 posts in this topic Last Reply

Recommended Posts

Description:  Mover causes IMG files to take up their full amount of storage despite actual size of vdisk

How to reproduce: Take a raw img file that isn't full size (i.e. 50G but 25G on disk) and then let mover move it from cache to array or array to cache

Expected results: IMG file retains all its parameters

Actual results: after the move, IMG file is the full size on disk verified with ls -lsh

Other information: see examples below

 

 13G -rwxrwxrwx 1 root root  50G Jan  9 03:05 vdisk1.img*

350G -rwxrwxrwx 1 root root 350G Jan  9 03:05 vdisk2.img*

 

The first disk was created on cache and never touched by mover.  The second disk was created on my array and then I used mover to move it on to cache.  Prior to move the img was only about 240G.  I've replicated this a few times.  I then have to use qemu-img commands to build a new img file that doesn't take up the slack space.

 

This also may have been an issue in 6.3.5 also but it took me a while to catch on to the fact that the img was now taking up the full size since you have to use du or ls -lsh to see the real size on disk.

Edited by nickp85

Share this post


Link to post

Good to know, but it looks like unraid has some of those commands that can be used to move sparse files.  Perhaps mover can be enhanced to handle it properly?  If it detects an img file maybe use dd or cp with the sparse options?  Just thinking out loud.

 

Would also be nice to have an easy way to load libguestfs since it has a command to sparsify an img file in place which is ideal for img on cache where you don't have the space to convert it using a second file.

Share this post


Link to post

Running into the same issue. Sparse files are created fine on the cache drive, but mover expands them to preallocated size.

 

I am on 6.5.2-rc2.

Share this post


Link to post
12 hours ago, MrChunky said:

Running into the same issue. Sparse files are created fine on the cache drive, but mover expands them to preallocated size.

 

I am on 6.5.2-rc2.

 

The lseek() command in Linux supports handling of sparse files by use of SEEK_DATA and SEEK_HOLE, which means it's possible to teach mover to query the source file system for the holes in moved files.

 

This would result in faster moves of sparse files and the file would continue to be sparse on the destination drive.

 

If mover doesn't support sparse files, then this should probably be added as an improvement suggestion in the specific subforum.

Share this post


Link to post
On 5/25/2018 at 1:18 AM, pwm said:

 

The lseek() command in Linux supports handling of sparse files by use of SEEK_DATA and SEEK_HOLE, which means it's possible to teach mover to query the source file system for the holes in moved files.

 

This would result in faster moves of sparse files and the file would continue to be sparse on the destination drive.

 

If mover doesn't support sparse files, then this should probably be added as an improvement suggestion in the specific subforum.

 

Thank you for the explanation. I have submitted a bug report a few weeks back. We'll see what the response is, possibly I will move it to feature request :)

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.