Accelerator drives


jonp

Recommended Posts

My idea here is to allow, via automatic process and/or manual definition, users to place data onto higher performance drives whilst still being part of the array (think file level tiering much like SANs often offer block level tiering)

 

Currently the expensive disks would obviously be SSD vs traditional spinners however as time passes and SSD becomes the norm it is likely that we will see slow (bulk storage SSD) and fast (expensive bleeding edge performance SSD) making the feature equally valuable long term.

 

This would be similar in nature to user shares that are currently allowed to use the cache drive but on a longer term basis maintaining  array RAID protection at all times. Done right this would give the user access to both real and perceived speed improvements (i.e. much as cache_dirs gives perceived real world speed improvements).

 

Basic examples of accelerator definitions would be:

 

      Array wide all files < 10kB.

      Array wide all files of a certain type e.g. nfo, sfv, md5

      Regex match off files in the xx user share

      Manual placement of important files e.g. kids movies they watch every day

 

It is expected that given the flexibility users would find clever ways to match their data storage and usage needs to the available features. At a minimum it would offer a simple and cost effective way to insert SSD into the array without risking it being eaten but less useful data.

 

Much of this could be done manually or in inelegant ways with the current v6x feature set which suggests it is not a huge leap to add this as a supported feature.

 

 

Link to comment
  • 6 months later...

I am keen to start some work on this since in theory the logic is simple enough (although I admit the details will add no small amount of complication).

 

As I see it there are a few hurdles:

 

1. Creation of a script that will move files from drive X to drive Y whilst maintaining metadata and folder structure

2. Create a means where users can define file types to be caught by this script (in such a  way as they dont have to resort to bash)

3. Create a means where this can be applied to existing on array data.

 

 

Luckily #1 can be almost completely handled by a variation of the current mover script (/usr/local/sbin/mover). Ideally rather than diverge the codebase here the mover script would be altered to allow the definition of file to be included/excluded and a different destination directory. The only other difference is the mover has some logic in it to check share settings etc which may also be applicable here.

 

#2 is trickier and I am tempted to suggest that at least in the first instance we have one simple INCLUSION definition of filesize < x KB. this is not very flexible but it is elegant.

 

#3 Again is variation on the current mover script where both the source and destination drivers can be set

 

At this point is anyone else intersted in this other than me? I imagine this being useful for people using on disk artwork, nfos, tbns, sfvs etc (so programs like Kodi etc) where it is beneficial to be able to open these small files without spinning up loads of mechanical drives.

  • Upvote 1
Link to comment

By means of an exmaple

 

find /mnt/user/tv.hd/ -type f -size -100000c -ls | wc -l
22186

 

root@SOLO:~# find /mnt/user/tv.hd/ -type f -size -100000c -ls | awk '{total += $7} END {print total}'
342382982

 

So for one share i can pull 22k files into the accelerator SSD array drive and only use up 350MB of it.

 

Or put another way

 

find /mnt/user/tv.hd/ -type f -ls | wc -l
37433

 

So for virtually no SSD hard disk cost I can make almost 60% of the files in that entire share instant super fast access.

 

  • Upvote 1
Link to comment

I think this would require modification of the shfs(usershare) code.

If there was a way to INCLUDE one of the disks, but make it read only that would allow manual placement of the smaller files.

Another option is to set a preferred fnmatch/regex/size match filter for writes to direct them to a specific disk.

Link to comment

Am I right in saying that setting a user share to exclude a disk only concerns the writing of the data to that disk. So if you write data to that disk by some other means it would still be seen on the user share.

 

If that is the case this becomes considerably easier as it is just an exercise in rsync and find.

 

For this reason I would suggest we make this a cache disk only feature since then we only care about initial population and the mover script form then on.

 

Does that make it simpler? I think it does

Link to comment

Am I right in saying that setting a user share to exclude a disk only concerns the writing of the data to that disk. SO if you write data to that disk by some other means it would still be seen ont he user share.

 

If that is the case this becomes considerably easier.

That is certainly the current behaviour.  You see data in a user share on any disk that contains a top level folder corresponding to the User share  regardless of the include/exclude disk settings for that share.

 

Whether LimeTech have any plans to change that I have no idea, but I have not seen anything suggesting it might change.

Link to comment

Am I right in saying that setting a user share to exclude a disk only concerns the writing of the data to that disk. SO if you write data to that disk by some other means it would still be seen ont he user share.

 

If that is the case this becomes considerably easier.

That is certainly the current behaviour.  You see data in a user share on any disk that contains a top level folder corresponding to the User share  regardless of the include/exclude disk settings for that share.

 

Whether LimeTech have any plans to change that I have no idea, but I have not seen anything suggesting it might change.

 

There have been requests to provide a configurable option to disable the automatic share outside of include/exclude.

 

In any case, the whole find/rsync would work, but it's kinda kludgy.

As files are found that match a pattern, the destination path has to be altered to be moved to the new location.

The problem with the rsync method is depth of directory.

 

Do-able in a .c program with ftw64().

Do-able in find, but it has to be passed to a script or program that will modify the supplied path.

 

Whereas, if this were done inside the usershare filesystem, the code inside would handle the output write delegation to the designated accelerator disk.

 

 

Frankly, I think this is becomes a feature for unRAID.

 

 

We have split level, but having the ability to direct files of a size,regex,fnmatch from the gui seems like it would be a desirable feature for a media server with tons of meta data files.

Link to comment

I agree this is definatelty the better approach however lets be realistic I first suggested this perhaps a year ago and it hasnt even made a 2015 milestone yet.

 

Unless we take this thing and run with it the chances of it making uNRAID in any reasonable time frame is small/none.

 

I am not sure I agree that the rsync way is kludgy though as thats exactly how the cache mover works surely?

 

Been playing with the numbers and I pretty considerably get 60%+ files are less than 100kB on my video shares. Quite frankly that amazes me.

Link to comment

It occured to me that rsync could do this pretty simply in theory "--max-size=", --remove-source-files and the various flags to preserve atributes.

 

So i dug out the attributes the mover uses and hit the manual:

 

rsync -i -dIWRpEAXogt --numeric-ids --inplace

 

which expands to

 

-d, --dirs                  transfer directories without recursing
-I, --ignore-times          don't skip files that match size and time
-W, --whole-file            copy files whole (w/o delta-xfer algorithm)
-R, --relative              use relative path names
-p, --perms                 preserve permissions
-E, --executability         preserve executability
-A, --acls                  preserve ACLs (implies -p)
-X, --xattrs                preserve extended attributes
-o, --owner                 preserve owner (super-user only)
-g, --group                 preserve group
-t, --times                 preserve modification times
--numeric-ids           don't map uid/gid values by user/group name

 

and a special mention for the last one which looks like its own little world of pain

 

--inplace
    This option changes how rsync transfers a file when the file's data needs to be updated: instead of the default method of creating a new copy of the file and moving it into place when it is complete, rsync instead writes the updated data directly to the destination file. 
This has several effects: (1) in-use binaries cannot be updated (either the
    OS will prevent this from happening, or binaries that attempt to swap-in their data will misbehave or crash), (2) the file's data will be in an inconsistent state during the transfer, (3) a file's data may be left in an inconsistent state after the transfer if the transfer is interrupted or if an update fails, (4) a file that does not have write permissions can not be updated, and (5) the efficiency of rsync's delta-transfer algorithm may be reduced if some data in the destination file is overwritten before it can be copied to a position later in the file (one exception to this is if you combine this option with --backup, since rsync is smart enough to use the backup file as the basis file for the transfer). 
WARNING: you should not use this option to update files that are being
    accessed by others, so be careful when choosing to use this for a copy. 
This option is useful for transfer of large files with block-based changes
    or appended data, and also on systems that are disk bound, not network bound. 
The option implies --partial (since an interrupted transfer does not delete
    the file), but conflicts with --partial-dir and --delay-updates. Prior to rsync 2.6.4 --inplace was also incompatible with --compare-dest and --link-dest. 

 

But I still think the theory is sound enough and pretty simple.

 

Update: the mover acutally uses a combination of find, fuser and rsync likely because of the --inplace switch. Does anyone know why we went this route?

  • Upvote 1
Link to comment

It's kludgy because the destination path's tree must fully exist or the rsync fails.

 

root@unRAID:/mnt/disk3/Music/music.mp3# ls -l filelist.txt
-rw-rw-rw- 1 root 522 21078930 2015-02-24 09:22 filelist.txt

Make the disk accelerator path.
root@unRAID:/mnt/disk3/Music/music.mp3# mkdir /mnt/diska

root@unRAID:/mnt/disk3/Music/music.mp3# rsync -avP  $PWD/filelist.txt /mnt/diska/Music/Music.mp3/
sending incremental file list
rsync: mkdir "/mnt/diska/Music/Music.mp3" failed: No such file or directory (2)
rsync error: error in file IO (code 11) at main.c(577) [Receiver=3.0.7]
rsync: connection unexpectedly closed (9 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(601) [sender=3.0.7]

root@unRAID:/mnt/disk3/Music/music.mp3# mkdir -p /mnt/diska/Music/Music.mp3/
root@unRAID:/mnt/disk3/Music/music.mp3# rsync -avP  $PWD/filelist.txt /mnt/diska/Music/Music.mp3/
sending incremental file list
filelist.txt
    21078930 100%  177.62MB/s    0:00:00 (xfer#1, to-check=0/1)

sent 21081584 bytes  received 31 bytes  42163230.00 bytes/sec
total size is 21078930  speedup is 1.00

 

So rsync would work, but the tree has to exist before hand.

 

I'm assuming the mover does not have that issue because the usershare creates the full destination path.

 

There's more then one way to skin a cat though. LOL.

 

I'm not really sure what's expected here, files within a certain regex/size range are staged differently via mover, or if you want to move/copy files on the array within those parameters to a consolidated accelerator drive.

 

Using find with cpio or tar a file list that matches the parameters can be made.

With cpio/tar the directory tree and permissions can be created/preserved (just like rsync).

 

The real issue is creating the whole directory tree to match the source before the files are rsynced.

 

I still think this is kludgy as the accelerator drive will need to be mounted before the array is started so that the usershare detects it.  This is why I think this type of filtering/designation should be done in the user share file system.

 

This would allow an SSD to be mounted as the accelerator drive.

Files writing via specific parameters can be written to the accelerator drive automatically.

 

Then we have the issue of protecting these files.

Will we want to keep the seed files on the protected array and only copy them to the accelerator cache drive?

 

If they do not exist on the protected array any more

Is the accelerator cache drive going to be part of the protected array?

 

Will it be read only or maybe another btrfs pool? (think I prefer the btrfs pool option).

Link to comment

I'm assuming the mover does not have that issue because the usershare creates the full destination path.

 

I don't think that is the case. The diskmv utility I made is able to create the full destination path with the find/rsync structure. I think the -R flag in rsync is what enables it. I'll try to verify that later today.

Link to comment

I'm assuming the mover does not have that issue because the usershare creates the full destination path.

 

I don't think that is the case. The diskmv utility I made is able to create the full destination path with the find/rsync structure. I think the -R flag in rsync is what enables it. I'll try to verify that later today.

 

Good to know. I looked closer at the mover script and replicated it. I stand corrected and we may have a winner folks.

 

Looks like it would work. Although you would need to cd to each /mnt/disk* folder and run the script / find for each file matching the pattern you want to move.

 

 

-name (some name) -size -4096c (sizes under 4096 bytes or whatever you choose).

 

root@unRAID:/mnt/disk3# mkdir /mnt/diska
root@unRAID:/mnt/disk3# find /mnt/diska -ls

root@unRAID:/mnt/disk3# find ./Music -depth -name folder.hash  \( \( -type f ! -exec fuser -s {} \; \) -o \( -type d -empty \) \) -print | tail -1
./Music/music.mp3/Meditative/Guna Sangah/Chakra Balancing/folder.hash

root@unRAID:/mnt/disk3# rsync -v -i -dIWRpEAXogt --numeric-ids --inplace './Music/music.mp3/Meditative/Guna Sangah/Chakra Balancing/folder.hash' /mnt/diska/                                                                          
building file list ... done
.d..t...... ./
cd+++++++++ Music/
cd+++++++++ Music/music.mp3/
cd+++++++++ Music/music.mp3/Meditative/
cd+++++++++ Music/music.mp3/Meditative/Guna Sangah/
cd+++++++++ Music/music.mp3/Meditative/Guna Sangah/Chakra Balancing/
>f+++++++++ Music/music.mp3/Meditative/Guna Sangah/Chakra Balancing/folder.hash

sent 1148 bytes  received 49 bytes  2394.00 bytes/sec
total size is 899  speedup is 0.75

root@unRAID:/mnt/disk3# find /mnt/diska -ls
693101    0 drwxrwxrwx   3 nobody   users           0 Nov 22 09:23 /mnt/diska
723527    0 drwxrwxrwx   3 rcotrone root            0 Dec 20 13:35 /mnt/diska/Music
723528    0 drwxrwsrwx   3 rcotrone 522             0 Feb 24 09:22 /mnt/diska/Music/music.mp3
723529    0 drwxrwsrwx   3 rcotrone 522             0 Jan 29 08:58 /mnt/diska/Music/music.mp3/Meditative
723530    0 drwxrwsrwx   3 rcotrone 522             0 Oct 25 10:41 /mnt/diska/Music/music.mp3/Meditative/Guna\ Sangah
723531    0 drwxrwsrwx   2 rcotrone 522             0 Oct 25 10:41 /mnt/diska/Music/music.mp3/Meditative/Guna\ Sangah/Chakra\ Balancing
723532    4 -rw-rw-rw-   1 rcotrone 522           899 Oct 25 10:42 /mnt/diska/Music/music.mp3/Meditative/Guna\ Sangah/Chakra\ Balancing/folder.hash

Link to comment

So it looks like as a proof of concept we have proven the big line items and this is not really that hard to do.

 

The only real kludge is there is no way to exclude an array drive from all user shares in one go and also proactively into the future. We can work around this for the time being but eventually this will mean the SSD accelerator drive will be contaminated with general user array data.

Link to comment

So it looks like as a proof of concept we have proven the big line items and this is not really that hard to do.

 

The only real kludge is there is no way to exclude an array drive from all user shares in one go and also proactively into the future. We can work around this for the time being but eventually this will mean the SSD accelerator drive will be contaminated with general user array data.

 

Then maybe that's a feature request that needs to be implemented.

 

Perhaps with a hidden file on the disk itself.

I.E.

 

If the root directory of a disk in /mnt/disk* has .excludeshfs all directories are added to the exclude list.

If the first top level subdirectory has a hidden file as .excludeshfs, that directory/disk is automatically added to the exclude list for the share.

 

In matching with other feature requests if the root directory of a disk in /mnt/disk* has .noexportshfs any top level directory is NOT included as a automatic user share for samba.

 

Is this understandable and worthy of a feature request?

Link to comment

I just noticed in beta14a, in the settings page there is a global share setting

 

enable/disable user shares

Include disks

Exclude disks

 

Perhaps there's a way to add global exclude for writes or read only include

Those settings have been around for a while. Here is a post from 2011 with a screenshot, for example

 

 

OK, thanks for the clarification, I had not noticed them before.

Link to comment

I must have looked at that setting countless times and never realsied what it is:

 

    This setting defines the set of array disk which are excluded from User Shares. Set this field to blank in order to not exclude any disks; otherwise, set this field as above to define the set of disks to exclude.

 

    Note: Each separate User Share also includes its own set of Included and Excluded disks which represent a subset of the Included/Excluded disks defined here.

 

It is late o'clock for me but we will need to check if an globally excluded disk is essentially RO as we need... I suspect it wont be part of user shares at all which if true doesnt help us

 

Link to comment

It is late o'clock for me but we will need to check if an globally excluded disk is essentially RO as we need... I suspect it wont be part of user shares at all which if true doesnt help us

 

I know.

I think if we can get the wording down to a clean and logical request, it'll be a reasonable request for consideration.

 

As Tom mentioned, it's about how many find this good for the product.

Link to comment

Just to confirm that if I add a disk to the global excluded list and then manually create files and folders on the disk via the disk share they DO show up in the user share.

 

So the wording for this feature is confusing but for us it means its currently an exact match for what we want namely:

 

have a SSD disk that is part of the array

not used it when writing data to the user share

available to use as a disk share

allow for files to be deleted via a user share.

 

So all we need now is a working rsync script and we have the feature.

 

 

Link to comment

I asked Freddie via PM if he would be interested in working on this script as I suspected it was a relatively small delta from his existing script (and he has expertise in this already). This was perhaps a bit cheeky of me but my worry was I missed something in the minutia of find/fuser/rsync that has an impact.

 

However saying that, at least in the interim, I am going to try and take this POC a bit farther and kludge up some script to see how we get on.

Link to comment

Sorry. Didn't know I had a PM.

 

I added an option in diskmv to only move small files from one disk to another.

 

https://github.com/trinapicot/unraid-diskmv

 

The limit is hard coded to 4k at the moment and after looking at my data, I realized that is really small. I'm thinking 1024k is a better default value.  Thoughts? You can change this line if you want to play with it:

maxsize=4k        # Default to 4096 byte max file size for small files

 

I'm planning to make the limit adjustable but I don't yet know how I'm going to process/validate the input value. I got annoyed with the `-size`  test in `find` because `-size -1024k` returns lots of expected files but `-size -1M` doesn't return any files. I think it is due to integer math. Perhaps as an easy next step I will require the limit be specified in kilobytes and must be greater than 1.

 

... in the minutia of find/fuser/rsync ...

 

 

fuser is involved to check if a file is in use. If it is in use, it will not be moved. So I assume bad things could happen if you try to move a file when in use.

Link to comment

Nice work that man.

 

Have been playing with it in its default "test" mode and it looks like it would do exactly as we would want. In the end it might be worth thinking about a similar option for consld8 but these are early days.

 

I am not sure what the best size would be. I am desperately trying to avoid making the logic too complicated by want to specify extension type but my feeling, backed up with some quick testing, is that 100kB is a sweet spot. However I am aware that is for my data and YMMV.

 

On my video shares 100kB catches the usual suspects of subs, crc type files, srr and nfo. These are accounting consistently for ~60% of file count and even multi TB shares when you match @<100kB the consolidated files are running to a only few 100MB of files. Quite amazing really and for those of us using cache_dirs this should really push the performance.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.