Jump to content
jonp

Accelerator drives

146 posts in this topic Last Reply

Recommended Posts

Jonathan => If you're moving between DIFFERENT user shares, you're okay.    But if you're using the same user share for both the source and destination, you're very likely to end up with truncated (zero length) files ... and the data's simply gone.

Could you give an example move statement that would clobber data within a user share?

Share this post


Link to post

... I was under the impression that moving files is currently completely safe as long as the source and destination are either both user or both disk shares, the risk occurs when you move from a disk share to a user share or vice versa.

 

No, the only "safe" way other than using the Global exclude as Tom noted is to ensure BOTH the source and destination are NOT user shares (i.e. both should be disk shares).    This is the "user share copy bug", which can result in significant data loss.  It's arguable whether it's really a "bug" or just a consequence of the structure of the share unions ... but nevertheless it's something to be avoided.

So if I understand you correctly Gary, you just told me that I can lose files if I move them using only user shares and not involving disk shares at all? If so, that's definitely a new bug that I was not aware of. I've been moving files around in the user share structure for as long as I've had unraid, and never had an issue.

 

Could you explain to me under what circumstances I would expect to see data loss when moving data using only the user shares?

The only time you could theoretically see a problem here is if you tried to copy a file to itself and the copy process did not detect the file already existed before attempting to copy the file (onto itself).  In practise if one is copying at the user share level one is almost invariably copying to either a different name or a different path so the problem never occurs.  Also all file managers I know of trap the condition of the source and target being the same file (i.e.copying a file onto itself)  and do not let you do it anyway.

 

Note the above scenario would actually cause data loss if the copy was allowed to proceed even if NOT using user shares.  This will be why it is already being trapped by the file managers/copying programs before it could occur.  However if one was doing such a copy in ones own code and did not include the check on source and target being the same it would be possible to force this to happen.

Share this post


Link to post

root@unRAIDb:/mnt/disk4/Video# ls -l /mnt/disk4/Video/testfile.dat /mnt/user/Video/testfile.dat
-r--r--r-- 1 1000 users 8544336788 Jan 13  2013 /mnt/disk4/Video/testfile.dat
-r--r--r-- 1 1000 users 8544336788 Jan 13  2013 /mnt/user/Video/testfile.dat

root@unRAIDb:/mnt/disk4/Video# stat testfile.dat
  File: testfile.data
  Size: 8544336788      Blocks: 16688160   IO Block: 4096   regular file
Device: 904h/2308d      Inode: 332         Links: 1
Access: (0444/-r--r--r--)  Uid: ( 1000/ UNKNOWN)   Gid: (  100/   users)
Access: 2015-02-10 10:23:31.239971436 -0500
Modify: 2013-01-13 20:08:32.000000000 -0500
Change: 2015-02-10 10:23:31.239971436 -0500
Birth: -

root@unRAIDb:/mnt/disk4/Video# stat /mnt/user/Video/testfile.dat
  File: /mnt/user/Video/testfile.data
  Size: 8544336788      Blocks: 16688160   IO Block: 131072 regular file
Device: 1ch/28d Inode: 5338        Links: 1
Access: (0444/-r--r--r--)  Uid: ( 1000/ UNKNOWN)   Gid: (  100/   users)
Access: 2015-02-10 10:23:31.239971436 -0500
Modify: 2013-01-13 20:08:32.000000000 -0500
Change: 2015-02-10 10:23:31.239971436 -0500
Birth: -

root@unRAIDb:/mnt/user/Video# mv -vi /mnt/user/Video/testfile.dat /mnt/user/Video/testfile.dat 
mv: /mnt/user/Video/testfile.dat and /mnt/user/Video/testfile.data are the same file
root@unRAIDb:/mnt/user/Video# cp -vi /mnt/user/Video/testfile.dat /mnt/user/Video/testfile.dat 
cp: /mnt/user/Video/testfile.data and /mnt/user/Video/testfile.data are the same file

 

The user share copy 'issue' is when you try to traverse different file systems, but the union of usershare/fuse actually points to the same files in the underlying layer.  As in disk to user share (if disk is not globally excluded).

 

mv and cp cannot detect the files are the same files because the device and inodes are different.

 

root@unRAIDb:/mnt/disk4/Video# stat testfile.dat 

File: testfile.data  Size: 8544336788     

Blocks: 16688160  IO Block: 4096  regular file

Device: 904h/2308d      Inode: 332        Links: 1

 

root@unRAIDb:/mnt/disk4/Video# stat /mnt/user/Video/testfile.dat 

File: /mnt/user/Video/testfile.data  Size: 8544336788     

Blocks: 16688160  IO Block: 131072 regular file

Device: 1ch/28d Inode: 5338        Links: 1

 

root@unRAIDb:/mnt/disk4/Video# mv -vi testfile.dat /mnt/user/Video/testfile.dat 
mv: overwrite /mnt/user/Video/testfile.data y
testfile.data -> /mnt/user/Video/testfile.data
mv: cannot open testfile.data for reading: No such file or directory

root@unRAIDb:/mnt/disk4/Video# stat testfile.dat
stat: cannot stat testfile.data : No such file or directory
root@unRAIDb:/mnt/disk4/Video# ls -l testfile.dat
/bin/ls: cannot access testfile.dat: No such file or directory
root@unRAIDb:/mnt/disk4/Video# ls -l /mnt/user/Video/testfile.dat
/bin/ls: cannot access /mnt/user/Video/testfile.dat: No such file or directory

 

 

I thought maybe LT/Tom could do some fancy locking or detection if the file is opened by the same process.

Apparently I'm incorrect in that.

Tom suggested presenting the usershare as symlinks instead of actual files. This could be a fix, but may raise other issues.

For now, the global share exclude setting is a method to prevent accidental overwrite.

Share this post


Link to post

Also all file managers I know of trap the condition of the source and target being the same file (i.e.copying a file onto itself)  and do not let you do it anyway.

 

The problem is that the file manager doesn't recognize that the file is the same if the path is different.

 

e.g.  if you try to copy \\Tower\Movies\Myfile  to  \\Tower\Movies\Myfile  most utilities would either not allow this, or would prompt you to see if that's really what you wanted to do.    But if you're copying \\Tower\Disk1\Movies\Myfile  to  \\Tower\Movies\Myfile  the copy utility wouldn't recognize that this was the same file.    What happens in this case is the file becomes a zero-length file -- so the data is lost.

 

 

Share this post


Link to post

I would still like to hear about any benefits you see from the accelerator drive, doesn't need to be quantitative. Are you spinning up drives less frequently? Are you seeing less lag in media players?

 

I am prepared to comment on this now. It is unscientific but I have always found that certain usage pattern would starve cache_dirs and result in a wide set of disk spinups. On the face of it populating the Accelerator drive reduces the impact of this considerably. The Accelerator drive is spun up a lot but being a SSD i dont really care about this as the spinners are down more than ever (as in almost always).

 

Update. I can no longer say definitively that the "spin up" state of the accelerator drive is likely related it being used as an accelerator drive as a change in 14b causes SSD never to go into standby.

 

I can still say with some confidence that drives spin up less and this makes sense given that a large percentage of all my files are on a single fast SSD with no rotational delay.

 

This is far to unscientific for my likeing but i can say with absolute confidece that even if this feature isnt picked up I am not going bcak to the way it was before I will kludge it together.

Share this post


Link to post

Stats update. I extended the caching process to include idx,ifo,jpg,log,md5,nfo,nzb,par2,pl,sfv,smi,srr,srt,sub,torrent,txt,url as long as they we < 100,000.

 

Applying this to everything excluding my photo collection

 

39GB

94,757 files

 

but looking beyond the raw numbers thats the artwork for everything including all my music, all the crc files and the nfo type files. On the face of it I now have several disks that just contain video and music files and nothing else. Huge win.

 

 

Share this post


Link to post

if and when i get an ssd, this will be something i'll be interested in.

 

look at this mess, lol.

 

9PQcGF8.png

Share this post


Link to post

Lets not turn this thread into a discussion about drive responsiveness. Thanks!

Share this post


Link to post

Lets not turn this thread into a discussion about drive responsiveness. Thanks!

OK. Sorry.  :-X

Share this post


Link to post

I came across something a bit concerning in the unRAID v6 manual:

Do not assign an SSD as a data/parity device. While unRAID won’t stop you from doing this, SSDs are only supported for use as cache devices due TRIM/discard and how it impacts parity protection. Using SSDs as data/parity devices is unsupported and may result in data loss at this time.

 

Anyone know how TRIM/discard on an SSD data device impacts parity?

Share this post


Link to post

As SSD is just a disk drive -- while the performance can degrade without TRIM support, that won't cause data loss.    I very much doubt that it will cause any issues if you did this.

 

Just as a data point => about 6 months ago I set up a 3-drive UnRAID system using 3 240GB SSD's ... just to see how it performed (VERY fast).    I only used it for a few days (the SSDs were for another purpose -- I just experimented with them a bit before moving them to the systems they were purchased for) ... but it worked perfectly.    Copied a bunch of data to it;  did a few parity checks; etc.  No issues at all -- both reads and writes were at full network speed (~ 120MB/s) and parity checks were ~ 250MB/s (under 20 minutes).

 

I wouldn't hesitate to use an SSD for a data or parity drive if I had SSDs large enough to meet my storage requirements.    At current prices, however, that's fairly unlikely unless you have a pretty small total capacity requirement.

 

Share this post


Link to post

I agree with Gary.  It sounds like a cover-your-butt statement, before the writer fully understood SSD's.  We'd have to ask whoever wrote it, to know for sure.

Share this post


Link to post

I suspect all the "non support" really means is they don't support TRIM.

 

Note that virtually all modern SSDs have very good garbage collection that will effectively "self TRIM" as long as the drive is idle for a reasonable length of time.    Clearly TRIM support is better ... but it's not absolutely necessary with newer SSDs => I wouldn't hesitate to build an all-SSD UnRAID setup if the price as reasonable [which is only true for very moderately sized arrays at the moment].

 

Share this post


Link to post

Just an update to say that this is still working out great for me. Since this topic started we have even seen 0.5TB affordable "slow for SSD" drives released which would be perfect partners to this.

 

For me, official support for this cheap performance boost cant come soon enough.

Share this post


Link to post

I agree with Gary.  It sounds like a cover-your-butt statement, before the writer fully understood SSD's.  We'd have to ask whoever wrote it, to know for sure.

 

Nope, it really depends on how the trim support is implimented, and its really a situation by situation thing.

 

Imagine the following:

 

You have a data file written to sectors 1 through 100.  You delete the file. Normally on a regular drive the delete simply does a write to the directory sectors (say sector 2000) to mark sectors 1 through 100 as available to store new data. This write is done through the OS so the change to sector 2000 is covered as part of the parity update and there is no change at all to sectors 1 through 100, they still contains the file data, it just doesnt show up in the directory listings.

 

Now on an OS with built in trim support for ssd, the OS sends an additional different command indicating sectors 1 through 100 can be trimmed but does not send any write all zeros to those sectors. Its up to the drive at this point how to handle things. Some drives will erase those ssd flash sectors, some might do nothing until a later point. There are no writes of zeros to sectors 1 through 100 from an OS perspective, so parity still has your orignal data file in the calculated parity. If the drive erases those sectors, then your parity is now incorrect.

 

Share this post


Link to post

Hmmm by that logic though (I get it, it makes sense, but i think there are missing facts) SSD can't work in any RAID situation but I'm not finding any issues in search.

 

Well I did find this http://en.wikipedia.org/wiki/Trim_%28computing%29#RAID_issues

As of January 2012, support for the Trim command has not been implemented in most hardware based RAID technologies. Software RAID implementations often do include support for TRIM. For example, TRIM has been supported for Mac OS X RAID volumes since 2011, using the SoftRAID application, including TRIM and RAID support for all non-Apple SSD devices. (Mac OS X does not officially offer TRIM support for third party SSD devices.) Another case where it has been implemented is in post-January-2011 releases of the Linux kernel's dmraid, which implements BIOS-assisted "fake hardware RAID" support, and now passes through any Trim requests from the filesystem that sits on the RAID array.[46] Not to be confused with dmraid, Linux's general-purpose software RAID system, mdraid, has experimental support for batch-based (rather than live, upon file deletion), Trim on RAID 1 arrays when systems are configured to periodically run the mdtrim utility on filesystems (even those like ext3 without native Trim support).[47] For a short time in March 2010, users were led to believe that the Intel Rapid Storage Technology (RST) 9.6 drivers supported Trim in RAID volumes, but Intel later clarified that Trim was supported for the BIOS settings of AHCI mode and RAID mode, but not if the drive was part of a RAID volume.[48]

and then this http://en.wikipedia.org/wiki/Mdadm#Features

Since version 3.7 of the Linux kernel mainline, md supports TRIM operations for the underlying solid-state drives (SSDs), for linear, RAID 0, RAID 1, RAID 5 and RAID 10 layouts.[2]
with reference  [2] linking to http://kernelnewbies.org/Linux_3.7#head-2fd9b183a4623d96e69ed24f88e0eb83217fa8df

 

Share this post


Link to post

Just thinking this through -

 

* We all know the LBA system, where the logical sectors we know and access are actually mapped at the internal drive level to a given set of physical sectors, and that even on spinning hard drives, that can change (sectors are remapped), without affecting our use of the LBA.  On SSDs, that is taken to the extreme, where the LBA's can float anywhere within the 'physical' sector map.  And we know that our LBA's are just a subset of the available physical sectors, and the drive can do anything it wants to with sectors not currently mapped into LBA's.  So far, no problem for us.

 

* What I believe you are saying is that LBA's that are part of a deleted file can be treated in a special way, and the SSD can be informed that it is free to change the contents of these LBA's, while they are still mapped as current LBA's.

 

* One point, since the concept of file deletion is not really an OS thing, this should really be referred to as a file system thing.  So the problem would be confined to those file systems with TRIM support, and how they implement it, whether they can instruct the SSD that it is OK to affect the content of a given set of LBA's.

 

* If true, 2 things are affected - parity is broken, and data recovery is hampered (since deleted files might be lost).

 

* If I'm understanding the issue correctly, then there's no problem with an SSD for a parity drive, since it does not have a file system.  But SSD's could not be data drives, at least not parity protected ones.  Correction: an SSD could not be a data drive formatted with a file system with a badly behaving TRIM (from our context).  So a ReiserFS data drive should be safe.

 

Edit: add correction at bottom.

Share this post


Link to post

I agree with Gary.  It sounds like a cover-your-butt statement, before the writer fully understood SSD's.  We'd have to ask whoever wrote it, to know for sure.

 

Nope, it really depends on how the trim support is implimented, and its really a situation by situation thing.

 

Imagine the following:

 

You have a data file written to sectors 1 through 100.  You delete the file. Normally on a regular drive the delete simply does a write to the directory sectors (say sector 2000) to mark sectors 1 through 100 as available to store new data. This write is done through the OS so the change to sector 2000 is covered as part of the parity update and there is no change at all to sectors 1 through 100, they still contains the file data, it just doesnt show up in the directory listings.

 

Now on an OS with built in trim support for ssd, the OS sends an additional different command indicating sectors 1 through 100 can be trimmed but does not send any write all zeros to those sectors. Its up to the drive at this point how to handle things. Some drives will erase those ssd flash sectors, some might do nothing until a later point. There are no writes of zeros to sectors 1 through 100 from an OS perspective, so parity still has your orignal data file in the calculated parity. If the drive erases those sectors, then your parity is now incorrect.

 

I went through a similar thought process trying to answer my own question, but my thoughts included one key difference. When a sector of data is trimmed, the SSD firmware can do whatever to the associated memory cells, but a read of the trimmed device sector MUST return zeros. That is a big assumption on my part, but it is the only thing that makes sense to me.

 

From an unRAID perspective, if a trim command to an md device is successful in reaching the SSD, then parity must be updated as well. I assume there is no mechanism to update parity with trim, so I would hope that any trim command to an md device is blocked.

 

In my experience, an SSD works fine as a data drive. I recently had my SSD data drive disabled (I think due to a bad sata cable going to a cache drive on the same controller). I ended up copying data from the emulated SSD and a verification of md5 sums after the fact showed no problems.

 

Keep in mind, when adding an SSD data drive, the clear or pre-clear process will fill the entire device with data. So write performance could be affected depending on how the firmware handles a full device. An SSD parity drive will always be full.

 

But we are talking about Accelerator drives here and I think the idea is to accelerate reads, so write performance shouldn't be a big factor.

 

Just an update to say that this is still working out great for me. Since this topic started we have even seen 0.5TB affordable "slow for SSD" drives released which would be perfect partners to this.

 

For me, official support for this cheap performance boost cant come soon enough.

 

It is working great for me also. I'm guessing for some people it is not worth the effort or risk of moving data around, but it just feels good to consume media with no disks spinning.

 

I spent a bunch of time tracking what was spinning up my disks unexpectedly. It was almost always access to the artwork I have stored next to video files. I export my Kodi database to single files and whenever a new thumbnail or fanart file shows up on disk, each Kodi client will access that file when it is needed for the gui. So when browsing through tv shows or movies, disks would spinup when I didn't expect. So I moved all .jpg and .tbn files to the accelerator drive. I also move tv show seasons to the accelerator drive for binge watching.

Share this post


Link to post

I suspect your usage model reflects quite a few people its just that most are not prepared to get up to this level of unRAID shenanigans to reap the benefits.

 

The recent cat5 video talks about a future tool to move game data between SSD and HDD which is essentially just this idea so we have some hope even though this hasnt made the roadmap yet it will some day, perhaps even soon after v6.

Share this post


Link to post

Can we have this either assigned to a milestone please or dropped i.e. get it out of limbo :)

Share this post


Link to post

Please post a "tl;dr" summary since I don't want to read through 5 pages at moment.

 

But would this work: At present there are several ways to control which device a new object (file or directory) is created on:

- include/exclude

- split level

- allocation method (most-free, fill-up, highwater)

- use cache (yes, no, only)

 

What if we had something like another allocation method, e.g, called "user-defined".  If that were selected then 'shfs' can call a user-defined script/program that is used to determine which device to create the new object on.  We'd expose an API that script could use to make and return the decision.

 

This would handle case of new object creation and could be leveraged by the mover to rearrange data.

Share this post


Link to post

tl;dr version

 

Dedicate protected array disk(s) (SSD) solely to certain types of data/files as defined by user (e.g file attributes, size, name etc). shfs should favour this disk when writing data that matches the user criteria and disfavour when no match.

 

 

This seems an exact match to your proposal above LT. What I am not clear about is where in the decision tree of methods this should be placed and the other complications about how this interacts with the other methods and if we should support more than one accelerator drive (i.e. multiple different match types)

 

But on the face of it the solution seems elegant.

Share this post


Link to post

Please post a "tl;dr" summary since I don't want to read through 5 pages at moment.

 

But would this work: At present there are several ways to control which device a new object (file or directory) is created on:

- include/exclude

- split level

- allocation method (most-free, fill-up, highwater)

- use cache (yes, no, only)

 

What if we had something like another allocation method, e.g, called "user-defined".  If that were selected then 'shfs' can call a user-defined script/program that is used to determine which device to create the new object on.  We'd expose an API that script could use to make and return the decision.

 

This would handle case of new object creation and could be leveraged by the mover to rearrange data.

 

Brilliant.  Love this idea!  Call a script/program, etc, etc that has the source file and the program returns a destination disk.

 

If you do this via an external application call, it will slow down object creation, but that yields better control to the user.

Please us exec() rather then system() or there will be all kinds of quoting issues.

 

Better yet would be to provide an example loadable shared library, however I believe that would complicate things for end users and require C expertise.

Share this post


Link to post

I just managed to shoe horn ~1M files onto 100GB SSD. Whilst this is a small % of array usage in terms of GB usage it is a huge % of the file count.

 

Considering how cheap this SSD was that a lot of bang for the buck. Happy

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.