Jump to content
jonp

unRAID Project Roadmap Announcements

414 posts in this topic Last Reply

Recommended Posts

All I want is a scalable, cost effective, low cost of ownership, reliable file server.

 

Right?

 

I think it should be agreed that unRAID must achieve the above baseline without Docker, Xen, or KVM. As long as it does this without any add-ons, then it's a real NAS. Pile on the virtualization all you want after that. It's called gravy!

 

I should say my definition of "reliable" means that I can RELY on it to safeguard my crap in a reasonably intelligent manner, not just sit there without crashing. At this very moment I'm moving data off of a failing disk. How do I know it's failing? Because I check the syslog every day. Why? Because I've learned that's what I have to do to keep my stuff safe on my unRAID box. When last night's parity check completed with zero errors and all green balls, most users would think that's a damn good indication of overall system awesomeness. But I know the deal. I know how this works. So I go to the syslog, like I do everyday, and there are bunch of read errors on disk 4. But the green balls!! :o I go back to Main... oh. Yes. There it is, the number forty-seven, nestled snugly next to other numbers like fifty-four million three hundred and fourteen thousand seven hundred and thirty-eight, and one billion eight hundred and ninety million blah blah blah. It's in the Errors column. That's how y'know it's bad. HOW can that be the only indication of a problem? I don't need it to do the RMA for me; I just need it to tell me something is wrong AND take some proactive steps to protect my stuff. I know this isn't a support thread but can someone tell me why a drive with 47 errors can get a green ball? The drive is obviously not dead because I'm reading it right now, but isn't there a yellow ball? Can we get some yellow balls in here? Just a few tiny yellow balls!?! ;D

 

If I understand this debate at all, it's that the V-word folks are thinking the way to build the more reliable / intelligent server we all want is through the V-word. Do I have that wrong? I'm hoping that's not the only way. If that ends up being the only way to do it, then I guess I'll do it. But get ready because just as between your grandparents and every electronic device they own, there are gonna be frickin problems and you are all now my grandkids.

 

(I'm just attaching my syslog in case anyone thinks I'm lame enough to make sh*t up.)

 

 

You read your syslog everyday because finding the 'Errors' column is too hard next to all those big numbers?

Share this post


Link to post
You read your syslog everyday because finding the 'Errors' column is too hard next to all those big numbers?
No, I keep an eye on my syslog and smart status because I've had failing drives not increment the error column. As long as a drive responds, even if it takes multiple retries, unraid doesn't give any overt indication that something is wrong.

Share this post


Link to post

Your statement is not accurate.  I have this exact same issue, but instead of Windows, I am watching a video via my popcorn hour player, my nzbdrone application runs in a VM, independent of popcorn hour and unraid.  When a new file is to be copied, my PCH locks the video, then suddenly comes back.  Most annoying thing ever, especially if the freeze occurs 3-4 times during the same video.

 

All I can say is from my experience with both my unRaid servers and multiple windows clients is that the only time I get a stutter on a video / music / etc is when the same client playing the video opens another file on the server(s) and a drive has to spin up.  If another client tries to open a file and a drive spins up, then the client playing the stream does not suffer any stuttering...  I regularily copy 100gig+ of data over to the server from one computer requiring spin ups while another is playing a stream and there is no stutters at all. 

 

When I had CP / SB running as background tasks on one of my HTPC's, I would always get a stutter when it started copying a file over which required spin up.  After moving them over to a VM on the same computer the stuttering issue disappeared.

 

I had the exact same result when I was using 2x Windows boxes as servers also.  Hence why I am blaming Windows

Share this post


Link to post

Obviously other people,  including myself,  have had different experiences than you.

 

Sent from my SM-N900V using Tapatalk

 

 

Share this post


Link to post
I had the exact same result when I was using 2x Windows boxes as servers also.  Hence why I am blaming Windows

 

I get the pause issue when streaming files to my Linux based HTPC boxes, so it's definitely not limited to Windows. On the flip side, I did not get this issue when using a Windows Server 2008 R2 based server solution (streaming to the same exact HTPC boxes), so if anything, Windows is the cure ;D

Share this post


Link to post

You read your syslog everyday because finding the 'Errors' column is too hard next to all those big numbers?
No, I keep an eye on my syslog and smart status because I've had failing drives not increment the error column. As long as a drive responds, even if it takes multiple retries, unraid doesn't give any overt indication that something is wrong.

 

That is valid but is not what the person who I was replying to said. It would be nice if there could be some sort of warning, perhaps an email on some keywords in the syslog if it's detected. I don't know... how could unraid handle failing drives better?

Share this post


Link to post

I don't have any pauses streaming from my server to a windows client, but when copying a new folder to a10 disk user share, i get a pause whilst it creates the folder of anything up to a minute, another pause before it starts writing the file, sometimes the pause is so long there is a timeout and I have to start over again.  There are often pauses in the middle of a long file copy for the same period before it resumes (usually).  I have just learned to live with it.

 

Incidentally, it makes no difference if I map a share to the drive and do the same copy, still get delays and sometimes server cannot be found error.  Ran Resierfsck on all disks, all fine.  I suspect it's something to do with writing to the flash, perhaps everything stops for that.

Share this post


Link to post

I don't have any pauses streaming from my server to a windows client, but when copying a new folder to a10 disk user share, i get a pause whilst it creates the folder of anything up to a minute, another pause before it starts writing the file, sometimes the pause is so long there is a timeout and I have to start over again.  There are often pauses in the middle of a long file copy for the same period before it resumes (usually).  I have just learned to live with it.

 

Incidentally, it makes no difference if I map a share to the drive and do the same copy, still get delays and sometimes server cannot be found error.  Ran Resierfsck on all disks, all fine.  I suspect it's something to do with writing to the flash, perhaps everything stops for that.

 

This is not a solution, but as a workaround it's worth setting up a cache disk and modify your shares to use it. Since the cache disk is completely outside of the array it shouldn't have these issues when you copy to the share (since it's redirected to the cache disk). It moves the data to the array at 3am or 4am or something by default, which means this should become somewhat invisible to you.

 

It's obviously preferable for the issue to not occur at all, but this could at least help mitigate the impact.

 

I used to have similar issues, but since adding the cache drive I don't (however I've also replaced CPU/MB/RAM twice since I last saw the issue so don't know the cache drive was 100% of the solution).

 

Share this post


Link to post

All I want is a scalable, cost effective, low cost of ownership, reliable file server.

 

Right?

 

I think it should be agreed that unRAID must achieve the above baseline without Docker, Xen, or KVM. As long as it does this without any add-ons, then it's a real NAS. Pile on the virtualization all you want after that. It's called gravy!

 

I should say my definition of "reliable" means that I can RELY on it to safeguard my crap in a reasonably intelligent manner, not just sit there without crashing. At this very moment I'm moving data off of a failing disk. How do I know it's failing? Because I check the syslog every day. Why? Because I've learned that's what I have to do to keep my stuff safe on my unRAID box. When last night's parity check completed with zero errors and all green balls, most users would think that's a damn good indication of overall system awesomeness. But I know the deal. I know how this works. So I go to the syslog, like I do everyday, and there are bunch of read errors on disk 4. But the green balls!! :o I go back to Main... oh. Yes. There it is, the number forty-seven, nestled snugly next to other numbers like fifty-four million three hundred and fourteen thousand seven hundred and thirty-eight, and one billion eight hundred and ninety million blah blah blah. It's in the Errors column. That's how y'know it's bad. HOW can that be the only indication of a problem? I don't need it to do the RMA for me; I just need it to tell me something is wrong AND take some proactive steps to protect my stuff. I know this isn't a support thread but can someone tell me why a drive with 47 errors can get a green ball? The drive is obviously not dead because I'm reading it right now, but isn't there a yellow ball? Can we get some yellow balls in here? Just a few tiny yellow balls!?! ;D

 

If I understand this debate at all, it's that the V-word folks are thinking the way to build the more reliable / intelligent server we all want is through the V-word. Do I have that wrong? I'm hoping that's not the only way. If that ends up being the only way to do it, then I guess I'll do it. But get ready because just as between your grandparents and every electronic device they own, there are gonna be frickin problems and you are all now my grandkids.

 

(I'm just attaching my syslog in case anyone thinks I'm lame enough to make sh*t up.)

 

 

You read your syslog everyday because finding the 'Errors' column is too hard next to all those big numbers?

 

My point was that it's an easy mistake to make. It's like a floor covered in banana peels: If I'm super careful, I'll be fine. But wouldn't it be better if the bananas peels weren't there?

 

Why does the system say the parity check completed with no errors when there were 47 read errors on one disk, all during the parity check? Why does the drive have a green ball next to it? Doesn't that deserve a yellow ball? Am I being unreasonable?

Share this post


Link to post

I don't have any pauses streaming from my server to a windows client, but when copying a new folder to a10 disk user share, i get a pause whilst it creates the folder of anything up to a minute, another pause before it starts writing the file, sometimes the pause is so long there is a timeout and I have to start over again.  There are often pauses in the middle of a long file copy for the same period before it resumes (usually).  I have just learned to live with it.

 

Incidentally, it makes no difference if I map a share to the drive and do the same copy, still get delays and sometimes server cannot be found error.  Ran Resierfsck on all disks, all fine.  I suspect it's something to do with writing to the flash, perhaps everything stops for that.

 

This is not a solution, but as a workaround it's worth setting up a cache disk and modify your shares to use it. Since the cache disk is completely outside of the array it shouldn't have these issues when you copy to the share (since it's redirected to the cache disk). It moves the data to the array at 3am or 4am or something by default, which means this should become somewhat invisible to you.

 

It's obviously preferable for the issue to not occur at all, but this could at least help mitigate the impact.

 

I used to have similar issues, but since adding the cache drive I don't (however I've also replaced CPU/MB/RAM twice since I last saw the issue so don't know the cache drive was 100% of the solution).

I used to use a cache drive, but found it too unreliable.  I ended up with files and folders stuck permanently on the cache drive.

Share this post


Link to post

I ended up with files and folders stuck permanently on the cache drive.

Do you have AFP enabled or ever enabled AFP, even once just to test or by accident?

Share this post


Link to post

I ended up with files and folders stuck permanently on the cache drive.

Do you have AFP enabled or ever enabled AFP, even once just to test or by accident?

 

If any folder on the cache drive has a folder with the same name on an array drive, the mover script can't move it off the cache onto the array.  if you move the folder from all array drives back to the cache drive, then the mover will move it to the array, like it's supposed to.

 

however, be careful, as unRAID has a bug which can permanently delete files if you try to move from the array to the cache, under some conditions which I'm not entirely sure I understand.

Share this post


Link to post

I ended up with files and folders stuck permanently on the cache drive.

Do you have AFP enabled or ever enabled AFP, even once just to test or by accident?

 

If any folder on the cache drive has a folder with the same name on an array drive, the mover script can't move it off the cache onto the array.  if you move the folder from all array drives back to the cache drive, then the mover will move it to the array, like it's supposed to.

 

 

however, be careful, as unRAID has a bug which can permanently delete files if you try to move from the array to the cache, under some conditions which I'm not entirely sure I understand.

 

I suspect the pauses I experience are probably causing the problem.  E.g., mover attempts to move a folder, creates the blank folder and then times out whilst trying to copy data files, next time it fails to move it because it already exists.

 

I probably did enable AFP at some time or other, I can't remember what I had for lunch let alone the last 3 years.

 

My current server only has 2 user shares, but they span over 60TB with thousands of files/folders.  I have a second server shipped this week from Greenleaf, then my servers will each only contain 1 user share.

Share this post


Link to post

however, be careful, as unRAID has a bug which can permanently delete files if you try to move from the array to the cache, under some conditions which I'm not entirely sure I understand.

I think you are thinking of the bug when moving to user shares from a disk.  I don't think going to the cache drive is a problem IF you move to /mnt/cache or \\<server>\cache.  However if you move to /mnt/user/<share> or \\<server>\<share> you can end up deleting your files.

Share this post


Link to post

For those who may have missed Tom's update in the other thread.

 

...<SNIP>...

 

In case anyone hasn't noticed, JonP has taken up much of the forum communication slack, and doing a great job.  Now I can sit here and spend the next couple hours I have left tonight as well as the next couple of full days running through each and every forum post "communicating" OR I can continue to get the coding done that you all are complaining about.

 

We are preparing a more detailed "status report" as well as adding more to the roadmap post, but let me throw out some items:

 

- WILL be adding multiple-device btrfs cache pool in unRaid 6; this will appear in the next beta currently under test internally

 

- WILL be integrating latest netatalk into unRaid 6 before 'final' (i.e., not dropping AFP support)

 

- WILL be integrating "native" UPS monitoring into unRaid 6 before 'final'

 

- WILL be integrating "native" email notifications into unRaid 6 before 'final'

 

- WILL be integrating "native" pre-clear utility into unRaid 6 before 'final'

 

- WILL be integrating docker containers to implement server applications into unRaid 6 before 'final' (this will largely replace the way application plugins are implemented today)

 

- WILL be generating TWO 'bzroot' images: one that contains the Xen/KVM user-space tools, one that does not.  This will happen before unRaid 6 'final'.  This will free up lots of memory for users who don't want/need to run VM's.

 

What about unRaid 5 you ask?  The plan is to NOT "backport" any work that has gone into unRaid 6.  Sorry, I know in the past I wanted to add some more features to unRaid 5 before marking it end-of-life.  But the situation has changed considerably.  Mainly it doesn't make sense to continue working on a 32-bit platform with a PAE kernel.  The sooner people can move to unRaid 6 the better.  Worried about more memory requirement?  That is why we are generating two bzroot images.  Worried about all your application plugins?  That is why we have spent many many man hours vetting docker for this purpose; and guess what? It's awesome!  Don't let all the chatter about VM for this, VM for that.  If you want to run a NAS with maybe a handful of specialized apps or even purely "stock" - moving to unRaid 6 will be easy.

Share this post


Link to post

Thanks for cross posting!  The other thread is too toxic to spend much time in :)  These details on unRAID 6 sound great.

Share this post


Link to post

Hi everyone!  JonP here with Lime Technology and some exciting new announcements ...

 

So, no plans to introduce block dev encryption support in unRAID 6?

If that's the case, it is quite disappointing. It also means I will be forced to steer away from unRAID on at least some of my systems.

I was under the impression that Tom was in fact "into" this feature, all the more so since adding it seems to be quite simple and straightforward technically (need to provide an interface for bringing up LUKS style crypto dm "under" the unRAID md).

 

Could you verify for us where this stands?

 

Thanks!!

Share this post


Link to post

Hi everyone!  JonP here with Lime Technology and some exciting new announcements ...

 

So, no plans to introduce block dev encryption support in unRAID 6?

If that's the case, it is quite disappointing. It also means I will be forced to steer away from unRAID on at least some of my systems.

I was under the impression that Tom was in fact "into" this feature, all the more so since adding it seems to be quite simple and straightforward technically (need to provide an interface for bringing up LUKS style crypto dm "under" the unRAID md).

 

Could you verify for us where this stands?

 

Thanks!!

Will check with Tom on this today.  To be totally honest, I haven't discussed that feature with him yet, but maybe its something we can add to either 6.0 or 6.1.  If you want to save me a step, can you provide a link to a forum post with Tom where this feature was discussed before?

Share this post


Link to post

:D

I wanted to checkup on the development of the roadmap,, but it seem to have drowned.

I love roadmaps as a great expectation synchronization tool, and together with budgets and spending reports. They tell where something is going, a view into the future.

So let me give my 2 cents.

 

A roadmap is about the "when" of things. Like Soon, Next, The 24. December, Far future. And if high focus features have been discussed heavily a Never, can be good to have.

So the discussions should relate to the timing of things.

 

The What, Why, Who and How belongs somewhere else.

 

/Rene

What is the one good thing about pedophiles? They slow down, when driving past schools.

Not sure what you mean by drowned.  We just posted this roadmap last month and it does provide a timeline for release as well.  In addition, Tom just added more details to it the other day.  Not sure what you are looking for...

Share this post


Link to post

I had the exact same result when I was using 2x Windows boxes as servers also.  Hence why I am blaming Windows

 

I get the pause issue when streaming files to my Linux based HTPC boxes, so it's definitely not limited to Windows. On the flip side, I did not get this issue when using a Windows Server 2008 R2 based server solution (streaming to the same exact HTPC boxes), so if anything, Windows is the cure ;D

 

I wonder if some devices, OSes or applications do any pre-caching.  Meaning, if I am playing a video and my player caches a handful of MB of data then when the spin-up happens, the video has no stutter because it plays from the cache as opposed to relying solely on streaming from disk.  So if I have a 1GB video file that is 40 minutes long, thats roughly 25MB per minute and .42 MB per second.  A 15 second pause would be 6MB, so if I cache even 10MB then I never see a stutter.

 

This is 100% just a random thought, but figured to put it out there for others to debunk maybe.  I mean it would make sense for the player to do that, especially if it is decrypting audio streams and such, but I have no idea, just hoping for a solid explanation.  To be honest if people just agree, I will likely accept this as the reason some do or don't have stutter.

Share this post


Link to post

Will check with Tom on this today.  To be totally honest, I haven't discussed that feature with him yet, but maybe its something we can add to either 6.0 or 6.1.  If you want to save me a step, can you provide a link to a forum post with Tom where this feature was discussed before?

 

Absolutely: here. It was mentioned in other threads as well, but this was the primary one.

 

BTW, some of the landscape has changed since then (mainly, Truecrypt has stepped down as an alternative), but LUKS is a solid, good option.

 

Thanks for looking into it!

Share this post


Link to post

can you fix some how the cache shares work? its very slow when you transfer to cached shares vs directly to cache drive share, same with doing any moving/copying on the system its self.

example you can test your self

 

file1 is in /mnt/cache/download  -  this is the cache drive

 

mv file1 /mnt/user/Tv\ Shows/show/ = very slow around 40-50MB/s i think  - Tv shows is a cached share so the file even though its going to user/tv shows it ends up on /mnt/cache/Tv Shows/Show/ which is correct but the slow speed is very bad it affects everything on the cache drive.

 

but when your in /mnt/cache/download and do this

mv file1 /mnt/cache/Tv\ Shows/show/ = instant

 

is there any way to make unraid understand where the file should be moving/copy to is the cache drive it self and instead make it a normal move/copy?

 

maybe there is a way to make a another cache mnt that includes everything thats on user so applications can see all the files and still be working off the cache device would make it not slow down for that but only the SMB cached shares i guess.

Share this post


Link to post

:D

I wanted to checkup on the development of the roadmap,, but it seem to have drowned.

I love roadmaps as a great expectation synchronization tool, and together with budgets and spending reports. They tell where something is going, a view into the future.

So let me give my 2 cents.

 

A roadmap is about the "when" of things. Like Soon, Next, The 24. December, Far future. And if high focus features have been discussed heavily a Never, can be good to have.

So the discussions should relate to the timing of things.

 

The What, Why, Who and How belongs somewhere else.

 

/Rene

What is the one good thing about unraid-fans? They slow down, when driving past a harddisk.

Not sure what you mean by drowned.  We just posted this roadmap last month and it does provide a timeline for release as well.  In addition, Tom just added more details to it the other day.  Not sure what you are looking for...

 

And "drowned" was what you chose to focus on :D

Never mind,, keep up the good work. Unraid continues to be an awesome product, that does a few things VERY well.

 

BTW I would totally support an annual maintenance fee. Something like 10-20USD to support the business.

Share this post


Link to post

Hi everyone!  JonP here with Lime Technology and some exciting new announcements ...

 

So, no plans to introduce block dev encryption support in unRAID 6?

If that's the case, it is quite disappointing. It also means I will be forced to steer away from unRAID on at least some of my systems.

I was under the impression that Tom was in fact "into" this feature, all the more so since adding it seems to be quite simple and straightforward technically (need to provide an interface for bringing up LUKS style crypto dm "under" the unRAID md).

 

Could you verify for us where this stands?

 

Thanks!!

Will check with Tom on this today.  To be totally honest, I haven't discussed that feature with him yet, but maybe its something we can add to either 6.0 or 6.1.  If you want to save me a step, can you provide a link to a forum post with Tom where this feature was discussed before?

 

Any news?

 

Thanks!

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.