Can/will this system convert to a new file system in 'real time', or will we need to 'copy' the data to a new drive/location and the conversion will happen as the files are moved?  Basically, do I need another large drive to facilitate this conversion?

 

You would convert a drive at a time.

 

1. Format Drive B with file system of choice.

 

2. Copy all of Drive A to Drive B.

 

Repeat 1 & 2 till all drives are done.

 

I'd think of it more like ripping blurays. You don't have to do it but if you want to... do it at your own pace / speed.

 

ResierFS isn't a self-healing File System so you would probably want to convert to one that is and one that has metadata checksums too.

 

On another related note, I've seen a few mentions of H.265 being able to reduce file sized from 35-55%, and that's if one doesn't crop files, and tweak encoding settings in addition.  I have most everything stored as mkv files without re-encoding anything, just removed unnecessary streams, subtitles, versions and extras.

 

Can the conversion to H.265 be done in conjunction with the file type conversion?

 

Let's talk software first...

 

ffmpeg already has H.265 support.

 

XBMC Gotham (Version 13) - Has H.265 support and should be in Beta any day now.

 

Let's talk about storage...

 

Repeated quality comparison tests have demonstrated that H.265 reduces file size by roughly 39-44% at the same quality compared to H.264.

 

This figure can change dramatically depending on how quality control is measured. In subjective viewing tests, subjects reported  quality was equal to or greater than H.264?s with a bitrate reduction of 51-74%.

 

All your downloads are going to be to in the 51 - 74% range and for those of you ripping blu-rays around 39-44% (or higher depending how you "tweak" it).

 

So yeah... Over the next year or so we are all going to get A LOT of free space if we convert our Media files.

 

Let's talk about what the hold up would be...

 

GPU / Video Cards doing the H.265 decoding otherwise your CPU will be doing to heavy lifting and not your GPU / Video Card like it today.

 

Should be a driver thing. I haven't checked in the last several months but AMD and nVidia might already have them out (assuming your GPU / Video Cards is capable).

 

It's not a realtime "RAID" like unRAID or ZFS but since most of you are storing TV Shows, Movies and Music... so what. Basically

What do you mean "back up your raid every..."?  I don't currently have enough space to backup the whole dataset, so do you mean 'back up the raid configuration', or something else?

 

Technically you aren't running a true RAID. You are running X number of drives individually (with whatever file system on each drive).

 

It basically does a "parity check" at X time (that you set) and any new files / changes on any of the drives are written to your parity drive(s).

 

It's really not that much different from what most of you do now. You have all your new stuff on a cache drive that copies into the RAID at X time. With SnapRAID it copies everything in the cache drives (each hard drive) and updates the parity drive(s).

 

With 2 Parity Drives... You could lose 2 drives at once and still not lose data.

 

Note: You would also need to run mhddfs or AUFS so you share all the various /movie folders on each drive as one. unRAID uses a program to do the same thing.

I did give up on trying to get CentOS 6.5 to work - I just couldn't get the kernel to run.

 

However I have had more luck with Arch Linux. I have been able to build the kernel, and get system to run emhttp. Resulting in being able to run the array.

 

Am pretty sure that a load of stuff won't work properly as I didn't know hat to do with all of the init scripts from /etc/rc.d. Obviously they need translating to work with system somehow?

 

Can anyone who has gotten unraid working on Arch (or any other system based distro) please advise me how to get the init scripts moved over?

 

Great news, however I'm pretty sure this is all futile as Tom has said a brand new x64 version of emhttp is coming 'any day now'.

I appreciate that it all might be futile, but I do feel I have learned a lot by attempting this.

 

I've actually re-done the Arch build, this time installing Arch 64 bit (silly mistake by me not to do 64 bit the first time). That has actually been a bit more of a learning experience - having to incorporate multilib to get emhttp running, and getting hold of a fuse package for multilib.

 

Now I've gotten the 64 bit build to the same state as the previous one - it runs, loads emhttp automatically, starts the array and the shares are available under /mnt/user

 

However I would really like to know how yourself and grumpy have handled the init scripts issue. Even if it is pointless in the long run, it would help me round off this little project nicely.

 

Pretty please? 

The init scripts aren't really required here because arch is doing all the heavy lifting. You need to install samba / NFS and enable those services in Arch. The difference comes as emhttp normally manages these things on unraid server, but as it stands these things are separated when taken out of context like this.

 

Make sense? Me neither.

 

Sent from my Nexus 5 using Tapatalk

 

 

Right - I was thinking you'd some something clever so that emhttp still managed samba etc.

 

But you're saying that you leave those to be managed outside of emhttp.

 

That makes sense, I just thought there was more to it. Thanks.

 

It's a shame that emhttp is hardcoded to use the rc.d scripts. Would have been nice if there was a bit of abstraction - e.g. emhttp calls emhttp-samba which in turn calls /etc/rc.d/rc.samba Then we'd be able to modify emhttp-samba to call what we wanted it to "systemctl smbd restart" for example.

 

Maybe if Tom sees this (or if you are in contact with him you could suggest it) he will keep something like that in mind for a future emhttp version

Tom has told me he is separating things up.

 

I think we will see progress on exactly this going forward if toms comments to me are anything to go by.

 

Sent from my Nexus 5 using Tapatalk

 

Separating the two, would open up a lot of potential in the web server side of things ...

 

I suppose no time indications are known !

 

Separating the two, would open up a lot of potential in the web server side of things ...

 

I suppose no time indications are known !

 

'Any day now' is what I've been told to a 64 bit version. Outa my hands though obvs.

 

 

Sent from my iPad using Tapatalk

Tom has told me he is separating things up.

 

I think we will see progress on exactly this going forward if toms comments to me are anything to go by.

 

I am happy to hear this. On this level it appears to match the "unRAID-NG" architecture discussed in the other threads before.

If Tom were to release the API, this would open up a new list of opportunities for integration and extension.

 

I am happy to hear this. On this level it appears to match the "unRAID-NG" architecture discussed in the other threads before.

 

 

To the Linux Gods / Pros...

 

WebGUI API / config / ini files is what we want. If Tom makes an unRAID "package" that we can install in our Distro of choice. I just need the ability through and conf / ini file to tell the WebGUI what the samba start / stop command is for whatever Distro I use.

 

Each Distro is going to need an unRAID kernel too. Users can really screw up their system / data easily with making API calls to RAID / Block devices if the various versions of an unRAID MD module and a normal Linux MD module aren't correct.

 

I do not need APIs to the RAID / Block Devices and I don't see why anyone else would. If we have various people making various tools making different API calls / doing things to the RAID / Block Devices (in either the unRAID Kernel or Regular Linux Kernel) and making a system do X, Y and Z... You will have people starting unRAID with the wrong drive for parity and making the parity a data drive (or worse) and lose data.

 

This also doesn't take into account that from time to time the APIs and unRAID kernel modules have to change. It did from Linux Kernel 3.9.11 to 3.10 and probably will again in the near future. So if I have Ubuntu installed and download an updated Linux Kernel from the package manager (which it would unless Tom has his own Distro and repo)...  I just hosed up my system / unRAID from working. If I have some apps that start up on boot that make API unRAID / RAID / Block Device calls that someone wrote expecting the unRAID md driver to be there and not the Regular Linux one... I could / would probably fry my data too.

 

Sometimes less is more and I bet what I just wrote above made sense to 3% of the people here and the other 97% doesn't understand it but saw they could / would lose data (perhaps everything) and crapped in their pants.

 

To those of you who do not wear pocket protectors like Ford / Ironic / Weebo / myself...

 

Unless you are a Linux God (Pros would even be crazy to do this)... I would NEVER install unRAID without a Distro and Repo (for the Linux Kernel / Kernel Headers / Base - Core Linux utilities / etc.) that Lime Technology produces / backs / supports / makes / manages / has "control" over.

 

All those reasons above are why I cannot and will not post a guide on how to install unRAID in a Distro. If I won't do it without any "skin" in the game... I cannot imagine Tom releasing an unRAID "package "that users simply install in their Distro of choice. It would be impossible to mange / support. There are so many things that will go wrong and many will lose data.

 

 

I do not need APIs to the RAID / Block Devices and I don't see why anyone else would.

 

I agree and it wasn't my intention to suggest that..

What I suggested in the other threads was that we need an API in userland, not kernel level to open the path to i.e. new front-ends.

 

But...and this is still vital to me as this was the reason I retired my two unRAID licenses in the past ...I *do* hope that the there will be the ability to

enable unRAID on (point md-module towards using) other block devices that that of real, physical local disks, as the ones a real Distro has to offer and that i.e.

are supplied by dm_mod (for dm_crypt) or the ones from LIO (iSCSI).

 

grumpy, good points about the config/ini files. That's exactly the type of thing I was referring to. It would make things much easier to configure for different distros like you say.

 

I have noticed that even though, on current unraid, there is the rc.samba file - emhttp doesn't always use it for stopping/starting samba. I've noticed in the logs that sometimes emhttp just kills smbd and starts it again directly.

 

Abstracting those kind of things out, so that there is one path for starting/stopping the various services, would be a good thing to do whether an unraid 'package' ever gets released or not.

 

Now that I've had a bash, and some success, at getting unraid running on top of Arch, I can understand your reluctance to produce a guide. You would get a storm of questions (the few I had were bad enough). Which would be made even worse by the scenario you describe - something getting updated that breaks the unraid integration.

 

Also in its current form, at least my version anyway, unraid on a distro isn't as complete/polished as it would need to be. If a user sees, in the unraid webui under emhttp, the ability to manage samba they are going to expect it to work 100%. Whilst is does kind of work it's not quite what would be expected from a commercial product.

 

I wonder if it would be possible for the unraid driver to be developed into a kernel module that could possibly be distributed alongside the unraid package and negate the need to have it compiled into the kernel? I have no knowledge or experience of such things, just speculating.

 

On the whole I'm pleased that I've managed to get my Arch build working. Although I'm not sure that I would want to run it on my actual server. I don't think I'd trust my data to my mad Linux skillz Think I'd rather wait for a 'proper' solution.

 

I'll probably carry on tinkering and learning though!

Peter, you hit the nail on the head. Couldn't have said it better myself.

 

Sent from my Nexus 5 using Tapatalk

 

 

grumpy

 

If it didn't come across in my previous post - I totally agree with you! In no way am I trying to argue or doubt your points or co-erce you into writing a guide or releasing anything - I agree wholeheartedly.

 

I've had a go at installing unraid on a distro and can now see for myself more clearly what problems might arise.

 

With that experience, and the points you've highlighted, it seems, to me, that an unraid package for installation on any distro is not the best way for things to proceed.

 

To my mind (and getting back onto something relating the actual topic of this thread ) the best solution would be for there to be a single unraid distro. The same as there is now, but not based on slackware of course

 

Essentially we're talking about unraid, as it is now, and swap slackware for CentOS/Arch/OpenSUSE/whatever. If the basic release from Limetech worked as the current one does (as an appliance) and there was the ability to install unraid on a full installation of whatever the base distro ends up being (as you can now with a full slackware install), then I think that would be most of the bases covered.

 

More modern and better supported base distro - Yep

Maintain appliance like ability - Yep

Better package management - Yep

Ability for hardcore/experienced users to do a full install - Yep

No increase in support headache - Yep

Happy forum users who have gotten something shiny and new to play with - Yep

Ooo shiny....

To my mind (and getting back onto something relating the actual topic of this thread ) the best solution would be for there to be a single unraid distro. The same as there is now, but not based on slackware of course

 

From what I can tell parsing together what Tom has said...

 

I think the intent is to enable KVM or Xen (Host and Guest) or perhaps both in unRAID as we know it. If there is something you want to do on your server that you can't using a plugin... Simply create a VM and have at it.

 

If I was him, that would be the way I would go about this. A lot easier / less work / less to manage / regular unRAID users are happy / Nerds like us have a great big sandbox to play in.

I voted yes, I like unraid but I like the idea of a full distro (i'm agnostic to which as i'd have to learn the vagaries of any distro anyways as i'm a relative linux noob) bringing with that the flexibility of a package manager and leveraging more of the power of my system.

I'm really intrigued by your bad drive getting a notification coming up.

 

 

Grumpy I would LOVE to see a guide on how to set this up.... 

 

I cannot do that. Tom hasn't given his approval for using emhttp or decided if he wanted to create a subforum with it's own section where it can be properly supported.

 

According to his last message he has more to say on this... Hopefully we will hear what that is soon.

 

I would even be interested in seeing this separate from unRAID, if the CentOS unRAID takes a while to materialize.

 

Hopefully if enough users say yes and provide positive feedback, Lime Technology will commit a lot of resources to development so we can see this happen more quickly than it took for 5.0 to be released.

 

This has possibilities to the broader XBMC community...

 

According to what Tom said in the other thread, 75% or more of his customers in the last 5 years or so learned about unRAID in the various XBMC, Plex, openELEC, etc. forums / blogs / sites.

 

However, based on previous threads and the PASSION that people had that opposes this endeavor... I'm not sure which customer base Lime Technology is pursuing or if he is going after both. I would think the "innovators" outweigh the "appliance only / no separate unRAID versions whatsoever" crowd.... but I could be wrong.

 

Guess we all will find out soon enough.

 

I found out about unraid from the XBMC hardware forum thread about building your first unraid server.

I was just getting started on gathering a fairly substantial media collection and suffered a hard drive failure and wanted something expandable and more robust, hence unraid.

From what I can tell parsing together what Tom has said...

 

I think the intent is to enable KVM or Xen (Host and Guest) or perhaps both in unRAID as we know it. If there is something you want to do on your server that you can't using a plugin... Simply create a VM and have at it.

 

Geez - I HOPE that isn't the plan. I hope Tom's got the itch to build a pkg that works for a standard distro (I don't care which one - pick one, but remember that the inexpensive ones will have a wider audience - that's good for business). Virtualized servers are great if you have the hardware, but the costs are unwarranted for the typical home user audience. (ECC memory, particular motherboards, etc.) If we got a package that works in Centos, Ubuntu, Mint, heck, even Steam OS - I much prefer that so that we can install whatever outboard packages in a standard OS and get them OUT of unRAID where they tend to wreak havoc. unRAID on its' own is stable, simple and it just works...Leverage that by making it available to a wider audience and figure out how to build a cohesive framework that supports whatever the end use wants to achieve, whether they have a huge audio or video collection and want to stream it over their network, or do automated backups, or whatever.

 

I am using about 1% of my current machine - I'd like to get lots more for my electricity usage and something that doesn't require exotic equipment to achieve that goal is always welcome. Getting away from the Limetech USB "key" license model would be even more welcome - they are too prone to damage for my tastes.

to be honest, i cant really think of that many reasons why Lime technology would NOT want to produce an unRAID CentOS 64bit based edition, other than potential support headaches caused by multiple linux distro's (which to be honest is far outweighed by the benefits IMO), i can't really see any con's! 

unRAID is NAS-like appliance-type learn-less no-brainer maintenance-free no-hassle don't-think-just-push-button product.

 

Any, and I repeat - any deviation from this model would be a different product. Including the deviation discussed here. Different product, aimed at different consumer base.

 

It may not look so, but it is not an easy decision to make.

 

Added: Thats' why I did not vote on this poll. It's second day I'm thinking about it and I simply can not make up my mind. Luckily it's not my decision to make.

an interesting point of view pkn, and yes your absolutely right, unraid as-is is an appliance type product, however i would question it being a "no hassle just push button product", there is plenty of times when you have to get down and dirty with telnet, esp as things like plugins can cause issues in themselves, something that an unRAID OS would actually solve.

 

keep in mind also that even if unRAID OS was to become a reality you would still be able to run your " NAS-like appliance-type learn-less no-brainer maintenance-free no-hassle don't-think-just-push-button product" as it is today, it would be baked into the OS so would be ready to use once you had installed the OS, hell you could even carry on using unraid plugins and ignore the fact you now could install them natively on the OS and get rid of plugin hell.

 

so yep your right this would be a diffferent product.......a MUCH better product.

.......a MUCH better product.

That is entirely your opinion.  I prefer the very small footprint and USB install of the current unRAID appliance for my main machine.  There is a reason I went ESXi...

 

Different strokes for different folks

prostuff1, i 100% agree with you, that is def only my humble opinion, but looking at this poll i would say a fair few other people agree with the idea at the very least.

prostuff1, i 100% agree with you, that is def only my humble opinion, but looking at this poll i would say a fair few other people agree with the idea at the very least.

Polls can be used effectively but they also only reflect the views of people that are on this forum, regularly.

 

I know quite a few customers I have built servers for that do not come on the forum much/at all but would not vote "Yes" in this particular case. Trying to explain the "benefit" of this to most of my customers would go completely over there head.

.......a MUCH better product.

That is entirely your opinion.  I prefer the very small footprint and USB install of the current unRAID appliance for my main machine.  There is a reason I went ESXi...

 

Different strokes for different folks

Would it be possible to still install and run this from a USB as a Live USB install?

Would it be possible to still install and run this from a USB as a Live USB install?

I assume so, but with all the things that can go on with a "full linux distro" I would not want to run such a thing from a USB drive for any extended period of time.