unRAID Server Release 5.0-beta10 Available

[Interstellar]

unRAID Server Release 5-beta has now passed its 1 year birthday.

 

I was of the understanding that one of the main developments of Unraid was to make it somewhat modular, and release 5 was a major step in that direction. However there seem to have been serious scopesliding, so now V5 is also dealing with other trivial protocol and driver issues, which could have been postponed to v5.x releases.

 

Not releasing a truely modular v5 is putting other developers on standby, and are actually holding back the true potential of Unraid.

 

I hope the future will show tighter project management, including release and scope control.

I love Unraid as a bitbucket, and I know of no real alternatives. So core product and technology is good.. but the project is not managed well.

 

Anyway,, Happy Birthday 5-beta !!

 

 

Getting v5 to work properly with AFP is hardly trivial given it was one of the targets of v5 in the first place?

 

I'm sure once AFP is 100% behaving with SL and Lion plus all the other bugs (some of them quite serious from what I gather) are sorted then your modular development will start

[disco]

Timemachine seems to work flawlessly in Lion with this release. Awesome!

[savestheday]

Saves the day,

 

Yes, that 'No such file or directory' message is normal, however...

 

As I stated in my previous post... you don't necessarily want to delete all of your .AppleDouble folders; there might have been some important resource fork information in there. To be clear, I don't recommend deleting the .AppleDouble folders.

 

Deleting .AppleDB folders is fine, because they only contain the netatalk CNID databases.

 

I'll summarise the current status of AFP from my point of view, now that Tom has made several improvements.

 

There used to be an issue that the inode numbers of files and folders in user shares changed every time the array was started, this caused the AFP databases to be invalid, requiring corrections, which took a while every array restart. This has now basically been fixed in beta10 (with some minor caveats that Tom mentions in a previous post).

 

There used to be an issue that extended attributes were not possible in user shares, and Netatalk wanted to use them in it's default configuration. Tom has fixed this as well.

 

• Netatalk creates a database of file and folder locations (based on inode numbers), by default in a .AppleDB folder in the root of each share.
  
• From Netatalk 2.1 onwards, some of this location data is also cached in the .AppleDouble folders that have always been used for storing resource forks.
  
• When Netatalk daemons are stopped (because you stopped the array, or rebooted), the databases are closed down, and must be restarted when the daemons start again.
  

 

In my testing, I have found that the process of caching this database data in the .AppleDouble folders takes an annoyingly long time, primarily because it's a lot of small reads and writes to the parity protected array.

 

In my system I run an additional drive outside the array, which stores not critical data, and runs my the add on applications that I've selected (Plex and Squeezebox Server).

 

To speed up the performance of these databases, I have chosen to modify the

/etc/netatalk/AppleVolumes.default-

file, by adding the

nocnidcache

option (which disables caching the database in .AppleDouble folders), and adding the

dbpath:

variable to move the .AppleDB folders onto my drive outside the array. This has improved initial browsing performance dramatically.

 

The standard unRAID setup works fine, and there are no bugs. I wasn't happy with the browsing performance, so I made those changes. I'm putting it out there as an idea, rather than advising anyone to do it.

 

Thanks for your help dude. I'm having slow initial browsing.  When I click on my server it takes 30-60 seconds to come up and then for each share or folder underneath it takes a little while as well. 

 

Looking at AppleVolumes.default- I see...

 

# dbpath:path         -> store the database stuff in the following path.

 

and

 

# nocnidcache         -> Don't store and read CNID to/from AppleDouble file.

 

Can I just change that to:

 

dbpath:/mnt/cache/.db

 

and

 

nocnidcache

 

??

 

In other words, do just remove '#' and the description after wards?  Pretty sure this is what needs to be done but wanted to check with you before I do this.  Thanks man!

[savestheday]

Tom,

I really think part of my problem is I rsync'd over my stuff from other servers that had AFP enabled.  I tried running this to remove the old AppleDB files:

 

 

/etc/rc.d/rc.atalk stop
cd /mnt
find disk*/ -name .AppleD* -exec rm -r {} \;

 

But I'm not sure that's helped much.  This is happening on Snow Leopard and Lion.  If you want, we could take it offline. 

 

On the other hand, I no longer get the CNID errors.

[Nezil]

Savestheday,

 

In my system, as I've said, I run an additional drive. You could use the cache drive instead, though I'm not sure what happens when you stop the array to that drive; if it's unmounted at that point, when AFP starts up again, once the array is stopped, it won't be able to find the folder containing the CNID databases, and could have un-expected results. Having said that, the user shares won't be available either, so the afp daemons will likely not have anything to do, and may not notice that the CNID path location doesn't exist - In short, it would be a bit of a hack to put the CNID databases on the cache drive.

 

Adding the nocnidcache option shouldn't be a problem at all, as it doesn't affect the location of the CNID databases. This will probably give you the biggest performance benefit anyway, so why not try that on it's own first.

 

If you're willing to take the risk (which is not much of a risk really) then go right ahead; this is how it's done.

 

Whenever Tom releases another beta build, I customise it by decompressing the bzroot file, make my changes, and then re-compressing it. I then put this bzroot file on my flash drive. This might sound complicated, but it really isn't; details in this old post from BubbaQ that I found in the wiki.

 

If you don't want to go down the bzroot customisation path, you might be able to add something to the go script to do a similar job.

 

In the

/etc/netatalk/

folder, you'll see that there are two files called 'AppleVolumes.default'. One has a - on the end of the file name, and it's this one that you want to edit. (Each time AFP starts, unRAID copies this file to AppleVolumes.default, adds the shares that you've configured, and then starts the AFP daemons).

 

Look in that file for a line at the bottom that by default says something like:

 

:DEFAULT: cnidscheme:dbd options:upriv,usedots

 

And change it to something like this:

 

:DEFAULT: cnidscheme:dbd options:upriv,usedots,nocnidcache dbpath:/apps/dbd/$v

 

My additional drive is mounted at /apps by the way. As you said, on the cache drive, you'll want to hide the dbd folder by prepending a dot to the name. The $v argument means that subfolders with the names of the shares (AFP 'V'olumes) will be created in the /apps/dbd folder.

 

If you just want to test the theory, you could edit the AppleVolumes.default- file when booted, start and stop the array, and try it out.

 

Please keep in mind my comments about putting it on the cache drive not being ideal; I'm not sure how this will work out for you.

[Nezil]

To do this with a go script, you could copy the AppleVolumes.default- file to somewhere on your flash drive, make you modifications to that file.

 

In your go script, just add:

 

cp /boot/custom/AppleVolumes.default- /etc/netatalk/AppleVolumes.default-

 

That should do it!

[savestheday]

To do this with a go script, you could copy the AppleVolumes.default- file to somewhere on your flash drive, make you modifications to that file.

 

In your go script, just add:

 

cp /boot/custom/AppleVolumes.default- /etc/netatalk/AppleVolumes.default-

 

That should do it!

 

Much thanks dude!

[parsec]

Lion came out during the development of v5, so im impressed if Tom had that in scope when he started.

 

Modularity opens for others to start contributing. These resources are on hold, apparently until some protocol for mac people is fixed. That is poor resource management.

 

Anyway, unraid is just a bitbucket for me and it works fine. Like like everything else, it will be used until something better comes along.

I just got my hopes up when I combined a modular v5 with the awesome contributions other have been making. Unmeny and Bubbaraid, showed the true potential.

Ive been selling and managing software projects for a long time, so the joke is truly on me.

[limetech]

Lion came out during the development of v5, so im impressed if Tom had that in scope when he started.

 

Modularity opens for others to start contributing. These resources are on hold, apparently until some protocol for mac people is fixed. That is poor resource management.

 

Anyway, unraid is just a bitbucket for me and it works fine. Like like everything else, it will be used until something better comes along.

I just got my hopes up when I combined a modular v5 with the awesome contributions other have been making. Unmeny and Bubbaraid, showed the true potential.

The last piece of the "5.0" puzzle is a "Plugin Manager" which I've been working on with more urgency now.  I'm sorry the 5.0-beta development has dragged on, but I think it's almost there.

 

Ive been selling and managing software projects for a long time, so the joke is truly on me.

I'm also sorry you feel that way.  In hindsight I probably should have implemented a Plugin Manager first, but I my thinking was that much careful design should be given to this because once implemented, probably largely cast in stone.  In the meantime, AFP, bug fixes, driver issues, support issues, etc, etc, and next thing you know it's been a year 

 

 

[prostuff1]

my thinking was that much careful design should be given to this because once implemented, probably largely cast in stone.

I agree with this completely.  Getting it as close to right the first time is going to be a huge step in the correct direction.  Getting all the hooks in place and being able to do it halfway easily, will be huge.  The unMenu package manager is pretty close and a sight to behold seeing as it is written in awk...

 

In the meantime, AFP, bug fixes, driver issues, support issues, etc, etc, and next thing you know it's been a year 

to true, stuff happens... going through some of this on my current work project right now.

[savestheday]

Lion came out during the development of v5, so im impressed if Tom had that in scope when he started.

 

Modularity opens for others to start contributing. These resources are on hold, apparently until some protocol for mac people is fixed. That is poor resource management.

 

 

In my situation, using ALL Mac's at home, Tom wouldn't have had my pro license if AFP wasn't implemented.  5.0 becoming more stable and adding AFP is the single biggest reason why my FreeNAS is turned off at home now.

[madburg]

I hope we all get to preview the plug-in manager before it is set in stone, so we can offer feedback.

[speeding_ant]

I've said it again and I'll say it now, you've done an amazing job trying to make everyone here happy! You're caught in an unfortunate rift,  trying to add consumer features as well as dealing with people in enterprise who want a rock solid solution with AD integration etc.

 

You've pushed out features faster than commercial products. Only netgear has bested your development speed, but look how big their company is.. And they don't have to deal with up-teen different hardware configurations!

 

Even without reliable communication (which I see you're improving on that), you've managed to hold onto an extremely knowledgeable community who are both reliant and loyal to your product. Hell, most of us offer to pay for the 5 upgrade...

 

In other words, keep doing what you're doing. Nothing to see here, move along.

[wix2]

Lion came out during the development of v5, so im impressed if Tom had that in scope when he started.

 

Modularity opens for others to start contributing. These resources are on hold, apparently until some protocol for mac people is fixed. That is poor resource management.

 

 

In my situation, using ALL Mac's at home, Tom wouldn't have had my pro license if AFP wasn't implemented.  5.0 becoming more stable and adding AFP is the single biggest reason why my FreeNAS is turned off at home now.

 

Ditto. AFP isn't just a nice-to-have, for me it's critical especially since Apple seems to have screwed up SMB so badly in Lion.

 

I'm extremely happy that Tom has been willing to put so much work into making AFP and Lion work with unRaid. It's much more important to me than plugins.

[squirrellydw]

AFP is huge to me also and while I wanted to see AFP implemented before 3TB support you don't always get what you want.  I think Tom is doing a great job and in the correct order.  You first build a house then you decorate it.  The plug-in manager should be the last thing you add.  What one person thinks is TRIVIAL, someone else think is VITAL.

[Auggie]

At least FINALLY there is some real, tangible progress being made to a final 5.0 release!

 

It's been several months since I visited the LT forums since it appeared that 5.0 had been stuck perpetual beta for a year with a feature set that didn't seem to be decided even after all that time.

 

Thanks Tom for moving forward, and including some big enhancements that many have been waiting for! 

[AwesomeAustn]

Any progress on 4.7.1?

 

I was hoping to get a little run time with recent 5.0 betas since the driver 'fix' is in there.  But I'm working on 4.7.1-rc1 which has this fix.

 

I upgraded a disk in 4.7 and got 4 parity errors. I put the old disk back and ran initconfig and then upgraded again but with 5.0-beta9 and did not get any parity errors.

[parsec]

AFP, Plugin, and what else is proposed in the roadmap, it is all a matter of opinion.

Making a roadmap is not a demokratic process, and using "who squeals the most" is also bad. "Squeaeling" has a profile, and cant be assumed to represent the biggest buying potential.

Tom decides, he sets the expectations, he chooses the segment he wants to approach and appeal to, and based on that he makes the roadmap.

When features is decided for releases, they get build and released often in iterations as he does, and when he decides that things are "good enough" he makes a final release.

"good enough" is a subjective evaluation, and decided by Tom.

 

The problem versions that contain several technologies is that it might take too long to complete, and during development new things happens in the surrounding systems, increasing the challenges. As an example is the release of Apple Lion. This makes it very difficult to complete the release, and the release just grows.

At the same time he now has to support two versions. V4 which he has to support, at it is last release, and V5x which is getting to be a huge and very noisy beta release.

 

Above is very basic, and Tom, as he writes, just got caught in challenges and time passed.

Projects are soo easy to evaluate in hindsight, and I dont think Tom made any serious mistakes.

 

Finishing releases is a huge job, and often you want to include just one more thing.. and then suddenly you have made hotfixes, which have to be included, which affects other things, and then that takes even more time.

This is usually a reason for huge conflicts between project managers and development, as too small iterations are too much work for too little progress, but too big iterations delays delivery and introduces challenges not known at scoping, and functional requirements might even change during the development time.

 

There are no shortcuts, and if any has the key for software development without any problems, then they will have a very bright future.

The rest of us, or at least I, fucks up on a regular basis.

[jamesbaker]

Any way to include support for USB sound devices?

Could make this into a multimedia monster without having to recompile every time

 

[prostuff1]

Any way to include support for USB sound devices?

Could make this into a multimedia monster without having to recompile every time

 

Um, what are you using a USB sound device for?

[lionelhutz]

The development is sounding really good. The feedback here is indicating this is the best 5.0b release to date.

 

However, I do wonder if the Apple support would have already been created as a plug-in if the plug-in manager was available 6+ months ago...

 

Peter

 

[Falcon]

I have replaced my parity disk from 2TB to 3TB and I have used the Preclear_disk script version 1.12beta to clear the disk with -A as 64k alignment.

 

When I am replacing the disk and start rebuilding i see these messages:

 

root@Fileserver:~# fdisk -lu /dev/sdk

 

WARNING: GPT (GUID Partition Table) detected on '/dev/sdk'! The util fdisk doesn't support GPT. Use GNU Parted.

 

 

Disk /dev/sdk: 3000.6 GB, 3000592982016 bytes

255 heads, 63 sectors/track, 364801 cylinders, total 5860533168 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 4096 bytes

I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disk identifier: 0x00000000

 

  Device Boot      Start        End      Blocks  Id  System

/dev/sdk1              1  4294967295  2147483647+  ee  GPT

Partition 1 does not start on physical sector boundary.

 

and from the mymain meny i see this line:

 

What should I do, is this normal ? or should I stop rebuilding the parity disk and do something with it ?

 

 

[Superorb]

I have replaced my parity disk from 2TB to 3TB and I have used the Preclear_disk script version 1.12beta to clear the disk with -A as 64k alignment.

 

 

I can't help with your issue, but it should read 4k alignment (or sector 64)

[Falcon]

The parity Settings page displays this information. Maybe it is ok ?

 

[Joe L.]

I have replaced my parity disk from 2TB to 3TB and I have used the Preclear_disk script version 1.12beta to clear the disk with -A as 64k alignment.

 

 

I can't help with your issue, but it should read 4k alignment (or sector 64)

3TB disks use their own alignment and an entirely different partitioning style.  They use a GPT partition.

The GPT partiton puts into place a "protective" partition table in the MBR to make legacy utilities think the disk is entirely allocated.  Many older utilities will just report the partition starting on sector 1.   The GPT partition is 4k aligned, so do not worry... and the -A had absolutely no effect on the preclear because the drive was over 2.2TB in size, but it did no harm either.  The "-A" option was ignored because of the size of the drive.

 

It is normal for fdisk to see the protective MBR partition and show it as starting on sector 1.  It just has no way to know of the actual GPT partition.

 

Joe L.