unRAID Server Release 5.0-beta9 Available


Recommended Posts

  • Replies 157
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Posted Images

Can I safely remove this dir or would it remove everything in /mnt/user as well?

 

You would probably be very unhappy if you try to delete this  ;D

 

Here's a rule of thumb: if it's not broke, don't try to fix it.

I almost fell off the chair when I read Tom's answer.

 

I agree.  The user would NOT be happy with the result.

Link to comment

I'm having problems with unMenu starting up. It doesn't, lol.

 

I'm at work now, so can't provide the syslog, but there were errors alone the lines of: "Divide by zero error", "Segmentation Fault", and "Port already in use."

 

I've tried with both no other packages, and a few essentials that I use. webGUI works ok, but seems to be a problem with emhttp/unmenu.

Link to comment

Can I safely remove this dir or would it remove everything in /mnt/user as well?

 

You would probably be very unhappy if you try to delete this  ;D

 

Here's a rule of thumb: if it's not broke, don't try to fix it.

I almost fell off the chair when I read Tom's answer.

 

I agree.  The user would NOT be happy with the result.

 

Good humor is always welcome, especially from Tom, makes him human ;D

Link to comment

Thanks for taking the time to respond to my posts Tom, though I do still have some questions:

 

Where did all that data come from?

 

If you look inside the .AppleDB folders, on both Netatalk 2.0.5 and 2.1.5, there are four files called lock, log.0000000001, cnid2.db and db_err.log. Netatalk 2.1.5 however also has several files labelled __db.00x, and it's these that make up the majority of the additional data. Perhaps 2.0.5 is storing these in RAM?

 

When a share is removed, added or changed by making changes to the AppleVolumes.default file, as happens when starting or stopping the array; all of this __db.00x files are deleted, and must be re-created again.

 

- As before, if a

killall -HUP afpd

command is called, performance is not affected really

- If the array is started and stopped, or the system is rebooted, the databases need to be completely rebuilt, slowing everything down again.

 

Nezil - nice analysis, but unfortunately a lot of it is wrong/misleading.

 

The .AppleDB folder contains the Berkeley Database files used to store CNID-to-file mappings required by AFP.  The files starting with "__" such as __db.001, are temporary files used during operation; in my testing they take very little time to create upon startup of AFP daemons.  The log.0000000001 file stores DB transactions (ie, a journal of sorts) that lets Berkeley DB recover from a system crash.  In my testing, I've found that "restoring" the DB from using this file is what's taking all the time, and I haven't determined why this is the case yet.

 

I think the point that I was making was that the time to "restore" the __db.00x files from the log.00000000x files is basically identical to having to re-create them.

 

More concerning of course is that they even have to be re-created at all. Your point is that they only need to be recovered during a system crash, but I have found that starting and stopping the array without a system crash requires them to be rebuilt / recovered as well.

 

The other interesting thing to note is that these files are not built upon startup of AFP daemons. The .AppleDB folder structure IS indeed built at that time, and takes less than a second to build, but it is not populated until you browse through all the shares, and this is what takes the time... every time your start / stop the array.

 

Conclusion / Suggestion

 

1. Storring the .AppleDB files on a disk outside the array, or possibly the RAM disk, will improve performance when initially creating the databases. There may also be some compatibility issues with having these on the Fuser disk system that's used for user shares. This can be changed with the dbpath: variable in AppleVolumes.default-

You definitely don't want these in a RAM disk, maybe the Cache disk, but probably the extra complication this entails is not worth it, but something I'm looking into.

 

I agree that you wouldn't want this in the RAM disk, but only because they don't persist with a reboot. The reason I suggested this is that as described above, the CNID database doesn't seem to survive a start / stop of the array currently, and so has to be re-built anyway. With this in mind, a RAM disk location would at least speed up the creation of the databases that seems necessary every start / stop.

 

If it's possible to prevent the CNID database needing to be re-created every start / stop / reboot (see point 3), then having these files on the array may not be a problem. They might take a long time to create, but that would only be necessary once, so wouldn't be an issue.

 

It's also interesting to note that having these files on the array means that the disk containing the CNID database needs to be spun up every time you browse the share, which would not be necessary with SMB.

 

2. Using the .AppleDouble files to store the CNID database cache (introduced in 2.1.5) causes performance and possibly again compatibility issues on user shares. This can be disabled by adding the nocnidcache option to the AppleVolumes.default- file

 

The .AppleDouble files have always been part of netatalk, and are required to implement resource forks.  If you use 'nocnidcache' option, and you lose the main cnid2.db, then all your resource forks are gone.  This is not recommended, and adds very little to decrease performance.

 

I think there is some confusion here; my understanding is that in Netatalk 2.1.x CNID files are stored inside .AppleDB AND the .AppleDouble folder in each directory. In previous versions of Netatalk, the CNID files were only stored in .AppleDB.

 

The 'nocnidcache' option does not stop .AppleDouble folders from being created when they're needed for resource forks, it just stops the CNID cache being stored there as well.

 

I believe that this does have a big performance impact, as when browsing into a directory that has several hundred sub-directories, several hundred new files need to be created on the array, which requires writing to the disks.

 

3. The current method of starting and stopping the array involves continual restarting of the AFP daemons, and moving and making changes to the AppleVolumes.default file. This causes all the CNID databases to be deleted / re-created, which may not be necessary. I would like to try and investigate what happens if you change the order of starting and stopping AFP daemons. For example, AFP daemons are stopped when the array stops, before the AppleVolumes.default- changes are carried out, and then only re-started after the array has started again, and any changes to the AppleVolumes.default- files have taken place.

 

My suggestion for 3. would mean that AFP services would not be running when the array is stopped, but I don't see this as being a problem, as nothing can be connected to in this case anyway.

 

Again, making changes to AppleVolumes.default does not cause "CNID databases to be deleted /re-created", nor should stop/start of the various AFP daemons.

 

A big change between netatalk 2.0 and 2.1 was requirement of using Berkeley DB 4.6 or later (previously DB 4.4 was requirement).

This required me to create a slack package for DB 4.6, which I did by modifying slack build scripts for 4.4.  It's possible I have some configure options incorrect for this version.

 

I think this suggestion was actually the most important one, as I was trying to solve the need to re-create / restore the databases every start / stop.

 

In my testing, when the array is stopped, in the logs, the AppleVolumes.default- file is copied to the AppleVolumes.default location, and on inspection you can see that all the shares are missing from the file. This makes sense so that AFP does not list the shares when the array is stopped.

 

When this process happens, all of the .__db00x files are deleted as well, which would explain why they need to be re-created / restored.

 

If I call

/etc/rc.d/rc.atalk

stop before stopping the array, the .__db00x files remain; but when I start the array again, the .__db00x files are deleted during the process. I would expect AFP to not work after starting the array, but it does; you must therefore be starting AFP daemons during the array startup process.

 

My guess therefore on the process of re-starting the array is:

 

Stopping array

- Stop array clicked in Web UI

- AFP daemon stopped

- AppleVolumes.default file modified to remove shares

- AFP daemon re-started

- AFP daemon sees that user shares are no longer required / present and deletes the CNID database files

- User Shares unmounted

 

Starting array

- Start array clicked in Web UI

- User Shares mounted

- AFP daemon stopped

- AppleVolumes.default file modified to add shares

- AFP daemon re-started

 

I may be wrong about the exact order, and was unable to test. I was simply trying to give you a pointer for something to look at as to the cause of the database loss on start / stop / reboot.

Link to comment

OK... an update, and Tom; I now see exactly what you mean about the issue.

 

My suggestions for the 'nocnid cache' and moving the dbpath for .AppleDB folders off the array were simply to improve the speed in creating and rebuilding the CNID database, something that should not be necessary once the main problem is fixed.

 

I tried the following things:

 

If you stop and then restart the AFP daemons while the array is running, all of the __db.00x files are removed, but they're rebuilt from the log.00000001 file at a pretty high speed (about 11 seconds for a full

find .

command on my setup).

 

The real problem is that the CNID backup in the log.00000001 file stops working when the array is stopped and then re-started.

 

I thought perhaps this was because the AFP daemons see that the shares no longer exist in the AppleVolumes.default file once the array is stopped, and perhaps mark the log file as no longer important - I was wrong! I prevented the AppleVolumes.default file from being changed, stoped and restarted the array and it didn't make any difference. The databases could not be recovered and took the long time (~4 mins) to be rebuilt.

 

Next I thought that maybe the AFP daemons noticed that the folder /mnt/users no longer existed, and somehow marked the log file as no longer important - I was wrong again!I stopped the AFP daemons manually, and moved the rc.atalk file to prevent the AFP daemons from starting when the /mnt/users folder was gone. I then stopped and restarted the array, moved the rc.atalk file back and manually restarted and it didn't make any difference. The databases could not be recovered and took the long time (~4 mins) to be rebuilt.

 

Now, like you Tom, I am stumped.

 

The only possible thing I can think of, is that the CNID databases know about what has changed by the modification time of the folders. When the array is stopped, these folders don't exist, and are created as part of starting the array. This makes the creation date different every time. I was going to try and test this, but I'm not sure how I can prevent emhttp from doing this on shut-down and start-up; I think it's over to you now!

Link to comment

Here's my syslog in regards to the unMenu start-up issues. A few lines which stand out to me (toward the bottom):

 

Jul 18 22:24:00 Tower unmenu[2192]: awk: ./unmenu.awk:1722: fatal: division by zero attempted
Jul 18 22:24:00 Tower unmenu-status: Exiting unmenu web-server, exit status code = 2
Jul 18 22:24:00 Tower unmenu[2192]: unmenu.awk unable to open port.  It  may already be running
Jul 18 22:24:10 Tower unmenu-status: Starting unmenu web-server

 

Had difficulty attaching it (even though it's only 80kb), so I used pastebin: http://pastebin.com/g8ymD8GQ

Link to comment

Here's my syslog in regards to the unMenu start-up issues. A few lines which stand out to me (toward the bottom):

 

Jul 18 22:24:00 Tower unmenu[2192]: awk: ./unmenu.awk:1722: fatal: division by zero attempted
Jul 18 22:24:00 Tower unmenu-status: Exiting unmenu web-server, exit status code = 2
Jul 18 22:24:00 Tower unmenu[2192]: unmenu.awk unable to open port.  It  may already be running
Jul 18 22:24:10 Tower unmenu-status: Starting unmenu web-server

 

Had difficulty attaching it (even though it's only 80kb), so I used pastebin: http://pastebin.com/g8ymD8GQ

This has nothing directly to do with the upgrade, but is as a result of the new release.

 

A basic patch is already been uploaded to google.code, so

cd /boot/unmenu

unmenu_install -u

Link to comment

Here's my syslog in regards to the unMenu start-up issues. A few lines which stand out to me (toward the bottom):

 

Jul 18 22:24:00 Tower unmenu[2192]: awk: ./unmenu.awk:1722: fatal: division by zero attempted
Jul 18 22:24:00 Tower unmenu-status: Exiting unmenu web-server, exit status code = 2
Jul 18 22:24:00 Tower unmenu[2192]: unmenu.awk unable to open port.  It  may already be running
Jul 18 22:24:10 Tower unmenu-status: Starting unmenu web-server

 

Had difficulty attaching it (even though it's only 80kb), so I used pastebin: http://pastebin.com/g8ymD8GQ

This has nothing directly to do with the upgrade, but is as a result of the new release.

 

A basic patch is already been uploaded to google.code, so

cd /boot/unmenu

unmenu_install -u

 

BEAUTIFUL!

 

Thank you sir. How can I make sure I always have the latest unmenu? Do you just run that command in your go script?

Link to comment

Hi there,

i have updated my server to 5.0beta9 yesterday and since then the write speed to the array dropped from 40MB/s to 5MB/s.

 

The thing is in the same time the server moved from my office to his new location, now it do not really know if it is a faulty LAN connection or the new Unraid Version (read speed are around 85-90MB/s so i dont think its the network).

 

Nevermind, to test this i would like to go back to beta6a, is that so easy like upgrading from it, or is it dangerous or not even possible?

Link to comment

Okay first thing i found out going back to 5.0beta6a solves the problem, after that i have 40-45MB/s write speed under 5.0beta9 only 3MB/s.

Reading speed in both Version is normal (85-95MB/s).

I am using Cool'n'Quite under both releases.

 

Syslog is attached.

 

Specs of my Server:

 

OS at time of building: 5.0-beta6a

CPU: AMD Athlon II X2 240e

Motherboard: Asus M4A78LT-M

RAM: 2GB KINGSTON CL9 ValueRAM PC3-10666

Case: Sharkoon Rebel 9 Economy

Power Supply: Super-Flower SF450P14XE Golden Green Pro (450W, 37A/12V Single Rail design, 80+ Gold)

 

Parity Drive: 2TB Hitachi 5K3000

Data Drives: 2x 2TB Hitachi 5K3000

Cache Drive: none

syslog.txt

Link to comment

BUG or FEATURE ?

 

Can't create usernames with capitals, it will only create users in lower case ?

 

 

Neither: it's a "limitation"  ;)

 

Most linux distros (including slackware) insist on only lowercase for usernames.  This is historic and really makes no sense, but sometimes other utilities get broken if not all lowercase, cf:

http://fixunix.com/samba/377744-%5Bsamba%5D-username-case-mangling-linux-username-mixed-case-samba-returns-lower-case.html

Link to comment

OK... an update, and Tom; I now see exactly what you mean about the issue.

 

BerkeleyDB has the concept of "checkpointing" where it commits transactions to disk and flushes them from the log.  I believe netatalk does this every 1800 seconds (3 min).  After running your 'find' do you wait more than 3 minutes before stopping/restarting array?  (Though one would think the afp daemon managing the DB would be "smart" enough to do an "explicit" checkpoint upon receiving a TERM signal, but I haven't look at the code for this yet.)

 

Some light reading for you:

http://sewm.pku.edu.cn/src/paradise/thirdparty/installed/docs/gsg_txn/CXX/BerkeleyDB-Core-Cxx-Txn.pdf

 

Also you can get lots of info logged by changing this line /etc/rc.d/rc.atalk

 

    /usr/sbin/cnid_metad

 

to

 

    /usr/sbin/cnid_metad -l LOG_MAXDEBUG

 

 

Link to comment

Will you ever had "man" pages to your releases, or do you exclude them to prevent filling up space in RAM/memory stick?

 

I know it's simple enough to look up most commands, but I have a habit of "man <command>" at times...

 

There are talk about this a while back and I think the conclusion and comment given was that they are specifically left out to save space.

 

As you said, most of it can be found easy enough online by type "man <command>" into a google search.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.