limetech Posted July 18, 2011 Author Share Posted July 18, 2011 Can I safely remove this dir or would it remove everything in /mnt/user as well? You would probably be very unhappy if you try to delete this Here's a rule of thumb: if it's not broke, don't try to fix it. Quote Link to comment
Joe L. Posted July 18, 2011 Share Posted July 18, 2011 Can I safely remove this dir or would it remove everything in /mnt/user as well? You would probably be very unhappy if you try to delete this Here's a rule of thumb: if it's not broke, don't try to fix it. I almost fell off the chair when I read Tom's answer. I agree. The user would NOT be happy with the result. Quote Link to comment
savestheday Posted July 18, 2011 Share Posted July 18, 2011 Yeah I know, I'm a dummy! Forgot your previous 8D post. He schooled me. on a serious note, I'm still seeing this Corrupt CNDB error even after deleting those appledb files. Something's still not right. Quote Link to comment
defected07 Posted July 18, 2011 Share Posted July 18, 2011 I'm having problems with unMenu starting up. It doesn't, lol. I'm at work now, so can't provide the syslog, but there were errors alone the lines of: "Divide by zero error", "Segmentation Fault", and "Port already in use." I've tried with both no other packages, and a few essentials that I use. webGUI works ok, but seems to be a problem with emhttp/unmenu. Quote Link to comment
madburg Posted July 18, 2011 Share Posted July 18, 2011 Can I safely remove this dir or would it remove everything in /mnt/user as well? You would probably be very unhappy if you try to delete this Here's a rule of thumb: if it's not broke, don't try to fix it. I almost fell off the chair when I read Tom's answer. I agree. The user would NOT be happy with the result. Good humor is always welcome, especially from Tom, makes him human Quote Link to comment
Nezil Posted July 18, 2011 Share Posted July 18, 2011 Thanks for taking the time to respond to my posts Tom, though I do still have some questions: Where did all that data come from? If you look inside the .AppleDB folders, on both Netatalk 2.0.5 and 2.1.5, there are four files called lock, log.0000000001, cnid2.db and db_err.log. Netatalk 2.1.5 however also has several files labelled __db.00x, and it's these that make up the majority of the additional data. Perhaps 2.0.5 is storing these in RAM? When a share is removed, added or changed by making changes to the AppleVolumes.default file, as happens when starting or stopping the array; all of this __db.00x files are deleted, and must be re-created again. - As before, if a killall -HUP afpd command is called, performance is not affected really - If the array is started and stopped, or the system is rebooted, the databases need to be completely rebuilt, slowing everything down again. Nezil - nice analysis, but unfortunately a lot of it is wrong/misleading. The .AppleDB folder contains the Berkeley Database files used to store CNID-to-file mappings required by AFP. The files starting with "__" such as __db.001, are temporary files used during operation; in my testing they take very little time to create upon startup of AFP daemons. The log.0000000001 file stores DB transactions (ie, a journal of sorts) that lets Berkeley DB recover from a system crash. In my testing, I've found that "restoring" the DB from using this file is what's taking all the time, and I haven't determined why this is the case yet. I think the point that I was making was that the time to "restore" the __db.00x files from the log.00000000x files is basically identical to having to re-create them. More concerning of course is that they even have to be re-created at all. Your point is that they only need to be recovered during a system crash, but I have found that starting and stopping the array without a system crash requires them to be rebuilt / recovered as well. The other interesting thing to note is that these files are not built upon startup of AFP daemons. The .AppleDB folder structure IS indeed built at that time, and takes less than a second to build, but it is not populated until you browse through all the shares, and this is what takes the time... every time your start / stop the array. Conclusion / Suggestion 1. Storring the .AppleDB files on a disk outside the array, or possibly the RAM disk, will improve performance when initially creating the databases. There may also be some compatibility issues with having these on the Fuser disk system that's used for user shares. This can be changed with the dbpath: variable in AppleVolumes.default- You definitely don't want these in a RAM disk, maybe the Cache disk, but probably the extra complication this entails is not worth it, but something I'm looking into. I agree that you wouldn't want this in the RAM disk, but only because they don't persist with a reboot. The reason I suggested this is that as described above, the CNID database doesn't seem to survive a start / stop of the array currently, and so has to be re-built anyway. With this in mind, a RAM disk location would at least speed up the creation of the databases that seems necessary every start / stop. If it's possible to prevent the CNID database needing to be re-created every start / stop / reboot (see point 3), then having these files on the array may not be a problem. They might take a long time to create, but that would only be necessary once, so wouldn't be an issue. It's also interesting to note that having these files on the array means that the disk containing the CNID database needs to be spun up every time you browse the share, which would not be necessary with SMB. 2. Using the .AppleDouble files to store the CNID database cache (introduced in 2.1.5) causes performance and possibly again compatibility issues on user shares. This can be disabled by adding the nocnidcache option to the AppleVolumes.default- file The .AppleDouble files have always been part of netatalk, and are required to implement resource forks. If you use 'nocnidcache' option, and you lose the main cnid2.db, then all your resource forks are gone. This is not recommended, and adds very little to decrease performance. I think there is some confusion here; my understanding is that in Netatalk 2.1.x CNID files are stored inside .AppleDB AND the .AppleDouble folder in each directory. In previous versions of Netatalk, the CNID files were only stored in .AppleDB. The 'nocnidcache' option does not stop .AppleDouble folders from being created when they're needed for resource forks, it just stops the CNID cache being stored there as well. I believe that this does have a big performance impact, as when browsing into a directory that has several hundred sub-directories, several hundred new files need to be created on the array, which requires writing to the disks. 3. The current method of starting and stopping the array involves continual restarting of the AFP daemons, and moving and making changes to the AppleVolumes.default file. This causes all the CNID databases to be deleted / re-created, which may not be necessary. I would like to try and investigate what happens if you change the order of starting and stopping AFP daemons. For example, AFP daemons are stopped when the array stops, before the AppleVolumes.default- changes are carried out, and then only re-started after the array has started again, and any changes to the AppleVolumes.default- files have taken place. My suggestion for 3. would mean that AFP services would not be running when the array is stopped, but I don't see this as being a problem, as nothing can be connected to in this case anyway. Again, making changes to AppleVolumes.default does not cause "CNID databases to be deleted /re-created", nor should stop/start of the various AFP daemons. A big change between netatalk 2.0 and 2.1 was requirement of using Berkeley DB 4.6 or later (previously DB 4.4 was requirement). This required me to create a slack package for DB 4.6, which I did by modifying slack build scripts for 4.4. It's possible I have some configure options incorrect for this version. I think this suggestion was actually the most important one, as I was trying to solve the need to re-create / restore the databases every start / stop. In my testing, when the array is stopped, in the logs, the AppleVolumes.default- file is copied to the AppleVolumes.default location, and on inspection you can see that all the shares are missing from the file. This makes sense so that AFP does not list the shares when the array is stopped. When this process happens, all of the .__db00x files are deleted as well, which would explain why they need to be re-created / restored. If I call /etc/rc.d/rc.atalk stop before stopping the array, the .__db00x files remain; but when I start the array again, the .__db00x files are deleted during the process. I would expect AFP to not work after starting the array, but it does; you must therefore be starting AFP daemons during the array startup process. My guess therefore on the process of re-starting the array is: Stopping array - Stop array clicked in Web UI - AFP daemon stopped - AppleVolumes.default file modified to remove shares - AFP daemon re-started - AFP daemon sees that user shares are no longer required / present and deletes the CNID database files - User Shares unmounted Starting array - Start array clicked in Web UI - User Shares mounted - AFP daemon stopped - AppleVolumes.default file modified to add shares - AFP daemon re-started I may be wrong about the exact order, and was unable to test. I was simply trying to give you a pointer for something to look at as to the cause of the database loss on start / stop / reboot. Quote Link to comment
PeterB Posted July 18, 2011 Share Posted July 18, 2011 .... especially from Tom, makes him human Are you saying that he's not a real person? I'd always wondered .... Quote Link to comment
PeterB Posted July 18, 2011 Share Posted July 18, 2011 .... especially from Tom, makes him human Are you saying that he's not a real person? I'd always wondered .... Quote Link to comment
Nezil Posted July 19, 2011 Share Posted July 19, 2011 OK... an update, and Tom; I now see exactly what you mean about the issue. My suggestions for the 'nocnid cache' and moving the dbpath for .AppleDB folders off the array were simply to improve the speed in creating and rebuilding the CNID database, something that should not be necessary once the main problem is fixed. I tried the following things: If you stop and then restart the AFP daemons while the array is running, all of the __db.00x files are removed, but they're rebuilt from the log.00000001 file at a pretty high speed (about 11 seconds for a full find . command on my setup). The real problem is that the CNID backup in the log.00000001 file stops working when the array is stopped and then re-started. I thought perhaps this was because the AFP daemons see that the shares no longer exist in the AppleVolumes.default file once the array is stopped, and perhaps mark the log file as no longer important - I was wrong! I prevented the AppleVolumes.default file from being changed, stoped and restarted the array and it didn't make any difference. The databases could not be recovered and took the long time (~4 mins) to be rebuilt. Next I thought that maybe the AFP daemons noticed that the folder /mnt/users no longer existed, and somehow marked the log file as no longer important - I was wrong again!I stopped the AFP daemons manually, and moved the rc.atalk file to prevent the AFP daemons from starting when the /mnt/users folder was gone. I then stopped and restarted the array, moved the rc.atalk file back and manually restarted and it didn't make any difference. The databases could not be recovered and took the long time (~4 mins) to be rebuilt. Now, like you Tom, I am stumped. The only possible thing I can think of, is that the CNID databases know about what has changed by the modification time of the folders. When the array is stopped, these folders don't exist, and are created as part of starting the array. This makes the creation date different every time. I was going to try and test this, but I'm not sure how I can prevent emhttp from doing this on shut-down and start-up; I think it's over to you now! Quote Link to comment
limetech Posted July 19, 2011 Author Share Posted July 19, 2011 .... especially from Tom, makes him human Are you saying that he's not a real person? I'd always wondered .... The jig is up, we're twins... see attached. Quote Link to comment
speeding_ant Posted July 19, 2011 Share Posted July 19, 2011 lol - explains a few things... Quote Link to comment
madburg Posted July 19, 2011 Share Posted July 19, 2011 Finally a face to a name Quote Link to comment
defected07 Posted July 19, 2011 Share Posted July 19, 2011 Here's my syslog in regards to the unMenu start-up issues. A few lines which stand out to me (toward the bottom): Jul 18 22:24:00 Tower unmenu[2192]: awk: ./unmenu.awk:1722: fatal: division by zero attempted Jul 18 22:24:00 Tower unmenu-status: Exiting unmenu web-server, exit status code = 2 Jul 18 22:24:00 Tower unmenu[2192]: unmenu.awk unable to open port. It may already be running Jul 18 22:24:10 Tower unmenu-status: Starting unmenu web-server Had difficulty attaching it (even though it's only 80kb), so I used pastebin: http://pastebin.com/g8ymD8GQ Quote Link to comment
Joe L. Posted July 19, 2011 Share Posted July 19, 2011 Here's my syslog in regards to the unMenu start-up issues. A few lines which stand out to me (toward the bottom): Jul 18 22:24:00 Tower unmenu[2192]: awk: ./unmenu.awk:1722: fatal: division by zero attempted Jul 18 22:24:00 Tower unmenu-status: Exiting unmenu web-server, exit status code = 2 Jul 18 22:24:00 Tower unmenu[2192]: unmenu.awk unable to open port. It may already be running Jul 18 22:24:10 Tower unmenu-status: Starting unmenu web-server Had difficulty attaching it (even though it's only 80kb), so I used pastebin: http://pastebin.com/g8ymD8GQ This has nothing directly to do with the upgrade, but is as a result of the new release. A basic patch is already been uploaded to google.code, so cd /boot/unmenu unmenu_install -u Quote Link to comment
defected07 Posted July 19, 2011 Share Posted July 19, 2011 Here's my syslog in regards to the unMenu start-up issues. A few lines which stand out to me (toward the bottom): Jul 18 22:24:00 Tower unmenu[2192]: awk: ./unmenu.awk:1722: fatal: division by zero attempted Jul 18 22:24:00 Tower unmenu-status: Exiting unmenu web-server, exit status code = 2 Jul 18 22:24:00 Tower unmenu[2192]: unmenu.awk unable to open port. It may already be running Jul 18 22:24:10 Tower unmenu-status: Starting unmenu web-server Had difficulty attaching it (even though it's only 80kb), so I used pastebin: http://pastebin.com/g8ymD8GQ This has nothing directly to do with the upgrade, but is as a result of the new release. A basic patch is already been uploaded to google.code, so cd /boot/unmenu unmenu_install -u BEAUTIFUL! Thank you sir. How can I make sure I always have the latest unmenu? Do you just run that command in your go script? Quote Link to comment
WARLOCK Posted July 19, 2011 Share Posted July 19, 2011 BUG or FEATURE ? Can't create usernames with capitals, it will only create users in lower case ? Quote Link to comment
darkside40 Posted July 19, 2011 Share Posted July 19, 2011 Hi there, i have updated my server to 5.0beta9 yesterday and since then the write speed to the array dropped from 40MB/s to 5MB/s. The thing is in the same time the server moved from my office to his new location, now it do not really know if it is a faulty LAN connection or the new Unraid Version (read speed are around 85-90MB/s so i dont think its the network). Nevermind, to test this i would like to go back to beta6a, is that so easy like upgrading from it, or is it dangerous or not even possible? Quote Link to comment
Nezil Posted July 19, 2011 Share Posted July 19, 2011 Warlock, that's a feature, nothing new to beta9. More details like this can be found in the wiki. Quote Link to comment
darkside40 Posted July 19, 2011 Share Posted July 19, 2011 Okay first thing i found out going back to 5.0beta6a solves the problem, after that i have 40-45MB/s write speed under 5.0beta9 only 3MB/s. Reading speed in both Version is normal (85-95MB/s). I am using Cool'n'Quite under both releases. Syslog is attached. Specs of my Server: OS at time of building: 5.0-beta6a CPU: AMD Athlon II X2 240e Motherboard: Asus M4A78LT-M RAM: 2GB KINGSTON CL9 ValueRAM PC3-10666 Case: Sharkoon Rebel 9 Economy Power Supply: Super-Flower SF450P14XE Golden Green Pro (450W, 37A/12V Single Rail design, 80+ Gold) Parity Drive: 2TB Hitachi 5K3000 Data Drives: 2x 2TB Hitachi 5K3000 Cache Drive: none syslog.txt Quote Link to comment
limetech Posted July 19, 2011 Author Share Posted July 19, 2011 BUG or FEATURE ? Can't create usernames with capitals, it will only create users in lower case ? Neither: it's a "limitation" Most linux distros (including slackware) insist on only lowercase for usernames. This is historic and really makes no sense, but sometimes other utilities get broken if not all lowercase, cf: http://fixunix.com/samba/377744-%5Bsamba%5D-username-case-mangling-linux-username-mixed-case-samba-returns-lower-case.html Quote Link to comment
limetech Posted July 19, 2011 Author Share Posted July 19, 2011 Syslog is attached. Did you capture this syslog after running some writes that were slow? Quote Link to comment
limetech Posted July 19, 2011 Author Share Posted July 19, 2011 OK... an update, and Tom; I now see exactly what you mean about the issue. BerkeleyDB has the concept of "checkpointing" where it commits transactions to disk and flushes them from the log. I believe netatalk does this every 1800 seconds (3 min). After running your 'find' do you wait more than 3 minutes before stopping/restarting array? (Though one would think the afp daemon managing the DB would be "smart" enough to do an "explicit" checkpoint upon receiving a TERM signal, but I haven't look at the code for this yet.) Some light reading for you: http://sewm.pku.edu.cn/src/paradise/thirdparty/installed/docs/gsg_txn/CXX/BerkeleyDB-Core-Cxx-Txn.pdf Also you can get lots of info logged by changing this line /etc/rc.d/rc.atalk /usr/sbin/cnid_metad to /usr/sbin/cnid_metad -l LOG_MAXDEBUG Quote Link to comment
darkside40 Posted July 19, 2011 Share Posted July 19, 2011 Syslog is attached. Did you capture this syslog after running some writes that were slow? Yes i startet the Upload of an 8GB ISO File via SMB, but stopped it after a few seconds when i saw that the transfer speeds were terribly slow. Quote Link to comment
defected07 Posted July 19, 2011 Share Posted July 19, 2011 Will you ever had "man" pages to your releases, or do you exclude them to prevent filling up space in RAM/memory stick? I know it's simple enough to look up most commands, but I have a habit of "man <command>" at times... Quote Link to comment
prostuff1 Posted July 19, 2011 Share Posted July 19, 2011 Will you ever had "man" pages to your releases, or do you exclude them to prevent filling up space in RAM/memory stick? I know it's simple enough to look up most commands, but I have a habit of "man <command>" at times... There are talk about this a while back and I think the conclusion and comment given was that they are specifically left out to save space. As you said, most of it can be found easy enough online by type "man <command>" into a google search. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.