NAS
-
Posts
5040 -
Joined
-
Last visited
Content Type
Profiles
Forums
Downloads
Store
Gallery
Bug Reports
Documentation
Landing
Posts posted by NAS
-
-
Ok. I've made a boo boo.
I updated main server to RC2. Have since upgraded through to RC4. I've just moved home and had to relocate servers to new address.
I've had to say bye bye to fibre and go back to ADSL2. As much of a bummer that is, it doesn't compare to this one ....
Can anyone guess???
Yep. My ADSL2 service connection has been delayed until Tuesday. Meaning the Server can't "call home" and therefore by all accounts I won't be able to start it.
Am I really " ... attached to another object by an inclined plane, wrapped helically around an axis?"
Sent from my iPhone using Tapatalk
I understand this was my fault. I also understand that the limitation of the RC requiring Internet access on boot was well documented.
However, as I sit in my new home waiting for my internet to be connected - unable to access my data - I am finding myself getting more annoyed with this "feature".
Firstly, I have to ask why was it included in the RC? A RC is a pre release version which has the potential to be a final product. Meaning surely it is almost ready to release. I would imagine all features to be included are in there already and we have already tested them thoroughly through the many beta cycles over a long period of time and we have had no known showstoppers.
Secondly, I know this is supposed to stop me from running a version which could result in a loss of data but as I sit here having another wine I'm finding that less and less palatable. You announce it is unstable software anyway and shouldn't be used in production so why should you worry this much about our data. This is our responsibility.
Maybe (if you still want to persist with this "feature" - allow us an opt in / opt out in future). If it truley is all about the users and preventing us from running at risk then allow us the ability to accept that risk if we want. It is in fact our risk to own and deal with anyway !!!
!!Finally, I do think (if you won't implement something like I suggest) then it's time you removed this little gem from the feature set. All it is doing for me is preventing me from accessing my array. Thankfully my backup strategy allows me access to my data in that I have a duplicate redundant server but that's not the point.
#Annoyed
100% agree. This code as-is cannot be put into the release product and since this is a candidate for release it should not be in it either.
I think the idea of being able to deprecate Alpha and Beta versions is good.... in fact I will go as far to say that I think its a critical new option and a great idea but if we still need it by the 4th RC then we are doing something wrong with the release cycle.
-
Sure, for the system you have. But it is very helpful for what-if situations.
The community wants to have a database of various setups and their performance so users on lower end systems know what to expect IF they go Dual-Parity. It also helps them if they're having lower performance than they'd like in knowing what a possible upgrade would give them.
Not disputing that, in fact we have something in the works. My objection is there's no point putting it in the Info box.
Not sure I follow. You add a new feature that is CPU intensive. There is a value buried in the logs that lets users know if their systems is capable of supporting the additional CPU workload vs their disk count.
I see no reason why we would ask users to use SSH and grep to find this figure.
However it sounds like you have decided this is the way so at the very least the procedure to check or dimension the CPU for the feature should be documented. This forum post is not documentation.
I think what Limetech is saying is that it's not a question of this being a useful feature for users (it is), it's just a matter of where to present it (within the WebUI).
I hope you are right and I am wrong
I dont feel passionately enough about this to make any stronger case than I have already so I will leave it out there for consideration.One ting I will say though is that my suggestion is for post RC. We should be adding nothing at this stage to get this thing live.
-
As a path of least resistance perhaps system info would be a sane place to present the figures in a human readable way.
You mean something like this ?
Meh - I don't see the value in presenting this info because there's nothing you can do with it. The rate "is what it is".
Sure, for the system you have. But it is very helpful for what-if situations.
The community wants to have a database of various setups and their performance so users on lower end systems know what to expect IF they go Dual-Parity. It also helps them if they're having lower performance than they'd like in knowing what a possible upgrade would give them.
Not disputing that, in fact we have something in the works. My objection is there's no point putting it in the Info box.
Not sure I follow. You add a new feature that is CPU intensive. There is a value buried in the logs that lets users know if their systems is capable of supporting the additional CPU workload vs their disk count.
I see no reason why we would ask users to use SSH and grep to find this figure.
However it sounds like you have decided this is the way so at the very least the procedure to check or dimension the CPU for the feature should be documented. This forum post is not documentation.
-
If you look near the top of your system log, there are a set of messages that output the rate at which various gen/xor algorithms run using different instruction sets. It then picks the best performing function to be used. Here 'gen' means "generate Q" and 'xor' means "generate P", ie, simple XOR.
The first ones that measure xor speed (P calculation) begin with "kernel: xor:", then there are a set that begin with "kernel: raid6:" which measure algorithm used to calculate Q.
Here is what I see for a lowly Atom D510 @ 1.66GHz:
Aug 18 10:13:21 Test1 kernel: xor: measuring software checksum speed Aug 18 10:13:21 Test1 kernel: prefetch64-sse: 5424.000 MB/sec Aug 18 10:13:21 Test1 kernel: generic_sse: 4912.000 MB/sec Aug 18 10:13:21 Test1 kernel: xor: using function: prefetch64-sse (5424.000 MB/sec) : : Aug 18 10:13:21 Test1 kernel: raid6: sse2x1 gen() 113 MB/s Aug 18 10:13:21 Test1 kernel: raid6: sse2x1 xor() 714 MB/s Aug 18 10:13:21 Test1 kernel: raid6: sse2x2 gen() 371 MB/s Aug 18 10:13:21 Test1 kernel: raid6: sse2x2 xor() 1193 MB/s Aug 18 10:13:21 Test1 kernel: raid6: sse2x4 gen() 632 MB/s Aug 18 10:13:21 Test1 kernel: raid6: sse2x4 xor() 1316 MB/s Aug 18 10:13:21 Test1 kernel: raid6: using algorithm sse2x4 gen() 632 MB/s Aug 18 10:13:21 Test1 kernel: raid6: .... xor() 1316 MB/s, rmw enabled Aug 18 10:13:21 Test1 kernel: raid6: using ssse3x2 recovery algorithm
The best it can do is 632 MB/s for gen and 5424 MB/s for xor. If we assume average throughput for attached HDD's is 100 MB/s, then barring other bottlenecks, this says that with 6 or fewer array devices, dual-parity sync/check will be disk bandwidth limited; but, if you go wider than that, CPU will start limiting performance. In single-parity setup, no CPU bottleneck until you go to 54 drives or higher (ie, never a bottleneck).
Here is what I see for a Core i5-4430 @ 3.00 GHz (a modest CPU by today's standards):
Aug 18 09:59:07 test2 kernel: xor: automatically using best checksumming function: Aug 18 09:59:07 test2 kernel: avx : 41780.000 MB/sec : : Aug 18 09:59:07 test2 kernel: raid6: sse2x1 gen() 9300 MB/s Aug 18 09:59:07 test2 kernel: raid6: sse2x1 xor() 7027 MB/s Aug 18 09:59:07 test2 kernel: raid6: sse2x2 gen() 11988 MB/s Aug 18 09:59:07 test2 kernel: raid6: sse2x2 xor() 8070 MB/s Aug 18 09:59:07 test2 kernel: raid6: sse2x4 gen() 14296 MB/s Aug 18 09:59:07 test2 kernel: raid6: sse2x4 xor() 9363 MB/s Aug 18 09:59:07 test2 kernel: raid6: avx2x1 gen() 18269 MB/s Aug 18 09:59:07 test2 kernel: raid6: avx2x2 gen() 21593 MB/s Aug 18 09:59:07 test2 kernel: raid6: avx2x4 gen() 25238 MB/s Aug 18 09:59:07 test2 kernel: raid6: using algorithm avx2x4 gen() 25238 MB/s Aug 18 09:59:07 test2 kernel: raid6: sse2x1 xor() 7031 MB/s Aug 18 09:59:07 test2 kernel: raid6: sse2x2 xor() 8330 MB/s Aug 18 09:59:07 test2 kernel: raid6: sse2x4 xor() 9363 MB/s Aug 18 09:59:07 test2 kernel: raid6: .... xor() 9363 MB/s, rmw enabled Aug 18 09:59:07 test2 kernel: raid6: using avx2x2 recovery algorithm
This should support a 25-drive array before CPU becomes bottleneck in dual-parity sync/check. Notice that for the xor calculation, if AVX instruction set is present, kernel uses it and doesn't bother measuring others.
Very interesting and useful information. Given that the forum, wiki and google have a decade of CPU recommendations that do not take this into account we should promote these figures out of the logs and into the GUI.
As a path of least resistance perhaps system info would be a sane place to present the figures in a human readable way.
-
The shares not being up is the only universal notification you know users have and we should always defer to that safest default.
To go beyind this actions like this need to be taken in the context of notifications being setup per install. We should not assume users here represent the actual userbase, most wont have more the the very basic setup.
-
The reality is that if you protect the ports with an external firewall then the risk of being breached is close to identical that if you hosted them direct. This is because the VM/docker layer is not really visible to the outside worlld (not 100% true but lets not go down that rabbit hole).
That is the good news.
The bad news is what happens if someone exploits vulnerability in your applications and gets root. At that point you are relying 100% on unRAID to protect you from attack escalation. The chances are you dont have a DMZ in this setup as current unRAID does not lend itself to this. Also this is where unRAID patching is critical. Docker/VM et all need to be up to date to reduce the risk of a known exploit being abused to breakout into the host. Currently there is no know exploit however there could be tomorrow and this is why it is critical you apply security patches.
Currently the non beta unRAID is approximately 6 months behind with all security patches and over 1 year behind with Docker.
I would not recommend the current unRAID stable be internet facing at all as it is just too old. I cannot also recommend the current beta line as it requires a call home working to boot.
It is not all doom and gloom. It is just a matter of risk and the risk is low IF you keep your applicaitons and VMs bang up to date with security patches but currently uNRAID stable fails a security audit very very badly.
-
On a related point. At this time the only even vaguely secure version of unRAID is the RC line however it requires internet access to boot. I would assume HTTPS to a LT domain but I havent looked into it.
In my personal opinion only; requiring an internet based server operated by a 3rd party be both up and contactable before your local server will boot is not a viable option for any production system
So until the next stable is released you have no elegant solution.
-
Why haven't i heard or found this before.
HUGE +1 for me, game changer.
-
Your recommendation to me was do not upgrate to beta builds without any experience with Linux / unraid system.
RC2 works stable now ? Can i upgrade without much experience...?
If it is for a production but non critical system I would normally say an absolute YES to upgrading to an unRAID RC. However as I understand it the beta call home is still in place and as such should the internet "licensing" server go offline or be uncontactable your production server would fail to boot.
Important: I am not 100% sure if this statement is true as I dont think anyone knows exactly what passes to between the systems but it seems to be the logical interpretation of "Your server must have access to the Internet to use the unRAID 6.2 rc." and if so that is a extra factor to take into account.
The more RC testers the better but eyes should be open going in.
-
New wording to cover smaller disks and SSDs:
* ReiserFS, essentially read-only usage - 99%
* ReiserFS, essentially read-write usage - 93%
* XFS/BTRFS, essentially read-only usage - 2% or 20GB free (whichever is smaller)
* XFS/BTRFS, essentially read-write usage - 5% or 50GB free (whichever is smaller)
-
Sometimes I despair over the internet.
So much discussion over one line of change log just to help out users who dont read the forum (most). Who cares if its not the 100% right thing to do when 7 words can save bunch of users a headache.
oh well, forget I asked
-
Normally I would agree however not this time.
Preclear has been a cornerstone of unRAID community for years now. If an unRAID release breaks something used by so many users should be informed.
Its just the right thing to do for the community.
-
there is a new preclear script that should be used with 6.2
http://lime-technology.com/forum/index.php?topic=39985.510
This needs added to the changelog. Most users dont read the forum and certainly not page 35 of a 50 page thread
-
Bumping the Kernel within a RC cycle is pretty unusual, assume this was an exception to fix something?
It's a kernel "maintenance" release, ie, bug fixes only. unRAID-6.2 will stay on kernel-4.4.
Excellent, thanks for the clarification.
-
Bumping the Kernel within a RC cycle is pretty unusual, assume this was an exception to fix something?
-
First off thanks to everyone for replying. I can confirm that my real world "feel" for RFS slowdowns matches what RobJ summarized above.
I see no reason not to base discussion now on RobJ recommendations:
* ReiserFS, essentially read-only usage - 99%
* ReiserFS, essentially read-write usage - 93%
* XFS, essentially read-only usage - 20GB free
* XFS, essentially read-write usage - 50GB free
* BTRFS, essentially read-only usage - 20GB free
* BTRFS, essentially read-write usage - 50GB free
as any difference between these and my personal recommendations would be within an error margin anyway.
So the first thing that catches my eye is based on the 93% recommendation for ReiserFS in read-write usage versus the 50GB recommendation for XFS just how superior in this respect XFS is. An 8TB drive which formats at approx 7.96TB of usable space, ReiserFS needs 507GB extra reservation per disk. In a full array that is a whopping 11.6TB of extra "wasted" space.
One areas we havent touched on is SSD which apart from being a different technology are much smaller and the flat numbers above for XFS could consume a much higher percentage of the disk than intended.
-
Minimum free space recommendations discussions forked here
-
Inspired by the following LT staffer personal recommendation:
Personally, I strive to keep disk usage below 90% because anything above that could degrade performance for writes on certain filesystems.
There are few follow on discussions but in general terms everyone seems to agree that:
- You should never fill up a drive to 100%
- You should ALWAYS reserve a certain amount of free space
- The amount of reserved free space depends on the filesystem type
- The amount of free space reserved depends on the use case (i.e. Predominately RO disks need less reservation that RW disks).
What is not agreed upon is:
- Should the reservation be a percentage, a GB amount or a combination of both.
- The differences each FS type makes (e.g. anecdotal evidence suggests XFS needs hardly any reservation whereas RFS needs comparitively lots).
This is an important topic as people will (and should) try to follow recommendations. More so, with the advent of 8TB + disks a strict adherence to the "10%" reservation recommendation amounts to 18.4TB of unused reserved space on a fully populated array (which is obviously a lot).
Google yields lots of other personal recommendations from a the tiny (few MB) to the insane (50%), but no hard facts.
The aim of this thread is to agree on a reservation amount recommendations for unRAID users of each supported filesystem type (if different). Ideally this should be a formal statement but failing hard facts a consensus of personal recommendations would best.
Thoughts?
- You should never fill up a drive to 100%
-
Agree it's useful to have a bit of free space if you're still writing to a drive.
I've got 3 servers (plus a test one that I change around a good bit) => of the 3 primary servers, one is RFS (my oldest media server) ... with 12 of the 14 drives 100% full (< 1GB of free space), and the other 2 have a modest amount of free space and are the only ones still written to. There's no ready penalty from full drives, so this works just fine.
My other 2 servers are both XFS -- one has a LOT of free space; the other has a few fairly full drives (> 99%), but they still have 30GB or so free, and I've seen NO degradation of writes to these drives. XFS is clearly MUCH better than Reiser in terms of writes to full drives.
My 6.2 test unit is all XFS ... I'll "play" a bit with it to see if this behavior is any different with 6.2, but I doubt that it will be, as the only real difference is the 2nd parity drive, which shouldn't have any impact on the file system behavior.
By the way, since this IS the 6.2 RC thread, I'll note that I just upgraded to the RC and all is working just fine
garycase, gubbgnutten and RobJ thanks for the replys and I have to say I agree with what you are saying. This is one reason why i quoted and queried the original recommendation to see if we could move this to a firmer non "personal" recommendation.
10% reservation if adhered to could represent two complete drives worth of space on a full populated unRAID server which seems quite high. Equally I havent seen any slow down with XFS and high fill levels but as most here my data on these drives ire relatively static.
However in among these assumptions are real uses cases where larger reservation would make a real world difference and I would like to nail that down to a point where a couple of paragraphs in the manual could explain it or even better the GUI could feed back to the user. Should we fork this thread/do me have enough interest to resolve it?
-
Unrelated, Disk1 is at 99% full (19GB free). Personally, I strive to keep disk usage below 90% because anything above that could degrade performance for writes on certain filesystems.
I believe this is a new recommendation? Can you expand on this in the context of RFS and XFS? Happy to start a new thread if needed as it sounds like quite an important consideration.
-
Question, is the call home still in the RC?
First time using 6.2? please read the Original 6.2-beta Announcement Post first.
IMPORTANT
[*]Your server must have access to the Internet to use the unRAID 6.2 rc.
Appreciated, my eye skipped right past that to the CVE list.
-
Lovely to see the CVE fixes and the documentation of such. Clear and precise so appreciated and nice work all.
Question, is the call home still in the RC?
-
Another angle, 33W is a low enough number that the money required to significantly better it would probably vastly outweigh the electricity costs of running as is
... at least in this decade -
Having a brain fart.
I have a umount script that Unassigned Devices calls and does a bunch of things like index etc and its great. Totally rely on it for my workflow so a big thanks.
However I just went to add a simple `smartctl -a /dev/sds > blah.txt` to it but it seems the partition device is set as a variable i.e. /dev/sds1 but the disk device partition device, e.g /dev/sda is not.
Before I go away an pull together a kludge and I missing something obvious?
Only partition is exported. You can extract the disk device using the following code:
DISK=$(echo $DEVICE | grep -Po "/dev/sd[a-z]$")
Appreciated as always.
!!
I dont feel passionately enough about this to make any stronger case than I have already so I will leave it out there for consideration.
unRAID "Phone Home" discussion
in Pre-Sales Support
Posted
To be fair this is a 99.999% open Linux product so I am not sure the 100% closed Microsoft activation analogy holds. More than that the feature is in place for deprecation and not activation. Two sides of the same coin but they are not the same.
I think it is fair to ask for this information, I cant see why what is being sent and received should be a secret.