When will the "official" release come?

My unRAID is a "productive" sys and i dont want to use a RC.

Edited June 7, 20179 yr by Zonediver

1 minute ago, dlandon said:

There are some well deserved kudos for the new goodies in the 6.4 version:

- The Azure and Gray themes are awesome.  My personal favorite is Azure.

- The responsiveness of the webUI is better than I've ever seen.

- The inclusion of IPV6 and the added networking features is excellent.  Being able to assign IP addresses to a docker is much cleaner in my mind than port management for one IP address.

 

Many thanks to the LT team, bonienl, and all others involved in getting this version ready.  Job well done!

 

Thanks for the kudos!  Most of the credit goes to bonienl!

14 minutes ago, Zonediver said:

When will the "official" release come?

My unRAID is a "productive" sys and i dont want to use a RC.

 

Normally is released when they feel comfortable with all RC releases and Patches that might follow. 

So in simple terms it could be tomorrow or it could be several months. Trust me your not the only one waiting. 

 

When it comes to your Data and the Data of many, nobody wants to rush anything so testing testing testing is very important. 

 

On 5.6.2017 at 4:00 PM, Benson said:

I got serious disk array throughput issue that ceiling ~700MB/s R/W, it will worse if increase in data disks.

I did some more investigaton and there is indeed a bottleneck which could be fixed easily. I seems that the parity calculation is single threaded. Even on modern CPUs with enough multi core power this can lead to to one cpu core being maxed out in 2 parity disk with many data disk situations. I verified this by changing cpu clock speed while looking at achievable parity check speeds and cpu load. I also made sure there are no other pci lane/a-link/sata-controller bottlenecks. While most cores were only at 10-20% load one core was maxxed out (process unraidd) and limited parity performance and performance scaled linearily with cpu clock speed. 

 

Conclusion: unraid uses only a single thread to do parity calculations, which can easily max out one cpu core, especially when 2 parity disks are in use, bottlenecking both parity checks and recontructive write speeds.

 

Good news: As parity calculation is easily parallizable, this bottneck can be entirely avoided when unraid devs finally make their code multithreaded.

 

Disclaimer: I cannot rule out that the maxxing out of one core is due to something else and that parity calculation is already multithreaded. Maybe the devs can shed some light on this.

Edited June 8, 20179 yr by Videodr0me

9 hours ago, Videodr0me said:

I did some more investigaton and there is indeed a bottleneck which could be fixed easily. I seems that the parity calculation is single threaded. Even on modern CPUs with enough multi core power this can lead to to one cpu core being maxed out in 2 parity disk with many data disk situations. I verified this by changing cpu clock speed while looking at achievable parity check speeds and cpu load. I also made sure there are no other pci lane/a-link/sata-controller bottlenecks. While most cores were only at 10-20% load one core was maxxed out (process unraidd) and limited parity performance and performance scaled linearily with cpu clock speed. 

 

Conclusion: unraid uses only a single thread to do parity calculations, which can easily max out one cpu core, especially when 2 parity disks are in use, bottlenecking both parity checks and recontructive write speeds.

 

Good news: As parity calculation is easily parallizable, this bottneck can be entirely avoided when unraid devs finally make their code multithreaded.

 

Disclaimer: I cannot rule out that the maxxing out of one core is due to something else and that parity calculation is already multithreaded. Maybe the devs can shed some light on this.

 

Thanks for interesting in this issue, the test haven't involve any parity calculation, I just read one file in one data disk at same time. For a upper low-end CPU, it already can calculate parity in several GB/s level.

 

The 700MB/s array throughput also not relate how many thread nor parallelize hash protocol, it may relate FUSE FS + Btrfs, there are many "kworker" process for Btrfs.

Edited June 8, 20179 yr by Benson

19 hours ago, Videodr0me said:

I did some more investigaton and there is indeed a bottleneck which could be fixed easily. I seems that the parity calculation is single threaded. Even on modern CPUs with enough multi core power this can lead to to one cpu core being maxed out in 2 parity disk with many data disk situations. I verified this by changing cpu clock speed while looking at achievable parity check speeds and cpu load. I also made sure there are no other pci lane/a-link/sata-controller bottlenecks. While most cores were only at 10-20% load one core was maxxed out (process unraidd) and limited parity performance and performance scaled linearily with cpu clock speed. 

 

Conclusion: unraid uses only a single thread to do parity calculations, which can easily max out one cpu core, especially when 2 parity disks are in use, bottlenecking both parity checks and recontructive write speeds.

 

Good news: As parity calculation is easily parallizable, this bottneck can be entirely avoided when unraid devs finally make their code multithreaded.

 

Disclaimer: I cannot rule out that the maxxing out of one core is due to something else and that parity calculation is already multithreaded. Maybe the devs can shed some light on this.

 

Single Parity Calculation is never an issue and is never a bottleneck.

Dual Parity Calculation might be an issue, but it depends on having recent hardware. For instance my rates seem to be around 9.7 GB/s [10,0003 MB/s]

 

What do you see in your syslog when it tests out Parity Calculations to decide on the best implementation of it? I've included the "limits" of the various XOR implimentations for my hardware. Yours will be different.

 

May 26 20:24:38 X kernel: xor: automatically using best checksumming function   avx
May 26 20:24:38 X kernel: raid6: sse2x1   gen()  6750 MB/s
May 26 20:24:38 X kernel: raid6: sse2x1   xor()  4970 MB/s
May 26 20:24:38 X kernel: raid6: sse2x2   gen()  8652 MB/s
May 26 20:24:38 X kernel: raid6: sse2x2   xor()  5937 MB/s
May 26 20:24:38 X kernel: raid6: sse2x4   gen() 10003 MB/s
May 26 20:24:38 X kernel: raid6: sse2x4   xor()  6843 MB/s
May 26 20:24:38 X kernel: raid6: using algorithm sse2x4 gen() 10003 MB/s
May 26 20:24:38 X kernel: raid6: .... xor() 6843 MB/s, rmw enabled
May 26 20:24:38 X kernel: raid6: using ssse3x2 recovery algorithm

 

Edited June 9, 20179 yr by BRiT
correcting the computed limit, silly decimal point issue...

LT, I'm still not getting any love from my Micron 320h NVMe drive, I'm happy to let dev's into the server if they want to play around?

Otherwise happy to ship one of the drives to you for development work


Sent from my iPhone using Tapatalk

1 hour ago, miniwalks said:

LT, I'm still not getting any love from my Micron 320h NVMe drive, I'm happy to let dev's into the server if they want to play around?

 

There's a Defect Report open on this device:

https://forums.lime-technology.com/topic/56350-633-mtip32xx-drives-dont-show/#comment-566975

 

I don't think it's fully NVMe compliant which is the problem.

4 hours ago, BRiT said:

 

Single Parity Calculation is never an issue and is never a bottleneck.

Dual Parity Calculation might be an issue, but it depends on having recent hardware. For instance my rates seem to be around 9.7 GB/s [10,0003 MB/s]

 

What do you see in your syslog when it tests out Parity Calculations to decide on the best implementation of it? I've included the "limits" of the various XOR implimentations for my hardware. Yours will be different.

 

8

As has been shown by other users here (see johnnie.black posts in the 6.20rc thread) the limits of the raid6 calculation are not a good indicator of real live parity speeds with unraid. Also people should not confuse modern cpus with high end fast cpus, there are a lot of "newer" low power cpu's which are very much suited for NAS duty. It's a crime against the efficiency gods not to use multiple cores if available and the problem can be parallized (as is the case here, just partition the stripes between the different cpu cores and let them do their work in parallel or - not quite as good - at least use one core per parity drive). As for the test system i used:

 

Jun  6 13:53:13 Tower kernel: raid6: using algorithm sse2x4 gen() 2039 MB/s
Jun  6 13:53:13 Tower kernel: raid6: .... xor() 2457 MB/s, rmw enabled
Jun  6 13:53:13 Tower kernel: raid6: using ssse3x2 recovery algorithm

with the dual parity setup and 5 data drives (7 8tb drives in total) it bottlenecked at 700mb/s with one core being maxed out by the unraidd process (100% load of this core) while the other 3 cores were idling or at 10%-20% load. In theory 2039/2 (for dual pairy) should be enough and yield 1 Gb/s througput but as almost always these numbers do not directly translate into real world unraid performance. These findings are stable if i concurrently produce more load on the pci bus by reading from the SSD cache disk and other UD disks - total disk speed exceeded 1 Gb/s while parity speed remained at approx 700mb/s.

 

My Conclusion: unraid's parity calculation is bottlenecked by being single threaded, which is very unfortunate for even some of the newer low power cpus. Even more so as these are exactly the kind of systems unraid was conceived for (making great use of old/low power hardware). Luckily this can be fixed by making the code multithreaded. Not a simple task if the devs have not had experience with multithreaded programming, but should be doable.. And as array sizes grow and many switch to dual parity, it will becomes even more relevant in the future.

 

Edited June 9, 20179 yr by Videodr0me

 
There's a Defect Report open on this device:
https://forums.lime-technology.com/topic/56350-633-mtip32xx-drives-dont-show/#comment-566975
 
I don't think it's fully NVMe compliant which is the problem.

Well that's a damn shame haha, ive got 4 of them sitting here now doing very little

I wonder if they're planning a firmware upgrade or something to resolve it or if that's it for those disks

The story continues


Sent from my iPhone using Tapatalk

6 hours ago, Videodr0me said:

8

As has been shown by other users here (see johnnie.black posts in the 6.20rc thread) the limits of the raid6 calculation are not a good indicator of real live parity speeds with unraid. Also people should not confuse modern cpus with high end fast cpus, there are a lot of "newer" low power cpu's which are very much suited for NAS duty. It's a crime against the efficiency gods not to use multiple cores if available and the problem can be parallized (as is the case here, just partition the stripes between the different cpu cores and let them do their work in parallel or - not quite as good - at least use one core per parity drive). As for the test system i used:

 

Jun  6 13:53:13 Tower kernel: raid6: using algorithm sse2x4 gen() 2039 MB/s
Jun  6 13:53:13 Tower kernel: raid6: .... xor() 2457 MB/s, rmw enabled
Jun  6 13:53:13 Tower kernel: raid6: using ssse3x2 recovery algorithm

with the dual parity setup and 5 data drives (7 8tb drives in total) it bottlenecked at 700mb/s with one core being maxed out by the unraidd process (100% load of this core) while the other 3 cores were idling or at 10%-20% load. In theory 2039/2 (for dual pairy) should be enough and yield 1 Gb/s througput but as almost always these numbers do not directly translate into real world unraid performance. These findings are stable if i concurrently produce more load on the pci bus by reading from the SSD cache disk and other UD disks - total disk speed exceeded 1 Gb/s while parity speed remained at approx 700mb/s.

 

My Conclusion: unraid's parity calculation is bottlenecked by being single threaded, which is very unfortunate for even some of the newer low power cpus. Even more so as these are exactly the kind of systems unraid was conceived for (making great use of old/low power hardware). Luckily this can be fixed by making the code multithreaded. Not a simple task if the devs have not had experience with multithreaded programming, but should be doable.. And as array sizes grow and many switch to dual parity, it will becomes even more relevant in the future.

 

 

I dont accept your conclusion. I dont think the two are connected one iota.

4 minutes ago, BRiT said:

 

I dont accept your conclusion. I dont think the two are connected one iota.

Care to elaborate? The facts on my system are:

1) Changes in CPU frequency affect parity speed linearily

2) process unraidd is at 100% maxing out 1 of 4 cores completely while the other 3 cores are aroung 10-20% load. These are real cores, not hyperthreaded.

3) PCI/SATA/A-LINK are not saturated as I can generate additonal concurrent system load (copying from cache ssd to UD disks) having a total system transfer speeds exceeding 1 Gb/s, while parity speed remains at 720 mb/s

4) My findings are in line with those of frank1940: https://forums.lime-technology.com/topic/49327-sudden-change-in-parity-check-times/?do=findComment&comment=489425

 

Maybe the unraidd process does not saturate one core completely because of parity caculation, but because of something else. Though not impossible it seems unlikely as with a single parity drive there was no bottleneck. Only the devs can shed more light on this, maybe they like to weigh in....  i would welcome a solution, be it in form of multithreaded parity calculation or by finding/optimizing unraidd to work better on low power/older hardware (sustainability, save the planet bla bla).

Upgraded today from 6.2.4 - all seems well and seems to have solved the odd timestamp modification caused in 6.3 series.

 

Although I don't like the new themes, Azure doesn't have enough contrast on some things, far too much space.

 

Back to 'white' for me, sorry!!!

2 hours ago, Interstellar said:

Upgraded today from 6.2.4 - all seems well and seems to have solved the odd timestamp modification caused in 6.3 series.

 

Although I don't like the new themes, Azure doesn't have enough contrast on some things, far too much space.

 

Back to 'white' for me, sorry!!!

 

I actually like the new theme but agree it needs work! I will stick with it in the hopes that they introduce some slider bars so we can adjust the colours to our liking. Hint Hint! I would also love to see a mobile friendly web theme or even a mobile app for managing unraid! here's hoping   

6 minutes ago, sarf said:

 

I actually like the new theme but agree it needs work! I will stick with it in the hopes that they introduce some slider bars so we can adjust the colours to our liking. Hint Hint! I would also love to see a mobile friendly web theme or even a mobile app for managing unraid! here's hoping   

 

Everybody is invited to work on the themes. They are publicly available on github.

Does this RC have the online licence check that stops VM's from booting before the licence is validated?

 

Not that having a license validated is an issue, just the fact i need my VM's to come online (virtualised pFsense) before my machine is able to gain internet access to complete the validation. 

Edited June 10, 20179 yr by bennymundz
spelling

13 hours ago, bennymundz said:

Does this RC have the online licence check that stops VM's from booting before the licence is validated?

 

Not that having a license validated is an issue, just the fact i need my VM's to come online (virtualised pFsense) before my machine is able to gain internet access to complete the validation. 

As far as I can tell no it does not have the check. My system booted before my network was established. I have a pFsense VM as well.

Only trial versions do the online check

Sent from my LG-D852 using Tapatalk

Cool, 6.4 is already RC. Looking forward when it becomes final!

 

You guys do me great pleasure with Docker and the possibility to select a custom ip adres. Kudos!

Edited June 10, 20179 yr by MvL

8 hours ago, captain134 said:

As far as I can tell no it does not have the check. My system booted before my network was established. I have a pFsense VM as well.

 

Perfect. 6.4 here we come :-)

And im online. I really like the grey theme. Tho i think the txt colour could be a little lighter. Overall look and feel is great. 

I want to try 6.4 rc but need to know which plugins are broken if any with it? Please report back if you are successfully using these plugins...

 

CA Auto Update Applications

CA Backup / Restore Appdata

CA Cleanup Appdata

Community Applications

Dynamix Active Streams

Dynamix Auto Fan Control

Dynamix Cache Directories

Dynamix Local Master

Dynamix Schedules

Dynamix SSD TRIM

Dynamix System Buttons

Dynamix System Information

Dynamix System Statistics

Dynamix System Temperature

File Activity

Fix Common Problems

Libvirt Hotplug USB

Nerd Tools

rclone

Tips and Tweaks

Unassigned Devices

User Scripts

I want to try 6.4 rc but need to know which plugins are broken if any with it? Please report back if you are successfully using these plugins...
 
CA Auto Update Applications
CA Backup / Restore Appdata
CA Cleanup Appdata
Community Applications
Dynamix Active Streams
Dynamix Auto Fan Control
Dynamix Cache Directories
Dynamix Local Master
Dynamix Schedules
Dynamix SSD TRIM
Dynamix System Buttons
Dynamix System Information
Dynamix System Statistics
Dynamix System Temperature
File Activity
Fix Common Problems
Libvirt Hotplug USB
Nerd Tools
rclone
Tips and Tweaks
Unassigned Devices
User Scripts

Need pack would be the only issue if you have to change any thing in it. Not sure if dlandon has updated it yet

Sent from my LG-D852 using Tapatalk

WebUI seems much more fluid now!

 

Just a warning to anyone using letsencrypt/nginx/etc dockers.  The unRAID webui now wont start if you're using port local 443, switch your container to use an alternate local port before upgrading to save yourself a headache.

Edited June 13, 20179 yr by Dephcon

I upgraded but unraid would not boot correctly, I could not putty but use ipmi to console, see attached diag logs.

tower-diagnostics-20170613-1611.zip