unRAID OS version 6.4.0-rc21b available


Recommended Posts

Upgrading: We have changed the way one checks for new unRAID OS releases.  Please refer to Update OS below.

Bugs: If you want to report an issue, please start a new topic in this board.

 

Notable changes (-rc21b):

  • Linux kernel 4.14.13.  More refinements for Meltdown.  Also an interesting patch was made to the kernel necessary to update Ryzen microcode, though no Ryzen microcode update has been posted publicly AFAIK.
  • Intel microcode update.  No one outside Intel knows what these firmware changes do, but probably necessary for upcoming Spectre patches.
  • The issue causing ttyd (web Terminal) to not work with IE/Edge has been fixed.  Safari is still broken.  If you use Safari, quit doing that.  Install and use FireFox instead.
  • Provide link to latest virtio-win drivers for Windows VM's.
  • Misc. bug fixes.

 

Notable changes (-rc20a):

Full implementation of patches to address Meltdown/Spectre is going to take some time.  We are closely watching progress and will make every effort to update unRAID OS as quickly as possible.

 

Notable changes (-rc19b):

  • Linux kernel 4.14.11.  The VM shutdown issue should be solved with this.  Important: If you notice any issue with VM's vs. how they behaved in 6.3.5 or earlier 6.4.0-rc releases, please post your experience in this separate VM Issues topic.
  • Refinements in the web-based Terminal app.  Clicking the Terminal button creates a separate window as before (and now with proper interpretation of /root/.bash_profile), but you may also open multiple windows (thanks @bonienl) also now right-click the button and Open link in new tab if you prefer (thanks @realies).
    NOTE: the web-based Terminal app only works with Firefox and Chrome.  Using Edge actually crashes the server-side ttyd daemon(!) and using Safari doesn't work due to longstanding Safari websockets issue.
  • Misc. bug fixes/improvements.

 

Notable changes (-rc18f):

  • Linux kernel 4.14.9 (with better patch).  Looks like the VM shutdown issue may be solved using latest patch referenced in the “Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover” Bugzilla topic.  Please check if this release solves this issue for you.  This patch will eventually find its way into an upcoming stable kernel release, maybe 4.14.10 or 4.14.11.  This is the last major issue to resolve before we can publish 6.4.0 'stable' release.
  • We now have a web-based Terminal application available by clicking the Terminal button on the Menu bar.  Kudos to @eschultz for integrating the ttyd application with our nginx install.  This is using websockets and appears to be quite fast, give it a try!
  • We ported a simplified version of the zenstates.py utility to C (to avoid including python in bzroot) which may be used to disable Ryzen C6 states (as workaround for Ryzen idle freeze issue).  We have found that sometimes bios option to disable C6 does not exist or does not do the right thing.  If you want to use this utility, we suggest that you edit your 'config/go' file on your USB flash device.  Add this line just before emhttp is invoked:
    zenstates --c6-disable

     

  • Misc. bug fixes

 

Finally, we also want to thank @Squid for some useful webGui improvements and fixes (in addition to all the help he provides around here): thanks a lot man!

 

Notable changes (-rc17b):

  • Linux kernel 4.14.7 (without "experimental" patch).  We cannot remain on 4.14.4 with a non-official patch set and it does not seem to completely solve the VM issues anyway.  This release (4.14.7) includes a patch (net: accept UFO datagrams from tuntap and packet) from a developer (Willem de Bruijn) who has been participating in the “Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover” Bugzilla topic.   We are thinking the VM freeze/unable to terminate issue may have to do with UDP fragmentation offload.  You can check this using the command:
    ethtool -k eth0
    which will report whether or not your h/w supports this feature.  If you see this, it doesn't:
    Cannot get device udp-fragmentation-offload settings: Operation not supported

     

  • Updated docker.
  • Misc. bug fixes.

 

Notable changes (-rc16b):

  • Linux kernel 4.14.4 (with "Revert - Remove UDP Fragmentation Offload support" patch).  This is an "experimental" patch that is meant to solve VM issues introduced after moving to 4.14 kernel.  Please report if this works for you.
  • Early microcode loading support.
  • Addition of background threads that auto-update local IP address with LimeTech DNS server when using Let's Encrypt SSL certs, and that auto-renew these certs if within 30 days of expiration.
  • Misc. bug fixes.

 

Notable changes (-rc15e):

  • Linux kernel 4.14.4.  This is a "longterm maintenance" kernel, which is nice.
  • There is now a Logout button for the webGUI which appears on right side of the menu bar when you have defined a root password and are logged in.  Clicking this will "log you out" from all tabs/windows in that browser.
  • Misc. bug fixes.

 

Notable changes (-rc14):

  • Fix "swap/disable" operation.
  • Restore proper interpretation of SAS device temperature.
  • Prevent mover from running when no cache volume present, or only the cache volume is present.

 

Notable changes (-rc13):

  • Correct handling of SMART ID 190.
  • Replaced 'tabs' for 'spaces' in the make_bootable.bat file (Windows doesn't like tabs).

 

Notable changes (-rc12a):

  • None, just bug fixes, well updated linux kernel minor patch release  ;)

 

Notable changes (-rc11i):

  • Let's Encrypt SSL certificate renewal: if your certificate is within 30 days of expiration the Renew button on Settings/Identification/SSLCertificate Settings page will be enabled.  LE certs have a lifetime of 90 days but it's free to renew.  We are working on a configurable daemon to make this automatic.
  • All encryption passphrase/keyfile handling is managed on the Main page.  For now the Encryption Settings page is gone but will eventually return as a way of changing the default cipher settings.  Also gone is the Restricted Start setting introduced in the last release (bad idea).
  • Yet another attempt to reliably report NVMe temperatures.

 

Notable changes (-rc10b):

  • Finally looks like the AMD Ryzen GPU pass-through performance issue has been solved.  We applied this patch and it does indeed appear to solve the problem.
  • Fixed issue in the 'mover' where it wasn't handling split-levels correctly in moving files from cache to array.
  • Further refinements in handling encryption keyfile.  Added new config setting Settings/Disk Settings/Restricted Start [Yes/No].  When set to Yes, array will not Start if there are encrypted volumes and the encryption key is missing (this is the default).  If set to No and encryption key is missing, array will Start but encrypted volumes will not be Mounted and also cannot be Formatted - any share data stored therein simply won't be available.
  • After attempted array Start with missing/wrong keyfile, you can enter the passphrase/upload keyfile directly from Main page.  If there are any encrypted volumes in your server, we recommend setting "Settings/Disk Settings/Enable auto start" to Yes.  Following boot of course there will be no keyfile present so autostart will fail.  But in this case s/w also now knows there are encrypted devices and you will see right on the Main page a place to enter the encryption passphrase or upload a keyfile.
  • Numerous package updates, bug fixes and webGui enhancements.

     

    Notable changes (-rc9f):

    • Improved handling of encryption passphrase/keyfile.  When Starting array with encrypted volumes, you only need to enter the encryption passphrase once on the Encryption Settings page - no more confusing "passphrase confirmation".  If no encrypted devices exist and you're trying to add some, then it will ask for passphrase confirmation.
    • Introduce new Disk Setting called 'Restricted Start - Yes/No'.  When set to 'Yes' then array will not Start if the encryption key is missing.  If set to 'No' then array will Start (including autostart) but encrypted volumes will not be 'mounted', meaning shares and/or share data stored on them will not be accessible.  The default (and normal) setting for this is 'Yes'.
    • The Let's Encrypt SSL provisioning is only available when 'Use SSL/TLS' on Identification page is set to 'Auto'.  Also, provisioning the cert no longer triggers complete restart of "services".
    • If using 'https' all 'http' is redirected to 'https'.  If not using 'https', all 'https' is redirected to 'http'.  The result of this is you can always enter servername in browser address bar to get to webGui, for example "Tower/" or "Tower.local" should always get you to the webGui.  In the case of SSL-enabled LE certificate, you will get redirected to the <hash>.unraid.net URL.
    • Added an 'Update DNS' button on Identification page.  If the IP address of your server changes and you're usng the LE certificate, you can click this button to tell unraid.net to update the DNS setting.  We have set TTL to 60 seconds so it might take this long to see the update.  Of course you have to already have the webGui open to do this.
    • Finally fixed reporting of temperature for NVMe devices (hopefully).
    • Updated OVMF firmware, tested with various OS types, seems to work.
    • Other misc. fixes an improvements, refer to Changes below.

    We're at the end of life for linux kernel 4.12. Next release will move to 4.13 kernel.

    Secure Access (-rc8q):

    Probably some explanation is in order.  The “major” feature we wanted to add into unRAID version 6.4 was block level device encryption.  However to get there we realized there needs to exist a secure way of entering information such as passphrases.  Hence phase 1 consisted of integrating nginx in order to leverage its support of SSL/TLS (https).

    Besides the benefit of https support, integration of nginx also lets us utilize websocket technology (which is an ongoing integration), and lets us greatly improve the overall responsiveness of the webGui.

    Phase 1 integration of nginx in unRAID only supports self-signed SSL certificates.  While in general, this may be OK to provide encrypted connections between a browser and a server in a trusted LAN, relying on self-signed certs is not good practice and is theoretically vulnerable to MITM attacks.

    With this release we have completed Phase 2 of nginx https integration by providing the ability for our users to provision a free SSL Certficate from Let’s Encrypt.  To obtain your certificate go to Settings/Identification, scroll to the bottom and click Provision.  In one operation this will allocate your certificate, upload it to your server, and switch nginx to redirect all http to https.  After clicking anywhere else in the webGui you should see a nice green lock icon in your browser address bar!

    The other thing you’ll notice in your address bar is a very funny looking URL consisting of a 40-hex-character subdomain of unraid.net.  We have set up a LimeTech DNS server that will resolve that URL to your servers IP address on your local network.  That FQDN is unique to your certificate.  When your browser resolves that URL it is given your local IP address which it then uses to perform the https connection handshake.  For this reason, we recommend that you give your server a static IP address because if the IP address changes, the browser will not be able to connect to your server.  This is like locking your keys in the car!  We plan on implementing a small daemon which wakes up upon such IP address change and tells the LimeTech DNS server to update its A-record, but this has not been done yet.  NOTE: if you do lock your keys in the car, the coat-hanger fix to restore http access is to telnet/ssh into the server and type:

    rm /boot/config/ssl/certs/certificate_bundle.pem
    /etc/rc.d/rc.nginx reload

    (You might also have to clear your browser cache.)

    Following re-enable of http, you can again Provision a certificate which will update the DNS entry.

    Device block level encryption (-rc8q):

    We have implemented full-device encryption as follows.  In unRAID, encryption is selected as another type of file system.  For example, with array Stopped, click on a Device link and then click on File system type.  Three new “types” are available:

    • xfs – encrypted
    • btrfs – encrypted
    • reiserfs – encrypted [should we get rid of this one?]

    If you change the File system type to one of these and then Start the array you will notice the device appears Unmountable and the Format button is available.  Formatting the device will result in creating an encrypted partition on that device with the specified file system type.  ALL PREVIOUS DATA ON THAT DEVICE WILL BE DESTROYED.  Hence it is not possible, in this release, to encrypt in-place.  We plan to add a utility in a future release to accomplish this however.

    The other thing you’ll notice when you click Format is that it may fail because there is no encryption key.  In this case, click on Settings/Encryption Settings and enter in a passphrase to be used to secure your encrypted devices.  At present we let you enter either a passphrase or upload a file which contains your passphrase (or binary data).  DO NOT FORGET YOUR PASSPHRASE OR LOSE YOUR KEYFILE.  Once a partition is encrypted, if you forget your passphrase or lose your keyfile, your data is forever lost - unless you know someone very high up in the NSA :ph34r:

    Also note that array Autostart following server boot will not succeed if any devices are encrypted.  This is because the keyfile (passphrase) is kept in RAM and thus lost upon reboot.  This means that following system reboot you must log into the webGui, go to Tools/Encryption and enter your passphrase (or upload your keyfile). Yes this is a nuisance and we have a few ideas for automating this, but at least you now have secure https access!

    In the case of a btrfs cache pool, all devices comprising the pool will be encrypted.

    For this release, we highly recommend using encryption only on a test server with test data which has been backed up.  We plan on many more refinements in future releases.

    4Kn Device Support (-rc8q):

    Yeah should work now.

    Other notes (-rc8q):

    • The /usr/local/sbin/emhttp line in your /boot/config/go file is no longer used to specify the ports where the webGui listens for connections.  Instead you must configure these on the Identification page.  Alternately if you need to set this up prior to server boot, you may add the port settings in /boot/config/ident.cfg.  Please refer to /usr/local/sbin/emhttp script for more information if you care about this.
    • It used to be that merely Starting the array would re-write a “unRAID standard partition layout”.  This surprises some users because one would expect nothing to be written to a new device unless Format was invoked.  This has been changed so that nothing is written to a device unless Format is invoked (except for Parity devices – those will still be written upon array Start if parity sync is indicated).
    • Moving devices around between cache pool and array or unassigned is handled much better now.
    • There are numerous webGui fixes and improvements.
    • Upgraded linux kernel and several base packages.
    • Where are releases -rc8a-rc8p you might ask?  Those were non-public test releases.

    Credits (-rc8q):

    • Thanks to @jonp for his work in securing us a Certifiate Authority (Let's Encrypt).
    • Thanks to @eschultz for an incredible amount of work involved in setting up DNS servers and integrating with Let's Encrypt API, among other vital tasks in this release.
    • Thanks to @bonienl for his continued dynamix amazing refinements and networking/IPv6 expertise.

     

    USB Flash boot device backup function (-rc7)

    Added "Flash backup" button on the flash device info page (Main/flash).  Click this button to download a zip file with the entire contents of your USB Flash boot device.  This zip file may be used to restore to a new unRAID USB Flash boot device either manually, or using our nifty new unRAID USB Creator tool.

     

    Linux 4.12 kernel (-rc7) - should provide better Ryzen support among other improvements.

     

    UEFI support (-rc5)

    It is now possible configure UEFI boot mode to boot unRAID OS.  The make_bootable.bat (Windows), make_bootable_mac (MacOS) and make_bootable_linux (Linux) scripts will output a prompt:

     

      Permit UEFI boot mode [Y/N]:

     

    If answered with 'Y' a new directory is included on the USB flash boot device called 'EFI'.  The presence of this directory along with its contents, and along with some additional linux kernel options permit UEFI boot.  This is done in such a way that you could choose either BIOS (legacy) or UEFI to boot off your USB flash device (that is, even if you answer 'Y' here you can still configure your motherboard to use Legacy boot).

     

    If answered with 'N' the directory and contents are still created, but named 'EFI-' (a dash at the end).  This will prevent UEFI firmware from considering this device.  You can manually rename the 'EFI-' directory to 'EFI' and permit possible UEFI boot (and rename back to 'EFI-' to prevent it again).

     

    Note: Even if the 'EFI' directory exists, whether or not your motherboard actually uses UEFI to boot is determined by BIOS settings.  In addition, some motherboards may present a strongly worded warning along the lines of "The system found unauthorized changes on the firmware, operating system or UEFI drivers."  In this case look for a "Secure Boot" BIOS setting and change to "Other OS" or "Disable".

     

    If you update your server using Check for updates on the Plugin page, an 'EFI-' directory and files will be automatically created on your USB flash boot device.  If you prepare a new USB flash using this release, the 'EFI-' directory and files will also be included.  If you use the "manual" method of updating by copying the bz* files from the release zip, beware you will need to manually also copy over the 'EFI-' directory (and modify the first line of syslinux.cfg and copy it to 'EFI-/boot' directory).

     

    There is also a webGui setting to permit UEFI boot located on the 'flash' device information page in the 'Syslinux Configuration' section.

     

    Update OS (-rc5)

    Instead of bundling an "unRAID Server" plugin on the Plugins page, there is a new page on the Tools menu in the About section called 'Update OS'.  Here you can check for a new unRAID OS release as well as switch between the latest release in the stable branch or the next branch.  In addition there is a separate control on the Notification Settings page that configures whether or not to automatically check for updates.


    enabling https (-rc3)

    To enable https support it's necessary to edit your 'config/go' file on your USB flash boot device.  Use the -p option to specify the port(s) and optionally include the -r option to redirect http request from your browser to using https.  Here's the detailed usage:

    # Usage:
    #   emhttp [-r] [-p port [,sslport]] [OPER]
    
    # OPER is start or stop.  Default is start.
    # By default nginx will be setup to listen only at port 80 (http).
    # The -p option may be used to define different listening ports and/or setup nginx
    # to listen at a specified port for https.  The -r option may be used to setup
    # nginx so that any http request is redirected to https (this requires that both
    # ports have been specified with -p option).  For example, to have nginx listen
    # at both standard ports but redirect all http to https use:
    #   emhttp -rp 80,443
    # To listen at only port 443 use:
    #   emhttp -p ,443
    
    # Note: the stop operation is only "safe" if the array has already been stopped
    # (this will be fixed).

    Improved shfs/mover (-rc1)

     

    The LimeTech user share file system (shfs) has been improved in two areas.  First, we now make use of FUSE read_buf/write_buf methods.  This should result in significant throughput increases.  Second, the mover script/move program no longer uses rsync to move files/directories between the cache pool and the parity array.  Instead the move program invokes a new shfs ioctl() call.  This should result in complete preservation of all metadata including atime and mtime.

     

    While this function has been fairly extensively tested, please keep an eye on mover activities - there shouldn't be any data loss, but it's a fairly significant code change.

     

    nginx http server (-rc1)

    We now use the nginx webserver as the front-end to the unRAID OS Management Utility (aka, webGui).  The emhttp process has been changed to a daemon (emhttpd) listening at a unix socket.  Incorporating nginx provides several features:

    • Multi-threaded access, though emhttpd is still single-threaded.
    • https (SSL) support.  At present unRAID OS will generate a self-signed certificate.  https works but you will get a scary warning from your browser about not being able to verify the certificate.  No worries.
    • nchan (websocket) support.  We have only just begun the process of converting many of the browser javascript polling functions to an event-driven websocket paradigm.  This opens the door for  us to create something like a process manager where we can have several background operations in process, all monitored in real-time via webGui dashboard.

     

    IPv6 support (-rc1)

    We want to again, give a big "thank you" to bonienl who has greatly improved unRAID OS networking with the addition of IPv6 support.  Give it a try and report any issues.

     

    Other (-rc1)

    • Two new webGUI themes: Azure and Gray.  Again, thanks to bonienl.
    • Expanded driver support (QLogic) and more hardware monitoring support.
    • Kernel modules and firmware are left on the Flash in a squashfs loopback and loaded into RAM on demand.
    • Many more misc. improvements

    Version 6.4.0_rc21b 2018-01-10 Changes

    Base distro:

    • intel-microcode: version 20180108
    • ttyd: version 20180110 (fixed IE/Edge browser compatibility)

    Linux kernel:

    • version 4.14.13

    Management:

    • mover: bug fix: if 'config/share.cfg' does not exist, cache is enabled by default if present
    • webgui: Correct validation error in UpdateDNS
    • webgui: Anonymize URL user hash
    • webgui: Use -x (extended all) option for generation of SMART reports
    • webgui: VM Manager: add 'virtio-win-0.1.141-1' to VirtIO-ISOs list
    Link to comment
    39 minutes ago, limetech said:

    Intel microcode update.  No one outside Intel knows what these firmware changes do, but probably necessary for upcoming Spectre patches.

     

    Fortuitous timing on adding microcode updates to unRAID :)  thanks!

     

    As a point of interest, for my Xeon E3-1240 v3:

    rc20: microcode updated early to revision 0x22, date = 2017-01-27
    rc21: microcode updated early to revision 0x23, date = 2017-11-20

     

    Link to comment

    Interesting with your printout.

     

    Kernel support Page Table Isolation (PTI): NO

    PTI enabled and active: YES <== how is this possible if kernel doesn't support PTI???

    STATUS: NOT VULNERABLE (PTI mitigates the vulnerability) <== how can a kernel that doesn't support PTI mitigate the vulnerability using PTI???

    Link to comment

    That at least some i3 and i5 processors has received microcode updates might indicate that Intel + M$ are close to releasing some Spectre fixes for normal Windows users.

     

    I would expect Xeon updates when the Linux community gets near to have good Spectre fixes for the server world and/or Microsoft starts to feel ready to roll out Spectre fixes for Server-edition Windows.

     

    The enterprise world will go nuts if server updates rolls out too early and some cloud computing center goes offline because of bugs in the fixes. And the enterprise world has more lawyers.

    Link to comment
    5 minutes ago, HellDiverUK said:

    So, this is odd.  My cache drive reports 5.92GB used, but clearly this is nonsense as the Docker.img is 20GB and is on the cache drive. 

     

    image.thumb.png.2dd923f3a06587645aa41db149f47896.png

     

    What does the usage show in Docker Settings under Docker Volume info?

    Link to comment
    3 minutes ago, johnnie.black said:

    His cache filesystem is xfs, the changes should only make a difference on a btrfs cache filesystem, maybe the docker image is sparse now?

     

     

     

    Oh, didn't see that.

    ls -lash

    The above command should show if it's sparse then.

    Link to comment
    29 minutes ago, HellDiverUK said:

    So, this is odd.  My cache drive reports 5.92GB used, but clearly this is nonsense as the Docker.img is 20GB and is on the cache drive. 

     

    image.thumb.png.2dd923f3a06587645aa41db149f47896.png

    This is normal result for sparse files - most Linux file systems supports sparse files where larger regions of zero values can be "punched out" from the file and not stored on disk.

    Link to comment

    I upgraded to 6.4.0-rc20 from 6.3.5 yesterday. Flawless upgrade. my Windows VM's start, stop, restart perfectly through the webGUI. Upgraded to 6.4.0-rc21b this morning and everything is still working great. Great job to all on the tremendous work that went into 6.4.0!!

     

    One minor thing I noticed (probably a WebGUI problem) is that when I stop a VM, the page does not automatically refresh to reflect the VM's current status. Issuing a start or force stop command through the WebGUI does automatically refresh the page to show the VM's new status.

     

    Attaching diagnostics in case they are of any use

     

    Thanks, Gary

    filesvr-diagnostics-20180111-0800.zip

    Link to comment
    2 hours ago, pwm said:

    This is normal result for sparse files - most Linux file systems supports sparse files where larger regions of zero values can be "punched out" from the file and not stored on disk.

     

    Yet a few hours later, with nothing at all happening on the server apart from a reboot....

     

    image.thumb.png.bd54dbba33b46fc4b2ad2ecc7207009e.png

     

    It'll probably have changed again later.  The inconsistency is bugging my OCD. :)

    Edited by HellDiverUK
    Link to comment
    12 hours ago, limetech said:

    Safari is still broken.  If you use Safari, quit doing that.  Install and use FireFox instead.

    Is this an unRAID issue (with an update coming soon to fix) or an Apple issue (with a fix who knows when)? Safari is my browser of choice mainly because it’s better supported across macOS and iOS devices. I occasionally use Chrome if I absolutely have too. It’s slow to launch but usually works ok after the first tab loads. I gave up on FireFox years ago. Got sick of the uncontrollable pop ups. Do I really need to reinstall it to be able to access the webUI in 6.4 or is there an expectation of a fix for Safari?

    • Like 1
    Link to comment
    1 minute ago, wgstarks said:

    Is this an unRAID issue (with an update coming soon to fix) or an Apple issue (with a fix who knows when)? Safari is my browser of choice mainly because it’s better supported across macOS and iOS devices. I occasionally use Chrome if I absolutely have too. It’s slow to launch but usually works ok after the first tab loads. I gave up on FireFox years ago. Got sick of the uncontrollable pop ups. Do I really need to reinstall it to be able to access the webUI in 6.4 or is there an expectation of a fix for Safari?

     

    Last I checked, Safari is the 2nd most popular browser, no reason to stop using it. 

    Link to comment
    Guest
    This topic is now closed to further replies.