limetech

Thanks for the "cheers" - I think I'm going to go fetch a cold one right now myself... If you don't mind, please post diagnostics.zip of your server in state with no plugins, no docker, no kvm. The logs are greatly simplified this way.

Ok but if you boot in safe mode, means no plugins, and also docker and kvm disabled, in this state, what fails?

What do you mean by "Fail" in this case?

Doesn't mean we won't add it

You can plug your USB flash device into a PC and then make a backup (just drag contents to a temp folder on your desktop). Next use the USB Creator Tool to re-install Unraid OS on your flash. You can put the last version you were running, or try 6.7 again. Then copy the contents of your 'config' folder backup to the USB flash 'config' folder. If you had any custom 'syslinux' settings, you could restore that too. Eject flash, reboot server and 'should work' 👍

That will happen when updating from an a 6.7.0-rc but should not happen in updating from 6.6.x. This particular issue will get fixed with the release of 6.8-rc series.

To upgrade: If you are running any 6.4 or later release, click 'Check for Updates' on the Tools/Update OS page. If you are running a pre-6.4 release, click 'Check for Updates' on the Plugins page. If the above doesn't work, navigate to Plugins/Install Plugin, select/copy/paste this plugin URL and click Install: https://s3.amazonaws.com/dnld.lime-technology.com/stable/unRAIDServer.plg Refer also to @ljm42 excellent 6.4 Update Notes which are helpful especially if you are upgrading from a pre-6.4 release. Bugs: If you discover a bug or other issue in this release, please open a Stable Releases Bug Report. New in Unraid OS 6.7 release: New Dashboard layout, along with new Settings and Tools icons. Designed by user @Mex and implemented in collaboration with @bonienl. We think you will find this is a big step forward. Time Machine support via SMB. To enable this feature it is necessary to first turn on "Enhanced OS X interoperability" on the Settings/SMB page. Next, select a share to Export for Time Machine in the share SMB Security Settings section. Note: AFP is now deprecated and macOS users are encouraged to use SMB only. Enhanced syslog handling. On Settings/Network Services page click on Syslog Server. Here you can designate this server to receive system logs from other Unraid OS servers, or forward this servers syslog to another local or remote server. Parity sync/Data rebuild/Check pause/resume capability. Main functionality in place. Pause/resume not preserved across system restarts yet however. Linux kernel 4.19. This is the latest Long Term Support kernel. Here are some other kernel-related updates: Added TCP "BBR Congestion control" and made it the default. This should improve network throughput but probably not too many users will notice anything different. Added Bluetooth support in the Linux kernel. We did not add the user-space tools so this will be mostly useful to support Bluetooth in docker containers. AMD firmware update for Threadripper. Ignore case in validating user share names. If there are multiple top-level directories which differ only in case, then we use the first such share name encountered, checking in order: cache, disk1, disk2, ..., diskN. Additional top-level directories encountered will be ignored. For example, suppose we have: /mnt/cache/ashare /mnt/disk1/Ashare /mnt/disk2/ashare The name of the exported share will be 'ashare' and will consist of a union of /mnt/cache/ashare and /mnt/disk2/ashare. The contents of /mnt/disk1/Ashare will not appear in /mnt/user/ashare. If you then delete the contents of /mnt/user/ashare followed by deleting the 'ashare' share itself, this will result in share 'Ashare' becoming visible. Similar, if you delete the contents of /mnt/cache/ashare (or gets moved), then you will now see share 'Ashare' appear, and it will look like the contents of 'ashare' are missing! Thankfully very few (if any) users should be affected by this, but handles a corner case in both the presentation of shares in windows networking and storage of share config data on the USB flash boot device. New vfio-bind method. Since it appears that the xen-pciback/pciback kernel options no longer work, we introduced an alternate method of binding, by ID, selected PCI devices to the vfio-pci driver. This is accomplished by specifying the PCI ID(s) of devices to bind to vfio-pci in the file 'config/vfio-pci.cfg' on the USB flash boot device. This file should contain a single line that defines the devices: BIND=<device> <device> ... Where <device> is a Domain:Bus:Device.Function string, for example, BIND=02:00.0 Multiple device should be separated with spaces. The script /usr/local/sbin/vfio-pci is called very early in system start-up, right after the USB flash boot device is mounted but before any kernel modules (drivers) have been loaded. The function of the script is to bind each specified device to the vfio-pci driver, which makes them available for assignment to a virtual machine, and also prevents the Linux kernel from automatically binding them to any present host driver. In addition, and importantly, this script will bind not only the specified device(s), but all other devices in the same IOMMU group as well. For example, suppose there is an NVIDIA GPU which defines both a VGA device at 02:00.0 and an audio device at 02.00.1. Specifying a single device (either one) on the BIND line is sufficient to bind both device to vfio-pci. The implication is that either all devices of an IOMMU group are bound to vfio-pci or none of them are. Other highlights: Added the '--allow-discards' option to LUKS open. This should only have any effect when using encrypted Cache device/pool with SSD devices. It allows a file system to notice if underlying device supports TRIM and if so, passes TRIM commands down. Added 'telegram' notification agent support - thank you @realies Updated several base packages, including move to Samba 4.9 and docker 18.09. Fixed a number of minor bugs. Finally: as always, a big "Thank You!" to everyone who contributed and helped with testing. Version 6.7.0 2019-05-08 Base distro: aaa_elflibs: version 15.0 (rev 3) acpid: version 2.0.31 adwaita-icon-theme: version 3.32.0 at-spi2-atk: version 2.32.0 at-spi2-core: version 2.32.1 at: version 3.1.23 atk: version 2.32.0 bash: version 5.0.007 bin: version 11.1 (rev 3) bluez: version 4.101 bridge-utils: version 1.6 btrfs-progs: version v4.19.1 ca-certificates: version 20190308 cairo: version 1.16.0 cifs-utils: version 6.9 coreutils: version 8.31 curl: version 7.64.1 (CVE-2019-8907, CVE-2019-3822, CVE-2019-3823) cyrus-sasl: version 2.1.27 dbus: version 1.12.12 dhcpcd: version 7.2.0 diffutils: version 3.7 dmidecode: version 3.2 dnsmasq: version 2.80 docker: version 18.09.5 (CVE-2019-5736) e2fsprogs: version 1.45.0 etc: version 15.0 (rev 9) ethtool: version 5.0 file: version 5.36 (CVE-2019-8906, CVE-2019-8907) findutils: version 4.6.0 freetype: version 2.10.0 fribidi: version 1.0.5 gdbm: version 1.18.1 gdk-pixbuf2: version 2.38.0 git: version 2.21.0 glib2: version 2.60.1 glibc-solibs: version 2.29 glibc-zoneinfo: version 2019a glibc: version 2.29 gnutls: version 3.6.7 (CVE-2018-16868) gptfdisk: version 1.0.4 graphite2: version 1.3.13 grep: version 3.3 gtk+3: version 3.24.8 gzip: version 1.10 harfbuzz: version 2.4.0 haveged: version 1.9.4 hdparm: version 9.58 hostname: version 3.21 hwloc: version 1.11.11 icu4c: version 64.2 infozip: version 6.0 (CVE-2014-8139, CVE-2014-8140, CVE-2014-8141, CVE-2016-9844, CVE-2018-18384, CVE-2018-1000035) inotify-tools: version 3.20.1 intel-microcode: version 20180807a iproute2: version 5.0.0 iptables: version 1.8.2 iputils: version 20190324 irqbalance: version 1.5.0 jansson: version 2.12 jemalloc: version 4.5.0 jq: version 1.6 (rev2) kernel-firmware: version 20190424_4b6cf2b keyutils: version 1.6 kmod: version 26 libSM: version 1.2.3 libX11: version 1.6.7 libXcomposite: version 0.4.5 libXcursor: version 1.2.0 libXdamage: version 1.1.5 libXdmcp: version 1.1.3 libXext: version 1.3.4 libXft: version 2.3.3 libXmu: version 1.1.3 libXrandr: version 1.5.2 libXxf86dga: version 1.1.5 libaio: version 0.3.112 libarchive: version 3.3.3 libcap-ng: version 0.7.9 libcap: version 2.27 libcroco: version 0.6.13 libdrm: version 2.4.98 libedit: version 20190324_3.1 libepoxy: version 1.5.3 libestr: version 0.1.11 libevdev: version 1.6.0 libgcrypt: version 1.8.4 libgpg-error: version 1.36 libjpeg-turbo: version 2.0.2 libnftnl: version 1.1.2 libpcap: version 1.9.0 libpng: version 1.6.37 (CVE-2018-14048 CVE-2018-14550 CVE-2019-7317) libpsl: version 0.21.0 libpthread-stubs: version 0.4 (rev 3) librsvg: version 2.44.11 libssh2: version 1.8.2 (CVE-2019-3855, CVE-2019-3856, CVE-2019-3857, CVE-2019-3858, CVE-2019-3859, CVE-2019-3860, CVE-2019-3861, CVE-2019-3862, CVE-2019-3863) libtirpc: version 1.1.4 libvirt: version 5.1.0 libwebp: version 1.0.2 libwebsockets: version 3.1.0 libxcb: version 1.13.1 libxkbfile: version 1.1.0 libxml2: version 2.9.9 libxslt: version 1.1.33 libzip: version 1.5.2 lm_sensors: version 3.5.0 logrotate: version 3.15.0 lsscsi: version 0.30 lvm2: version 2.03.02 lz4: version 1.8.3 lzip: version 1.21 mc: version 4.8.22 mcelog: version 162 mesa: version 18.3.0 miniupnpc version: 2.1 mkfontscale: version 1.2.1 mozilla-firefox: version 66.0 (CVE-2018-18500, CVE-2018-18504, CVE-2018-18505, CVE-2018-18503, CVE-2018-18506, CVE-2018-18502, CVE-2018-18501, CVE-2018-18356, CVE-2019-5785, CVE-2018-18511, CVE-2019-9790, CVE-2019-9791, CVE-2019-9792, CVE-2019-9793, CVE-2019-9794, CVE-2019-9795, CVE-2019-9796, CVE-2019-9797, CVE-2019-9798, CVE-2019-9799, CVE-2019-9801, CVE-2019-9802, CVE-2019-9803, CVE-2019-9804, CVE-2019-9805, CVE-2019-9806, CVE-2019-9807, CVE-2019-9809, CVE-2019-9808, CVE-2019-9789, CVE-2019-9788) mpfr: version 4.0.2 nano: version 4.2 ncompress: version 4.2.4.5 ncurses: version 6.1_20190420 netatalk: version 3.1.12 (CVE-2018-1160) nettle: version 3.4.1 (CVE-2018-16869) nghttp2: version 1.38.0 nginx: version 1.14.2 (+ nchan 1.2.3) (CVE-2018-16843, CVE-2018-16844, CVE-2018-16845) ntp: version 4.2.8p13 (CVE-2019-8936) oniguruma: version 6.9.1 (CVE-2017-9224, CVE-2017-9225, CVE-2017-9226, CVE-2017-9227, CVE-2017-9228, CVE-2017-9229) openldap-client: version 2.4.47 openssh: version 8.0p1 openssl-solibs: version 1.1.1b (CVE-2019-1559) openssl: version 1.1.1b (CVE-2019-1559) p11-kit: version 0.23.15 pciutils: version 3.6.2 pcre2: version 10.33 pcre: version 8.43 php: version 7.2.18 (CVE-2019-11034, CVE-2019-11035, CVE-2019-11036) pixman: version 0.38.4 pkgtools: version 15.0 (rev 23) pv: version 1.6.6 qemu: version 3.1.0 (rev 2) patched pcie link speed and width support rpcbind: version 1.2.5 rsyslog: version 8.40.0 samba: version 4.9.7 (CVE-2018-14629, CVE-2018-16841, CVE-2018-16851, CVE-2018-16852, CVE-2018-16853, CVE-2018-16857) sdparm: version 1.10 sed: version 4.7 sg3_utils: version 1.44 shadow: version 4.6 shared-mime-info: version 1.12 smartmontools: version 7.0 spice-protocol: version 0.12.14 spice: version 0.14.1 sqlite: version 3.28.0 sudo: version 1.8.27 sysvinit-scripts: version 2.1 (rev 26) sysvinit: version 2.94 talloc: version 2.2.0 tar: version 1.32 tdb: version 1.4.0 tevent: version 0.10.0 tree: version 1.8.0 ttyd: version 1.4.2 ttyd: version 20190223 util-linux: version 2.33.2 wget: version 1.20.3 (CVE-2019-5953) xauth: version 1.0.10 (rev 3) xfsprogs: version 4.20.0 xkeyboard-config: version 2.25 xprop: version 1.2.4 xterm: version 341 xtrans: version 1.4.0 zstd: version 1.4.0 Linux kernel: version: 4.19.41 added drivers: CONFIG_USB_SERIAL_CH341: USB Winchiphead CH341 Single Port Serial Driver CONFIG_X86_MCELOG_LEGACY: Support for deprecated /dev/mcelog character device added TCP BBR congestion control kernel support and set as default: CONFIG_NET_KEY: PF_KEY sockets CONFIG_TCP_CONG_BBR: BBR TCP CONFIG_NET_SCH_FQ: Fair Queue CONFIG_NET_SCH_FQ_CODEL: Fair Queue Controlled Delay AQM (FQ_CODEL) added Bluetooth kernel support: CONFIG_BT: Bluetooth subsystem support CONFIG_BT_BREDR: Bluetooth Classic (BR/EDR) features CONFIG_BT_RFCOMM: RFCOMM protocol support CONFIG_BT_RFCOMM_TTY: RFCOMM TTY support CONFIG_BT_BNEP: BNEP protocol support CONFIG_BT_BNEP_MC_FILTER: Multicast filter support CONFIG_BT_BNEP_PROTO_FILTER: Protocol filter support CONFIG_BT_HIDP: HIDP protocol support CONFIG_BT_HS: Bluetooth High Speed (HS) features CONFIG_BT_LE: Bluetooth Low Energy (LE) features CONFIG_BT_HCIBTUSB: HCI USB driver CONFIG_BT_HCIBTUSB_AUTOSUSPEND: Enable USB autosuspend for Bluetooth USB devices by default CONFIG_BT_HCIBTUSB_BCM: Broadcom protocol support CONFIG_BT_HCIBTUSB_RTL: Realtek protocol support CONFIG_BT_HCIUART: HCI UART driver CONFIG_BT_HCIUART_H4: UART (H4) protocol support CONFIG_BT_HCIUART_BCSP: BCSP protocol support CONFIG_BT_HCIUART_ATH3K: Atheros AR300x serial support CONFIG_BT_HCIUART_AG6XX: Intel AG6XX protocol support CONFIG_BT_HCIUART_MRVL: Marvell protocol support CONFIG_BT_HCIBCM203X: HCI BCM203x USB driver CONFIG_BT_HCIBPA10X: HCI BPA10x USB driver CONFIG_BT_HCIVHCI: HCI VHCI (Virtual HCI device) driver CONFIG_BT_MRVL: Marvell Bluetooth driver support CONFIG_BT_ATH3K: Atheros firmware download driver firmware: added BCM20702A0-0a5c-21e8.hcd added BCM20702A1-0a5c-21e8.hcd md/unraid: version 2.9.7: setup queue properties correctly support sync pause/resume fix: kernel BUG if read phase of read/modify/write with FUA flag set fails on stripe with multiple read failures OOT Intel 10Gbps network driver: ixgbe: version 5.5.5 OOT Tehuti 10Gbps network driver: tn40xx: version 0.3.6.17 patch: support Mozart 395S chip patch: hpsa: change scsi_host_template.max_sectors from 2048 to 1024 per request Management: add early vfio-bind utility restore PHP E_WARNING in /etc/php/php.ini support Apple Time Machine via SMB acpi: silence undefined ACPI event logging docker: preserve container fixed IPv4 and IPv6 addresses across reboot/docker restart emhttp: bug fix: cache-only/cache-prefer share not initially created on cache emhttp: ignore *.key files that begin with "._" emhttp: properly dismiss "Restarting services" message emhttp: use mkfs.btrfs defaults for metadata and SSD support emhttpd: Add --allow-discard luksOpen option emhttpd: Increase number of queued inotify IN_MOVED_TO events from 16 to 1024 for /var/local/emhttp directory. fix: docker log rotation fix: inconsistent share name case fix: terminal instances limited to 8 (now lifted) fstab: mount USB flash boot device with 'flush' keyword networking: pass user-specified MAC address through to bridge rc.nginx: eliminate unnecessary 10 sec delays rc.nginx: implement better status wait loop - thanks ljm42 rc.sshd: only copy new key files to USB flash boot device smartmontools: update drivedb and hwdata/{pci.ids,usb.ids,oui.txt,manuf.txt} smb: when Enhanced OS X interoperability set, include "fruit:nfs_aces = no" to be compatible with Unraid security model smb: disable samba auto-register with avahi for now webgui: Add GameServers to category for docker containers webgui: Add log-size and log-file options to docker run command webgui: Added new font icons webgui: Added parity pause/resume button webgui: Added syslog server functionality webgui: Allow optional notifications on background docker update checks webgui: Allow plugins to use font awesome for icon webgui: Dashboard: add settings shortcuts webgui: Dashboard: added control buttons webgui: Dashboard: create more space for Dokcer/VM names (3 columns) webgui: Dashboard: cut off long container and VM names webgui: Dashboard: fix color consistency webgui: Dashboard: fix incorrect memory type webgui: Dashboard: fixed display of Wattage in UPS load webgui: Dashboard: fixed hanging when no share exports are defined webgui: Dashboard: fixed wrapping of long lines webgui: Dashboard: fixed wrong display of memory size webgui: Dashboard: include links to settings webgui: Dashboard: replace inline style statements for style section webgui: Dashboard: table adjustment in three columns view webgui: Dashboard: table right adjustment in two columns view webgui: Dashboard: use disk thresholds for utlization bars webgui: Dashboard: wrap long descriptions webgui: Diagnostics: dynamic file name creation webgui: Do not capitalize path names in title of themes Azure and Gray webgui: Docker: single column for CPU/Memory load webgui: Docker: Add More Info link (docker registry) to context menus webgui: Docker: textual update webgui: Docker: usage memory usage in advanced view webgui: Escape quotes on a containers template webgui: File browser: force download of files webgui: Fix Background color when installing container webgui: Fixed share/disk size calculation when names include space webgui: Fixed version display in system information webgui: Fixed: slots selection always disabled after "New Config" webgui: Keep status visible for paused array operations webgui: Main: make disk identification mono-spaced font webgui: Minor textual changes webgui: Move "Management Access" directly under Settings webgui: New icon reference webgui: OS update: style correction webgui: Open link under Unraid logo in new window webgui: Per Device Font Size Setting webgui: Permit configuration of parity device(s) spinup group. webgui: Plugin manager: add .png option to Icon tag webgui: Plugin manager: align icon size with rest of the GUI webgui: Plugin manager: enlarge readmore height webgui: Plugin manager: table style update webgui: Position context menu always left + below icon webgui: Prevent update notification if plugin is not compatible webgui: Replace string "OS X" with "macOS" webgui: Replaced orb png icons by font-awesome webgui: Revamped dashboard page webgui: Share settings: fixed exclude "All" from write function webgui: Suppress PHP warnings from corrupted XML files webgui: Switch button: use blue color in ON state webgui: Switch plugins to a compressed download webgui: Syslinux config: replace checkbox with radio button webgui: Syslog: add '' entry in local folder selection webgui: Syslog: added log rotation settings webgui: Syslog: added viewer webgui: Syslog: included rsyslog.d conf files and chmod 0666 webgui: Syslog: sort logs webgui: Updated Unraid icons webgui: Updated icons and cases webgui: Updated jquery cookie script from 1.3.1 to 1.4.1 webgui: Use cookie for display setting font size webgui: VM manager: remove and rebuild USB controllers webgui: VM page: allow long VM names webgui: added new case icons webgui: other GUI enhancements webgui: prevent dashboard bar animations from queuing up on inactive browser tab webgui: sort notification agents alphabetically, add telegram notifications webgui: syslog icon update webgui: telegram notification agent bug fixes

Something else to try: create the file /boot/config/vfio-bind with a line that specifies the PCI ID of your nvme device and then reboot for it to take effect. Take a look at /usr/local/sbin/vfio-pci script for more details. If you post diags I can tell you the exact string to use but you can prob figure it out. This might not make any difference but would be good to know that.

This patch has been removed in 6.7.0-rc8 and a different workaround is recommended: https://bugzilla.kernel.org/show_bug.cgi?id=202055#c42

Probably updating to 6.6.7 failed somehow. I suggest you plug usb flash into PC, make a backup, reinstall using Flash Creator Tool and then restore your 'config' directory contents from your backup.

On your new rig, once it gets to the Login prompt, login (as root user) and type the 'df' command, should look something like this: root@Test1:~# df Filesystem 1K-blocks Used Available Use% Mounted on rootfs 1970548 530088 1440460 27% / tmpfs 32768 224 32544 1% /run devtmpfs 1970564 0 1970564 0% /dev tmpfs 2021920 0 2021920 0% /dev/shm cgroup_root 8192 0 8192 0% /sys/fs/cgroup tmpfs 131072 172 130900 1% /var/log /dev/sda1 3781076 212928 3568148 6% /boot /dev/loop0 8832 8832 0 100% /lib/modules /dev/loop1 6016 6016 0 100% /lib/firmware In particular /lib/modules and /lib/firmware should be mounted on loop devices.

Sounds like a flash corruption problem.

Reformat the flash device and reinstall your flash backup zip.

..for so long it's not true...

No other changes.

To upgrade: If you are running any 6.4 or later release, click 'Check for Updates' on the Tools/Update OS page. If you are running a pre-6.4 release, click 'Check for Updates' on the Plugins page. If the above doesn't work, navigate to Plugins/Install Plugin, select/copy/paste this plugin URL and click Install: https://s3.amazonaws.com/dnld.lime-technology.com/stable/unRAIDServer.plg Refer also to @ljm42 excellent 6.4 Update Notes which are helpful especially if you are upgrading from a pre-6.4 release. BIOS: Especially if using Virtual Machines, we highly recommend keeping your motherboard bios up-to-date. Bugs: If you discover a bug or other issue new to this release, please open a Stable Releases Bug Report. This is a security release that updates docker from version 18.06.1 to 18.06.3. Version 6.6.7 2019-02-22 Base distro: docker: version 18.06.3 (CVE-2019-5736)

Thanks for the report!

Right, our testing didn't show much speed difference but maybe not configuring properly...

qemu 3.1.0 with aforementioned patch will be available starting with 6.7.0-rc4.