limetech

Administrators
  • Posts

    10011
  • Joined

  • Last visited

  • Days Won

    178

Everything posted by limetech

  1. The primary purpose of this release is to address an issue seen with many HP Microserver Gen8/9 servers (and other platforms) where data corruption could occur if Intel VT-d is enabled. ALL USERS are encouraged to update. As always, please make a flash back up before upgrading: Main/Flash/Flash Backup. While we have not identified the exact kernel commit that introduced this issue, we have identified a solution that involves changing the default IOMMU operational mode in the Linux kernel from "DMA Translation" to "Pass-through" (equivalent to "intel_iommu=pt" kernel option). At first, we thought the 'tg3' network driver was the culprit; however, upon thorough investigation, we think this is coincidental and we have also removed code that "blacklists" the tg3 driver. Special thanks to @JorgeB who helped characterize and report this issue, as well as helping many people recover data when possible. Please refer to the Unraid OS 6.10.3-rc1 announcement post for more information. Version 6.10.3 2022-06-14 Improvements Fixed data corruption issue which could occur on some platforms, notably HP Microserver Gen8/9, when Intel VT-d was enabled. This was fixed by changing the Linux kernel default IOMMU operation mode from "DMA Translation" to "Pass-through". Also removed 'tg3' blacklisting when Intel VT-d was enabled. This was added in an abundance of caution as all early reports of data corruption involved platforms which also (coincidentally) used 'tg3' network driver. If you created a blank 'config/modprobe.d/tg3.conf' file you may remove it. Plugin authors: A plugin file may include a tag which displays a markdown formatted message when a new version is available. Use this to give instructions or warnings to users before the upgrade is done. Brought back color-coding in logging windows. Bug fixes Fix issue detecting Mellanox NIC. Misc. webGUI bug fixes Change Log vs. Unraid OS 6.10.2 Base distro: no changes Linux kernel: version 5.15.46-Unraid CONFIG_IOMMU_DEFAULT_PASSTHROUGH: Passthrough Management: startup: improve network device detection webgui: Added color coding in log files webgui: In case of flash corruption try the test again webgui: Improved syslog reading webgui: Added log size setting when viewing syslog webgui: Plugin manager: add ALERT message function webgui: Add INFO icon to banner webgui: Added translations to PageMap page webgui: Fix: non-correcting parity check actually correcting if non-English language pack installed webgui: Updated azure/gray themes Better support for Firefox Move utilization and notification indicators to the right
  2. limetech

    NVME error

    Changed Status to Retest
  3. limetech

    NVME error

    Also, you are running an old release, please update to Unraid OS 6.10.2 and retest.
  4. limetech

    NVME error

    Please post output of this command: v /dev/disk/by-id Along with diagnostics.zip please
  5. The primary purpose of this release is to address the issue seen with many HP MIcroserver Gen8/9 servers where data corruption could occur if Intel VT-d is enabled. As always, please make a flash back up before upgrading: Main/Flash/Flash Backup. While we have not identified the exact kernel commit that introduced this issue, we believe there is a viable solution. The solution involves changing the default IOMMU operational mode in the Linux kernel from "DMA Translation" to "Pass-through" (equivalent to "intel_iommu=pt" kernel option). At first we thought the 'tg3' network driver was the culprit; however, upon thorough investigation, we think this is coincidental and we have removed code that "blacklists" the tg3 driver. We have decided to publish this release on the Unraid OS next branch so that those users with test servers may give this release a try. To update to this release, navigate to Tools/Update OS and select 'next' under Branch. As soon as we have confirmation from more HP Microserver users that no more "DMAR ERROR" syslog messages are generated, we will publish 6.10.3 stable release. Similarly, since we have effectively changed the intel_iommu mode, we would be interested to know if any VM issues arise - in all our testing there are no issues. More info by @JorgeB a few posts down: Version 6.10.3-rc1 2022-06-10 Improvements Plugin authors: A plugin file may include a tag which displays a markdown formatted message when a new version is available. Use this to give instructions or warnings to users before the upgrade is done. Changed default kernel IOMMU operation mode from "DMA Translation" to "Pass-through". - removed 'tg3' blacklisting Brought back color-coding in logging windows. Bug fixes Fix issue detecting Mellanox NIC. Misc. webGUI bug fixes Change Log vs. Unraid OS 6.10.2 Base distro: no changes Linux kernel: version 5.15.46-Unraid CONFIG_IOMMU_DEFAULT_PASSTHROUGH: Passthrough Management: startup: improve network device detection webgui: Added color coding in log files webgui: In case of flash corruption try the test again webgui: Improved syslog reading webgui: Added log size setting when viewing syslog webgui: Plugin manager: add ALERT message function webgui: Add INFO icon to banner webgui: Added translations to PageMap page webgui: Fix: non-correcting parity check actually correcting if non-English language pack installed webgui: Updated azure/gray themes Better support for Firefox Move utilization and notification indicators to the right
  6. As of yesterday does exactly that, except if that condition exists (tg3 installed + iommu enabled), the update exits.
  7. Do you have all 240 devices hooked up? If so, please capture output of this command: v /dev/dsk/by-id There is no limitation inside Unraid OS for max number of devices, except for: max 30 devices in unRAID array (future feature will permit more than 1 unRAID array) max 30 devices per pool max 35 pools That permits up to 1080 theoretically managed within pools and the array. Additional devices would be managed as Unassigned Devices.
  8. Note: to be safe you should add that to all "boot modes" indicated on Main/Flash page, in case you ever need to switch boot modes. This is an easy thing to forget.
  9. When you click 'Check for Updates" it downloads 'unRAIDServer.plg' file from our download server. When this file is 'executed' and detects tg3 present and iommu enabled it does this: echo "NOTE: combination of NIC using tg3 driver and Intel VT-d enabled may cause DATA CORRUPTION on some platforms." echo "Please disable VT-d in BIOS or pass 'intel_iommu=off' on syslinux kernel append line." echo "Alternaltely create 'config/modprobe.d/tg3.conf' file:" echo " touch /boot/config/modprobe.d/tg3.conf # if your platform is not affected" echo "or" echo " echo 'blacklist tg3' > /boot/config/modprobe.d/tg3.conf # to blacklist the tg3 driver" echo exit 1 The script only checks for existence of modprobe.d/tg3.conf file, not it's content. Hence user can choose to blacklist or not.
  10. AFAIK it's not possible to programmatically disable VT-d. The way the kernel initializes is based on whether VT-d is enabled or not. The current approach was taken in an abundance of caution. Going into a 3-day holiday here in the US I decided it's better for users to lose network connection (which I agree sucks) than to suffer data loss, when we know about possible data loss (that would suck even more). I've just added some code to the downloaded 'unRAIDServer.plg' file that will detect the combination of 'tg3' module loaded and VT-d enabled, and will bail out of the upgrade unless ./config/modprobe.d/tg.conf file exists. This should greatly help those upgrading but new users on affected platform will still see no ethernet. This is going to take us some time to get this fixed; probably will have to go purchase a known-affected platform. The issue is acknowledged here: https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-c04565693 Why this has suddenly happened is a mystery.
  11. Please boot in 'safe mode' and then post diagnostics. This topic from last year might be relevant:
  12. Your Stardate seems off https://www.stoacademy.com/tools/stardate.php
  13. Thanks for the instructions! Added to release notes: https://wiki.unraid.net/Unraid_OS_6.10.2#How_to_Disable_Intel_VT-d_on_HP_MicroServer_Gen_8_with_a_E3-1265LV2
  14. To confirm: which motherboard is this? Does onboard NIC use the 'tg3' driver?
  15. By "change" if you mean code added in this release to blacklist 'tg3' - it does not enable/disable Intel IOMMU. It detects if Intel VT-d is enabled, and if so, auto-blacklists 'tg3' - that' s it. I think the issue you are referring to is unrelated. Can't remember atm, but isn't this issue with older Dell's? Older platforms, such as GEN8 Microservers have buggy bios which does not reset/virtualization correctly and manufacturers have no appetite to go fix.
  16. Issue was introduced at the beginning of 6.10-rc series when we introduced code to require a root password and add brute-force mitigations. Does not have to do with sanitizing input from a form but rather server-side extraction of the record from /etc/shadow file. You can examine the one-line fix here.
  17. This release contains Security fixes, a Data Corruption mitigation, bug fixes and other minor improvements. To upgrade: First create a backup of your USB flash boot device: Main/Flash/Flash Backup If you are running any 6.4 or later release, click 'Check for Updates' on the Tools/Update OS page. If you are running a pre-6.4 release, click 'Check for Updates' on the Plugins page. If the above doesn't work, navigate to Plugins/Install Plugin, select/copy/paste this plugin URL and click Install: https://unraid-dl.sfo2.cdn.digitaloceanspaces.com/stable/unRAIDServer.plg Bugs: If you discover a bug or other issue in this release, please open a Stable Releases Bug Report. Thank you to all Moderators, Community Developers and Community Members for reporting bugs, providing information and posting workarounds. Please remember to make a flash backup! Data Corruption possible with tg3 driver when Intel VT-d is enabled. The combination of Linux 5.15 kernel, tg3 driver, and Intel VT-d enabled appears to be causing data corruption. This has been verified on several platforms which include a Broadcom NetXtreme Gigabit Ethernet NIC (note: there may be others). This release includes the following workaround: Very early in server startup (rc.S) if Intel VT-d is detected enabled, then the script will unconditionally create the file: /etc/modprobe.d/tg3.conf with following content: blacklist tg3 Hence by default if VT-d is enabled, which is to say, it has not been disabled in either bios or via kernel "intel_iommu=off", then we are going to blacklist the tg3 driver on all platforms. What if someone has a platform where tg3 does not give them any trouble with VT-d enabled? In this case they must create an empty file on their flash device: config/modprobe.d/tg3.conf When the startup sequence continues it will get to the point where it executes: install -p -m 0644 /boot/config/modprobe.d/* /etc/modprobe.d A blank tg3.conf file stored on the flash then effectively un-blacklists it. There will be users who will lose network connectivity because their NIC is blacklisted. If you are running on a problematic platform you should go into your bios and disable VT-d. If this is a platform without issue, then you will need to create the blank tg3.conf file on your flash config/modprobe.d directory. It may take some time to identify and integrate a proper fix for this issue, at which point we will remove the auto-blacklisting code. I want to thank @JorgeB for his incredible help in identifying and isolating this issue. Security-related Changes The Firefox browser and has been updated to version 100.0.2 to address a very nasty security vulnerability. If you use Firefox we also suggest upgrading on all platforms. We fixed an issue where webGUI login could accept a password from a user other than 'root', if that username included the string 'root'. The Linux kernel was updated to 5.15.43 to address a "security bypass" vulnerability. Other Changes On Management Access page, for the "Use SSL/TLS" setting we changed the word "Auto" to "Strict" in the drop-down menu. This better describes the action of this setting. Docker manager now uses Docker label for icons as fallback. VM manager now gives the option of using LibVirt networks in addition to bridges without having to edit the VM's XML. Improved handling of custom SSL certificates. [6.10.1] Fix regression: support USB flash boot from other than partition 1 other misc. bug fixes Version 6.10.2 2022-05-27 Base distro: firefox AppImage: version: 100.0.r20220519220738 (CVE-2022-1802 CVE-2022-1529) Linux kernel: version 5.15.43-Unraid (CVE-2022-21499) Management: nginx: avoid appending default port number to redirect URLs nginx: self-signed cert file: accept common name and all alternate names startup: fix multiple network interfaces being assigned the same MAC address startup: blacklist tg3 by default if Intel VT-d is enabled webgui: Management Access: Use SSL/TLS setting: change the word 'Auto' to 'Strict' webgui: Fixed: smGlue not included when selecting a controller webgui: Fixed: allow share names with embedded ampersand webgui: add LXC terminal support (for LXC Plugin) webgui: Docker Web UI to use Docker label for icons as fallback webgui: VM Manager: support libvirt networks (make libvirt networks accessible via gui) webgui: fix issue where 'root' login works with password from another username which includes string 'root' webgui: Update OS page spelling corrction: warninging -> warning webgui: helptext review: minor corrections Version 6.10.1 2022-05-21 Management: startup: fix regression: support USB flash boot from other than partition 1
  18. Agreed, this is almost certainly a plugin issue. Are they all up-to-date? For example: May 20 08:15:14 SUN nginx: 2022/05/20 08:15:14 [error] 7603#7603: *458574 open() "/usr/local/emhttp/main" failed (2: No such file or directory) while sending to client, client: 192.168.1.122, server: , request: "GET /main HTTP/1.1", host: "192.168.1.200" The file "/usr/local/emhttp/main" correctly does not exist. The endpoint "/usr/local/emhttp/Main" does exist.
  19. I think you win the prize for the longest continuously running USB flash device!