• Unraid OS version 6.9.0-beta30 available


    limetech

    Changes vs. 6.9.0-beta29 include:

     

    Added workaround for mpt3sas not recognizing devices with certain LSI chipsets. We created this file:

    /etc/modprobe.d/mpt3sas-workaround.conf

    which contains this line:

    options mpt3sas max_queue_depth=10000

    When the mpt3sas module is loaded at boot, that option will be specified.  If you add "mpt3sas.max_queue_depth=10000" to syslinux kernel append line, you can remove it.  Likewise, if you manually load the module via 'go' file, can also remove it.  When/if the mpt3sas maintainer fixes the core issue in the driver we'll get rid of this workaround.

     

    Reverted libvirt to v6.5.0 in order to restore storage device passthrough to VM's.

     

    A handful of other bug fixes, including 'unblacklisting' the ast driver (Aspeed GPU driver).  For those using that on-board graphics chips, primarily Supermicro, this should increase speed and resolution of local console webGUI. 

     


     

    Version 6.9.0-beta30 2020-10-05 (vs -beta29)

    Base distro:

    • libvirt: version 6.5.0 [revert from version 6.6.0]
    • php: version 7.4.11 (CVE-2020-7070, CVE-2020-7069)

    Linux kernel:

    • version 5.8.13
    • ast: removed blacklisting from /etc/modprobe.d
    • mpt3sas: added /etc/modprobe.d/mpt3sas-workaround.conf to set "max_queue_depth=10000"

    Management:

    • at: suppress session open/close syslog messages
    • emhttpd: correct 'Erase' logic for unRAID array devices
    • emhtppd: wipefs encrypted device removed from multi-device pool
    • emhttpd: yet another btrfs 'free/used' calculation method
    • webGUI: Update statuscheck
    • webGUI: Fix dockerupdate.php warnings

     

    • Like 5
    • Thanks 5


    User Feedback

    Recommended Comments



    and it looks now its completely stalled ... cant even reboot via ssh, so i have to hard reboot when im at home

     

    seems docker service is blocking from reboot

     

    also stalls when i do

    /etc/rc.d/rc.docker stop

    so i guess docker service is borked ;) i can tell when im at home, if you have anymore ideas what todo via ssh ... i ll try

    Edited by alturismo
    Link to comment
    Share on other sites
    18 minutes ago, alturismo said:

    and it looks now its completely stalled ... cant even reboot via ssh, so i have to hard reboot when im at home

     

    seems docker service is blocking from reboot

     

    also stalls when i do

    
    /etc/rc.d/rc.docker stop

    so i guess docker service is borked ;) i can tell when im at home, if you have anymore ideas what todo via ssh ... i ll try

    It's because of open file handles on the mounts i think.

    If you crazy you could manually unmount all array, cache or docker image mountpoints. But if the WebGui is still responsive click on stop array before you do this...

     

    umount -f -l /path

    While the -f is to force, the -l is the lazy option, which doesn't allow new handles on that mountpoint.

     

    And start with the docker image first...

     

    Worked for me several times, because of stalled docker service which won't close, even if you send sigseg or sigkill...whatever.

     

    But keep in mind, this is only a workaround with potantally dataloss!

    Edited by DarkMan83
    Link to comment
    Share on other sites

    tried it all, stopped libvirt service fine, just docker service is ...  ;)

     

    /sbin/reboot ... lets see what happens

     

    and thanks for the tipp with unmount ... you prolly right ;) but was too fast now

    Edited by alturismo
    Link to comment
    Share on other sites

    ok, server did reboot, but webgui is still gone ...

     

    oot@AlsServer:~# uptime
     17:20:18 up 2 min,  1 user,  load average: 8.85, 4.20, 1.58
    root@AlsServer:~# 
     

    Link to comment
    Share on other sites
    36 minutes ago, alturismo said:

    cat didnt work out, always "no such file .." on target

    There's a typo in that command, a missing ">" 

    Try this:

    cat /var/log/syslog > /boot/syslog.txt

     

    Link to comment
    Share on other sites

    attached, syslog before and after reboot now

     

     

    also, i can only ssh from externally now via port forwarding, internal LAN 192.168.1.2 ... like not listening ...

    dockers mostly running, not all ... the ones which are related to mysql which is running on host which is not available anymore local ...

     

    VM's looking good

     

    very very weird

    syslog.txt

    i hope someone can help ;)

     

    i also added the network.cfg file, but looking good to me ... 

    syslog_rebooted.txt network.cfg

    Edited by alturismo
    Link to comment
    Share on other sites

    so overall, looks more like a network issue due my srver ip address is not available at all,
    as i have almost all dockers on custom br:0 with their own ip's, thats why they are mostly working ...

     

    so, somehow local binding is not working in network ... like described

     

    ssh to 192.168.1.2  <<-- doesnt work

    ssh to ssh.mydomain.de (port 22 forwarded to 192.168.1.2) <<-- working

     

    from the same VM in LAN running on unraid ;)

    Edited by alturismo
    Link to comment
    Share on other sites

    as i am at home now i made another hard reboot, power off / on.

     

    boot sequence etc looking all good, i just cant reach my services anymore related to server ip 192.168.1.2 at all,

    all services besides that are working fine.

     

    Shares (not reachable)

    dockers running on host in bridge mode like mariadb (not reachable)

    webgui not reachable

    VNC to a VM using unraid vnc the same, not reachable

     

    and here also, i can forward external port to unraid server and webgui is there ...

     

    so, i can proceed any local connection to the host server (unraid), only via external connections ...

    tested from local VM running on top of unraid, also tested from laptop @home ...

     

    may a hint what could be broken here ? no hardware changes, no changes on network config, ...

     

    image.thumb.png.bcd4281d0ec132f6289d4a95833a4bbd.png

    image.thumb.png.679e0f0701b9f293ed6ab2bf1d2ef2ac.png

     

    Edited by alturismo
    Link to comment
    Share on other sites
    3 hours ago, John_M said:

    There's a typo in that command, a missing ">" 

    Try this:

    
    cat /var/log/syslog > /boot/syslog.txt

     

    Oops right, meant to have:

    cp /var/log/syslog /boot/syslog.txt

     

    Cats were on my mind I guess ;)

    • Haha 1
    Link to comment
    Share on other sites
    20 minutes ago, alturismo said:

    may a hint what could be broken here ? no hardware changes, no changes on network config, ...

    Too much going on in there.  You need to simplify the config and see what makes it fail.  Start by booting in safe mode.

    Link to comment
    Share on other sites

    well, cant imagine its a config thing as it stopped working while running ...

     

    i mean its all working besides the system (unraid host on 192.168.1.2) is not accepting connections from LAN, ONLY from external ... which is a bit weird to me ...

     

    also all other connections in the bridge are working, all dockers now (i changed the mysql docker to its own ip) and now the dependant dockers like nextcloud and guacamole are also up and running, all VM's are fine, all connections in LAN from VM's, Dockers, other devices are all fine ...

     

    so like a firewall blocking unraid host from internal access, no http, no smb, no ... 

     

    so, when i boot in safe mode, what should i look for ? debug what ? look for ? or just check if system is then reachable from LAN ?

    Link to comment
    Share on other sites

    ok, 1st booted in safe mode GUI, no plugins loaded.

    -> system is reachable again in LAN, VM and Docker also working

    just no plugins loaded ...

     

    now, changed my append on regular boot, removed isolcpu and stubs as they where managed through unraid meanwhile anyway, rebooted

    -> all working again

    just the isolcpu was gone, probably due i setted it up through appnd only

     

    changed now isolcpu in GUI to get back as before, rebooted

    -> looking all good again now

     

    closing now my external ports again for webgui and ssh usage ;)

     

    but honestly, i cant understand why its working again like this ...

     

    this was my append line before (like the last 1 - 2 years ...)

    append pci-stub.ids=10de:1b81,10de:10f0,10de:1d01,10de:0fb8,8086:a2af,1106:3483 pcie_acs_override=downstream isolcpus=2,3,4,5,8,9,10,11 kvm-intel.nested=1 initrd=/bzroot mitigations=off

    Link to comment
    Share on other sites

    Hello,

    I upgraded to beta 30 from 6.8.3 when it became available.

    In 6.8.3 i had the ssd write problem which killed 2 brand new ssd in 9 months.

    I replaced the pool  with a single xfs formatted ssd.

    Yesterday i decided to setup a new pool with 2 brand new ssd (western blue 500Gb).

    I transfered all data from the cache (appdata, domains and system share, no other data) to the new pool (around 310Gb). It was 15 hours ago.

    Now here's a screenshot of smart data.

    with iotop -ao i can see that each container startup write almost instantanously 4G on loop2 i have around 35 containers running. All are stopped each night for backup of appdata.

     

      TID  PRIO  USER     DISK READ DISK WRITE>  SWAPIN      IO    COMMAND                                                            
    25403 be/0 root          4.71 M   1263.98 M  0.00 %  1.00 % [loop2]
    25946 be/4 root          0.00 B    111.59 M  0.00 %  0.09 % qemu-system-x86_64 -name guest=Her~rol=deny -msg timestamp=on [worker]
    17588 be/4 root          0.00 B     90.50 M  0.00 %  0.07 % qemu-system-x86_64 -name guest=Her~rol=deny -msg timestamp=on [worker]
     2795 be/4 root          0.00 B     87.25 M  0.00 %  0.00 % [kworker/u65:7-btrfs-endio-write]

    above is the result of iotop -ao after 10 min (containers are already started).

    Of course the new pool is formatted in btrfs with 1Mib aligned partition.

     

    I thought the write problem was solved in beta 25. Do i miss something obvious?

     

    edit: i decided to dig a bit more.  I looked the ssd which was used as single xfs formatted cache drive.

    I set it up around the end of may until yesterday. I had 2 identicals ssd i used one for cache and keep the other.

    They wera around 41728932997 lba written (19TB with 512 byte sector)

    Today the ssd i used as single cache drive is at 107379774578 lba written (50TB)

    So i guess during 5 months unraid has written 31TB. It's around 200GB/day.

     

     

     

    2020-10-16 09_01_49-godzilla_Device.png

    Edited by caplam
    Link to comment
    Share on other sites

    Do you have any docker containers that if you look at the Docker tab and switch on the Advanced view show they are ‘healthy’?    It has been noticed that such containers have been set up by their authors to write to the docker image every few seconds as part of a health check, and although the writes are small the write amplification inherent in using BTRFS means this can add up.   I believe that an issue has been raised against docker in case this is a bug in the Docker engine rather than the container authors simply mis-using the health check capability.

     

    In addition there are other options available in the Docker settings that can reduce this load such as using an XFS formatted image instead of BTRFS, or not using an image at all and storing directly into the file system.  The array has to be stopped to see these options.    Have you tried any of these?

    Link to comment
    Share on other sites

    i've checked my containers against the "healthy" mention. No one have that.

    I haven't read anything about the format of docker image. I thought btrfs was mandatory. I assume it has nothing to do with the pool format which is also btrfs.

    You can store image directly on file system ? Does this mean you don't have a loop2 device anymore ?

    I didn't see such options.

     

    edit i missed that option (docker image) as it was explained in beta 25 release post. I only installed beta starting with 30.

    I think i will give a try to the directory option.

    I have one question though. In the release post squid writes that if choosing directory option it's better to make a dedicated share but in his example (and default path when choosing this option) the directory is /mnt/cache/system/docker/docker which is in the system share.

    I guess if i create a docker share the path is /mnt/cache/docker and it's ok?

    Edited by caplam
    Link to comment
    Share on other sites
    On 10/14/2020 at 4:40 PM, atconc said:

    Is there anything else I can try to troubleshoot this (issue with very slow apps in docker and high SHFS cpu use)? or if i roll back to beta 25 where I didn't have this issue will the new partition layout be recognized or will I have to rebuild my cache again?

    Edited by atconc
    clarify context as i didn't quote the original post
    Link to comment
    Share on other sites
    45 minutes ago, atconc said:

    if i roll back to beta 25 where I didn't have this issue will the new partition layout be recognized or will I have to rebuild my cache again?

    It will be recognized, -beta25 is where the new alignment was introduced, curiously just recently found out that the new alignment works even in previous releases, but only for pools, though you can still have a single device btrfs "pool".

     

     

    Link to comment
    Share on other sites

     

    On 10/16/2020 at 12:10 AM, caplam said:

    I transfered all data from the cache (appdata, domains and system share, no other data) to the new pool (around 310Gb). It was 15 hours ago.

    How did you do the transfer?  It's possible the "NoCOW" bit is not set for disk images, that is, the docker.img and all your VM vdisk images.  If this is the case, all writes to these image files will result in btrfs copy-on-write operations.  Since btrfs uses 1GiB block-group size, if say a docker container writes a single 4K block that will result in btrfs actually writing a 1GiB block (copy block group which contains the 4K block, update the 4K block, write the entire 1GiB block group to a new location).  This is why we turn on NoCOW bit when images are created via webGUI.  But if you simply used 'cp' command or similar, the target file will probably not have NoCOW bit set unless the directory you're copying to has NoCOW set.  You can use the 'lsattr' command to see if NoCOW bit is set for a file, for example:

    # here is C bit (nocow) set on directory
    root@test-1:/mnt/images/domains# lsattr
    ---------------C---- ./win10-1
    ---------------C---- ./ubuntu
    
    # here is C bit set on file
    root@test-1:/mnt/images/domains# lsattr win10-1/
    ---------------C---- win10-1/vdisk1.img

    Note also you can create different pools, for example, to separate VM vdisk images from the Docker image.

     

    The downside with having NoCOW set is that btrfs turns off checksumming for the file (but contrary to misinformation out there, you can still create reflink copies of NoCOW files and snapshots of subvolumes with NoCOW files).

    • Thanks 1
    Link to comment
    Share on other sites

    To clarify a bit: if you want to copy say, "vdisk1.img" from one share to another you need to first set the NoCOW bit on the directory which will receive the file.  For example:

    # /mnt/backup/domains/win10/vdisk1.img - is backup of img file
    # /mnt/cache/domains - is where we want to copy the vdisk.img file
    mkdir /mnt/cache/domains/win10
    chattr +C /mnt/cache/domains/win10
    cp --sparse=always /mnt/backup/domains/win10/vdisk1.img /mnt/cache/domains/win10  # --sparse-always is optional

    By virtue of C bit set on /mnt/cache/domains/win10 all files henceforth copied there will have the C bit set also.

    Link to comment
    Share on other sites

    Also for completeness: You cannot turn on the C bit for an existing file.  Well you can turn it on but btrfs will not now revert to NoCOW for the file.  It must be turned on at the time the file is created before any data is written to it.  The easiest way to accomplish this is to set the C bit on the parent directory.  Though if you look at /usr/local/sbin/mount_image you will see what we do is create a zero-length file, set the C bit, then use fallocate to extend to the requested size.

    Link to comment
    Share on other sites
    On 10/14/2020 at 9:57 AM, JorgeB said:

    I’ve been ruminating on this SAMBA aio issue because the very large read performance difference first reported by @trypowercyclereminded me of an issue I’ve seen before, but I was having trouble finding that post, now I know why, because those forums are gone? I did finally find it in my content:

     

    39923698_rc4post.thumb.PNG.8e301f7fc3363825a3b8852af79b6e15.PNG

     

    And this is the comparison I posted at the time:

    b21_rc4_xfs.thumb.png.b1fbc793d562bedd1542f4e2e140f5c7.png

     

    So I believe I noticed this issue at around the same time aio was introduced in Samba, and at the time disabling smb3 fixed it, now I wonder if it was already the same issue and disabling smb3 was also disabling aio, symptoms are very similar, and the problem wasn’t controller related but device related, some brands/models perform worse than others, so I now did some more tests with –beta30 and different disks.

     

    Ignore the normal max speed difference from brand to brand, I used whatever disks I had at hand, so some disks are older and slower than others, the important part is the aio on/off difference, tested with disk shares so no shfs interference, all connected to the same Intel SATA controller, each test was repeated 3 times to make sure results are consistent, read speed reported by robocopy after transferring the same large file.

     

    imagem.png.a17b2e645dec94d2860f75258adf6811.png

     

     

    I think the results are very clear, and by luck (or bad luck) the tests this past weekend were done with the only disk that now doesn’t show a significant difference, note that I don’t think this is a disk brand issue, but a disk model issue, likely firmware related, possibly worse in older disks?

     

    I know you already plan to leave aio disable, but still one more data point that I believe really confirms it should be left disable.

     

     

    On 10/14/2020 at 4:40 PM, atconc said:

     

    5 hours ago, JorgeB said:

    It will be recognized, -beta25 is where the new alignment was introduced, curiously just recently found out that the new alignment works even in previous releases, but only for pools, though you can still have a single device btrfs "pool".

     

     

     

    I rolled back to beta 25 and the apps in docker containers are noticeably significantly more responsive again so there's definitely a regression in beta29 and beta30 for me.  Happy to help troubleshoot, let me know if there's anything I can do.

    Link to comment
    Share on other sites
    4 hours ago, limetech said:

    if say a docker container writes a single 4K block that will result in btrfs actually writing a 1GiB block

    Further clarification: it's not necessarily that bad because btrfs will gather other such writes into the newly allocated block group, but depending on I/O it can end up writing more due to metadata updates as well.

    Link to comment
    Share on other sites
    13 hours ago, limetech said:

     

    How did you do the transfer?  It's possible the "NoCOW" bit is not set for disk images, that is, the docker.img and all your VM vdisk images.  If this is the case, all writes to these image files will result in btrfs copy-on-write operations.  Since btrfs uses 1GiB block-group size, if say a docker container writes a single 4K block that will result in btrfs actually writing a 1GiB block (copy block group which contains the 4K block, update the 4K block, write the entire 1GiB block group to a new location).  This is why we turn on NoCOW bit when images are created via webGUI.  But if you simply used 'cp' command or similar, the target file will probably not have NoCOW bit set unless the directory you're copying to has NoCOW set.  You can use the 'lsattr' command to see if NoCOW bit is set for a file, for example:

    
    # here is C bit (nocow) set on directory
    root@test-1:/mnt/images/domains# lsattr
    ---------------C---- ./win10-1
    ---------------C---- ./ubuntu
    
    # here is C bit set on file
    root@test-1:/mnt/images/domains# lsattr win10-1/
    ---------------C---- win10-1/vdisk1.img

    Note also you can create different pools, for example, to separate VM vdisk images from the Docker image.

     

    The downside with having NoCOW set is that btrfs turns off checksumming for the file (but contrary to misinformation out there, you can still create reflink copies of NoCOW files and snapshots of subvolumes with NoCOW files).

    i transferred it with cp command (docker and vm service stopped). 

    I checked and the no cow bit is not set for directory and img files. Share have enable COW set to auto.

    But what you explain is for the initial transfer. How does it act once the transfer is completed and services restarted ?

    I dropped the usage of docker.img. I'm now using directory option with dedicated share.

    I must admit i don't fully understand COW things and let the option on auto in shares settings.

    Until now i hadn't had to think about it. When i was using proxmox my vdisk were qcow2 and i could take snaphots.

     

    For now i decided to use 2 single ssd pools. Right now i unassigned one ssd and restart array (balance is running).

    I think the right path is then to convert to single pool. After that i will create the second pool and assign the second ssd, format it to xfs and transfer all data to it. The do the same thing for the first ssd (formatting xfs) and retransfer back the data need.

     

    I'll wait to see the future of zfs and unraid (interesting for me as i have plenty of ram ecc); but for sure i won't use anymore unraid with btrfs.

    Link to comment
    Share on other sites
    6 hours ago, caplam said:

    But what you explain is for the initial transfer. How does it act once the transfer is completed and services restarted ?

    Depending on timing maybe you were seeing the results of a 'sync' to write that blocks of your copy to storage?

    Are most of the writes during normal operations (where you say 200GB/day) originating from docker containers or your VM's?

     

    Also, this discussion really should be a bug report because I will lock this topic with next release and not refer back to it.

    Link to comment
    Share on other sites

    Is there any chance that a task can be added to the scheduler options to run a scrub on BTRFS pools?

     

     

    Link to comment
    Share on other sites



    Guest
    This is now closed for further comments

  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.