Dav3

This may or may not be relevant: I switched from unraid to ubuntu for various reasons, among which was that I was experiencing this SMB slowdown issue. So this is using the exact same hardware but somewhat different software. I'm experiencing this exact same problem in ubuntu that I had in unraid: In a Windows VM <-> Linux SMB (Samba) all on the same host I see lousy read & write throughput, typically around 15MB/s on hardware that's easily capable of >50MB/s. Typically I see a burst of up to 200MB/s for a few seconds then a crash to 15MB/s. I assume this is write-caching. It may be an entirely different issue, but it does seem to have followed me. I've just lived with the problem; in my case I'm hoping to move to vfio at some point and bypass the whole SMB bottleneck. I don't want to confuse or sidetrack the issue, but to me this seems to suggest it's a more general upstream performance issue and not unraid-specific. Has anyone skimmed the ubuntu/debian buglists for similar performance degradation complaints?

Unfortunately for this and other reasons I also determined that Unraid just wasn't suited for my needs. I moved to Ubuntu with ZFS and use Looking Glass to simplify my GPU-passthru VM setup. Although it's far from perfect, it does everything I need.

I don't really have much to add other than to mention that I'm also having serious performance regressions post-upgrade.

Yeah, I spent about three hours trying to troubleshoot the issue late last night. Results were inconsistent and contradictory so I decided not to bring it up here, just live with it with the hope it resolves or roll-back to 6.7 in in desperation. 🤷‍♂️ Paying for those three hours of lost sleeep today. Hope the gurus figure it out. I'm kinda done losing sleep over the stuff.

Well, I stand by the statement that while switching from cache=off to cache=preferred works as one would expect, switching from cache=off to cache=on does not. It's a corner-case, but an initial sync should be at least offered. I think it's that initial transition that's confusing, not the behavior going forward. Beyond that, a little criticism is probably deserved. It's just been more than a little frustrating spending epic time troubleshooting issues in my attempt to move my development workstation platform off of Vmware Workstation and onto unraid. I had expected to get it done over the Christmas break. It's not a learning curve, it's been a learning El Capitan face-route climb. Far too many times have I been sucked into the linux weeds, fun topics like dumping VGA BIOSes, iptables & go scripts. As it is, I'm in multi-boot hell trying to get work done in the day, 'transitioning' at night & 'breaks' and not enjoying the experience although when the virtualization stuff works it really shows great potential. My wife & kids (& sleep!) have been the ultimate losers. In hindsight, I think my mistake was trying to shoehorn what is (or was) essentially designed to be a media builk-storage NAS product into being a workstation virtualization platform. And my multi-boot 'solution', expecting a short transition period, was woefully mistaken. Nuf said, what I think obviously isn't going to change anything, it's an unproductive religious argument, so on to more productive things...

I see what you're saying, and I did read the forums, but the built-in help often misses my attention. Also the use of 'cache' terminology is a bit of a misnomer, to me cache=yes implies initial synchronization similar to cache=prefer. Like usual I over-thought the issue. As apparently others have. I'd suggest surfacing help a little better, perhaps in a side-bar table element.

I was having the same problem but @Silverbolt's solution fixed it. Thanks @Silverbolt!! +10 UPDATE: Argh, I was wrong. May be a caching thing - I saw normal performance and assumed it was fixed and canceled the copy. And posted the above. Then I restarted the copy and it was normal until the cancel-point, when transfer rate dropped to the 5MB/s I'm seeing. Which sucks. Did anyone ever figure out a fix? Do I need to roll back to 6.6.x?

A note to whoever takes care of an feeds the mover script: I can see in the /usr/local/sbin/mover script that it only looks for 'prefer' not 'yes' to initiate a sync from array to cache.

omg I am stupid... got it thanks.

Oddly, I see 'mount' returning '/mnt/disk1/system/libvirt/libvirt.img on /etc/libvirt type btrfs (rw)' even though no VMs are running... I think this is holding the img file open. Is this correct? Is there some 'VMs are enabled' setting I'm missing? I'm just stopping them.

Ok, I admit I don't understand. Now that I have mover logging enabled, I'm testing simple scenarios and not seeing behavior I expect. Currently all shares are set to cache = no. I set share 'system' to cache = yes and trigger mover (via Main / Move Now button). I see mover start & stop in syslog with no additoinal mover messages. Shouldn't I see mover copying files from /mnt/user/system to /mnt/cache/system? Update, I see that switching from cache = no to cache = prefer does trigger mover copy but setting from no to yes does not. (?)

I agree but in this monopoly-dominated world we live in we need to work with what we get. Until recently I was using charter cable until suddenly without warning AT&T deployed fiber to my area. Yay! Being 40% cheaper & 3x faster, I made the jump. Also 1-Gb + fixed IP address blocks being available, this is everything I always wanted. I was pretty happy. Then after deployment, examining the router (which isn't actually half bad) I realized it was a bit of a Faustian Bargain. However it looks like I can punt the router into 'passthrough' mode, turning it into more of a physical bridge device where I can put my router in front of it and filter LAN traffic away from it. This little task is on my to-do list...

Thanks! I just happened to be reading & considering this post right now. It's very helpful. I'm now seeing the libvirt.img file is being skipped by the mover in the syslog. Yay I can see! Is this 'by design or something wrong? Are there any rules for mover 'skip' decisions besides 'file in use'?

Yeah it sounds like this is the piece of the puzzle I'm missing. I can't find the mover settings... Looking in the usual places, searching the web. Where are mover settings? UPDATE: Found it. under 'Scheduler'... Yep, this should help. Thanks for the tip.

Well, I'm not sure what happened, but after rebooting the server mover now completes instantly. Not sure if this is a good or bad thing, but even though all VMs & docker are stopped I still see /mnt/user/system/libvirt/libvirt.img hasn't been moved to /mnt/cache as expected. Damn, this stuff is opaque. How can I figure out what mover is & is not doing??

I have unraid setup on an 'array' currently consisting of a 4GB HDD (no parity), formatted xfs, currently 139GB is used. I just added an 500GB SSD as a cache drive, also formatted xfs. I stopped VMs & docker and manually started the mover. The cache immediately filled up to 100%. How can this be if I only have 139GB in the array only contains 138GB of data? Note: I do have a VM set up to use a 1TB SSD (configured as a qemu raw drive, not vfio controller passthrough). It has a smallish .img file associated with the SSD that sits in the domain share. Could this be causing the problem, acting as a proxy for the for the entire 1TB SSD and triggering a 1TB copy that fails? Also, I do have the new 'pass-though=true' setting for the passthrough disk in the unassigned disks plugin (@Squid) could this be effecting the issue? If it's the VM's passthrough disk that's causing the problem, what is the recommended best practice for this scenario? Additionally, how does a user gein insight into what the mover is doing? I've seen forum mentions to watch the syslog, but I'm not seeing anything there. I've tried grepping lsio output but is there any better way?

Hi, Can the community comment on the following? So I've finally got unraid set up how I want it, to serve as my new virtualized windows workstation platform. Now I'd like to dial in the storage. Is ZFS a good solution for a workstation scenario? My plan is to use the following hardware: 8700K CPU (rock stable clocked @4.7GHz hosting stress tests in a unraid KVM, btw) ASUS Z570-E Gaming motherboard 64GB RAM Nvidia GTX-1080 GPU Storage --> Purchase a 8 Port SATA III PCI-e x4 Controller Card, probably this: https://www.sybausa.com/index.php?route=product/product&path=64_181_85&product_id=1006&filter=62 here: https://www.amazon.com/dp/B07NFRXQHC/ Re-purpose 8x Seagate ST4000D 'desktop' (slow SMR!) HDDs Re-purpose 1x 1TB Samsung 970-EVO+ NVMe SSD that my old bare-metal system runs from Re-purpose 1x 1TB Samsung 840-EVO SATA SSD Continue using 1x Seagate ST4000MB 'performance' (non SMR) HDD (unraid xfs) Re-purpose 1x 500GB Samsung 860 EVO SATA SSD Plan is to: 1. Plug the new PCIe x4 sata controller into the empty PCIe x16 slot at the bottom of the motherboard (thus allowing the GPU to remain in x16 mode) 2. Put the NVMe SSD into x4 mode (thus disabling two sata ports on motherboard) 3. Plug the 840-EVO SSD into a motherboard SATA port 4. Unraid is installed on ST4000MB HDD plugged into motherboard, leave it there. 5. Plug the 860-EVO SSD into motherboard, configure as unraid cache. 6. Plug the 8x ST4000D drives into the newly purchased Syba (dual Marvell 9215), configure all drives into a zfs zpool. 7. Configure 1TB NVMe drive as zpool read-cache (for performance). 8. Plug the 1TB 840-EVO nto motherboard, configure as write-cache for zpool (since SMR drives are so slow for writes). Note: I'm running my primary GPU-passthrough VM from another SATA SSD (not stubbed, just vfio by-disk device) From what I understand, unraid doesn't support running out of zfs, thus #4 & #5 above. But I intend to move the 'domains' share to the zfs pool, hopefully the .img files & shares I'll be hitting will be mostly be in-cache on the NVMe SSD. (right?) The primary reason I'm seriously looking at zfs is that from what I understand (wongly?) the unraid caching method is by-file, not by-block, but xfs does do block-level caching. I expect that multiple large .img files won't get along together well in the unraid cache but will in a zfs block cache. My expectation is that this setup should give close to ideal performance (for the given hardware), where typical reads should run from the big 1tb & super-fast nvme cache, have a large & fairly resilient (if slow smr disk) storage pool, and writes should be fast (enough) & buffered by the 1TB sata ssd. So no bottlenecks for typical (cached IO pattern) operations. Right??? My general use-case is I do a lot of heavy-duty CPU + GPU + IO, generally a lot of c++ compiles followed by heavy testing (I do 3D app development). If the zfs storage proves to be performant, I'll consider moving my primary OS KVM from it's dedicated virt-io ssd into the pool. Finally, I plan to (intermittently) back up everything to a USB-3 storage enclosure with a bunch of rando-drives running either JBOD or ideally another zfs array. This looks ideal to me since I only need to buy a new 8-port sata adapter and reconfigure a bunch of underutilized hardware I have laying around. The plan from there I plan on upgrading my primary workstation <--> server network to 10GB, then use the zpool to host a bunch of work files while I give my Windows Server 2012R2 box the unraid treatment, then move the files back. Then probably dedicate some chunk of the zpools on workstation & server to an IPFS node living in a docker. Comments, feedback, suggestions welcome. Just not the crickets, I hate the crickets... @steini84? @Squid? @SpaceInvaderOne?

Well, my 'three ideas' feature request post only interested the crickets. So something a little more prosaic: I just spent several hours troubleshooting a windows credentials cache -> unraid samba share permissions issue. It was really hit or miss but a few breadcrumbs would have really helped me resolve the issue faster. I see a lot of share permissions questions in the forums that this might help address as well. A big problem is there was no indication of what was happening in the syslog. I ended up having to drop down to sniffing network traffic with my protocol analyzer box to pinpoint the issue, something us newbs may or may not enjoy spending their saturday on. So, Request: When the samba fileserver service generates a 'permission denied' reply, generate a log message in the syslog of the event, including details on the request username and the share involved. Maybe manage log spamming by throttling, and/or make the logging of such events toggleable via the unraid UI. Ideally 'fix common problems' would catch the issue to help resolve this issue for us noobs. While their saturday morning isn't yet wasted. What would Chode do?? @Squid

Ok so I ended up back-burnering setting up a virtual switch and defining it in net-define XML. Using iptables in /boot/config/go is good enough for now. Spent far too much time on it. So thanks anyway, that lab-network.xml snippet was really helpful and lead me to the virsh command interface which is a goldmine.

Hey I just wanted to say thanks for your help. Just FYI, I did get some weird results when switching container to "host" network; at first it worked but after a server reboot starting a docker container & selecting WebUI actually resulted in opening a VNC session to a running VM (!) I think that had to do with DHCP assignment cruft. I ended up putting all VM's into static IP assignment since that was a to-do anyway. I also ended up putting dockers into "custom" network with static IP addresses. That made accessing dockers from br1 reliable, which is my primary need. I still can't seem to get dockers to be accessible from the LAN (via br0) but as long as I can still access the main unraid web interface externally I'm going to back-burner this task and come back to it.

Automating mounting disk images would be a nice addition to a plugin like Unassigned Devices... @dlandon

Ok thanks dont know how I missed that. eth1 = None in network settings now. iptables matchs yours (set in go file). docker WebUI still doesn't work... Is using 'virsh net-define' xml better than /boot/config/go iptables commands?

Great! I'm trying to figure out how to isolate a vm to a few lan addresses & ports, block everything else to them. Been using unraid /boot/config/go file to tweak iptables of br1/eth1 on 2nd nic. Instead I want to define it using 'virsh net-define'. (better) Would prefer to use virtual interface not eth1. To get filtering it needs NAT type (?) Not clear on xml

Apologies for resurrecting an old thread, but this seems relevant enough to avoid creating a duplicate: @ken-ji I've been reading the libvirt Network XML spec but can't figure out how to so something similar to this. I want to create a private bridge that filters a VM to a few addresses & ports on the LAN. My XML currently looks like: <network > <name>private</name> <bridge name="private0" stp="on" delay="0"/> <forward mode="nat" dev="br1"> <nat> <port start='500' end='1000'/> </nat> </forward> <ip address="192.168.1.202" netmask="255.255.255.0"/> </network> But I get an error that br1 is already taken when I try to virsh net-start the XML file. Can anyone provide clues for how to do this?

Hey @bonienl I think I might have figured it out. And I think may be a corner-case bug having to do with my issue where br1 somehow became the unraid default gateway. I noticed that "iptables -S FORWARD -v" returns "-A FORWARD -d 192.168.1.0/24 -o br1 -j ACCEPT" but not the expected "-A FORWARD -d 192.168.1.0/24 -o br0 -j ACCEPT". (br1 is no longer useful to unraid) Executing "iptables -A FORWARD -d 192.168.1.0/24 -o br0 -j ACCEPT" gave me access to the docker WebUI. Meaning when I removed the IP address from br1, although unraid reset the default gateway, but docker did not. Can you suggest how to fix this? I can add it to the unraid go script but disabling & enabling docker seems to lose the rule. Update. Uh, no. It was working but adding the above iptables statement to the go file and rebooting, its no longer working...