Hanfufu

It was the second m.2 NVME drive in the cache pools But ty for the info, will try to grow balls enough to try to update UNRAID 🥴

I had some trouble with a disk suddenly not being recognized, and when it booted up it didnt start the array, and i had the option of selecting the missing disk.

Am on 6.9.2, and yes the pool is redundant

Hey all! I just bought 2 new 1TB NVME SSDs for an existing cache pool in unraid. The old ones are also 1TB each and M.2. Since VMS and dockers are running from that cache pool, i wanna make sure it just works. Is it correct, that I can simply swap 1 of the drives, and on reboot, select the new drive, in the dropdown where it says missing disk. Start the array, let it sync the old drive to the new. Rinse and repeat with drive 2, when the sync is done?

Omg thank you so much, have had this happen multiple times, just made a new VM every time, but this just fixed it!

Hey Jonp, thanks again for your time, its invaluable! My main router is an ASUS RT-AX86U. I have a cable running from that, to a no-name 5 port gigabit switch, which then connects to my server and a wifi extender setup. I just tried to remove the switch as a possible issue, so the cable was running straight from my router to the server. The results are extremely confusing... When removing that switch, the problems still persists, but the download speed was like capped at 30MB/sec - makes absolutely no sence to me. Also tried: using just ethernet 1 (onboard Intel I211 something) using just ethernet 2 (PCIe Intel Gigabit generic adapter) using just ethernet 1 without the switch using just ethernet 2 without the switch using both ethernet simultaneous. used another brand new cable. Still the issue persits. About it being a bad cable/setup, we have to remember that: Downloading to c-drive (vdisk file) in VM runs PERFECTLY @ 60MB/sec - it is only when i save it the the \\tower\media network share from within the VM that the problems show up. Downloading to array through qBittorrent running in Docker, also runs perfectly @ stable 60MB/sec. Nothing has been changed with my setup, going from WS2016 to UNRAID, same cables, setup etc and it always ran perfect. So if this was a bad cable etc, im sure it would have come up as a problem, in the past 2.5 years where my setup (LAN wise) has been exactly the same. But anyway i have now tried different cable etc with no luck, but it was indeed worth a shot. Also as mentioned before, the problem appeared on my old Intel Xeon X99 board, which only had a 3m cable directly into my router, so it is very strange indeed

Sorry for the delay in posting, have been pretty busy the past few days. Sadly its still the same behaviour when testing on that VM, with the other shut down I have however, discovered that the stuttering was misinterpreted by me. it is only RDP/VNC over ethernet that chokes it. VNC via VM Manager in unraid gui does not stutter, it is only the download speeds that are affected. This also makes sence, since there are no CPU usage what so ever, so I am confident that it is NOT a CPU issue. Also i found out, that limiting the amount of open connections pr torrent to 6 Peers only, and only downloading 1 torrent at a time, runs ALOT better. The download still goes way down to sub 5 MBps, but generelly stays in the 50MBps range, with an average download speed of around 35-45MBps. It is NOT a permanent fix, but makes it possible to download at somewhat respectable speeds. Starting 2 downloads at once, makes it choke the LAN completely once again - but as stated without any stuttering per say, it is just network related, and the more open connections there are, the more it chokes.

Thanks for your clarification on the CPU pinning system! I was however considering some misconfig of these settings, but everywhere i found topics related to my problem, there was always 1 core that was maxing out and bottlenecking the VM, but I am not experiencing anything of the sorts. No load of any kind on any of the cores, CPU usage is only a few % with no cores showing full or even high usage. Only 2 VMs are running, the Windows Server 2016 old is running an iis/SQL Server and cannot be shut down atm. And the Windows Server 2016 SATA, is the one i am currently testing on. Linux Mint was only to test if the problem would also occur in a Linux VM, just to cross that off the list. I just tried shutting down all dockers and still only running the 2 VMS i mentioned above. Same result, download spends 95% of the time at around 0kb/sec and VM non responsive via RDP It really baffles me if it is not related to the virtualization network stuff, since downloading to the vdisk file aka c drive inside the VM, runs perfectly smooth @ full speed. It is only, and only if i download/copy to the array/cache from inside the VM, the problems occur.

I just found a tool called ifconfig, and running that shows ALOT of dropped packets

I also saw this, under IPv6 there are lots of discarded packets. I also disabled IPv6 on the VM a while back, to make sure that it didnt cause problems. Also i posted a screenshot of Network Settings, showing something odd with IPv6 route, but im not a network expert so dont know if its relevant.

Holy sh*t that gives a lot of information, didnt know that app, thanks! I dont know what to look for, but i found a lot of packet drops, check the screendump, no idea if its relevant. I will check if i can see anything, if the network error in qbittorrent shows up again soon.

Not sure if it helps, but here are a few screenshots of an error that often happens after a long time of VM being frozen. Also from the eventlog in Windows.

Thanks a lot for the response! I posted a few more screenshots of CPU usage while the VM is frozen, connected via RDP. And as i mentioned earlier, i discovered that connection via UNRAID WEBui VNc (VNC Remote), the stuttering is non existing, but the download speeds are still displaying the same behaviour with the download speed hitting < 1Mbits 98% of the time. tower-diagnostics-20220224-0953.zip

I will post another diagnostics when im at my pc again, with no cpu intensive tasks running

Not sure what you mean, i was not running folding@home or anything, i may have been running some handbrake encoding, but that does not affect anything, as my VMS run fine until there are IO between it and network/user shares on the UNRAID server, no matter how many plex transcodes are running. Shutting down all other VMs/dockers, doesnt affect this behaviour, it still happens.

Update once again: When connecting to the server through UNRAID gui VNC, the stutter does not happen, but the download speeds still jumps up and down, but stays mostly down in the sub 10KB/secs. But as mentioned the server responds fine also when downloads drop to 0. Not sure what this means, but connectiong via RDP to the server, reintroduces the stuttering and freezing. Im seriously getting more and more perplexed by this problem, and am baffled that it can even exist in software that is paid and costs 100$+...

Update: Tried a few other things today, from various other forum posts, none had any effect at all. It seems that a lot of people experiencing this behaviour, have problems with 1 of the CPU cores hitting 100% - This is NOT happening to me. None of the cores on my WS2016 server are doing anything when it stutters, hovers around a few % usage, both in Task Manager in the VM, and on Dashboard on the UNRAID WEBUI. This topic and the fixes in it, did nothing: Also i tried setting up a brand new VM, using both Q35-5.1/5.2 + i440fx-5.1/4.x The new VM still jumps up and down in speed when downloading, HOWEVER it DOES NOT stutter and become unresponsive all the time. Tried setting CPU Scaling Governor to On Demand instead of Power Save, no effect from that either. Tried changing USB to USB 3 (both options), no effect at all. Tried everything i could find here, also did nothing.. Nothing here either:

Anyone....please...??

Does all destination not array ? Nothing goes to the array, i have 2x1TB SSDs as a cache pool, and I can see the files are being created and downloaded onto the array, so this is not the issue. And I was downloading the exact same file in the VM and qbittorrent Ib docker, and to the same path. Docker runs perfect, VMS run terrible. Docker: /user/media/Temp, VM \\tower\media\Temp to make sure that the only difference was VM or Docker. BTW, I think most problem will solve if array out of equation. As stated, it is out of the equation, as the cache pool works fine and as it should. Even downloading (from VM) to the cache-ssd disk share, exchibits the same strange freezing behaviour, and I have read that doing it through disk shares, could be faster than via user share. No difference at all.

So i have spent the past 2 months, ripping out my remaining hair, and pondering to toss my server out the window... I made the unfortunate decision to switch from WS2016 to UNRAID. Very bad decision apparently. I have problems with VMs being completely unusable when downloading/copying files from a VM on UNRAID, to a share on the UNRAID server (disk/user shares both tried, no difference). Quick view of my server: Ryzen 9 3950x, Asrock x570 Steel Legends 48Gigs RAM 2x1TB M2 as cachepool for VMs 2x4TB Sata SSD cachepool for misc files 1x240GB SSD for misc projects. Intel I211 LAN card I attached some screenshots, showing the problems as best as i can. As soon as there is network traffic to any share/cache pools from my VM (Currently running WS2016, but have tried with WS2012, Windows 10 and even a Linux Mint VM), everything starts to lockup and freeze totally, for sometimes several minutes. If i try to navigate anything while downloading, nothing will respond, and clicking a button might take several minutes to actually open. Download speeds occasionally goes up to 60MB/sec (500Mbit connection), but only for a second, then it drops to 0 and the server freezes once again. It is completely unusable and i have tried EVERYTHING i can think of/find online. Nothing works, if anything it just gets worse. If i manage to get it to pause the download, everything immediately runs fine with no hiccups. As soon as i start the download again, the speed goes up to like 30-40MB/sec, then after a few secs, the freezes start once again, and DL speeds drop to 0KB/secs. Most of the time, the machine is unresponsive when the DL speed goes to 0, but at rare occasions it still responds fine, but the DL speeds are just 0, and spikes up to 20MB/secs for a few seconds then down again. Sometimes it locks up completely for a long time (10+ mins), and when i check again, all running downloads has stopped, with a network error: Couldnt write to file, no space on disk or just network error. Copying a file from the VM to a share, results in some short lockups, and the speed goes up and down sometimes stops completely for a few secs - tops at around 50MB/secs and goes down to sub 10MB/sec and then up again, over and over. Copying the SAME file from my gaming PC to the same share, gives a rock solid 113-114MB/secs, all through the transfer, exactly as it is supposed to be. Using WinRAR to unpack files to a share, also makes the VM lockup/freeze constantly. Also it is ONLY the VMs that becomes unresponsive, everything else on the server runs fine, dockers etc while it freezes the VM. Checking CPU usage when it happens, shows its doing pretty much nothing and there is sub 5% load on the cores for the VM. Yes i am using CPU PINNING, so the cores for the VM is only for that. I am also running Plex in Docker, and it runs VERY well with 10+ transcodes with no issues at all, even when using Handbrake to convert x264 -> x265 on 12 cores/24 threads. Running qBittorrent in Docker as a test - rock solid 60MB/secs speeds with no issues at all. I tried running the VMs from all my 3 drive pools - same result, freezes and becomes unusable. Tried changing VM network to virtio or virtio-net - Same results Tried changing everything under Tips and tricks pluging, with no success whatsoever: Disable NIC Flow Ctr - yes/no Disable NIC Offload - yes/no Ethernet NIC Rx/Tx buffer: 256,512.......4096 Also tried to change DirectIO to yes under Global Share settings - does absolutely nothing. So i am 100% certain, that this is an issue between the VM networking and connections to UNRAID shares, as using qBittorrent in docker works perfectly fine, and copying files to/from the unraid shares from physical machines on my network runs totally fine with no issues at all. Another important note here, is that if i download to the C drive on the VM (aka the vdisk file), there are absolutely NO PROBLEMS at all, 55MB/sec solid download, and NO slowdowns. Again, its ONLY when transferring/downloading to/from UNRAID network shares from within the VMs. I am totally stumped, and this issue is making my server completely useless, and i even paid money for that..... I sincerely hope someone has an idea, as i am ready to throw in the towel.... tower-diagnostics-20220219-1115.zip

I just checked all the individual disks, and only on disk1 are there domain and system folder, which have not been changed for more than 3 days, so they should be running from the SSDs

You could be right about that, but i can see the vdisk files etc physically on the SSDs when i view the file system on them, also they are growing in size as i install stuff. Tried downloading from a VM to the c-drive (vdisk), and i could see the writes on the SSD in the Main tab. But i was under the impression that i could keep everything related to VMs solely on the SSDs by selecting Yes: cache_ssd/VMs. How do i make sure that they are kept only on the SSDs then?

There shouldnt be active files on other than the cache_ssd pool, and the WS VM that is problematic, runs off of its own 240GB SSD on a second pool. Or have i set it up incorrectly, according to my screenshot?

On another note, im running a Plex server in Docker, and while all these slowdowns take place, the server responds pretty quickly, and plays almost immediately when transcoding, and does not seem to suffer from slowdowns when transcoding. The Windows Server VM has access to 12 cores and 8GB of RAM så should be plenty for a smooth user experience. Even though there is no transfers ongoing to/from the server, my torrent download speeds will atm not exceed 1MB/sec on my 500Mbit connection, with 0 other LAN traffic interferring

Thanks for the quick answer, and here is the diagnostics ZIP file tower-diagnostics-20211217-1522.zip