Marshalleq

Of course! So not that then. The speed test came out OK. Also @johnnie.black I'd suggest that an impact to performance that brings a systems to it's knees in the main area it is designed for should not be categorised as minor. Perhaps we should increase the ticket rating which may also get more visibility?

Yes, simultaneous write session while multiple read sessions to spinning disk is what I did. There was multiple plex sessions ongoing while doing a large multi-terabyte copy from SSD cache to the array. But I could try be a little harder on it and try again with even more writes and reads. Regarding the speed, my reading on this forum indicated that 53MB/s was fairly normal for writing with Parity. If it's not, the only thing I can think of it being is a faulty cable, but I have run speed tests on all my drives and they perform at their rated speed individually - so I don't think it's that. I'm doing another speed test now to make sure nothing hasn't gone wrong. I'd be interested in knowing what your configuration is. my drives are mainly on a Dell PERC310 in IT mode, which seems to have more than enough bandwidth for the job, but perhaps it's that. Edit: Quick calculation: The Dell Perc H310 supports 8 drives and runs on the PCI Express 2.0 bus. PCI Express 2.0 supports 500MB/s. So dividing by 8 means each drive would get a maximum of 62.5MB/s. This could be the reason why I guess. Individual drive speed tests wouldn't be restricted by the bus speed, so that would be why I hadn't seen the issue. I also assume read would not be impacted as I don't think read needs to calculate across all drives. Perhaps I should look into reconstruct write mode again.

Right, so initial testing with the release candidate 6.7.3-rc2. To test I played a plex video, preloaded many gigabytes of files to a cached share. Invoked the mover manually, added an additional copy of another large set of files from a disk to the cache and with all this going simultaneously I don't seem to have any issues. The wa (io wait) in top only get's up to 8.0 instead of 20.0 under the previous kernel. (Gotta edit that as I think I wrote 0.20 which was incorrect, it was 20. Also my write speed (from HDD to SSD) is about normal at 53MB/s - yes it's slow, it's always been slow, even with Seagate enterprise capacity disks - seems to be an overhead of the unraid parity. This is my first and only test so far (I'll try tomorrow when someone is watching plex on the Apple TV in the lounge where the issue was visible today) I'd be interested if anyone else can test though, by upgrading to 'next' - there are very very few changes in it so it should be quite safe. If the problem goes away for you I'd say very lucky we are. Otherwise we shall need to investigate further. Fingers crossed!

Just updating this that today I installed this RC again and this time my GPU passthrough is working fine.

That’s why I upgraded to the release candidate of the next version. Probably my post is a bit confusing cause at the beginning of it I wasn’t. But I try not to post too many messages in a row.

So I'm now back on 6.7.2 and already I'm seeing issues again. Specifically, while playing something on Plex, the mover process created a repeating image freeze / resume scenario on the client. I had the opportunity therefore to look at top and saw the wa (I/O wait) reach approximately 0.20 vs the idle wait of 0.03. While this may be indicative of the issue below, I'm looking deeper into it as it's not entirely unusual for moving data from SSD to HDD to create a high I/o wait obviously. Perhaps someone can check it on 6.6.7 for me, my recollection was this only got to about 0.10 on that. The patch mentioned above was put into mainline kernel from 4.19.1. So I've upgraded to the beta of unraid which has kernel 4.19.60. I assume that is later and therefore a good way to test if this resolves the issue. Will keep you posted.

Well you say never, but twice it did it. The first time you could argue I did it via the console by not specifying a Raid level, but the second time, it did it all on its own. Adding empty SSD's to the cache pool using the unraid GUI, is all I've ever done before, so far this has been unsuccessful. I honestly don't trust it. At some point I'll pull out one of the cables - I suspect from this experience that the mirror, doesn't work when you've got encryption enabled but could be wrong.

The best thing about unraid is the pleasant community. Actually, I worry that I won't live up to it sometimes! And seriously, that's not how some other communities are. I moved to unraid to consolidate into a smaller system in preparation for an eventual move overseas, for the most part, it's done that admirably. The killer feature for me is the relative ease of GPU and associated device passthrough e.g. for gaming. I definitely miss some of the features of the more pro end NAS units like directory integration, snapshotting, branded apps and tools to access the NAS e.g. the QNAP / Apple integration to mention the main areas. I look forward to Unraid plugging these gaps more and more in the future.

I'd start with Proxmox. Or even Freenas for your use case. unless you actually want Virtual Machine Graphics pass through to game in or similar, those solutions probably fit you better. Unraid isn't actually a great NAS either, it's a great all rounder that tries to do everything with a very nice non raid disk setup. As far as everything else goes, it's kind of a really polished average capability. I'd add though that the community support is very good generally - much better than others.

Hi I'm running similar setup, working fine, though NVIDIA, not AMD GPU. I would check your BIOS version, there are known issues with the very latest BIOS's affecting GPU passthrough e.g per below. I'm not aware of thread ripper systems being affected yet, but it's worth a shot. Alternatively, if you're running a very old BIOS, you should definitely update that as there's been all sorts of AMD microcode updates that impact virtual machines and passthrough. I'm not sure about point 2, I'd suggest maybe the system has crashed due to point 1. Regarding point 3, I see this happening too, when the system has crashed. So again, maybe start with point 1 and take it from there. Also, unraid has a nice system diagnostics utility you can run and post to this forum to help others to help you, optional of course.

This conversation got my interest. I'm surprised to read that it works. Particularly when you say some SSD's don't have parity support, which is really like saying it doesn't work. Can you elaborate how many, which type of SSD's don't work with Parity? My understanding was that SSD arrays are still unsupported and generally not recommended? Thanks.

After a downgrade to 6.6.7 to isolate some issues, then an upgrade to 6.7.2 a lot of my configuration was reset. One of those things is the cache and I again find myself in the situation where it's not set up correctly. In this instance I have stopped the array to add the cache drive and reassign disk 2 etc as per @johnnie.black above, but unfortunately it's set it up as a RAID 0 for some reason. I'm not sure why, but just noting that it seems I again need to run through the steps above, because I don't wish to format the whole cache again, which may not get different results anyway.

In an effort to begin some testing I have upgraded back to latest 'stable' - if we can call it that. Unfortunately, my mirrored cache has mounted one drive into unassigned devices and the other is being reported as having no file system. This is not the expectation I have of a set of mirrored SSDs. In addition, even though it is enabled, I have lost SSH access. Sure I can get in via the server, but this is really quite unexpected. I have no idea if I had data on my cache or not. Actually with the amount of issues with the BTRFS cache I've had with it not setting up a true mirror etc, I am now of the opinion it is safer to be unmirrored, because frankly it doesn't work for the purpose it was intended. This unraid has a few quirks doesn't it. If it weren't for the excellent KVM GPU passthrough, I'd probably migrate to FreeNAS now. The silence from @limetech on this issue is just not OK. Fixed SSH by deleting the keys in /boot/config/ssh/ Fixed BTRFS Cache, by stopping array, and changing cache to have two disks again (it had reset to 1). Then starting the array, confirming the data existed, then stopping array again, adding in second disk that had been ejected and starting array one more time. An auto balance then automatically ran, which incorrectly turned the btrfs into a RAID 0. Running the below command balanced it back to raid 1. Done. Now to monitor to see what we can find in terms of iowait etc. # btrfs balance start -dconvert=raid1 -mconvert=raid1 /mnt/cache

Doing a bit of googling - I did find the below, which seems to coincide with the kernel versions of the 6.7 series, similar symptoms and be related to ATA disks. I'll need to upgrade back to 6.7 to do testing to see if there are any IOWait related issues or if this is totally unrelated, but posting here so others can weigh in. Equally according to this thread, some things seem to wake up the bug and cause it to revert to normal performance again. With all the plugins and dockers within Unraid, I could see that happening quite consistently, making it hard to pin down. As far as I can tell, the kernel versions Unraid use, even in the latest RC3, do not include a fix for this issue. https://bugzilla.kernel.org/show_bug.cgi?id=202353

I'm still not sure @limetech have actually seen this. While we're waiting, a good idea might be to all comment on what motherboards / sata cards we're using to see if there's any commonality. I'm surprised more people aren't reporting this to be honest, but maybe they've just not been that observant yet. I've got an Asus Prime X399-A board with 6 SATA's on it and a Dell Perc 310 controller flashed to IT mode, detected as a Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-s [Falcon] (rev 03). This has an additional 8 SATA ports. If we all had that exact card, all had AMD, or even all had threadripper for example, I'd be suspicious. Also, has anyone uploaded diagnostics, looking back through the thread it seems even the original poster hasn't done this, though maybe I'm not looking properly... If not, we should attach, just make sure you're not on 6.6.7 when you do it.

Forgive the dumb question, I don't actually see where I can upvote this, I can see others that have upvoted, but no option in there for me to do the same, other than to 'like' it. Edit - found it. Hovering over the like button shows an upvote button. Not exactly intuitive but all good.

Yeah, I was having issues with backups crashing too. I'm not enjoying the downgraded KVM though.

Found it copied relevant snippet below: "....some observations that seem odd to me include at times the disk is reading and writing from the same disk at the same speed in both read and write columns of 75MB/s and simultaneously the drive it's copying from is only running at 10 or 20MB/s sometimes less. Other behaviour that seems odd to me, is it cycles between reading from the source drive (and not writing to the target), then not reading from the source drive and writing to the target. So it's like copying it to a buffer somewhere. Something I'm sure is not normal for a normal move or copy operation." This was using unbalance. However, more unusual stuff even on 6.6.7 shows I really don't know how unraids raid works. I have disabled both the docker and the virtual machines services, so nothing is running. Doing a console copy from my VM drive (unassigned devices) to the Btrfs cache mirror, is running nice at 490MB/s, yet there is a raid array disk constantly at read of 245MB/s for the whole copy - it stopped when the copy stopped. And no, I am not writing to /mnt/usr/something I'm just writing directly to /mnt/cache.

I think it's a weekend for @limetech, probably be another 24 hours before they notice this has blown up a bit. Your post reminds me of another one too, where I had the same 'burst' file copying issue - I had forgotten about that, but exactly as you describe. I think I'll go hunt for my thread on it, pretty sure everyone thought I was mad!

I suddenly had the realisation that this bug is probably what's been causing me so many headaches with my Crashplan backup. I mean, I nearly cancelled the service because it was so slow and it kept crashing. So I had enough and downgraded. Yes, crashplan (docker based) is now suddenly faster and so far working much better, other things I noticed included that it booted a lot quicker and didn't sort of pause before the login screen, Plex is more responsive, the disks seem to be 'quieter', before there was sort of random reads and writes happening which I couldn't track down, but now seem to have dissappeared, the Unraid GUI is much faster, I'd even say my SSD is running cooler. (Call me paranoid but I've had two SSD's unexpectedly die and this brand new one already has unrecoverable sectors after only a month. Perhaps some of this is in my mind, but the primary function of a NAS is, well to serve files to multiple people concurrently in a performant way. Right now that doesn't happen on 6.7. I'd bet many people have this bug and haven't realised it yet.

Is it possible something in the upgrade process breaks something? After I downgraded I actually had to reboot twice to get the GPU to even show up as a PCI device in Unraid at all. Something isn't right. I don't normally have any issue with GPU passthrough disappearing. And I turn it on and off most days along with reboots every now and then, never an issue. Along comes rc2 and straight away an issue. Some kind of differing KVM version / kernel match perhaps?

I have a 1070TI and after installing beta 2 seemed to break my GPU passthrough (I only got a black screen) and one CPU thread of the passed through ones to the same VM was also stuck on 100%. Going back to stable seemed to fix it. Only tried it once though. Perhaps the two are related.

I'm just going to downgrade until someone sorts something out I think. There is a beta out with a newer kernel which could be worth a go though. Happy to help out with testing, but doesn't seemlike lime tech are listening for some reason. They're usually pretty good right?

Talking about differing CPU's - I have been having issues on my thread ripper system, I'm now going to perform the same test as above as I hadn't noticed it exactly like that, but then I do have two disk controllers which may change things a little. I hadn't realised it until just now, but on top of the normal Plex issues with mover, I had the Apple TV plex client just dumping out of a movie yesterday while copying a large amount of data from my Unraid server to an iMac. Other things I'm trying to understand the cause of, is since this version I've had two SSD's die (one enterprise) and the new enterprise that is only a month old, written only 8TB (rated at 1TB per day for 5 years) already has re-allocated sectors on it). I'm pretty sure I've changed that cable which is about the only thing left I can think of doing - I don't suppose it has anything to do with this, but thought I'd throw it out there. It does say it's had 11 unsafe shutdowns (which it definitely hasn't) - however a bad cable is a possibility or maybe with all these I/o problems it starving the SSD into thinking it's had a disconnection? Just throwing it out there as 2 dead SSD's and a third new one with issues is not normal.

Yikes, that's terrible speed.