Marshalleq

@jaylo123It's a known fact that Unraid's 'Unraid' array has dawdling speed. There is no workaround for this. The only solution I can think of (which I have done) is to not use the unraid array. So pretty much on unraid that means use ZFS array. From experience the speed increase was notable. -add to that the remainder of benefits and (to me at least) it's a no brainer. However, despite being very well implemented into unraid, you would need to be comfortable with the command line to use it and be prepared to do some reading on how it works. So it isn't for everyone and I'm not trying to push you one way or the other. I'm just saying the 'unraid' array is known to be extremely slow.

Also try removing and re-adding the zfs plugin. Could also try stable and do it again. But it does work on at least r3 because I'm running that. (Sorry not sure what you're aware of here so I'm just going to say it - make sure you wait until the zfs plugin updates it's module before rebooting after an unraid version change). And if you can boot into normal mode (not safe mode), perhaps you need to drop to the command line and reimport the pools.

It sounds to me like either the server didn't wait for the updated zfs module to get built, or it wasn't able to be built. The quite solution to that might be to downgrade the server firmware to the previous version and go from there. That shouldn't be too hard. Failing that, ping either steini84 (or possibly others know) where there is a downloadable matchable version along with where to put it to force the process. Sorry I don't know the process, it may be indicated somewhere in this thread though. Marshalleq

No problem. I haven't actually ever used the GUI options and my setup may be very slightly different. For example when I create an array, I set the mount point of the array simply into /mnt - it seems like you've put yours into /mnt/user/zpool. I am not sure if that's in a guide or what, but it doesn't seem like a sensible way of doing it as you may have permissions problems being that it's a user folder. Also, clearly you're also talking about SMB sharing. I found that the unraid smb sharing doesn't work with ZFS really, but luckily zfs has it's own SMB built in - so you can edit the smbextra file in /etc somewhere - sorry can't look it up exactly at the moment - I think it's like /etc/samba/smb.conf/smbextra.conf or something like that. Don't worry the smb format has examples and is super easy. Also for me, any zfs shares in unraid under /mnt I just set them to nobody.user and that sorts them out. having them as root definitely doesn't work. Just always bear in mind with linux, folders must always be set to 777, and the files can be whatever you need. Hope that sort of points you in the right direction.

Command line. The useful commands I've noted for the disk based (L2) cache are: Add Cache Drive: zpool add data cache /dev/nvme0n1 Info to see how good it's functioning (also good for the in memory cache) arc_summary arcstat How to Remove Cache Drive zpool remove data cache /dev/disk/by-id/ata-xxx (Not sure if /dev/disk/by-id was what actually removed it in the end) The second time this command removed it zpool remove DISKNAME /dev/nvme0n1 It is sort of diminishing returns but fun to try out because you can actually remove it. I tried it out for a while but the hit rate was quite low. Could be good if you had VM's stored on slow disks or something though. One nice thing is it's persistent now, so it remembers what it cached after a reboot.

Basically, I created a new dataset and copied it over with rsync after setting it all up. Have been doing ZFS for a while now. Same for the array with special vdev - that was a while ago now and wasn't a small task, but got there in the end. My main gripes are not so much with the web pages, more to do with load times e.g. startup from docker and the forever chugging away in the background. It may just be that my library is big. Plex says I have 114000 tracks / 1092 artists / 8463 albums. I hadn't seen ioztat before - I'm guessing that better than zpool iostat by going down to dataset level of something? I'm talking about how ZFS will store in the default 128k block a block of up to 128k. it's variable. I believe it will literally turn a 128k block into a 64k one if it sees fit to do that. I've always been suspicious about this though, particularly with databases. The thing with ZFS is that just because one group of people told me, doesn't mean it's true - there are a lot of details to work through. But I do know that ZFS has variable record sizes. That's very kind thank you. I may take you up on it in future as it would be fun to see your process of figuring it out. Also, this is actually not on the 1950x (actually now it's a 2950x so I should update that), this is on the dual xeon machine. Either way I'm basically unavailable for around 3 weeks due to things going on in my life so would have to be after that if it's anything more than comments in a forum.

I'd recommend setting up some sensible defaults in the root that you think will apply to that whole drive, e.g. like turn compression on as most things will use it - probably the default record size can be left alone, xattr=sa and whatever else you want but those are the main ones from memory. Then you tweak them per dataset.

I've done what you're suggested on Lidarr, because that's by far the worst performing app for me, but don't really notice any difference so far. In particular the updating of the library (which annoying seems to be entirely scanned now when triggering only a single artist) My library is probably 6x the size of yours. What I was asking above was, does your understanding of zfs include why or why not the variable record sizes cover performance of different table sizes in a database? Because I've gone down this path before of optimising record sizes, jumped on some forums and been shot down because they were adamant that it's not needed with the variable record size feature of ZFS. Also, by reducing the record size, you apparently reduce the available compression (which I checked and my DB went from 1.3G to 1.6G with the smaller record size so it appears that at least that comment was correct. Personally, I think you're onto something here, because I assume ZFS cannot be aware of database page size in a large single file like it can be with individual files and cannot align a page to a record without some help, but I could be wrong. Since the official ZFS page has an example covering I think it's Postgres, that would seem to confirm it. So I applaud your efforts here and await more commentary from others around any speed changes. What I was hoping for was an increased refresh and scan speed and the subsequent 'reading file' which seems to happen twice on the whole library afterward to be a little quicker. But there are external factors with that. In addition to the Lidarr DB on SSD, my audio is stored on a 6 disk Raidz2 but with a special vdev where all the metadata is stored, so it's about as good as I'm going to get without going all SSD. I also use a product called Roon, which is a fantastic but expensive Music player. That is the absolute slowest app I have. It runs on Google Leveldb. Any experience with that out of interest?

I run some Darr apps and notice on my ZFS array they are probably some of the more heavy I/o requirements. It's a 6 disk Enterprise INTEL SSD set so it has the IOPS. Most of the time I've looked to optimise it, I come back to the advice that ZFS has variable record sizes so there's no point. But if we use postgres perhaps there's an optimisation opportunity to match the page size to the record size. I'd be interested in your thoughts on that - thanks for the link to the Radarr guide.

Fantastic info! This will be a great addition!

Hey, just eluding the various places where Limetech have already eluded ZFS is coming - .e.g the video interview I saw quite a while back now, the poll for next feature which included (a winning) zfs as highest vote etc...

Just to be clear - nothing in that post says ZFS will be available in 6.11. It only says they're laying the groundwork for it. Yes, that does officially elude that it's coming - but that's happened in a few places already.

Does anyone know if this is specific to the unraid array and or a specific filesystem, or does it apply to any filesystem? I'm having some weird issues and wondering if this could be the culprit. Thanks.

Just letting you know I got thunderbolt storage working in unraid. Details of how here:

OK, I got it working. Details here:

Just adding a note here as I created a page for people to list their experiences with different devices and I'll summarise them and keep it all up to date. I've also included a sort of live install instructions, which will no doubt need work, but we can update it as we go. I am managing to see storage devices minus the disks so far, there's probably some trick to mounting the disks once the SATA connector is detected by thunderbolt. Anyway, the page is here if you'd like to contribute. Many thanks, Marshalleq

This section to become the howto for how to get this going. For now I will put my experience so far, which is to say it looks like it's working but am yet to get any fruit. Jump pins 3 and 5 as outlined here , install the card and connect your devices. The card should now show up in Unraid as a PCIe device lspci shows a number of lines similar to: 03:00.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] (rev 06) Also for your storage you should see a line similar to: [ 44.231417] thunderbolt 0-1: LaCie Rugged THB USB-C Add the following line to the /etc/udev/rules.d/99-local.rules file: ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0", ATTR{authorized}="1" Navigate to /sys/bus/thunderbolt/devices/domain0/yourdevicefolder Display the contents of the authorized file: cat authorized If this is currently a 0 then: echo 1 > authorized Confirm it is now a 1: cat authorized Reboot Success, storage is now shown in the unraid GUI Note that line 4 effectively disables all thunderbolt security if you're worried about that. I've requested the bolt package be added to nerd tools as this apparently is the security manager for thunderbolt in linux. There is also a package called thunderbolt-tools that provides tbtadm that seems to do something similar. I am still to understand how much value these tools are considering this works now.

Sorry, I'm not sure if we're allowed to ask here - but could we get the bolt tools for thunderbolt added to this? I think I have thunderbolt working in unraid by disabling the security, but I understand the bolt package will allow that to happen more elegantly. Many thanks. Marshalleq.

Wow that was a lot to take in. On that two sided argument I’m going for the awesome read performance mentioned and think it isn’t going to make much difference for writes. Especially when I am using a 32 thread threadripper and a 24 thread dual xeon setup.

If I recall correctly the performance is better and it’s multi threaded where as lz4 is very old and single threaded. Could be wrong about the threads. Zstd has differing levels of performance you can set obviously. I just read up on it at the time and chose it. I don’t use slogs I thought they were really only beneficial in rare cases and would need more redundancy because it’s writes? I don’t know much about them sorry. Very nice drive though!

I can tell you what I use, then you can go and read up on those bits. To me these are the key bits in setting up an array. I'm not going to disagree entirely with BVD, but really like all things technical, research and experimentation is valuable and it's no reason to fear ZFS and not use it. There are some people that dive in without doing even a minor bit of forethought and I assume his commentary is really aimed toward that rarer group whom will likely get themselves into trouble with everything not just ZFS. Anyway, here's the basic command I use first. If it's an ssd pool, I add -o autotrim=on some people are still scared of this, but I've never ever had even one issue with it - compare that to btrfs and the issues were quite a few - though that was years ago now. zpool create HDDPool1 -f -o ashift=12 -o -m /mnt/HDDPool1 raidz2 /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg See below about special vdev before creating the above. Then the optional tweaking stuff I use: zfs set xattr=sa atime=off compression=zstd-1 HDDPOOL1 And depending on your media a dataset could be changed thus: zfs set recordsize=1M HDDPool1 Default is 128k (Don't get caught with this too much as I keep forgetting ZFS does variable record sizes - 1M might be good if you have a lot of large video files for example.) dedup=on (I use this a lot, but only because I have a special vdev because IMO it means that no significant extra ram is required, however I've had quite the discussion about that so not all will agree - definitely do your own research before enabling this one). Works great if you have lots of VM's and ISO's. My RAM is not used in any way that I've ever been able to notice. More options zfs set primarycache=metadata HDDPool1/dataset zfs set secondarycache=all HDDPool1/dataset Some of the cache options are actually dealt with automatically. The promise with them is to optimise how much of your data will be cached in ram depending on e.g. if you have big files or not and whether it is valuable and even possible to cache them. And finally, the special vdev mentioned above is very cool. It will store metadata on a second set of disks assigned to the array. So for example, if you have slow hard drives in a raidz2, you could have 3 SSD's (for same redundancy level), which speeds up seeking and such. It optionally also will store on the SSD's, any small file up to the size you specific (which must be less than the recordsize or you'd be storing everything). As you probably know, small files on HDD's aren't very fast to read and write to, so the advantage here is obvious if you have that kind of setup. zpool add HDDPool1 -o ashift=12 special mirror /dev/sdp /dev/sdr /dev/sdq I actually also set up a fast SSD / NVME as a level 2 cache - this can be done at any time and it's advantage is just that anything the doesn't get hit in ram, fails over to SSD and again is a way of speeding up reads from HDD's. zpool add HDDPool1 cache /dev/nvme0n1 useful commands: arc_summary arcstat So what you can probably see is there is a default way of doing things and the 'tweaking' mentioned above is really more about understanding your data and how you want to address it via ZFS. Some settings need to be done at array creation and some can be done later. Most settings that are done later will only apply to newly written data so you end up having to copy the data off and on again if you get it wrong. I found it super fun to go on the journey and learn it all, I expect you will too. If you're like me, you'll want to be doing some more reading now! Have a great day. Marshalleq.

Hi thanks for posting, I'm interested in what advantage this is to you. I have run the img file on a zfs dataset and also run it as a folder which creates a host of annoying datasets within it. Does this somehow get the best of both worlds? Thanks.

Right out of the gate I can say that you'd be better off with 4 drives in two mirrors. You have the same space and more speed vs 4 drives in raidz2. Secondly, I assume that an i3 is powerful enough to calculate the parity needed without bottleneck, but it might pay to double check that. In the above configuration, do you have all of that RAM spare or not much ram? I have seen it where having not much ram spare slows things down a lot - this can somewhat be mitigated by setting the available ram in the go file. It also may pay to performance test each drive individually, in case one of them is slowing the others down. I had this same problem on a thunderbolt connected ZFS cabinet the other day, then found out that running non-zfs file system on a single (or multiple disk) was also slow. I am yet to progress but suspect I have one drive that is playing up. It's surprising (and annoying) how often this can be a faulty SATA cable. ZFS will be slower that other raided file systems, but not by that much so I agree something is wrong. It's probably close to what unaid performance is though as that is actually very slow. Can't think of anything else right now sorry, and I appreciate you may have thought of these things already - but sometimes it can trigger a thought for a solution right? Hope you figure it out and let us know - it might help me for my one!

Sorry just rushing out, yeah this does seem a little slower than normal, though unraid is typically quote slow for disk transfers. I'd suggest to see how fast it copies locally on the box first to get a baseline, (maybe use midnight commander (mc is used to activate it) as I think that shows the speed. Then if that works OK it means it's probably a combination of networking and perhaps the SMB protocol, which is also a bit slow on unraid for some reason - depending on what you're connecting to it with. There are some tweaks for SMB. But those speeds almost look like you're running a 10Mb/s hub in there somewhere or a faulty cable slowing it down.

This is a really good question. I think there has been a few times when I have wondered the same and done something probably wrong to get it going again. Perhaps a few of us (or anyone with a standard install that hasn't been messed with should post back here what they have. Mine is set to nobody.users with 777 on everything - so I guess I did that in the past some time. I do note that preferences.xml plexmediaserver.pid and the scripts folder are set to 644. I have a myriad of different permissions in the cache folder - which I guess is right because that will create it's defaults as it creates them. Probably a good idea will be to set one up from scratch and have a look at those defaults.