JorgeB Posted July 11, 2019 Share Posted July 11, 2019 7 minutes ago, sdamaged said: So is there no way at all for unRAID to be able to silently fix file corruption? (even in the future?) I can't say for sure but doubt it, one of the big Unraid advantages is that each disk is a separate filesystem, but this also bring some limitations. 12 minutes ago, sdamaged said: it becomes almost impossible to have backups (50+TB of movies on mine) that i can't back up without another array. Yes, it gets expensive, I have a full backup of all my servers, each server has a clone synced at least once a month with btrfs send/receive, largest one is at 102TB currently, and I'm far from wealthy, for me it's a question of priorities, though I'm now at a point where the main expenses are done, just need to upgrade a few disks each year as more capacity is needed. 8 minutes ago, sdamaged said: Incidentally, what benefits could BTFRS give me over XFS, as i may be building a new server early next year Main btrfs advantages for me are snapshots, send/receive and checksums, I've been using btrfs exclusively on all my Unraid servers for years now without issues, but won't deny that ZFS is more mature and stable, I also have one FreeNAS server, though it's not perfect, it's not flexible at all, and if there's a problem, there are no tools to fix it or recover, just nuke pool and restore from backup. Quote Link to comment
BRiT Posted July 11, 2019 Share Posted July 11, 2019 53 minutes ago, sdamaged said: it becomes almost impossible to have backups (50+TB of movies on mine) that i can't back up without another array. This entire plugin thread says otherwise: Quote Link to comment
sdamaged Posted July 11, 2019 Share Posted July 11, 2019 55 minutes ago, BRiT said: This entire plugin thread says otherwise: Oooh thanks for this! I did try an older video where you could backup to Google Drive, and then it stopped working as Google apparently wouldn't allow it? If this works then great, as i currently have a 2TB volume added to a VM (which is basically duplicating data) with the google drive client installed Quote Link to comment
Alphahelix Posted July 15, 2019 Share Posted July 15, 2019 Ok I know that ZFS is at the moment not possible to implement in unRAID due to legal issues. But... what if someone (unfortunately I lack the programming skills) was to take some of the features and make a rudimentary construct? Would that still be ok legal-wise? Let us pretend it is, and let me theorize how it could benefit us unRAIDers... Today we have individual disks for parity and data, only cache can consist of multiple disks. To my knowledge (and i do not claim to burdened with in-dept-knowledge) of ZFS uses vdev(s) that consists of on or more disks that can vary in sizes. With multiple vdevs you are able to expand the ZFS. vdev(s) uses ZFS own data protection single, mirror, RaidZ1, RaidZ2, RaidZ3 etc. where the number indicates how many parity disks in the pool(s). Now if we use a somewhat similar approach in unRAID where we bundle individual disk to a pool-of-disks (Abbreviated as POD from this point on) running in stripped-mode via "mdadm" and assigning PODs different roles like individual disks are today. In theory the benefits should be increased speed. Also we should be able to make a POD that consist of SSD's for parity-POD and a data-POD, thus get a very high transfer speed outside the cache and still have data protected in parity. For those wanting multiple arrays, this could be seen as each POD could be seen as one array. I have tried to create a visual of how disk assignment is today, and with the POD design: Today: 8tb Disk0-parity0 6tb Disk1-parity1 4tb Disk2-data0 4tb Disk4-data1 4tb Disk5-data2 4tb Disk6-data3 4tb Disk7-data4 4tb Disk8-data5 6tb Disk9-data6 6tb Disk10-data7 6tb Disk11-data8 6tb Disk12-data9 4tb Disk13-data10 4tb Disk14-data11 4tb Disk15-data12 POD-design: 4tb Disk2 POD0-parity0 4tb Disk4 POD0-parity0 4tb Disk5 POD0-parity0 4tb Disk6 POD1-parity1 4tb Disk7 POD1-parity1 4tb Disk8 POD1-parity1 6tb Disk9 POD2-data0 6tb Disk10 POD2-data0 6tb Disk11 POD3-data1 6tb Disk12 POD3-data1 4tb Disk13 POD4-data2 4tb Disk14 POD4-data2 4tb Disk15 POD4-data2 8tb Disk0 POD5-data3 6tb Disk1 POD5-data4 In the example above overall performance in the array should in theory be faster, even with unRAID's speed penalty. Further POD4 should have superior read/write speed (given that no other PODs are active at that moment). And one could argue that you now have 4 smaller arrays. This is will no doubt make the "master"-array more vulnerable to disk fails, but by using mdadm you are not limited to RAID0, but can in theory use RAID6 in one POD and RAID0 in another should there be a need for that. Also if memory serves me correctly, ZFS has no way of expanding a vdev. It is possible to expand a POD using mdadm. (with limitations in RAID10) When using mdadm it is also possible for RAID6 (most likely also RAID5) to add a hot spare disk for minimal "rebuild-time" if needed. So data residing in a POD with RAID6 will have "internal POD protection" in form of RAID6 but also be protected by the parity POD(s). Data residing in a POD with RAID0 will only have the protection of the parity POD(s), and thus be more vulnerable than a setup where you have 1 disk for parity and 1 disk for data. The reason is if ONE disk in a RAID0 array breaks down ALL data is lost. So with the POD design using RAID0 you are more vulnerable than if you have 1 disk for parity and 1 disk for data. So I think a 1 disk for parity and 1 disk for data solution should still be possible, at least as having a POD with only 1 disk in it. Now I know some may think this is a bad idea, but I am sure some think it can be useful. I think that if Limetech chooses to implement this it should be for PLUS & PRO licenses only as 6 disks and below will get little benefit from this. I have tried to list pros and cons for this POD design: PRO: 1a) if theory meets practice we should see a read speed bump for each disk added to a POD 1b) if theory meets practice we should see a write speed bump for each(* 2 for RAID10) disk added to a RAID0/RAID10*/RAID5/RAID6 POD 2) possibility to "kind of" having multiple arrays 3) possibility to have hot spares for selected PODs 4) selective elevated data protection for each POD CON: 1) more vulnerable to data loss when RAID0 is used in POD 2) less available space from disks where anything else but RAID0 is used in POD 3) CPU usage may be higher due to the extra "layer" of RAID This suggestion is to give inspiration I have only used what I know/heard of. I know it is a bit far out. But at least no one got hurt. /Alphahelix Quote Link to comment
glennv Posted September 20, 2019 Share Posted September 20, 2019 (edited) Personaly i am happy with the main array as it is. But have been bitten now about 3 times with corrupt btrfs cache pool. I use a large multi TB ssd (btrfs raid1) cache pool for speed and direct fast (via 10GB ether) access to video data . I keep projects that i am working on in the pool and once done trigger mover to move them to the (still fast 7200rmp) spindels (xfs). Back and forth with unbalance keeps it very workable. For my vm's i enjoy the snapshots feature of btrfs But 2 of these times i was hit it resulted in corrupt VM images that btrfs could not fix and the last one into complete pool corruption, that was not fixable by any of the btrfs methods i researched. Of course have good backups but it sucks having to rebuild everything and drops my sense of btrfs reliabilty to near zero (while i was bragging about btrfs to most of my friends untill then) At that time i did not know much if any about zfs. Started reading up on it, installed the plugin, moved most data i kept on the btrfs cache to ssd only zfs raid1 pools and they have been stellar since. Also zfs literay (in plain english rather then useless syslog messages) told me the main reason that i had issues with btrfs already in the first weeks , which was a wobbly connection to 2 of 8 ssds in a hotswap cage, but in contrast to btrfs it only reported it but kept fixing any issue continously and kept the pool healthy, where btrfs broke in same setup / same disks / same cables / same problems. I unplugged and replugged live ssd's to see which drive it was and all the time zfs kept smiling like nothing happend. There is just "zero" comparison or competition with btrfs . Specs or having a system runnig for years without issues are nice but real world behavior when there "are" issues and how you recover from them is what counts. The argument that btrfs has and zfs does not have recovery tools was mute in my case as none of the tools helped me at all other then to save some part of the data but not it fixing the pool, while zfs just fixed it for me while helping me also identify the problem. So i have become a complete convert from a btrfs advocate to a zfs fanboy. So a great option would be to be able to have zfs for cache only so get the best of both worlds. I can live without it for the main array as i see less benefits and more complications on implementing it there as already discussed. Nice pool features but still a fast and super reliable cache pool (with raidzx options as i did not even mention btrfs lack of anything (reliable) more then raid1/10) Edited September 20, 2019 by glennv Quote Link to comment
Marshalleq Posted October 14, 2019 Share Posted October 14, 2019 I've just installed the ZFS plugin on unraid and love it. I decided to take two drives out of my unraid array and assign them to a ZFS mirror to store important photos and documents on. This makes unraid nearly the perfect system. I agree it would be great to have ZFS cache pool. I've been bitten several times with btrfs cache and I note depsite some contrary views, the latest Unraid 6.8 rc1 changelog which confirms some of these bugs. Would be great if ZFS was included - even if only hidden under an advanced option, then among other things we'd probably find e.g. proper integration with plugins to help e.g. the scheduling of scrubs and such. Licensing doesn't seem to be a concern any more either: E.g. ubuntu and software freedom legal council Actual releases in Ubuntu and proxmox Others if you google it. Quote Link to comment
chris_netsmart Posted December 3, 2019 Share Posted December 3, 2019 I have just discovered this function, and I am doing a lot of reading. I feel the same as @marshalleq that I also like to have a area that I can place my important data, but I don't like the ideal of relaying on a plugin to do the work, as I feel that this should be a core element. To to ZFS users can you please tell me how you have found the ZFS environment and setting up. and also did you find any good reference "Step by step Guide" Quote Link to comment
Marshalleq Posted December 4, 2019 Share Posted December 4, 2019 I wouldn't worry about the plugin aspect of it - yes it would be better to be built into unraid, but it works at native speeds and is very robust and reliable. I've not had a single issue in regards to performance or stability since I installed it - in that regard I do consider it native. The only thing it means is that after an unraid upgrade, the developer has to recompile it for the latest kernel. Not a big deal - he's very responsive. Hopefully one day, limetech will find a way to make this integrated more as others such as ubuntu are now doing (legacy licencing concerns that no longer apply - but some are still cautious). I think there's a step by step guide a few posts up. You gotta be comfortable with the command line is the only thing. Quote Link to comment
zepheris Posted December 5, 2020 Share Posted December 5, 2020 Will OpenZFS be a viable option with Unraid? https://github.com/openzfs/zfs Quote Link to comment
PyCoder Posted December 27, 2020 Share Posted December 27, 2020 +1 I would like to see a UI for ZFS in unraid and a one-click installation for the module (so we avoid the license issues)! PS. No, ECC not requiered it's only recommanded! Unraid isn't protecting us at the moment from bit-rot neither in ram nor disk. ZFS without ECC would at least protect the data that is already on the disk from bit-rot! Quote Link to comment
Marshalleq Posted November 12, 2021 Share Posted November 12, 2021 (edited) On 12/28/2020 at 10:57 AM, PyCoder said: PS. No, ECC not requiered it's only recommanded! Unraid isn't protecting us at the moment from bit-rot neither in ram nor disk. ZFS without ECC would at least protect the data that is already on the disk from bit-rot! Seems like someone has been having a few trying discussions! Two more: No ZFS doesn't eat your RAM, some distro's just haven't configured the memory limits properly Yes dedup is useful and no it doesn't eat your RAM if applied properly, especially with the special metadata vdev. We should make a list of the FUD BTW, my entire system is now ZFS and has been for quite some time. I won't be going back, the benefits are just too huge. The speedup of the system was quite noticeable also. Edited November 12, 2021 by Marshalleq Quote Link to comment
guybrush2012 Posted November 17, 2021 Share Posted November 17, 2021 +1Gesendet von meinem SM-G998B mit Tapatalk Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.