NasOnABudget

Members
  • Posts

    28
  • Joined

  • Last visited

Everything posted by NasOnABudget

  1. So, I put my plans on actually using my UnRaid server on the backburner because I have figured out that I have a hard requirement that UnRaid cannot fill out of the gate. The problem is, there is no recycle bin. So, after searching around somewhat I found out that there is a recycle bin, through a plugin, however that also does not do what I want, as I don't want the recycle bin or its contents to be able to be deleted by users accessing the server via SMB, but only through the Unraid web interface or a VM. I want to say ahead of time, that I'm not using this in place of backups, I want this feature to both cover what backups can't cover (extremely recent files), and to protect the user from themselves with accidental deletions. So what are my potential solutions. It's been suggested that I use a cron job every minute to change the permissions of deleted files in the recycle bin, however that does not work as a solution because a user could potentially delete all of the files at once. The only solution I have been able to think of is to have the main share shared to a virtual machine, which then shares that share to the end user. Im uncertain of how much overhead this will have, but I'm not concerned about cpu performance if that's the worry. I suppose another way this could be done is with a Docker container, however, I am even more unfamiliar with those than I am setting up a virtual machine (which is to say not that experienced). I was hoping someone might have insight into either: How to accomplish the overarching task of having a recycle bin end users cannot access What pitfalls I might run into with the VM plan of action Is it possible to for instance, use the recycle bin of the VM and have the share/unraid pool as a mounted drive How a Docker container might compare
  2. That sounds like it could work, but sounds fiddly to what Im thinking where Ill have a vm with a share.... to a folder in the share. This way the deleted files just wont be there for the end user (me) or I can set it up as read only, and wont have to rely on a script. The worry with that script is Im not sure how the permissions work or if that would break through unraid changing.
  3. This is really disappointing to me, as I consider protecting me from myself a really important feature of a nas. This protects you in a way deeper than a backup can. Not having this puts a pretty big damper on unraid as a whole for me. Im wondering what can be done about it, like through a VM or something similar. I guess Ill have to start poking around in ubuntu or something. I was really hoping there would be some way @dlandon might have of having the recycle bin for one share be a different share or similar. (btw I only mentioned again because I edited in the name the last time so Im not sure if that still pings the user)
  4. Is there a way to make the recycle bin inaccessible on the share @dlandon ? I want to make dumping the recycle bin/restoring only available from unraid itself and not over the share/eliminate the ability to delete the recycle bin.
  5. I forgot to ask here, what issues did you have I feel like I've seen a bunch of people talk about issues, but rarely is it elaborated on.
  6. The reason I leave off ZFS, is that would require a couple thousand, whereas the reason I went with unraid was because I mostly had the drives id be using (and its inflexibility in general). I'll probably be buying some more drives soon to have a proper offline local backup, but yea, basically unraid, due to my not needing performance, but having a lot of differently sized drives is sort of the option, plus I've already bought it and it seems more turn key than ZFS. I'll definitely revisit it in a few years though. I'll look more into Dynamix File Integrity though, but I generally think I would prefer to avoid relying on third party plugins due to how that usually works with updates and because its people donating time that they may or may not have in the future to keep them up to date.
  7. I've been agonizing over this choice for weeks, unable to pick between the 2 file systems for my main array (I'm already decided on going dual raid 1e for ssd cache/light vms). I have found general answers for many of the questions I have about the choice but there are still ones that just don't seem to be answered that I kinda want to know the answers to, hopefully definitively. What questions do I have? When a file is corrupted with a BTRFS system, is it unable to be transferred? I feel like I have read something similar to this a few times with there being an error, but also that there might be a way to get around it if for instance, you don't have another good copy of the file, like if it is unsafe or between backups. Is anyone familiar with this experience with perhaps a link to some sort of relevant resource on the topic specifically? I've read multiple times that BTRFS doesnt have a working FSCK. Apparently though, this isn't actually a problem because it has a replacement in btrfs-check. Is my understanding there more or less correct? While it can detect errors due to checksumming, it does not have a method of fixing these errors in unraid, specifically because unraid is using it on singular disks inside of the unraid array so those features would not be feasible. Luckily, from what I can tell, BTRFS has a method for relatively easily figuring out which errors were found so that a use can potentially restore the file from backups. Is there any problem with the assertions I've just made? The write hole issues with BTRFS mostly come into play with sudden ungraceful shutdowns due to a lack of proper atomicity. Also, its now really just a concern for the Raid 5/6 implementations that Unraid does not use in favour of the proprietary Unraid solution. Is this correct (this is one that seems really important and related to the first question I asked)? I'm sure that having a UPS (Which I do have) helps, but even still, ungraceful shutdowns can happen for many ungraceful reasons Why should I/Shouldnt I use BTRFS (something I'll answer a bit myself)? OK, so why do I care about BTRFS and what benefits will it potentially bring to me to warrant all the fuss and agonizing about the choice. The main things are: Data integrity - I at least know when a file is bad, even though I may not be able to fix it/will have to get a copy. Snapshots - Something to aid in protecting me from user error once set up, something I feel is a bit lacking in general. Both of these points have large asterisks as well of course. For data integrity, or at least more data integrity rather than less. I'm not even running ECC, and quite frankly, if the data integrity of a regular pc has been good enough for me so far, I probably don't have data that is so susceptible to corruption that I would notice a flipped bit or 2. The data im storing is mostly media, and some backups of other systems anyways. On top of that and as I've read (I think on this forum) a likely place for data corruption to come from on a modern system is ram, so while this data integrity feature is a nice thing to have, it may not be all that I hope that it is, at least unless I in the future decide to upgrade to ECC. For Snapshots, as far as I can tell, this will take a lot of time and effort to figure out how to set up correctly (I'm no fan of running someone else's long script unless I've read through it and understand fully what its doing, and even then, Id prefer knowing how to make my own rather than simply trusting my assumptions). This is why if I'm honest with myself, this feature would sit unused until hopefully, at some point in the future, Unraid natively supported snapshots. OK, that was a lot, and I know posting too much leads to people being uninterested in answering, so Il stop right there, but hopefully I've included enough detail, and evidence that I've put in the leg work that my questions will get answered and I can worry not. Oh, and also, here is a previous post with my specs roughly, if anyone feels they are relevant to my post TL:DR; I basically have 4 questions that have been holding me off from configuring my new server related to BTRFS. Thanks in advance.
  8. Is this due to a change in unraid or a change in their upses. Id be interested to know if others who buy the CP1000PFCLCD can reliably expect this, because I've been pouring over options trying to figure out what to get without unnecessarily overspending.
  9. Thanks that is pretty definitive, now I really only have the question of : Does encryption affect the recoverability of data at all? I suppose though, that that could be a question all on its own.
  10. Is there any verification of this by unraid? I've seen multiple people describe what unraid does in these sorts of situations differently really was hoping for a more definitive answer. Maybe I should send them an email.
  11. lol I have no idea where you got that from ha. Im in Canada. Hmm, sort of unraid scrubbing I suppose. Though, unlike btrfs scrubbing I imagine by default with XFS there is no data repair that goes on when problems are discovered. So if Im not missing something a corrupted disk, if not outright failing could pass off bad data as the true data which would then migrate to the replacement disk as well. Unless throwing an error implies something more that Im not thinking of.
  12. UREs As far as I can tell this is really the only thing that does not line up with what I posted? The hardware information about the background of the disks is cool, though I'm not sure the obfuscation of the inner workings of drives is all that detrimental to me because the actual methods employed don't really matter to the end user, only the resulting conditions. We have an interface, where we know what we can get out for what we put in. That doesnt mean this isnt interesting, but incidental to my queries. A partial problem though with this is that the resulting conditions are unknown. Looking into UREs previously and now, the only way you can get meaningful statistics on how many UREs occur is through working at a large company under nda or having access to a statistically relevant number of drives. What you said about a bad sector/block sounds reasonable, but a third of the disk? That sounds like catastrophic disk failure, like a head crash or sand paper intrusion. A statistical anomaly covered by parity drives or backups (which I will get to later). As a side note, a reddit comment from a claimed to be Oracle employee said something quite interesting. There was a part about reading all of the data more frequently leading to higher chances of fixing errors before they are even registered to the user, sort of related to the inner working of the drive and the basically magic proprietary error correcting/read technologies that go into them. From the same chain, they also talk about how many errors are hidden until spare disk space hidden to the user is used up. All sort of incidental to what I'm really looking for but interesting nonetheless. Encryption Ok, so encryption. How the parity works is all well and good and I've looked through all the articles on the wiki I could find on the topic before making this post trying to see if this was answered in the process. What I'm curious about is the interaction with encryption and data. So basically what the effects of what I am garnering the most likely type of data integrity issue to be. So for a video file, you might see just a blip usually but my question about practicality is if encryption, particularly the common place block level encryption in AES 256 will exacerbate that issue in any meaningful way. As for losing the encryption key, It shouldn't be too hard to avoid losing it by keeping multiple separate backups. The point about access to it are valid absolutely, however, my reason for using encryption isn't because I'm worried about any high intel or trade secrets, but because of the use of my server as a backup for other devices. The thought is that if I become the victim of theft, I don't also need to worry about becoming the victim of identity theft, at least due to digital data. Now, for my portable devices this is way more important, and is the reason I have bitlocker enabled on laptops, as I imagine that theft in a public area is far more likely than theft at home. At home its less important given that if someone has access to your home, they probably have access to a lot of the information they might want anyways, but keyly not things like credit cards, or email passwords. I imagine that so long as a potential bad actor does not have access to my online presence, thats job done for data security. If a government entity, large corporation or anyone else wants at my backups, linux distros and general media, well, firstly, they are going to get it no matter what unless I devote my life to data security, and secondly I'll have bigger problems than my identity being stolen Backups Firstly, I want to say you are 100% right, parity is not a backup, raid is not a backup, backups in the same locations arent the most useful backups. 3-2-1 is absolutely a great standard to keep. That being said, I am not an enterprise. Only some of my data is important enough for me to be able to financially justify backing it up using the 3-2-1 technique, and so I only do that with that data. In a perfect world, absolutely, and indeed I plan on building a more or less identical server when reasonable to function solely as a backup to this one (which would add to my protections, malware protection, and true lightening strike protection), but that's not currently possible. The risk of losing this data (relatively small), vs the financial penalty for dedicating funds towards this insurance dont balance out currently. I realize that for some people with less data or more money, that may sound ridiculous, but unfortunately not everyone is in that position, so compromise must be made here as with every area in life. To give some perspective, I have about ~50TB of data that I would need to find 3 homes for (you can see my in progress build in this post). Doing some rough estimates, and leaving some room for expansion, that would add up to owning a (low end) new car to follow the 3-2-1 method with data storage costs that would be unfathomable. That's simply not feasible. When I have proper insulation from the life altering events listed myself, Ill do so for my unimportant (non show stopper) data as well. The amount of data that is more important though, (pc backups (I dont really hold sentimental value with any of my other data)), Is much smaller and I plan to adhere to 3-2-1 with this data, likely with an online backup solution at much more manageable prices via more hands off services (ex. backblaze + this server). To put it another way, second priority data (~48TB out of 50), simply will have to be second class, while priority data can follow 3-2-1/ Reading through various forums, I see many users are in this situation and often are made to feel poorly, perhaps due to a lack of elaboration and explanation, so I thought id include that if not just to say, yes, I understand best practices, and they are important, particularly for data related to income, clients etc, but sometimes best practices aren't the best (or most feasible) options. Ok so that was long, so let me give a short summary of the above. 3-2-1 on ~49TB of non essential, non income related data is a small new car (completely out of budget/reason) 3-2-1 on ~1TB of essential, still non income related data is doable and I'm doing it. That out of the way lets talk parity. Parity Ok, so one thing I'm very unclear on is what unraid parity can and cannot do with regards to UREs. I understand that they can indeed recreate a drives contents using parity with up to 2 drive failures. What isn't clear is whether or not they can deal with URE's in a way that recovers data. I think some systems are able to make a best guess out of 3 sources (ex. zfs raidz2/3), but as far as I've read, unraid is not able to do this and while it wont fail a rebuild due to bad data, it also wont be able to fill in the gaps. That isnt the end of world for me, as like i've said, none of my data (except that which is backed up multiple times) needs to be absolutely 100%, but I would still like to know authoritatively what current unraid does, and if encryption worsens it. Since I haven't already said it, thanks for the lengthy reply. I found the wiki link interesting.
  13. Hi, I'm about to build my first server and had a question about UREs (Unrecoverable Read Errors) and encryption with unraid I was hoping to figure out before configuration. So if I have this right, with regular unraid, no encryption, when you have a URE during rebuild, and I supposed outside of rebuild as well, it simply means the file with the URE will be corrupted at the byte where the error is, but if you are using AES for instance, will the same be true? As AES is a block level cipher, I imagine that would mean entire blocks of 128 bits would be lost in a file, but otherwise it would be the same result. Does anyone know if this is true? I also wonder if there are any measures in place, with 2 levels of parity to remedy this type of situation without data loss or a method for easily figuring out what file was effected.
  14. As Unraid Supports BTRFS, does that mean that I can say, have 3 ssds, of 240GB in size, and get 360GB of useful space using Raid 1E? Has anyone used this? Is it possible?
  15. Hi, so after looking around for a long while, and few weeks more in depth, I've planned out what I think will be the nas build I'll go with. The goal of this nas is to have a convenient way to backup other devices, to store media files (unbacked up, at least temporarily till I build another due to how much data I have) and to stream that media (likely without transcoding) to one or 2 devices. If Im wasting money ridiculously in someway that isn't reasonably justifiable to me/the goal, Im hoping that gets caught here, before I buy. I've included my logic for all major components in the parts list below. Please note that I have marked where I already own hardware (such as all the hard drives listed). PCPartPicker Part List CPU: Intel Core i5-10400 2.9 GHz 6-Core Processor ($279.09 @ Mike's Computer Shop) Thoughts: I actually am not sure what I want to do with this thing more than holding media, or if Il ever transcode or use vms much or anything, so this, with its integrated iommu capable, quicksync capable gpu should give me enough rope to hang myself messing around with stuff. The only reason Im getting this over ryzen, is that integrated gpu. I plan to use it with the stock cooler, unless otherwise alerted to temp issues. I doubt itll see heavy use frequently. Motherboard: MSI MAG B460 TOMAHAWK ATX LGA1200 Motherboard ($203.38 @ Memory Express) Thoughts: Its the cheapest 2 16x slot board that had enough usb ports to ill out the front io... thats why I picked it Memory: G.Skill Ripjaws V Series 32 GB (2 x 16 GB) DDR4-3200 CL16 Memory ($163.83 @ Memory Express) Thoughts: Im actually only putting in 8gb, this is to swap with a different system Storage: ADATA Premier SP550 240 GB 2.5" Solid State Drive (Purchased For $0.00) Thoughts: I already have it sitting around so I might as well do something with it. I think I'll run vms or.... something on it. I really have no express purpose for having it here and might not include it, especially since the drive technology I think would make it a very poor caching drive. Case: Fractal Design Define 7 XL ATX Full Tower Case ($305.08 @ Newegg Canada) Thoughts: Big case with spots for up to 19 drives plus a couple ssds. Expensive though considering Il need to buy the trays separately, but I don't have a server rack, and likely wont need one for anything else anytime soon. Power Supply: Corsair HX Platinum 750 W 80+ Platinum Certified Fully Modular ATX Power Supply ($237.28 @ Amazon Canada) Thoughts: Highly rated, not too high inrush, many sata ports for minimal expansions needed, efficient Out of stock so replacement: Power Supply: Corsair RM (2019) 850 W 80+ Gold Certified Fully Modular ATX Power Supply ($203.38 @ Memory Express) Has 12 sata connections requiring 2 4 way sata splitters to achieve 18 total, but thats alright. External Storage: Western Digital My Book 8 TB External Hard Drive (Purchased For $0.00) Thoughts: All part of the current pool External Storage: Western Digital My Book 8 TB External Hard Drive (Purchased For $0.00) External Storage: Western Digital My Book 8 TB External Hard Drive (Purchased For $0.00) External Storage: Western Digital My Book 8 TB External Hard Drive (Purchased For $0.00) External Storage: Western Digital My Book 8 TB External Hard Drive (Purchased For $0.00) External Storage: Western Digital My Book 8 TB External Hard Drive (Purchased For $0.00) External Storage: Western Digital ELEMENTS 14 TB External Hard Drive (Purchased For $418.09) Thoughts: These 3 are new for unraid and I'll be testing the basic setup on these before transferring data. 2 are for parity External Storage: Western Digital ELEMENTS 14 TB External Hard Drive (Purchased For $418.09) External Storage: Western Digital ELEMENTS 14 TB External Hard Drive (Purchased For $418.09) Custom: Various Cables ($40.00) Thoughts: Sata extensions (non molded) and the sas to sata cables for the hba card. I also plan to use the kapton tape method for drive incompatibility since its non destructive, fairly simple, and I dont plan on moving these drives very much if at all. Custom: Unraid ($191.96) Custom: Adaptec 71605He Used off ebay ($150.00) Thoughts: This is the thing I'm shakiest about. It should have unraid support, and its great because it only takes up one x16 pcie slot, but used off ebay for a critical component is something I don't like, but everyone recommends, so I suppose Ill have to get used to that if I want to build the next nas for less than this one. Total without drives (with taxes): $1570.62 Total: $2824.89 (with taxes) Prices include shipping, taxes, and discounts when available For reference, my current data hoarding set is a bunch of external drives in a pool, with a backup pool of a similar size I was supposed to use for backup but never did with any reasonable frequency because it was too cumbersome. I imagine that this will have the net effect of making the backing up of other devices more frequent at the cost of being unable to back up the media stored, though as I said, I never did that with enough frequency to matter anyways. As for media needs, I just need to read the files I suppose. I've never found any reason to transcode them, but then I've never really used other devices to consume the media anyways or shared any of it. I also have only one used part, and that's honestly just because that's the only place I can find a 16 internal port card that people recommend. I want a reliable system that's straightforward to setup where it wont take too long to get all the parts as I'll probably be too busy soonish to put in more effort and it is the first nas Ill have built, so reliability and straightforwardness with room to expand is what I want.
  16. How can you be sure they are legitimate? The price discrepancy seems so high between new prices and those prices. The best value I could find for what looked like a good fast, reliable new card was almost 400 dollars for a Highpoint RocketRAID 2840A
  17. Hi, I've been window shopping my first nas build for a while and a potential issue I've had is with picking out an appropriate HBA card setup. For perspective, Im thinking of building in a Define 7 XL, and assuming a typical motherboard with 6 sata ports on it directly, id need 12 connected via other means to complement it to hit the 18 total potential far hard drives. What my issue is, is that Im looking for new (as in not used) sata controllers/HBA cards and im not sure which card I should really get. I've looked at the wiki and at other builds but the wiki seems to have many options that are now discontinued making me think perhaps its not updated often and with respect to other builds, people often use cards that are really expensive new and fractions of that cost used from ebay etc and that price difference scares me off. Thats why Im hoping to hopefully find a card/s I can get that will help me with this that both doesnt break the bank and has enough connections. I suppose it'd be useful to define what I mean by not breaking the bank, so I guess Il say I dont expect whatever solution be it one card or multiple to cost more than the case. As it may be relevant I am in Canada with Canadian pricing. Thanks in advance for your suggestions/input.
  18. I get that. The difference Im talking about is with the wiki is where it seems to saying that the parity data on 1 disk is practically the same use wise as the parity data of the other one which means well basically here the answer for the third picture according to the wiki would be that the data is lost from what Im getting, but that isnt the case now
  19. So then I take it the wiki doesnt match the current version? Oh I know, but backup is expensive. I plan to, but itll have to be later. For now Il have to settle with a recycle bin and a backup of only more important stuff Is this enough to just play with other apps (testing things out) or should I just start with 8 off the bat then. I dont think il end up using it as anything else, but plans change. * I mean really 4gb more is just 30 bucks more so
  20. Not necessarily motherboard specific. For that Ive just tried to search for users who use similar or the same motherboards as Im looking at (lan adapter/cpu etc). I actually found confirmation that a board I was looking at worked fine for unraid in the form of an amazon review. Things that are quite hard to be sure on for me: How does unraid dual parity work? Ive seen a post saying its P + Q but im honestly not sure what it means here. Basically I just want a confirmation that any 2 drives could fail in that event but either Im confused, the wikipedia is inaccurate or its more like you only get dual redundancy for parity drives. Then there's usb keys. Ive looked on the list of approved drives and card readers and Ive yet to find one thats not eol. That was sort of solved by looking up more modern builds and just picking one someone else has used though so I suppose its not a problem, but I must say the card reader option did look appealing. More related to hardware though, I dont really have an idea of how much power/ram etc is needed. Is the 4gb I have picked out there enough to mess around with? I see some users with a whole lot, and some with less than that. I guess actually in hindsight I could have really put that in the post, along with more specifically what I plan to do with it, but Im not even 100% sure about that ha. Essentially, if my current listed out build can max out gigabit/have no lag and maybe even run a *docker* itll be fine. I guess its possible with even more research trying to piece it together by finding a relevant post here and there Il find more information. I just was hoping itd be more plainly laid out/unified/centralized/official/quick.
  21. Im only thinking of making one server any time soon so Il probably just get a regular stick
  22. 66.7% feedback is... oof. Its also like 2-3x the cost of a decent usb stick shipping included for me =/
  23. Are there any updates? All of the listed readers seem to be eol
  24. Ive now added a third option for better drive support as Ive read the Node 804 has drive screw hole placement that doesnt work with the placement on newer large drives. I have to admit Im a bit disappointed with the lack of comments or suggestions. I have no idea whether this is reasonable, good, or has glaring incompatibility issues. Its giving me a bit of analysis paralysis. Is this good? Will this work without a fuss? I kinda wish there was maybe a discord or chat where you could just ask all the small questions that might not warrant their own thread or even an updated FAQ. Every turn there are questions I cant find definitive answers to and I havent yet found a section for complete beginners on the forum if there is one.