comet424 Posted July 12, 2018 Share Posted July 12, 2018 hi i been reading about xfs zfs btrs etc and i read that unraid doesnt support checksum error checking for something called silent errors.. but zfs does.. and that your data still may not be recoverable. even with double parity i was reading it from https://www.snapraid.it/compare but my question is is there a plugin to make sure my data is checked for integratary as i dont wanna loose data.. . do i need to use a server grade motherboard and ram.. as i currently using a asus gaming board.. i just want reliablity and i like unraid its simipler no having to map multiple shares just for 1 share.. and plex updates automaticlly freenas doesnt but the file checking bothers me anyone can explain it better.. or a plugin i need to add so my data is better... as they even mentioned flexraid i looked them up but that just confused me i like the simpliness of unraid Quote Link to comment
JorgeB Posted July 12, 2018 Share Posted July 12, 2018 11 minutes ago, comet424 said: but my question is is there a plugin to make sure my data is checked for integratary You can use btrfs or the file integrity plugin with xfs. 11 minutes ago, comet424 said: do i need to use a server grade motherboard and ram If you care about data integrity ECC RAM is a must. 1 Quote Link to comment
comet424 Posted July 12, 2018 Author Share Posted July 12, 2018 oh ok so ill need a server motherboard anyones you recommend... as i been reading zfs is better then xfs but i left freenas cuz of the pain of setting up damn jails and shares and plex would always go down at times never updates the os never updates on its own.. and i just get confused i just wanted simple and secure data integratery and i read btrfs is not a good format? and file integrity plugin does that do this silent checksum stuff.. like i have photos of my kids so i want to make sure they dont get corrupt.. and then my movies tv show collection and music and all that stuff.. so i dont want it corrupting but ill look into server grade motherboard as i only know gaming boards from asus Quote Link to comment
comet424 Posted July 12, 2018 Author Share Posted July 12, 2018 oh and kinda off topic. but would it be better i want 2 servers 1 is my main server i access and 2nd be the important data backuped a 2nd like my photos would it be like main server unraid and then 2nd server be like freenas just for secure data or like windows server 2016 or another unraid.. i only have 1 computer a asus crosshair motherboard with 16 gig ram in and a 8 core cpu thats my current unraid with several hard drives and 1 parity drive but going to get a 2nd parity drive Quote Link to comment
JorgeB Posted July 12, 2018 Share Posted July 12, 2018 Look for Supermicro server boards, latest ones from the X11 series, e.g., X11SSM, or X9/X10 series if buying used from previous generations. I use btrfs for all my unRAID servers and I find very stable and reliable for single disk usage, it can be more problematical in RAID configuration, like in the cache pool. Note that unRAID with both btrfs (or xfs with the plugin) can detect data corruption but can't fix it, you'll need to restore from backups. It's also my experience that silent data corruption, AKA bit rot, is extremely rare, but having a checksum enable filesystem or having checksums can be very useful when something happens that makes you question the data, like for example errors on another disk during a disk replacement. Quote Link to comment
comet424 Posted July 12, 2018 Author Share Posted July 12, 2018 is btrfs a good format i read it wasnt there is much info in google i get confused as they say zfs is the best.. but what i didnt like in freenas even when i moved files off the drives the drives didnt clear up disk space it was using more data got fed up and when you say single disk usage... is that mean like xfs its disk 1 2 3 and files just get copied to each one... and when do you use a cache pool.. is that that copying to the cache drive then to the disks later. i dont think i need that as its just me and my mom at home just to watch movies and save my data so what is actually this bit rot or silent data corrupt how that happen... i also make sure my unraid is on a ups... and how do you know what files corrupted to restore from what backups like a 2nd computer... does lime tech people plan on having data corruption checksums in there next version or patchs or is that a limitiation of xfs... im used to windows home server 2003 2011 for years ... so im realtivity new to freenas i tried for 6 months and now moved over using unraid.. i just wanna make sure no problems ever.. but so many options so many nas programs file formats.. gets over whelming and i did try this parity stuff i removed one of my data disks and put a blank one in .. had to reboot the computer for it to notice my hotswap drive but it rebuilt it.. which was cool i liked that but then when i read this silent data then i wondered does it tell me if it rebuilt it right from the parity drive and i guess its better to have 2 parity drives for extra security Quote Link to comment
JorgeB Posted July 12, 2018 Share Posted July 12, 2018 19 minutes ago, comet424 said: is btrfs a good format i read it wasnt there is much info in google i get confused as they say zfs is the best. IMO btrfs is good, but zfs is without a doubt superior. 19 minutes ago, comet424 said: and when you say single disk usage... unRAID doesn't stripe disks, each disk is an independent filesystem, that's why you can for example read any data disk outside the array with other Linux distros. 20 minutes ago, comet424 said: so what is actually this bit rot or silent data corrupt how that happen... Like I said, extremely rare, never happened to me, you're much more likely to get data corruption from a RAM stick going bad if not using ECC RAM. 21 minutes ago, comet424 said: does lime tech people plan on having data corruption checksums in there next version or patchs or is that a limitiation of xfs... It's limited by the filesystem used, if xfs implements checksums it will be also be available in unRAID Quote Link to comment
comet424 Posted July 12, 2018 Author Share Posted July 12, 2018 ah ok im currently in xfs mode should i be switching to btrfs i just reading alot of articles and like windows has refs didnt even know.. so it gets over whelming.. so i should stick with what i got and look for a supermicro board.. and the 1151s there 2 kinds plain and 8th generation well i seen ll51 8th by other brand and i thought unraid invented xfs is btrfs better then xfs.... i appreciate all the help so far... it does get over whelming Quote Link to comment
JorgeB Posted July 12, 2018 Share Posted July 12, 2018 7 minutes ago, comet424 said: is btrfs better then xfs.... That's always debatable, btrfs is more advanced and includes checksum and snapshots capabilities, xfs is more mature and has a functioning fsck, despite btrfs being much improved on latest kernels still lacks that, depends on your priorities. Quote Link to comment
comet424 Posted July 12, 2018 Author Share Posted July 12, 2018 my priorities is data never gets corrupt even this what i read refs from microsoft does after time magnets on the drives wear out and you loose data and i dont want that so i want data to make sure its still the right data.. i want if a hard drive goes bad it can fix itself with a new drive i like the the shares as 1 folder.. like Windows home extender from 2003 server 1 drive letter and bunch of drives made it 1 size.. so i like that i just want to make sure my files intacked and movies and tv shows always work with plex Quote Link to comment
comet424 Posted July 12, 2018 Author Share Posted July 12, 2018 and id like to be able to defrag the hard drives so they access files faster as freenas there was no defrag but windows has one Quote Link to comment
JorgeB Posted July 12, 2018 Share Posted July 12, 2018 Defrag is not really important for WORM data, but both xfs and btrfs have it. Quote Link to comment
comet424 Posted July 12, 2018 Author Share Posted July 12, 2018 not sure what worm data is but ok.. i always liked defraged so when asking files off my home server it be as fast as you can with it not all over the place.. and for my specs above that message which file format be better... i gonna pick up a hard drive today a 10tb wd gold as i having multiple old drives failing off freenas so im transfering files from it but going to start using 10TB that way i get a couple of those later when prices come down i appreciate your help... better then sometimes trying to read all the articles online as ur mind wants to explode Quote Link to comment
JorgeB Posted July 12, 2018 Share Posted July 12, 2018 12 minutes ago, comet424 said: not sure what worm data is WORM = write once, read many 13 minutes ago, comet424 said: and for my specs above that message which file format be better... Without ECC you're at an increased risk of data corruption, but btrfs is still better for data integrity, it can't protect you from writing corrupted data to the disk if it's corrupted by bad RAM, but once written it will detect any future corruption. Quote Link to comment
bman Posted July 12, 2018 Share Posted July 12, 2018 One of the biggest take-aways here should be the idea of backups. You say you do not want to lose data; the only way that happens is with rigorous, and functional backup. My own experience over the past 9 years with unRAID is losing data only because I did it unintentionally myself. That was true even before I ran backup servers, but it didn't take me long to come around to the idea that at least one backup server is a sanity saver. The backup servers I run (also unRAID, because why would I want different server types?) have proven their worth by allowing me to ignore the minutiae of things like dual parity drives, the virtues of ZFS over BTRFS or XFS, and even server-grade hardware. Even without ECC RAM it has been very rare that I have needed to retrieve any files from backup (perhaps half a dozen times). That said, I have settled on (and would recommend) ECC RAM, enterprise-grade hard drives, a battery backup for safe server shutdown, and quality drive chassis choices and cabling (like Supermicro 24-bay hot swap cases) for best reliability. I also perform regular parity checks to ensure all drives get exercised, I use the File Integrity plugin (my drives use XFS) and I run at least 3 Preclear cycles on all unknown/new drives. The headache of having drives just days old go belly-up because I was too hasty to preclear it first has made me a believer in this practice. Quote Link to comment
pwm Posted July 13, 2018 Share Posted July 13, 2018 18 hours ago, comet424 said: would it be like main server unraid and then 2nd server be like freenas just for secure data Note that checksummed file systems (or file integrity plugins) are great for detecting errors. And server boards with ECC are great at reducing the probability of file corruption during file copying. But the really important part is that you always need to store your important files on multiple storage media and on multiple physical locations. RAID, parity, ECC, checksums etc will never ever replace the need for backup. But the checksums means that after a silent corruption, you can figure out if it's the file on the main server or on the backup server that has the correct copy of the file. 18 hours ago, comet424 said: does lime tech people plan on having data corruption checksums in there next version or patchs or is that a limitiation of xfs... File integrity plugins can be used for archival-type files. But for dynamic data that is regularly changed, it doesn't work well with plugin-based file integrity - only checksumming within the file system can work really well with dynamic data. It is the file system itself that knows the actual meaning of all the bytes that represents the meta-data and the internal data structures that describes the content of the disk and what blocks belongs to a file and what blocks are free for reuse. 17 hours ago, comet424 said: and id like to be able to defrag the hard drives so they access files faster as freenas there was no defrag but windows has one The need for defrag software depends on used file system. Originally, Microsoft didn't have any defrag for NTFS because they claimed there was no need. It was quickly concluded that Microsoft was wrong. So they had to release a defrag software. Some other, more modern, OS have better algorithms for positioning the data, reducing the need for defrag. And some file systems perform on-the-fly defrag. So it isn't obvious that the available of a defrag program should be a selection choice when selecting what file system to use. Quote Link to comment
comet424 Posted July 16, 2018 Author Share Posted July 16, 2018 not sure what worm data is but ok.. i always liked defraged so when asking files off my home server it be as fast as you can with it not all over the place.. and for my specs above that message which file format be better... i gonna pick up a hard drive today a 10tb wd gold as i having multiple old drives failing off freenas so im transfering files from it but going to start using 10TB that way i get a couple of those later when prices come down i appreciate your help... better then sometimes trying to read all the articles online as ur mind wants to explode Quote Link to comment
comet424 Posted July 16, 2018 Author Share Posted July 16, 2018 i see my postings didnt work it just posted now im using my old Acer H340 4 bay for file backup i going to use my asus rog board as the main server i bought 2 wd 12TB gold drives. just backed it up all the files off corrupted drives from freenas to it i running btrs with a 6 and 2tb drive copying to it. i not sure what you mean the preclear cycles whats that and there was another question i had i forgot and do you guyts understand what the logs mean as i get the triangle on several drives and then its hight lighted too if you scroll down on each drive Quote Link to comment
comet424 Posted July 16, 2018 Author Share Posted July 16, 2018 is the preclear cycles the extended smart scan as i have done that a few times on each drive Quote Link to comment
bman Posted July 17, 2018 Share Posted July 17, 2018 preclear is a script you can run on your unRAID server to read from and write to every sector on a hard drive, then read again to verify the writes happened properly. It is an optional step in preparing a drive for use with a server (I preclear all new drives whether for unRAID or not). The original purpose of the script is now irrelevant, but it was also very useful for weeding out weak hard drives before you entrusted them with your data, which presumably you do not wish to lose. By forcing a drive to read and write every sector multiple times (I use a cycle count of 3 - for 12TB drives that would likely take a week) you are forcing the drive to find any sectors that are perhaps bad from the factory. It's a burn-in test. For me it has found a few drives that I was able to return within the 15 days my local computer shop allows me to walk in and replace a failed drive with them. After that time period I'd have to deal with the manufacturer's RMA process which takes much longer. It's also useful for assessing the relative health of a drive that has been used before. It erases everything on the drive, so it's something you only do before or in between "uses". preclear is unrelated to SMART data and testing except to say that SMART counters will definitely be altered after preclear has been run. SMART test results after the completion of preclear can be decent indicators of whether you can rely on your drive to keep your data safe, but nothing is foolproof. SMART data will never reveal the full story: Almost regardless of the SMART data, your drive may live a very long life, or it may fail a week later without any hint of trouble beforehand. The majority of the time (75%, give or take, if you believe Backblaze data on the subject) SMART results are good enough to determine the relative health of a hard drive. Quote Link to comment
comet424 Posted July 17, 2018 Author Share Posted July 17, 2018 ah ok well right now I running just a smart test on the 1 12tb drive the other one I moving the data back off the 12tb drive to to my red 6tb pro and a black 2tb drive that didn't fail any smart checks.. as if I scroll down on some of these I get uncorrect check etc they highlighted there is one drive I dunno if you know what all it means reported a lot in its logs.. but as for this test.. I can still do the test is this script somewhere in unraid.. or I gotta download it.. id run both 12tbs at the same time and not sure what backblaze data is I still new to all this so used to windows ntfs lol but id rather test the drives before I loose data.. and I using the brts like your running took a while to figure where it was to set it. Quote Link to comment
comet424 Posted July 17, 2018 Author Share Posted July 17, 2018 found the pre clear pluggin so I doing a 3 cycle pre clear on my one computer be slow since its only a sata 2 not sata 3 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.