question about error checking

[comet424]

hi 

i been reading  about xfs zfs btrs etc  and 

i read  that unraid doesnt support checksum error checking for something called silent errors.. but zfs does..

and that your data still may not be recoverable. even with double parity 

i was reading it from https://www.snapraid.it/compare

 

but my question is is there a plugin  to make sure my data is checked for integratary  as i dont wanna loose data.. .  do i need to use a server grade motherboard and ram.. as i currently  using a asus gaming board..  i just want reliablity  and i like unraid its simipler  no having to map multiple shares just for 1 share.. and plex updates automaticlly  freenas doesnt 

but the file checking bothers me  anyone can explain it better..

or a plugin i need to add so my data is better... as they even mentioned flexraid i looked them up but that just confused me  i like the simpliness of unraid

[JorgeB]

11 minutes ago, comet424 said:

but my question is is there a plugin  to make sure my data is checked for integratary

You can use btrfs or the file integrity plugin with xfs.

 

11 minutes ago, comet424 said:

do i need to use a server grade motherboard and ram

If you care about data integrity ECC RAM is a must.

[comet424]

oh ok so ill need a server motherboard  anyones you recommend... as i been reading zfs is better then xfs  but i left freenas cuz of the pain of setting up damn jails and shares  and plex would always go down at times never updates  the os never updates on its own..  

and i just get confused  i just wanted simple  and secure data integratery  and i read btrfs is not a good format?   and file integrity plugin  does that do this silent checksum stuff..  like i have photos of my kids so i want to make sure they dont get corrupt..  and then my movies tv show collection  and music and all that stuff..  so i dont want it corrupting

 

but ill look into server grade motherboard as i only know  gaming boards from asus

[comet424]

oh and kinda off topic.  

but would it be better  i want 2 servers 1 is my main server i access and 2nd  be the important data  backuped a 2nd like my photos

would it be like  main server unraid and then 2nd server be like freenas  just for secure data      or like windows server 2016 or another unraid..

 

i only have 1 computer  a asus  crosshair motherboard with 16 gig ram in  and a 8 core cpu  thats my current unraid    with several hard drives and 1 parity drive but going to get a 2nd parity drive

[JorgeB]

Look for Supermicro server boards, latest ones from the X11 series, e.g., X11SSM, or X9/X10 series if buying used from previous generations.

 

I use btrfs for all my unRAID servers and I find very stable and reliable for single disk usage, it can be more problematical in RAID configuration, like in the cache pool.

 

Note that unRAID with both btrfs (or xfs with the plugin) can detect data corruption but can't fix it, you'll need to restore from backups.

 

It's also my experience that silent data corruption, AKA bit rot, is extremely rare, but having a checksum enable filesystem or having checksums can be very useful when something happens that makes you question the data, like for example errors on another disk during a disk replacement.

 

 

[comet424]

is btrfs a good format i read it wasnt there is much info in google i get confused as they say zfs is the best.. but what i didnt like in freenas  even when i moved files off the drives the drives didnt clear up disk space it was using more data   got fed up

 

and when you say single disk usage...  is that mean  like xfs    its disk 1  2 3 and files just get copied to each one...  and when do  you use a cache pool.. is that that  copying to the cache drive then to the disks later.  i dont think  i need that as its just me  and my mom  at home  just to watch movies and save my data   

so what is actually this bit rot or silent data corrupt how that happen... i also make sure my unraid is on a ups...  and how do you know what files corrupted to restore from what backups  like a 2nd computer...

does  lime tech people plan on having data corruption checksums in there next version or patchs  or is that a limitiation of xfs...    im used to windows home server 2003 2011  for years ... so im realtivity new to freenas i tried for 6 months and now moved over  using unraid..   i just wanna make sure no problems ever..  but so many options   so many nas programs   file formats..  gets over whelming 

and i did try this parity stuff  i removed one of my data disks and put a blank one in  .. had to reboot the computer for it to notice my hotswap drive  but it rebuilt it.. which was cool i liked that  but then when i read this silent data then i wondered does it tell me if it rebuilt it right from the parity drive

 

and i guess its better to have 2 parity drives for extra security

 

[JorgeB]

19 minutes ago, comet424 said:

is btrfs a good format i read it wasnt there is much info in google i get confused as they say zfs is the best.

IMO btrfs is good, but zfs is without a doubt superior.

 

19 minutes ago, comet424 said:

and when you say single disk usage...

unRAID doesn't stripe disks, each disk is an independent filesystem, that's why you can for example read any data disk outside the array with other Linux distros.

 

20 minutes ago, comet424 said:

so what is actually this bit rot or silent data corrupt how that happen...

Like I said, extremely rare, never happened to me, you're much more likely to get data corruption from a RAM stick going bad if not using ECC RAM.

 

21 minutes ago, comet424 said:

does  lime tech people plan on having data corruption checksums in there next version or patchs  or is that a limitiation of xfs...

It's limited by the filesystem used, if xfs implements checksums it will be also be available in unRAID

 

 

[comet424]

ah ok  im currently  in xfs mode should i be switching to btrfs    

i just reading alot of articles  and like windows has refs  didnt even know..  so it gets over whelming..  

so i should stick with what i got and look for a supermicro board.. and the 1151s there 2 kinds plain and 8th generation  well  i seen  ll51 8th by other brand

 

and i thought unraid invented xfs

is btrfs  better then xfs....   

 

i appreciate all the help so far... it does get over whelming 

[JorgeB]

7 minutes ago, comet424 said:

is btrfs  better then xfs....   

That's always debatable, btrfs is more advanced and includes checksum and snapshots capabilities, xfs is more mature and has a functioning fsck, despite btrfs being much improved on latest kernels still lacks that, depends on your priorities.

 

 

[comet424]

my priorities is  data never gets corrupt  even this what i read refs from microsoft does  after time  magnets on the drives wear out and you loose data and i dont want that

 

so i want data to make sure its still the right data..

i want if a hard drive goes bad it can fix itself with a new drive

i like the the shares  as 1 folder.. like Windows home extender from 2003 server  1 drive letter  and bunch of drives made it 1 size..  

so i like that i just want to make sure my files intacked and movies and tv shows always work with plex

 

[comet424]

and id like to be able to defrag the hard drives so they access files faster as freenas there was no defrag  but windows has one

[JorgeB]

Defrag is not really important for WORM data, but both xfs and btrfs have it.

[comet424]

not sure what worm data is  but ok.. i always liked defraged so when asking files  off my home server it be as fast as you can  with it not all over the place..

and for my specs  above that message which file format be better...

i gonna pick up a hard drive today  a 10tb wd gold  as i having multiple old drives failing off freenas  so im transfering files from it but going to start using 10TB  that way i get a couple of those later when prices come down  

 

i appreciate your help...  better then sometimes trying to read all the articles online as  ur mind wants to explode

[JorgeB]

12 minutes ago, comet424 said:

not sure what worm data is

WORM = write once, read many

 

13 minutes ago, comet424 said:

and for my specs  above that message which file format be better...

Without ECC you're at an increased risk of data corruption, but btrfs is still better for data integrity, it can't protect you from writing corrupted data to the disk if it's corrupted by bad RAM, but once written it will detect any future corruption.

[bman]

One of the biggest take-aways here should be the idea of backups.  You say you do not want to lose data; the only way that happens is with rigorous, and functional backup.

 

My own experience over the past 9 years with unRAID is losing data only because I did it unintentionally myself. That was true even before I ran backup servers, but it didn't take me long to come around to the idea that at least one backup server is a sanity saver.

 

The backup servers I run (also unRAID, because why would I want different server types?) have proven their worth by allowing me to ignore the minutiae of things like dual parity drives, the virtues of ZFS over BTRFS or XFS, and even server-grade hardware.  Even without ECC RAM it has been very rare that I have needed to retrieve any files from backup (perhaps half a dozen times).

 

That said, I have settled on (and would recommend) ECC RAM, enterprise-grade hard drives, a battery backup for safe server shutdown, and quality drive chassis choices and cabling (like Supermicro 24-bay hot swap cases) for best reliability. 

 

I also perform regular parity checks to ensure all drives get exercised, I use the File Integrity plugin (my drives use XFS) and I run at least 3 Preclear  cycles on all unknown/new drives. The headache of having drives just days old go belly-up because I was too hasty to preclear it first has made me a believer in this practice.

 

 

[pwm]

18 hours ago, comet424 said:

would it be like  main server unraid and then 2nd server be like freenas  just for secure data

Note that checksummed file systems (or file integrity plugins) are great for detecting errors.

And server boards with ECC are great at reducing the probability of file corruption during file copying.

But the really important part is that you always need to store your important files on multiple storage media and on multiple physical locations. RAID, parity, ECC, checksums etc will never ever replace the need for backup. But the checksums means that after a silent corruption, you can figure out if it's the file on the main server or on the backup server that has the correct copy of the file.

 

18 hours ago, comet424 said:

does  lime tech people plan on having data corruption checksums in there next version or patchs  or is that a limitiation of xfs...

 

File integrity plugins can be used for archival-type files. But for dynamic data that is regularly changed, it doesn't work well with plugin-based file integrity - only checksumming within the file system can work really well with dynamic data. It is the file system itself that knows the actual meaning of all the bytes that represents the meta-data and the internal data structures that describes the content of the disk and what blocks belongs to a file and what blocks are free for reuse.

 

17 hours ago, comet424 said:

and id like to be able to defrag the hard drives so they access files faster as freenas there was no defrag  but windows has one

The need for defrag software depends on used file system. Originally, Microsoft didn't have any defrag for NTFS because they claimed there was no need. It was quickly concluded that Microsoft was wrong. So they had to release a defrag software.

Some other, more modern, OS have better algorithms for positioning the data, reducing the need for defrag. And some file systems perform on-the-fly defrag. So it isn't obvious that the available of a defrag program should be a selection choice when selecting what file system to use.

[comet424]

not sure what worm data is  but ok.. i always liked defraged so when asking files  off my home server it be as fast as you can  with it not all over the place..

and for my specs  above that message which file format be better...

i gonna pick up a hard drive today  a 10tb wd gold  as i having multiple old drives failing off freenas  so im transfering files from it but going to start using 10TB  that way i get a couple of those later when prices come down  

 

i appreciate your help...  better then sometimes trying to read all the articles online as  ur mind wants to explode

[comet424]

i see my postings didnt work it just posted now

 

im using my old Acer H340 4 bay for file backup  i going to use my asus rog board as the main server

i bought 2 wd 12TB gold drives.

just backed it up all the files off corrupted drives from freenas to it

 

i running btrs with a 6 and 2tb drive  copying to it.

i not sure what you mean the preclear cycles whats that  

and there was another question  i had i forgot

and do you guyts understand what the logs mean  as i get the triangle on several drives  and then its hight lighted too if you scroll down on each drive

 

[comet424]

is the preclear  cycles

 the extended smart scan  as i have done that a few times on each drive  

[bman]

preclear is a script you can run on your unRAID server to read from and write to every sector on a hard drive, then read again to verify the writes happened properly. 

 

It is an optional step in preparing a drive for use with a server (I preclear all new drives whether for unRAID or not). The original purpose of the script is now irrelevant, but it was also very useful for weeding out weak hard drives before you entrusted them with your data, which presumably you do not wish to lose.

 

By forcing a drive to read and write every sector multiple times (I use a cycle count of 3 - for 12TB drives that would likely take a week) you are forcing the drive to find any sectors that are perhaps bad from the factory. It's a burn-in test.  For me it has found a few drives that I was able to return within the 15 days my local computer shop allows me to walk in and replace a failed drive with them. After that time period I'd have to deal with the manufacturer's RMA process which takes much longer.

 

It's also useful for assessing the relative health of a drive that has been used before.  It erases everything on the drive, so it's something you only do before or in between "uses".

 

preclear is unrelated to SMART data and testing except to say that SMART counters will definitely be altered after preclear has been run.  SMART test results after the completion of preclear can be decent indicators of whether you can rely on your drive to keep your data safe, but nothing is foolproof. 

 

SMART data will never reveal the full story: Almost regardless of the SMART data, your drive may live a very long life, or it may fail a week later without any hint of trouble beforehand. The majority of the time (75%, give or take, if you believe Backblaze data on the subject) SMART results are good enough to determine the relative health of a hard drive.

 

[comet424]

ah ok  well right now I running just a smart test on the 1 12tb drive  the other one I moving the data back off the 12tb  drive to to my red 6tb pro and a black 2tb drive  that didn't fail any smart checks.. as if I scroll down on some of these I get uncorrect   check etc   they highlighted  

there is one drive I dunno if you know what all it means reported a lot in its logs..

 

but as for this test.. I can still do the test is this script somewhere in  unraid..  or I gotta download it..  id run both 12tbs at the same time 

and not sure what backblaze  data is  I still new to all this  so used to windows ntfs  lol

 

but id rather test the drives before I loose data.. and I using the brts like your running took a while to figure where it was to set it.

[comet424]

found the pre clear pluggin  so I doing a 3 cycle pre clear  on my one computer  be slow since its only a sata 2  not sata 3