Hello, I have been using Unraid NAS for a couple of years now with three 3TB drives. Was working fine all this time. I then had to upgrade to three 8TB drives which I have done. Took about a week and a half to let all the drives rebuild. Then I first had upgraded to the latest v6.2 Unraid version. Before I replaced each drive I ran the preclear on all of them successfully.Then I removed my original Seagate 3TB Parity drive and replaced it with a new larger Seagate Enterprise NAS 8TB drive and let it rebuild the Parity drive.

After it reported successful completion of rebuilding the Parity drive, I proceeded to remove and replace Disk1 and let it rebuild. It reported successful completion also.  Then I proceeded to remove and replace my final Disk2 and let it rebuild. It too reported successful completion.  I didn't think anything more of it and thought all was successful. File listings showed all my original directories and files all there on each respective drive.  So I thought cool all went well. NOT!

Today, I just noticed that when I go to pull my .pdf documents from the Unraid drives, that just about all of them will not open in Adobe Reader and report that the file may be corrupted! And indeed they all are.  This perturbs me immensely.  I am a paid subscriber to Unraid and thought it a reliable NAS system.  Now I have my doubts and concerns as to its integrity and reliability. 

Luckily I have most of my .pdf files backed up and stored on my other PC's and other older Netgear ReadyNAS NV+ server disks and can recover them.

 

I recopied these .pdf files from my Windows PC over to the new Unraid disks to restore them and see if they were written properly. They were. So if I copy and add new files to the Unraid disks they are all written okay and recoverable.  It's just that the UNRAID drive replacement procedure and rebuilding the disks FAILED to work properly when I replaced the hard disks.  This is a serious serious bug as far as I am concerned and I want to report it here in case anyone else has the same issue.

 

I have no idea how many other files might also be corrupted during the rebuild process.  So far I've noticed all my .pdf documents won't open. I've tried playing some of my many video mp4 files and they seem to be intact and play. I am concerned about other files types that might surprise me one day when I try to use them now... too many to list like (.exe, .doc, etc.).

Anyone else have this problem after replacing hard drives and letting Unraid rebuild them from Parity drive? 

 

Oh, No! I just tried checking some .zip and .exe files and they too are all corrupt! BAD!

Oddly enough, I can open .txt and .docx documents.

I'm really sorry you have had this trouble.

 

Someone can correct me, but I've never heard of anyone else having something like this happen to them.  And it's hard to imagine what could have happened.  Try running the reiserfsck check on your data drives (Check Disk File systems), see if it turns up anything.  I suspect it will, although I don't know how it could have happened.  If it does, then there's a command that may restore everything (or almost everything) back to normal.  It will take awhile though.

 

Do corrupted files appear on both data drives?

 

It may be useful to see both current diagnostics and any diagnostics or syslogs saved that cover the periods of the rebuilds.

1. Did you run any of the 6.0-beta's?  In particular a silent data corruption bug snuck into the linux kernel used in unRAID OS version 6.0-beta7/8 around September of 2014:

http://lime-technology.com/forum/index.php?topic=35161.0

 

If your array devices are ReiserFS and you might have installed one of these versions, it's possible the corruption orriginated there.

 

2. Do you have any of your original 3TB drives that you can compare to see if corruption is on those drives?

Hi, I did not use any Unraid beta versions.  Never have. All the drives were default Unraid ReiserFS format. I have not run any of the diagnostics on them.

I have two of my original 3TB Unraid drives untouched since I removed them (the Parity drive and the Disk1 drives). I saved them just in case the new 8TB drives were bad.  The 8TB drives are not bad and are working fine.

I have already begun overwriting back my files on the 8TB Disk1 to restore them... So I don't think running a diagnostics on Disk1 will give a true indication of its state when all the files were corrupted.  Also, Yes, the new 8TB disk2 also has the same file corruptions after it was rebuilt from Parity drive.

Can I just mount my old 3TB Disk1 and Unraid will read it to check if the files can be read from the original files?

You say that the corruption appears to be related to file type (pdf, zip, exe are corrupted; mp4, txt, docu are not corrupted).    I find it strange if the corruption is related to the file type.  More likely it is related to file size (a larger file being more likely to be corrupted), and/or its location (are the corrupted files all on the same data disk?)

 

As has already been suggested you really should at this stage run a ReiserFS check on your data disks before copying more data to them, as if there is corruption in the file system you should correct that before writing new/updated files to them.  For ReiserFS this can take a while to complete the check.

 

Keep your old Parity and Data1 disk to one side until you have advice on what to do with them .... What have you done with Data2 disk?  Can it be freed up ...... There is a possibility of rebuilding the old Data2 disk using the other 2 disks if it proves necessary to see if the data corruption occurred there provided you make no changes what so ever to the old Parity and Data1 disk.  Wait for more advice from other experienced people before doing anything with the original drives.

 

Also, what version of Unraid did you upgrade from?

 

I also suggest running a memory check from the boot screen .... In case the corruption was introduced by memory corruption whilst rebuilding the new disks, if the newer unraid is using memory that previously was not being accessed.

 

Whatever you do, don't rush into making changes unless you are sure you need to ... Get as much advice as possible so that you don't reduce your recovery options.

 

How experienced are you at running Linux commands, as this may effect what advice people give you to try out recovery options?

* Since it seems much more likely that there is file system corruption than that the files are corrupted, I want to emphasize the importance of first running a file system check on both of the disks, before copying anything else to them, just as remotevisitor said.

 

* I completely forgot about your original 3TB drives, so that means you haven't lost anything.  You could at any time just put them back online, do a New Config, reassign them, and start the array with "Parity is already valid".  You would be back where you started, but this time we could better monitor the process, determine what goes wrong and when.

 

* There is no possibility that I know of that the actual rebuild operation could have corrupted the files or the file system, as it is a bit to bit exact copy.  I believe the process is - copy all of the bits of the 3TB drive to the 8TB drive exactly, then copy zeroes to the rest of the 8TB drive, then expand the file system from 3TB to 8TB.  Only the last step offers any chance of corruption, and there aren't any reports of such that I know of.

 

* The possibilities for corruption seem to me to be the following:

- it was already corrupted (we'll want to check files on the 3TB drives, and do a file system check of them)

- the parity drive was corrupted somehow, perhaps when it was rebuilt?  (seems extremely unlikely, but you could do a parity check)

- the expansion of the file system from 3TB to 8TB somehow failed (a file system check should show issues)

- some other agent (malware or bad software of some kind) has caused the corruption

- memory issues, faulty RAM (run Memtest)

- user/disk copying bug - a file destroying bug caused by copies between a user share and disk involved in the share (creates the files with zero bytes I believe, something you could check)

 

* Other than the last one (the user/disk copy bug), there aren't any reports of this happening, so I don't as of yet believe this is an unRAID bug.

 

* You don't have to listen to me, but my suggested course of action would be to -

- Install the Unassigned Devices plugin.

- Install the original 3TB Disk 1.

- Mount it read-only with Unassigned Devices, and check its files for correctness.

- Unmount it, then use Check Disk File systems to do a file system check on that drive.

- Repeat the above with the original Disk 2 (install it, mount and inspect it, then check its file system).

- Based on what you find above (files already corrupt, or file system corruption), the following instructions may need revision.

- Do the New Config procedure, then assign the 8TB parity drive and the 2 3TB data drives.

- Start the array and let parity build again.

- Do a parity check to verify parity is fine.

- Optional but do a last verification that files are correct.

- Unassign Disk 1, start the array, stop it, assign an 8TB as Disk 1, and start the array again to rebuild Disk 1.

- Check the files on Disk 1...

 

* As remotevisitor said though, probably the very first operation should be to make absolutely sure there are no RAM issues.  Please run the built in MemTest from the unRAID boot menu, at least several passes, preferably over night.  It's probably not a memory issue, but it's an easy test, and you always want to make sure your RAM is perfect, can be completely trusted, before you waste hours on other tests.

From what wyee said I believe he currently only has the original Parity and Data1 disk .... He has not responded on whether he has the original Data2 disk available (unmodified, or changed but available for a rebuild)

 

I therefore think at this stage there is the possibility of rebuilding the original Data2 disk if necessary provided no changes are made to the original Parity & Data1 disk.  So while I agree the mounting Disk1 read-only is OK, and running a file system check to see if there was any file corruption on the original Data1 disk, it is important to not perform any action to correct any corruption on the original Data1 disk at this stage.

 

Hi, I appreciate everyone's suggestions above. I don't have the time to go through all these long time consuming procedural checks right now but will keep it all in mind.  No I do not have the original Disk2 drive in its original Unraid data contents or format as I have installed it in another Windows 10 PC already and reformatted it NTFS for Windows use.  I do have the original 3TB Parity and Disk1 drives untouched. I am myself an IT professional software developer and I am familiar with Linux command line commands. I have not run any diagnostics disk checking commands on these drives as I didn't want to affect any changes on them just in case. I have not run RAM memory tests either.  I have 16GB RAM installed on an ASUS H87I-Plus mini-ITX motherboard with an Intel I5 CPU.  I will one day go through all this down time testing and verification of the old 3TB disks when I can afford to.

In the meantime, I have already begun and copied restored all of my critical files back onto the Unraid 8TB Disk1 from my other backup copies and all files once copied back over to Unraid are opening and reading back fine.

Yes, to the question asked that only certain file types seem to be corrupted (.pdf, .exe, .tar, .zip, .png) while other file types seem to have not been corrupted like (.doc, .txt, .ini, .java, .log).  Also, NO file size does not matter on which files were corrupted.  They are all different sizes both very small and very large. And yes, both new 8TB drives (Disk1 and Disk2) had the same kind of file corruptions all over the drive in every different folder and subfolders. It does not matter where on the drive the files were placed.  I don't recall exactly what previous version of Unraid was running before I had let it update to v6.2.x versions.  I am pretty sure it was version unRAIDServer-5.0.6-i386. Hope this info helps in any further diagnosis ideas.

Also if it matters, I had these plugins and apps installed, Dynamix webGui, Nerd Tools, Preclear Disks, docker, plex.

I can't remember off hand whether I had installed the "Nerd Tools" and "Preclear Disks" plugins after removing the old 3TB Parity Drive or before I removed it and installed the new 8TB Parity Drive in order to run preclear on the drives.  I am wondering if this might have had the corruption effect possibly or if it should not have mattered.

If it is only certain file types that have been corrupted then I would think it is almost certainly nothing to do with unRAID (which is not file type aware).  Sounds more like the type of effects one gets from ransomware or other malignant software.  If that is the case one will definitely want to identify the source as otherwise it is likely to occur again.

Nothing you have mentioned has been reported as ever causing anyone any data corruption, that I know of.  But I have to say, from the file types you had corrupted, and from the textual file types that weren't corrupted (except for .doc), this sounds very much like the effects of a ransomware attack.  You really need to investigate that further.  Perhaps you have or have had an infected machine on your network.

 

As an IT professional, you surely would not want to use any file system that was suspect.  PLEASE check the file systems on your data drives!  You mention how long it might take, I assume you sleep at night, right?    You can start a test then!  And these file system checks don't take that long.  As an IT professional, you've been running Check Disks and scandisks all your life.

I ran the parity check last evening and it completed in about 12 hours 45 minutes with 0 errors this morning.

I am running the reiserfschk (read only mode) now on my 8TB data disks. 

Data Disk2 report completed with 0 errors and no corruptions.  So the disk seems fine.

I am now running the check on Disk1 and I suspect it too will be fine.  This tells me that the Parity Disk that was built from the original 3TB Disk1 and Disk2 drives image is probably corrupt. 

I will have to reinstall those old 3TB Parity drive and Disk1 drive to check and see if they had a corruption situation before I replaced them.

Update: Disk1 reiserfschk just finished and reports no errors nor corruptions.  All disks are fine.

This tells me that whaterver image that got built to the new 8TB Parity drive is most likely corrupt.

I looked at several of the corrupt files on my Disk2 drive using a hex editor and many of the files that can't be opened contain all zeros (0)! I wonder if that routine to zero out the rest of the drive during disk rebuild might have a bug.

I ran the parity check last evening and it completed in about 12 hours 45 minutes with 0 errors this morning.

I am running the reiserfschk (read only mode) now on my 8TB data disks. 

Data Disk2 report completed with 0 errors and no corruptions.  So the disk seems fine.

I am now running the check on Disk1 and I suspect it too will be fine.  This tells me that the Parity Disk that was built from the original 3TB Disk1 and Disk2 drives image is probably corrupt. 

I will have to reinstall those old 3TB Parity drive and Disk1 drive to check and see if they had a corruption situation before I replaced them.

Update: Disk1 reiserfschk just finished and reports no errors nor corruptions.  All disks are fine.

This tells me that whaterver image that got built to the new 8TB Parity drive is most likely corrupt.

I looked at several of the corrupt files on my Disk2 drive using a hex editor and many of the files that can't be opened contain all zeros (0)! I wonder if that routine to zero out the rest of the drive during disk rebuild might have a bug.

i think you misunderstand how parity works (there is a good write-up in the online documentation).  The parity disk does not have any data in the normal sense, it just contains sectors that are a bit Level xor of the corresponding sectors on the data drives.

I wonder if that routine to zero out the rest of the drive during disk rebuild might have a bug.

 

Very doubtful, rebuilding a disk is one the most used functions of unRAID, be it to replace a bad disk or upgrade to a bigger one, I myself have probably done more than 100 between all my servers, all my files are checksummed, never had a single corruption.

 

Did you used to do regular parity checks?

I just noticed that on my Disk2 drive that many other file types I thought were unaffected are indeed corrupted too.  That is I just found that my .txt, .jpg and .doc are all corrupted also on this drive!  So it seems to not be limited to file types that got corrupted.

 

I do understand how the Parity drive is backing up the disk drives.  What I was trying to convey in earlier statement is the part that someone mentioned that the Parity rebuilds the data bit for bit on the new drive(s) and then it zeroes out the remaining space on the larger drives.  So I was talking about that zeroing out part of the process... which might have somehow lost track of the indexing and zeroed out all these corrupt files.  I say that because when I use my hex editior to view these corrupt files all of them contain all zeroes for their contents! Bug?

Can it be something with unRAID upgrading from old 3TB drives to new larger 8TB drives and losing track of the index spacing somehow?

Anyone else actually did what I did moving from 3TB drives to 8TB drives? Just wondering.

I just noticed that on my Disk2 drive that many other file types I thought were unaffected are indeed corrupted too.  That is I just found that my .txt, .jpg and .doc are all corrupted also on this drive!  So it seems to not be limited to file types that got corrupted.

 

I do understand how the Parity drive is backing up the disk drives.  What I was trying to convey in earlier statement is the part that someone mentioned that the Parity rebuilds the data bit for bit on the new drive(s) and then it zeroes out the remaining space on the larger drives.  So I was talking about that zeroing out part of the process... which might have somehow lost track of the indexing and zeroed out all these corrupt files.  I say that because when I use my hex editior to view these corrupt files all of them contain all zeroes for their contents! Bug?

Can it be something with unRAID upgrading from old 3TB drives to new larger 8TB drives and losing track of the index spacing somehow?

Anyone else actually did what I did moving from 3TB drives to 8TB drives? Just wondering.

 

No.

 

The thing is, unRAID is not really zeroing the added space in the way you think, it is just filling the whole drive based on parity and the other drives. There is no separate zeroing process or indexing for such process.

 

In any case, it is extremely unlikely that unRAID writing zeros to lots of wrong places on drives would only affect file contents and not mess up the filesystems at all. I would mount the old intact drive as read-only and verify some of the corrupted files.

Okay, I am beginning to think that the original 3TB drives were corrupted somehow now.  I will try to mount the old 3TB disk1 drive to see if those files are okay or also corrupted before the rebuild. If they are corrupted, then I still have a concern on how unRAID corrupted all those files when the unRAID NAS is hardly ever used short of occasional PLEX server access to my shares and occasional writing some files to the unRAID disk1 drive.  As far as I can tell, the 3TB drives setup was working fine before I performed the drive upgrades to 8TB drives.  Something along the way might have caused the 3TB drive corruptions and I don't know what step along the way might have done that.  I run no apps or software on this server at all other than PLEX.

I am wondering if letting the unRAID GUI perform the automatic software updates including unRAID new version updates can cause disk corruptions due to some incompatibile setting between versions that could have occurred.

Question: to mount my old 3TB drive along side my 8TB array drives, do I just plug the 3TB drive into a spare SATA3 port and mount it from the GUI or must I issue the CLI command from console terminal to mount it outside the array in order to view its contents?

You are not really doing anything to troubleshoot this.

 

Please review the advice RobJ gave in reply #5 in this thread.

What do you mean I am not doing anything to troubleshoot this? I am performing the disk checks and parity verification checks on the current 8TB drives and all passed okay. Then I want to now mount the old 3TB disk1 drive to verify if those files are intact on that original drive and also run the reiserfschk on it. I am doing these checks a step at a time...

In looking at the suggestions in post #5, one item does stand out to me that might have caused the file corruptions.  That is where it says this:

 

- user/disk copying bug - a file destroying bug caused by copies between a user share and disk involved in the share (creates the files with zero bytes I believe, something you could check)

 

* Other than the last one (the user/disk copy bug), there aren't any reports of this happening, so I don't as of yet believe this is an unRAID bug.

 

As I have mentioned I do have PLEX Server installed and it has defined shares of several of my disk1 directories with the same directory names. Some files  that were corrupted are in some of those shared directories but then other corrupted files were not at all in a named shared directory.

 

Anyway, I am about to mount the old 3TB disk1 drive and see how it looks.

 

If you hit the "User Share Copy bug", then the symptoms are that you end up with zero-length files rather than files that contain zeroes and are not zero length.. 

 

As was mentioned the only known case that would cause corruption where files ended up containing zeroes was one of he early 6.0 betas which had a ReiserFS bug in the Linux ReiserFS driver that could silently corrupt files in this manner.  However you indicated that you never ran any of these betas so that should not be the cause.

What do you mean I am not doing anything to troubleshoot this?

 

From your posts it is easy to get the impression that you already have decided that it is a bug in unRAID, and that you are focusing your attention on making everything fit your idea of what the bug is rather than on figuring out what the problem is.

 

I am performing the disk checks and parity verification checks on the current 8TB drives and all passed okay. Then I want to now mount the old 3TB disk1 drive to verify if those files are intact on that original drive and also run the reiserfschk on it. I am doing these checks a step at a time...

 

I would mount the disk as read-only and check the files. I would absolutely not run any correcting checks on the disk, or anything else that writes to it. If the check on the rebuilt drives passed I would expect the originals to pass as well.

 

In looking at the suggestions in post #5, one item does stand out to me that might have caused the file corruptions.  That is where it says this:

 

- user/disk copying bug - a file destroying bug caused by copies between a user share and disk involved in the share (creates the files with zero bytes I believe, something you could check)

 

* Other than the last one (the user/disk copy bug), there aren't any reports of this happening, so I don't as of yet believe this is an unRAID bug.

 

As I have mentioned I do have PLEX Server installed and it has defined shares of several of my disk1 directories with the same directory names. Some files  that were corrupted are in some of those shared directories but then other corrupted files were not at all in a named shared directory.

 

The last paragraph probably could to be rephrased and clarified.

Okay, so I have connected my old #TB disk1 drive to a free sata3 port now and stopped the array.  In the webGUI I can assign it as disk3... but I can't find anywhere in the configuration screens to specify mount it as Read Only.  Can I use the GUI to mount this outside of the array or must I use CLI commands?  I don't want to add this old 3TB disk1 drive into my current unRAID config where it keeps showing up even after I remove the drive after testing it.

The easiest thing to would be to install the Unassigned Devices plugin which would give you GUI support for mounting drives that are not part of the array.

I've installed the Unassigned Devices plugin and it shows my unmounted 3TB drive.  But there is nowhere that I see for selecting to mount the drive as "Read Only".  How do you do that with this utility? Do I have to write a script to tell this plug how to mount the drive?  I only see a button next to the listed drive that says "mount" at this time.  I would guess the default is to mount the drive full R/W which I don't want to do at this time.

I mounted the old 3TB disk1 drive anyway and looked at the contents and it seems a lot of the files and directories are missing from it from what  remember should be there.

I think that the drive was indeed corrupt before the unRAID rebuilding.  So when I rebuild the new 8TB drives from these old 3TB drive images, they just copied the corruptions over to the new 8TB drives.  But that still leaves me with the question on how did my 3TB drives become so corrupted in the first place. My unRAID server for the most part is hardly accessed or used that much with really very little activity other than the times I login to the GUI and see updates and let it update automatically on its own. I've read about flaws in the reiserfs file system causing corruptions in certain situations. Maybe my server encountered one of those situations, I don't know.

I am now thinking of moving away from reiserfs format and trying to reformat to XFS.