A few months ago, there were lots of (grumpy) posts stating that we should get and stay on the latest versions of Linux, and other programs, because they contain bug fixes, and are generally more secure than the old version of Linux unRAID had been using for years.  The posts sounded convincing, and made some sense, but I always had this possibility in the back of my mind.  That the latest aren't as well tested, and although they have bug fixes, they may also contain unknown bugs.  This situation has shown that to be a real, and frustrating possibility.

 

Perhaps there needs to be some more thought about how 'bleeding edge' unRAID wants to run, in regards to Linux and other critical programs.

 

it's a balancing act, for sure, but when we were living with known bugs, I don't think we were subjected to data-destroying bugs, and now we've run into one.

 

Obviously getting beta9 tested and out the door should remain the top priority, but once done, maybe a rethink about latest versioning is necessary.

 

Just thinking out loud.

Both corruptions of which I'm aware have been within my appdata share (associated with docker).  One was a truncated source file in an LMS plugin.  The second was a corrupted (or missing?) .conf file for a deluge plugin.

 

The first one was, almost certainly, corrupted as I copied it from my previous LMS configuration into appdata.  The second one was a file created under beta8 and I believe that it had existed but became corrupted after creation.  Both were relatively small text files.

 

I have not encountered any problems with larger media files (mainly .mkv) which were downloaded by deluge and subsequently processed by mkvmerge.

 

Does this help to identify where to concentrate your testing?

 

We are finding the same.  The biggest areas of impact seem to be metadata related to plugins / containers that are of small size.  Larger files (media content) seem to be unaffected, but I cannot claim that for certain at this point, just by the nature of how things seem to be working fine for me on video (not a single video-related problem yet, but I have a lot of content to test  )

We're trying to get more information on what this bug directly affects, but large files seem to be unaffected so far.

 

If you can help determine of the stat block size varys from the actual data read, we can throw some kind of read test to compare the two. See my prior post with cat copy ls and stat commands.

 

Weebo, I'm going through your suggestions right now and testing (as well as building documentation at the same time so hopefully we can provide a more comprehensive guide for detecting affected users with this).

 

It just takes some time to repeat the tests on each system and update docs and check here for more feedback...

A few months ago, there were lots of (grumpy) posts stating that we should get and stay on the latest versions of Linux, and other programs, because they contain bug fixes, and are generally more secure than the old version of Linux unRAID had been using for years.  The posts sounded convincing, and made some sense, but I always had this possibility in the back of my mind.  That the latest aren't as well tested, and although they have bug fixes, they may also contain unknown bugs.  This situation has shown that to be a real, and frustrating possibility.

 

Perhaps there needs to be some more thought about how 'bleeding edge' unRAID wants to run, in regards to Linux and other critical programs.

 

it's a balancing act, for sure, but when we were living with known bugs, I don't think we were subjected to data-destroying bugs, and now we've run into one.

 

Obviously getting beta9 tested and out the door should remain the top priority, but once done, maybe a rethink about latest versioning is necessary.

 

Just thinking out loud.

 

The reality is that these types of bugs can happen with betas.  It sucks...plain and simple.  That said, let's not let one bug during beta all of the sudden make us second guess our vision / direction.

There is one report, and only one so it's rare, of very minor file system damage, found here.  Reiserfsck easily fixes it, and it only involves a backup copy of a bitmap within the ReiserFS data.  (But don't run it until -beta9!)

 

Since the bug involves a pointer within the ReiserFS tree not being advanced properly, I can't help wondering if the wrongly changed file size happens to match a different file on that same drive.  Perhaps it's left pointing to the wrong metadata.  Just curious.

 

I have to agree with Tom, messing with the basic core code that manages the tree and leafs is unconscionable.

 

Obviously getting beta9 tested and out the door should remain the top priority, but once done, maybe a rethink about latest versioning is necessary.

 

I strongly agree with this.  Let's stay at least a release or 2 behind.  Let others bleed.

The reality is that these types of bugs can happen with betas.  It sucks...plain and simple.  That said, let's not let one bug during beta all of the sudden make us second guess our vision / direction.

 

I don't disagree, and I'm not arguing, or even saying anything should change, I was just thinking out loud.

 

however, this is more than 'one bug'.  A bug is the GUI not updating, or something equally annoying, this is data loss, and actually has nothing to do with beta, so much as being a result of the latest, less tested, Linux kernel.  If unRAID was running 3-5 versions behind the latest Linux kernel, I think this situation would not have happened.

 

Again, I'm not saying this 'should' happen, just pointing out that this potential data loss is a result of being on the very cutting edge of Linux, not the unRAID beta, per se.

2 weeks ago i had corruption on my disk 1 (the disk stuff will fill up since the rest are already filled.) I had to use rebuild tree. I suspect it was due to this bug as I have not had any hard shutdowns and it just decided to happen. Also, yesterday I was moving files around from disk 1 to another drive to fill up more and out of the 400gb i moved only one mkv (25gb) got corrupted. So I think it can affect large files just rarely.

 

There is one report, and only one so it's rare, of very minor file system damage, found here.  Reiserfsck easily fixes it, and it only involves a backup copy of a bitmap within the ReiserFS data.  (But don't run it until -beta9!)

 

Since the bug involves a pointer within the ReiserFS tree not being advanced properly, I can't help wondering if the wrongly changed file size happens to match a different file on that same drive.  Perhaps it's left pointing to the wrong metadata.  Just curious.

 

I have to agree with Tom, messing with the basic core code that manages the tree and leafs is unconscionable.

 

Obviously getting beta9 tested and out the door should remain the top priority, but once done, maybe a rethink about latest versioning is necessary.

 

I strongly agree with this.  Let's stay at least a release or 2 behind.  Let others bleed.

OMG.... I have been copying TERRABYTES of data from REISERFS to REISERFS to make room for migrating disks to XFS :-(

 

Ok, relax...breathe...  We will get through this together.  I am in a similar situation with multiple systems.

 

What type of data do you have?  Is this just video content or photos + documents + other things?

 

We're trying to get more information on what this bug directly affects, but large files seem to be unaffected so far.

 

Quantity wise it is movies and series... All downloadable stuff.. I moved my personal photos and documents to onedrive some time ago so they are ok.. I have some home movies, kids stuff on there though that I hope is ok. I have stopped my mediaserver so no more stuff is downloaded and/or changed..

 

My main concern would be that I do not know what might be corrupted..

 

Btw: I fully understand the risk of running beta software... stepped into this willingly.. But must admit that I was not expecting major stuff like this.. But noone has ofcourse.. including Limetech.

 

 

With apologies for being one more useless announcement post interruption...

 

The patches to the Reiser file system do not appear to be backing out shoddy code.  They are new code, fixes of core functionality.  So technically, the Reiser file system has to be considered a BETA file system, for awhile!

Pfew... A lot of shaking up at LimeTech... Used to be such a peacefull little world :-)

 

I just figured something out: This afternoon I cleared a REISERFS 3TB (copied everything off it, I can no longer change that), and have after that formatted it to XFS.

 

Since it is advised now to to write TO REISERFS .. I can safely (I guess) move my files OFF another 3TB reiserfs TO the now cleared XFS.. And so on... That way I would be writing on the XFS drives and not to the REISERFS drives... Therewith moving away from REISERFS to XFS..

 

Have started this up now.

With apologies for being one more useless announcement post interruption...

 

The patches to the Reiser file system do not appear to be backing out shoddy code.  They are new code, fixes of core functionality.  So technically, the Reiser file system has to be considered a BETA file system, for awhile!

With this in mind, what are the downsides of reverting back to the kernel that was used in beta6?

I have quite a lot of files that I copied from one disk to another one where I still have the original (mainly because I forgot to remove the original!).

 

I have been using the presence of these to check for duplicates using the script I posted earlier today.  I am in the process of adding an option to also doing a compare on files with duplicate names.  My initial results while testing this change suggest that in a significant number of cases there are differences in content even when the file size is the same.  I am assuming that is probably the result of the bug mentioned in this thread.  I have not so far looked at the scale of the corruption - just identified that the two copies do not have identical content.  I intend to now start looking in more detail at the files identified as different to see if I can identify by how much they differ.  As they are media files if it is just the odd byte it would probably not end up being noticeable - but it would still be worrying that it is occurring at all.

 

I am lucky in that I should be able to recover all my files from backups (although that is likely to be time consuming).  This is where a simple way of creating checksums for the files on both the server and the backup disks will be useful to identify these corrupted files.

 

I should be able to upload an updated version of the script within a few hours for anyone who might be interested in using it to test for such corruption.  It is easy enough to set up a test case by copying a set of files from one disk to another (making sure you do not write to the source disk so it remains intact).

 

 

Pfew... A lot of shaking up at LimeTech... Used to be such a peacefull little world :-)

 

I just figured something out: This afternoon I cleared a REISERFS 3TB (copied everything off it, I can no longer change that), and have after that formatted it to XFS.

 

Since it is advised now to to write TO REISERFS .. I can safely (I guess) move my files OFF another 3TB reiserfs TO the now cleared XFS.. And so on... That way I would be writing on the XFS drives and not to the REISERFS drives... Therewith moving away from REISERFS to XFS..

 

Have started this up now.

 

Can't speak to your personal situation here, but given the lack of what all is known right now, I can't help but wonder if the best thing to do right now is to stop the array, power down and wait for beta 9 (or revert to beta 6).  What is described above probably would work, but given all the unknowns I get a "playing with fire" vibe... then again I'm not a linux pro or anything like that so don't take my comment too seriously.

itimpi - I am eager to know whether only the new files that were copied to the reiserfs drives could be corrupted or untouched existing files could get corrupted as well? I am worried about the latter since I have about 17TB of data sitting on reiserfs disks. I migrated to V6b8 a week ago, I can live with potential corruption of newer files in the last week.

 

I have quite a lot of files that I copied from one disk to another one where I still have the original (mainly because I forgot to remove the original!).

 

I have been using the presence of these to check for duplicates using the script I posted earlier today.  I am in the process of adding an option to also doing a compare on files with duplicate names.  My initial results while testing this change suggest that in a significant number of cases there are differences in content even when the file size is the same.  I am assuming that is probably the result of the bug mentioned in this thread.  I have not so far looked at the scale of the corruption - just identified that the two copies do not have identical content.  I intend to now start looking in more detail at the files identified as different to see if I can identify by how much they differ.  As they are media files if it is just the odd byte it would probably not end up being noticeable - but it would still be worrying that it is occurring at all.

 

I am lucky in that I should be able to recover all my files from backups (although that is likely to be time consuming).  This is where a simple way of creating checksums for the files on both the server and the backup disks will be useful to identify these corrupted files.

 

I should be able to upload an updated version of the script within a few hours for anyone who might be interested in using it to test for such corruption.  It is easy enough to set up a test case by copying a set of files from one disk to another (making sure you do not write to the source disk so it remains intact).

Pfew... A lot of shaking up at LimeTech... Used to be such a peacefull little world :-)

 

I just figured something out: This afternoon I cleared a REISERFS 3TB (copied everything off it, I can no longer change that), and have after that formatted it to XFS.

 

Since it is advised now to to write TO REISERFS .. I can safely (I guess) move my files OFF another 3TB reiserfs TO the now cleared XFS.. And so on... That way I would be writing on the XFS drives and not to the REISERFS drives... Therewith moving away from REISERFS to XFS..

 

Have started this up now.

 

Can't speak to your personal situation here, but given the lack of what all is known right now, I can't help but wonder if the best thing to do right now is to stop the array, power down and wait for beta 9 (or revert to beta 6).  What is described above probably would work, but given all the unknowns I get a "playing with fire" vibe... then again I'm not a linux pro or anything like that so don't take my comment too seriously.

 

You are correct, I am talking a bit of a leap here.. But I do try and combine several things I have heard in the near past.. With the whole grumpy situation he made a point in stating that running reiserfs was a risk since it is old and support is limited, after that we see that Limetech states that it becomes more important to move away from reiserfs (not majorly important but getting more important), now we see that a reiserfs in combination with new updates is causing issues with potential data loss.. This makes me feel I need to either

 

- stick with non beta ( I do not want to do this as I want to move forward, I am full aware I am taking risks here)

- get rid of reiserfs... More commonly used filesystems will run less risk of these kind of errors, simply because these errors will be noticed sooner because of the larger user base.

 

I also will be less quick in using new beta versions for unraid in the future..

itimpi - I am eager to know whether only the new files that were copied to the reiserfs drives could be corrupted or untouched existing files could get corrupted as well? I am worried about the latter since I have about 17TB of data sitting on reiserfs disks. I migrated to V6b8 a week ago, I can live with potential corruption of newer files in the last week.

 

They've stated a few times that corruption of existing files is very possible, so it's not just new files only that have been corrupted (at least that's what has been reported).

 

That being said, I'm in process of moving to XFS, of my drives in my array, the newest 3 drives have been the only ones written to with b7 (haven't jumped on b8 yet), so I'd be willing to bed most of my data is fine.

 

 

Maybe this has been already answered .. but how do I stop files to being written to the drives? Modify crontab since all the scheduled mover processes happen from there or is there any definite way of doing this?

With this in mind, what are the downsides of reverting back to the kernel that was used in beta6?

 

Just my take, but it should be very straightforward, *if* you qualify.  If you have converted any drives to other file systems or are using the new Cache Pool feature, then you don't qualify, and will need to wait for -beta9.  Reverting to -beta6 costs you the multiple file system support, Cache Pool support, a number of bug fixes, a number of library updates (like Samba, php, others), improved Docker support, some webGUI enhancements, and the improved driver support of kernel 3.16.  Each user will have to evaluate that cost for themselves.

 

If you qualify (all your drives are ReiserFS and you don't mind reverting), then just copy bzimage and bzroot from the -beta6 distro to the flash drive, and reboot.  Once booted with -beta6, I would Check Disk File systems on all of your data drives first, then determine a best strategy for locating suspect files, such as the most recently written files, and any files with suspect lengths.  Par files and MD5 or any other CRC info would be most welcome here, and file backups even more so.  Parity is NOT an issue, no need to check it.

itimpi - I am eager to know whether only the new files that were copied to the reiserfs drives could be corrupted or untouched existing files could get corrupted as well? I am worried about the latter since I have about 17TB of data sitting on reiserfs disks. I migrated to V6b8 a week ago, I can live with potential corruption of newer files in the last week.

I am not sure if my script would necessarily help in this case.  It is looking for a file with the same name on more than one disk, and then if found can optionally compare them to see if they are the same.  It will not identify files that have been corrupted, but which are not duplicated.

 

It might be possible to set up a deliberate test scenario (probably by setting it up under v5) where one had some files duplicated on two disks, and then switched to v6 beta 7/8 and copied files from a third disk onto one of the ones set up above.  You could then run the scripts to see if both copies set up in the test case are identical.

 

However, the only real way to test if files are in fact intact is to check them against your backups.  Ideally one has checksums for the files both on the array (and on the backup disks).  One can then see if the checksums remain unchanged.

 

It occurs to me that it would not be difficult to add to the script the ability to specify the path to a backup location that would be compared to files on the array.  It would require the backup to have the same folder structure as the array.  I am not sure if adding that would be worth the effort as the overhead of reading through each file to do the binary compare would be huge.  Developing a checksum based method of validation would be much more useful, and I believe there are already those working on such a solution.

 

Maybe this has been already answered .. but how do I stop files to being written to the drives? Modify crontab since all the scheduled mover processes happen from there or is there any definite way of doing this?

 

Safest way is shut it down and wait.  Otherwise, you can disable the Mover, disable any plugins that write to Reiser formatted disks, and change the Disk and User Share permissions to read-only.

All,

 

Just wanted to provide a quick update before we go back to testing.  We believe the bug is completely squashed and are working to get a release out as soon as possible.  Sorry for the lack of updates, but it's a lot to test especially when the very nature of the bug makes it silent.

All,

 

Just wanted to provide a quick update before we go back to testing.  We believe the bug is completely squashed and are working to get a release out as soon as possible.  Sorry for the lack of updates, but it's a lot to test especially when the very nature of the bug makes it silent.

 

Thanks for the update, as well as keeping your priorities in order, now back to work!!!!

Question... When I was copying data between my ReiserFS disks and my BTRFS disks, I was using rsync.  Would rsync be able to recognize and correct data corruption? I copied to and from ReiserFS during my migration.

 

I've looked through my files and haven't found any obvious data corruption issues.  I found 7 corrupt audio files, but I believe these predate my return to UnRaid from WHS.

With this in mind, what are the downsides of reverting back to the kernel that was used in beta6?

 

Just my take, but it should be very straightforward, *if* you qualify.  If you have converted any drives to other file systems or are using the new Cache Pool feature, then you don't qualify, and will need to wait for -beta9.  Reverting to -beta6 costs you the multiple file system support, Cache Pool support, a number of bug fixes, a number of library updates (like Samba, php, others), improved Docker support, some webGUI enhancements, and the improved driver support of kernel 3.16.  Each user will have to evaluate that cost for themselves.

 

If you qualify (all your drives are ReiserFS and you don't mind reverting), then just copy bzimage and bzroot from the -beta6 distro to the flash drive, and reboot.  Once booted with -beta6, I would Check Disk File systems on all of your data drives first, then determine a best strategy for locating suspect files, such as the most recently written files, and any files with suspect lengths.  Par files and MD5 or any other CRC info would be most welcome here, and file backups even more so.  Parity is NOT an issue, no need to check it.

Sorry, I wasn't asking about reverting to b6 (personally I'm still using it, I didn't upgrade yet) I was asking what the impact would be on unraid if all the enhancements and upgrades were put in place on the kernel that was used in b6.

 

Put another way, are there any features in the 6.0 roadmap that require 3.16? Why not just fall back to 3.15 for 6.0 final, and put the 3.16 kernel with patches into the 6.1 beta cycle? Which common devices are effected by the 3.16 hardware support differences?

 

(I left your reply intact because it is a good comprehensive answer to the question I didn't ask, but I'm sure others want to know)

Question... When I was copying data between my ReiserFS disks and my BTRFS disks, I was using rsync.  Would rsync be able to recognize and correct data corruption? I copied to and from ReiserFS during my migration.

 

I've looked through my files and haven't found any obvious data corruption issues.  I found 7 corrupt audio files, but I believe these predate my return to UnRaid from WHS.

It would depend on the options you used.  I believe the default is to use timestamp/filesize.  You would have to use -c to use checksum which is alot slower...