unRAID6-beta7/8 POSSIBLE DATA CORRUPTION ISSUE: PLEASE READ

[Helmonder]

Ehm.. I did it by making sure I had one large empty drive, formatting that to xfs and then copying the contents of a complete other drive onto the just formatted one and then formatting the just cleared one.

 

To avoid issues with stopped transfers and such I do them on console.

 

It takes my five minutes every 24 hours this way..

[dikkiedirk]

Ehm.. I did it by making sure I had one large empty drive, formatting that to xfs and then copying the contents of a complete other drive onto the just formatted one and then formatting the just cleared one.

 

To avoid issues with stopped transfers and such I do them on console.

 

It takes my five minutes every 24 hours this way..

 

So you add a new precleared disk to the array, format it as XFS, copy over data from disk-x, when this is done, disk-x is formatted to XFS, the data from disk-x+1 is copied over and so you work through the entire array? Is there any verification on file integrity? Is the array taken off-line?

 

Which console do you mean? Do you have a keyboard and display attached during this procedure? Or are you using IPMI KVM?

[Helmonder]

I was able to free a disk in the array to start with.

 

Using ipmi kvm and MC.

 

Array remains online.

 

If regular copies needed extra verification I would have had bitrot way before this..

[dikkiedirk]

Can a freed up disk be formatted be formatted to XFS without taking the array off-line? Did you copy or move the data?

 

Another plus for IPMI, luckily both my servers have it.

 

The biggest disk in my array is 4 TB, all are pretty much filled up. Guess I need to add a 4TB disk first.

 

Might postpone this operation a bit.

[itimpi]

Can a freed up disk be formatted be formatted to XFS without taking the array off-line?

You have to stop the array to be allowed to change the format in the GUI.    However once the format has been changed the array can immediately be started again so it is only off-line for a minute or so.  Formatting is done after the array is re-started so does not affect availability.

[Moussa]

Can a freed up disk be formatted be formatted to XFS without taking the array off-line?

You have to stop the array to be allowed to change the format in the GUI.    However once the format has been changed the array can immediately be started again so it is only off-line for a minute or so.  Formatting is done after the array is re-started so does not affect availability.

 

How long does this formatting take roughly? I've only ever formatted a precleared disk before.

[Interstellar]

I've been looking at the scripts being posted and I think the corz script might do the job.

 

I can't however figure out a way of making the script run at the root of a disk, generate a recursive hash and save that at the disk root.

 

I.e. then I can every so often run the script against the previously generated file at the root of the disk...

 

Edit: I was running beta 7 for a few weeks, I've run a number of checksum verifies against my source and the copy on the server, not found anything out of the ordinary though so I might have just been spared...

[Helmonder]

Can a freed up disk be formatted be formatted to XFS without taking the array off-line? Did you copy or move the data?

 

Another plus for IPMI, luckily both my servers have it.

 

The biggest disk in my array is 4 TB, all are pretty much filled up. Guess I need to add a 4TB disk first.

 

Might postpone this operation a bit.

 

You take the array offline for 10 seconds and set filesystem tot XFS. Then restart the array and while it is running unraid will be able to format to xfs..

[Helmonder]

Can a freed up disk be formatted be formatted to XFS without taking the array off-line?

You have to stop the array to be allowed to change the format in the GUI.    However once the format has been changed the array can immediately be started again so it is only off-line for a minute or so.  Formatting is done after the array is re-started so does not affect availability.

 

How long does this formatting take roughly? I've only ever formatted a precleared disk before.

 

15 seconds... The array is availalble during this time, as soon as the disk is formatted I go ahead and copy a REISERFS's disks data to the newly formatted XFS disk, and when that is finished I have another empty REISERFS disk and the whole thing starts again..

 

It takes a lot of time, but not a lot of work.. like 5 minutes every day..

[SSD]

Can a freed up disk be formatted be formatted to XFS without taking the array off-line?

You have to stop the array to be allowed to change the format in the GUI.    However once the format has been changed the array can immediately be started again so it is only off-line for a minute or so.  Formatting is done after the array is re-started so does not affect availability.

 

How long does this formatting take roughly? I've only ever formatted a precleared disk before.

 

Don't confuse formatting with preclearing.

 

Preclearing zeroes out the whole dirve and inserts a special signature so that unRAID recognizes the disk as precleared. Since "zeroes" have no affect on parity, a precleared disk can be added to an array without rebuilding parity.

 

Formatting lays down a structure for storing data. To unRAID, the format is nothing more than data that is reflected in parity.

 

If your drive is formatted for RFS, let's say, and you want to change it to XFS, the parity already is taking into consideration all of the data on the disk. Switching the file system can be done without rebuilding parity. The format itself if very fast.

 

What is time consuming is to add a NEW disk to the array which does not have a preclear signature. In this case unRAID will clear it (which takes many hours during which the array is offline) and then add it to the array. From there you can format it any way you like.

 

Hope this clears up any confusion.

[itimpi]

How long does this formatting take roughly? I've only ever formatted a precleared disk before.

A couple of minutes (exact time varying with disk size).  It takes exactly the same length of time as it would if the disk had been pre-cleared.

[Moussa]

Can a freed up disk be formatted be formatted to XFS without taking the array off-line?

You have to stop the array to be allowed to change the format in the GUI.    However once the format has been changed the array can immediately be started again so it is only off-line for a minute or so.  Formatting is done after the array is re-started so does not affect availability.

 

How long does this formatting take roughly? I've only ever formatted a precleared disk before.

 

Don't confuse formatting with preclearing.

 

Preclearing zeroes out the whole dirve and inserts a special signature so that unRAID recognizes the disk as precleared. Since "zeroes" have no affect on parity, a precleared disk can be added to an array without rebuilding parity.

 

Formatting lays down a structure for storing data. To unRAID, the format is nothing more than data that is reflected in parity.

 

If your drive is formatted for RFS, let's say, and you want to change it to XFS, the parity already is taking into consideration all of the data on the disk. Switching the file system can be done without rebuilding parity. The format itself if very fast.

 

What is time consuming is to add a NEW disk to the array which does not have a preclear signature. In this case unRAID will clear it (which takes many hours during which the array is offline) and then add it to the array. From there you can format it any way you like.

 

Hope this clears up any confusion.

 

Oh I wasn't confused about the difference between formatting and preclearing, but thank you for your explanation. I wasn't sure about the effect of having a disk with data already on it, and parity protected, on a format. But now I see it makes no difference, thanks for the clear up!

[Helmonder]

Just to be sure: data on the drive will be gone as a result of the format !

[Moussa]

Just to be sure: data on the drive will be gone as a result of the format !

 

Haha thanks for the warning Now to a week of moving data between disks!

[dikkiedirk]

Just to be sure: data on the drive will be gone as a result of the format !

 

Haha thanks for the warning Now to a week of moving data between disks!

 

In my case more like 2 to 3 weeks . Just hope Murphy and his Law don't show up.

[gshlomi]

A few questions:

1. When you move the data from ReiserFS to XFS, do you move or copy and then delete?

2.Does it leave the source disk all zeroed out? if not - how come formating the disk leaves the parity intact?

3. The biggest question - who said XFS is better then the fixed ReiserFS regarding corruptions? I remember there was a thread about XFS lockup/race condition or something like that. Isn't it a risk to move to XFS at the time?

 

[jphipps]

I use rsync to move the data.  With that you can always kill it and restart without issue.  I use the --remove-source-files option, so once the file is successfully moved, it will be deleted from the source.  For example:

 

rsync -av --progress --remove-source-files /mnt/diskX/ /mnt/diskY/

 

It does leave the directory structure, but all files will be moved.

 

I started moving on of my unraid servers over to XFS early on, and have had some issues with hard lockups.  I still haven't found out the exact cause, because I have converted about 20% of my drives in my other unraid server to XFS without any issues.  I have found on the first server that for some reason the BTRFS filesystem is seemingly more stable for what ever reason.  I am actually going to switch all the XFS filesystems over to BTRFS and see how it runs.

 

My suggestion would to not be in a hurry to move over to one or the other.  I would just convert a few over to the various format for testing and see if any one works better or worse. 

[Helmonder]

A few questions:

1. When you move the data from ReiserFS to XFS, do you move or copy and then delete?

2.Does it leave the source disk all zeroed out? if not - how come formating the disk leaves the parity intact?

3. The biggest question - who said XFS is better then the fixed ReiserFS regarding corruptions? I remember there was a thread about XFS lockup/race condition or something like that. Isn't it a risk to move to XFS at the time?

 

1) I do moves with MC over console.

2) The source disk is empty when it completes so I can safely change filesystem and format it

3) Dunno... REISERFS was rockstable because it did not get touched, but support was limited and usage low. Now it appears changes are made to REIERFS. I feel more comfortable with a widely used filesystem.

[PeterB]

A few questions:

1. When you move the data from ReiserFS to XFS, do you move or copy and then delete?

 

I use rsync to copy, without deleting.  If you delete, the parity will be written twice - once for the drive being written to, and once for the drive being deleted from(albeit only clearing a directory entry) - this will slow down the operation.  The format operation, which is performed when the drive FS is changed, will make the drive 'empty' without having to delete files.

 

2.Does it leave the source disk all zeroed out? if not - how come formating the disk leaves the parity intact?

 

The entire operation is carried out with the array running, and parity active - hence there is no need to zero.

 

3. The biggest question - who said XFS is better then the fixed ReiserFS regarding corruptions? I remember there was a thread about XFS lockup/race condition or something like that. Isn't it a risk to move to XFS at the time?

 

XFS has to be the most sensible choice at this time - it is the default FS for many major Linux dstributions.  No major distribution uses RFS as default - indeed, many distros no longer include RFS as standard.  BTRFS will become the most widely used FS, but is still in a state of flux and, while supported by most distros, few, if any, use BTRFS as the default FS.

 

A couple of points from my own experience - converting/copying a fullish 2TB drive takes approximately 24 hours.  RFS uses slightly more disk space than RFS.  For example, free space on two of my drives reduced from 413 to 400 and 122 to 86.6 GB.

[hackztor]

Just rsync one disk to a blank disk, stop array click drive you want to switch to different file system, start array and then click the checkbox for format and click format (check to make sure only one disk is unformatted). It will format the drive while the array is started and usually only takes a short time (1 min or less). Then rysnc data back to that disk. I went with xfs as btrfs is still being worked on and not worth risking data yet. I will switch to that file system in probably a year to 2. Id rather spend a week getting drives prepared than risk any data. The thing that is lacking from btrfs right now is any kind of checkdsk if the data does go corrupt, but it does have the benefit of detecting bitrot.

 

A few questions:

1. When you move the data from ReiserFS to XFS, do you move or copy and then delete?

 

I use rsync to copy, without deleting.  If you delete, the parity will be written twice - once for the drive being written to, and once for the drive being deleted from(albeit only clearing a directory entry) - this will slow down the operation.  The format operation, which is performed when the drive FS is changed, will make the drive 'empty' without having to delete files.

 

2.Does it leave the source disk all zeroed out? if not - how come formating the disk leaves the parity intact?

 

The entire operation is carried out with the array running, and parity active - hence there is no need to zero.

 

3. The biggest question - who said XFS is better then the fixed ReiserFS regarding corruptions? I remember there was a thread about XFS lockup/race condition or something like that. Isn't it a risk to move to XFS at the time?

 

XFS has to be the most sensible choice at this time - it is the default FS for many major Linux dstributions.  No major distribution uses RFS as default - indeed, many distros no longer include RFS as standard.  BTRFS will become the most widely used FS, but is still in a state of flux and, while supported by most distros, few, if any, use BTRFS as the default FS.

 

A couple of points from my own experience - converting/copying a fullish 2TB drive takes approximately 24 hours.  RFS uses slightly more disk space than RFS.  For example, free space on two of my drives reduced from 413 to 400 and 122 to 86.6 GB.

[jumperalex]

You said:

The reason why there is no checkdsk is that there is quite a bit of debate as to whether it's needed.

 

What I heard was: "No one needs more than 640k" ;-)

[gshlomi]

You said:

The reason why there is no checkdsk is that there is quite a bit of debate as to whether it's needed.

 

What I heard was: "No one needs more than 640k" ;-)

[jbartlett]

Here's the hash tool I mentioned earlier. Generates and stores a SHA256 hash in the user attributes for a file.

 

bitrot.sh

 

thanks for your work. But I have a question, since I did have hash from before de the corruption, this tool cannot help me to detect what file is corrupt or not? But it can help me for future possible errors? thanks

 

That is correct.

[lars]

Can a freed up disk be formatted be formatted to XFS without taking the array off-line? Did you copy or move the data?

 

Another plus for IPMI, luckily both my servers have it.

 

The biggest disk in my array is 4 TB, all are pretty much filled up. Guess I need to add a 4TB disk first.

 

Might postpone this operation a bit.

well i hope your 4tb drive is the parity drive! otherwise you are in self-made trouble.

if that sounds strange to you - back to the basics reading about unraid!

 

[hackztor]

Ya that is what I am doing. I have 4 drives of the 8 converted to xfs now and cache on btrfs.

 

Will btrfs tell you if you have a corrupted file or bit rot automatically or only after running scrub?

 

A notice will be placed in the DMESG queue. IF the file is read and the checksum is incorrect.

I'm not sure of syslog at this point.

From there you have to identify the file and/or initiate recovery from a snapshot or backup.

 

I am trying to decide xfs or btrfs for all my array devices, leaning towards xfs for now since even redhat 7 uses that as the default now, I have a feeling btrfs still has about a year- two to be considered stable enough.

 

I would suggest XFS at this point in time. Many of the hardware PVR vendors use it (from what I remember at least).

Using BTRFS for the cache would probably be OK.