copy files, wait 30min, hard reboot = data loss

[Joe L.]

Just to be clear this is only affecting disk shares and not user shares, correct?

It affects anything using the reiserfs file-systems on the disks, including user-shares.

[WeeboTech]

Just to be clear this is only affecting disk shares and not user shares, correct?

It affects anything using the reiserfs file-systems on the disks, including user-shares.

 

Including reiserfs disks outside of the array.

Apps drive, cache drive.

 

EXT3 disks outside of the array have not shown to be affected.

[Riot]

@binhex: Yes, from our tests, depending on version, etc... there is the possibility to lose all or partial data written since last 'sync' or since last array stop. To make absolute sure data is committed you may either telnet the server and use the 'sync' command or just stop/start the array on the web interface.

 

Could you list versions affected etc according to tests??

[Joe L.]

So as an example, say on the 1st of January 2013 user X performed their last sync of the array, anything they have written to disk direct (not via user share) since that date is not written to the array safely until another sync is performed?

Don't confuse a parity "sync" with the low-level command "sync" used to flush all the disk buffers to a disk prior to it being un-mounted.  They really have nothing to do with each other.

 

Joe L.

[jaybee]

So as an example, say on the 1st of January 2013 user X performed their last sync of the array, anything they have written to disk direct (not via user share) since that date is not written to the array safely until another sync is performed?

Don't confuse a parity "sync" with the low-level command "sync" used to flush all the disk buffers to a disk prior to it being un-mounted.  They really have nothing to do with each other.

 

Joe L.

 

Apologies Joe, perhaps you could explain exactly what happens in each and under what circumstances either one would be called by what, and why and how often. Not all of us are particularly familiar with this but still trying to follow and assist with beta/RC's.

I'd like to know when it would be normal - if at all - for a user to perform a low level sync command and how often? Suppose that someone does not for a long time, ALL of the copied data since the last one is at risk until they perform another? This is what I was getting at.

[Joe L.]

So as an example, say on the 1st of January 2013 user X performed their last sync of the array, anything they have written to disk direct (not via user share) since that date is not written to the array safely until another sync is performed?

Don't confuse a parity "sync" with the low-level command "sync" used to flush all the disk buffers to a disk prior to it being un-mounted.  They really have nothing to do with each other.

 

Joe L.

 

Apologies Joe, perhaps you could explain exactly what happens in each and under what circumstances either one would be called by what, and why and how often. Not all of us are particularly familiar with this but still trying to follow and assist with beta/RC's.

I'd like to know when it would be normal - if at all - for a user to perform a low level sync command and how often? Suppose that someone does not for a long time, ALL of the copied data since the last one is at risk until they perform another? This is what I was getting at.

A "parity sync" (also known as a parity check) is a verification that the data written to the parity disk accurately reflects the data written to the data disks.    All disks are read, and if everything matches the stored parity calculations on the parity disk, nothing is written to the disks.

 

However... the data being read from the disks is actually going through the disk buffer cache, so there is a possibility for a parity check to never read the physical disk for the disk buffer blocks still in the disk buffer cache.

 

Tom: Is there a "sync" performed to flush the data to the disks immediately before a parity check?  Might we get misleading results with this bug giving false sense of security the data is being protected by the parity calculation?, since the data read was from the un-written buffers in the disk cache rather than the disks themselves.

 

A disk "sync" flushes all un-written data to the physical disks. 

 

Joe L.

[nars]

@Riot: Ok, going to sum it again:

 

According MY tests I could reproduce it on rc8a, rc11a, rc14 (all these with 3.4.x kernel), rc15a and rc16 (these with 3.9.x kernel patched to fix periodic writes after sync). I could not reproduce any problem with rc5 (3.0.x kernel) and rc13 (3.9.x kernel without the patch to fix periodic writes).

 

Others on this thread (limetech, garycase, mejutty) reproduced the problem on rc15a easily, but not on rc16, this may mean either the problem is harder to happen on rc16 (and probably in any version other than rc15a) or there is more than one problem here. Also WeeboTech was able to reproduce an apparently similar problem on rc16 but with a more 'stressing' test, however we are not yet fully sure that this is really the same problem that I can reproduce or yet some other problem.

 

@jaybee: there is no downside you telnet and do 'sync', it will be a very fast thing (it should take 1 second or not much more than that), and it will ensure all info is commited/flushed on the disk's. If you don't have an UPS I think until this is all surely sorted it may be a good idea to a 'sync' after you copy some important new data to your unRaid system.

 

Btw, I think this thread is getting a bit bloated with these questions... what may make it hard for Tom to check test posts, etc... maybe would be worth splitting it on two one for test results, etc and one for "end user" questions about the problem?

[WeeboTech]

I'd like to know when it would be normal - if at all - for a user to perform a low level sync command and how often? Suppose that someone does not for a long time, ALL of the copied data since the last one is at risk until they perform another? This is what I was getting at.

 

It's not normal to require a user or user process to initiate the manual sync.

However, to be on the safe side, anytime you move over a large amount of data, you can initiate the sync to be sure and confident all buffers are written out to the disk. At the current time we do not have a known time frame of when you can wait to be sure your buffers have been flushed.

 

When you stop and start the array, the buffers and superblock are flushed with the umount,mount command automatically.

 

I've just read that doing a mount -o remount,rw does this also, but I don't want to go into details with this right now.

 

Keep in mind when you do a sync, all your disks will spin up.

 

In looking at the mover, I do not see a sync initiated.

That would be a good place to add it "temporarily" so that you can be sure at least one sync is done on a daily basis, more if you manually trigger the mover.

This will also trigger all your disks to spin up.  Spin up once a day to be confident your superblock is updated and flushed is a small price considering the potential silent corruption.

 

We can also add it to the daily cron stream by dropping an executable job in the /etc/cron.daily folder.

ie.

# Run daily cron jobs at 4:40 every day:

40 4 * * * /usr/bin/run-parts /etc/cron.daily 1> /dev/null

I was reading somewhere that it may be possible to sync a single filesystem, but we may be going to far in that direction. Let's see what Tom brings to the picnic table.

[nars]

Tom: Is there a "sync" performed to flush the data to the disks immediately before a parity check?  Might we get misleading results with this bug giving false sense of security the data is being protected by the parity calculation?, since the data read was from the un-written buffers in the disk cache rather than the disks themselves.

I may be wrong but doesn't 'sync' flush's data just for mounted file systems? not for any lower level disk access... and parity works at block level then it should not be an issue, I think? And at least I can say that in all my hard reset tests I did never noticed parity errors after restart.

[WeeboTech]

Interesting read for those participating in the testing.

 

http://unix.stackexchange.com/questions/48235/can-i-watch-the-progress-of-a-sync-operation

[garycase]

Interesting read ... but regardless of how we can "watch" the dirty buffers get flushed; what we really need is a fix for UnRAID to ensure syncs are performed within a reasonable time.

 

Question for anyone who might know:  Is a sync performed before the drives are spun down?  That would certainly seem to be a good idea !!  (and would put a nice upper limit on the period of "danger" for this issue)

 

My earlier comments that systems with a UPS aren't impacted by this, since they will do a clean powerdown after a power failure and thus are never exposed to a "pull the plug" power outage does have one flaw:  the current issues some are having with Powerdown means the "clean" shutdown may not in fact occur in the case.    I guess the only way to really be "safe" with regard to this is to either manually run a Sync command; or Stop and Start the array.  [WeeboTech:  Does an array Stop/Start also resolve the potential issues with drives "outside" the array that you've noted?]

 

[JonathanM]

Might we get misleading results with this bug giving false sense of security the data is being protected by the parity calculation?, since the data read was from the un-written buffers in the disk cache rather than the disks themselves.

Actually, this gives an opportunity to test exactly what happens. Set up a failure mode situation, where you are guaranteed to have incorrect data on a protected drive. REMOVE the protected drive before you start the machine back up, and do the corruption check on the parity emulated disk.

[WeeboTech]

Interesting read ... but regardless of how we can "watch" the dirty buffers get flushed; what we really need is a fix for UnRAID to ensure syncs are performed within a reasonable time.

 

While the obvious is stated, the information is relevant.

Turning off the system during testing, when you KNOW there are dirty buffers,  is going to trigger the condition.

In that respect it's not really testing if we have a bug,  or we have not waited long enough.

 

The fix that is required is to insure that dirty buffers ARE actually flushed and the superblock updated.

What is a reasonable time?

by watching the dirty buffers, you can get an idea of what that is on your system with the tests being performed.

 

I asked this same question earlier. No response, I found this information and presented so that people understand, if you have dirty buffers, you are highly likely to reveal issues. Issues which may not be the bug, but are simply part of how the write-behind buffer cache works.

 

 

Question for anyone who might know:  Is a sync performed before the drives are spun down?  That would certainly seem to be a good idea !!  (and would put a nice upper limit on the period of "danger" for this issue)

 

This is something for Tom.

maybe emhttp  can do a sync for only a specific filesystem before the drive is spun down.

I have always wished for that.

 

My earlier comments that systems with a UPS aren't impacted by this, since they will do a clean powerdown after a power failure and thus are never exposed to a "pull the plug" power outage does have one flaw:  the current issues some are having with Powerdown means the "clean" shutdown may not in fact occur in the case.    I guess the only way to really be "safe" with regard to this is to either manually run a Sync command; or Stop and Start the array. 

 

This is not really a golden rule in that systems with a UPS are not impacted from this.

Currently if you have processes sitting on the array writing it wont shut down.

If you run out of battery, you crash.

plus... other s**t happens.

Sometimes the kernel crashes and does not sync, or a machine locks up.

 

The real issue is outstanding data is not written.

Once that is resolved, ups or no ups, you should loose very little unless you KNOW there have been outstanding dirty buffers.

 

I would really appreciate if you stop pacifying the situation in that if you have a UPS you are not impacted.

This is not OK.

You can be impacted until you reach the point where filesystems have been unmounted.

 

unRAID is OK if

manually sync.

stop and start the array.

power down gracefully from the emhttp interface and it succeeds with umounting the filesystems.

There is a sync being done in there which can diminish the gravity of it, however if you have allot of outstanding data, that can take time.

Drives still have to spin up, and then data has to be flushed.

 

I'm sure. I am doing a sync in the powerdown script. I just don't remember where.

Plus I can add one if need be.

 

Does an array Stop/Start also resolve the potential issues with drives "outside" the array that you've noted?

 

I have confirmed that on rc15a, a sync is in the syslog when you stop the array.

This will insure all dirty buffers are written and superblocks updated (or should) for all filesystems mounted.

From what I see in the log, the sync is done before attempting to umount the filesystems.

 

Still,  drives have to spin up,  and data has to be flushed at the speed of which your array can perform.

[limetech]

You guys who sent me email asking for -rc16, I will have an -rc16a ready to test this afternoon and will send you email notification when it's ready.

[nars]

Interesting read for those participating in the testing.

 

http://unix.stackexchange.com/questions/48235/can-i-watch-the-progress-of-a-sync-operation

 

Interesting... though I just checked and either on rc15a or rc16 right after copy my test files, after some seconds I can see dirty and writeback going to 0, without me doing anything (no sync, nothing...), and despite both 0 I did hard reset and files missing... then this seems useless for this problem.

[WeeboTech]

Interesting read for those participating in the testing.

 

http://unix.stackexchange.com/questions/48235/can-i-watch-the-progress-of-a-sync-operation

 

Interesting... though I just checked and either on rc15a or rc16 right after copy my test files, after some seconds I can see dirty and writeback going to 0, without me doing anything (no sync, nothing...), and despite both 0 I did hard reset and files missing... then this seems useless for this problem.

 

it's usefull because,

if it's > 0 and you loose power, you most likely will have some kind of loss.

if it's 0 and you loose power, you should NOT have such a large amount of loss.

 

Try the version that doesn't exhibit the loss of data.

Wait until you see dirty buffers at 0, then do the hard reset.

 

If we are loosing data while dirty buffers are 0, then the number is actually useless.

 

 

Since I have an idea of how this works I did a small test on a normal linux system.

 

I did a find down the whole tree.

While i did not write to the filesystem, the process of accessing an inode causes access time to be updated. Yes, every read later on becomes a write.

 

I watched the simple act of a find cause the dirty buffers to elevate, then 10 seconds later decrement to 0.

 

unRAID mounts filesystems with nodiratime and noatime.

We wont see this exhibited on unRAID.

It did give me an idea, or line in the sand, that I cannot power off until that number is 0 or I will be surely tainting the test.

 

So to be fair to this test, do not power off until that number is 0 or very low.

On some systems it never went to 0.

[nars]

Ah, ok, can understand your idea now. I was expecting to see it just don't go to 0 on a problematic system without a manual sync... forget

 

Will just wait 16a for now then I can do so more tests on it.

[theone]

What could be the potential damage if I do a hard reset?

http://lime-technology.com/forum/index.php?topic=28222.0

 

[WeeboTech]

What could be the potential damage if I do a hard reset?

http://lime-technology.com/forum/index.php?topic=28222.0

 

Wait a few minutes and lets see if Tom has a moment to ask you some questions.

I posted in the other thread how to manually unmount the filesystems should you decide to do a hard reset.

 

Their might be something your condition reveals that Tom could use.

 

 

As far as your question, what could happen is anything written to the server in the last hour or so may have the potential to be a partial transfer or lost.

[theone]

As far as your question, what could happen is anything written to the server in the last hour or so may have the potential to be a partial transfer or lost.

 

Anything older - day older - won't be lost?

 

[WeeboTech]

As far as your question, what could happen is anything written to the server in the last hour or so may have the potential to be a partial transfer or lost.

 

Anything older - day older - won't be lost?

 

I cannnot guarantee that.

 

I would make the attempt to umount the filesystems manually first.

 

Supposedly the filesystems can be remounted and that forces a superblock flush.

 

I would wait for Tom's input first.

 

[theone]

I cannnot guarantee that.

 

I would make the attempt to umount the filesystems manually first.

 

Supposedly the filesystems can be remounted and that forces a superblock flush.

 

I would wait for Tom's input first.

 

OK, I am waiting for Tom's suggestion...

[aaronwt]

I've been copying 26 Terabytes of data over to my unRAID2 for the last few weeks. I had a couple of sudden shut downs during that period. The shutdowns were not during the copying process but I have no idea if I lost anything. I guess when I finish copying the data I need to try and figure out if any of my BD ISO files are missing.

 

Hopefully nothing is missing since I've been deleting the files from my WHS after I verified the folder sizes match on unRAID2 each day. Plus I initially needed to move some empty drives from my WHS to get my capacity to 51TB on my unRAID2 setup. And then then the remaining fifteen, 2TB, drives can be moved to my unRAID3 setup when the final files are finished being copied tonight or tomorrow.

[WeeboTech]

I had a couple of sudden shut downs during that period.

 

if unraid was shutdown gracefully, i.e. via normal browser interface powerdown/reboot button or UPS trigger, you should be good.

if you had a sudden power loss without protection of a controlled shutdown then files closest to the shutdown are suspect. Check sizes, you may have lost data at the end of the files.

 

as a precaution, after a significant amount of activity you can do a manual sync.

After the sync command returns, you're good.

 

Also what version of unRAID are you using. it seems that some do not exhibit the issue revealed in the most recent candidates.

[aaronwt]

I had a couple of sudden shut downs during that period.

 

if unraid was shutdown gracefully, i.e. via normal browser interface powerdown/reboot button or UPS trigger, you should be good.

if you had a sudden power loss without protection of a controlled shutdown then files closest to the shutdown are suspect. Check sizes, you may have lost data at the end of the files.

 

as a precaution, after a significant amount of activity you can do a manual sync.

After the sync command returns, you're good.

 

Also what version of unRAID are you using. it seems that some do not exhibit the issue revealed in the most recent candidates.

 

I'm using r15a in all my unRAID setups. How do I perform a manual sync?