Broken mp4 video file

[Sypower]

Some of the videos from my wedding that were saved to my unraid server will not play through until the end anymore.  The shorter ones are fine, but all of the longer ones will play for some time and then freeze.  

 

I've tried copying the file from the server onto my Windows machine and playing the video from there in case it was an issue of the network file sharing.  I've tried playing the video in Window's default "Films & TV" app, and also in VLC media player, always the video will stop at the same timestamp.

 

To be clear, this is not a data recovery post.  If I need help with that I'll make a new thread.  I want to know why or how this could happen so I can avoid it in the future.  How am I supposed to trust this software for permanent file storage, if it can't even handle a few videos for one year?

 

Please let me know what other information I should provide in order to help troubleshooting.

[Frank1940]

Was there any error when you copied the the file from your server to your PC?  If you can, this would indicate that the actual data in the file is the problem and not an error in the file itself.

 

Do you still have access to the original source?

 

IF nothing else, try to move past the stopping point and see of you can play the balance of the file.  (I know that VLC will allow to position the start point for play by "pulling" the circle when playback is stopped...)

 

Please be advised that the parity protection in Unraid is not a backup if the only version of a file is on the server.  You need at least two copies of anything which is irreplacable!  And for even more security, one of those copies should be off-site. 

[Sypower]

1 hour ago, Frank1940 said:

Was there any error when you copied the the file from your server to your PC? 

No errors or warnings when copying the file from the Unraid server to my PC.

 

1 hour ago, Frank1940 said:

Do you still have access to the original source?

Yes - the USB that was given to us by the videographer still has the original videos and they will play just fine off the USB.

 

1 hour ago, Frank1940 said:

IF nothing else, try to move past the stopping point and see of you can play the balance of the file.  (I know that VLC will allow to position the start point for play by "pulling" the circle when playback is stopped...)

I can drag the "circle" on the slider in VLC in an attempt to start playback at a different point in the video.  The video will begin playback if started from any moment before when it usually stops playing (the "broken point").  I can drag it forward past the broken point, press play, see nothing, then drag it back to the start again and it plays the beginning fine again. I tried many places after the broken point, and it is always the same.

 

When I do drag the circle to get playback after the broken point, there is no sound and the video is stuck on the last frame that was played, but the slider keeps creeping forward and the timer keeps counting up as though it were playing.  This is in contrast to how Window's "Films & TV" app behaves - when I try to get playback past the broken point, the timer stops counting up and I cannot get it to play near the start again without closing and re-opening the app.

 

1 hour ago, Frank1940 said:

Please be advised that the parity protection in Unraid is not a backup if the only version of a file is on the server.  You need at least two copies of anything which is irreplacable!  And for even more security, one of those copies should be off-site. 

I understand this, and I agree this is a good time to bring up that reminder.  Still, I believe that there is more to explore here.  If Unraid is going to keep one of my copies of my files, I would like to know that it is reliable and that is not my impression so far.  I want to know if this sort of thing has happened to someone before.  I want to know if there is something I should be doing to monitor for errors or broken files like this.  I wouldn't have known there was an issue with this backup if I had not tried to watch some of the video.  Are there other files on my server that are broken and I just don't know it yet?  If that is the case then Unraid is giving me a false sense of security that I have two backups when really I only have one.  Is there anything else that can be done to find a root cause to this problem?

[Michael_P]

1 hour ago, Sypower said:

I want to know if there is something I should be doing to monitor for errors or broken files like this.

 

You can use the file integrity plugin, assuming the file is copied correctly in the first place then any changes to the file fail the hash check. 

 

If you have backups, I wouldn't stress too much about it - for family photos/videos I keep to the 3-2-1 backup stratagy, 2 on site and 1 off site. If I happen across a corrupted file (which, knock on wood, I haven't yet), just restore from the backup. It's very likely not somthing that would be Unraid's "fault", far more likely that the error was introduced in RAM and/or compunded by the ECC on the drive "correcting" the error. Tho a correcting parity check at any point could also "correct" bits in an unexpected manner.

[Frank1940]

Copy the files off that USB drive to your PC.  Check that those copies play.  Then copy them up to the Unraid server again.  (You can either delete those original copies that don't play and copy replacements to the same point on your Unraid file structure-- or to a new diectory on the server.)  Again, check to see that they play.

 

2 hours ago, Sypower said:

Are there other files on my server that are broken and I just don't know it yet?  If that is the case then Unraid is giving me a false sense of security that I have two backups when really I only have one.  Is there anything else that can be done to find a root cause to this problem?

 

It is very unusual.  (Data errors that are on a HD are (1) either corrected by error-correcting algorithms or (2) flagged as read errors.  A HD will only send out the exact bits that were written to it or it sends an error message.)   There are a lot of checks on data integrity during the movement of data between storage devices.  As @Michael_P indicated, when these errors occur, it is usually attributed to data being corrupted in RAM.  It could happen either on your PC or in the server.  There is a Unraid bootup option to check server memory using MEMTST that will check standard RAM.  (ECC RAM requires that you go to the MEMTST website and download the free testing suite for this type of RAM.)  Run the test for 24 hours.  You should have zero errors. 

 

PS--- Don't overclock memory or the CPU on your server. Either of these can cause silent crashes...

Edited January 9, 2023 by Frank1940

[Sypower]

21 hours ago, Michael_P said:

You can use the file integrity plugin, assuming the file is copied correctly in the first place then any changes to the file fail the hash check. 

I am using the file integrity plugin.  I never noticed any warnings about these files, though I might have missed the warning.  I find the plugin throws a lot of false positives for file edits or new files or even just things being written to a log file, so maybe I never read the message that warned me about it.

 

21 hours ago, Michael_P said:

a correcting parity check at any point could also "correct" bits in an unexpected manner.

This is the sort of thing I was worried about.  

 

21 hours ago, Frank1940 said:

Copy the files off that USB drive to your PC.  Check that those copies play.  Then copy them up to the Unraid server again.  (You can either delete those original copies that don't play and copy replacements to the same point on your Unraid file structure-- or to a new diectory on the server.)  Again, check to see that they play.

I've put the original files onto the server now and they will play without issue, even directly off the server.

 

21 hours ago, Frank1940 said:

It is very unusual.  (Data errors that are on a HD are (1) either corrected by error-correcting algorithms or (2) flagged as read errors.  A HD will only send out the exact bits that were written to it or it sends an error message.)   There are a lot of checks on data integrity during the movement of data between storage devices.  As @Michael_P indicated, when these errors occur, it is usually attributed to data being corrupted in RAM.  It could happen either on your PC or in the server.  There is a Unraid bootup option to check server memory using MEMTST that will check standard RAM.  (ECC RAM requires that you go to the MEMTST website and download the free testing suite for this type of RAM.)  Run the test for 24 hours.  You should have zero errors. 

 

PS--- Don't overclock memory or the CPU on your server. Either of these can cause silent crashes...

I can double check that my RAM is good.  Is XMP enough of an overclock that you would recommend disabling that as well?   Other than that I have no overclocks enabled.

 

In terms of other potential hardware issues, I did have an old, old hard drive in there when the files became broken.  The drive did error out and stop working and has since been replaced, but I was assuming that the parity drive (brand new drive) would protect all my data in that scenario.  Is it possible that the drive took some data with it on its way out?

Edited January 10, 2023 by Sypower

[Frank1940]

1 hour ago, Sypower said:

In terms of other potential hardware issues, I did have an old, old hard drive in there when the files became broken.  The drive did error out and stop working and has since been replaced, but I was assuming that the parity drive (brand new drive) would protect all my data in that scenario.  Is it possible that the drive took some data with it on its way out?

IF that old disk had the file with the data problem in that file (Regardless of how the  file was corrupted) and parity was totally rebuilt for any reason, rebuilding that disk from parity would have restored the corrupted file to the new disk.  Remember what rebuilding parity does.  When parity is rebuilt, (1) all of the data disks are read byte-by-byte, (2) a new parity information is calculated, and (3) that results of that calculation are written to the parity disk.  (Having said that allow me to say that if a parity error is found during a non-correcting parity check, the standard recommendation to check the logs and SMART data for any possible reason for the error.  If none is found, then a parity rebuilt is that proper action.  This recommendation is based on the assumption in the next paragraph about data corruption on Hard Disks.)

 

What we don't know is where and when that file was corrupted.  The HD manufacturers have a lot of  proprietary error detecting and error correcting software to prevent it from happening on the hard disk. I am one of the folks who believe that a HD will always return the exact data that was originally written to it or it returns an error code.  I am convinced that any error that corrupts data occurs when the data in some place besides on the HD.  

 

The statements in the previous two paragraphs have logic flaws in them.  There are folks who believe in "bit-rot"  (which I consider to be folk lore) but I can not completely rule it out as I have never seen that any HD manufacturer software code have been audited/analyzed to guarantee that silent errors might not slip thorough undetected and what the probability of such an occurrence from happening IF the algorithms might allow such situation. 

 

With regard to something like your wedding video, might I suggest that you consider multiple backup on several different media materials if its retention is of prime concern to you.  Consider using not only that USB drive and a hard drive in a parity protected server, but also consider using burning the files to DVD and uploading them to the cloud.  Consider an off-site storage for one (or more) of your physical copies in case you have a natural or man-made disaster at your home. 

 

Be careful of the storage conditions.  I have writable DVDs that are over ten years old that are still playable but they are stored at room temperature in a dark environment.  Do a bit of research on media life and ideal storage conditions for any storage media that you use. 

 

Nothing can guarantee that you won't lose critical files but you can minimize the risks.  Test that each copy is readable when you make it and periodically test that it is still readable.  Make new copies on new media as time passes.  (Thing about the problem of trying to find a floppy disk reader today...) 

Edited January 10, 2023 by Frank1940