random file corruption


Recommended Posts

hi all

Over the last few days I have started to see movie files that are copied to my unraid share via SMB in Windows become corrupt.

By this I mean during playback I get skipping and pixelation. 

In order to rule out the files I playback the local copy on my PC and it plays fine but then I go to the same file via SMB on unraid and it has skipping and pixelation. 

Any ideas?

Link to comment

It seems to me, a few years ago, that there was a particular SATA card that could do this.  If you are using any SATA expansion cards, list the Manufacturer(s) and model number(s). 

 

One thing to do is to post up your Diagnostics file  (  Tools   >>>   Diagnostics   in a new post) and let the real Gurus have a look at it.

Link to comment

You have a lot of errors in the syslog that look like this:

 

Apr 21 05:50:05 Tower avahi-daemon[10039]: Withdrawing address record for 2a00:23c4:4e9c:5700::1f27 on br0.
Apr 21 05:50:06 Tower dhcpcd[14448]: br0: DHCPv6 REPLY: Address not bound to this interface.
Apr 21 05:50:06 Tower dhcpcd[14448]: br0: soliciting a DHCPv6 lease
Apr 21 05:50:07 Tower dhcpcd[14448]: br0: ADV 2a00:23c4:4e9c:5700::2000/128 from fe80::4262:31ff:fe01:528c
Apr 21 05:50:07 Tower dhcpcd[14448]: br0: REPLY6 received from fe80::4262:31ff:fe01:528c
Apr 21 05:50:07 Tower dhcpcd[14448]: br0: adding address 2a00:23c4:4e9c:5700::2000/128
Apr 21 05:50:07 Tower dhcpcd[14448]: br0: renew in 2250, rebind in 3600, expire in 7200 seconds
Apr 21 05:50:07 Tower avahi-daemon[10039]: Registering new address record for 2a00:23c4:4e9c:5700::2000 on br0.*.
Apr 21 05:50:09 Tower ntpd[1636]: Listen normally on 518 br0 [2a00:23c4:4e9c:5700::2000]:123
Apr 21 05:50:09 Tower ntpd[1636]: new interface(s) found: waking up resolver

Hopefully, someone who is much smarter about this type of error will jump in and give you a hand.  

 

You might not have a file corruption issue.  You should compare the md5 sums for the same file on both the server and your PC.  Here  is one article about how to do that:

 

        https://www.lifewire.com/validate-md5-checksum-file-4037391

 

If you want more information about md5sum, Google    md5 checksum   

Edited by Frank1940
Link to comment
13 hours ago, Frank1940 said:

I would suggest that you run memtst  (found on the UnRAID boot menu)  for 24 hours.

I found files that are are downloaded by containers like sonarr for example are fine. 

they playback without any corruption, its only files copied over from my PC.

could it still be memory?

Link to comment

Sounds like a network issue. Unraid would not corrupt files, the only way to tell is to play the file on your pc, it should play fine.  Copy said file to unraid, and try to read.  If it fails, take the md5 sum of the working file and compare it to the one you copied.  This will tell you if it got corrupted during the network transfer.

Link to comment
1 minute ago, Jclendineng said:

Sounds like a network issue. Unraid would not corrupt files, the only way to tell is to play the file on your pc, it should play fine.  Copy said file to unraid, and try to read.  If it fails, take the md5 sum of the working file and compare it to the one you copied.  This will tell you if it got corrupted during the network transfer.

yup done that and the md5s are different

what does not make sense is why all of a sudden this happened. 

no changes to network have been made

Link to comment

I have a suggestion for one testing mode that you might want to try.  Copy a file the works on your PC to a second PC.  Check both the playback and the MD5 checksum of the file on the second PC.   Now copy the file from the second PC to your Unraid server.  Again check the playback and the MD5 Checksum.  

Link to comment
20 hours ago, Frank1940 said:

I have a suggestion for one testing mode that you might want to try.  Copy a file the works on your PC to a second PC.  Check both the playback and the MD5 checksum of the file on the second PC.   Now copy the file from the second PC to your Unraid server.  Again check the playback and the MD5 Checksum.  

So I did a test.

I copied a file downloaded from PC to Unraid over SMB to another machine on the same switch and same corruption.

However files downloaded on unraid entirely without PC are fine.

Edited by nekromantik
Link to comment
20 hours ago, nekromantik said:

I copied a file downloaded from PC to Unraid over SMB to another machine on the same switch and same corruption.

Please reword this as I can not figure out what you did.  

 

What I asked you to do was:  

 

     (1) On your PC, select a file of the type that plays correctly on the PC and not after it is copied to the server.

 

     (2) Get the MD5 checksum of the file on that PC.

 

     (3) Copy that file to a second PC.  Now get the MD5 checksum of that new copy on this second PC.

 

    (4) Copy that same file from the second PC to the Unraid server.

 

    (5) Get the MD5 checksum of that file on the Unraid server.

 

All of the MD5 checksums should be the same!  If they are not, on which transfers are they different?  (By the way, it is possible that could be different before and after any copy.)  What we are trying to see is where file corruption is occuring.  Knowing where it is happening will suggest the possible sources/cause of the problem.    (You could have bad RAM in one of computers involved, a defective switch/router, etc.) 

Link to comment
36 minutes ago, Frank1940 said:

Please reword this as I can not figure out what you did.  

 

What I asked you to do was:  

 

     (1) On your PC, select a file of the type that plays correctly on the PC and not after it is copied to the server.

 

     (2) Get the MD5 checksum of the file on that PC.

 

     (3) Copy that file to a second PC.  Now get the MD5 checksum of that new copy on this second PC.

 

    (4) Copy that same file from the second PC to the Unraid server.

 

    (5) Get the MD5 checksum of that file on the Unraid server.

 

All of the MD5 checksums should be the same!  If they are not, on which transfers are they different?  (By the way, it is possible that could be different before and after any copy.)  What we are trying to see is where file corruption is occuring.  Knowing where it is happening will suggest the possible sources/cause of the problem.    (You could have bad RAM in one of computers involved, a defective switch/router, etc.) 

Ok

 

So I copied a file from PC to second PC ---- MD5 matches with first PC

Copied file from second PC to Unraid --- MD5 Does not match with second PC/first PC

Edited by nekromantik
Link to comment
52 minutes ago, Frank1940 said:

Now copy that same file from the first PC to the Unraid server.  Get the MD5 checksum of this copy on the Unraid server.  Does that match the MD5 checksum of the file that was copied from the second PC to the Unraid server?  

Nope MD5 does not match between file copied from second PC to Unraid and file copied from first PC to Unraid.

 

So I then copied file from second PC to Unraid using NFS directly from second PC rather then via my windows first PC and MD5 matches.

So issues is copying over SMB to Unraid on my Windows machine is corrupting the files.

Any hints? I would rather use NFS but I do not have Win 10 Pro.

Edited by nekromantik
Link to comment

Let's try one more test.  Copy the file from the Unraid server to the PC that you uploaded it from and check the MD5 checksum again.  

 

My best guess at this point would be that there is a RAM issue on your server hardware--- but ECC RAM should catch this.  (I assume that you are using ECC RAM in this server!)   

 

As I understand it, the  memtst   program that is installed with Unraid will not catch ECC RAM issues.  But there is a version which will.  You can find it here:

 

   https://www.memtest86.com/

 

It will run from a bootable flash drive.  Be sure to run it for 24 hours.  

 

Does anyone else have any ideas of what could be causing this problem? 

Link to comment
4 minutes ago, Frank1940 said:

Let's try one more test.  Copy the file from the Unraid server to the PC that you uploaded it from and check the MD5 checksum again.  

 

My best guess at this point would be that there is a RAM issue on your server hardware--- but ECC RAM should catch this.  (I assume that you are using ECC RAM in this server!)   

 

As I understand it, the  memtst   program that is installed with Unraid will not catch ECC RAM issues.  But there is a version which will.  You can find it here:

 

   https://www.memtest86.com/

 

It will run from a bootable flash drive.  Be sure to run it for 24 hours.  

 

Does anyone else have any ideas of what could be causing this problem? 

Surely it cannot be RAM issue if file copied directly from second PC to Unraid matches md5?

All the mis matches are coming when my windows PC is involved, as both my Unraid server and my second PC are accessed via SMB so all the testing apart from the last test has been via my windows PC. By that I mean, when I copied file from second PC to unraid the first time round I used my windows PC to do that. While the second time I did not use my windows machine and did it directly from second PC.

Link to comment

From what I gather you have two PC's ---  The first one has WIN10 as the OS and the second PC is running some other OS.  Is this correct? And that second PC does not use SMB.  Is this correct?

 

You said you used the first PC to copy the file from the second PC to the Unraid server.  Why did you do this?  The object was to use a separate, distinct SMB path between the second PC and the Unraid server.  You have to initiate the transfer from the second PC.  

 

Itis important to realize that your network could be the cause of this problem.  Your router and/or switch have either a dedicated SOC or a CPU controlling the flow of data through them.  I doubt if SMB is the root cause of your problem.  If it were, this board would have hundreds (if not thousands) of folks screaming...

Link to comment
51 minutes ago, Frank1940 said:

From what I gather you have two PC's ---  The first one has WIN10 as the OS and the second PC is running some other OS.  Is this correct? And that second PC does not use SMB.  Is this correct?

 

You said you used the first PC to copy the file from the second PC to the Unraid server.  Why did you do this?  The object was to use a separate, distinct SMB path between the second PC and the Unraid server.  You have to initiate the transfer from the second PC.  

 

Itis important to realize that your network could be the cause of this problem.  Your router and/or switch have either a dedicated SOC or a CPU controlling the flow of data through them.  I doubt if SMB is the root cause of your problem.  If it were, this board would have hundreds (if not thousands) of folks screaming...

Correct first PC is Windows 10 and I mount shares for Unraid.

Second PC is a Intel NUC with Ubuntu on it. That runs my media player and connects to Unraid via NFS.

 

So it looks like my Windows PC is the issue then as when initiating transfer from Ubuntu to Unraid MD5 remains same but when using my Windows PC to copy to Unraid MD5 does not match. 

My network setup is Windows is wireless and Ubuntu and Unraid are wired in. My router is intel mini pc with pfsense and wifi access point connected.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.