The short question is - why did this happen?  I found instructions to fix it with xfs_repair (I'll have to use -L) but how can I prevent this from happening again? Does this indicate an issue with the drive, or just the filesystem?

 

The longer version...

 

I'm weeks into this hardware migration and I'm at my wits end.  All the troubleshooting steps take days to run and are inconclusive.

 

I've migrated to new hardware (used 730xd) with my old drives (2x2TB) and added four new ones (4x12TB refurbished, from two different sources). All 4 new drives were precleared and tested.

 

I began migrating files from windows drivepool (over samba) with Teracopy. Teracopy validation passed. Later I noticed issues with some files. VLC gave index errors for AVI files. Media files would play, but not to the end. File sizes were correct, but MD5 on source/destination was now different.

 

I tried again with rsync. The results were the same - rsync did not report any errors, but afterwards MD5 did not match.

 

I posted on the forums and got a recommendation to run memtest, which I did over a weekend. No errors.

 

I tried a few other things (creating single disk shares to try to isolate the issue to a specific drive) but those have been inconclusive, mostly because I started to get lost with which files had been tested.

 

The majority of files had been assigned to disk2, so I set the share to exclude disk2 and copied a large subset of files. I spot checked files with manual MD5 as the copy was in progress and they were all fine. The copy finished yesterday and today I was getting ready to run the batch MD5 on all files. Sometime this morning I got the corruption error.

 

One more odd thing I noticed right before this... I was using gvim to edit a file on the share, over samba.  I would make some changes, save the file, make some more changes ... and when I went to save again, vim warned me the file had changed on disk. If I reloaded the file I would find that my last change had not actually been saved.

 

Any insight or suggestions on any of this would be greatly appreciated.

Thanks, C

 

tangerine-diagnostics-20240724-0954.zip

Both issues suggest a RAM related problem, memtest is only definitive is it finds errors, though the server appears to be running ECC RAM, so it could also be a board/CPU problem, I would retest with half the RAM installed, if issues continue try the other half, that will basically rule out the RAM.

10 minutes ago, JorgeB said:

Both issues suggest a RAM related problem, memtest is only definitive is it finds errors, though the server appears to be running ECC RAM, so it could also be a board/CPU problem, I would retest with half the RAM installed, if issues continue try the other half, that will basically rule out the RAM.

 

Thanks for the analysis.  It is ECC; that and memtest had me thinking the ram was OK.  I have two fully new RAM chips coming this week I will test with. I can't run just one chip because minimum 1/processor is required (I think).

 

Are there any diags I can run to rule the cpu(s) out?

13 hours ago, carlel said:

Are there any diags I can run to rule the cpu(s) out?

AFAIK does not easy to test, you may be able to test by using just one CPU at a time, unlikely that both would have issue.

With new memory installed I recopied all the data then logged and compared an md5 on each file against the original. No corruption yet.  Today I turned docker back on and am slowly re-enabling containers one at a time.  I'll keep validating MD5 as I turn containers back on.  I still wonder if some bad configuration could have caused this but I'm not sure I'm interested in taking a chance on putting the old RAM back in.

 

I'll update back here if anything changes. Thanks again.

8 hours ago, carlel said:

I still wonder if some bad configuration could have caused this

I think that is unlikely, but please keep us updated.

I should add that the Vim issues I was seeing are apparently a problem between Vim and Samba; (Suppress/bypass "file has changed" errors when editing cifs/Samba files).

 

That leaves the only unexplained issue as the media file corruption, which hasn't yet reoccurred. A few more dockers are running now and in addition to the manual MD5 indexes I've built I'm running Dynamix File Integrity to help keep an eye on it.