[6.11.5] Silent file corruption when using the nvme cache (trimmed file, parts of file replaced with zeroes)

Not sure if it is related to your error, but there are a lot of 

May  6 00:21:58 vs-tower  shfs: share cache full

In the syslog.   You do not have a Minimum Free Space value set for your ‘cache’ pool - it should be set to be more than the largest file you expect to write as file systems getting too full can have unpredictable effects.

 

There are also a lot of lines of the form:

May  6 03:05:19 vs-tower kernel: CIFS: __readahead_batch() returned 870/1024

In the syslog.  Not sure what they mean but they are not normal.

 

neither of these, though, explain why there should be corruption   if you run a scrub is the corruption detected as since the pool is formatted btrfs it should automatically have checksums associated with it to check integrity.

 

CIFS: __readahead_batch()

 

I only see it when I mount a SMB share using unassigned devices and pass it into a docker volume as read/write-slave. But in my case I do it only to verify the already transferred files using freefilesync docker in unraid. When I copy files from synology I don't see such messages and don't mount anything with unassigned devices plugin.

I will set the minimum free space and try the transfer again.

 

 

About the 1 megabyte of zeroes that is inserted into a file.

I remember  that I was getting warnings that the nvme was hot (50C or more) so I went to the cache settings and changed Warning/Critical disk temperature threshold (°C) while synology was transferring data, maybe changing this setting caused sometething to the nvme and it just wrote a megabyte of zeroes.

 

It should not be a SATA controller issue because the mover placed 2 movies on 2 different HDDs, one was connected to the Motherboard, 2nd to the PCIE LSI Broadcom controller. Both eded with a megabyte or zeroes in the middle.

 

When I didn't touch any cache settings, the only issue was the file size, I didn't find any files with an "empty" region.

 

The SSDs are cheap SiliconPower

 

 

This looks more like a device problem, if you transfer directly to the array do you still find any corruption issues?

If I set cache to "No" then there's no corruption. The main issue is that both file truncation and zero block insertion isn't detected in any way even in raid1 and the copy isn't aborted with an error. If cache is set to "prefer" then synology gets an error that there's no free space and aborts the copy.

 

Yesterday I tested with  the Minimum free space set to 10GB on the Cache and the 1.9TB were transferred correctly, in previous tests  it was set to 0 (default). 

Edited May 7, 2023 by vasiby

57 minutes ago, vasiby said:

If cache is set to "prefer" then synology gets an error that there's no free space and aborts the copy.

Cache=yes and Cache=prefer have the same behavior regarding that, data is written to cache, if there's a minimum free space set the transfer will overflow to the array, if there's not the cache will get completely full and the transfer abort with an ENOSPC error.

Looks like in my case the transfer abort error didn't happen and synology just kept copying the file and the rest of the movie library did overflow to the array.

Edited May 7, 2023 by vasiby

I don't know how that could be an Unraid issue, please retry transferring from a PC, if it does as expected retry with the Synology, what were you using to tranfer the files?

I've repeated the transfer several times more, there are no problems if the minimum cache size is set to 10G or 100G, so that there's some free space left on the pool.

when it's set to 0 the movie can't fit and only half of it remains in the cache, but the synology continues copying and reports success at the end and doesn't receive the abort error.

 

I couldn't reproduce the 1 megabyte of zeroes again.

Edited May 10, 2023 by vasiby

Did you try with a PC? i.e. is the problem only transferring from the Synology?

I mounted syno and unraid folders (1.9TB)  into a windows11 PC and it aborted the copy when the cache is full, also there's no partially copied file in the cache. Don't know why it asks for 892gb free space, the folder itself is 57.7GB, but anyways I got the error unlike synology.

 

 

Also I get this in syslog when copying from windows

 

 

Edited May 10, 2023 by vasiby

You can ignore those SMB warnings, and Synology should also abort the copy, if it doesn't it's probably a Synology issue, or of the application you are using to make the copy.