Unresponsive shares and empty files

[20_100]

Hi

 

I use Unraid to backup files for a web app. 

During some peaks, hundreds of files can be queued to be processed.

The app downloads the file to a temp storage (unrelated to Unraid), then parses and analyses it, then copy it to the cloud for storage and to unraid for backup.

4 service workers actually work concurrently to process these queued files.

 

These files are between 50MB to 150MB

 

During the peaks, I have two problems

 

- Copying the files from the service worker drive to the share seems to work, but the target file is actually empty (0 byte). That one problem doesn't even happen only during peaks. It happens frequently even when the web app traffic is slow and there are not that many files to process.

 

- It gets worse and the share become unresponsive 

 

It happens multiple times a day, and I have many files which are 0 byte. They aren't originally 0, and the files stored in the primary location in the cloud are not empty. Only the unraid copies are empty.

 

One episode happened recently between 3:13 and 3:25 today

 

 

 

The thing is for the first empty file, the one named AC0DDDF... , the app workers log don't show an error : the program thinks the file has been correctly copied to its destination, the unraid share.

 

For some of the other files, the worker crashed during the copy :

 

After a few minutes the shares become responsive again.

 

A few remarks :

- I tried with and without using a cache pool. There are empty files on the cache pool drives if I use one.

- 2 workers (of the 4 total) run on a VM hosted on the same Unraid instances. 2 other workers are on a different servers. The 4 workers have the same issues.

- I have been having this issue for 2 years now

 

 

 

 

 

 

sunraid1-diagnostics-20211025-2059.zip

Edited October 26, 2021 by 20_100

[trurl]

31 minutes ago, 20_100 said:

Copying the files from the service worker drive to the share

Is this "service worker drive" part of the array or cache?

[20_100]

Hi , it's a RAM drive created by a VM hosted on the same unraid instance

Edited October 26, 2021 by 20_100

[trurl]

3 hours ago, 20_100 said:

RAM drive

Have you done memtest?

[20_100]

Yes, there is 128gb of ram and a memtest takes 3 days but it pass. The files are never 0 MB on the ramdrive. Also the vm uses the ramdrive for some other heavier processing and it works flawlessly