Lost 6,000 files during server crash

[wilddog]

Hi, I'm was uploading a large set of video files when my Unraid server become unresponsive. 

I do have a monitor and keyboard on it, but it would not respond.  Finally I power cycled it get get it back online.

However, the folder 'PeriscopeFilms', into which I was upload all the files, was not there. And all the files in that folder seem to be gone.

 

1. Is there a way to recover the missing files?

2. I need to solve why it crashed.

 

Thanks for any advice!

wilddog-diagnostics-20220305-0951.zip

[Squid]

Are you supposed to have a cache drive / pool?

[wilddog]

I have the SDDs installed, but not configured as a cache yet. 

[Squid]

Then what share is PeriscopeFilms supposed to be in?  How were you copying?

[wilddog]

I have a share, utube, in which I created a folder PeriscopeFilms.  Into that folder I was moving files over the network from a linux client machine.  

It's weird that after booting back into the machine, that folder was no longer there and the files in it don't seem to be anywhere obvious. It's about 1 TB of files.  I was hoping they're hiding somewhere. I used QDirStat to see if any big chuck of data appeared someplace, but I didn't see anything unusual.   I'm wondering if there's an audit tool or someway to find abandoned files or at least some indication that 6,000 files had been uploaded, but may become 'nuked' when the parent folder goes away. Why the parent folder would go away in a crash was weird. I'm hoping it's all still there, but just not visible to the file system for some reason. 

[Frank1940]

How are you looking for the missing files? 

 

The reason for this query is that Windows ignores capitalization in file names.  Thus, Movies and movies are consider to be the same resource and only the first 'discovered' will be accessible.  (Windows would not allow the second one to be created.)   With Linux, capitalization is used.  So  Movies and movies are two different resources.   Thus, Linux will allow both of these to be created.  The problem is that one of them can not be seen by Windows.    (As I understand it, it is arbitrary which one is accessible!)  

 

So have a look for the missing files from your Unraid server.  You can use  the Linux  ls utility from the command line, the builtin  mc  file manager, or the Krusader Docker. 

 

You want to start by going to    /mnt/user     and go down the file structure to see if you can find the missing files.

[wilddog]

Good point to use mc or Krusader to look in /mnt/user for the files (instead of from the network share on a client, where it may look different). But not finding the /utube/PerspectiveFilms folder or files that were under it. I doubt there's a journal log or something similar that would indicate all those files were written to the array of the last week. But that's probably too resource intensive. But maybe some other type of log/indicator all those files were recently created?

 

The share settings for utube:  High-water allocation, automatically split any dir as required, includes all disks. Cache pool: no.

 

This system:   Last year, I converted my previous desktop, a mini tower system76.com about 6 years ago; with an i7 and 64 GB of RAM its good for virtualization (for a windows desktop and a few lnx VMs).  I've mainly been uploading archives up to it (drives: 16 (parity), 16, 14, 12) for 42 TB of storage.   But during this week of lots of uploading, I noticed the upload of a few hundred GB stalled part way through.  And then the next day it had the crash.  So maybe a hardware issue going on.

 

 

 

[remotevisitor]

When you said you were moving files over the network from another Linux machine, how were you doing the actual moving?

 

I’m wondering if you copied them into /utube/PerspectiveFilms instead of /mnt/user/utube/PerspectiveFilms.

 

This would explain both your system becoming unresponsive (root file system becoming full) and why they have disappeared after the reboot (because the root file system is in RAM)

[wilddog]

Thanks for the question, but I moved the files to the server like all my other uploads, by accessing the network share from a linux (mint 20.3) machine.  

 

I do see all my other folders under /mnt/user/utube. 

 

But the folder to which I was uploading to, /mnt/user/utbue/PeriscopeFilms, disappeared when the server became unresponsive and I power-cycled it.  Doing a global search of example files I had previously uploaded there across the whole file system seems to confirm they are not hiding somewhere.

 

So far the suggestions seem cover good checks such as making sure the user didn't access the server in a odd way and somehow put the files where not expected.

 

I was going to run a backup of that folder since it was so large to an offline 16TB drive that I'm currently used for my 2nd level backup. 

I never would have expected Unraid would 'burp' during a freeze-up of the system with a hard reboot and remove this folder, resulting in the loss of the thousands of files uploaded to it over a week. 

[trurl]

Are you sure you didn't transfer them to one of the several Unassigned Devices?

[wilddog]

Yes, as the Unassigned Devices are smallish SDDs (1TB and 500GB) and the volume of data I uploaded to the share is around 2.5 TB.

Plus the process of moving data to the server is the same as all the data already there. Those folders are still in /mnt/user/utube. 

It's just the open folder where I was uploading to during the system-freeze event that disappeared after the hard boot.

[wilddog]

I appreciate everyone helping to eliminate if this data loss was a user error (are you looking in the wrong place? etc.).

Does not appear to be a user error.

 

I've not uploaded anything else to this server after having such a hard data loss, hoping to find the root problem.

I'll move on to looking at the hardware and any I/O errors. Check cabling and such.  Any other logs I could check?

 

I'm starting on building a new server as I'm struggling to trust this one now.