I have replaced a drive in my array yesterday and parity was rebuilding the contents.  Before the rebuild, the files were emulated and accessible.  But today, about 10 hours into the rebuild, it appears all files were deleted from the array.  I don't really understand what is happening.. All the modified times for the folders with missing data is around 2:15pm 12/30.  So, it appears something in the rebuild process deleted the files?  Not sure what to do.  It is still rebuilding and at 80% complete.

freenasmedia-diagnostics-20221230-1622.zip

I stopped the rebuild and checked what was emulated, and it didn't emulate anything.  Going to chalk this up to something incorrect I did last night.  It doesn't make sense, but I don't see a way to get back the data.

@guythnick

Rebuild can't delete files since parity doesn't know anything about files, just bits.

 

Those diagnostics show it was rebuilding disk2, and that emulated disk2 was Unmountable. So, it was rebuilding an unmountable filesystem.

 

Since you stopped the rebuild not clear where you are now but I expect whether it was rebuilt or not, disk2 is still unmountable.

 

It should be possible to fix the filesystem on disk2 and get some if not all of its data back.

 

Attach new diagnostics to your NEXT post in this thread.

 

 

 

 

56 minutes ago, guythnick said:

It doesn't make sense, but I don't see a way to get back the data.

Out of memory occur, some process have crash, you should reboot first. 

 

Dec 30 08:28:46 FreenasMedia nginx: 2022/12/30 08:28:46 [crit] 10382#10382: ngx_slab_alloc() failed: no memory
Dec 30 08:28:46 FreenasMedia nginx: 2022/12/30 08:28:46 [error] 10382#10382: shpool alloc failed
Dec 30 08:28:46 FreenasMedia nginx: 2022/12/30 08:28:46 [error] 10382#10382: nchan: Out of shared memory while allocating message of size 522. Increase nchan_max_reserved_memory.
Dec 30 08:28:46 FreenasMedia nginx: 2022/12/30 08:28:46 [error] 10382#10382: *208992 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/cpuload?buffer_length=1 HTTP/1.1", host: "localhost"

 

Want to put a pin on this - after the second rebuild finished the drive was still unmountable.  I did some searching online and it looks like running xfs_repair was the right way to try to fix it.  I started the array in maintenance mode and tried to just run 'xfs_repair -n', but it showed an error stating basically I needed to rebuild the log with the -L switch.  I then re-ran with 'xfs_repair -L' , which completed in a couple minutes. Then after starting the array, I now have all the missing data! 

Thanks for the replies.

Do you have a lost+found share after repair?

No, it appears that the directory structure was maintained.