PLEASE HELP! - I think I may have deleted most of my array :(

[KptnKMan]

Hi, if anyone can help me please I would be appreciative.

I just got my Unraid server up and running and everything has been running pretty sweet for a few weeks.

Today, I have attempted to delete the '.DS_Store' and '._.DS_Store' files from my server so I ran a few commands in the unraid terminal.

The commands I ran:

find . -name '.DS_Store' -type f -delete && find . -name '._.DS_Store' -type f -delete

find . -type f -delete -name '.DS_Store' -name '._.DS_Store'

I definitely didn't run any other commands, or had any other malicious process running.

Luckily I managed to stop the process when I noticed some files had gone. As it is, a bunch of critical stuff got deleted.

I'm not sure how this happened, but it seems that a bunch of personal and critical data has been deleted. In a panic, I stopped everything, and rebooted the server.

Now it seems as though the system files were deleted also, and the server has come up without any of my VMs or Docker containers present.

The array drives are formatted using XFS.

Does anyone know how this could have happened using these commands?

Does anyone know how I can restore my XFS drives?

I've stopped the array and trying to research what to do.

I know I should backup, but I had not yet configured a Cloud Backup for this server. It's definitely been high on my list.

[Squid]

I don't run random commands, so not going to particularly analyze what you typed in, but will say that the script included with User Scripts that does the same thing is this:

find /mnt/user -maxdepth 9999 -noleaf -type f -name ".DS_Store" -exec rm "{}" \;

 

As far as recovering, you can pull the drive and attach it to your Mac and then install and run (and unfortunately pay for) https://www.ufsexplorer.com/ufs-explorer-standard-recovery.php  (I've run this on Windows and it works great (albeit I used it for a different reason) )

[KptnKMan]

I'm not using a Mac anymore, hence why I wanted rid of these files.

I didn't have an issue with this before, not a random command, but this is very odd behaviour I think.

I'm still pretty shocked this happened, I think it may have been a syntax error.

 

Hey thanks, I'm currently looking at:

https://www.ufsexplorer.com/raise-data-recovery-xfs.php

 

I think this is an older version though, so best I scan with the newest.

 

Thanks for your help.

[KptnKMan]

As I suspected, there was a syntax error in my command, in particular with the '-delete' placement. It should always be at the end or will ignore the filter and assume deletion of everything. I find this pretty stupid, but it is what it is, I should have checked the small print. So I'm the stupid one.

Ironically, what seems really lost was my 'backup' share that I had setup. This was intended to be the source of a clone to my old decommissioned server, when that was brought online as a backup server later this week. I'm trying to restore this, as the array has not been overwritten.

Unfortunately, this happened first.

 

As it turns out, after more careful review, most of what is lost can be replaced or regenerated. The 'system' and 'appdata' is basically lost, and locked onto the cache drives in btrfs, which I'm trying to reconstruct. I noticed that all my VM and Docker configurations disappeared from the UI.

 

I'm reconstructing the cache drives, and trying to pull the file structure off it. I'm having some success using recovery tools.

ReclaiMe Pro and UFS Recovery seem to be doing pretty well so far, but we'll see.

Saying that, is there any way I can restore my docker settings using a backup (Meaning what I pull of)?

Also, for example in a normal backup/restore scenario, how do I backup/restore 'system' data and configuration?

 

[Squid]

1 hour ago, KptnKMan said:

Saying that, is there any way I can restore my docker settings using a backup

If you've made a backup of the appdata, then restore it.  All of the settings for the templates though are on the flash drive, and will be restore-able via Apps - Previous Apps

[KptnKMan]

Thanks @Squid I managed to get everything restored.

I used UFS Explorer Standard Recovery and ReclaiMe Pro to recover my files.

For anyone who might want to do this in future, using UFS Explorer for each drive individually (Not as RAID) works fine, as the virtual share filesystem is split out across the actual separate filesystems of the disks.

 

I restored each disk separately, to another location, then recombine the filesystem manually.

I'm starting to copy them back to the array now.

[KptnKMan]

If anyone is able to help though, I'm experiencing a strange networking problem bringing my docker containers back online.

Before restore, I was able to run containers that used VNC, but now I seem to be getting some strange port conflicts.

 

I have a VM running, with VNC enabled, and now any dockers using VNC redirect to it.

 

I'm also unable to run multiple dockers with VNC, like binhex-crusader and binhex-preclear.

I get an "execution error", and I can't see the logs showing anything useful.

[Squid]

Just now, KptnKMan said:

binhex-crusader and binhex-preclear.

Change one of the host ports.  IIRC, they both use 6080 by default

[KptnKMan]

Yes, they use the same port.

Why was I able to run them together before without changing anything?

Seems like something has changed.

[KptnKMan]

I manually changed the ports to 6081, and restarted containers.

Still redirecting to my VM, and it doesnt seem to reflect in Unraid UI.

[KptnKMan]

Its the strangest thing, I set new ports in the UI.

I've ven tried deleting the container, and starting again.

 

And it still shows as 6080, even after deleting, new, set correctly.

I'm not sure what's going on.

[KptnKMan]

Docker log for the container is repeating:

2019-10-12 21:19:49,920 DEBG 'start' stderr output:
No protocol specified
tint2: could not open display!

2019-10-12 21:19:59,926 DEBG 'start' stdout output:
[info] tint2 not running

2019-10-12 21:19:59,935 DEBG 'start' stderr output:
No protocol specified
tint2: could not open display!

2019-10-12 21:20:09,939 DEBG 'start' stdout output:
[info] tint2 not running

2019-10-12 21:20:09,946 DEBG 'start' stderr output:
No protocol specified
tint2: could not open display!

 

[Squid]

You're running them as host.  Switch to bridge

[KptnKMan]

Switched them back to bridge, and they are still inaccessible.

New port shows up in UI, but nothing is accessible on those ports.

 

Emptied my browser cache, etc, which I've seen suggested in other threads.

I'm starting to think that this installation is borked.

Edited October 12, 2019 by KptnKMan

[Squid]

And hitting webUI when you click on the icon also does nothing

[KptnKMan]

I'm afraid not, it's very strange. It seems like the docker UI is somehow only half connected to the config. 

 

When I changed the port, I noticed the the Web UI link did not update, but I fixed that and it still doesn't work when pointed to correct port. 

 

My VMs start fine, and the dockers can start, but the docker networking seems to be all weird.

[KptnKMan]

At this point, I'm wondering if I should cut my losses here, and just backup my USB, reformat USB and start over as a "new server", then import my array disks. 

 

Is that possible?

Edited October 12, 2019 by KptnKMan