Hi everyone. I don't know what went wrong tonight, but today I noticed that almost all my shares are empty.
I checked with the built-in file manager, and I also checked the individual disks by running some ls commands in the terminal. It's like something selected almost all the files and hit Delete.

Dumbly enough, I have a script that runs rsync to another mini NAS for my most important data, of course mirroring everything, because I don't have that much space on that machine.
I see that those folders are empty too (but fortunately, this NAS has a recycle bin, and I'm finding some files in there), so the script did its job... 😣

I really can't imagine what went wrong. The last parity check was run about two weeks ago, so I don't think it was that, although I have Parity Check Tuning that cut the task in pieces, and stops if the HDDs become too warm etc.

I also had the Txtify Docker container running and working on transcribing some text. I was testing it in the afternoon, and I noticed that the progress windows were stuck at a random percentage, but I don't know if that could have been the culprit.

I have several HDDs, two of which are set as parity drives.

So, the question is: I'll probably never find out what happened (although I hope you might be able to find something in the Diagnostics...), but is there any terminal command that can perform some sort of undelete, scanning all those HDDs for files that MAYBE are still physically there?

Of course, thanks in advance!

nastower-diagnostics-20260917-1929.zip

The diagnostics show that all five data disks mounted cleanly. Combined with your direct-disk checks and the mirrored backup loss, this suggests deletion, although the cause is still unknown.

First, pause the rsync job and any cleanup schedules. Stop Docker containers and other writers, then stop the array to preserve recovery options. Preserve the files in your backup NAS recycle bin.

You also have Unraid’s Recycle Bin configured on several shares. Look for the missing files there too. It covers deletions through SMB; direct container or script deletions bypass that protection.

A few questions could help narrow this down:

- Is the WebGUI, SMB, or any application directly accessible from the internet through port forwarding, UPnP, or a router DMZ?

- Which host folders can Txtify write to? Does it have access to an entire share or /mnt/user?

- Are any media applications, cleanup tools, or scheduled scripts allowed to delete files? Was Appdata Cleanup Plus used recently?

- Can you post the rsync command with credentials and personal details removed, while preserving its options and source/destination direction?

A parity check would never delete any data, and the parity-check date does not provide a recovery point. Parity updates with writes, including deletions, so rebuilding a disk will not restore its earlier contents.

There is no built-in command that reliably undeletes everything across the array. If the recycle bins cannot supply the files, recovery needs a separate XFS recovery plan, preferably using disk images. Avoid formatting or filesystem repair as an undelete attempt, and save recovered files to separate storage.

Thanks for the first reply.

I stopped schedule for:
- Appdata Backup
- User Scripts
I also have this one, don't know if could be problematic: Dynamix Cache Directories

This is the script I used for mirroring:
#!/bin/bash

if mountpoint -q /mnt/remotes/NASQNAP112_backups; then

rsync -avh --progress --delete --stats /mnt/user/1/ /mnt/remotes/NASQNAP/1_backup/

rsync -avh --progress --delete --stats /mnt/user/2/ /mnt/remotes/NASQNAP/2_backup/

rsync -avh --progress --delete --stats /mnt/user/3/ /mnt/remotes/NASQNAP/3_backup/

else

echo "NAS QNAP ko!"

exit 1

fi

- Is the WebGUI, SMB, or any application directly accessible from the internet through port forwarding, UPnP, or a router DMZ?

No, I don't think so.

I use Nextcloud that has linked mounts to several unraid shares, however I use it thru Tailscale.
Thinking about it, apparently the three shares I mounted for use them as "external" in Nextcloud are those totally zeroed, while I can't be totally sure about it.

And/or maybe the Mariadb that it's installed 'cause it's necessary for Nextcloud?

Txtify has very few settings in the configuration. No mentions about the shares it can use. The elevated setting is off.

I'm also trying to do some elementary check with the help of chatgpt. Sadly, I'm not a pro-linux user...
The various .Recycle.Bin and .Trash-99 are empty, just one of them has some file, not important, for a share not important.

Edited September 17Sep 17 by Andrea P.

With this command:
docker logs --since 48h Nextcloud 2>&1 | grep -Ei 'delete|deleted|unlink|remove|scan|external|trash|error|exception' | tail -500

I got several lines that left me baffled, like those ones (I changed the names of the folders, here):

rsync: [generator] delete_file: rmdir(ext_1) failed: Device or resource busy (16)

rsync: [generator] delete_file: rmdir(ext_2) failed: Device or resource busy (16)

cannot delete non-empty directory: ext_2/Photo/Cat

cannot delete non-empty directory: ext_2/Photo/Dog

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1347) [sender=3.4.1]

rsync: [generator] delete_file: rmdir(ext_1) failed: Device or resource busy (16)

rsync: [generator] delete_file: rmdir(ext_2) failed: Device or resource busy (16)

rsync: [generator] delete_file: rmdir(ext_3) failed: Device or resource busy (16)

Also, several:

=> Configuring PHP session handler...

==> Using default PHP session handler

Initializing nextcloud 34.0.4.1 ...

Upgrading nextcloud from 34.0.3.2 ...

Don't tell me that nextcloud did a mess with the mounts ext_1 ext_2 ext_3 while it was trying to update himself... 😨
Also, I see that when I start it, after a while it goes down, and of course it doesn't work.

This was the setting I used in the Docker setup:

This is the first part of the Docker setup:

Honestly, I can't remember if I selected /var/www/html/etcetc as the "base" folder to mount my external mounts, and/or if I followed some guide, however if the Nextcloud rsync did something in those folders, surely it got my files too :-/

Edited September 17Sep 17 by Andrea P.

Those Nextcloud logs are a strong clue. The container’s upgrade script uses rsync --delete to update /var/www/html. Nextcloud’s container documentation specifically warns that custom storage mounted inside this directory can lose data during upgrades unless excluded.

If your shares were mapped to /var/www/html/ext_1, /var/www/html/ext_2, and /var/www/html/ext_3, that would fit what you describe. Those mappings expose the original Unraid files inside the container. Deleting files there deletes them from the shares too.

The “Device or resource busy” errors do not mean the files were protected. The mounted directory itself can resist removal after files inside it have already been deleted. The errors do not establish exactly how much was deleted.

Keep Nextcloud stopped and disable its autostart. Repeated starts could attempt the upgrade again. Keep the backup script disabled and preserve the current container configuration and logs before reinstalling or changing anything.

Please post the Nextcloud image repository/tag and the three Host Path → Container Path mappings, including their read/write settings. You can replace personal folder names, but preserve the container path structure.

Your backup script’s --delete option can explain the subsequent backup loss: it removes destination files that are missing from the source. As posted, it does not request deletion of the Unraid source files.

For now, preserve the backup NAS recycle bin and recover available files to separate storage. Keep the Unraid array stopped to preserve recovery options for anything still missing.

So, let's say that probably / almost certainly the Nextcloud update caused that mess. Of course, it's my fault for having installed it so easily, thinking that a Docker container could be relatively safe as a QNAP app on my former NAS...

Now, what can I do to try a restore / undelete? Surely the files didn't move to any recycle bin. In my old, myopic understanding, they're still on the HDDs, just marked as "deleted" and their space marked as available for reuse (of course, some block could have been overwritten, no problem about that).

Is there any tool / terminal command I can use to search for them?

Edited September 18Sep 18 by Andrea P.

You can use a file recovery app, like UFS Explorer, the free trial should show what it can recover, the license is I belive around $60 last time I checked. Don't write anything to the disks before that.

I’d avoid running any undelete or repair command for now, especially if the files may still be recoverable. First check the Unraid recycle bin and make sure nothing is writing to the affected disks. It may also be worth checking the Docker/container activity and system logs around the time the files disappeared. If the disks use fnaf XFS, recovery tools can sometimes help, but they should be used carefully on unmounted disks to avoid overwriting recoverable data. The fact that the rsync backup still has some files in its recycle bin is a useful clue.

Edited October 1Oct 1 by dampthieve

Yeah, thank both of you. I still have Unraid stopped and "frozen". I tried to remove a disk and attach it to a different PC to try some recovery, however I'm partially prone to delete everything and restart. The files I found after the software analysis are totally without structure, so no folders, no filenames (not even partial).

Surely, I recovered all the really critical stuff thanks to the recycle bin in the old Qnap that I used as timed backup, because the ignorant Rsync I set up deleted everything in there too, but fortunately Qnap moved all the deleted files to its bin.

All the other old archives aren't critical, and it will be a pain to rebuild them. First because I'd need additional storage to do the recovery, and I don't have it (maybe I could remove one of the two Parity HDDs I use in Unraid, but it's still too little). Then, I could end up with hundreds of old copies, and/or partially unreadable pieces, so I'm really discouraged and I'm leaving all the stuff as it is right now.

Well, I'd say to close the topic and mark it as solved, at least I learned a thing or two about the risks with mounting folders in the Dockers... damn, that Docker update got me by surprise...

Edited October 3Oct 3 by Andrea P.
wrong copy/paste