Many Missing Files


Go to solution Solved by Cybernaut,

Recommended Posts

Hi Folks.

 

I'm hoping someone can help me resolve this.

 

So, I have Radarr/Sonarr set up to notify me of changes via Discord.  I get up this morning and Discord tells me 10 - 15 TV episodes were deleted.  I did some investigating but couldn't find the problem.  Shrugged it off.  Then, this afternoon, Discord tells me 80 movies were deleted.  After further investigation, most of the files in my Documents share are missing.  I don't think it's Radarr, Sonarr or Plex, because none of them have access to the Documents share.  The shares are still there.  The folder structure is still there, just missing files.  Looking at folder modify dates, they appear to happen in batches.

 

I read somewhere to try ls -al to look for hidden files and get this: 

root@Tower:/mnt/user/data/media/movies/Ad Astra (2019)# ls -al
total 12
drwxrwxrwx 1 mike users   10 Jun 30 20:56 ./
drwxrwxrwx 1 mike users 8192 Jun 24 15:55 ../

 

Thanks for any guidance.

tower-diagnostics-20220701-1632.zip

Link to comment

Lots of this in syslog

Jun 17 11:54:07 Tower winbindd[15571]: [2022/06/17 11:54:07.633588,  0] ../../source3/passdb/pdb_smbpasswd.c:1251(build_sam_account)
Jun 17 11:54:07 Tower winbindd[15571]:   build_sam_account: smbpasswd database is corrupt!  username admin with uid 1001 is not in unix passwd database!

Any idea what that is about?

Link to comment
16 hours ago, trurl said:

Lots of this in syslog

Jun 17 11:54:07 Tower winbindd[15571]: [2022/06/17 11:54:07.633588,  0] ../../source3/passdb/pdb_smbpasswd.c:1251(build_sam_account)
Jun 17 11:54:07 Tower winbindd[15571]:   build_sam_account: smbpasswd database is corrupt!  username admin with uid 1001 is not in unix passwd database!

Any idea what that is about?

Thank you for replying, trurl.

I'm not sure.  I went back and looked at syslog form December, when I first moved to unRAID, and it was there too.  I was playing around with things then, and might have created a user admin and later deleted it.

Quote

Dec 15 14:13:39 Tower userdel[7355]: delete user 'test'
Dec 15 14:14:03 Tower sudo:     root : TTY=pts/0 ; PWD=/root ; USER=root ; COMMAND=/usr/sbin/userdel -r test
Dec 15 14:14:35 Tower userdel[7554]: delete user 'admin'
Dec 15 14:16:08 Tower winbindd[5820]: [2021/12/15 14:16:08.947523,  0] ../../source3/passdb/pdb_smbpasswd.c:1250(build_sam_account)
Dec 15 14:16:08 Tower winbindd[5820]:   build_sam_account: smbpasswd database is corrupt!  username test with uid 1003 is not in unix passwd database!

I didn't find a way to fix this with a few minutes searching, but will continue looking.  This wouldn't be causing my missing files issues, would it?  Best I can tell that just started on 6/27.

As far as FTP, I didn't intentionally set it up, and as far as I can tell, it's not.

 

Link to comment

Arrgggg!  I caught it after another 20 movies deleted this afternoon.  That's 188 movies, a couple dozen TV shows and most of my documents share!  I immediately shutdown Plex, Prowlarr, Radarr, Readarr as well as Mariadb and nexcloud dockers, and it continued deleting movies every couple minutes.

 

I rebooted the server which seems to have stopped the bleeding.  I did the Check Disk Filesystem procedure (without repair).  All disks looked pretty much the same.  Is this a normal result??

Quote

Phase 1 - find and verify superblock...

        - block cache size set to 1471600 entries

Phase 2 - using internal log

        - zero log...

zero_log: head block 2676720 tail block 2676720

        - scan filesystem freespace and inode maps...

        - found root inode chunk

Phase 3 - for each AG...

        - scan (but don't clear) agi unlinked lists...

        - process known inodes and perform inode discovery...

        - agno = 0

        - agno = 1

        - agno = 2

        - agno = 3

        - agno = 4

        - agno = 5

        - agno = 6

        - agno = 7

        - process newly discovered inodes...

Phase 4 - check for duplicate blocks...

        - setting up duplicate extent list...

        - check for inodes claiming duplicate blocks...

        - agno = 0

        - agno = 4

        - agno = 1

        - agno = 3

        - agno = 2

        - agno = 5

        - agno = 6

        - agno = 7

No modify flag set, skipping phase 5

Phase 6 - check inode connectivity...

        - traversing filesystem ...

        - agno = 0

        - agno = 1

        - agno = 2

        - agno = 3

        - agno = 4

        - agno = 5

        - agno = 6

        - agno = 7

        - traversal finished ...

        - moving disconnected inodes to lost+found ...

Phase 7 - verify link counts...

No modify flag set, skipping filesystem flush and exiting.

 

        XFS_REPAIR Summary    Sat Jul  2 15:44:34 2022

 

Phase                   Start                     End                       Duration

Phase 1:               07/02 15:44:23  07/02 15:44:23

Phase 2:               07/02 15:44:23  07/02 15:44:23

Phase 3:               07/02 15:44:23  07/02 15:44:29  6 seconds

Phase 4:               07/02 15:44:29  07/02 15:44:29

Phase 5:               Skipped

Phase 6:               07/02 15:44:29  07/02 15:44:34  5 seconds

Phase 7:               07/02 15:44:34  07/02 15:44:34

 

Total run time: 11 seconds

Please help me figure this out before I lose everything!

Link to comment
17 minutes ago, trurl said:

Are you sure you aren't being hacked?

 

Can your server be accessed from the internet?

 

 

Check in Settings - FTP.

That did cross my mind, though.  I am accessible through the My Servers plugin.  I guess I could disable that and change my pw.  But the nature of the problem doesn't suggest it.  With Ransomware, they work really fast to get your stuff encrypted before you notice.  To delete files during sporadic time frames? I would think they'd be more efficient in their destruction, but still anything's possible.

 

FTP Servers is disabled.  It's not normal for that avahi-daemon to load that?  Maybe some plugin?

 

Damn! happened again while typing this.  I stopped all dockers except pihole, unbound, swag and cloudflare and it stopped. 

Link to comment

Before you loose any more files, ssh into your server or open a terminal in the webui and run this command on all your disks:

find /mnt/disk1/ -type f -exec chattr +i "{}" \;

 Now all your files can not be deleted, renamed or edited. When you figure out what is causing the issue and figured out how to solve it you can run this command on all your disks to "unlock" your files again.

find /mnt/disk1/ -type f -exec chattr -i "{}" \;

 

Too figure out what is causing it start one container at the time and watch the logs. Maybe your radarr/sonarr etc has been hacked. You're probably reverse proxying them with swag right? Look at the nginx access and error logs. Check every container that has access to your files. And change your password for all containers you're accessing through swag asap. 

Link to comment
23 minutes ago, strike said:

Before you loose any more files, ssh into your server or open a terminal in the webui and run this command on all your disks:

find /mnt/disk1/ -type f -exec chattr +i "{}" \;

 Now all your files can not be deleted, renamed or edited. When you figure out what is causing the issue and figured out how to solve it you can run this command on all your disks to "unlock" your files again.

find /mnt/disk1/ -type f -exec chattr -i "{}" \;

 

Too figure out what is causing it start one container at the time and watch the logs. Maybe your radarr/sonarr etc has been hacked. You're probably reverse proxying them with swag right? Look at the nginx access and error logs. Check every container that has access to your files. And change your password for all containers you're accessing through swag asap. 

Thank you strike.

Good point on the container PW's, I didn't think of that.  I'm on the same page with starting one container at a time.  And, I'll definitely search those logs.

 

The odd thing is that my array hasn't noticeably changed from 14.1/32TB (44%).  I figure at least a TB is missing (Probably more).  I'm holding out hope that it's recoverable.

Link to comment

Thank you all for suggestions and making me think in different directions.

 

Hacking:  Files weren't being encrypted, no evidence of theft, no ransomware demand, no-one trying to F* up my system, just data files missing.  I'm not exactly a highly sought target.  So, not likely, yet still possible.  Took some steps to mitigate.

 

Dockers:  Shutdown most of them and it seemed to stop.  Only left a few up that shouldn't be able to access my shares (maybe appdata).  Through my searching, I saw the Plex people swearing that Plex can't delete your files (other than manually in a client).  I removed write access to my media share for the Plex container, and it didn't have access to any other shares.  So, I started that Container last night.  Nothing lost overnight.

 

Laying in bed I thought about trurl and strike's latest comments.  It seems to be a container.  Plex & the 'arrs don't have access to my documents share.  But, Nextcloud does!  While searching this problem I did see one thread where Nextcloud had moved the users files.  I looked around in my Nextcloud share but didn't notice anything there.  I recalled (about a week ago) that I was having trouble opening some files on my desktop.  The Nextcloud client was connected to the admin account., and I changed it to my user account.  This morning I got up and looked at my Nextcloud client on my laptop and see the files with a red x saying synced.  I fired up Krusader and did a global search (why didn't I think of this earlier?) for one of the missing files, and found it here nextcloud\user\files_trashbin\files, along with the others.

 

 

So, now I just need to figure out the why, and best way to move everything back.

 

Thanks again for helping me sort through this.  I caught the 'rona last week, so probably wasn't thinking to clearly to start.  I'll do some more searching from the Nextcloud aspect and come back and update/mark resolved when I get things restored. 

Link to comment
  • 2 weeks later...
  • Solution

So, I never did figure out why Nextcloud moved/deleted so many of my files.  It seems to have moved a bunch to the userprofile\files folder of the Nextcloud share, and deleted a bunch that went to the pruserprofile\files_trashbin\files folder.  I would start a restore from Nextcloud trashbin, and my laptop fans would spin up and the browser would freeze for several minutes.  It would end up restoring 500 to 1000 files before stopping.  It seemed to be using the browsers resources, so I moved to my desktop.  Worked a little better, and after many tries, finally got all those restored.  It took a couple days of going through the userprofile\files files and manually moving them back where they belonged, but finally done.

 

In the end, I don't think it was an unraid issue, but appreciated those who helped me get things restored.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.