Unable to delete a folder on the system

[dmk0]

It's me again with another topic that really wrecks my brain.

 

I created a - let us say - faulty directory in my first round of setting up my media server. Now I have a directory, I don't need and want anymore:

├── data
│   ├── Download
│   ├── downloads
│   │   ├── radarr
│   │   ├── sonarr
│   └── media
│       ├── anime
│       │   ├── movies
│       │   ├── renamed anime
│       │   └── series
│       ├── movies
│       └── tv

This is my share setup for /data. The folder I don't need anymore is the /Download folder. I've already spent a considerable amount of time to debug this, but to no avail.

 

When trying to

rm -rf /mnt/user/data/Download

I get the error: rm: cannot remove '/mnt/user/data/Download': Device or resource busy

So, I do what everyone would do and do a "ps" in the directory with the result:

  PID TTY          TIME CMD
17207 pts/0    00:00:00 bash
18864 pts/0    00:00:00 ps

After doing a ps -ef I get the following results for the first PID:

UID        PID  PPID  C STIME TTY          TIME CMD
root     17063     1  0 12:20 ?        00:00:00 /usr/bin/ttyd -d0 -t disableLeaveAlert true -t theme {'background':'black'} -t fontSize 15 -t font
root     17103     2  0 12:23 ?        00:00:00 [metaslab_group_]
root     17204     2  0 12:23 ?        00:00:00 [metaslab_group_]
root     17205     2  0 12:23 ?        00:00:00 [metaslab_group_]
root     17206     2  0 12:23 ?        00:00:00 [metaslab_group_]
root     17207 17063  0 12:20 pts/0    00:00:00 bash --login

I can't kill PID 17207. After consulting a friend of mine, he said it could be a startup script for the server as the CMD is "bash --login" so I restarted the server (just for good measure) and left the disks unmounted. I searched far and wide for the /Download path - as my friend suggested - but couldn't find anything of substance - esp. no shell scripts that would do anything with the path.

 

Any idea what I have done and how I can fix this? I just want to delete that directory, as I don't need it anymore. I've already tried everything with the permissions, to no avail.

 

Currently running on Version: 6.12.0-rc4 with ZFS as file system.

ymir-diagnostics-20230510-1214.zip

 

EDIT: I found another command:

root@Ymir:/mnt/user/data/Download# df -h /mnt/user/data/Download
Filesystem      Size  Used Avail Use% Mounted on
shfs             51T  640K   51T   1% /mnt/user

When I try to unmount the Download directory with "unmount /mnt/user/data/Download" it just says bash: unmount: command not found.

Edited May 10, 2023 by dmk0

[Frank1940]

Looking at your problem, I see two directories in 'Download'---  'radarr'  and 'sonarr'.  I would  start by trying to delete them.  I would suspect that one (or. perhaps, both) will be the cause of the problem. 

 

It could be that one of those Dockers (radarr and sonarr) has something open...   Stop the Dockers- either all of them or one at a time.  (Your choice)

 

Finally brute force the problem, keeping moving down the tree until you are working only on files.  You will eventually find the culprit(s).

 

EDIT:  Before you run a rm -rf command, be sure to test the path first using   ls -al  and double check that you have the correct path.  ( rm is ancient Unix command written by the original UNIX Gurus that does exactly what it is told without question!!!!)  You have been warned

Edited May 10, 2023 by Frank1940

[dmk0]

Thanks for the reply

Quote

Looking at your problem, I see two directories in 'Download'---  'radarr'  and 'sonarr'.

radarr and sonarr are under /data/downloads (downloads with lowercase d is my new directory that I created because the uppercase Download gave me some issues). /data/Download (uppercase D) is already completely empty and nothing is in there anymore (either been moved or deleted).

Quote

Stop the Dockers- either all of them or one at a time.  (Your choice)

I tried it with this suggestion, yesterday when I also rebooted the whole server and looked at some start-up scripts. Docker was not running and I also doublechecked in every docker container config that nothing is using the old path.

 

I am just confused as hell why I can't seem to delete that directory, i.e., what process is using this (and from where).

[Frank1940]

11 minutes ago, dmk0 said:

I created because the uppercase Download gave me some issues

Interesting.  What is the output of

ls -al /mnt/user/data

and

ls -al /mnt/user/data/Download

 

Directory permissions are funny animals and I always have problems when I need to figure why 'funny things' are happening in finding good documentation on what each parameter (r.w.x) really does...

 

You can blank out all the directory names except for Download as we aren't really that interested in them if you want.

[dmk0]

root@Ymir:~# ls -al /mnt/user/data
total 86766
drwxrwxrwx 1 nobody users        5 May 10 11:42 ./
drwxrwxrwx 1 nobody users        5 May  9 09:33 ../
drwxrwxrwx 1 nobody users        2 May  8 17:29 Download/
-rwxrw-rw- 1 dmk0   users 88660562 May 10 13:33 Everything.db*
drwxrwxrwx 1 nobody users        6 May  9 22:46 downloads/
drwxrwxrwx 1 nobody users        5 May  4 16:42 media/

And

root@Ymir:~# ls -al /mnt/user/data/Download
total 26
drwxrwxrwx 1 nobody users 2 May  8 17:29 ./
drwxrwxrwx 1 nobody users 5 May 10 11:42 ../

 

I don't know the parameters as well, but as far as I can see, /data/Download doesn't have any different permissions than the other ones.

 

Do you have an idea how the bash --login is part of this whole thing? That is the command that I discovered to be behind the PID keeping /data/Download busy thus making it undeletable.

 

EDIT: Everything.db is an indexing file created by my Search Everything application on my windows PC to index the drives and making file discovery easier.

Edited May 10, 2023 by dmk0
Clarification

[Frank1940]

Download is empty and it has the proper permissions.  The permissions on the path down to Download look correct.  So something else is going on.   (Which you already knew!)

 

I assume that you have Googled    bash --login    I did and didn't find anything in a quick search/review that looked like as answer.

 

I know just enough about Linux to make me really dangerous to be offering any suggestions when I start to move out of comfort zone of knowledge.  So I am going to ping a couple of fellows who are probably more knowledgeable than I am---  @trurl  and @itimpi .

 

 

[dmk0]

Yeah it is kind of strange with the whole bash --login command. Thanks for pinging the fellows and thanks for offering your help Much appreciated

[Frank1940]

DUH!  Try booting to the Safe Mode (a boot time option) and see if you can delete the file from there.  You are not making any connections via NFS are you?

[dmk0]

@Frank1940sorry for the late reply. I booted into Safe Mode and tried deleting it in the console, to no avail. Still the same "Device or resource busy" message. I disconnected the drive connection from windows and tried again, still the same result.

[dmk0]

After many moons: still the same issue with this. Can @trurlor @itimpi maybe help with this?

[Frank1940]

On 5/10/2023 at 11:42 AM, dmk0 said:

root@Ymir:~# ls -al /mnt/user/data
total 86766
drwxrwxrwx 1 nobody users        5 May 10 11:42 ./
drwxrwxrwx 1 nobody users        5 May  9 09:33 ../
drwxrwxrwx 1 nobody users        2 May  8 17:29 Download/
-rwxrw-rw- 1 dmk0   users 88660562 May 10 13:33 Everything.db*
drwxrwxrwx 1 nobody users        6 May  9 22:46 downloads/
drwxrwxrwx 1 nobody users        5 May  4 16:42 media/

 

Have another look at the output of this command above.   If the date on the   Download    directory is more recent than the May 8th date, something is using that directory.  You will have to figure what Docker app, VM or plugin it is.  It could be a remote connection by one of the Share users via a SMB or NFS connection as well. 

[dmk0]

On 11/26/2023 at 9:46 PM, Frank1940 said:

 

Have another look at the output of this command above.   If the date on the   Download    directory is more recent than the May 8th date, something is using that directory.  You will have to figure what Docker app, VM or plugin it is.  It could be a remote connection by one of the Share users via a SMB or NFS connection as well. 

It is still May 8th. This is curious...

[dmk0]

@Frank1940 As you've been with me through the whole journey: I have finally found the culprit and the solution.

 

After doing a quick google search (again) I stumbled upon this post

This caught my eye but led me in a strange direction. I am not great with managing zfs and I already stumbled because of step 1. Thus, I consulted the discord channel where I got into a dialogue with @jcofer555 (thanks again) reiterating my problem. And at some point mentioning that maybe it could be zfs. I also finally found out what the bash process is: it's the terminal! So, killing it, just closes the terminal

 

Now, experimenting with zfs commands I did the following:

root@Ymir:/mnt/user/data# zfs list -r
NAME                  USED  AVAIL     REFER  MOUNTPOINT
disk1                9.07T  7.16T      112K  /mnt/disk1
disk1/.Trash-99        96K  7.16T       96K  /mnt/disk1/.Trash-99
disk1/data           9.07T  7.16T     9.07T  /mnt/disk1/data
disk1/data/Download    96K  7.16T       96K  /mnt/disk1/data/Download
disk1/isos             96K  7.16T       96K  /mnt/disk1/isos
disk1/other          2.18G  7.16T     2.18G  /mnt/disk1/other
disk2                9.06T  7.17T       96K  /mnt/disk2
disk2/data           8.75T  7.17T     8.75T  /mnt/disk2/data
disk2/data/Download   120K  7.17T      120K  /mnt/disk2/data/Download
disk2/other           317G  7.17T      317G  /mnt/disk2/other
disk3                8.86T  7.37T      104K  /mnt/disk3
disk3/data           8.67T  7.37T     8.67T  /mnt/disk3/data
disk3/other           196G  7.37T      196G  /mnt/disk3/other
disk4                1.75T  14.5T      112K  /mnt/disk4
disk4/data           1.72T  14.5T     1.72T  /mnt/disk4/data
disk4/other          24.5G  14.5T     24.5G  /mnt/disk4/other
disk5                2.66M  5.33T       96K  /mnt/disk5

 

Combining this with the above post, I just tried to do a

zfs destroy -r disk1/data/Download

Which removed it from the list once. I did it again for disk 2. Now the list was empty. I went into the share and the folder was finally GONE.

[bmartino1]

I had a ghosting issues once with docker 1 to 1.

 

I had to run the unmount command

 

umount /folder/path to release init hold/use on the folder. Then I could delete it.