Weird permissions issue

[k0d3g3ar]

I have an Unraid server running the latest software version, with about 80TB of storage.  I have had this for a very long time, but recently started to notice issues with permissions.  It seems that if I log into the server with SSH and look at the file & directory permissions in /mnt/user/* and /mnt/disk?/* they show as owned by "nobody" with the group "users".  All content has been copied to this server via NFS over the years.  The perms are showing routinely as -rw-rw-r--  but this is not consistent.  Some directories are 777 as well.

 

I've tried to chmod 777 on disks and it works on some, but on others (even though I'm logged in as root) give me permission errors when I attempt to change this.  The same is true of changing ownerships.

 

Is there a "best practice" way to set perms up so that I can delete content I no longer need?  I tried to do this via the very same NFS mounts that created the content and I'm getting the same errors there as well.

 

Thanks in advance for any suggestions.

[JorgeB]

Try running the new permissions tool.

[k0d3g3ar]

Thank you.  I was not aware that this had been addressed.  I found the tool and have started to run it on the raw disks first.  Any idea how long this takes on 80TB of storage, over 8 drives?

[JorgeB]

Should be fairly quick.

[k0d3g3ar]

Unfortunately that didn't work.  Ran for about 9 hours and no difference.  Seems to be just staying on drive1 in the list and going over and over on it.  Then I stopped it to do the shares and it did the same thing.  Any ideas?  The content appears to be fine and readable.  Just the permissions are messed up.

[JorgeB]

Post the output from:

ls -l /mnt/disk1

 

[k0d3g3ar]

Here you go.

[JorgeB]

Those permissions are correct, except backup_windows, IIRC the + means there are additional ACL permissions, but I don't think Unraid supports that, though it may not be a problem.

[k0d3g3ar]

I think I might have stumbled on the issue.  One of the folders has no content in it (backup_offsite2).  I tried to delete it with the share, and got this:

root@Tower:/mnt/user# rm -Rf /mnt/user/backup_offsite2
rm: cannot remove '/mnt/user/backup_offsite2': Read-only file system

This was the problem occurring repetitively.  However if I go into one of the disks and try and delete it, it worked:

root@Tower:/mnt/user# rm -Rf /mnt/disk1/backup_offsite2
root@Tower:/mnt/user#

So I went through each of the 7 disks in the array, and everything let me delete the folder, except disk 3:

root@Tower:/mnt/user# rm -Rf /mnt/disk3/backup_offsite2
rm: cannot remove '/mnt/disk3/backup_offsite2': Read-only file system

Although there are no errors reported in the array for disk3, clearly this is the issue.  I'm stopping the array and putting the system into maintenance mode, so I can disable disk3 and then rebuild from parity.  I'm away from the location where the server is, so I'm trying to do this remotely but I think at least I have the issue isolated.  If it is just a matter of a disk replacement, and the entire array can be used again, I'll be a very happy camper.

 

Thanks for your help in helping me dig deeper on this.  Not sure I'm totally solved yet, but I think I'm on the right track now.
 

 

 

[k0d3g3ar]

Well I thought I was solved, but no so fast....

 

I removed the disk from the array under maintenance mode, then started the array.  It comes up in degraded state which is to be expected.  But since the parity disk contained the content from before with /mnt/disk3 it still shows up and attempts to remove the content from that disk continues to fail as "Read-only file system".  

 

Is it that the parity disk copied the corrupted file system from disk3?  If so, is there any chance of recovery now?

 

[itimpi]

8 minutes ago, k0d3g3ar said:

Is it that the parity disk copied the corrupted file system from disk3?  If so, is there any chance of recovery now

I think you misunderstand how parity works?   Parity has no concept of data or of file systems!  Parity works at the raw sector level and just knows how to reconstruct any particular sector on a failed/missing drive (in conjunction with all the good drives).   If that sector contained corrupt data parity will quite happily emulate that corrupt data.

[JorgeB]

Assuming the disk is xfs you should check filesystem, if it's not, post the diagnostics.

[Frank1940]

5 hours ago, k0d3g3ar said:

Here you go.

The ACL parameter has caused problem in the past for SMB users.  Here is a post from someone who found a way to fix it:

 

       https://forums.unraid.net/topic/116389-access-denied-printer-unraid-share/#comment-1059803

 

You will want to review the switches for that command that the poster used to make sure what each one does.  Google  "manual setfacl"    (Many times the    -R    switch will do the action recursive down the directory tree from the starting point.)   

 

[k0d3g3ar]

I wanted to follow up on this, because I solved it.  The issue was that the Unraid server exhibiting the problems was an upgraded server from a long time back (probably v5 or so) and the disks in there were formatted with ReiserFS (now deprecated).  Well 75% of them were - two were XFS and 6 were ReiserFS.  On one of the disks when I ran diagnostics on it, it came back with a small file integrity issue during the ReiserFS check run, and was fixable.  When I then ran the auto fix on it, it corrected it and then all of a sudden I could rsync to the server without any file permission errors anymore.  Of course that has meant that about 15TB of content is being replaced now from our production server, but that's ok - at least we have the backup running correctly now.

 

Thank you for anyone who has helped (in particular @JorgeB) who helped me get to the point where I could identify the issue and fix it.