unraidster Posted March 6, 2023 Share Posted March 6, 2023 (edited) Hello Everyone, Apologies in advance for the long post. I have been troubleshooting the upgrade of an Active Directory integrated 6.9.2 environment on and off for a few months now. Each time I attempt to upgrade I end up reverting back to 6.9.2 to get access to shares working again. During my troubleshooting I have found two key differences in the environment after the upgrade (most recently to 6.11.5). I have setup a configuration within a lab environment that closely mimics the main environment to help me revert the configuration and try different configuration changes out. I am able to reproduce the issues I have in the main environment within this lab. The changes I have identified after the 6.9.2 > 6.11.5 upgrade are: Access to a "private" share using windows no longer works. A User on a windows client is presented with a "Windows cannot access…" "Network Name cannot be found" 0x80070043 Error. The Unraid log records the following error: Mar 5 00:48:23 UR-Lab smbd[6151]: [2023/03/05 00:48:23.788115, 0] ../../source3/smbd/smb2_service.c:168(chdir_current_service) Mar 5 00:48:23 UR-Lab smbd[6151]: chdir_current_service: vfs_ChDir(/mnt/user/PrivateShare) failed: Permission denied. Current token: uid=1278739537, gid=1278738945, 4 groups: 1278738945 1278739544 1278739542 1278739546 Access to private shares using the Krusader docker container (default configuration) no longer works. This was also experienced with other docker containers that have mapped host paths. I don't see any errors in the Unraid log when this access attempt fails. I think there might be a couple of root cases here, but so far have failed to identify what those are. I have tried a variety of changes that included re-applying permissions (using the new permissions feature) without being able to resolve both issues above (please see below for more detail). Having read other posts on the forum, I think the following notes may be useful info/context to folks looking at this post: Spoiler The AD User accounts can be looked up after the upgrade to 6.11.5 (using ID {username}) and wbinfo -n {username}. The IDMAP of the environment is still the system's original default HASH based setting. I intend to change this to a RID IDMAP in the future to prevent future issues. But I do not think this is the root cause of the issues I see above due to the lookups working correctly. Once I have a root cause/solution to the issues above I will probably test a change to the IDMAP (and inevitable re-permission) in concert with any other changes that need to be made. Private Share Configuration - How the shares are created: UR SMB setting - Initial AD user is set to an AD admin account. Initial AD Group is set to an AD access group that only an admin account is a member of. Private Share is created. Accessed using the Initial AD User admin account from Windows. The Initial AD Admin account and Initial AD Group ACL entries are set to "This folder, all subfolders and files" Additional share specific RO and RW AD groups are added with respective read-only or Full control permissions. The scope is again set to "This folder, all subfolders and files". The "Replace all child object permission entries with inheritable permission entries from this object" is selected. Click on apply. Remove the Everyone permission. The "Replace all child object permission entries with inheritable permission entries from this object" is selected. Click on apply. This results in a share that only members of the Initial AD Group, RO, or RW groups are able to access. Docker containers with the path mapped are able to essentially perform any action (RWX). The Krusader docker container is configured with: UMASK 000, PUID 99, PGID 100 I have tried to reset the permissions with the "New Permissions" tool. The share did become accessible (Windows and Docker), but that was because an "Everyone" permission was added. I checked the remainder of the permissions, which included those that were added previously as part of the configuration detailed above. On removal of the everyone permission, Windows reports it is unable to remove the permissions from some of the folder (I think those that were previously created by Krusader). The share became inaccessible to windows users, but was accessible to Krusader (presumably because an explicit nobody or users permission had been added to the ACL by the tool). Happy to explore this if recommended to do so. I have included the diagnostics and some outputs from testparm and permissions listings as an attachment to this post. The 6.9.2 prefixed files are from the environment before the upgrade (working state), and those pre-fixed 6.11.5 are those after the upgrade where the issues occur. Public share access functions as expected (accessible for domain joined users in Windows, and host path mapped docker containers) both before and after the upgrade. In summary, I suspect there are a couple of changes to the way things work between 6.9.2 and (6.10.3-6.11.5). I would like to understand if any differences are resulting in my experiences and then work out what to change to get back to the same functionality. Thank you to anyone that endured reading through all that and for any assistance they are able to provide. Please let me know if there is any additional information I can provide or things you would like me to try. Thanks. 6.9.2_PrivateShare_Permissions.txt 6.9.2_testparm.txt 6.9.2_ur-lab-diagnostics-20230305-0024.zip 6.11.5_Permissions.txt 6.11.5_testparm.txt 6.11.5_ur-lab-diagnostics-20230305-0053.zip Edited December 28, 2023 by unraidster 1 1 Quote Link to comment
unraidster Posted March 9, 2023 Author Share Posted March 9, 2023 Hi, To add to the information above, here are some outcomes from other tests. The environment can be rolled back to 6.9.2 and the errors cease. The errors occur on upgrade from 6.9.2 to 6.10.3. This makes me think that the errors I am encountering are introduced with 6.10.3. Any thoughts/help are welcome. Thanks, Quote Link to comment
psychofaktory Posted March 10, 2023 Share Posted March 10, 2023 Thanks for the write up on this topic. It sounds like the exact same error pattern as I had described here: https://forums.unraid.net/bug-reports/stable-releases/6103-intermittent-smb-issues-after-6102-upgrade-r2028/?do=findComment&comment=20618 I also think that this is a different additional problem than the one mainly described here. I sincerely hope that the Unraid team will take the information you have provided and finally fix this bug. The bug has been known for several months now. And although this is a business issue, nothing has been done about it yet. Even more: it was even noted that the priority of the bug fix was set very low here. Especially for customers who use Unraid in a business environment, this is an absolute core function with the highest security relevance. The question of how many Unraid users use this function and would benefit from the bug fix is irrelevant. Therefore, immediate bug fixing should be absolutely in the interest of the Unraid team. 2 Quote Link to comment
unraidster Posted March 10, 2023 Author Share Posted March 10, 2023 Thanks @psychofaktory, appreciate the feedback and glad to see that it is not just isolated to my environment. 1 Quote Link to comment
unraidster Posted March 16, 2023 Author Share Posted March 16, 2023 (edited) Hi, I noticed that 6.12.0-rc1 is out (LINK), and out of curiosity I thought I would give it a go by upgrading the working 6.9.2 build to 6.12.0-rc1. Unfortunately I continue to encounter the same error. Thought it was worth a try. The modified GUI looks nice though Edited March 16, 2023 by unraidster 1 Quote Link to comment
unraidster Posted April 23, 2023 Author Share Posted April 23, 2023 Hi everyone, No solution yet, but managed to find some specific log entries in Samba that may be useful for the diagnosis. I set the samba logging level to 10 and performed the same share access action in 6.9.2 and 6.11.5. 6.9.2 (successful access to share) [2023/04/20 23:02:15.891814, 5, pid=5659, effective(0, 0), real(0, 0)] ../../source3/auth/token_util.c:873(debug_unix_user_token) UNIX token of user 1278739538 Primary group is 1278738945 and contains 4 supplementary groups Group[ 0]: 1278738945 Group[ 1]: 1278739543 Group[ 2]: 1278739547 Group[ 3]: 1278739545 [2023/04/20 23:02:15.891836, 4, pid=5659, effective(1278739538, 1278738945), real(1278739538, 0), class=vfs] ../../source3/smbd/vfs.c:923(vfs_ChDir) vfs_ChDir to /mnt/user/PrivateShare [2023/04/20 23:02:15.892966, 5, pid=5659, effective(1278739538, 1278738945), real(1278739538, 0), class=vfs] ../../source3/smbd/vfs.c:985(vfs_ChDir) vfs_ChDir: vfs_ChDir got /mnt/user/PrivateShare [2023/04/20 23:02:15.893078, 5, pid=5659, effective(1278739538, 1278738945), real(1278739538, 0)] ../../source3/smbd/uid.c:293(print_impersonation_info) print_impersonation_info: Impersonated user: uid=(1278739538,1278739538), gid=(0,1278738945), cwd=[/mnt/user/PrivateShare] [2023/04/20 23:02:15.893083, 5, pid=5659, effective(1278739538, 1278738945), real(1278739538, 0)] ../../lib/dbwrap/dbwrap.c:142(dbwrap_lock_order_lock) dbwrap_lock_order_lock: check lock order 1 for /var/cache/samba/smbXsrv_tcon_global.tdb [2023/04/20 23:02:15.893086, 10, pid=5659, effective(1278739538, 1278738945), real(1278739538, 0)] ../../lib/dbwrap/dbwrap.c:129(debug_lock_order) lock order: 1:/var/cache/samba/smbXsrv_tcon_global.tdb 2:<none> 3:<none> 4:<none> 6.11.5 (access denied) [2023/04/21 22:58:11.191052, 5, pid=6250, effective(0, 0), real(0, 0)] ../../source3/auth/token_util.c:873(debug_unix_user_token) UNIX token of user 1278739538 Primary group is 1278738945 and contains 5 supplementary groups Group[ 0]: 1278739538 Group[ 1]: 1278738945 Group[ 2]: 1278739543 Group[ 3]: 1278739547 Group[ 4]: 1278739545 [2023/04/21 22:58:11.191074, 4, pid=6250, effective(1278739538, 1278738945), real(1278739538, 0), class=vfs] ../../source3/smbd/vfs.c:938(vfs_ChDir) vfs_ChDir to /mnt/user/PrivateShare [2023/04/21 22:58:11.191533, 0, pid=6250, effective(1278739538, 1278738945), real(1278739538, 0)] ../../source3/smbd/smb2_service.c:168(chdir_current_service) chdir_current_service: vfs_ChDir(/mnt/user/PrivateShare) failed: Permission denied. Current token: uid=1278739538, gid=1278738945, 5 groups: 1278739538 1278738945 1278739543 1278739547 1278739545 [2023/04/21 22:58:11.191616, 3, pid=6250, effective(1278739538, 1278738945), real(1278739538, 0), class=smb2] ../../source3/smbd/smb2_server.c:3955(smbd_smb2_request_error_ex) smbd_smb2_request_error_ex: smbd_smb2_request_error_ex: idx[1] status[NT_STATUS_ACCESS_DENIED] || at ../../source3/smbd/smb2_server.c:3247 [2023/04/21 22:58:11.191630, 10, pid=6250, effective(1278739538, 1278738945), real(1278739538, 0), class=smb2] ../../source3/smbd/smb2_server.c:3841(smbd_smb2_request_done_ex) smbd_smb2_request_done_ex: mid [6] idx[1] status[NT_STATUS_ACCESS_DENIED] body[8] dyn[yes:1] at ../../source3/smbd/smb2_server.c:4005 [2023/04/21 22:58:11.191637, 10, pid=6250, effective(1278739538, 1278738945), real(1278739538, 0), class=smb2_credits] ../../source3/smbd/smb2_server.c:969(smb2_set_operation_credit) smb2_set_operation_credit: smb2_set_operation_credit: requested 1, charge 1, granted 1, current possible/max 8160/8192, total granted/max/low/range 33/8192/7/33 [2023/04/21 22:58:11.194902, 10, pid=6250, effective(1278739538, 1278738945), real(1278739538, 0), class=smb2] ../../source3/smbd/smb2_server.c:4995(smbd_smb2_io_handler) smbd_smb2_request idx[1] of 5 vectors I have attached full copies of the samba log in case the additional debug information is useful to anyone, Thanks, Samba-6.9.2-Level10.log samba-6.11.5-Level10.log 1 Quote Link to comment
unraidster Posted December 28, 2023 Author Share Posted December 28, 2023 HI Everyone, I thought I would take another look at this (as I want to upgrade from 6.9.2). Tried 6.12.6 and can confirm the issue persists when upgrading from 6.9.2. Will be taking another look to the lab to see if I can find a workaround but no successes yet. Has anyone else with this issue been able to move on from 6.9.2? if so, are you able to explain what you did? TIA! 1 Quote Link to comment
unraidster Posted February 1 Author Share Posted February 1 Hi, I sent some queries into the Samba mailing list and received some feedback from the Samba team. Unfortunately I haven’t been able to resolve the issues I have highlighted in this thread but still have some troubleshooting to work through. In the meantime the thread may be useful to others working on the same problem. https://lists.samba.org/archive/samba/2024-January/247763.html Unraidster 2 1 Quote Link to comment
psychofaktory Posted February 2 Share Posted February 2 Wow, many thanks for your tireless efforts. I'm stunned that so little attention has apparently been paid to the Samba configuration by the Unraid team for some time now. Hopefully your meticulous analysis will lead to better attention being paid to this issue in the near future and thus make the whole Samba integration under Unraid more reliable. 2 1 Quote Link to comment
unraidster Posted February 12 Author Share Posted February 12 Hello Everyone, I have separated the latest update into two posts, this one is some more detail on the error the next post is looking at another troubleshooting approach. I reproduced the error in 6.12.6 using the smbclient on the Unraid server (AD user credential used). I found this cuts down on the amount of "noise" in the logs vs using a Windows client. I also setup strace to record the calls made during the connection. Not being familiar with strace, I have managed to configure it to capture any processes spawned by the main smbd process and save the output to a file. I found that strace was not installed by default so used the following processes for 6.9.2 and 6.12.6 to install. Appreciate any feedback on the approach to install strace (Note: I was installing on a lab system that I tear down frequently so was not too concerned about where files were being saved etc.) Process used to install strace on 6.9.2 (run from CLI) Spoiler cd /mnt/user/isos curl https://slackware.uk/slackware/slackware64-15.0/slackware64/l/elfutils-0.186-x86_64-1.txz -o elfutils-0.186-x86_64-1.txz curl https://slackware.uk/slackware/slackware64-15.0/slackware64/l/libunwind-1.6.2-x86_64-1.txz -o libunwind-1.6.2-x86_64-1.txz curl https://slackware.uk/slackware/slackware64-15.0/slackware64/d/strace-5.16-x86_64-1.txz -o strace-5.16-x86_64-1.txz curl https://slackware.uk/slackware/slackware64-15.0/slackware64/l/glibc-2.33-x86_64-5.txz -o glibc-2.33-x86_64-5.txz installpkg elfutils-0.186-x86_64-1.txz installpkg libunwind-1.6.2-x86_64-1.txz installpkg strace-5.16-x86_64-1.txz #This seemed to be needed to allow strace to execute in 6.9.2 installpkg glibc-2.33-x86_64-5.txz Process used to install strace on 6.12.6 (run from CLI) Spoiler cd /mnt/user/isos curl https://slackware.uk/slackware/slackware64-15.0/slackware64/l/elfutils-0.186-x86_64-1.txz -o elfutils-0.186-x86_64-1.txz curl https://slackware.uk/slackware/slackware64-15.0/slackware64/l/libunwind-1.6.2-x86_64-1.txz -o libunwind-1.6.2-x86_64-1.txz curl https://slackware.uk/slackware/slackware64-15.0/slackware64/d/strace-5.16-x86_64-1.txz -o strace-5.16-x86_64-1.txz installpkg elfutils-0.186-x86_64-1.txz installpkg libunwind-1.6.2-x86_64-1.txz installpkg strace-5.16-x86_64-1.txz I have included a snippet of the calls made around the error in 6.12.6 below. I have also included what I think are the similar calls made in 6.9.2 (where the SMB connection works). 6.9.2 - strace excerpt Spoiler 6920 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3648, ...}) = 0 6920 getpid() = 6920 6920 getgid() = 0 6920 getuid() = 1278739538 6920 getegid() = 1278738945 6920 geteuid() = 1278739538 6920 geteuid() = 1278739538 6920 write(17, "[2024/02/10 18:17:24.749347, 4,"..., 150) = 150 6920 geteuid() = 1278739538 6920 write(17, " vfs_ChDir to /mnt/user/Private"..., 38) = 38 6920 chdir("/mnt/user/PrivateShare") = 0 6920 stat(".", {st_mode=S_IFDIR|0770, st_size=152, ...}) = 0 6920 getcwd("/mnt/user/PrivateShare", 4096) = 23 6920 stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=3648, ...}) = 0 6920 getpid() = 6920 6920 getgid() = 0 6920 getuid() = 1278739538 6920 getegid() = 1278738945 6920 geteuid() = 1278739538 6920 geteuid() = 1278739538 6920 write(17, "[2024/02/10 18:17:24.756082, 5,"..., 150) = 150 6920 geteuid() = 1278739538 6920 write(17, " vfs_ChDir: vfs_ChDir got /mnt/"..., 50) = 50 6920 stat(".", {st_mode=S_IFDIR|0770, st_size=152, ...}) = 0 6920 stat("/mnt/user/PrivateShare", {st_mode=S_IFDIR|0770, st_size=152, ...}) = 0 6.12.6 - strace excerpt Spoiler 11254 newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0644, st_size=3664, ...}, 0) = 0 11254 getpid() = 11254 11254 getgid() = 0 11254 getuid() = 1278739538 11254 getegid() = 1278738945 11254 geteuid() = 1278739538 11254 geteuid() = 1278739538 11254 writev(4, [{iov_base="[2024/02/10 18:30:03.772118, 4,"..., iov_len=151}, {iov_base=" vfs_ChDir to /mnt/user/Private"..., iov_len=38}], 2) = 189 11254 chdir("/mnt/user/PrivateShare") = -1 EACCES (Permission denied) 11254 newfstatat(AT_FDCWD, "/etc/localtime", {st_mode=S_IFREG|0644, st_size=3664, ...}, 0) = 0 11254 getpid() = 11254 11254 getgid() = 0 11254 getuid() = 1278739538 11254 getegid() = 1278738945 11254 geteuid() = 1278739538 11254 geteuid() = 1278739538 11254 writev(4, [{iov_base="[2024/02/10 18:30:03.777285, 0,"..., iov_len=161}, {iov_base=" chdir_current_service: vfs_ChD"..., iov_len=220}], 2) = 381 11254 getpid() = 11254 11254 sendto(6, "<27>Feb 10 18:30:03 smbd[11254]:"..., 194, MSG_NOSIGNAL, NULL, 0) = 194 11254 getpid() = 11254 Note: The "11254 chdir("/mnt/user/PrivateShare") = -1 EACCES (Permission denied)" line in the 6.12.6 output and the similar "6920 chdir("/mnt/user/PrivateShare") = 0" line in the 6.9.2 output. I tried to interpret the two outputs, but only really established that the EACCES error is thrown when "Search permission is denied for one of the components of path". [REF] Directory Permissions/ACL Spoiler Directory Permissions ========================= / drwxr-xr-x 20 root root /mnt/ drwxr-xr-x 9 root root /mnt/user/ drwxrwxrwx 1 ur_admin ur-lab_access /mnt/user/PrivateShare/ drwxrwx---+ 1 ur_admin ur-lab_access ACL root@UR-Lab:~# getfacl /mnt/user/PrivateShare getfacl: Removing leading '/' from absolute path names # file: mnt/user/PrivateShare # owner: ur_admin # group: ur-lab_access user::rwx user:ur_admin:rwx group::rwx group:ur-lab_access:rwx group:ur-lab-privateshare-ro:r-x group:ur-lab-privateshare-rw:rwx mask::rwx other::--- default:user::rwx default:user:ur_admin:rwx default:group::--- default:group:ur-lab_access:rwx default:group:ur-lab-privateshare-ro:r-x default:group:ur-lab-privateshare-rw:rwx default:mask::rwx default:other::--- ========================= Based on: 1) the ACL previously working in 6.9.2 and 2) the folder permissions (for /, /mnt, and /mnt/user) and 3) the ACL (for /mnt/user/PrivateShare) allowing a minimum of execute access to the test AD users (members of the -ro and -rw groups) I suspect something is causing the ACL to get evaluated incorrectly. Unraidser. 2 Quote Link to comment
unraidster Posted February 12 Author Share Posted February 12 (edited) Hi, As part of my troubleshooting I tried the following test and found an interesting result. Test: Attempt to access the share contents using a disk share (rather than a user share). Note: This is a lab system, so not concerned about creating a disk share for a disk that also has user shares configured. The system only contains one disk. Method: Using a working, configured 6.9.2 installation. Upgrade to 6.12.6. Use the same test user accounts used to validate 6.9.2 to access 6.12.6 Confirmed error is thrown (see previous posts) accessing the user share. Setup a disk share (disk1). Browse to the disk share and evaluate the folder that represents the "PrivateShare" user share. Outcomes: The test users are able to access the PrivateShare folder within the disk share but are still unable to access the PrivateShare user share. Can view the ACL and can see valid ACLs (as we saw in 6.9.2) when browsing the PrivateShare folder within the disk share. I don't have a detailed knowledge of the architecture of Unraid, but I am guessing that there is a difference in the way a SMB call to the user share (/mnt/user/PrivateShare in this example) is made vs the same folder in a disk share (/mnt/disk1/PrivateShare). From what I have read so far, I believe Fuse may be one of those differences. Any knowledge/advise on the above would be appreciated :). Regards, Unraidster Edited February 13 by unraidster To clarify the method and outcomes. 3 Quote Link to comment
WormTX Posted February 13 Share Posted February 13 Thank you @unraidster for all the work you've been putting into this! I would love to see some type of response from the Unraid team that they are at least aware of this and can use the information you've gathered to help troubleshoot this issue. 1 Quote Link to comment
PatMatt Posted February 13 Share Posted February 13 1 hour ago, WormTX said: Thank you @unraidster for all the work you've been putting into this! I would love to see some type of response from the Unraid team that they are at least aware of this and can use the information you've gathered to help troubleshoot this issue. Yes, I've been watching this closely as I had been planning on using unRaid as an integral part of AD infrastructures for cheap data space. But until this issue is resolved, the best wrapped I have for those environments is to use a NextCloud instance to actually get the data there. It'll work, but a solid "yes" or "no" as to whether this will be fixed will let me put the idea to rest or continue to monitor. It can't be said enough, but @unraidster has earned my respect for his dedication to this issue, wow. Thank you for your hard work! 1 Quote Link to comment
unraidster Posted February 13 Author Share Posted February 13 Hi, thank you both. In looking into Fuse from my last post I found a thread in this forum reporting a bug related to 6.10.0 rc5 (Link). The timing correlates with when ACLs stopped working for me with AD users and may explain that difference in behaviour between user share access and disk share access. I have provided some additional information in that post and am waiting to see if someone with some additional knowledge on Unraid and Fuse is able to advise/reply to that post. Unraidster. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.