Jump to content

golli53

Members
  • Content Count

    67
  • Joined

Community Reputation

7 Neutral

About golli53

  • Rank
    Advanced Member

Recent Profile Visitors

The recent visitors block is disabled and is not being shown to other users.

  1. This fixes the stat issue for very large folders - thanks for your hard work! Unfortunately, SMB is still quite slow - I think the listdir calls are still ~2x slower than with prior versions, despite Hard Links disabled. With the tweaks, my scripts now run instead of stalling, though are still noticebly slower. I'll try to reproduce and compare when I get a chance to try 6.8.2 again. Regardless, thanks for your efforts here.
  2. All on cache (which is 2xSSD RAID1 btrfs for me). Same issue occurs with folder that's on array though (spread across disks). Seems to be SMB issue because I don't see extra lag when calling stat from unRAID shell or through NFS from Linux client.
  3. @limetech First of all, thank you for taking the time to dig into this. From my much more limited testing, the issue seems to be a painful one to track down. I upgraded yesterday and while this tweak solves listdir times, stat times for missing files in large directories is still bugged (observation 2 in the below post): For convenience, I reproduced in Linux and wrote this simple script in bash: # unraid cd /mnt/user/myshare mkdir testdir cd testdir touch dummy{000000..200000} # client sudo mkdir /myshare sudo mount -t cifs -o username=guest //192.168.1.100/myshare /myshare while true; do start=$SECONDS; stat /myshare/testdir/does_not_exist > /dev/null 2>&1 ; end=$SECONDS; echo "$((end-start)) "; done On 6.8.x, each call takes 7-8s (vs 0-1s on previous versions), regardless of hard link support. The time complexity is nonlinear with the number of files (calls go to 15s if I increase the number of files by 50% to 300k).
  4. Got this error randomly as well for the first time after accessing SMB. Wonder if anyone found out whether this is a symptom of a serious problem?
  5. I don't have a test server for unRAID so can only try out these suggestions on a weekend when I don't need my production environment up and running. For now, I'm going back to 6.6.7 to avoid slow SMB and the concurrent disk problem in 6.7.2. Also, I think there was something else going on in addition to the 3-4x slower directory listings. Some of my apps would lag for 20 minutes compared to 5 seconds, so I think there were additional SMB performance regressions. I detailed some other slow behavior in the Prerelease thread, but those were just the regressions I happened to notice from debugging the code in a couple of my apps one weekend, so I may have missed others.
  6. I guess it's the definition of near instantaneously. In my testing over many thousands of calls, I was averaging 2.5s vs 0.7s (for 6.7.2.) for 3k items. When 2 programs are accessing SMB simultaenously, that becomes 5s vs. 1.4s. For 10 programs, 25s vs 7s. I think it's common for services to access SMB shares on a server simultaenously.
  7. Will Wireguard eventually be integrated into unRAID? I think Wireguard is very cool and definitely appreciate the work on new functionalities, but am curious about the design decision here. It's unrelated to storage (and I would personally hesitate to host my VPN server on my storage server in case I need to reboot my storage remotely). Wireguard can also be run as a docker container if the kernel supports (here's a popular one: https://github.com/cmulk/wireguard-docker). The Plugin/Community App/Docker route may be cleaner and less demanding on developer resources.
  8. Security measures like fail2ban are much easier to set up on Ubuntu than unRAID (to prevent brute force attacks into password-protected services). Otherwise, there's no difference as long as you only expose the port for the specific service.
  9. There are two issues and the sample code has a section for each (preceded by a comment header) Part 1 of the code is getting listings. That seems to be slower on 6.8.0 for all directories and is noticeable to a human on a single call without concurrency starting with a couple thousand files. Part 2 of the sample code is only calling stat. I can only reproduce this issue for very large directories, but maybe that's because it requires large directories to produce a measurable difference.
  10. I never call a directory listing in that directory. I only open specific files by naming convention. So, adding subdirs would make things more inefficient because I would have to check for a file in each subdir. My current setup works very fast using a normal Samba server eg 6.7.2 or Ubuntu. The first issue is a problem for much smaller directories (a few thousand).
  11. 😀Yes it's a very big one for automatically archiving json files. There's no natural categorization for assigning subdirectories, so wouldn't improve the speed for my app.
  12. smb672.txtsmb680.txt Attaching the testparm outputs. I also tried some debugging after rolling back again to the previous version. I see two differences in behavior - Each os.listdir call is ~3-4x slower on average on v6.8 vs v6.72 (2.5s vs 0.7s for a 6k file directory) - When calling os.stat on a single NONEXISTENT file in a 250k file directory on 6.8 (should be microseconds per call), every 100 or so calls, it hangs for ~5s Concurrent calls are simply additive in terms of execution time and concurrency itself doesn't seem to be the problem, but it makes it more apparent. Note that under the hood, Python is just using Windows native protocol for listing / accessing stats on these files, but using Python just makes it easier to debug many requests. My code for reproducing this below: from datetime import datetime as dt import os # observation 1 while True: start = dt.now() os.listdir('//192.168.1.100/share/path') print((dt.now() - start).total_seconds()) # observation 2 while True: start = dt.now() try: os.stat('//192.168.1.100/share/bigpath/nonexistent.txt') except: pass print((dt.now() - start).total_seconds())
  13. Thanks- I will try as soon as I get a chance (running production environment). I'm essentially calling code like below on several (~5) network folders with 5k subdiectories each from a Win10 client. That slows SMB directory listing down to a halt for that client, including normal browing using Windows file explorer. def recursive_ls(path): files = os.listdir(path) for f in files: subpath = os.path.join(path, f) if os.path.isdir(subpath): recursive_ls(subpath) [edit] Each subdirectory only has a couple files in it, so the number of files may not be an issue, but rather just concurrent ls requests, so calling ls on a loop from multiple threads on 1 client may do the same thing (may be easier to setup using a shell script)
  14. Gotcha. Will replace the cable just in case