NLS Posted November 5, 2020 Share Posted November 5, 2020 I have a seemingly huge issue with my unRAID. Seems the larger the folder in number of files, the slower the file access inside it. And I mean disproportionally. I don't mean that double the files the double the file access. And I am not just referring to listing the contents. I am talking about actual file access, edits etc. A tiny file with a small change that in a small unRAID folder takes non-measurable time after saving, may take... 1 minute (!!!) for the same tiny file if it resides in a huge folder (containing let say 100k files). I have read about case sensitivity and SAMBA issues with it, but I cannot experiment with it (yet) as I need to take down the array (else smbd cannot restart). But I DID try mounting a share using NFS, both case sensitive and non case sensitive... Supposedly NFS doesn't have such issues. Results? Even worse! Actually listing the folder of 100k files in windows explorer didn't even give me any contents after 5 minutes (then I dismounted the share). Both with case sensitive and non. I tried listing the folder in command line (where "dir" is more... erm... direct) and indeed started listing files but after around "c" letter it started having hiccups! With SAMBA at least, listing the folder (in windows) does work, but takes almost 2 minutes. There is definitely an issue. But large folders is common for me (as I handle large emulation collections) and this is major. For example I started a batch edit of simple text files in 7200 folders... normally it should take less than a second per file (a single file per folder). It took more than 2 hours for the first 400 files in folders!!! (so will take almost 2 days for a simple task!) Note that "normal" share access (of smaller folders, file copies etc.) is "ok-ish", not fantastic (40-50MB/s?) but given my unRAID setup it is ok. Any ideas??? Quote Link to comment
Frank1940 Posted November 5, 2020 Share Posted November 5, 2020 (edited) In Windows Explorer, right click on the Share and select Properties and look for the Customize tab. Now look for the "Optimize this folder for:" and set it to 'General items". Click the check box to "Also apply this template to all subfolders". I am not surprised at the sort times for a folder with a 100,000 items in it. I don't know what sort routine was used by the Windows developers but I would bet they never tested it to see how it handled a 100,000 items! (The simple bubble sort routine is very fast for a few items but the time increases by the square as the number of items increases. And this does not even consider the time required to fetch directory data for these 100,000 items across a network.) @JorgeB's question about the filesystem is a very valid one. The old Reiserfs has real significant problems under certain conditions! Edited November 5, 2020 by Frank1940 Quote Link to comment
NLS Posted November 5, 2020 Author Share Posted November 5, 2020 (edited) I think you are missing the point. My problem is not if folder browsing is slow. It is yes, ok, but not a major problem. Again a "demo" of the problem: Editing a pack of let say 400 x 100kb files within a 400 files folder = 150 seconds. Editing a pack of same 400 x 100kb files within a 100000 files folder = 2 hours. (numbers are not real, but they are close to scale) This is the problem. I am using latest non-beta release for unRAID and using the default fs it installs on data disks. Client is Windows 10 x64 with all patches. Edited November 5, 2020 by NLS Quote Link to comment
trurl Posted November 5, 2020 Share Posted November 5, 2020 35 minutes ago, NLS said: using the default fs it installs on data disks You have been a member of this forum since 2007, so it still isn't clear from your answer of "default" what fs you have. If your data disks were created before V6 then the only choice back then was ReiserFS. Quote Link to comment
Frank1940 Posted November 5, 2020 Share Posted November 5, 2020 Look on the MAIN page of the GUI. It plainly tells you what the File System is for each disk. Quote Link to comment
NLS Posted November 5, 2020 Author Share Posted November 5, 2020 Yes sorry... this is not an install since then, is a fresh one (less than a year) data xfs cache btrfs Quote Link to comment
NLS Posted January 7, 2022 Author Share Posted January 7, 2022 (edited) I wake up my own old topic... Because I never fixed that problem, neither did someone actually give me an idea of what could be wrong. Help? No I cannot really break the folders to smaller ones (they are part of a specific collection that needs the files in place). This is crazy. I am trying to modify (through some program) around 100000 small files (much less than a mb each) and looks like it will take DAYS! This is insufferable. Edited January 7, 2022 by NLS Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.