The traversal of the file system in Unraid is very slow.

[competent-bailout3425]

The speed of traversing the /mnt/user file system is very slow. For instance, in my array, I have a top-level directory ASD, which contains a large number of nested small files. The entire ASD directory is in the cache disk. When I use time find /mnt/cache/ASD -type f, it takes roughly 8 seconds. However, time find /mnt/user/ASD -type f takes 168 seconds, consuming 20 times more time, even though there is no ASD directory under /mnt/user0 at all.

 

I believe that the algorithm in Unraid should cache whether each directory exists on each disk, especially for directories that have been accessed recently. This should at least be an option, even though it might bring about some issues. For instance, writes through /mnt/disk1 might require a delay to be observable through /mnt/user0 and /mnt/user. However, this is acceptable in many scenarios because it can significantly speed up the opening of files and the traversal of the user file system.

 

If such an operation is too aggressive, at least for the top-level directories, acceleration should be applied automatically according to the settings in the share tab. Typically, I would establish a special directory to store some data that would only be stored in the cache (i.e., only primary storage is set in the share). For example, the compilation directory of OpenWRT, which would contain a large number of files stored in a multi-level directory structure. If every level of directory and file access requires check the existence on each array disk, it would waste a lot of unnecessary time.

Edited December 3, 2023 by competent-bailout3425

[trurl]

Dynamix Cache Dirs plugin

 

 

[ChatNoir]

And for shares only on a pool, the Exclusif access share option could be helpful for file access time.

Also, this has nothing to do with the forum's config or whatever.

 

Let's move that to general

[competent-bailout3425]

12 hours ago, trurl said:

Dynamix Cache Dirs plugin

 

 

This plugin does indeed improve access speed, but it's still quite slow. The command time find /mnt/user/fast -type f|wc -l takes 106s, while time find /mnt/cache/fast -type f|wc -l takes 6.5s. Compared to the original speed（160）, this improve a lot, but it's still significantly slow compare to the original 7s.

[competent-bailout3425]

Optimizing access through plugins, in my opinion, won't yield very good results. My memory is already quite large, and I'm using ZFS, so I basically don't trigger IO. I believe that the algorithm should internally add a cache to store which files and directories have been recently accessed and on which hard drives they reside. Instead of trying to test whether the directory exists from all hard disks every time, this would greatly improve the access performance of a multi-disk file system.

[competent-bailout3425]

12 hours ago, ChatNoir said:

And for shares only on a pool, the Exclusif access share option could be helpful for file access time.

Also, this has nothing to do with the forum's config or whatever.

 

Let's move that to general

what do you mean "Exclusif access"?my share option is already pool only.

[ChatNoir]

4 hours ago, competent-bailout3425 said:

what do you mean "Exclusif access"?

https://docs.unraid.net/unraid-os/release-notes/6.12.0/#exclusive-shares

 

[itimpi]

5 hours ago, competent-bailout3425 said:

what do you mean "Exclusif access"?my share option is already pool only.

You do need to enable Exclusive Access option under Settings->Global Share settings to get this option enabled for user shares which meet the criteria.

[competent-bailout3425]

2 hours ago, itimpi said:

You do need to enable Exclusive Access option under Settings->Global Share settings to get this option enabled for user shares which meet the criteria.

After I  enable the Exclusive Access option  Settings->Global Share setting, the Exclusive access option in share tab is still unconfigable.

 

[JorgeB]

1 hour ago, competent-bailout3425 said:

After I  enable the Exclusive Access option  Settings->Global Share setting, the Exclusive access option in share tab is still unconfigable.

It's not configurable, it's automatic if the share only exists in a pool, it won't work for shares that exist only (or also) in the array.

[trurl]

4 hours ago, competent-bailout3425 said:

Exclusive access option in share tab is still unconfigable.

Try stopping and starting the array.

[competent-bailout3425]

13 hours ago, trurl said:

Try stopping and starting the array.

that work,thank