I've been plagued this issue in both 6.12 and 7.0.0-beta.1/2 and I have finally narrowed down the exact configuration cause.

 

In all of my instances of this docker container, I am mapping the root of one of my shares (R/W). Normally, the list action on the webui will take 700-1000ms to respond back but it's been take 10+ seconds and it's when any ZFS pool is configured to be the primary storage with the array being secondary.

 

Here is the results of some configurations for this share that I've tried, where-in it's the ONLY change that is made:

• Primary: Array, Secondary: None - 700-1000ms response time
• Primary: single 2TB NVME encrypted btrfs, Secondary: Array - 700-1000ms response time
• Primary: single 2TB NVME encrypted zfs, Secondary: Array - 6000-8000ms response time
• Primary: 3x 2-Mirror 10TB encrypted zfs, Secondary: Array - 10000-12000 response time

 

I have done testing with the cachedirs plugin and it does not affect these outcomes at all. I originally thought this was the cause because my zfs pool wasn't named "cache" at the time and the plugin only seems to cache /mnt/disk* and /mnt/cache, but later renaming the pool had zero effect.

 

Now this is also when the zfs cache/primary drive is completely empty, it still affects the performance. I also tried with direct_io turned on and it had zero effect (though I didn't expect it to).

 

I am going to be setting up a container to test whether this also occurs with the zfs pool as an exclusive share or if this is entirely cause by zfs behind fuse.

I couldn't find any similar issues so I don't know if this is just extremely specific to whatever rtorrent is doing. I'd like to determine what exact command(s) are taking so long to return back when the share is configured this way, but that would be a whole other deep dive to get some proper profiling on the process because the debug logs didn't give me any useful information.

ecba-diagnostics-20240715-1508.zip

Fuse has per-file overhead as it has to check each drive for where it is, so anything trying to list large numbers of files over fuse will be pretty slow.

Edited July 15, 20241 yr by Kilrah

1 minute ago, Kilrah said:

Fuse has per-file overhead as it has to check each drive for where it is, so anything trying to list large numbers of files over fuse will be pretty slow.

 

This doesn't happen when it's just the array (27 drives) or when the nvme cache drive as primary is formatted as btrfs. As soon as that same cache drive is formatted to zfs, the performance tanks, with no other changes.

I can confirm that a container mapped to an exclusive share on my mirror ZFS pool is extremely fast and responding in <200ms with 1000 torrents.

 

There is definitely a weird interaction happening with zfs behind fuse, and it's not inherent to zfs (which I didn't expect anyway).

Just to tell the story of what I am encountering, visually. This is how I can dynamically recreate this problem.

Note: Neither cache or temp pools have any files for the software share. Not even the folder. Everything is on the Array currently.

 

But now when I swap the temp (btfrs) pool with the cache (zfs) pool:

 

This result is reflected immediately after swapping the pools and it makes no difference if I restart the container or the entire machine.

 

I'll see if I can replicate this performance drop in a container but outside of rtorrent and then see if that also can be replicated outside of docker.

I decided to take another look at this with v7.2-rc1 and openzfs v2.3.4 to see if any of the latest improvements happened to alleviate this problem.

TL;DR statfs() for zfs behind fuse is 15-20x slower than anything other filesystem, even with mitigations.

I first attempted to recreate the scenario in the OP as-is with just the updated tooling. There was zero changes and I saw the exact same performance degradation.

The problem seems to occur when statfs is used by rtorrent to get properties like free disk space. I was able to somewhat solve the problem by removing the eliminating the free space property the exists on every torrent so it wouldn't be requested for every xmlrpc query, but there were still statfs calls being made periodically.

Since I'm pretty sure that is the issue I decided to see if I could just demonstrate the degradation with minimal additional context. On a new share with just generated directories and files here is the average time for a statfs call to take in different scenarios:

ZFS - Exclusive Share (4 drive mirror w/ nvme special vdev)

Total elapsed time: 179.99 ms (0.18 seconds)

Successful calls: 1000

Failed calls: 0

statfs() timing breakdown:

Min time: 0.153 ms

Max time: 0.248 ms

Avg time: 0.180 ms

Total statfs() time: 179.85 ms

Overhead (non-statfs): 0.14 ms (0.1%)

Array Only (28 mixed HDDs)

Total elapsed time: 585.95 ms (0.59 seconds)

Successful calls: 1000

Failed calls: 0

statfs() timing breakdown:

Min time: 0.372 ms

Max time: 1.166 ms

Avg time: 0.586 ms

Total statfs() time: 585.84 ms

Overhead (non-statfs): 0.10 ms (0.0%)

Array + ZFS (Primary: Above ZFS pool, Secondary: Above array)

Total elapsed time: 9387.06 ms (9.39 seconds)

Successful calls: 1000

Failed calls: 0

statfs() timing breakdown:

Min time: 9.045 ms

Max time: 12.728 ms

Avg time: 9.387 ms

Total statfs() time: 9386.79 ms

Overhead (non-statfs): 0.28 ms (0.0%)

It doesn't matter if the files are on the array, or the cache, or even if the dataset is not yet even created. It didn't matter how many settings I changed or tuned. Every single time there is that much of a hit just having any zfs pool on a non-exclusive share. And I tested many time again with using the same devices formatted as btrfs and the performance almost always was the same as the array only tests.

I also tested this on a more modern system and it just shift all the numbers upwards with the same relative degradation. 0.159ms avg for array only and ~3.5 ms avg for array + zfs.

To me this seems much more extreme than just fuse overhead + zfs overhead. For now I am just going to have to move ahead with a different pool topology than I would have liked. Here is the benchmarking code used:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/vfs.h>
#include <sys/time.h>
#include <dirent.h>
#include <unistd.h>
#include <errno.h>

#define MAX_PATHS 10000
#define MAX_PATH_LEN 4096

double get_time_ms() {
    struct timeval tv;
    gettimeofday(&tv, NULL);
    return (tv.tv_sec * 1000.0) + (tv.tv_usec / 1000.0);
}

int collect_directories(const char *base_path, char paths[][MAX_PATH_LEN], int *count, int max_count, int max_depth) {
    if (max_depth == 0 || *count >= max_count) {
        return 0;
    }

    DIR *dir = opendir(base_path);
    if (!dir) {
        return -1;
    }

    struct dirent *entry;
    while ((entry = readdir(dir)) != NULL && *count < max_count) {
        if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0) {
            continue;
        }

        char full_path[MAX_PATH_LEN];
        snprintf(full_path, sizeof(full_path), "%s/%s", base_path, entry->d_name);

        if (entry->d_type == DT_DIR) {
            strncpy(paths[*count], full_path, MAX_PATH_LEN - 1);
            paths[*count][MAX_PATH_LEN - 1] = '\0';
            (*count)++;

            collect_directories(full_path, paths, count, max_count, max_depth - 1);
        }
    }

    closedir(dir);
    return 0;
}

void perform_statfs_burst(char paths[][MAX_PATH_LEN], int path_count, int calls_per_path) {
    printf("\n=== Performing statfs() burst ===\n");
    printf("Paths: %d\n", path_count);
    printf("Calls per path: %d\n", calls_per_path);
    printf("Total calls: %d\n", path_count * calls_per_path);
    printf("\n");

    double start_time = get_time_ms();
    double total_statfs_time = 0;
    double min_time = 999999;
    double max_time = 0;
    int success_count = 0;
    int error_count = 0;

    printf("Sample timings (first 10 calls):\n");

    for (int i = 0; i < path_count; i++) {
        for (int j = 0; j < calls_per_path; j++) {
            struct statfs buf;

            double call_start = get_time_ms();
            int result = statfs(paths[i], &buf);
            double call_end = get_time_ms();
            double call_time = call_end - call_start;

            if (result == 0) {
                success_count++;
                total_statfs_time += call_time;

                if (call_time < min_time) min_time = call_time;
                if (call_time > max_time) max_time = call_time;

                if (success_count <= 10) {
                    printf("  [%d] %s: %.3f ms (type=0x%lx, blocks=%lu, free=%lu)\n",
                           success_count, paths[i], call_time,
                           (unsigned long)buf.f_type,
                           (unsigned long)buf.f_blocks,
                           (unsigned long)buf.f_bfree);
                }
            } else {
                error_count++;
                if (error_count <= 10) {
                    printf("  ERROR: %s: %s\n", paths[i], strerror(errno));
                }
            }
        }
    }

    double end_time = get_time_ms();
    double total_elapsed = end_time - start_time;

    printf("\n=== Results ===\n");
    printf("Total elapsed time: %.2f ms (%.2f seconds)\n", total_elapsed, total_elapsed / 1000.0);
    printf("Successful calls: %d\n", success_count);
    printf("Failed calls: %d\n", error_count);

    if (success_count > 0) {
        printf("\nstatfs() timing breakdown:\n");
        printf("  Min time: %.3f ms\n", min_time);
        printf("  Max time: %.3f ms\n", max_time);
        printf("  Avg time: %.3f ms\n", total_statfs_time / success_count);
        printf("  Total statfs() time: %.2f ms\n", total_statfs_time);
        printf("  Overhead (non-statfs): %.2f ms (%.1f%%)\n",
               total_elapsed - total_statfs_time,
               ((total_elapsed - total_statfs_time) / total_elapsed) * 100);
    }
}

int main(int argc, char *argv[]) {
    if (argc < 2) {
        fprintf(stderr, "Usage: %s <base_path> [iterations_per_path] [max_paths] [max_depth]\n", argv[0]);
        fprintf(stderr, "\n");
        fprintf(stderr, "Examples:\n");
        fprintf(stderr, "  %s /downloads/seed 2 500 2\n", argv[0]);
        fprintf(stderr, "  %s /mnt/user/downloads 2 2000 3\n", argv[0]);
        fprintf(stderr, "\n");
        fprintf(stderr, "Arguments:\n");
        fprintf(stderr, "  base_path         : Root directory to scan (e.g., /downloads/seed)\n");
        fprintf(stderr, "  iterations        : Number of statfs() calls per directory (default: 2)\n");
        fprintf(stderr, "                      (rtorrent calls statfs ~2 times per torrent per request)\n");
        fprintf(stderr, "  max_paths         : Maximum directories to check (default: 2000)\n");
        fprintf(stderr, "  max_depth         : Recursion depth (default: 3)\n");
        return 1;
    }

    const char *base_path = argv[1];
    int iterations = (argc > 2) ? atoi(argv[2]) : 2;
    int max_paths = (argc > 3) ? atoi(argv[3]) : 2000;
    int max_depth = (argc > 4) ? atoi(argv[4]) : 3;

    if (iterations < 1) iterations = 1;
    if (iterations > 10) iterations = 10;
    if (max_paths < 1) max_paths = 1;
    if (max_paths > MAX_PATHS) max_paths = MAX_PATHS;
    if (max_depth < 1) max_depth = 1;
    if (max_depth > 5) max_depth = 5;

    printf("=== statfs() Benchmark (rtorrent simulation) ===\n");
    printf("Base path: %s\n", base_path);
    printf("Iterations per path: %d\n", iterations);
    printf("Max paths: %d\n", max_paths);
    printf("Max recursion depth: %d\n", max_depth);
    printf("\n");

    char (*paths)[MAX_PATH_LEN] = malloc(sizeof(char[MAX_PATHS][MAX_PATH_LEN]));
    if (!paths) {
        fprintf(stderr, "Failed to allocate memory\n");
        return 1;
    }

    int path_count = 0;

    printf("Scanning directories...\n");
    if (collect_directories(base_path, paths, &path_count, max_paths, max_depth) < 0) {
        fprintf(stderr, "Failed to scan directory: %s\n", strerror(errno));
        free(paths);
        return 1;
    }

    printf("Found %d directories\n", path_count);

    if (path_count == 0) {
        fprintf(stderr, "No directories found in %s\n", base_path);
        free(paths);
        return 1;
    }

    perform_statfs_burst(paths, path_count, iterations);

    free(paths);
    return 0;
}

Edited October 14, 2025Oct 14 by sjrahn

Was trying to run the script but got an error, any idea why?

root@Test:/# ./test
./test: line 13: syntax error near unexpected token `('
./test: line 13: `double get_time_ms() {'

53 minutes ago, JorgeB said:

Was trying to run the script but got an error, any idea why?

That's C code, not a script... would need to be compiled

33 minutes ago, Kilrah said:

That's C code, not a script...

lol, thanks, no wonder then, I'm afraid compiling stuff is way out of my wheelhouse.