sjrahn
Members
-
Joined
-
Last visited
-
I decided to take another look at this with v7.2-rc1 and openzfs v2.3.4 to see if any of the latest improvements happened to alleviate this problem. TL;DR statfs() for zfs behind fuse is 15-20x slower than anything other filesystem, even with mitigations. I first attempted to recreate the scenario in the OP as-is with just the updated tooling. There was zero changes and I saw the exact same performance degradation. The problem seems to occur when statfs is used by rtorrent to get properties like free disk space. I was able to somewhat solve the problem by removing the eliminating the free space property the exists on every torrent so it wouldn't be requested for every xmlrpc query, but there were still statfs calls being made periodically. Since I'm pretty sure that is the issue I decided to see if I could just demonstrate the degradation with minimal additional context. On a new share with just generated directories and files here is the average time for a statfs call to take in different scenarios: ZFS - Exclusive Share (4 drive mirror w/ nvme special vdev) Total elapsed time: 179.99 ms (0.18 seconds) Successful calls: 1000 Failed calls: 0 statfs() timing breakdown: Min time: 0.153 ms Max time: 0.248 ms Avg time: 0.180 ms Total statfs() time: 179.85 ms Overhead (non-statfs): 0.14 ms (0.1%) Array Only (28 mixed HDDs) Total elapsed time: 585.95 ms (0.59 seconds) Successful calls: 1000 Failed calls: 0 statfs() timing breakdown: Min time: 0.372 ms Max time: 1.166 ms Avg time: 0.586 ms Total statfs() time: 585.84 ms Overhead (non-statfs): 0.10 ms (0.0%) Array + ZFS (Primary: Above ZFS pool, Secondary: Above array) Total elapsed time: 9387.06 ms (9.39 seconds) Successful calls: 1000 Failed calls: 0 statfs() timing breakdown: Min time: 9.045 ms Max time: 12.728 ms Avg time: 9.387 ms Total statfs() time: 9386.79 ms Overhead (non-statfs): 0.28 ms (0.0%) It doesn't matter if the files are on the array, or the cache, or even if the dataset is not yet even created. It didn't matter how many settings I changed or tuned. Every single time there is that much of a hit just having any zfs pool on a non-exclusive share. And I tested many time again with using the same devices formatted as btrfs and the performance almost always was the same as the array only tests. I also tested this on a more modern system and it just shift all the numbers upwards with the same relative degradation. 0.159ms avg for array only and ~3.5 ms avg for array + zfs. To me this seems much more extreme than just fuse overhead + zfs overhead. For now I am just going to have to move ahead with a different pool topology than I would have liked. Here is the benchmarking code used: #include <stdio.h> #include <stdlib.h> #include <string.h> #include <sys/vfs.h> #include <sys/time.h> #include <dirent.h> #include <unistd.h> #include <errno.h> #define MAX_PATHS 10000 #define MAX_PATH_LEN 4096 double get_time_ms() { struct timeval tv; gettimeofday(&tv, NULL); return (tv.tv_sec * 1000.0) + (tv.tv_usec / 1000.0); } int collect_directories(const char *base_path, char paths[][MAX_PATH_LEN], int *count, int max_count, int max_depth) { if (max_depth == 0 || *count >= max_count) { return 0; } DIR *dir = opendir(base_path); if (!dir) { return -1; } struct dirent *entry; while ((entry = readdir(dir)) != NULL && *count < max_count) { if (strcmp(entry->d_name, ".") == 0 || strcmp(entry->d_name, "..") == 0) { continue; } char full_path[MAX_PATH_LEN]; snprintf(full_path, sizeof(full_path), "%s/%s", base_path, entry->d_name); if (entry->d_type == DT_DIR) { strncpy(paths[*count], full_path, MAX_PATH_LEN - 1); paths[*count][MAX_PATH_LEN - 1] = '\0'; (*count)++; collect_directories(full_path, paths, count, max_count, max_depth - 1); } } closedir(dir); return 0; } void perform_statfs_burst(char paths[][MAX_PATH_LEN], int path_count, int calls_per_path) { printf("\n=== Performing statfs() burst ===\n"); printf("Paths: %d\n", path_count); printf("Calls per path: %d\n", calls_per_path); printf("Total calls: %d\n", path_count * calls_per_path); printf("\n"); double start_time = get_time_ms(); double total_statfs_time = 0; double min_time = 999999; double max_time = 0; int success_count = 0; int error_count = 0; printf("Sample timings (first 10 calls):\n"); for (int i = 0; i < path_count; i++) { for (int j = 0; j < calls_per_path; j++) { struct statfs buf; double call_start = get_time_ms(); int result = statfs(paths[i], &buf); double call_end = get_time_ms(); double call_time = call_end - call_start; if (result == 0) { success_count++; total_statfs_time += call_time; if (call_time < min_time) min_time = call_time; if (call_time > max_time) max_time = call_time; if (success_count <= 10) { printf(" [%d] %s: %.3f ms (type=0x%lx, blocks=%lu, free=%lu)\n", success_count, paths[i], call_time, (unsigned long)buf.f_type, (unsigned long)buf.f_blocks, (unsigned long)buf.f_bfree); } } else { error_count++; if (error_count <= 10) { printf(" ERROR: %s: %s\n", paths[i], strerror(errno)); } } } } double end_time = get_time_ms(); double total_elapsed = end_time - start_time; printf("\n=== Results ===\n"); printf("Total elapsed time: %.2f ms (%.2f seconds)\n", total_elapsed, total_elapsed / 1000.0); printf("Successful calls: %d\n", success_count); printf("Failed calls: %d\n", error_count); if (success_count > 0) { printf("\nstatfs() timing breakdown:\n"); printf(" Min time: %.3f ms\n", min_time); printf(" Max time: %.3f ms\n", max_time); printf(" Avg time: %.3f ms\n", total_statfs_time / success_count); printf(" Total statfs() time: %.2f ms\n", total_statfs_time); printf(" Overhead (non-statfs): %.2f ms (%.1f%%)\n", total_elapsed - total_statfs_time, ((total_elapsed - total_statfs_time) / total_elapsed) * 100); } } int main(int argc, char *argv[]) { if (argc < 2) { fprintf(stderr, "Usage: %s <base_path> [iterations_per_path] [max_paths] [max_depth]\n", argv[0]); fprintf(stderr, "\n"); fprintf(stderr, "Examples:\n"); fprintf(stderr, " %s /downloads/seed 2 500 2\n", argv[0]); fprintf(stderr, " %s /mnt/user/downloads 2 2000 3\n", argv[0]); fprintf(stderr, "\n"); fprintf(stderr, "Arguments:\n"); fprintf(stderr, " base_path : Root directory to scan (e.g., /downloads/seed)\n"); fprintf(stderr, " iterations : Number of statfs() calls per directory (default: 2)\n"); fprintf(stderr, " (rtorrent calls statfs ~2 times per torrent per request)\n"); fprintf(stderr, " max_paths : Maximum directories to check (default: 2000)\n"); fprintf(stderr, " max_depth : Recursion depth (default: 3)\n"); return 1; } const char *base_path = argv[1]; int iterations = (argc > 2) ? atoi(argv[2]) : 2; int max_paths = (argc > 3) ? atoi(argv[3]) : 2000; int max_depth = (argc > 4) ? atoi(argv[4]) : 3; if (iterations < 1) iterations = 1; if (iterations > 10) iterations = 10; if (max_paths < 1) max_paths = 1; if (max_paths > MAX_PATHS) max_paths = MAX_PATHS; if (max_depth < 1) max_depth = 1; if (max_depth > 5) max_depth = 5; printf("=== statfs() Benchmark (rtorrent simulation) ===\n"); printf("Base path: %s\n", base_path); printf("Iterations per path: %d\n", iterations); printf("Max paths: %d\n", max_paths); printf("Max recursion depth: %d\n", max_depth); printf("\n"); char (*paths)[MAX_PATH_LEN] = malloc(sizeof(char[MAX_PATHS][MAX_PATH_LEN])); if (!paths) { fprintf(stderr, "Failed to allocate memory\n"); return 1; } int path_count = 0; printf("Scanning directories...\n"); if (collect_directories(base_path, paths, &path_count, max_paths, max_depth) < 0) { fprintf(stderr, "Failed to scan directory: %s\n", strerror(errno)); free(paths); return 1; } printf("Found %d directories\n", path_count); if (path_count == 0) { fprintf(stderr, "No directories found in %s\n", base_path); free(paths); return 1; } perform_statfs_burst(paths, path_count, iterations); free(paths); return 0; }
-
Just to tell the story of what I am encountering, visually. This is how I can dynamically recreate this problem. Note: Neither cache or temp pools have any files for the software share. Not even the folder. Everything is on the Array currently. But now when I swap the temp (btfrs) pool with the cache (zfs) pool: This result is reflected immediately after swapping the pools and it makes no difference if I restart the container or the entire machine. I'll see if I can replicate this performance drop in a container but outside of rtorrent and then see if that also can be replicated outside of docker.
-
I can confirm that a container mapped to an exclusive share on my mirror ZFS pool is extremely fast and responding in <200ms with 1000 torrents. There is definitely a weird interaction happening with zfs behind fuse, and it's not inherent to zfs (which I didn't expect anyway).
-
This doesn't happen when it's just the array (27 drives) or when the nvme cache drive as primary is formatted as btrfs. As soon as that same cache drive is formatted to zfs, the performance tanks, with no other changes.
-
I've been plagued this issue in both 6.12 and 7.0.0-beta.1/2 and I have finally narrowed down the exact configuration cause. In all of my instances of this docker container, I am mapping the root of one of my shares (R/W). Normally, the list action on the webui will take 700-1000ms to respond back but it's been take 10+ seconds and it's when any ZFS pool is configured to be the primary storage with the array being secondary. Here is the results of some configurations for this share that I've tried, where-in it's the ONLY change that is made: Primary: Array, Secondary: None - 700-1000ms response time Primary: single 2TB NVME encrypted btrfs, Secondary: Array - 700-1000ms response time Primary: single 2TB NVME encrypted zfs, Secondary: Array - 6000-8000ms response time Primary: 3x 2-Mirror 10TB encrypted zfs, Secondary: Array - 10000-12000 response time I have done testing with the cachedirs plugin and it does not affect these outcomes at all. I originally thought this was the cause because my zfs pool wasn't named "cache" at the time and the plugin only seems to cache /mnt/disk* and /mnt/cache, but later renaming the pool had zero effect. Now this is also when the zfs cache/primary drive is completely empty, it still affects the performance. I also tried with direct_io turned on and it had zero effect (though I didn't expect it to). I am going to be setting up a container to test whether this also occurs with the zfs pool as an exclusive share or if this is entirely cause by zfs behind fuse. I couldn't find any similar issues so I don't know if this is just extremely specific to whatever rtorrent is doing. I'd like to determine what exact command(s) are taking so long to return back when the share is configured this way, but that would be a whole other deep dive to get some proper profiling on the process because the debug logs didn't give me any useful information. ecba-diagnostics-20240715-1508.zip
-
I just upgraded to 7.0.0-beta.1 from 6.12.10. I am adding a new 6 drive ZFS pool so I wanted to do it with the latest version. Upon configuring and letting it go to format, it fails on the format and put the disk in an error state. These 6 drives were all previously working fine in the Array (as XFS Encrypted) and replaced with larger drive only in the last week. I tried a couple more times and with a 4 drive BTRFS mirror instead and received the same (or similar) errors. After multiple reboots I tried the drives individually with with XFS, ZFS - encrypted, and BTRFS - encrypted and it's the same story, so it's independent of the filesystem it seems. I then tried a SATA drive and it works totally fine... Log from formatting one of the SAS drives (via UD plugin): Jul 9 15:08:46 ecba unassigned.devices: Format device '/dev/sdg'. Jul 9 15:08:46 ecba unassigned.devices: Device '/dev/sdg' block size: 19532873728. Jul 9 15:08:46 ecba unassigned.devices: Clearing partition table of disk '/dev/sdg'. Jul 9 15:08:46 ecba unassigned.devices: Clear partition result: 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0379989 s, 55.2 MB/s Jul 9 15:08:48 ecba unassigned.devices: Reloading disk '/dev/sdg' partition table. Jul 9 15:08:48 ecba unassigned.devices: Reload partition table result: /dev/sdg: re-reading partition table Jul 9 15:08:48 ecba unassigned.devices: Creating Unraid compatible gpt partition on disk '/dev/sdg'. Jul 9 15:08:50 ecba kernel: sdg: sdg1 Jul 9 15:08:50 ecba unassigned.devices: Create gpt partition table result: Creating new GPT entries in memory. The operation has completed successfully. Jul 9 15:08:52 ecba unassigned.devices: Reloading disk '/dev/sdg' partition table. Jul 9 15:08:52 ecba kernel: sdg: sdg1 Jul 9 15:08:52 ecba unassigned.devices: Reload partition table result: /dev/sdg: re-reading partition table Jul 9 15:08:52 ecba unassigned.devices: Formatting disk '/dev/sdg' with 'xfs' filesystem. Jul 9 15:08:52 ecba unassigned.devices: Format drive command: /sbin/mkfs.xfs -f '/dev/sdg1' 2>&1 Jul 9 15:08:53 ecba kernel: sd 7:0:5:0: device_block, handle(0x000f) Jul 9 15:08:54 ecba kernel: sd 7:0:5:0: device_unblock and setting to running, handle(0x000f) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:08:55 ecba kernel: sd 7:0:5:0: Power-on or device reset occurred Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: sd 7:0:5:0: [sdg] tag#1215 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=15s Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: sd 7:0:5:0: [sdg] tag#1215 CDB: opcode=0x93 93 00 00 00 00 02 80 09 00 20 00 00 ff f8 00 00 Jul 9 15:09:08 ecba kernel: I/O error, dev sdg, sector 10738008096 op 0x9:(WRITE_ZEROES) flags 0x8000000 phys_seg 0 prio class 0 Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: sd 7:0:5:0: [sdg] tag#1214 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=15s Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: sd 7:0:5:0: [sdg] tag#1214 CDB: opcode=0x93 93 00 00 00 00 02 80 04 00 48 00 00 ff f8 00 00 Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: I/O error, dev sdg, sector 10737680456 op 0x9:(WRITE_ZEROES) flags 0x8000000 phys_seg 0 prio class 0 Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: sd 7:0:5:0: [sdg] tag#1180 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=15s Jul 9 15:09:08 ecba kernel: sd 7:0:5:0: [sdg] tag#1180 CDB: opcode=0x93 93 00 00 00 00 02 80 08 00 28 00 00 ff f8 00 00 Jul 9 15:09:08 ecba kernel: I/O error, dev sdg, sector 10737942568 op 0x9:(WRITE_ZEROES) flags 0x8000000 phys_seg 0 prio class 0 Jul 9 15:09:08 ecba kernel: sd 7:0:5:0: [sdg] tag#1183 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=15s Jul 9 15:09:08 ecba kernel: sd 7:0:5:0: [sdg] tag#1183 CDB: opcode=0x93 93 00 00 00 00 02 80 2f fe e8 00 00 ff f8 00 00 Jul 9 15:09:08 ecba kernel: I/O error, dev sdg, sector 10740563688 op 0x9:(WRITE_ZEROES) flags 0x8000000 phys_seg 0 prio class 0 Jul 9 15:09:08 ecba kernel: sd 7:0:5:0: [sdg] tag#1189 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=15s Jul 9 15:09:08 ecba kernel: sd 7:0:5:0: [sdg] tag#1189 CDB: opcode=0x93 93 00 00 00 00 02 80 2a ff 10 00 00 ff f8 00 00 Jul 9 15:09:08 ecba kernel: I/O error, dev sdg, sector 10740236048 op 0x9:(WRITE_ZEROES) flags 0x8000000 phys_seg 0 prio class 0 Jul 9 15:09:08 ecba kernel: sd 7:0:5:0: [sdg] tag#1175 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=15s Jul 9 15:09:08 ecba kernel: sd 7:0:5:0: [sdg] tag#1175 CDB: opcode=0x93 93 00 00 00 00 02 80 0d 00 00 00 00 ff f8 00 00 Jul 9 15:09:08 ecba kernel: I/O error, dev sdg, sector 10738270208 op 0x9:(WRITE_ZEROES) flags 0x8000000 phys_seg 0 prio class 0 Jul 9 15:09:08 ecba kernel: sd 7:0:5:0: [sdg] tag#1176 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=15s Jul 9 15:09:08 ecba kernel: sd 7:0:5:0: [sdg] tag#1176 CDB: opcode=0x93 93 00 00 00 00 02 80 0b 00 10 00 00 ff f8 00 00 Jul 9 15:09:08 ecba kernel: I/O error, dev sdg, sector 10738139152 op 0x9:(WRITE_ZEROES) flags 0x8000000 phys_seg 0 prio class 0 Jul 9 15:09:08 ecba kernel: sd 7:0:5:0: [sdg] tag#1177 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=15s Jul 9 15:09:08 ecba kernel: sd 7:0:5:0: [sdg] tag#1177 CDB: opcode=0x93 93 00 00 00 00 02 80 00 00 68 00 00 ff f8 00 00 Jul 9 15:09:08 ecba kernel: I/O error, dev sdg, sector 10737418344 op 0x9:(WRITE_ZEROES) flags 0x8000000 phys_seg 0 prio class 0 Jul 9 15:09:08 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:08 ecba kernel: sd 7:0:5:0: [sdg] tag#1153 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=15s Jul 9 15:09:08 ecba kernel: sd 7:0:5:0: [sdg] tag#1153 CDB: opcode=0x93 93 00 00 00 00 02 80 15 ff b8 00 00 ff f8 00 00 Jul 9 15:09:08 ecba kernel: I/O error, dev sdg, sector 10738859960 op 0x9:(WRITE_ZEROES) flags 0x8000000 phys_seg 0 prio class 0 Jul 9 15:09:08 ecba kernel: sd 7:0:5:0: [sdg] tag#1166 UNKNOWN(0x2003) Result: hostbyte=0x0b driverbyte=DRIVER_OK cmd_age=15s Jul 9 15:09:08 ecba kernel: sd 7:0:5:0: [sdg] tag#1166 CDB: opcode=0x93 93 00 00 00 00 02 80 07 00 30 00 00 ff f8 00 00 Jul 9 15:09:08 ecba kernel: I/O error, dev sdg, sector 10737877040 op 0x9:(WRITE_ZEROES) flags 0x8000000 phys_seg 0 prio class 0 Jul 9 15:09:08 ecba kernel: sd 7:0:5:0: device_block, handle(0x000f) Jul 9 15:09:10 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:10 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:10 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:10 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:10 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:10 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:10 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:10 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:10 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:10 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:10 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:10 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:10 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:10 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:10 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:10 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:10 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:10 ecba kernel: sd 7:0:5:0: device_unblock and setting to running, handle(0x000f) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:14 ecba kernel: sd 7:0:5:0: Power-on or device reset occurred Jul 9 15:09:21 ecba kernel: scsi_io_completion_action: 54 callbacks suppressed Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2655 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=13s Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2655 Sense Key : 0x5 [current] [descriptor] Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2655 ASC=0x24 ASCQ=0x0 Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2655 CDB: opcode=0x8a 8a 00 00 00 00 02 80 00 3c 68 00 00 04 00 00 00 Jul 9 15:09:21 ecba kernel: blk_print_req_error: 54 callbacks suppressed Jul 9 15:09:21 ecba kernel: critical target error, dev sdg, sector 10737433704 op 0x1:(WRITE) flags 0x0 phys_seg 128 prio class 0 Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2656 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=13s Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2656 Sense Key : 0x5 [current] [descriptor] Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2656 ASC=0x24 ASCQ=0x0 Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2656 CDB: opcode=0x8a 8a 00 00 00 00 02 80 00 38 68 00 00 04 00 00 00 Jul 9 15:09:21 ecba kernel: critical target error, dev sdg, sector 10737432680 op 0x1:(WRITE) flags 0x4000 phys_seg 128 prio class 0 Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2657 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=13s Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2657 Sense Key : 0x5 [current] [descriptor] Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2657 ASC=0x24 ASCQ=0x0 Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2657 CDB: opcode=0x8a 8a 00 00 00 00 02 80 00 34 68 00 00 04 00 00 00 Jul 9 15:09:21 ecba kernel: critical target error, dev sdg, sector 10737431656 op 0x1:(WRITE) flags 0x0 phys_seg 128 prio class 0 Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2658 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=13s Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2658 Sense Key : 0x5 [current] [descriptor] Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2658 ASC=0x24 ASCQ=0x0 Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2658 CDB: opcode=0x8a 8a 00 00 00 00 02 80 00 30 68 00 00 04 00 00 00 Jul 9 15:09:21 ecba kernel: critical target error, dev sdg, sector 10737430632 op 0x1:(WRITE) flags 0x4000 phys_seg 128 prio class 0 Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2659 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=13s Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2659 Sense Key : 0x5 [current] [descriptor] Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2659 ASC=0x24 ASCQ=0x0 Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2659 CDB: opcode=0x8a 8a 00 00 00 00 02 80 00 2c 68 00 00 04 00 00 00 Jul 9 15:09:21 ecba kernel: critical target error, dev sdg, sector 10737429608 op 0x1:(WRITE) flags 0x0 phys_seg 128 prio class 0 Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2660 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=13s Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2660 Sense Key : 0x5 [current] [descriptor] Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2660 ASC=0x24 ASCQ=0x0 Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2660 CDB: opcode=0x8a 8a 00 00 00 00 02 80 00 28 68 00 00 04 00 00 00 Jul 9 15:09:21 ecba kernel: critical target error, dev sdg, sector 10737428584 op 0x1:(WRITE) flags 0x4000 phys_seg 128 prio class 0 Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2661 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=13s Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2661 Sense Key : 0x5 [current] [descriptor] Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2661 ASC=0x24 ASCQ=0x0 Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2661 CDB: opcode=0x8a 8a 00 00 00 00 02 80 00 24 68 00 00 04 00 00 00 Jul 9 15:09:21 ecba kernel: critical target error, dev sdg, sector 10737427560 op 0x1:(WRITE) flags 0x0 phys_seg 128 prio class 0 Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2662 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=13s Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2662 Sense Key : 0x5 [current] [descriptor] Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2662 ASC=0x24 ASCQ=0x0 Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2662 CDB: opcode=0x8a 8a 00 00 00 00 02 80 00 20 68 00 00 04 00 00 00 Jul 9 15:09:21 ecba kernel: critical target error, dev sdg, sector 10737426536 op 0x1:(WRITE) flags 0x4000 phys_seg 128 prio class 0 Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2663 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=13s Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2663 Sense Key : 0x5 [current] [descriptor] Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2663 ASC=0x24 ASCQ=0x0 Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2663 CDB: opcode=0x8a 8a 00 00 00 00 02 80 00 1c 68 00 00 04 00 00 00 Jul 9 15:09:21 ecba kernel: critical target error, dev sdg, sector 10737425512 op 0x1:(WRITE) flags 0x0 phys_seg 128 prio class 0 Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2664 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=13s Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2664 Sense Key : 0x5 [current] [descriptor] Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2664 ASC=0x24 ASCQ=0x0 Jul 9 15:09:21 ecba kernel: sd 7:0:5:0: [sdg] tag#2664 CDB: opcode=0x8a 8a 00 00 00 00 02 80 00 18 68 00 00 04 00 00 00 Jul 9 15:09:21 ecba kernel: critical target error, dev sdg, sector 10737424488 op 0x1:(WRITE) flags 0x4000 phys_seg 128 prio class 0 Jul 9 15:09:23 ecba kernel: mpt3sas_cm0: log_info(0x31120101): originator(PL), code(0x12), sub_code(0x0101) Jul 9 15:09:23 ecba unassigned.devices: Format disk '/dev/sdg' with 'xfs' filesystem failed: mkfs.xfs: libxfs_device_zero write failed: Remote I/O error meta-data=/dev/sdg1 isize=512 agcount=10, agsize=268435455 blks = sectsz=4096 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=1 = reflink=1 bigtime=1 inobtcount=1 nrext64=1 data = bsize=4096 blocks=2441609203, imaxpct=5 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=521728, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 Jul 9 15:09:23 ecba kernel: sd 7:0:5:0: device_block, handle(0x000f) Jul 9 15:09:26 ecba kernel: sd 7:0:5:0: device_unblock and setting to running, handle(0x000f) Jul 9 15:09:27 ecba kernel: mpt3sas_cm0: log_info(0x31110e05): originator(PL), code(0x11), sub_code(0x0e05) Jul 9 15:09:27 ecba kernel: scsi_io_completion_action: 4068 callbacks suppressed Jul 9 15:09:27 ecba kernel: sd 7:0:5:0: [sdg] tag#2856 UNKNOWN(0x2003) Result: hostbyte=0x01 driverbyte=DRIVER_OK cmd_age=4s Jul 9 15:09:27 ecba kernel: sd 7:0:5:0: [sdg] tag#2856 CDB: opcode=0x88 88 00 00 00 00 04 8c 3f ff 40 00 00 00 08 00 00 Jul 9 15:09:27 ecba kernel: Buffer I/O error on dev sdg1, logical block 2441609184, async page read Jul 9 15:09:27 ecba kernel: sd 7:0:5:0: [sdg] Synchronizing SCSI cache Jul 9 15:09:27 ecba kernel: sd 7:0:5:0: [sdg] Synchronize Cache(10) failed: Result: hostbyte=0x01 driverbyte=DRIVER_OK Jul 9 15:09:27 ecba kernel: mpt3sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x5000cca2517194f1) Jul 9 15:09:27 ecba kernel: mpt3sas_cm0: removing handle(0x000f), sas_addr(0x5000cca2517194f1) Jul 9 15:09:27 ecba kernel: mpt3sas_cm0: enclosure logical id(0x500304801ebaa5bf), slot(5) Jul 9 15:09:27 ecba kernel: mpt3sas_cm0: enclosure level(0x0000), connector name( ) Jul 9 15:09:32 ecba emhttpd: offline: HUH721010AL4200_7PJ0GEUC_35000cca2517194f0 (sdg) 512 19532873728 Log from formatting a SATA drive (via UD plugin): Jul 9 15:17:01 ecba unassigned.devices: Format device '/dev/sdg'. Jul 9 15:17:01 ecba unassigned.devices: Device '/dev/sdg' block size: 15628053168. Jul 9 15:17:01 ecba unassigned.devices: Clearing partition table of disk '/dev/sdg'. Jul 9 15:17:01 ecba unassigned.devices: Clear partition result: 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.0913783 s, 23.0 MB/s Jul 9 15:17:03 ecba unassigned.devices: Reloading disk '/dev/sdg' partition table. Jul 9 15:17:03 ecba unassigned.devices: Reload partition table result: /dev/sdg: re-reading partition table Jul 9 15:17:04 ecba unassigned.devices: Creating Unraid compatible gpt partition on disk '/dev/sdg'. Jul 9 15:17:06 ecba kernel: sdg: sdg1 Jul 9 15:17:06 ecba unassigned.devices: Create gpt partition table result: Creating new GPT entries in memory. The operation has completed successfully. Jul 9 15:17:08 ecba unassigned.devices: Reloading disk '/dev/sdg' partition table. Jul 9 15:17:08 ecba kernel: sdg: sdg1 Jul 9 15:17:08 ecba unassigned.devices: Reload partition table result: /dev/sdg: re-reading partition table Jul 9 15:17:08 ecba unassigned.devices: Formatting disk '/dev/sdg' with 'xfs' filesystem. Jul 9 15:17:08 ecba unassigned.devices: Format drive command: /sbin/mkfs.xfs -f '/dev/sdg1' 2>&1 Jul 9 15:17:20 ecba unassigned.devices: Format disk '/dev/sdg' with 'xfs' filesystem: meta-data=/dev/sdg1 isize=512 agcount=8, agsize=268435455 blks = sectsz=4096 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=1 = reflink=1 bigtime=1 inobtcount=1 nrext64=1 data = bsize=4096 blocks=1953506633, imaxpct=5 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 blocks=521728, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 Jul 9 15:17:23 ecba unassigned.devices: Reloading disk '/dev/sdg' partition table. Jul 9 15:17:23 ecba kernel: sdg: sdg1 Jul 9 15:17:23 ecba unassigned.devices: Reload partition table result: /dev/sdg: re-reading partition table I have never experienced this in 6.12 but it's been a while since I've needed to format a SAS drive. I may downgrade back just to make sure it does work. Any assistance would be appreciated. Also there are no SMART errors on these drives before or after. The only other thing I do is enable write caching on these drives via user script (at array start), but this happens whether or not that has run. ecba-diagnostics-20240709-1532.zip
-
You have too much memory. But seriously I too have 128GB and encounter similar issues and I believe the only fix is to adjust the write caching to ram. I have been searching this issue for weeks and lots of people have similar issues and the only resolution seems to be the following: Download the tips and tweaks plugin and adjust down the vm.dirty_ratio and vm.dirty_background_ratio values. Or set those values yourself from the command line. I need to do some more indepth testing but I currently have mine set to 4 and 2. I haven't fully tested if it fixes my specific symptoms when running the mover since I have undertaken other migitations to resolve them but I do know I am seeing similar I/O waits with the new values too. If you want to find previous topics search for 'io wait' or 'dirty_ratio' but here is the one in particular that led me to this fix:
-
In my ideal world it would work how I initially thought zpool and zfs worked. I love the concept of vdevs and in that context it makes total sense why drives need to be the same. Before further reading I thought with redundancy happening at that level, then you could freely mix and match different vdev topologies in a zpool to combine them together. Not only can you not (or at least it's very much not recommended), but I came to realize that losing one vdev loses all of the data. That freaks me out. I think it would be great to have a pool that can stripe files across many vdevs, then you could control at the share level where data can be will be placed. For example, if you had an array consisting of a 6-wide RAID-Z2 and 4-wide RAID-Z1 you could mark the 'photos' share to only or prefer the Z2, but 'media' could go anywhere similar to how we have include/exclude disk options.
-
I am assuming the write cache checking is done using hdparm. Would it be possible to use smartctl instead so that SAS drives can also be checked? Not sure what the equivalent sdparm command is, if there even is one, but this one below works for both SATA and SAS drives. smartctl -g wcache /dev/... I had run into write issues for the last couple of days and I finally narrowed it down to only 3 out of my 6 SAS drives have writeback cache enabled by default. I had no idea. Simple add to my write cache enabling script but I just assumed they were all enabled by default if it wasn't reported by this plugin. From the man page for smartctl: -g NAME, --get=NAME, -s NAME[,VALUE], --set=NAME[,VALUE] wcache[,on|off] - [ATA] Gets/sets the volatile write cache fea- ture (if supported). The write cache is usually enabled by de- fault. wcache[,on|off] - [SCSI] Gets/sets the 'Write Cache Enable' (WCE) bit (if supported). The write cache is usually enabled by default.
-
I am encountering this problem and I feel like it's only just started happening today. The SMB share from my unraid box is mounted to a linux machine connected over a 2.5GBe link. As soon as I invoke the mover it totally distrupts that mount. Doing a simple list just hangs indefinitely. If I stop the mover and wait ~5 minutes it becomes responsive again. I just switched the mount to nfs4 (which I wanted to do anyway) and I am not experiencing these same issues. EDIT: Nevermind it's happening with the NFS shares as well. A little while after turning off the mover it magically comes back to life. Aug 12 17:26:56 sjr-plex kernel: nfs: server ecba.local OK Aug 12 17:26:56 sjr-plex kernel: nfs: server ecba.local OK Aug 12 17:26:41 sjr-plex kernel: nfs: server ecba.local not responding, still trying Aug 12 17:26:41 sjr-plex kernel: nfs: server ecba.local not responding, still trying Aug 12 17:25:57 sjr-plex kernel: nfs: server ecba.local not responding, still trying Aug 12 17:25:33 sjr-plex kernel: nfs: server ecba.local not responding, still trying Aug 12 17:21:41 sjr-plex kernel: nfs: server ecba.local not responding, still trying
-
In one of those threads I linked someone tried without plugins and it still happened. I never bothered to verify so I can't say either way for sure but it would honestly make the most sense if it were the ZFS Master or some other plugin.
-
That may have been my comment. The TL;DR I came away with was that it was temporary files being written to these disks and that it was not coming from any plugin (maybe). I didn't want to deal with it so I reformatted them, but yes in the few days that I them in the array they never spun down. It was very tiny writes happening every 10-15 seconds on each of the ZFS disks in the array. Here is some further discussions of the same issues I experienced:
